KeyError: 1152562 Dev Branch

[Stratos-Crimson]

Hello,
I tried to use the dev branch to export so that I could bypass the #44 issue but now I have this.
I installed using pip install git+https://github.com/KnugiHK/Whatsapp-Chat-Exporter.git@dev.

Traceback (most recent call last):
  File "/home/user/.local/bin/wtsexporter", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.9/site-packages/Whatsapp_Chat_Exporter/__main__.py", line 241, in main
    media(db, data, args.media)
  File "/home/user/.local/lib/python3.9/site-packages/Whatsapp_Chat_Exporter/extract.py", line 439, in media
    data[content["key_remote_jid"]].messages[content["message_row_id"]].media = True
KeyError: 1152562

[Stratos-Crimson]

@KnugiHK ?

[KnugiHK]

Can you confirm if there is a row in message table with "1152562" in _id field?

[Stratos-Crimson]

Yes there is.

[Stratos-Crimson]

It's NULL though.

[KnugiHK]

If it is NULL it cannot be 1152562.

[Stratos-Crimson]

You misunderstand, there is a row with that id but the message content is NULL

[Stratos-Crimson]

Perhaps message contents that are NULL should simply be ignored just like non UTF-8 weirdness?

[KnugiHK]

A NULL message doesn't necessarily be invalid (e.g., the row represents other metadata or media). It shouldn't be ignored.

[Stratos-Crimson]

Then why did It error out on that?

[KnugiHK]

The corresponding message entry didn't get added into the chat store probably.

[Stratos-Crimson]

So perhaps the ones that error out like this can be skipped too instead of stopping the whole process?

[Stratos-Crimson]

Is there a way to detect If something didn't get added to the chat store? If so, then you could just skip ones that didn't.

[KnugiHK]

Could you provide me the outputs of SELECT * FROM message WHERE _id=1152562; and SELECT * FROM message_media WHERE message_row_id=1152562? I would like to further investigate the problem before I skipping it.

[Stratos-Crimson]

Can I just send you the row again? This didn't work out well last time.

[KnugiHK]

Sure.

[Stratos-Crimson]

Done

[KnugiHK]

I couldn't reproduce the bug after I inserted the rows you sent to me. What's the output of the following SQL?

SELECT jid.raw_string as key_remote_jid,
       message_row_id,
       file_path,
       message_url,
       mime_type,
       media_key,
       file_hash,
       thumbnail
FROM message_media
INNER JOIN message
    ON message_media.message_row_id = message._id
LEFT JOIN chat
    ON chat._id = message.chat_row_id
INNER JOIN jid
    ON jid._id = chat.jid_row_id
LEFT JOIN media_hash_thumbnail
    ON message_media.file_hash = media_hash_thumbnail.media_hash
WHERE message_row_id = 1152562 OR key_remote_jid = 1152562
ORDER BY jid.raw_string ASC

[KnugiHK]

Since 3ed269e, the invalid flag is removed, and all possible messages are included in the output. You can try to re-install and re-run the exporter from dev to see if the problem persists.

[Stratos-Crimson]

It's persisting

[Stratos-Crimson]

I got a lot of gibberish but I did get audio/mp4 with this "|" around It. The quotes weren't there, of course.

I couldn't reproduce the bug after I inserted the rows you sent to me. What's the output of the following SQL?

SELECT jid.raw_string as key_remote_jid,
       message_row_id,
       file_path,
       message_url,
       mime_type,
       media_key,
       file_hash,
       thumbnail
FROM message_media
INNER JOIN message
    ON message_media.message_row_id = message._id
LEFT JOIN chat
    ON chat._id = message.chat_row_id
INNER JOIN jid
    ON jid._id = chat.jid_row_id
LEFT JOIN media_hash_thumbnail
    ON message_media.file_hash = media_hash_thumbnail.media_hash
WHERE message_row_id = 1152562 OR key_remote_jid = 1152562
ORDER BY jid.raw_string ASC

[Stratos-Crimson]

@KnugiHK ?

[KnugiHK]

Could you show me the output?

[Stratos-Crimson]

Sure and sorry for taking a while.
�B.�Q�i)9~�23G�|H07O6FjcvxRvp+M1OKqJyqIG5ulpIGlfYDjv6SQA8T0=|/AvdxZMCJ2RXhBpJLY2IDedZ9ZaYfsqTWmSguEZFGaC6b.enc|audio/mp4|�_A6��k#�[3+�

[qosch]

Had the same issue with latest main branch, but dev worked.

[Stratos-Crimson]

I will try again then.