Crash when content id is not found #61

andrp92 · 2023-08-18T06:31:30Z

When you have Whatsapp and migrate to Whatsapp business or the opposite, some records will lose its ids and python doesn't like it here.

Traceback (most recent call last):65)
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/runpy.py", line 198, in _run_module_as_main return _run_code(code, main_globals, None,^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/runpy.py", line 88, in _run_code exec(code, run_globals)
File "/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module> cli.main()
File "/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run()
File "/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="__main__")
File "/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals,
File "/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals)
File "/WhatsApp-Chat-Exporter/Whatsapp_Chat_Exporter/__main__.py", line 382, in <module> main() File "/Whatsapp_Chat_Exporter/__main__.py", line 305, in main messages(db, data, args.media)
File "/dev/itunes-backup/.venv/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract_iphone.py", line 93, in messages path = f'{media_folder}/Media/Profile/{_id.split("@")[0]}' ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

The text was updated successfully, but these errors were encountered:

KnugiHK · 2023-10-01T13:31:36Z

Hi. Could you post the value of _id before the program crash?

LoSunny · 2023-11-28T13:08:20Z

Sorry for overtaking this issue, but I have just tried and face the same bug.
Device: iOS
WhatsApp / Business: WhatsApp

The _id is indeed NULL for some reason, I have opened an sql editor and try to run the command, adding the WHERE _id is NULL clause to check if it is an issue with the python sqlite reader or the query statement.
Here is a photo for reference: https://imgur.com/a/tYDeXLm

The issue seems to be also affecting the media function too, apart from the messages function andrp92 faced above

To fix the issue temporary, I have currently added the WHERE clause to both statement, i.e.
Messages query

    c.execute("""SELECT COALESCE(ZFROMJID, ZTOJID) as _id,
                        ZWAMESSAGE.Z_PK,
                        ZISFROMME,
                        ZMESSAGEDATE,
                        ZTEXT,
                        ZMESSAGETYPE,
                        ZWAGROUPMEMBER.ZMEMBERJID,
						ZMETADATA,
                        ZSTANZAID
                 FROM ZWAMESSAGE
                    LEFT JOIN ZWAGROUPMEMBER
                        ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
					LEFT JOIN ZWAMEDIAITEM
						ON ZWAMESSAGE.Z_PK = ZWAMEDIAITEM.ZMESSAGE
                 WHERE _id is NOT NULL;""")

Media query

    c.execute("""SELECT COALESCE(ZWAMESSAGE.ZFROMJID, ZWAMESSAGE.ZTOJID) as _id,
                        ZMESSAGE,
                        ZMEDIALOCALPATH,
                        ZMEDIAURL,
                        ZVCARDSTRING,
                        ZMEDIAKEY,
                        ZTITLE
                 FROM ZWAMEDIAITEM
                    INNER JOIN ZWAMESSAGE
                        ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
                 WHERE ZMEDIALOCALPATH IS NOT NULL AND _id is NOT NULL
                 ORDER BY _id ASC""")

However, I could not verify if it causes any corruption due to my message history being insanely huge

(venv) sunnylo@SunnyLos-Laptop % ./venv/bin/wtsexporter -i -b ~/Library/Application\ Support/MobileSync/Backup/asdf
WhatsApp directory already exists, skipping WhatsApp file extraction.
Processing contacts...(548)
Processing messages...(245452/245452)
Processing media...(128019/128019)
Processing vCards...(51/51)
Generating chats...(564/564)
Copying media directory...

Everything is done!

Hope it will be fixed soon, thanks for the great library, love it

KnugiHK · 2023-11-28T16:47:23Z

Hi @LoSunny. Thanks for your workaround. Since _id served as an identifier for each chat, it is crucial for the whole operation. Therefore, your workaround will most likely be accepted as a fix unless a better solution pops up. I will work on it soon.

This fix also changed how to determine if the message belongs to a group or PM.

KnugiHK · 2023-12-03T02:31:40Z

Good news! A more reliable way to determine which chat the message belongs to is shipped with the latest fix, 3847836.
@LoSunny @andrp92
Can you try the fix in dev branch and see if it works as expected?

LoSunny · 2023-12-03T03:16:42Z

Yep it "works"

sunnylo@SunnyLos-Laptop WhatsApp-Chat-Exporter % wtsexporter -i -o result2 -b ~/Library/Application\ Support/MobileSync/Backup/asdf
WhatsApp directory already exists, skipping WhatsApp file extraction.
Processing contacts...(548)
Processing messages...(245452/245452)
Processing media...(128019/128019)
Processing vCards...(51/51)
Generating chats...(548/548)
Copying media directory...

Everything is done!

However, it seems to have generated less chats than before, from 564 to 548? I have ran a diff to see the file difference

(part of the result)
Only in result: 1545670533.html
Only in result: 1620178621.html
Only in result: 1628765705.html
Only in result: 1630636817.html
Only in result: 1630920124.html
Only in result: 1634260290.html
Only in result: 1635758267.html
Only in result: 1639734958.html
Only in result: 1641020304.html
Only in result: 1648629655.html
Only in result: 1661274036.html
Only in result: 1661601743.html
Only in result: 1662791013.html
Only in result: 1671960498.html
Only in result: 1675172235.html
Only in result: status-.html

On the other hand, the dev branch seems to have solve some of the errors that displayed before when I ran a diff -r command

$ diff -r result result2
161c195
< 									<p>Not supported WhatsApp internal message</p>
---
> 									<p>The group name changed to asdf</p>

KnugiHK · 2023-12-03T03:41:20Z

However, it seems to have generated less chats than before, from 564 to 548?

I believe 548 is the correct number. AFAIK, each message should be mapped to the table ZWACHATSESSION by the column ZWAMESSAGE.ZCHATSESSION. Hence, it should be accurate. You can see if messages in chats that only appear before the fix are exported into another HTML file. (sor for 1999)

On the other hand, the dev branch seems to have solve some of the errors that displayed before when I ran a diff -r command

Well😄That's unintended.

LoSunny · 2023-12-03T03:55:50Z

I believe 548 is the correct number. AFAIK, each message should be mapped to the table ZWACHATSESSION by the column ZWAMESSAGE.ZCHATSESSION. Hence, it should be accurate. You can see if messages in chats that only appear before the fix are exported into another HTML file. (sor for 1999)

After I double check with the messages, it indeed exists in other group chat, so yeah it should be correct now. hou yea 🤩

KnugiHK · 2023-12-03T03:57:56Z

For example:

Assuming the user 85200000000@s.whatsapp.net created a broadcast channel 1111111111@broadcast and you were being added to the channel. And you do not have any PM message with 85200000000@s.whatsapp.net.

Before the fix:

Message sent by 85200000000@s.whatsapp.net to the broadcast channel 1111111111@broadcast will probably appear in the chat HTML for 85200000000@s.whatsapp.net, since 85200000000@s.whatsapp.net is the chat identifier. Therefore, chat HTML for 85200000000@s.whatsapp.net is generated.

After the fix:

Message sent by 85200000000@s.whatsapp.net to the broadcast channel 1111111111@broadcast will be included in the chat HTML for 1111111111@broadcast, which is the expected and correct behaviour. Therefore, chat HTML for 85200000000@s.whatsapp.net is not generated.

LoSunny · 2023-12-03T04:10:15Z

okie. learn a lot of new stuff tdy.
Btw, for both versions the timestamp shown in the HTML seems to be mostly the same for the same date. I.e. the hour:minute shown for both pm and grp chat will be the same for the same date. Is it a bug?

KnugiHK · 2023-12-03T05:12:41Z

okie. learn a lot of new stuff tdy. Btw, for both versions the timestamp shown in the HTML seems to be mostly the same for the same date. I.e. the hour:minute shown for both pm and grp chat will be the same for the same date. Is it a bug?

If it is related to #64, could you provide some more details & screenshots to that issue? Also, as I mentioned in another issue, I don't currently have access to the latest iOS WhatsApp, so its difficult for me to reproduce the problem and debug that.

KnugiHK · 2023-12-03T05:13:34Z

I will leave this open until next release.

KnugiHK added the bug Something isn't working label Aug 18, 2023

KnugiHK added the inactive Waiting for more information label Oct 23, 2023

KnugiHK removed the inactive Waiting for more information label Nov 28, 2023

KnugiHK added a commit that referenced this issue Dec 3, 2023

Fix: use a more reliable way to determine chat #61

3847836

This fix also changed how to determine if the message belongs to a group or PM.

KnugiHK self-assigned this Dec 3, 2023

KnugiHK added the pending for release Feature or bug fix to be released label Dec 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash when content id is not found #61

Crash when content id is not found #61

andrp92 commented Aug 18, 2023 •

edited by KnugiHK

KnugiHK commented Oct 1, 2023

LoSunny commented Nov 28, 2023

KnugiHK commented Nov 28, 2023 •

edited

KnugiHK commented Dec 3, 2023

LoSunny commented Dec 3, 2023

KnugiHK commented Dec 3, 2023

LoSunny commented Dec 3, 2023

KnugiHK commented Dec 3, 2023 •

edited

LoSunny commented Dec 3, 2023

KnugiHK commented Dec 3, 2023

KnugiHK commented Dec 3, 2023

Crash when content id is not found #61

Crash when content id is not found #61

Comments

andrp92 commented Aug 18, 2023 • edited by KnugiHK

KnugiHK commented Oct 1, 2023

LoSunny commented Nov 28, 2023

KnugiHK commented Nov 28, 2023 • edited

KnugiHK commented Dec 3, 2023

LoSunny commented Dec 3, 2023

KnugiHK commented Dec 3, 2023

LoSunny commented Dec 3, 2023

KnugiHK commented Dec 3, 2023 • edited

For example:

Before the fix:

After the fix:

LoSunny commented Dec 3, 2023

KnugiHK commented Dec 3, 2023

KnugiHK commented Dec 3, 2023

andrp92 commented Aug 18, 2023 •

edited by KnugiHK

KnugiHK commented Nov 28, 2023 •

edited

KnugiHK commented Dec 3, 2023 •

edited