No such table: messages (Edit: Support for WhatsApp databases with "message" table) #9

Takepy · 2022-02-18T13:40:05Z

When running the script I get following error message

Gathering contacts...(97)
Traceback (most recent call last):
  File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\******\AppData\Local\Programs\Python\Python310\Scripts\wtsexporter.exe\__main__.py", line 7, in <module>
  File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\__main__.py", line 147, in main
    messages(db, data)
  File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\extract.py", line 88, in messages
    c.execute("""SELECT count() FROM messages""")
sqlite3.OperationalError: no such table: messages

My database doesn't contain the table messages, I only have one called message which contains the data but the schema of the table looks different from what it's supposed to look like.

What could be the issue here?

The text was updated successfully, but these errors were encountered:

KnugiHK · 2022-02-18T13:53:33Z

What is the WhatsApp version and which platform (Android/iOS) are you using?

Takepy · 2022-02-18T13:59:53Z

I'm using Android.
WhatsApp versions that have been used are 2.22.3.77 and 2.22.5.7 (Beta)
2.22.3.77 was running on an emulator so that I could pull the /com.whatsapp/ directory.
On the 2.22.5.7 version I used the previously pulled key to decrypt the db.

KnugiHK · 2022-02-18T14:04:41Z

Are both versions and environments result in same database schema?

Takepy · 2022-02-18T14:05:39Z

Yes, they both resulted in the same database schema

KnugiHK · 2022-02-18T14:08:47Z

Interesting. I will try to reproduce it with 2.22.3.77 on emulator.

KnugiHK · 2022-02-19T04:11:56Z

I tried to login WhatsApp with 2.22.3.77 on emulator. However, the database includes the "messages" table. Do you remember when you installed WhatsApp on the emulator for the first time (i.e. creation date of the database)?

Takepy · 2022-02-19T13:47:17Z

I installed WhatsApp on the emulator on the 9th of this month and then imported my chat history using the chat backup in WhatsApp.
On my actual phone, WhatsApp has been installed like 6 months ago.

KnugiHK · 2022-02-21T05:11:45Z

I checked some similar projects and suspect that "message" table is a legacy table for storing messages. On your phone, is the WhatsApp conversation restored from old backup after installation?

Takepy · 2022-02-21T12:49:13Z

I restored the WhatsApp conversations from Google Drive (doing weekly backups).
Never deleted my WhatsApp chat history though so the messages go all the way back to 2013

KnugiHK · 2022-02-21T15:05:54Z

As far as I observed, it is possible that "message" was used in the early years of WhatsApp. Could you check if new installation without backup will result the same situation?

Takepy · 2022-02-21T15:11:20Z

I'll give it a try later and report back.

Takepy · 2022-02-22T18:41:53Z

Sorry, it took a little bit longer until I had time to try it out.
Setup a new emulator, installed WhatsApp and didn't import my backup.
The table messages exists now.

So I'm guessing my backup is just too old for it to work then? :(

KnugiHK · 2022-02-23T05:24:57Z

Sorry, it took a little bit longer until I had time to try it out.

No problem!

So I'm guessing my backup is just too old for it to work then? :(

I think so.
I would like to add support for older version. However, I don't have such database, hence, I am not able to do it. If you have time and you want the script to work with your database, you can take a look inside the database and fine tune the script.
Anyway, I will keep my eyes on the opportunity for supporting old databases. Thanks for your report!

Unbrick · 2022-04-06T09:45:48Z

Hey everyone! I just stumbled upon the same issue with WhatsApp version 2.22.7.73 on Android on two different devices using two different phone numbers without restoring any backup.

I'll try to reinstall WhatsApp and try it again, but it seems like the database scheme is changing. The new messages table has the following schema:

CREATE TABLE message (
  _id INTEGER PRIMARY KEY AUTOINCREMENT, 
  chat_row_id INTEGER NOT NULL, from_me INTEGER NOT NULL, 
  key_id TEXT NOT NULL, sender_jid_row_id INTEGER, 
  status INTEGER, broadcast INTEGER, 
  recipient_count INTEGER, participant_hash TEXT, 
  origination_flags INTEGER, origin INTEGER, 
  timestamp INTEGER, received_timestamp INTEGER, 
  receipt_server_timestamp INTEGER, 
  message_type INTEGER, text_data TEXT, 
  starred INTEGER, lookup_tables INTEGER, 
  message_add_on_flags INTEGER, sort_id INTEGER NOT NULL DEFAULT 0
);

Edit: Tried a second time, database scheme is still the same.

Edit2: In WhatsApp version 2.21.21.17 both tables messages and message do seem to exist.

KnugiHK · 2022-04-06T15:43:29Z

I'll try to reinstall WhatsApp and try it again, but it seems like the database scheme is changing. The new messages table has the following schema:

I can produce the same result on 2.22.7.73 and "messages" table no longer exist.

@Takepy As I can reproduce your situation now, I will work on supporting "message" table.

KnugiHK · 2022-04-06T15:46:44Z

I am curious about how WhatsApp decides which table to store messages 🤨. Is "message" a legacy table? Or "messages" is the legacy one?

Unbrick · 2022-04-06T16:03:43Z

I am curious about how WhatsApp decides which table to store messages 🤨. Is "message" a legacy table? Or "messages" is the legacy one?

I'm not certain, using the outdated WhatsApp version both tables exist, in the more recent version (downloaded from whatsapp.com yesterday) only the message table existed.

Edit: Just checked, in my current setup (outdated version) messages are only stored in the messages table (which works awesome with your script!) and the message table appears to be empty.

KnugiHK · 2022-04-06T17:18:25Z

WhatsApp using "message" table spread different types of message to many other tables. Good chance for me to rewrite the project :)

lordfeck · 2023-03-01T20:30:52Z

I checked out the message_table branch and used it to extract my newer 'message' DB successfully. All conversations including group conversations were converted to HTML. Many thanks @KnugiHK !

nahoj · 2023-03-17T10:45:35Z

I had to cherry-pick the fix for #32 and then it worked perfectly. Thank you for making this tool!

➜  Whatsapp-Chat-Exporter git:(message_table)$ git cherry-pick 2ba5571
➜  Whatsapp-Chat-Exporter git:(message_table)$ pip install '.[android_backup]'

KnugiHK · 2023-03-25T04:02:42Z

Hello everyone! We are finally moving to the last step before releasing this update.

I merged the logic for processing old and new schema in extract.py in commit fb88124. As it is not a commit specifically for the message table, it will not be merged into the message_table branch, and the branch will not be updated from now on. And changes will be committed to the dev branch as usual.

Feel free to try it, especially for those with databases with both message and messages tables, to see if there is any compatibility issue.

umjediqualquer · 2023-04-08T17:42:06Z

Hey there! Not sure if it's related, but I was having the same issue related to this thread, downloaded both the commit and the dev branch versions and now I'm getting a different error:

Traceback (most recent call last):
  File "/home/user/.local/bin/wtsexporter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/__main__.py", line 229, in main
    messages(db, data)
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py", line 249, in messages
    data[content["key_remote_jid"]].add_message(content["_id"], Message(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'add_message'

Again, not sure if it's related, but what should I do in this case, please?

KnugiHK · 2023-04-08T17:45:10Z

Hey there! Not sure if it's related, but I was having the same issue related to this thread, downloaded both the commit and the dev branch versions and now I'm getting a different error:
Traceback (most recent call last):
  File "/home/user/.local/bin/wtsexporter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/__main__.py", line 229, in main
    messages(db, data)
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py", line 249, in messages
    data[content["key_remote_jid"]].add_message(content["_id"], Message(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'add_message'
Again, not sure if it's related, but what should I do in this case, please?

@umjediqualquer
What's the command you issued and does wa.db exists?

umjediqualquer · 2023-04-08T18:03:23Z

Hiya there @KnugiHK, I ran wtsexporter -a. I have msgstore.db, wa.db and the WhatsApp folder (containing the Backups, Databases and Media folders) in the directory I tried to run the command.

KnugiHK · 2023-04-08T18:07:09Z

Hiya there @KnugiHK, I ran wtsexporter -a. I have msgstore.db, wa.db and the WhatsApp folder (containing the Backups, Databases and Media folders) in the directory I tried to run the command.

@umjediqualquer I did a quick fix on it (b371587) in the dev branch. Would you please check if the fix works?

umjediqualquer · 2023-04-08T18:14:31Z

@umjediqualquer I did a quick fix on it (b371587) in the dev branch. Would you please check if the fix works?

Managed to make some progress, getting a different error message now:

Gathering messages...(694764/694764)
Gathering media...(43618/43618)
Traceback (most recent call last):
  File "/home/user/.local/bin/wtsexporter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/__main__.py", line 231, in main
    vcard(db, data)
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py", line 491, in vcard
    file_name = "".join(x for x in media_name if x.isalnum())
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable```

KnugiHK · 2023-04-08T18:17:22Z

@umjediqualquer I did a quick fix on it (b371587) in the dev branch. Would you please check if the fix works?

Managed to make some progress, getting a different error message now:

Gathering messages...(694764/694764)
Gathering media...(43618/43618)
Traceback (most recent call last):
  File "/home/user/.local/bin/wtsexporter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/__main__.py", line 231, in main
    vcard(db, data)
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py", line 491, in vcard
    file_name = "".join(x for x in media_name if x.isalnum())
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable```

Do you know if your msgstore.db use message or messages table?

umjediqualquer · 2023-04-08T18:20:18Z

Do you know if your msgstore.db use message or messages table?

I'm assuming it's message because when running the release version, I get the sqlite3.OperationalError: no such table: messages error, but I can open this with a db viewer to double check.

KnugiHK · 2023-04-08T18:27:15Z

Do you know if your msgstore.db use message or messages table?

I'm assuming it's message because when running the release version, I get the sqlite3.OperationalError: no such table: messages error, but I can open this with a db viewer to double check.

Attempt on fixing it is pushed (06a1d34). See if that works (the error may disappear but it is not a good fix IMO).

umjediqualquer · 2023-04-08T18:31:27Z

Attempt on fixing it is pushed (06a1d34). See if that works (the error may disappear but it is not a good fix IMO).

The previous error has been fixed, but it threw a new error a few lines after:

Gathering messages...(694764/694764)
Gathering media...(43618/43618)
Traceback (most recent call last):
  File "/home/user/.local/bin/wtsexporter", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/__main__.py", line 231, in main
    vcard(db, data)
  File "/home/user/.local/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py", line 496, in vcard
    data[row["key_remote_jid"]].messages[row["message_row_id"]].data = media_name + \
                                                                       ^^^^^^^^^^^^^^
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Is there a way to extract only the chat without the media? It's weekend and I don't to keep bothering you with this haha

KnugiHK · 2023-04-08T18:43:21Z

Is there a way to extract only the chat without the media?

There is no way to extract chat only. Maybe it is a good option to add.

It's weekend and I don't to keep bothering you with this haha

That's fine. The bug has to be fixed anyway. And I reply only when I am available.

As for the bug, I committed another attempt to fix it (7c0b90d). This time I modified my database to test the problem myself and it works for me. Also, maybe the file name should rely on the information inside the vcards instead of the message data (the current implementation).

umjediqualquer · 2023-04-08T19:02:23Z

@KnugiHK it worked! Seems to have generated everything correctly and didn't throw any errors :D

Thank you very much for your help, especially on a weekend! May I buy you a cup of coffee (or tea, it that's your cup, heh) as a token of gratitude?

KnugiHK · 2023-04-09T09:15:08Z

@KnugiHK it worked! Seems to have generated everything correctly and didn't throw any errors :D

Thank you very much for your help, especially on a weekend! May I buy you a cup of coffee (or tea, it that's your cup, heh) as a token of gratitude?

Glad that works for you too! And thanks for your report! The point for not releasing this update yet is to let everyone test it and spot problems.

Also, I appreciate your offer to buy me a cup of coffee or tea, but it's not necessary. If you'd like to show your support, consider staring this repo instead. That meant a lot to me😆. Thanks again.

Signum21 · 2023-04-17T22:53:37Z

Hi, I cloned the "message_table" branch and run the tool with this command:
python3 __main__.py -a --db ./msgstore.db

The messages are recovered correctly but i noticed that two identical "WhatsApp" folders are created.
One outside and one inside the "result" folder.
This might be a bug.

The two folders structure are:

./WhatsApp/
./WhatsApp/vCards/

./result/
./result/messages.html
./result/WhatsApp/
./result/WhatsApp/vCards/

KnugiHK · 2023-04-18T14:32:25Z

Hi, I cloned the "message_table" branch and run the tool with this command: python3 main.py -a --db ./msgstore.db

The messages are recovered correctly but i noticed that two identical "WhatsApp" folders are created. One outside and one inside the "result" folder. This might be a bug.

The two folders structure are:

./WhatsApp/ ./WhatsApp/vCards/

./result/ ./result/messages.html ./result/WhatsApp/ ./result/WhatsApp/vCards/

@Signum21
Did "Copying media directory..." printed in your terminal? I guess your problem is related to #25.

Also, "message_table" branch is no longer maintained. Try dev branch instead.

Signum21 · 2023-04-18T16:19:48Z

I switched to dev branch.
"Copying media directory..." printed in the terminal.
I think the problem is not related to #25 cause I don't have the media folder so it's not a problem of moving instead of copying, it's creating two copies of the same folder from zero.
I'm working only with the msgstore.db file.

KnugiHK · 2023-04-18T17:12:50Z

I switched to dev branch. "Copying media directory..." printed in the terminal. I think the problem is not related to #25 cause I don't have the media folder so it's not a problem of moving instead of copying, it's creating two copies of the same folder from zero. I'm working only with the msgstore.db file.

vCards will be exported to ./WhatsApp/vCards and copied to the output directory by default. If you don't want two identical folders, try -c (--move-media) flag.

Related to #39 and #9

KnugiHK · 2023-04-25T05:07:19Z

The output treats deleted message as not supported message in the new schema. It is fixed in c7a01bb.

UsamaAshfaq · 2023-05-08T10:51:35Z

WhatsApp using "message" table spread different types of message to many other tables. Good chance for me to rewrite the project :)

Kindly do it, I am also getting same error : sqlite3.OperationalError: no such table: messages
In my case Messages are stored in "message" table.

KnugiHK · 2023-05-08T17:54:55Z

WhatsApp using "message" table spread different types of message to many other tables. Good chance for me to rewrite the project :)

Kindly do it, I am also getting same error : sqlite3.OperationalError: no such table: messages In my case Messages are stored in "message" table.

The support of the message table is completed and will be released soon.

KnugiHK · 2023-05-16T13:00:17Z

Support with new schema has been released in 0.9.0. I am closing this issue. Further problems should be reported on a dedicated issue. Thanks to everyone who contributed to the support of the new schema!

KnugiHK/WhatsApp-Chat-Exporter#9

KnugiHK added bug Something isn't working help wanted Extra attention is needed labels Feb 18, 2022

KnugiHK added enhancement New feature or request and removed bug Something isn't working labels Feb 23, 2022

KnugiHK changed the title ~~no such table: messages~~ No such table: messages Feb 23, 2022

KnugiHK changed the title ~~No such table: messages~~ No such table: messages (Old WhatsApp database not supported) Feb 23, 2022

KnugiHK changed the title ~~No such table: messages (Old WhatsApp database not supported)~~ No such table: messages (Old WhatsApp databases are not supported) Feb 23, 2022

KnugiHK changed the title ~~No such table: messages (Old WhatsApp databases are not supported)~~ No such table: messages (Edit: old WhatsApp databases are not supported) Feb 23, 2022

KnugiHK removed the help wanted Extra attention is needed label Feb 24, 2022

KnugiHK self-assigned this Apr 6, 2022

KnugiHK pinned this issue Apr 25, 2022

KnugiHK changed the title ~~No such table: messages (Edit: old WhatsApp databases are not supported)~~ No such table: messages (Edit: WhatsApp databases with "message" table are not supported) Apr 25, 2022

KnugiHK mentioned this issue Mar 25, 2023

TypeError: Object of type ChatStore is not JSON serializable #37

Closed

AlexanderRitter02 mentioned this issue Apr 24, 2023

Provide reasons for "message not supported" #39

Closed

KnugiHK added a commit that referenced this issue Apr 25, 2023

Handle deleted message in new schema

c7a01bb

Related to #39 and #9

KnugiHK mentioned this issue May 6, 2023

No such table messages #40

Closed

KnugiHK closed this as completed May 16, 2023

KnugiHK removed the priority Priority to be fixed or implemented label May 16, 2023

Goblincomet added a commit to Goblincomet/WhatsApp-Chat-Exporter that referenced this issue Sep 15, 2023

Support new WhatsApp database schema

cab4898

KnugiHK/WhatsApp-Chat-Exporter#9

No such table: messages (Edit: Support for WhatsApp databases with "message" table) #9

No such table: messages (Edit: Support for WhatsApp databases with "message" table) #9

Comments

Takepy commented Feb 18, 2022

KnugiHK commented Feb 18, 2022

Takepy commented Feb 18, 2022

KnugiHK commented Feb 18, 2022 • edited

Takepy commented Feb 18, 2022

KnugiHK commented Feb 18, 2022

KnugiHK commented Feb 19, 2022

Takepy commented Feb 19, 2022

KnugiHK commented Feb 21, 2022 • edited

Takepy commented Feb 21, 2022

KnugiHK commented Feb 21, 2022

Takepy commented Feb 21, 2022

Takepy commented Feb 22, 2022

KnugiHK commented Feb 23, 2022

Unbrick commented Apr 6, 2022 • edited

KnugiHK commented Apr 6, 2022 • edited

KnugiHK commented Apr 6, 2022 • edited

Unbrick commented Apr 6, 2022 • edited

KnugiHK commented Apr 6, 2022

lordfeck commented Mar 1, 2023

nahoj commented Mar 17, 2023

KnugiHK commented Mar 25, 2023 • edited

umjediqualquer commented Apr 8, 2023 • edited by KnugiHK

KnugiHK commented Apr 8, 2023 • edited

umjediqualquer commented Apr 8, 2023

KnugiHK commented Apr 8, 2023 • edited

umjediqualquer commented Apr 8, 2023

KnugiHK commented Apr 8, 2023

umjediqualquer commented Apr 8, 2023

KnugiHK commented Apr 8, 2023

umjediqualquer commented Apr 8, 2023 • edited by KnugiHK

KnugiHK commented Apr 8, 2023 • edited

umjediqualquer commented Apr 8, 2023

KnugiHK commented Apr 9, 2023

Signum21 commented Apr 17, 2023 • edited

KnugiHK commented Apr 18, 2023 • edited

Signum21 commented Apr 18, 2023 • edited

KnugiHK commented Apr 18, 2023

KnugiHK commented Apr 25, 2023

UsamaAshfaq commented May 8, 2023

KnugiHK commented May 8, 2023

KnugiHK commented May 16, 2023

KnugiHK commented Feb 18, 2022 •

edited

KnugiHK commented Feb 21, 2022 •

edited

Unbrick commented Apr 6, 2022 •

edited

KnugiHK commented Apr 6, 2022 •

edited

KnugiHK commented Apr 6, 2022 •

edited

Unbrick commented Apr 6, 2022 •

edited

KnugiHK commented Mar 25, 2023 •

edited

umjediqualquer commented Apr 8, 2023 •

edited by KnugiHK

KnugiHK commented Apr 8, 2023 •

edited

KnugiHK commented Apr 8, 2023 •

edited

umjediqualquer commented Apr 8, 2023 •

edited by KnugiHK

KnugiHK commented Apr 8, 2023 •

edited

Signum21 commented Apr 17, 2023 •

edited

KnugiHK commented Apr 18, 2023 •

edited

Signum21 commented Apr 18, 2023 •

edited