If you ever lost your phone due to an accident, or just had to use a temporary phone for some reason...
And you're one of those who hates to lose history, even if you know you hardly look at it...
This tool is for you.
Pre-requisites:
Neither of your phones where your whatsapp is installed need be rooted.
You must enable USB debugging on both devices
You must have Android SDK, we will be using the Android Debug Bridge (ADB)
Have Android Backup Extractor ready, we will use it.
Instructions:
Connect the phone which holds your old database to your computer, and enable USB debugging.
Use ADB to make a backup of the whatsapp database
Code:
adb.exe backup -f whatsapp.ab com.whatsapp
Use 'Android Backup Extractor' to extract whatsapp.ab to whatsapp.tar
Open whatsapp.tar with 7zip (or any other compression utility) and extract the file called 'msgstore.db'
Rename the file to 'old.db'
Repeat the above steps for your new phone, but rename the file to 'new.db'
Run merjapp in the following fashion:
Code:
java.exe -jar merjapp.jar -old old.db -new new.db
You can also use the -simulation mode if you like, it will not create any output database, and you will be able to read the log created in the working directory.
If you want extra debug information you can use the -debug mode.
If everything went correctly and "Done!' is displayed, rename output.db (which is your merged database) to msgstore.db
Connect your new phone to your computer, open the Whatsapp/Databases directory on the internal SD card and backup its contents to some other location.
Copy msgstore.db created in the previous steps to this directory
Clear whatsapp's data from your new phone and force stop it.
Open whatsapp and when prompted, elect to restore from backup, and voila!
Download
Download merjapp v141201 (runnable jar file)
Backup
Backup
Just registered to say thank you. It works!
Official whatsapp method wasn't working for me: "unable to recover chat history". So your tool saved me. Thanks!
namnoops said:
If you ever lost your phone due to an accident, or just had to use a temporary phone for some reason...
And you're one of those who hates to lose history, even if you know you hardly look at it...
This tool is for you.
Pre-requisites:
Neither of your phones where your whatsapp is installed need be rooted.
You must enable USB debugging on both devices
You must have Android SDK, we will be using the Android Debug Bridge (ADB)
Have Android Backup Extractor ready, we will use it.
Instructions:
Connect the phone which holds your old database to your computer, and enable USB debugging.
Use ADB to make a backup of the whatsapp database
Code:
adb.exe backup -f whatsapp.ab com.whatsapp
Use 'Android Backup Extractor' to extract whatsapp.ab to whatsapp.tar
Open whatsapp.tar with 7zip (or any other compression utility) and extract the file called 'msgstore.db'
Rename the file to 'old.db'
Repeat the above steps for your new phone, but rename the file to 'new.db'
Make back-ups of both old.db and new.db, as merjapp will make changes in them
Run merjapp in the following fashion:
Code:
java.exe -jar merjapp.jar -old old.db -new new.db
You can also use the -simulation mode if you like, it will not change your databases and you will be able to read the log created in the working directory.
If you want extra debug information you can use the -debug mode.
If everything went correctly and "Done!' is displayed, rename old.db (which is your merged database now) to msgstore.db
Connect your new phone to your computer, open the Whatsapp/Databases directory on the internal SD card and move its contents to some other location
Copy msgstore.db created in the previous steps to this directory
Clear whatsapp's data from your new phone and force stop it.
Open whatsapp and when prompted, elect to restore from backup, and voila!
Download
Download merjapp v141005 (runnable jar file)
Click to expand...
Click to collapse
A pretty neat tool!
I was trying to merge two databases. It seemed that everything went okay and all with the "Done!" message. However, there seems to be no changes made two any of the database. Their filesizes remains the same for old.db and new.db.
I tried loaded the old.db anyway since you said that is the merged one. However, I am getting only the old one.
Please advise.
And you didn't use the '-simulation' flag by chance, right?
Can you attach the log file?
Hi,
I think I have done everything as instructed above. But the phone doesn't seem to recognize the new mgstore.db backup file.
I notice that the new file isn't crypt7 as the original mgstore.db file from the internal drive. Is that causing my problems? How do I fix it?
Greetz
clevernam said:
mgstore.db
Click to expand...
Click to collapse
Did you make a typo with the file like you did here? It should be msgstore.db, not mgstore.db.
No, it doesn't have to be encrypted in order for whatsapp to recognize it.
namnoops said:
Did you make a typo with the file like you did here? It should be msgstore.db, not mgstore.db.
No, it doesn't have to be encrypted in order for whatsapp to recognize it.
Click to expand...
Click to collapse
Nope, that's not it, just checked.
Do i have to put it into the msgstore.tar file? Where i extracted new and old from?
Greetz (thanks for helping btw., if this works, its a realy big help to me!!)
Nope, you just have to place msgstore.db inside the 'whatsapp' directory on your internal SD card (I'd make sure it's the only file there).
Then clear data on whatsapp, force close it, and run it.
Is that exactly what you're doing?
Multiple Databases
Can this app merge multiple databases...
Like if i want to merge 3-4 old databases into one? is it possible? if either one of these databases have same data, will that be duplicated?
It can merge more than one, but that's under the assumption they're mutually exclusive (in terms of messages - i.e., no message appears twice)
So if you used one phone (1), started with a clean whatsapp on a second phone (2), then another clean whatsapp on a third (3), and so on...
You'll merge (1) as 'old' and (2) as 'new', then use the output as 'old' and merge it with (3) as 'new', etc.
namnoops said:
Nope, you just have to place msgstore.db inside the 'whatsapp' directory on your internal SD card (I'd make sure it's the only file there).
Then clear data on whatsapp, force close it, and run it.
Is that exactly what you're doing?
Click to expand...
Click to collapse
Yes, thats exactly what i did. Any other idea what could be wrong?
I'm sorry - it should be under WhatsApp/Databases like the opening post says...
namnoops said:
I'm sorry - it should be under WhatsApp/Databases like the opening post says...
Click to expand...
Click to collapse
That's what i figured, because I was already thinking about all the other files with dates like msgstore-2014-10-12.1.db.crypt7 that i took out of the Database folder.
namnoops said:
I'm sorry - it should be under WhatsApp/Databases like the opening post says...
Click to expand...
Click to collapse
The output file, that I renamed to msgstore.db has "gear wheels" as an icon, but the msgstore file from the database of the phone has no icon at all. Is that somehow significant? Does that maybe lead you to my mistacke?
The fact the the output file has the 'gear wheel' icon just means Windows detects it as a database, I think it's quite meaningless...
The other files it didn't give this icon are *.db? Or are you referring to the encrypted databases? They encrypted ones probably won't have this icon.
It seems to me you're doing everything right:
Renaming the file to 'msgstore.db'
Put it in your internal SD card under WhatsApp/Databases
Making sure there's no other file there except this file
Force closing whatsapp, clearing data, opening the app...
So you're doing all this, and when whatsapp loads it initializes from scratch and doesn't tell you a backup has been found?
It just goes on to initialize an empty database as though you just installed whatsapp for the first time?
namnoops said:
The fact the the output file has the 'gear wheel' icon just means Windows detects it as a database, I think it's quite meaningless...
The other files it didn't give this icon are *.db? Or are you referring to the encrypted databases? They encrypted ones probably won't have this icon.
It seems to me you're doing everything right:
Renaming the file to 'msgstore.db'
Put it in your internal SD card under WhatsApp/Databases
Making sure there's no other file there except this file
Force closing whatsapp, clearing data, opening the app...
So you're doing all this, and when whatsapp loads it initializes from scratch and doesn't tell you a backup has been found?
It just goes on to initialize an empty database as though you just installed whatsapp for the first time?
Click to expand...
Click to collapse
Precisley. Nothing comes, when I reinstall whatsapp.
It is not like when I put the normal backup data back in the databse folder. Then whatsapp realises that there is a backup file in the internal memory and gives me the option to restore the history. Not so with the merged file.
Btw. I have no sd card. I connect the phone with usb. I figured that it is an unessencial diffrence? Am I wrong? My phone btw. is Motorola Moto G. If that helps
I'll check, if the old an new are mutually exclusive. I think in the progress I got a message on the new and old one. What would happen in that case? Would your program do the job, but then creat a bad "output" file?
If the only problem was, that I messed up the historys, I'm sorry, namnoops.
Are you using the latest version of whatsapp?
iclumyc Doctrine
namnoops said:
Are you using the latest version of whatsapp?
Click to expand...
Click to collapse
Well if I didn't then, now I do. And it still doesn't work
I checked if there are any intersecting messages... It turns out there weren't any.
@namnoops: Does the oldest message in the new history have to be more recent then any old message from the old history? I mean: Is it a problem for merjapp if the timelines intersect in any way, although there are no double messages?
Thanks for your help in advance
Background
I recently installed the CyanogenMod12 on my Nexus 5 Lollipop and made sure to perform a full backup first. But I did not realize the difficulty I would be experiencing in attempting to use the backup file's contents to restore my previous SMS messages to my new install. Long story short, I've located the files I need to restore onto my hard drive from the backup and found a thread detailing the commands I would need to execute: http://forum.xda-developers.com/galaxy-nexus/help/restoring-sms-db-via-adb-push-t2146640
My problem
I cannot find the path of these two files on my new phone. I've confirmed that they in fact do exist by backing up my phone's current contents with adb then extracting the backup and searching for the files but the path provided in the backup file do not help me. I started off using airdroid to locate these files but eventually resorted to running a command to dump the outputs of ls via adb shell to my computer and searching for the file name... They were not listed there. Can anybody explain to me why a backup would contain the files yet adb shell isn't printing them out?
Files I'm attempting to find on my hard drive: messaging.db and messaging.db-journal
Path of these files within my extracted backups, both old and new: \apps\com.textra\db
Command I ran to dump adb shell listing: adb shell ls -lR > C:\file_output.txt
Unfortunately all of the sms backup/restore utilities don't actually use the db file format utilized by Android. I am aware I can use SQL Lite to view the db's contents, but I'm interested in restoring these messages to my new operating system install.... Any insight or assistance would be most appreciated.
I did not have root access enabled on my device... Now that I can find and access these files I plan on using Titanium to create a backup, modifying the Titanium backup by inserting my older files, and restoring the Titanium backup. I'll update my post and edit the Title to reflect the resolved state momentarily.
Edit: This did the trick... (just found out I can't update the title too).
I did the following in case anybody stumbles upon this post with this issue in the future: After turning on root access I performed an app data backup of Textra using Titanium backup. I transferred over the backup tar.gz to my computer using Airdroid, inserted the older SMS files using 7zip, force stopped Titanium and Textra just because and then replaced the original Textra Titanium backup file with my modified one. Once I restored the backup all my SMS texts from my previous install were loaded into Textra.... Phew
dentonid said:
Background
I recently installed the CyanogenMod12 on my Nexus 5 Lollipop and made sure to perform a full backup first. But I did not realize the difficulty I would be experiencing in attempting to use the backup file's contents to restore my previous SMS messages to my new install. Long story short, I've located the files I need to restore onto my hard drive from the backup and found a thread detailing the commands I would need to execute: http://forum.xda-developers.com/galaxy-nexus/help/restoring-sms-db-via-adb-push-t2146640
My problem
I cannot find the path of these two files on my new phone. I've confirmed that they in fact do exist by backing up my phone's current contents with adb then extracting the backup and searching for the files but the path provided in the backup file do not help me. I started off using airdroid to locate these files but eventually resorted to running a command to dump the outputs of ls via adb shell to my computer and searching for the file name... They were not listed there. Can anybody explain to me why a backup would contain the files yet adb shell isn't printing them out?
Files I'm attempting to find on my hard drive: messaging.db and messaging.db-journal
Path of these files within my extracted backups, both old and new: \apps\com.textra\db
Command I ran to dump adb shell listing: adb shell ls -lR > C:\file_output.txt
Unfortunately all of the sms backup/restore utilities don't actually use the db file format utilized by Android. I am aware I can use SQL Lite to view the db's contents, but I'm interested in restoring these messages to my new operating system install.... Any insight or assistance would be most appreciated.
Click to expand...
Click to collapse
I have some files in /data/data/com.something that are again and again deleted or rather overwritten by a program every time I run it. I'd really love a way to delete protect those files of mine.
Points:
I know that the files are overwritten because they are read only files so the only logical explanation is that they are first deleted and then replaced with a new file.
The files should be prevented from being deleted or edited but I also want that the data still be executable and viewable.
The files are in .XML format.
The name of the files and their current location should not change.
Last but not the least, this change should be reversible.
Can any human being help me achieve this or is this something that only the almighty can do?
Evan Jongske Sangma said:
I have some files in /data/data/com.something that are again and again deleted or rather overwritten by a program every time I run it. I'd really love a way to delete protect those files of mine.
Points:
I know that the files are overwritten because they are read only files so the only logical explanation is that they are first deleted and then replaced with a new file.
The files should be prevented from being deleted or edited but I also want that the data still be executable and viewable.
The files are in .XML format.
The name of the files and their current location should not change.
Last but not the least, this change should be reversible.
Can any human being help me achieve this or is this something that only the almighty can do?
Click to expand...
Click to collapse
Greetings and welcome to assist. If the files are in data then they can be overwritten if they are in system/root then they could only be ovewritten by a root application.
Hope this helps
Good Luck
Sawdoctor
I have a BLU phone that was my only source of contacts with no Google or backup (yes, I know). It was dropped a significant distance. It won't show any indication of charging or powering up, no lights or sounds. However when I plugged in the USB port I noticed the toggling of the USB driver install, typical for a MTK device so I decided to try to read from it with SPFlash tool.
To try to get at the contacts I downloaded a factory restore ROM and loaded the scatter file to see what location I needed to seek to find the user data for download. I first found userdata.img location from the factory scatter file and set up SPF under the ReadBack tab and used the Add button to specify a range beginning with and including the same numbers where the scatter file indicated the user data was, plus with SPF tool I stated I wanted the user data area read and saved as userdata.img. Now clicking on the read and then plugging the phone in with the read button pre-pushed, the SPF tool took off reading as soon as the dead phone USB was plugged in, operating just like I was flashing or writing to the phone. (Excitement was mounting.)
From here on my Windows machine I installed DiskInternals Linux Reader and mounted the userdata.img with that software and was able to navigate to the db file containing the contacts and the android call list located at
/data/data/com.android.providers.contacts/databases/contacts2.db.
Now the DiskInternals software allowed me to save the contacts2.db as a Windows file anywhere on the laptop but when I tried to open it with SQLite, it said the file is encrypted. I viewed the file using a Linux friendly editor (Context) and saw some names and numbers in readable text, but they were surrounded by tons of formatting and unprintable characters. (Looked something like a MS Word docx file.) I also tried the FireFox plugin for SQLite in addition to the SQLiteManager and the SQLite browser software. All of them said the file had a password that was needed to open it. (Exit excitement.)
Now it looks like my options are to either try to find someone or thing that can hack the password of the contacts2.db, locate the password perhaps hidden somewhere in a system.img that I can attempt to extract via the same method as I did the userdata, or I can try to assemble a working group of restore "parts" which include this userdata.img, adding it to some mix of the restore file and other IMG files I can possibly extract from the "bricked all but for flash tool" mostly dead phone. Theoretically, with such a reassembled package of IMG files I would then go look for an exact same model of this bricked phone and then try to load the new assembly of IMG files onto that good used phone.
I am open for suggestions of how to solve the puzzle. The objective is to get a list of readable contacts from the crashed phone and ideally get them exported into a format that can be input into another phone.
Thanks in advance.
P.S. Android 5.0 on the phone
NOTE THIS IS NOT "UNABLE TO BACK UP" THIS IS "UNABLE TO RESTORE"
Lots of people on this forum, and lots of people on the internet have errors trying to back up because of this dreaded error 255. But, for me, the backup was fine, but now I can't restore. This is a huge problem because I just tried out a custom ROM (which turned out to be a huge fail), and I want to go back to stock.
So I switched back to the other slot (A/B ftw), wiped all data, formatted SD card, set up the OS to have the same lock pattern as the backup, transferred the backup onto the phone and tried to restore, and, well, fuck.
Code:
[RESTORE STARTED]
Restore folder: '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR'
Skipping Digest check based on user setting.
Calculating restore details...
Restoring 4 partitions...
Total restore size is 11276MB
Restoring Vendor Image...
[Vendor Image done (3 seconds)]
Restoring Boot...
[Boot done (0 seconds)]
Wiping Data (excl. storage)
Wiping data without wiping /data/media ...
Done.
Restoring Data (excl. storage)...
extractTarFork() process ended with ERROR: 255
It failed, and there is absolutely nothing I can do about it. TWRP is using buggy commands to extract the backup and I don't know what's triggering the bug.
As a result, right now all my data is in limbo. TWRP cannot restore this backup as-is. I can't just go back in time and use `pm remove-user 999` or `pm remove-user 10` because the backup has already been created. Any solution that has to be done before the backup phase is inaccessible to me now.
My hope is that some smart person here can help me find a way to get this backup back onto the phone. Right now the internet does not have a solution to this problem, no matter how many naive users spam discussion boards with "pm remove-user 999 totally worked". Serious answers only.
Relevant section in the recovery log:
Code:
==> extracting: //data/vendor_de/10/ (mode 40771, directory)
Cannot find key for 10
error looking up proper e4crypt policy for '//data/vendor_de/10/' - 1DE10
tar_extract_file(): failed to extract //data/vendor_de/10/ !!!
I:Unable to extract tar archive '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR/data.ext4.win000'
Error during restore process.
I:Error extracting '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR/data.ext4.win000' in thread ID 0
I:Error extracting split archive.
Error during restore process.
pigz: write error code 32
pigz: abort: write error on <stdout>
extractTarFork() process ended with ERROR: 255
I suppose the solution is to make it "find a key for 10", but how? Do I just need to somehow recreate the 10 user before restoring the backup?
EDIT NOTICE: THIS DIDN'T WORK, DON'T MAKE IT WORSE BY TRYING
ORIGINAL POST IS BELOW
Alright. Well, I found a solution. That was quick.
The error is triggered by the backup containing a user that isn't present during the restore, I think. One solution is (theoretically) to create the missing user... another solution is to remove the missing user from the backup.
Turns out, backups are not so hard to modify, even in their compressed form. However, it is a little bit unintuitive at first.
You can't use the `tar` command to modify the backup, at all. TWRP's tar doesn't even support the --delete flag, nor does the BSD tar command (what ships with macOS). Even after installing GNU tar, it still refuses to work on a compressed archive (probably for the better).
You either need a Windows system with WinRAR/7zip (allows opening/modifying compressed files without extracting them), or a macOS/Linux system with FUSE and archivemount. If you're on Windows, you will have to manually search every single folder for 10 or 999 folders and delete every single one. If you're on macOS/Linux, the process is a bit more technical but also a bit more automated. And in both cases you will have to do this many times, and it can easily take half an hour or more and many restore attempts.
First and foremost, you need the recovery log to find out where the process failed. Every time it fails you will need to go to Advanced > Copy Log, copy them to the SD card, then use the flipping horrible Android File Transfer program to transfer the recovery.log to your PC, then use the cat/tail command or a text editor of your choice to read the log file.
Somewhere around the end of that file you will find something that looks like this:
Code:
==> extracting: //data/misc_de/10/ (mode 41771, directory)
Cannot find key for 10
error looking up proper e4crypt policy for '//data/misc_de/10/' - 1DE10
tar_extract_file(): failed to extract //data/misc_de/10/ !!!
I:Unable to extract tar archive '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR/data.ext4.win003'
This contains all the info you need: the archive where the tar command gave up, and the file it gave up on.
Destroy that file with extreme prejudice.
Make a mount point with `mkdir mnt`, and mount the archive with `sudo archivemount data.ext4.win003 mnt`. (Of course, substitute data.ext4.win003 with the culprit archive file.)
If you're like me and have this problem with 10 folders, or 999 folders, you can find every single one of its kind with `sudo find mnt -name 10`. You can then use `sudo rm -rf` to delete every one of them. If you're brave, try `sudo find data -name 10 -exec sudo rm -rf \{\} \;`. I'm not responsible for any data losses incurred by using powerful commands that don't prompt you for confirmation.
Notice how I'm repeatedly using `sudo` for this: FUSE is kind of quirky and so is the kernel about multi-user filesystems. If you don't use sudo to mount the archive, then the user won't have permission to represent files it doesn't own, even if root is the one asking. You'll get weird "No such file or directory" errors for files and directories that are contained in parent directories' file listings. If you don't use sudo for the find or rm commands, you won't have permission to look inside folders or delete folders. Really, do everything with sudo because it's the only way to do this correctly.
Once you're done scrubbing the problem files away (be careful, do not use a file explorer of any kind, especially macOS's, which generally craps .DS_Store files all over your filesystem), unmount and save the archive with `sudo umount mnt`.
The next part is very important because recompressing files is not instant.
You will need to run `watch -n 1 du -hs data.ext4.win003` and WAIT.
Watch the file size slowly grow; only continue once it has stayed the exact same for at least one minute. Yes, it can definitely stop for 30 seconds at a time and then resume. You will need to give it time to recompress. It's also probably normal for the recompressed file to be about half the size of the initial file; that's just an artifact of TWRP not using very good compression to begin with. archivemount will create a backup of the archive with .orig appended, so you can try again if your backup ends up getting corrupted. (I just created an entirely separate copy of the backup to use archivemount on)
Of course, if you're on Windows you can skip all this and simply delete the problem file from WinRAR/7zip, and it'll be gone instantly and you can just proceed to copy the archive over to your device and retry the backup. You will have to do all the searching manually though, or perhaps you could try your luck with WSL. (Does WSL support FUSE?)
Expect this process to fail multiple times, but all at different points; whenever it fails at a different file, you know it got past the one you just removed. all you need to do is repeat this process over and over, slowly fixing your backup until it restores properly. This is what needs to be done if you can't reproduce the event that created that user in the first place. I'm sure if the user is important, it'll get created again.
Also, looks like user 10 is what happens when Google Play updates Device Personalization Services. Is this the generic "updates for system apps" user or is it specifically for that app?
EDIT: But updating that app doesn't create user 10. Interesting...
EDIT 2: User 10 is the guest user!! All I needed to do was create user 10 and that allowed me to restore my backup!
My post above is still good to see what I tried at first but it actually bricked my device until I factory reset it again, so :/
LoganDark said:
Also, looks like user 10 is what happens when Google Play updates Device Personalization Services. Is this the generic "updates for system apps" user or is it specifically for that app?
EDIT: But updating that app doesn't create user 10. Interesting...
EDIT 2: User 10 is the guest user!! All I needed to do was create user 10 and that allowed me to restore my backup!
My post above is still good to see what I tried at first but it actually bricked my device until I factory reset it again, so :/
Click to expand...
Click to collapse
Glad you managed to get your device back!
I had the same problem on another phone. The reason I got the error was because I tried to restore a decrypted backup on an encrypted device. After choosing the option "Format" in TWRP and typing 'yes' to confirm the decryption process, I was able to restore my backup.
Im having the same error in my realme xt...
sir..please help me.. I have a twrp backup of system image vendor image data boot recovery persist but im unable to recover because it shows an "error extract tar fork process ended up with error 255".during backup i had a pincode for screenlock.
i also have a backup of my internal storage in twrp but im not able to restore because of this error
LoganDark said:
Also, looks like user 10 is what happens when Google Play updates Device Personalization Services. Is this the generic "updates for system apps" user or is it specifically for that app?
EDIT: But updating that app doesn't create user 10. Interesting...
EDIT 2: User 10 is the guest user!! All I needed to do was create user 10 and that allowed me to restore my backup!
My post above is still good to see what I tried at first but it actually bricked my device until I factory reset it again, so :/
Click to expand...
Click to collapse
Hi. Did you create user 10 in Source, or in destination?
How did you do that in source?
If in destination, then is it possible to do it via TWRP?
EDIT: disregard. Mine user in the error path is 0 :-( '//data/system_de/0/'
just use option format data when entering wipe selection
erosman23 said:
just use option format data when entering wipe selection
Click to expand...
Click to collapse
doesnt work
redmi k20 davinci