Related
Hi there,
I was wondering (and searching different boards) if there is no option to copy files from internal/sdcard flash by schedule. I do a titanium and sms backup every sunday and would like zip/rar these folders and copy the saved archives to any folder on my network (a nas, which I have already mounted in my android).
Commandline with the terminal emulator wouldn´t be a problem, but I dont know how to run scripts by schedule.
Edit: If someone knows how to copy these archives to the installed dropbox sync, this would be awesome too!
Thanks!
shrewm said:
Hi there,
I was wondering (and searching different boards) if there is no option to copy files from internal/sdcard flash by schedule. I do a titanium and sms backup every sunday and would like zip/rar these folders and copy the saved archives to any folder on my network (a nas, which I have already mounted in my android).
Commandline with the terminal emulator wouldn´t be a problem, but I dont know how to run scripts by schedule.
Edit: If someone knows how to copy these archives to the installed dropbox sync, this would be awesome too!
Thanks!
Click to expand...
Click to collapse
For scheduled commandlines , google for cron / cronjob.
Well I did a simple script with GScript Lite and copied the dirs to a smb-share which I mounted with mountmanager. Did work pretty well, but then I stumbled up on the DropSpace-App which works exact the way I want to.
Thanks anyway, darken
Some time ago I sold my old android phone and forgot to make a backup of SMS messages and call logs, but kept a complete image backup made by clockworkmod recovery (system.img, data.img, etc.). I wanted to import my SMS messages and call logs to a new phone, but without a reasonable backup this seemed to be impossible. Some quick search over the internet showed that a question about extracting/importing SMS messages from mmssms.db is not so uncommon, but there are no tools to do this. So I wrote one, and decided to share
Sources are available at github.com abbot/android-restore-tools.
It requires Python 2.6 and above to run.
I've also made compiled binaries for windows, attached to this post.
[Binaries last updated on 12.04.2011]
Thanks for the tools. Nice work.
Sent from my GT-I8150 using XDA
Hello,
I'm trying to use your tool to extract SMS from a nandroid backup.
I'm using the data.yaffs2.img file from the backup.
When I choose #2 for mmssms.db and then "s" to extract SMS I get the following error:
Failed to extract messages: file is encrypted or is not a database
DatabaseError('file is encrypted or is not a database',)
Warning: failed to remove temporary file...
What does this mean, and is there a solution?
Hi,
This may be caused by two things: either my tool can't properly read/extract the image file, or it can't read the database.
Please try to extract the image (extract -x data.yaffs2.img). If this does not produce any errors, find the mmssms.db file in the extracted data, it will probably be in data/com.android.providers.telephony/databases/mmssms.db. Then try to run extract -s mmssms.db. Please post if you get any errors doing these steps.
There was an error while extracting the image...it got through partway, but then failed at some bluetooth directory with colons in the path.
So I used a different tool to extract it, which went successfully.
Then I ran extract -s mmssms.db which came with the same error.
I'm pretty sure there isn't some weird encryption since I opened it up with Notepad ++ and I could read bits and pieces of conversations.
Anything else to try?
This might be caused by an older sqlite3 version bundled with binaries. I have updated the binaries in the first message to a newer version, could you download it and try again, extract -s mmssms.db?
Different error this time:
Failed to extract messages: no such column: failure_cause
OperationalError('no such column: failure_cause',)
Warning: failed to remove temporary file...
FYI the first thing at the top of the file when I open in Notepad++ is SQLite format 3 and this is from an HTC device running Android 4.x does that help at all?
This is much better and now makes sense: android 4.0 usually has sqlite 3.7.x, previous binary build of this tool had sqlite 3.6.21, and that was the reason for the 'file is encrypted or is not a database' error.
Now it looks like mmssms.db format in Android 4.0 has changed a little bit. I will have a look on these changes and update the app accordingly. Hope it will not take too much time
abbot2 said:
This is much better and now makes sense: android 4.0 usually has sqlite 3.7.x, previous binary build of this tool had sqlite 3.6.21, and that was the reason for the 'file is encrypted or is not a database' error.
Now it looks like mmssms.db format in Android 4.0 has changed a little bit. I will have a look on these changes and update the app accordingly. Hope it will not take too much time
Click to expand...
Click to collapse
That would be awesome, perhaps keep the old version available for other folks too though
If you can get this working I'll be sure to send a couple bucks your way. Thanks!
I have updated the extractor again, did some limited testing with android 4.0.3 on the emulator - seems to work. This database has a number of fields removed on android 4 compared to android 2, however everything required for xml dump is still there - just had to remove some unused stuff. Download the new version and try again.
Regarding the older versions, no reason to keep them - new one works fine with old database formats.
Awesome works perfectly! Send me a PM where to donate.
Sent from my GT-I9000 using xda premium
Thanks! It really works perfectly. Any plans to include MMS?
Sorry for the noobish question but I'm not familiar with Python.
Installed newest version of Python and also got the extracted mmssms.db and contacts2.db (used Nandriod Browser to extract them), which command lines do I need/what do I have to do? :s
€dit: Downloaded the zip from github and got as far as opening mmssms2xml.py which gives me an error when opening the mmssms.db
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Python27\lib\lib-tk\Tkinter.py", line 1410, in __call__
return self.func(*args)
File "D:\Downloads\abbot-android-restore-tools-bc8584d\abbot-android-restore-t
ools-bc8584d\mmssms2xml.py", line 99, in open_file
self.messages = read_messages(filename)
File "D:\Downloads\abbot-android-restore-tools-bc8584d\abbot-android-restore-t
ools-bc8584d\mmssms2xml.py", line 33, in read_messages
c.execute("SELECT _id, thread_id, address, person, date, protocol, read, pri
ority, status, type, callback_number, reply_path_present, subject, body, service
_center, failure_cause, locked, error_code, stack_type, seen, sort_index FROM sm
s ORDER BY date DESC")
OperationalError: no such column: priority
Click to expand...
Click to collapse
€dit2: No problems with contacts2.db?!
€dit3: Well..this should be the last, found this explanation of rani2001 over here (http://forum.xda-developers.com/showpost.php?p=25173166&postcount=7). That did it for me!
Dude you just made my day
Can you please compile a new windows version with the latest files from github?
Or give us a manual how to use the .py files.
I think this will help many of us.
Thank you very much.
BTW: Your tool worked perfectly with my mmssms.db but failed with contacts2.db. Maybe the new files requested above will help me.
Got all my sms back thanks to you.
Amazing script, Thanks for sharing and explaining and updating.
60% There
I was just wondering if there was anything special I had to do to merge my old SMS list and the ones from the new list. I didn't see an option there for it and I don't want to just paste it in there and load it. Many thanks ahead of time
Hi I want to extract the data.yaffs2.img from my HTC Desire nandroid backup but get the error message:
> extract.exe -s data.yaffs2.img
Failed to extract messages: file is encrypted or is not a database
Any ideas?
I tried this tool yesterday evening and I'm truly grateful for your work!
I was ready to spend the whole night to work with these damn tables when I found your work. Many thanks! :highfive:
Where is the file stored after i run it? This is what I got.
Code:
C:\Users\Damastah>C:\Users\Damastah\Downloads\yaffs-mmssmsdb-calls-extractor\ext
ract.exe -s D:\S3_recovery\mmssms.db
Read 1782 messages
Save as (empty=sms-20130330111807.xml): y
If you ever lost your phone due to an accident, or just had to use a temporary phone for some reason...
And you're one of those who hates to lose history, even if you know you hardly look at it...
This tool is for you.
Pre-requisites:
Neither of your phones where your whatsapp is installed need be rooted.
You must enable USB debugging on both devices
You must have Android SDK, we will be using the Android Debug Bridge (ADB)
Have Android Backup Extractor ready, we will use it.
Instructions:
Connect the phone which holds your old database to your computer, and enable USB debugging.
Use ADB to make a backup of the whatsapp database
Code:
adb.exe backup -f whatsapp.ab com.whatsapp
Use 'Android Backup Extractor' to extract whatsapp.ab to whatsapp.tar
Open whatsapp.tar with 7zip (or any other compression utility) and extract the file called 'msgstore.db'
Rename the file to 'old.db'
Repeat the above steps for your new phone, but rename the file to 'new.db'
Run merjapp in the following fashion:
Code:
java.exe -jar merjapp.jar -old old.db -new new.db
You can also use the -simulation mode if you like, it will not create any output database, and you will be able to read the log created in the working directory.
If you want extra debug information you can use the -debug mode.
If everything went correctly and "Done!' is displayed, rename output.db (which is your merged database) to msgstore.db
Connect your new phone to your computer, open the Whatsapp/Databases directory on the internal SD card and backup its contents to some other location.
Copy msgstore.db created in the previous steps to this directory
Clear whatsapp's data from your new phone and force stop it.
Open whatsapp and when prompted, elect to restore from backup, and voila!
Download
Download merjapp v141201 (runnable jar file)
Backup
Backup
Just registered to say thank you. It works!
Official whatsapp method wasn't working for me: "unable to recover chat history". So your tool saved me. Thanks!
namnoops said:
If you ever lost your phone due to an accident, or just had to use a temporary phone for some reason...
And you're one of those who hates to lose history, even if you know you hardly look at it...
This tool is for you.
Pre-requisites:
Neither of your phones where your whatsapp is installed need be rooted.
You must enable USB debugging on both devices
You must have Android SDK, we will be using the Android Debug Bridge (ADB)
Have Android Backup Extractor ready, we will use it.
Instructions:
Connect the phone which holds your old database to your computer, and enable USB debugging.
Use ADB to make a backup of the whatsapp database
Code:
adb.exe backup -f whatsapp.ab com.whatsapp
Use 'Android Backup Extractor' to extract whatsapp.ab to whatsapp.tar
Open whatsapp.tar with 7zip (or any other compression utility) and extract the file called 'msgstore.db'
Rename the file to 'old.db'
Repeat the above steps for your new phone, but rename the file to 'new.db'
Make back-ups of both old.db and new.db, as merjapp will make changes in them
Run merjapp in the following fashion:
Code:
java.exe -jar merjapp.jar -old old.db -new new.db
You can also use the -simulation mode if you like, it will not change your databases and you will be able to read the log created in the working directory.
If you want extra debug information you can use the -debug mode.
If everything went correctly and "Done!' is displayed, rename old.db (which is your merged database now) to msgstore.db
Connect your new phone to your computer, open the Whatsapp/Databases directory on the internal SD card and move its contents to some other location
Copy msgstore.db created in the previous steps to this directory
Clear whatsapp's data from your new phone and force stop it.
Open whatsapp and when prompted, elect to restore from backup, and voila!
Download
Download merjapp v141005 (runnable jar file)
Click to expand...
Click to collapse
A pretty neat tool!
I was trying to merge two databases. It seemed that everything went okay and all with the "Done!" message. However, there seems to be no changes made two any of the database. Their filesizes remains the same for old.db and new.db.
I tried loaded the old.db anyway since you said that is the merged one. However, I am getting only the old one.
Please advise.
And you didn't use the '-simulation' flag by chance, right?
Can you attach the log file?
Hi,
I think I have done everything as instructed above. But the phone doesn't seem to recognize the new mgstore.db backup file.
I notice that the new file isn't crypt7 as the original mgstore.db file from the internal drive. Is that causing my problems? How do I fix it?
Greetz
clevernam said:
mgstore.db
Click to expand...
Click to collapse
Did you make a typo with the file like you did here? It should be msgstore.db, not mgstore.db.
No, it doesn't have to be encrypted in order for whatsapp to recognize it.
namnoops said:
Did you make a typo with the file like you did here? It should be msgstore.db, not mgstore.db.
No, it doesn't have to be encrypted in order for whatsapp to recognize it.
Click to expand...
Click to collapse
Nope, that's not it, just checked.
Do i have to put it into the msgstore.tar file? Where i extracted new and old from?
Greetz (thanks for helping btw., if this works, its a realy big help to me!!)
Nope, you just have to place msgstore.db inside the 'whatsapp' directory on your internal SD card (I'd make sure it's the only file there).
Then clear data on whatsapp, force close it, and run it.
Is that exactly what you're doing?
Multiple Databases
Can this app merge multiple databases...
Like if i want to merge 3-4 old databases into one? is it possible? if either one of these databases have same data, will that be duplicated?
It can merge more than one, but that's under the assumption they're mutually exclusive (in terms of messages - i.e., no message appears twice)
So if you used one phone (1), started with a clean whatsapp on a second phone (2), then another clean whatsapp on a third (3), and so on...
You'll merge (1) as 'old' and (2) as 'new', then use the output as 'old' and merge it with (3) as 'new', etc.
namnoops said:
Nope, you just have to place msgstore.db inside the 'whatsapp' directory on your internal SD card (I'd make sure it's the only file there).
Then clear data on whatsapp, force close it, and run it.
Is that exactly what you're doing?
Click to expand...
Click to collapse
Yes, thats exactly what i did. Any other idea what could be wrong?
I'm sorry - it should be under WhatsApp/Databases like the opening post says...
namnoops said:
I'm sorry - it should be under WhatsApp/Databases like the opening post says...
Click to expand...
Click to collapse
That's what i figured, because I was already thinking about all the other files with dates like msgstore-2014-10-12.1.db.crypt7 that i took out of the Database folder.
namnoops said:
I'm sorry - it should be under WhatsApp/Databases like the opening post says...
Click to expand...
Click to collapse
The output file, that I renamed to msgstore.db has "gear wheels" as an icon, but the msgstore file from the database of the phone has no icon at all. Is that somehow significant? Does that maybe lead you to my mistacke?
The fact the the output file has the 'gear wheel' icon just means Windows detects it as a database, I think it's quite meaningless...
The other files it didn't give this icon are *.db? Or are you referring to the encrypted databases? They encrypted ones probably won't have this icon.
It seems to me you're doing everything right:
Renaming the file to 'msgstore.db'
Put it in your internal SD card under WhatsApp/Databases
Making sure there's no other file there except this file
Force closing whatsapp, clearing data, opening the app...
So you're doing all this, and when whatsapp loads it initializes from scratch and doesn't tell you a backup has been found?
It just goes on to initialize an empty database as though you just installed whatsapp for the first time?
namnoops said:
The fact the the output file has the 'gear wheel' icon just means Windows detects it as a database, I think it's quite meaningless...
The other files it didn't give this icon are *.db? Or are you referring to the encrypted databases? They encrypted ones probably won't have this icon.
It seems to me you're doing everything right:
Renaming the file to 'msgstore.db'
Put it in your internal SD card under WhatsApp/Databases
Making sure there's no other file there except this file
Force closing whatsapp, clearing data, opening the app...
So you're doing all this, and when whatsapp loads it initializes from scratch and doesn't tell you a backup has been found?
It just goes on to initialize an empty database as though you just installed whatsapp for the first time?
Click to expand...
Click to collapse
Precisley. Nothing comes, when I reinstall whatsapp.
It is not like when I put the normal backup data back in the databse folder. Then whatsapp realises that there is a backup file in the internal memory and gives me the option to restore the history. Not so with the merged file.
Btw. I have no sd card. I connect the phone with usb. I figured that it is an unessencial diffrence? Am I wrong? My phone btw. is Motorola Moto G. If that helps
I'll check, if the old an new are mutually exclusive. I think in the progress I got a message on the new and old one. What would happen in that case? Would your program do the job, but then creat a bad "output" file?
If the only problem was, that I messed up the historys, I'm sorry, namnoops.
Are you using the latest version of whatsapp?
iclumyc Doctrine
namnoops said:
Are you using the latest version of whatsapp?
Click to expand...
Click to collapse
Well if I didn't then, now I do. And it still doesn't work
I checked if there are any intersecting messages... It turns out there weren't any.
@namnoops: Does the oldest message in the new history have to be more recent then any old message from the old history? I mean: Is it a problem for merjapp if the timelines intersect in any way, although there are no double messages?
Thanks for your help in advance
So yesterday My phone, Sony Xperia S, decided to go into bootloop and not boot at all, no matter what I did. Tried flashing stock PRF, stock firmware, some stock based firmware, but it would get stuck in bootloop. The reason for this being I was not formatting the data partition, which I needed to preserve since it had all My sms, contacts, call logs and other app data. Luckily, I had a working recovery(TWRP) and was able to take a backup. Then using I was able to recover My contacts, sms, call logs and app data. After a lot of Google searches, I was finally able to work it out. The process is as follows:
Needed software:
Code:
yaffs-mmssmsdb-calls-extractor.zip from [URL="http://forum.xda-developers.com/showthread.php?t=1585957"]here[/URL].
Firefox Browser
Firefox addon of SQLite manager : [URL="https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager"]link[/URL].
Excel macro to convert .db contacts to .csv : [URL="http://thydzik.com/downloads/convert-contacts2.db-data-to-vcard-vcf.xlsm"]link[/URL].
Software for csv to vcard conversion (Windows) : [URL="http://www.csvtovcard.com/"]link[/URL].
CSV to vcard conversion (online) : [URL="http://labs.brotherli.ch/vcfconvert/"]link[/URL].
SMS Backup & Restore android app : [URL="https://play.google.com/store/apps/details?id=com.riteshsahu.SMSBackupRestore"]link[/URL].
Call Log Backup and Restore app : [URL="https://play.google.com/store/apps/details?id=com.riteshsahu.CallLogBackupRestore"]link[/URL].
The procedure is a lot crude as of yet, I would update as more universal and convenient options are found.
I know there is Titanium backup app which can restore contact, sms etc from nandroid backups, but You need a paid version of the app.
First we need to extract the data partition of nandroid
Rename the data.ext4.win to data.ext4.tar.gz . If Your files are like data.ext4.win000, data.ext4.win001, then rename each of them to data.ext4.tar.gz in separate folders.
On windows, now You can directly extract them using WinRaR or 7zip application, although You don’t need to extract it entirely.
On linux, You can do so using the following command. It is recommended to store the file in a separate folder as the file gets extracted directly.
Code:
tar –xf data.ext4.tar.gz
Now we need to find two files contacts2.db and mmssms.db . The contacts2.db files is for contacts as well as the call logs and the mmssms.db is for messages. The contacts2.db is constant across most ROM’s and OEM firmware apps. The mmssms.db file may vary as per app and ROM’s. So what You need to do is look for a .db file inside the SMS app You have been using with the firmware/ROM.
We will restore the SMS and Call logs first.
Currently it is limited to Windows OS only.
Extract the contents of yaffs-mmssmsdb-calls-extractor.zip to a folder.
Copy the mmssms.db file and contacts2.db file to the above folder.
Now open a command prompt at location of the above folder. This can be done by opening the folder, right clicking using shift key pressed in blank space. You get the “Open command window here” option.
In it type:
Code:
extract -s mmssms.db
It will ask You to enter a name for the output .xml file, You can press enter to let it give a default name. This will convert Your sms to a .xml file .
Then type:
Code:
extract –c contacts2.db
It will also ask You to enter a name for the output .xml file, You can press enter to let it give a default name. This will convert Your call logs to a .xml file .
Copy both of these files to Your newly formatted phone and then use the two before mentioned apps (Call Log backup& Restore and SMS Backup & Restore) to restore Your SMS and call logs.
Now retrieving contacts. It is a bit lengthy, but simple process.
Start firefox browser. In it, after You have installed the addon do following to enable the addon
Go to options (3 horizontal lines).
Select “Customize” and drag the “SQLite Manager” to the blue bordered box.
Now open options and select SQLite Manager. In it select Database>Connect Database. In the opened dialog box, select All Files(*.*). Then select Your contacts2.db .
From the left side tree structure in the SQLite Manager window, right click on “data” and select “Export Table”.
Now in right side part of the window, You will get CSV tab and SQL tab. Under CSV tab, simply press OK button without changing any other options.
It will ask You to save Your .csv file. Save it.
Now if You try to open this file directly, You will notice that file is not properly arranged.
To arrange it properly, open the convert-contacts2.db-data-to-vcard-vcf.xlsm file. If You don’t know how to enable Macros don the following:
It will show You a warning “Security Warning Macros have been disabled. Options..”
Press the Options button. Select “Enable this content” and press ok.
Now Press the office button present at left top corner. In it select “Excel Options”.
Select Customize tab from left side. Then from “Choose commands from:” drop down menu, select Macros.
Now add convertDatatoContacts and writeContactstovCard to right pane from left pane by selecting them and pressing add. Press OK button.
Now as You can see two white square icons have been added at top right corner next to save undo redo options.
Select the first white square icon. Select “Enable Macros” button. Now select Your .csv file in the opened dialog box.
Now the file must have opened in a separate excel. The format should be 4 rows with they being First Name, Last Name, Phone 1 and Phone 2.
Then select Save As>Excel Workbook>Save Type as CSV(Comma delimited).csv . Press ok and yes on the two warnings generated.
At the left bottom, right click on “contacts” sheet and press delete. Save and close the file.
Run CSV to vCard software and open the above saved .csv file. Here You can save all contacts as single vCard file or separate files.
In the “Properties” section on the window, in front of “Phone 1” and “Phone 2” select “Mobile Phone” from options. Press “Convert” button.
Alternative to above two steps, You can also convert .csv to vcard using the website.
Now a .vcf file will be created where the .csv file was stored.
Now simply copy the .vcf file to phone and import using the phone contacts app!
Done!
Credits and sources:
@abbot2
[TOOL] yaffs extractor, mmssms.db & contacts2.db converter
http://thydzik.com/export-android-contacts-contacts2-db-to-vcard-vcf-on-windows/
Reserved.
Awesome concept
Thanks for posting this! Im working on a sprint S4 that is bootlooping. ive extracted all data and am attempting to recover contacts with your method. I am getting an error with the yaffs extract.exe that im hoping you could give me some direction on.
**EDIT**
I was never successful using yaffs (dependency issue perhaps) but i downloaded firefox and was able to extract everything with the sqlite extension. THANK YOU!!!! This tool is one of those things you dont need often, but when you do it's a godsend!
when i issue the contacts extraction command i get this output
C:\Users\PartSwap\Desktop\DB_xtract>extract -c contacts2.db
Read 0 calls
Failed to extract calls: list index out of range
contacts2.db is present in the yaffs folder.
Any ideas/suggestions would be greatly appreciated!
This doesn't work anymore, just puts 0 in all of the phone number fields, you can even see the numbers before hand, just jumbled up, so this script needs to be updated.
Stillworks July 2020
This worked in July 2020 to restore SMS messages (I didn't need contacts). I tried to restore selectively from a full OrangeFox (OF) NANDroid backup (a full restore didn't work, encryption/password issues) with Titanium Backup (TB), but this didn't work (maybe I simply didn't know what to restore?). I tried the old Nandroid Manager's facility to restore SMS from a NANDdroid backup, but it hung.
I then successfully extracted mmssms.db as described here and further extracted the xml file with the yaff extractor. TB's "Restore data from XML" hung, but SMS Backup & Restore worked fine.
Xiaomi Redmi Note 7 (lavender), all software current as of July 2020: xiaomi.eu 20-07-23 weekly ROM, TB 8.4.0.2, others downloaded today. A useful way to find the SMS file if it has a different name than mmssms.db: after extracting from the tar.gz, search the extracted files for a string that only appears in your SMS (I used "bekonscot").
Incidentally, a note on Titanium Backup, which can restore either from its own files or a NANDroid backup: the directory for the NANDroid file is hard-wired to the directories used by TWRP and CWM. I use the current OrangeFox instead of TWRP, and it uses a different directory. But the OF backup is compatible with TWRP; moving it to /storage/emulated/0/TWRP/BACKUPS/ works.
HTH
I used a Note 4 (Android 6) full nandroid to Note9 (Android 10). For contacts I just copied the .db to the new location (the directories where .db are located might not be the same) and rebooted - it worked (on rooted), but this will remove any contacts already made, replacing them with the backup.
SMS and Call logs were restored as described, but I used a newer version of the app (which combines both SMS and logs restoration), searched the backup archives with ZArchiver, and I ran the python2 version of the converter (https://github.com/abbot/android-restore-tools) on TermuxArch (meaning that, unlike the Windows version in the tutorial, this should work on any OS supporting python2, allowing me to do all steps on my phone).
Also, in the directories where. db files are located there are big .db-wal files, too, which might contain some data that aren't in the .db yet, so you might need to VACUUM with sqlite3 (https://stackoverflow.com/questions...f-sqlite-3-7-wal-file-into-main-database-file). I did it on TermuxArch as well, but the size of the db didn't change much, while the wal disappeared, so I don't know if that did anything.
P.S. Sqlite3 wouldn't open db at all just on Termux for some reason, opened on TermuxArch. Python2 also exists for Termux, didn't try but should work.
NOTE THIS IS NOT "UNABLE TO BACK UP" THIS IS "UNABLE TO RESTORE"
Lots of people on this forum, and lots of people on the internet have errors trying to back up because of this dreaded error 255. But, for me, the backup was fine, but now I can't restore. This is a huge problem because I just tried out a custom ROM (which turned out to be a huge fail), and I want to go back to stock.
So I switched back to the other slot (A/B ftw), wiped all data, formatted SD card, set up the OS to have the same lock pattern as the backup, transferred the backup onto the phone and tried to restore, and, well, fuck.
Code:
[RESTORE STARTED]
Restore folder: '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR'
Skipping Digest check based on user setting.
Calculating restore details...
Restoring 4 partitions...
Total restore size is 11276MB
Restoring Vendor Image...
[Vendor Image done (3 seconds)]
Restoring Boot...
[Boot done (0 seconds)]
Wiping Data (excl. storage)
Wiping data without wiping /data/media ...
Done.
Restoring Data (excl. storage)...
extractTarFork() process ended with ERROR: 255
It failed, and there is absolutely nothing I can do about it. TWRP is using buggy commands to extract the backup and I don't know what's triggering the bug.
As a result, right now all my data is in limbo. TWRP cannot restore this backup as-is. I can't just go back in time and use `pm remove-user 999` or `pm remove-user 10` because the backup has already been created. Any solution that has to be done before the backup phase is inaccessible to me now.
My hope is that some smart person here can help me find a way to get this backup back onto the phone. Right now the internet does not have a solution to this problem, no matter how many naive users spam discussion boards with "pm remove-user 999 totally worked". Serious answers only.
Relevant section in the recovery log:
Code:
==> extracting: //data/vendor_de/10/ (mode 40771, directory)
Cannot find key for 10
error looking up proper e4crypt policy for '//data/vendor_de/10/' - 1DE10
tar_extract_file(): failed to extract //data/vendor_de/10/ !!!
I:Unable to extract tar archive '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR/data.ext4.win000'
Error during restore process.
I:Error extracting '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR/data.ext4.win000' in thread ID 0
I:Error extracting split archive.
Error during restore process.
pigz: write error code 32
pigz: abort: write error on <stdout>
extractTarFork() process ended with ERROR: 255
I suppose the solution is to make it "find a key for 10", but how? Do I just need to somehow recreate the 10 user before restoring the backup?
EDIT NOTICE: THIS DIDN'T WORK, DON'T MAKE IT WORSE BY TRYING
ORIGINAL POST IS BELOW
Alright. Well, I found a solution. That was quick.
The error is triggered by the backup containing a user that isn't present during the restore, I think. One solution is (theoretically) to create the missing user... another solution is to remove the missing user from the backup.
Turns out, backups are not so hard to modify, even in their compressed form. However, it is a little bit unintuitive at first.
You can't use the `tar` command to modify the backup, at all. TWRP's tar doesn't even support the --delete flag, nor does the BSD tar command (what ships with macOS). Even after installing GNU tar, it still refuses to work on a compressed archive (probably for the better).
You either need a Windows system with WinRAR/7zip (allows opening/modifying compressed files without extracting them), or a macOS/Linux system with FUSE and archivemount. If you're on Windows, you will have to manually search every single folder for 10 or 999 folders and delete every single one. If you're on macOS/Linux, the process is a bit more technical but also a bit more automated. And in both cases you will have to do this many times, and it can easily take half an hour or more and many restore attempts.
First and foremost, you need the recovery log to find out where the process failed. Every time it fails you will need to go to Advanced > Copy Log, copy them to the SD card, then use the flipping horrible Android File Transfer program to transfer the recovery.log to your PC, then use the cat/tail command or a text editor of your choice to read the log file.
Somewhere around the end of that file you will find something that looks like this:
Code:
==> extracting: //data/misc_de/10/ (mode 41771, directory)
Cannot find key for 10
error looking up proper e4crypt policy for '//data/misc_de/10/' - 1DE10
tar_extract_file(): failed to extract //data/misc_de/10/ !!!
I:Unable to extract tar archive '/data/media/0/TWRP/2020-11-08--22-06-46 PRE-RR/data.ext4.win003'
This contains all the info you need: the archive where the tar command gave up, and the file it gave up on.
Destroy that file with extreme prejudice.
Make a mount point with `mkdir mnt`, and mount the archive with `sudo archivemount data.ext4.win003 mnt`. (Of course, substitute data.ext4.win003 with the culprit archive file.)
If you're like me and have this problem with 10 folders, or 999 folders, you can find every single one of its kind with `sudo find mnt -name 10`. You can then use `sudo rm -rf` to delete every one of them. If you're brave, try `sudo find data -name 10 -exec sudo rm -rf \{\} \;`. I'm not responsible for any data losses incurred by using powerful commands that don't prompt you for confirmation.
Notice how I'm repeatedly using `sudo` for this: FUSE is kind of quirky and so is the kernel about multi-user filesystems. If you don't use sudo to mount the archive, then the user won't have permission to represent files it doesn't own, even if root is the one asking. You'll get weird "No such file or directory" errors for files and directories that are contained in parent directories' file listings. If you don't use sudo for the find or rm commands, you won't have permission to look inside folders or delete folders. Really, do everything with sudo because it's the only way to do this correctly.
Once you're done scrubbing the problem files away (be careful, do not use a file explorer of any kind, especially macOS's, which generally craps .DS_Store files all over your filesystem), unmount and save the archive with `sudo umount mnt`.
The next part is very important because recompressing files is not instant.
You will need to run `watch -n 1 du -hs data.ext4.win003` and WAIT.
Watch the file size slowly grow; only continue once it has stayed the exact same for at least one minute. Yes, it can definitely stop for 30 seconds at a time and then resume. You will need to give it time to recompress. It's also probably normal for the recompressed file to be about half the size of the initial file; that's just an artifact of TWRP not using very good compression to begin with. archivemount will create a backup of the archive with .orig appended, so you can try again if your backup ends up getting corrupted. (I just created an entirely separate copy of the backup to use archivemount on)
Of course, if you're on Windows you can skip all this and simply delete the problem file from WinRAR/7zip, and it'll be gone instantly and you can just proceed to copy the archive over to your device and retry the backup. You will have to do all the searching manually though, or perhaps you could try your luck with WSL. (Does WSL support FUSE?)
Expect this process to fail multiple times, but all at different points; whenever it fails at a different file, you know it got past the one you just removed. all you need to do is repeat this process over and over, slowly fixing your backup until it restores properly. This is what needs to be done if you can't reproduce the event that created that user in the first place. I'm sure if the user is important, it'll get created again.
Also, looks like user 10 is what happens when Google Play updates Device Personalization Services. Is this the generic "updates for system apps" user or is it specifically for that app?
EDIT: But updating that app doesn't create user 10. Interesting...
EDIT 2: User 10 is the guest user!! All I needed to do was create user 10 and that allowed me to restore my backup!
My post above is still good to see what I tried at first but it actually bricked my device until I factory reset it again, so :/
LoganDark said:
Also, looks like user 10 is what happens when Google Play updates Device Personalization Services. Is this the generic "updates for system apps" user or is it specifically for that app?
EDIT: But updating that app doesn't create user 10. Interesting...
EDIT 2: User 10 is the guest user!! All I needed to do was create user 10 and that allowed me to restore my backup!
My post above is still good to see what I tried at first but it actually bricked my device until I factory reset it again, so :/
Click to expand...
Click to collapse
Glad you managed to get your device back!
I had the same problem on another phone. The reason I got the error was because I tried to restore a decrypted backup on an encrypted device. After choosing the option "Format" in TWRP and typing 'yes' to confirm the decryption process, I was able to restore my backup.
Im having the same error in my realme xt...
sir..please help me.. I have a twrp backup of system image vendor image data boot recovery persist but im unable to recover because it shows an "error extract tar fork process ended up with error 255".during backup i had a pincode for screenlock.
i also have a backup of my internal storage in twrp but im not able to restore because of this error
LoganDark said:
Also, looks like user 10 is what happens when Google Play updates Device Personalization Services. Is this the generic "updates for system apps" user or is it specifically for that app?
EDIT: But updating that app doesn't create user 10. Interesting...
EDIT 2: User 10 is the guest user!! All I needed to do was create user 10 and that allowed me to restore my backup!
My post above is still good to see what I tried at first but it actually bricked my device until I factory reset it again, so :/
Click to expand...
Click to collapse
Hi. Did you create user 10 in Source, or in destination?
How did you do that in source?
If in destination, then is it possible to do it via TWRP?
EDIT: disregard. Mine user in the error path is 0 :-( '//data/system_de/0/'
just use option format data when entering wipe selection
erosman23 said:
just use option format data when entering wipe selection
Click to expand...
Click to collapse
doesnt work
redmi k20 davinci