So a while ago I grabbed a Motorola G2, made a complete physical memory dump to file “Full_Phone_Backup.image” I also have a .vhd which will not mount, but I remember very clearly extracting the contacts list from it.
I didn’t keep records of how I went about doing it, but I’m sure it was some busybox or cygwin and I’m very sure the file contains all the partitions. Don’t know what happened to .vhd, but I also can’t make a new one either, so bad luck I guess. Anyway, now I need to extract SMS messages from it, and so I turned to Foremost. With the following in a foremost-db.conf:
Code:
db n 4000000 \x53\x51\x4c\x69\x74\x65\x20\x66\x6f\x72\x6d\x61\x74\x20\x33\x00
I hoped to get a good rundown of the databases and extract from there. Now, I don’t need deleted messages or anything, I need to find the SMS messages that would be there as if the phone was on right now. Well, that and search them.
So, Foremost carved a whooping 36 GiB of .db files, which I think suggests that the file header is actually not for SQLite 3 file as much as for a part of SQLite3 file, and one file may contain many. Secondary evidence of that is that massive number of files contain the same data shifted a more or less uniform number of lines (like an entire block shifted 1k lines down across 10 files before it completely disappears).
Anyway, from the carved DB files I got meaningful e-mail messages, from the carved photos and videos, I got meaningful pictures (ones which would be on this phone), so I am sure data is there. Problem is, I could not find the messages I’m looking for. I was unable to find a single SMS message, I was unable to browse any of the recovered databases with sqlite database browsers, and the most useful thing I was able to do was to use Ransack in windows to search for relevant text in those recovered files.
What am I doing wrong that I’m not finding SMS messages or any relevant text in this mess?
KYKYLLIKA said:
So a while ago I grabbed a Motorola G2, made a complete physical memory dump to file “Full_Phone_Backup.image” I also have a .vhd which will not mount, but I remember very clearly extracting the contacts list from it.
I didn’t keep records of how I went about doing it, but I’m sure it was some busybox or cygwin and I’m very sure the file contains all the partitions. Don’t know what happened to .vhd, but I also can’t make a new one either, so bad luck I guess. Anyway, now I need to extract SMS messages from it, and so I turned to Foremost. With the following in a foremost-db.conf:
Code:
dbn 4000000\x53\x51\x4c\x69\x74\x65\x20\x66\x6f\x72\x6d\x61\x74\x20\x33\x00
I hoped to get a good rundown of the databases and extract from there. Now, I don’t need deleted messages or anything, I need to find the SMS messages that would be there as if the phone was on right now. Well, that and search them.
So, Foremost carved a whooping 36 GiB of .db files, which I think suggests that the file header is actually not for SQLite 3 file as much as for a part of SQLite3 file, and one file may contain many. Secondary evidence of that is that massive number of files contain the same data shifted a more or less uniform number of lines (like an entire block shifted 1k lines down across 10 files before it completely disappears).
Anyway, from the carved DB files I got meaningful e-mail messages, from the carved photos and videos, I got meaningful pictures (ones which would be on this phone), so I am sure data is there. Problem is, I could not find the messages I’m looking for. I was unable to find a single SMS message, I was unable to browse any of the recovered databases with sqlite database browsers, and the most useful thing I was able to do was to use Ransack in windows to search for relevant text in those recovered files.
What am I doing wrong that I’m not finding SMS messages or any relevant text in this mess?
Click to expand...
Click to collapse
You can't get into the phone and use SMS backup app or PC to backup your SMS? Can you back them up to your Google account and then recover them from there?
There are several ways to recover SMS from a device, is this method the only one you've tried?
Are you recovering your SMS or someone else's?
Sent from my SM-S903VL using Tapatalk
KYKYLLIKA said:
So a while ago I grabbed a Motorola G2, made a complete physical memory dump to file “Full_Phone_Backup.image”
.
.
and I’m very sure the file contains all the partitions.
Click to expand...
Click to collapse
How did you make this Backup.image? In my question over here I dreamt of something like 'dd if=/dev/sda of=/image.bak' for a really complete backup. Did you make your complete memory dump in such a way and would i be possible to write it back to the phone?
Thanks!
Droidriven said:
You can't get into the phone and use SMS backup app or PC to backup your SMS? Can you back them up to your Google account and then recover them from there?
There are several ways to recover SMS from a device, is this method the only one you've tried?
Are you recovering your SMS or someone else's?
Sent from my SM-S903VL using Tapatalk
Click to expand...
Click to collapse
This is my sister’s phone. The backup is from 16 months back or so. I can’t go search the SMS it has now, because updates been around since then, including a new version of android, and all that. It does not have the old messages.
What I tried is a sqlite forensics utility called “sqlite forensics reporter”, but no luck with that either. Piriform recuva did not work, and I was unable to mount it as a virtual hard drive or find a part of it that I could mount as a virtual hard drive.
andy_ross said:
How did you make this Backup.image? In my question over here I dreamt of something like 'dd if=/dev/sda of=/image.bak' for a really complete backup. Did you make your complete memory dump in such a way and would i be possible to write it back to the phone?
Thanks!
Click to expand...
Click to collapse
This is entirely possible. However, I cannot find the boot record in there or a partition table, which is very strange. I do remember dumping all the partitions in that file, though. It’s been over a year now, so details kind of gone fuzzy. I am sure I could write it back to a phone and use like that, but I don’t have a suitable surrogate phone or a virtual machine to try that on. I will not do it on the device itself, seeing as how it’s in use and all that. I just want to find the text of some messages.
Related
Playing around with the ls -R command, I found where the SMS database is kept, and it's somewhat readable in a text editor...
UPDATE (thanks staulkor): This database is viewable, and searchable using an SQLite database viewer.
Code:
/data/data/com.android.providers/telephony/databases/mmssms.db
Just run the following command to back it up to the SD Card:
Code:
busybox cp /data/data/com.android.providers.telephony/databases/mmssms.db /sdcard
Happy hacking
You can use an SQLite viewer to read the database correctly
Would seem simple enough to write a script to run on a PC/Mac to back up the database and load it into some sort of viewer... searchable perhaps?
I'm not too familar with databases, but would seem simple enough to make a lightweight app to keep the SMS archive...
Cool, maybe we'll soon see a port of PIM Backup. Would be nice...
Hmm...
/data/data/com.android.providers/telephony/databases/mmssms.db
Click to expand...
Click to collapse
and
busybox cp /data/data/com.android.providers.telephony/databases/mmssms.db /sdcard
Click to expand...
Click to collapse
are two different paths - I think both are supposed to be
Code:
/data/data/com.android.providers.telephony/databases/
(since I can't cd to the other one).
At any rate - thanks for the tip, but can anyone else confirm this works for them? On my android (R30) I get "opendir failed, Permission denied" if I try to read the contents of that directory.
Seeing as it's fully possible to write an application to back up the database to the sdcard (or even to a website), wouldn't that make more sense?
Afterall, there's no need to have the actual database -- and the method is highly restrictive to both people with root on their phones and those with a USB cable and computer handy.
Also, backing up this database will not backup any MMS attachments, which are stored here:
/data/data/com.android.providers.telephony/app_parts/
Refer to the "part" table for an index matching the files there to messages/filenames. (For each MMS, there should be an XML file that helps identify the type of message, and then the file itself, such as a jpeg).
is this the best and safest way to back up sms database as of yet? I have about 1000 sms and it slows down my messaging so I've been try to find a way to free that up. Backing them up would be great so i can clear them.
If anyone has tried this, please share with the rest of us on its effect on the phone.
Thank you,
Hbguy
P.S: This site is truely GREAT because pros are patient with noobs
i don't understand this...i downloaded the file managers on marketplace, and cannot when i click on data file, there is nothing in there..so i dowonloaded a terminal program and still don't understand how to use the commands to move sms from memory to sim...can u do a quick step-by-step? do i have to use any specific programs to do this? sorry, so used to the simplicity of WM6 on my old phone, having trouble adapting to this platform..thanks for any help.
jazzmanmonty said:
i don't understand this...i downloaded the file managers on marketplace, and cannot when i click on data file, there is nothing in there..so i dowonloaded a terminal program and still don't understand how to use the commands to move sms from memory to sim...can u do a quick step-by-step? do i have to use any specific programs to do this? sorry, so used to the simplicity of WM6 on my old phone, having trouble adapting to this platform..thanks for any help.
Click to expand...
Click to collapse
You need to have root on your phone to follow the instructions given above (if you don't know what that means, you don't have it). For non-root users, most of the Android system on the G1 is locked down -- including /data and all its subfolders.
If you do have root on the phone, you can follow the instructions at the top from a terminal while logged in as root. You can verify you are root by running the program "id":
# id
uid=0(root) gid=0(root)
From there, to copy the SMS/MMS database to your sdcard, you can follow the above instructions. If you don't have busybox (ie, "busybox: not found"), you can use the "cat" command as so:
cat /data/data/com.android.providers.telephony/databases/mmssms.db > /sdcard/mmssms.db
You may also want any files stored here:
/data/data/com.android.providers.telephony/app_parts/
Which are from MMSs (if you've no MMSs saved, then you've no need to bother).
Also, not sure if there is a clarification needed for you. The instructions provided offer no way to access SMSs stored on the SIM, as Android uses a PDU interface that doesn't use the SIM's SMS storage space. It is possible to read SMSs off the SIM, however, and into the internal memory storage. See the settings window in the messaging application.
If someone does take the time to create an application to do this, then it would not require root access. Applications are allowed to access the SMS/MMS database, provided the user grants them permission, and could then be used to read the data from the database and into a new database or other file (eg, csv).
do ya think it would be posible to move these permanently to the memory card? ummmm.... ill try if i get a chance ..... ya can connect to phone via adb in recovery mode right ? an undo whatwver i break?
how would you view your backed up sms messages using staulkors sql database?
step by step would be great
hbguy
Just to be clear - for most people on RC30, this probably will not work.
Per http://forum.xda-developers.com/showthread.php?t=442480 the hacks to gain root access have been blocked since RC30 was deployed in early November. Someone should write an application to support this, but instead everybody is busy creating new useless calculators and niche geolocation applications.
If I'm wrong about the above, it would be terrific for anyone to chime in and clarify that.
goodwinm said:
Cool, maybe we'll soon see a port of PIM Backup. Would be nice...
Click to expand...
Click to collapse
https://github.com/jberkel/sms-backup-plus
dear all!!!
really it's so hard to restore many pimbackup.pib files on a device...
i have been searching for an extracor, or kind of emulator that alow me to view the backuped file on my pc...
thankyou
best regards...
moiisse
If I remember correctly, you can rename the .PIB file as a .ZIP and then look at it with your "favourite archive tool".... Within the ZIP/PIB file there are loads of CSV format files for each of the messages/contacts/phone logs etc.
The only problem may/might/will be if you did the PIM backup in binary format; your best bet would be to post on the PIMbackup thread (here) and see if Dotfred or anyone else can help
Cheers,
Mark.
Dear Mark
I have tried this trick but the files I got were not .CSV one some other extension files. I am afraid if I am doing anything wrong.
The extension is not .CSV, but they are in CSV format. If you rename them to .CSV and open them with Microsoft Excel or Openoffice Calc (or whatever), they can be imported as semicolon delimited files (so strictly speaking, they're SSV - "Semicolon Separated Values" rather than CSV )
Hope that helps,
Mark.
That was a great tip!
Any idea how to import these into an Android Phone - just got an HTC Hero today. Took a backup of my existing contacts, sms etc from WM6.5 (HTC Touch HD) and could see the .pib (renamed as .zip) and individual files (which I renamed as .csv)..
Guess there might be a simple trick to restore them as well in HTC Hero?
I think I may have found something........
sam_htc_touch said:
Any idea how to import these into an Android Phone - just got an HTC Hero today. Took a backup of my existing contacts, sms etc from WM6.5 (HTC Touch HD) and could see the .pib (renamed as .zip) and individual files (which I renamed as .csv)..
Guess there might be a simple trick to restore them as well in HTC Hero?
Click to expand...
Click to collapse
I was about to say I have no idea and I don't really care (about Android phones that is, not your problem getting your messages onto your Hero!), but then.......
I found this - Sprite Software's FREE (at the moment) migration software - which looks like the answer to your prayers
I've seen a report from someone who used it - apparently, it's pretty slow, but as it's a "one-time" transfer operation.....who cares how long "free" takes!!!???
Let us know how you get on - and post something in the Android-related forums so that other users can benefit.....
Cheers,
Mark.
(At times like this, I wish I had a Paypal "donate" button.....this gem of wisdom is surely worth a buck/quid/euro or two ....Ho hum!!!)
Excellent! That seems to be promising!
Just tried using that and tried to restore only the sms on my android, however it gave an error. Have emailed sprite software the relevant log file, lets see if they come up with the reason of failure. Guess it might have something to do with the threaded view on my win mob 6.5
PS : Paypal donate button shudn't be that difficult to get
And yes just thought that I would mention about the contacts - its actually very easy - just get them into csv file using the trick that you mentioned above and then import the csv file in gmail contacts! that's it! gmail automatically syncs the contacts thereafter!
Mark Crouch said:
I was about to say I have no idea and I don't really care (about Android phones that is, not your problem getting your messages onto your Hero!), but then.......
I found this - Sprite Software's FREE (at the moment) migration software - which looks like the answer to your prayers
I've seen a report from someone who used it - apparently, it's pretty slow, but as it's a "one-time" transfer operation.....who cares how long "free" takes!!!???
Let us know how you get on - and post something in the Android-related forums so that other users can benefit.....
Cheers,
Mark.
(At times like this, I wish I had a Paypal "donate" button.....this gem of wisdom is surely worth a buck/quid/euro or two ....Ho hum!!!)
Click to expand...
Click to collapse
Hi Mark (Am I Noob to Android)
It seems like, you have solved the issue here. I did have one question for you:
What if you allready had a PIM back of your old phone. Then the phone you have sold it. But you still the the PIM backup file on your SD card. Can you still go on to follow and start from 2 step and then step 3.
Or is the another way of doing this then?
Many Thanks.
yes u should be able too as long as the PIM file is on your SD card. you can just directly transfer it onto your PC, then rename it to a ZIP file and proceed as instructed above.
sam_htc_touch said:
Any idea how to import these into an Android Phone - just got an HTC Hero today. Took a backup of my existing contacts, sms etc from WM6.5 (HTC Touch HD) and could see the .pib (renamed as .zip) and individual files (which I renamed as .csv)..
Guess there might be a simple trick to restore them as well in HTC Hero?
Click to expand...
Click to collapse
tried it..
i just took the contacts.csv file ,tried to extract with gmail...
then
An unknown error occurred while importing your contacts.Learn more
.....oohhhh.....really needed help
Hello to everyone!
I had an HTC Touch HD and after a serious accident it's officially dead! I had a backup of contacts and messages to my sd card and i did what is described. I renamed it to .zip and i extracted it to my pc.. The result i get is files .pba, .pbl, .pbc, .pbx, .pbd and .pbt
Does anyone know what is all that and how i can use them in android phone????
please help me!
isovitis33 said:
Hello to everyone!
I had an HTC Touch HD and after a serious accident it's officially dead! I had a backup of contacts and messages to my sd card and i did what is described. I renamed it to .zip and i extracted it to my pc.. The result i get is files .pba, .pbl, .pbc, .pbx, .pbd and .pbt
Does anyone know what is all that and how i can use them in android phone????
please help me!
Click to expand...
Click to collapse
that is what I got also, I tried importing data to excel (with / without renaming to csv or ssv) but the text was not understandable in any language so I did not even proceed in selecting the separator and actually import the data.
Mr Black'd said:
that is what I got also, I tried importing data to excel (with / without renaming to csv or ssv) but the text was not understandable in any language so I did not even proceed in selecting the separator and actually import the data.
Click to expand...
Click to collapse
Come on guys read a little more on the web, if you would have done so you would have figured out you have the binary backup files, not the text file, so what you need to do is convert. Now if you press the search button for the forum you would have found this interesting link:
http://forum.xda-developers.com/showthread.php?t=424641
the title says: 'Convert PIM Backup binary file to text without using windows mobile device'
(considering yours broke like mine)
You can read there that you can convert it to a text file using an emulator on your pc and subsequentlyfollow the steps that were described here...
First of all what makes you think that none has bothered to use search button or at least Google the issue?
Second and last ok you have proved us stupid.
Anyway thanks for the link even though a late reply cause personally I have solved my issues.
Sent from my HTC HD2 using XDA app.
This week I finally got around to backing up my text messages and clearing the text message cache on my HTC Touch Pro 2 (Verizon). (my God, it had become slow) I'm running WM 6.1 and it seems that the MMS text messages are stored separately in proprietary ".dat" files in the "MMS UA" folder in My Documents. I searched around xda-dev and couldn't find any way to extract the files contained within. From what I understand, they are referred to as "Arcsoft MMS".
Anyway, the file format is somewhat obfuscated so a true extractor wasn't possible but I was able to produce a tool that could extract all the files out of my .dat files. It is a total hack-job, but there don't seem to be any options. Like I said, I searched around for a few minutes and couldn't find anything else so I am now posting to ask the following:
Does anyone want the tool?
If there is no other extractor out there, I will post it to this thread. Also, anyone who has some Arcsoft MMS files handy that don't contain any private information could help by sending me a copy if the tool can't handle them. With some input, it could possibly become a valuable tool for the community.
Lend me your input.
Thanks
Sounds awesome, please post the tool!
Yes, would appreciate this as well - so many devices and MMS are cluttered among all of them.
Cool. Well here it is guys.
Remember that I couldn't completely discover the proprietary format, and although it works on all of my attachments, it may not work on all of yours. It is easy to update the tool, though, so just message me.
metalliqaz said:
this week i finally got around to backing up my text messages and clearing the text message cache on my htc touch pro 2 (verizon). (my god, it had become slow) i'm running wm 6.1 and it seems that the mms text messages are stored separately in proprietary ".dat" files in the "mms ua" folder in my documents. I searched around xda-dev and couldn't find any way to extract the files contained within. From what i understand, they are referred to as "arcsoft mms".
Anyway, the file format is somewhat obfuscated so a true extractor wasn't possible but i was able to produce a tool that could extract all the files out of my .dat files. It is a total hack-job, but there don't seem to be any options. Like i said, i searched around for a few minutes and couldn't find anything else so i am now posting to ask the following:
Does anyone want the tool?
If there is no other extractor out there, i will post it to this thread. Also, anyone who has some arcsoft mms files handy that don't contain any private information could help by sending me a copy if the tool can't handle them. With some input, it could possibly become a valuable tool for the community.
Lend me your input.
Thanks
Click to expand...
Click to collapse
thanks for this mate...s
Thanks, works - not on all, though. Will check further if there is a common cause for the failing ones.
Improvement proposal:
1.) touch the extracted files with the timestamp of the .dat file
2.) check the content of the smil file in the body section if there are elements supposed to be present. afaik, the .smil is a tag-language which can deliver the MMS content with an appropriate viewer
Thanks for the quick feedback.
Improvement #3... GUI
As for the .smil file, you are absolutely correct. However, the issue is not knowing that there are files present, but finding the boundaries of the data. Arcsoft's binary packed format uses header bytes to describe how data is stored. Unfortunately, even though I was able to determine many of the header fields, without documentation I have no idea what they mean or what others may be used in a given archive. It does seem to depend on what kind of phone sent the MMS. Still, I think you may have a good idea going. I could perform a second pass after extracting the .smil, which is just XML, using file names and a lazy search.
One thing I didn't have on my phone was an MMS containing a movie file. I should try that out...
This is a great tool!
It works under desktop Windows.
How about a tool that can be used in WM itself? Anyone thought of that?
Hi,
So I have a VX6700 w/WM5 which the screen broke on yesterday. I was able to extract the contracts from the pim.vol file, however I do not see my SMS Messages. So here is my question and some of the directions I think this can go. I want to be able to extract the SMS Messages, but I have yet to find a file that they are located in nor hear anyone talk about where exactly.
So, is there db file I can grab and export
Or, is there a way I can run SMSBackup or another tool remotely on there? I can't see the screen but I can of course still communicate with it over the data cable, so if there was a program I could run remotely without needing to click on anything, I'm in the good. I don't really know much about WM but I think both of these should be possible.
Thanks!
Edit: so it looks like the cemail.vol file is what I need, but it is protected by the OS. And it looks like just about every tool I've came across needs to be ran on the phone, so I will likely need to be able to do some remote execution somehow.
Hello,
I want to merge 2 Whatsapp Backups. I don't know anything about this but I am good in reading tutorials. But unfortunately I couldn't find any tutorials helping me to merge Whatsapp Backups. Can anyone help me? I've found out that my Backups are crypt8 files. Is this a problem?
I can not give you a definitive answer but I could get you some information that might help you.
First off you would need Root access to be able to grab the non-encrypted Databasefiles.
You might be able to decrypt the current database format (crypt8?). However they changed that quite a few times and I didn't keep up with what they changed, maybe try searching here on xda for that.
!!If you don't want to loose your messages I suggest to backup the files somewhere just in case!!
Now, Whatsapp uses two main database files: msgstore.db - for the actual messages and wa.db for the contacts.
Now what you would need to do in theory is to export the messages from one database (the first msgstore.db) and import them into the second one (the new msgstore.db), while making sure that the _id has not been used yet. (this field should be on a auto-increment if I remember correctly)
I hope you have some knowledge of databases?
You could try to dump the Database and then reimport it (http://stackoverflow.com/questions/75675/how-do-i-dump-the-data-of-some-sqlite3-tables)
Good luck
if you have any further questions, i'll try to answer them as good as I can