Database Info

HOWTO: Unpack, Edit, and Repack Boot Images

Several people have already figured out the details on their own, but I have gotten requests to do a more comprehensive tutorial on how the boot and recovery images are structured, and how you can edit them.
Background
Your phone has several devices which hold different parts of the filesystem:
Code:
#cat /proc/mtd
dev: size erasesize name
mtd0: 00040000 00020000 "misc"
mtd1: 00500000 00020000 "recovery"
mtd2: 00280000 00020000 "boot"
mtd3: 04380000 00020000 "system"
mtd4: 04380000 00020000 "cache"
mtd5: 04ac0000 00020000 "userdata"
In this tutorial, we will deal with "recovery" and "boot". The "boot" device holds the files that are automatically loaded onto the root of your filesystem every time you boot (details below).
"system" holds everything that gets mounted in your system/ directory, and userdata/ is everything that shows up in data/ (this is all the apps you've installed, your preferences, etc).
The recovery and boot partitions are at /dev/mtd/mtd1 and /dev/mtd/mtd2, and before you do anything else you should back these up (note: this may not be the best way of doing this because it may not deal properly with bad blocks etc, but it's all we've got until somebody comes up with a better method, and besides you will probably be restoring from update.zip anyway):
Code:
# cat /dev/mtd/mtd1 > /sdcard/mtd1.img
# cat /dev/mtd/mtd2 > /sdcard/mtd2.img
The other thing you should do is put your favorite update.zip file into the root directory of your sd card so that if you screw up your boot partition you can boot into recovery mode and re-apply the update. You probably want one of the pre-rooted recovery images found elsewhere on the forums.
There is also another important file you should know about. In /system/recovery.img there is a full copy of everything that is loaded on mtd1. This file is automatically flashed onto mtd1 every time you shut down. That means two things: 1. Any changes you make directly to /dev/mtd/mtd1 get blown away on reboot and 2. If you want to change /dev/mtd/mtd1 you're probably better off just sticking the image in /system/recovery.img and rebooting. When creating your own custom update.zip files (especially when adapting the stock images), you can get tripped up if you forget to replace /system/recovery.img and it ends up overwriting /dev/mtd/mtd1 unbeknownst to you. Watch out.
Structure of boot and recovery images
The boot and recovery images are not proper filesystems. Instead, they are a custom android format consisting of a 2k header, followed by a gzipped kernel, followed by a ramdisk, followed by a second stage loader (optional, we have not seen these in the wild yet). This structure is outlined in mkbootimg.h:
Code:
+-----------------+
| boot header | 1 page
+-----------------+
| kernel | n pages
+-----------------+
| ramdisk | m pages
+-----------------+
| second stage | o pages
+-----------------+
n = (kernel_size + page_size - 1) / page_size
m = (ramdisk_size + page_size - 1) / page_size
o = (second_size + page_size - 1) / page_size
0. all entities are page_size aligned in flash
1. kernel and ramdisk are required (size != 0)
2. second is optional (second_size == 0 -> no second)
A ramdisk is basically a small filesystem containing the core files needed to initialize the system. It includes the critical init process, as well as init.rc, which is where you can set many system-wide properties. If you really want to know more about it, here is the documentation. Here's a list of files on a typical ramdisk:
Code:
./init.trout.rc
./default.prop
./proc
./dev
./init.rc
./init
./sys
./init.goldfish.rc
./sbin
./sbin/adbd
./system
./data
The recovery image typically has a few extra files, which constitute the recovery binary and supporting files (the application that gets run if you hold down home+power when rebooting). These files are:
Code:
./res
./res/images
./res/images/progress_bar_empty_left_round.bmp
./res/images/icon_firmware_install.bmp
./res/images/indeterminate3.bmp
./res/images/progress_bar_fill.bmp
./res/images/progress_bar_left_round.bmp
./res/images/icon_error.bmp
./res/images/indeterminate1.bmp
./res/images/progress_bar_empty_right_round.bmp
./res/images/icon_firmware_error.bmp
./res/images/progress_bar_right_round.bmp
./res/images/indeterminate4.bmp
./res/images/indeterminate5.bmp
./res/images/indeterminate6.bmp
./res/images/progress_bar_empty.bmp
./res/images/indeterminate2.bmp
./res/images/icon_unpacking.bmp
./res/images/icon_installing.bmp
./sbin/recovery
Unpacking, Editing, and Re-Packing the images
Note: below I give you the details for unpacking and repacking manually, but I have attached two perl scripts that do most of this for you
If you are good with a hex editor, you can open up any of these images and strip off the first 2k of data. Then, look for a bunch of zeroes followed by the hex 1F 8B (which is the magic number of a gzip file). Copy everything from the first line of the file, through the zeroes, and stopping at the 1F 8B. That is the kernel. Everything from the 1F 8B through the end is the ramdisk. You could save each of these files separately. In order to see the contents of the ramdisk, you need to un-gzip it and then un-cpio it. You could use a command like this (ideally after creating a new directory and cd'ing into it):
Code:
gunzip -c ../your-ramdisk-file | cpio -i
That will place all of the files from the ramdisk in your working directory. You can now edit them.
In order to re-create the ramdisk, you need to re-cpio them and re-gzip those files, with a command like the following (remember, cpio will include everything in the current working directory, so you probably want to remove any other cruft you might have in there):
Code:
find . | cpio -o -H newc | gzip > ../newramdisk.cpio.gz
The final step is to combine the kernel and your new ramdisk into the full image, using the mkbootimg program (which you should download and compile from the git repository):
Code:
mkbootimg --cmdline 'no_console_suspend=1 console=null' --kernel your-kernel-file --ramdisk newramdisk.cpio.gz -o mynewimage.img
Now, there's a lot of hassle in pulling apart files in hex editors and remembering all of these commands, so I wrote unpack and repack perl scripts for you (attached). Hooray.
Flashing your new image back onto the phone
You will probably only ever be flashing boot images directly to the phone, given the fact that /system/recovery.img automatically flashes the recovery device for you (as noted above). If you have created a new recovery image, just stick it in /system/recovery.img and reboot. If you are flashing a boot image, stick it on your phone via adb (a tool included in the Android SDK):
Code:
adb push ./mynewimage.img /sdcard
Then, open a shell to your phone via 'adb shell', get root, and do the following two commands to flash your new boot image:
Code:
# cat /dev/zero >> /dev/mtd/mtd2
write: No space left on device [this is ok, you can ignore]
# flash_image boot /sdcard/mynewimage.img
Reboot.
If your phone starts all the way up, congratulations. If not, you did something wrong and you'll need to boot into recovery mode and apply your update.zip file (reboot while holding down home+power, when you get the recovery screen press alt+L and then alt+S).
Something fun to do with your new found power
If you place a file titled initlogo.rle in the root directory of your boot image, the phone will display this image upon boot (after the "G1" image and before the Android animation). In order to create this file, you need to create a 320x480 image in Photoshop or Gimp and save it as a "raw image" file. You then need to compress that image with the program to565. More details on that here.
This is not the same thing as applying an update.zip
You will see other places on the forums that describe how to create customized update.zip files, as well as update.zip files that people are sharing. For example, there is a recent update.zip which is a modified version of rc30 (with the anti-root aspects disabled). The update.zip files include new boot images, recovery images, and typically replacements for the entire system/ directory as well as other updates. If you are creating a custom boot or recovery image, it is typically a good idea to start with the image distributed with the most recent update you have applied (flashing an image from an older release could have unintended consequences).
Questions?

hooray! you're awesome

Where does boot.img flash? What is the corresponding part of the system?

Dimath said:
Where does boot.img flash? What is the corresponding part of the system?
Click to expand...
Click to collapse
I'm not sure what exactly you mean, but when you do flash_image boot imagefile.img it will write imagefile.img to /dev/mtd/mtd2, which is where your phone looks for the boot files. Did that answer your question?

For your command...
Code:
cat /dev/zero >> /dev/mtd/mtd2
Do you mean
Code:
cat /dev/zero > /dev/mtd/mtd2
?
The idea being that you erase flash in the version with one '>', whereas you... append to the end of a device in the version with two '>'s? I can see the utility of erasing flash with one '>' but appending seems... odd. Am I missing something?

Is this any different than using the Dalvik Debug Monitor (DDMS) file manager found the the Android SDK? I'm able to push, pull, and delete files on my G1 with no problem.

Would this be the only way to rebuild a system app (i.e. Settings.apk) with more debug (Log. to extract via adb logcat over usb), then rebuild the entire system.img, then flash into the G1?

eckzow said:
For your command...
Code:
cat /dev/zero >> /dev/mtd/mtd2
Do you mean
Code:
cat /dev/zero > /dev/mtd/mtd2
?
The idea being that you erase flash in the version with one '>', whereas you... append to the end of a device in the version with two '>'s? I can see the utility of erasing flash with one '>' but appending seems... odd. Am I missing something?
Click to expand...
Click to collapse
I think you're right. I copied that from somebody else's instructions and it certainly seems to make more sense with one '>'. Anybody know for sure?
In any event, this is unnecessary in most cases because flash_image should overwrite the whole thing. The only exception is when you have an identical header on your image to the one that is already on the device. This shouldn't happen in my instructions (mkbootimg creates a header that includes the build timestamp) but I kept the instruction there just for good measure.

andonnguyen said:
Is this any different than using the Dalvik Debug Monitor (DDMS) file manager found the the Android SDK? I'm able to push, pull, and delete files on my G1 with no problem.
Click to expand...
Click to collapse
The adb push command I gave is no different, but you would still have to unpack/repack and flash_image according to my instructions.

I've tried these instructions on RC30 1.3. Basically I extracted, unpacked, and repacked, just to see if it would work. The resultant file is too large; it fails when you run flash_image.

I'm trying to modify my boot image of my ADP1 using the perl scripts, but I receive the following warning while decompressing the ramdisk:
Code:
$ unpack-bootimg.pl mtd2.img
kernel written to mtd2.img-kernel.gz
ramdisk written to mtd2.img-ramdisk.cpio.gz
removed old directory mtd2.img-ramdisk
[B]gzip: ../mtd2.img-ramdisk.cpio.gz: decompression OK, trailing garbage ignored
462 blocks[/B]
extracted ramdisk contents to directory mtd2.img-ramdisk/
Is this warning expected? Is safe to continue?
Also, I've found that the size of the modified packed image is far smaller than the original one
Code:
$ ll mtd2.img mtd2-modified.img
-rwx------ 1 ris ris 2621440 2009-01-07 19:19 mtd2.img
-rw-r--r-- 1 ris ris 1533952 2009-01-07 20:49 mtd2-modified.img
Update: I've tried anyway.
For the record, I've obtained the boot.img using cat, uncompressed with the perl script, modified default.prop, repacked with the perl script.
As you see from the code above, the img file is much smaller (Opening with a hex editor you can see that the end of the original image is full of 0xFF, so I believe it's ok, both the gzip warning and the different file sizes).
Reflashed it from recovery mode, using
fastboot flash boot mt2-modified.img
fastboot reboot
... and worked flawlessly
I'm leaving the coment for future references.
Thanks for the tutorial

[RiS] said:
gzip: ../mtd2.img-ramdisk.cpio.gz: decompression OK, trailing garbage ignored
462 blocks
Is this warning expected? Is safe to continue?
Click to expand...
Click to collapse
Yes, you would expect trailing zeroes, which would give you that error. The trailing zeroes exist in order to pad the image size to the nearest page boundary. They are added by mkbootimg.
[RiS] said:
Also, I've found that the size of the modified packed image is far smaller than the original one
As you see from the code above, the img file is much smaller (Opening with a hex editor you can see that the end of the original image is full of 0xFF, so I believe it's ok, both the gzip warning and the different file sizes).
Click to expand...
Click to collapse
Yep, that's exactly why. Nothing to worry about.
Thanks for the informative clarification.

alansj said:
There is also another important file you should know about. In /system/recovery.img there is a full copy of everything that is loaded on mtd1. This file is automatically flashed onto mtd1 every time you shut down. That means two things: 1. Any changes you make directly to /dev/mtd/mtd1 get blown away on reboot and 2. If you want to change /dev/mtd/mtd1 you're probably better off just sticking the image in /system/recovery.img and rebooting.
Click to expand...
Click to collapse
I'm using an stock ADP1, and the file /system/recovery.img does not exist. Is this expected?
Also, I've found out in JFv.1.31 that the recovery image is in /data/recovery.img (although there is no /data/recovery.img in my ADP1 neither..)

[RiS] said:
I'm using an stock ADP1, and the file /system/recovery.img does not exist. Is this expected?
Also, I've found out in JFv.1.31 that the recovery image is in /data/recovery.img (although there is no /data/recovery.img in my ADP1 neither..)
Click to expand...
Click to collapse
It deletes it after it flashes on the first bootup after you apply the update.

JesusFreke said:
It deletes it after it flashes on the first bootup after you apply the update.
Click to expand...
Click to collapse
Are you refering to /data/recovery.img or /system/recovery.img? or both?

[RiS] said:
Are you refering to /data/recovery.img or /system/recovery.img? or both?
Click to expand...
Click to collapse
I'm referring to /data/recovery.img in JFv1.2 and up (although there was a bug in the RC8 version of JFv1.2 that prevented it from being deleted)

So is there any reason why there is no /system/recovery.img on my ADP1?

When I use the unpack script on JF1.31 boot.img, it prints out like this:
"Could not find any embedded ramdisk images. Are you sure this is a full boot image?"
Any help?

[RiS] said:
So is there any reason why there is no /system/recovery.img on my ADP1?
Click to expand...
Click to collapse
Stock or JFv1.3?

JesusFreke said:
Stock or JFv1.3?
Click to expand...
Click to collapse
"Stock". I've just modified boot img to change default.prop

[Q] NVFlash

So I never used it before, since I thought it would be fairly straightforward. But nope, it turns out more complicated. First I wanted to backup the device, yet when I downloaded the zip file, I only see scripts. How the hell do you run them. Do I replace the files in the original folder and run the download.bat? Videos online show it formatting and installing prime when you run download.bat. After backing up I wanted to dual boot Ubuntu, yet those instructions aren't much more clear. Again do I replace those files in the zip with the original nvflash folder? How do I run the script once I put them in there? The Nvflash.exe doesnt help. And he says to put recovery.img, boot.img, and system.img in the /image folder (for prime), yet I have no idea where those are. NVFlash hasnt been explained well. I see videos online showing how to use it for basic root, but when it comes to anything else, everyone is silent.

Here:
www.androidroot.mobi/2011/06/13/nvflash-on-asus-transformer/
There is a link to the prime rom 1.4 needed too.
1- download nvflash
2-download the primerom 1.4
3-extract both zips
4- in the extracted primerom folder, copy system.img, boot.img, recovery.img
5-paste them in your nvfash folder
6-connect your tablet to ur pc n enter APX mode
7-double click on download.bat(yes it looks like a script)
8-sit and relax and your tablet will boot with thw prime rom.
9-now, boot into recovery (POWER+ DWN) THEN UP. wipe everything and install the rom of your choice.
Sent from my Transformer TF101 using Tapatalk

dude2k5 said:
So I never used it before, since I thought it would be fairly straightforward. But nope, it turns out more complicated. First I wanted to backup the device, yet when I downloaded the zip file, I only see scripts. How the hell do you run them. Do I replace the files in the original folder and run the download.bat? Videos online show it formatting and installing prime when you run download.bat. After backing up I wanted to dual boot Ubuntu, yet those instructions aren't much more clear. Again do I replace those files in the zip with the original nvflash folder? How do I run the script once I put them in there? The Nvflash.exe doesnt help. And he says to put recovery.img, boot.img, and system.img in the /image folder (for prime), yet I have no idea where those are. NVFlash hasnt been explained well. I see videos online showing how to use it for basic root, but when it comes to anything else, everyone is silent.
Click to expand...
Click to collapse
It's not easy to explain something that has so many uses and so little documentation. Try to find anything over at nvidia where this thing comes from and let us know how that goes:
Here is all I've gotten and it is just by typing in (either in linux or Win) this:
Code:
nvflash --help
Nvflash started
nvflash action [options]
action (one or more) =
--help (or -h)
displays this page
--cmdhelp cmd(or -ch)
displays command help
--resume (or -r)
send the following commands to an already-running bootloader
--quiet (or -q)
surpress excessive console output
--wait (or -w)
waits for a device connection (currently a USB cable)
--create
full initialization of the target device using the config file
--download N filename
download partition filename to N
--setboot N
sets the boot partition to partition N
--format_partition N
formats contents of partition N
--read N filename
reads back partition N into filename
--getpartitiontable filename
reads back the partition table into filename
--getbit filename
reads back BIT into filename
--getbct
reads back the BCT from mass storage
--odm C Data
ODM custom 32bit command 'C' with associated 32bit data
--go
continues normal execution of the downloaded bootloader
options =
--configfile filename
indicates the configuration file used with the following commands:
--create, --format_all
--bct filename
indicates the file containing the BCT
--sbk 0x00000000 00000000 00000000 00000000
indicates the secure boot key for the target device
--bl filename
downloads and runs the bootloader specified by filename
--odmdata N
sets 32bit customer data into a field in the BCT, either hex or
decimal
--diskimgopt N
sets 32bit data required for disk image convertion tool
--format_all
formats all existing partitions on the target device using the config file,
including partitions and the bct
--setbootdevtype S
sets the boot device type fuse value for the device name.
allowed device name string mentioned below:
emmc, nand_x8, nand_x16, nor, spi
--setbootdevconfig N
sets the boot device config fuse value either hex or decimal
--verifypart N
verifies data for partition id = N specified. N=-1
indicates all partitions
Intended to be used with --create command only.
--setbct
updates the chip specific settings of the BCT in mass storage to
the bct supplied,used with --create, should not be with --read,and
--format(delete)_all,format(delete)_partition,--download, and--read
--sync
issues force sync commad
--rawdeviceread S N filename
reads back N sectors starting from sector S into filename
--rawdevicewrite S N filename
writes back N sectors from filename to device starting from sector S
--updatebct <bctsection>
bctsection should refer to the section of the bct we are updating.
Curently we suport updates for following sections
<SDRAM> updates SdramParams and NumSdramSets fields
<DEVPARAM> updates DevParams, DevType and NumParamSets
<BOOTDEVINFO> updates BlockSizeLog2, PageSizeLog2 and PartitionSize
And this crazy nvflash utility has been the most useful small program I've had for the transformer and I think will remain that way. Some of the commands (e.g. when you insert the SBK or Secure Boot Key) seem like magic, and I've no idea where people have come up with these things. Other pieces of it are pretty clear like using --read {filename} to put a partition .img file on the PC as a backup of that numbered partition.
Sorry: I'm still learning this as well, and will be well . forever.

use dd to root p970!

Op afraid that lg might know this and patch it. I call that's a bull. I don't think lg engineer didn't know how to use dd. As long as we have unecrypted bin file we can patch it. Btw, how can one conceal a patching method using simple dd when you have unpatch file and patched file to compare? Even I managed to figure it out. Op true intention known only to himself.
So, to patch your bin file: (this is done on ubuntu, i don't know if windows got dd)
1. Download cwm at http://download.clockworkmod.com/rec...8.1.3-p970.img to the folder where you keep your bin file.
2. Open terminal and cd to the same folder as above
3. What we need to find is the location where we're suppose to copy cwm into the bin file. dd is bit-stream reader and it will show us the location in term of bytes. We need to search for ANDROID! string. So type:
dd if=LGP970AT-00-V20b-SEA-XXX-JAN-02-2012+0_AP.bin| strings -n 8 -t d| grep ANDROID!
result:
1436661 ANDROID!
1703936 ANDROID!
6422528 ANDROID!
bla..bla..bla...
4. Look of the last entry which contain the ANDROID! string and take note of its byte location (6422528 in this case)
5. Now it's time to copy cwm into the bin file. Type:
dd if=<cwm.img> bs=1 seek=<byte location> conv=notrunc of=<rom.bin>
So in this case:
dd if=recovery-clockwork-touch-5.8.1.3-p970.img bs=1 seek=6422528 conv=notrunc of=LGP970AT-00-V20b-SEA-XXX-JAN-02-2012+0_AP.bin
6. That's it. Your rom is ready to be flash using smartflash tool. You can run the same command in step 3 to verify that cwm is succesfully copied into the bin file. If at the same byte location the ANDROID! string change to [email protected] then you're good to go.
Don't waste your bandwidth to download the op's file if you can do this yourself and beside there's nothing to test, not that op modified the rom in any way. You would be better off testing huexxx zues rom imho.
http://forum.xda-developers.com/showthread.php?t=1480392&page=11

So here's the first question: where from can we find an unecrypted bin rom for our O3D?
3rd step,after the command I don't get such lines
"1436661 ANDROID!
1703936 ANDROID!
6422528 ANDROID! "
I get a count of entries instead.

[GUIDE] How to extract, create or edit android adb backups

What is an android adb backup?
An adb backup is a file with ab extension, generated by android's backup manager when we request it via adb shell. This allows you to backup some data of the phone, but is not a replacement of a clockworkmod backup:
- Java 7 or higher is required because of SYNC_FLUSH mode for the Deflater.
- If the backup contains apk+data for an app, restore will work. If contains only the data, you must install the app first on the device and then restore. Installing the app later won't work.
- Some apps include a policy where the apk is never backed up even if specified.
- Inside an ab file is a tar file, which contains files and folders in a certain order. You have you respect that order, which is not necessarily alphabetical like tar does by default. Such order is listed in "Full local backup infrastructure".
- Inside the tar file, directories must not have trailing slash, for that reason pax, star or equivalent has to be used.
- There are some bugs present in the android source code.
- 'adb backup' or 'adb restore' are pretty slow, between 1 and 2.6 MBps.
- The adb backups (usually with .ab extension) can be password protected or not. If the device is encrypted this is a must and has to be the same used for the device.
- Star is recommended instead of pax, which supports path length only up to 100 characters, so will fail in some cases. This usually happens with browsers history.
- ADB backups are not a replacement for a Nandroid backup. The whole /data partition is not backed up, only a part of it. Also, other partitions like /system, /preload, /cache, modem, RIL, efs, kernel or recovery are never backed up because are not user data. This is to prevent issue when restoring on a different device. It has also the biggest
- If you have the device encrypted with a password, you must use that particular one for backup creation and restore. You can't create a backup without password or a different one in that particular case, or if you try to restore will fail.
- star for cygwin for windows is attached (move it to C:\cygwin\bin\star.exe) since there is no package available. It can be compiled from the schily (ftp://ftp.berlios.de/pub/schily/).
- The best way to test if an adb backup has errors, is to convert it to tar and then check.advantage that doesn't require root to operate, so is totally compatible with stock roms, locked bootloaders and device encryption.
- There's also a bash script called adb-split.sh that creates individual adb backups for each recognized app from the full one, so you can restore apps individually. Same encryption is preserved to them, if any.
You can use java or perl, although with perl can be more complicated because requires downloading some modules from cpan and some ssl headers.
To know more information about types of android backups:
[Guide] Types of Android backups
Software needed
The entire process can be done from Windows, but is better to use a unix-like operating system, like Linux, OS X or BSD, since we should extract the files on a filesystem that preservers file permissions and owners and repack the tar archive. Furthermore some tools like star are easier to get for linux. In such cases, using Virtualbox or VMWare Workstation is highly recommended. Using Windows should work in any case whatsoever.
Cygwin
Preferably 32 bits: has more packages and is less prone to errors. Furthermore, the star for cygwin attached is 32 bits.
Java 7 or higher
Java 7 or OpenJDK 7 (more builds here)
Oracle Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 if you are going to work with password encrypted backups.
You need to install the files local_policy.jar and US_export_policy.jar under jre's lib/security folder, for example:
- For Windows:
C:\Program Files\Java\jdk1.7.0_09\jre\lib\security\
C:\Program Files\Java\jre7\lib\security\
C:\Program Files (x86)\Java\jdk1.7.0_07\jre\lib\security\
C:\Program Files (x86)\Java\jre7\lib\security\
- For Linux or BSD:
/usr/local/jdk1.7/jre/lib/security/
/usr/lib/jvm/java-7-openjdk-*/jre/lib/security/
/usr/local/openjdk7/jre/lib/security/
- For OS X:
/Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/jre/lib/security/
Perl
Perl is available for several operating systems
libssl or openssl headers. If you are using Linux or Cygwin is much more easier
cpan modules required by use functions
pax or star
pax is an archiving utility that compresses in ustar by default. In this case is extremely useful because stores directories without trailing slash. Is available for all operating systems, and in windows can be installed via cygwin's setup.exe or download a native version like gnuwin32 (sometimes available via bsdtar with pax option). pax doesn't work when paths are more than 100 characters lenght, so I recommend star instead.
star allows storing directories without trailing slash. You can get the ubuntu version 1.5 here. For Windows you can use Cygwin's version included.
Android Backup Extractor
Android Backup Extractor is the java application that does all the job. It includes the perl scripts.
How it works, better with an example
I will use an example to demonstrate how it works, with java version. If you like to use perl, just grab the perl scripts and is nearly the same.
This is extracted from the readme file, and will extract a whole adb backup and repack only the data for the game Grand Theft Auto III for android.
1) Convert the original adb backup to tar format:
Code:
java -jar abe.jar unpack nexus7.ab nexus7.tar <password>
2) Extract the contents of the tar archive. This should be done on a filesystem where the permissions of the files inside the tar are preserved, for example using linux, mac or bsd. Up to two folders may appear, apps and shared:
Code:
tar -xvf nexus7.tar
3) Make a list of all the contents of the original archive in the order they are archived:
Code:
tar -tf nexus7.tar > nexus7.list
4) Create a new list only with the files and folders you want, in proper order. For example for the GTA 3 (you can try savegames instead of all data):
Code:
cat nexus7.list | grep com.rockstar.gta3 > gta3.list
5) Create the new tar archive. The directories stored on tar shouldn't contain trailing slashes, so I use pax or star instead of tar. Pax works also if paths are up to 100 of lenght:
Code:
cat gta.list | pax -wd > gta3.tar
OR
Code:
star -c -v -f gta3.tar -no-dirslash list=gta3.list
6) Create the adb backup from the tar archive. Password is optional:
Code:
java -jar abe.jar pack gta3.tar gta3.ab <password>
Note: if the backup is not encrypted zlib can be used instead for both unpack and pack the ab archive:
- Quick unpacking:
Code:
dd if=nexus7.ab bs=24 skip=1 | openssl zlib -d > nexus7.tar
- Quick packing:
Code:
dd if=nexus7.ab bs=24 count=1 of=gta3.ab ; openssl zlib -in gta3.tar >> gta3.ab
Split an Android backup
Use Android Backup Splitter (is a shell script). Works on Cygwin too:
sh adb-split.sh backup.ab [password if needed]
Resulting files go in app-ab folder
# ADB Backup Splitter:
#
# This bash script utility generates an adb backup for each item
# on apps/ folder. That equals to generating an adb backup for
# each app.
#
# shared/0 and shared/1 (sdcards) are ignored, so that means that
# external data and OBB files, although may be backed up, may not
# be restored. For doing so backup the folder Android of the sdcards.
# Creating a backup with -shared flag has know issues (corruption)
# so is not recommended. Future releases of this script may add
# the option to automatically detect things from sdcards inside
# the adb backup file.
#
# This script works on Cygwin also, and in general where star does.
#
# There may be apps in the backup that don't include the apk. In that
# particular case, the data won't be restored, so the individual backup
# is generated for testing purposes only. Installing the apk afterwards won't work.
# There is generated an html file called apk-missing.html so you can open it and
# install all applications from Play Store on any of your devices.
Patch apps to allow adb backup
Some apps specify in the AndroidManifest.xml android:allowBackup="false" so it won't be backed up by adb backup. If you have root and use Titanium Backup or CWM or TWRP they will do the job, but if you don't want root the only solution is to modify the apk to edit that string. Read here how to do that and the pros and cons:
[GUIDE] How to enable adb backup for any app changing android:allowBackup
Links
Android Backup Splitter (@ Sourceforge)
Android Backup Extractor (@ Sourceforge)
Android Backup Extractor (@ GitHub)
Documentation at Blogspot
Full local backup infrastructure
[Guide] Types of Android backups
Backing Up Android Games
Titanium Backup Decrypter
Perl scripts to encrypt/decrypt adb backup files
[GUIDE] Full Phone Backup without Unlock or Root
[GUIDE] Internal Memory Data Recovery - Yes We Can!
[Q] [adb] backup not working for Temple Run
Related adb backup and restore bugs:
Issue 55860: adb backup skips the apk for certain apps and doesn't prompt for download when restoring
Issue 48126: device adb connecting to localhost adbd cannot execute commands anymore since 4.2.2 (error device offline)
Issue 3254: adb shell doesn't return exit code of program
Issue 53065: Debugging via ADB stopping the onClick action on ImageView widget
Issue 53668: ADB : An existing connection was forcibly closed by the remote host
Issue 54781: adb install: single quotes in filenames cause quoting issues
Issue 55173: (Game) Offroad Legends does not detect obb when restored via adb restore
Issue 55178: (Game) Carmageddon Promo does not detect obb when restored via adb restore
Issue 28303: adb backup doesn't respect -noshared flag
Issue 32830: adb restore errors not displayed on device
Issue 34311: Galaxy Nexus gets stuck when restoring adb backup
Issue 25780: BackupManager causes reboot when BackupAgent missing
Issue 40691: ab backup and restore Ski Safari does not restore saved data
Issue 16286: Restoration of phone not working properly
Issue 39014: Nexus 7 adb restore freezes when restoring udk.android.reader
Warning about Helium (Carbon): helium backups are just android adb backups that are not compressed with Java Deflater. It means they were created with the option static final boolean COMPRESS_FULL_BACKUPS = false.
A normal adb restore should work.

Great guide, and can confirm that it does work!
I managed to use this method to restore my angry birds data from a Froyo phone (HTC Desire), to a Jelly Bean phone (HTC One X+). This is no easy task since you cannot use adb backup/restore on Froyo, but you can use adb pull, and you can't use adb push on Jelly Bean, but you can use adb backup/restore. Just in case anyone is looking to do the same, here is the process I used (requires having adb up and running but does not require root)...
As the adb backup function doesn't work on Froyo, I had to use adb pull to grab the angry birds data files (highscores.lua and settings.lua). Then, making sure that you have angry birds installed on the jelly bean phone and have played the first level, use the adb backup command to backup just the angry birds app. You can then use scandiuns guide to extract a tar file from the backup file, and then extract the "app" folder from the tar file.
Once this is done, you will find the highscores.lua and settings.lua files in the apps/com.rovio.angrybirds/f directory. Replace these two files with your original ones that you pulled from the old device. Then package up the app folder using the guide. Then just use adb restore to restore the modified backup file to your new device, and you should have all your old scores restored!
Hope that helps and thanks to scandiun for the great guide! :good:

cn198 said:
Great guide, and can confirm that it does work!
I managed to use this method to restore my angry birds data from a Froyo phone (HTC Desire), to a Jelly Bean phone (HTC One X+). This is no easy task since you cannot use adb backup/restore on Froyo, but you can use adb pull, and you can't use adb push on Jelly Bean, but you can use adb backup/restore. Just in case anyone is looking to do the same, here is the process I used (requires having adb up and running but does not require root)...
As the adb backup function doesn't work on Froyo, I had to use adb pull to grab the angry birds data files (highscores.lua and settings.lua). Then, making sure that you have angry birds installed on the jelly bean phone and have played the first level, use the adb backup command to backup just the angry birds app. You can then use scandiuns guide to extract a tar file from the backup file, and then extract the "app" folder from the tar file.
Once this is done, you will find the highscores.lua and settings.lua files in the apps/com.rovio.angrybirds/f directory. Replace these two files with your original ones that you pulled from the old device. Then package up the app folder using the guide. Then just use adb restore to restore the modified backup file to your new device, and you should have all your old scores restored!
Hope that helps and thanks to scandiun for the great guide! :good:
Click to expand...
Click to collapse
I have tried similar attempt to restore my save of another game but that doesn't work.
How did you replace the files in the .tar file? I am using 7z to replace the .tar file but nothing happen after i restored.
Thanks.

ball3t said:
I have tried similar attempt to restore my save of another game but that doesn't work.
How did you replace the files in the .tar file? I am using 7z to replace the .tar file but nothing happen after i restored.
Thanks.
Click to expand...
Click to collapse
Read the guide. 7z is not compatible, you have to use star or equivalent and meet the requirements.

scandiun said:
Read the guide. 7z is not compatible, you have to use star or equivalent and meet the requirements.
Click to expand...
Click to collapse
star
star is required since GNU Tar does not allow storing directories without trailing slash. You can get the ubuntu version 1.5 here. For windows or Cygwin I haven't found it yet, if exists.
Because i am using windows, so does it mean that i couldn't modify the .tar file in any way?
thanks.

ball3t said:
Because i am using windows, so does it mean that i couldn't modify the .tar file in any way?
thanks.
Click to expand...
Click to collapse
Yes, from windows you can use pax. You can download it from cygwin's setup.exe. I've updated the guide, see the step 5.

scandiun said:
Yes, from windows you can use pax. You can download it from cygwin's setup.exe. I've updated the guide, see the step 5.
Click to expand...
Click to collapse
it worked!!
thank you so much.

Maybe I'm missing something.
I've installed Java, ran cygwins setup.exe and I think it installed everything. Also downloaded the Android Backup Extractor. I try running a windows command prompt with:
java -jar abe.jar unpack nexus7.ab nexus7.tar
All I get is "java is not an internal or external command". I'm running the command prompt in the android backup extractor folder. Does everything have to be in the same folder? Where do I dump everything?

AAhrens said:
Maybe I'm missing something.
I've installed Java, ran cygwins setup.exe and I think it installed everything. Also downloaded the Android Backup Extractor. I try running a windows command prompt with:
java -jar abe.jar unpack nexus7.ab nexus7.tar
All I get is "java is not an internal or external command". I'm running the command prompt in the android backup extractor folder. Does everything have to be in the same folder? Where do I dump everything?
Click to expand...
Click to collapse
Probably java is not in your path, try
Code:
which java
It should result something like
Code:
$ which java
/cygdrive/c/Windows/system32/java
Add Java to the Windows Path
Then restart cygwin

Omg omg omg! Thanks for the guide you just saved my ass, i love you!!!!!

scandiun said:
2) Extract the contents of the tar archive. This should be done on a filesystem where the permissions of the files inside the tar are preserved, for example using linux, mac or bsd. Up to two folders may appear, apps and shared:
Code:
tar -xvf nexus7.tar
Click to expand...
Click to collapse
Hi,
I'd like to try this out on a Windows 7 (virtual) machine. All software installed an clear what to do. But how do I manage to preserve the file permissions? Does Cygwin take care of that?
Thanks

ttv said:
Hi,
I'd like to try this out on a Windows 7 (virtual) machine. All software installed an clear what to do. But how do I manage to preserve the file permissions? Does Cygwin take care of that?
Thanks
Click to expand...
Click to collapse
You can't preserve the exact permissions because of the operating system and the filesystem. While permissions will be somewhat similar the owner will change completely. While android has the owner encoded by a number windows will use your windows username. I made simple modifications of backups on windows and luckily worked, but of course can't promise you that will apply for all the cases.
If you know how to use virtual machines, why just don't use linux / unix? For example if you use PC-BSD or Linux Mint you can download directly the Virtualbox image and then share a folder or use ftp server.
http://virtualboximages.com/LinuxMint+14+KDE+Desktop+32bit+Virtual+Appliance

scandiun said:
If you know how to use virtual machines, why just don't use linux / unix? For example if you use PC-BSD or Linux Mint you can download directly the Virtualbox image and then share a folder or use ftp server.
http://virtualboximages.com/LinuxMint+14+KDE+Desktop+32bit+Virtual+Appliance
Click to expand...
Click to collapse
Thanks for the answer.
I didn't understand the bit about sharing a folder on Unix before, but now I think it clicks.... The Linux/Unix VM is only used to share a folder in which I can extract the Android backup from my Windows machine right? So that way I can still use the installed software and drivers for my Nexus 4 on the Windows machine. Do I understand this correctly?
If that's the case I am going to try right away. Otherwise I have to make the Nexus 4 software installation work on the Unix/Linux VM and that will take a lot more time I think.....

ttv said:
Thanks for the answer.
I didn't understand the bit about sharing a folder on Unix before, but now I think it clicks.... The Linux/Unix VM is only used to share a folder in which I can extract the Android backup from my Windows machine right? So that way I can still use the installed software and drivers for my Nexus 4 on the Windows machine. Do I understand this correctly?
If that's the case I am going to try right away. Otherwise I have to make the Nexus 4 software installation work on the Unix/Linux VM and that will take a lot more time I think.....
Click to expand...
Click to collapse
Yes, I mean to access from linux the folder you share in windows, but then remember to copy the backup inside linux to extract it there, to preserve permissions.
You have the option to share folders in Virtualbox and VMWare Workstation. In case you use virtualbox, make sure the guest has Guest Additions installed. Some virtualbox images already come with it.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}

Well I got the java command to work, but now the "cat" command will not work? Your link to add java to the path worked for the latter, but how do I get 'cat' to work?

AAhrens said:
Well I got the java command to work, but now the "cat" command will not work? Your link to add java to the path worked for the latter, but how do I get 'cat' to work?
Click to expand...
Click to collapse
What exact error are you getting?

scandiun said:
What exact error are you getting?
Click to expand...
Click to collapse
"Cat is not an internal or external command"

AAhrens said:
"Cat is not an internal or external command"
Click to expand...
Click to collapse
What shell are you using? cat is included on any default linux installation.

scandiun said:
What shell are you using? cat is included on any default linux installation.
Click to expand...
Click to collapse
Windows 8

On Cygwin you have cat and the other utilities you have to install them from the setup.

[Q] unpack userdata***.backup from cwm on windows

Hi/Hello/good evening!
I have a broken MTK phone waiting for spareparts and one backup file from its SDCard made with CWM
name userdata_20130615_140224.backup where I need to extract contact data from.
How to do it?
I have been searching a whole while now but could not find one single post about opening this archive.
Pls help!
Thank you,
Erwin

If u have any spare phone with Titanium Backup installed then u can transfer ur backup to tat mobile's sd card and use TB to extract data from Nanoid Backup..
Sent from my WT19i using xda premium

expanding file
hi,
unyaffs doesnt work on that .backup file and I dont have a spare phone lying around. I got the file on windows
and please after reading 20.000 tuts on "backups" can there be one that explains how to extract the file
with windows?
thank you,
Erwin

NOW!!!
ewoewo said:
hi,
unyaffs doesnt work on that .backup file and i dont have a spare phone lying around. I got the file on windows
and please after reading 20.000 tuts on "backups" can there be one that explains how to extract the file
with windows?
Thank you,
erwin
Click to expand...
Click to collapse
tell me the tec solution internet!

ewoewo said:
Hi/Hello/good evening!
I have a userdata#######.backup and I cant enter into recovery mode. once i try toenter recovery mode in boots and shuts down without entering into recovery mode. I want to restore this backup file. Help me out
Thank you!!
Click to expand...
Click to collapse

It's a 512 byte header in front of a "tar.gz"
hexdump -C userdata_xxxxxxxx_yyyyyy.backup | less​showed the magical zip header "1f8b0800 00000000" at offset 0x200
So, skipping 512 bytes, the rest can be gunzip | tar
dd if=userdata_xxxxxxxx_yyyyyy.backup bs=512 skip=1 | gunzip -c | tar xv​

Code:
dd if=/home/user/_backups/E380-android-phone/userdata_20160126_223433.backup bs=512 skip=1 > /home/user/_backups/E380-android-phone/userdata_20160126_223433.backup.dd-out.tar.gz
Worked for me. Then just used KDE's archive opener (Ark) to view the files, looks to have everything in there except keys and whatnot.

if you prefer a linux solution use this script it converts android backup to nandroid backup
http://forum.xda-developers.com/showthread.php?p=65374561

Hello! Super-Developers, mega-programmers and smart hackers!
Nobody does not know how to open *. backup file?
No one can't help in this problem suffering millions of android/windows users?
You suggest them to install Unix/Linux system, - everybody to become hackers and programmers?!
maybe in the World exists GUI- programme to open *. backup file?
Someone can solve this problem of the century?

Philoandr said:
Hello! Super-Developers, mega-programmers and smart hackers!
Nobody does not know how to open *. backup file?
No one can't help in this problem suffering millions of android/windows users?
You suggest them to install Unix/Linux system, - everybody to become hackers and programmers?!
maybe in the World exists GUI- programme to open *. backup file?
Someone can solve this problem of the century?
Click to expand...
Click to collapse
What do you expect. You are trying to edit a Linux based file. Not to mention a closed sourced compressed file.

There are free programs like OSFMount or DiskInternals Linux Reader for mounting (android) linux file system images on Windows. Also 7-Zip for Windows can extract files from linux file system images.
However, there is metadata like user identifier and group ownership, file permissions, SELinux context (user role type range) stored for each file, which gets lost once you have saved linux files to windows disk. Without this metadata you can not restore files to phone.
Regardless you can extract files in windows if you just care about pictures, music files etc. Maybe one day there is a developer who may write such a program no matter its useless for restoring complete user data to phone.
Regarding the userdata*.backup files (no cwm) this is a chinese feature not provided by google android open source project. Without source code its nearly impossible to analyze the checksum algorithm. You can not re-create a userdata*.backup file.
What you can do is unpacking it. make a guess. cut off the first 512 byte header of each file which is probably the checksum. then merge the files together. Now you have a file for further analysis. Its probably either a ext4 disk image or a (gz compressed) tarball archive. a disk image can be mounted as a new drive, metadata keeps preserved in this case. a tarball archive must extracted with gnu tar to linux file system. when extracting with 7-Zip you will lose metadata.
There is a risk of tarball archive is unheadered Tar, in this case 7-zip will not extract it. If your phone is encrypted, the disk image is probably encrypted disk image (thats where the hacker stuff starts).
If you want to remove the checksum in windows, there is a small problem. There is no such command. You can use a third party command line utility Trunc to try it. (truncate deletes only from tail)
Another way for Windows, you can use some linux utils with cygwin. I don't know exactly how to do it, but @Doc_cheilvenerdi.org released a Windows script Odinatrix which converts unheadered Tar to Ustar Tar. From this you can use some linux utils (i don't know if it works for converting backups, too)
1. unpack the Odinatrix-WINDOWS.zip file. copy the folder named "files" containing cygwin linux utils. rename the folder to bash (or any meaningful name)
2. create a new folder on Windows Desktop
3. move the backup files to <some folder>
4. open the Windows Command Prompt
goto Windows Start - Run...
type: %SystemRoot%\system32\cmd.exe - press OK
now type the following commands in Windows terminal console:
5. add bash folder to windows environment variable %PATH%
(example: PATH=%PATH%;C:\Program Files\bash)
Code:
PATH=%PATH%;<path to bash folder>
6. change directory and list backup files
Code:
cd %UserProfile%\Desktop\<some folder>
dir /a userdata*.backup*
7. cut off 512 byte header from each file using (cygwin) gnu "dd" linux util like @ektoric suggest in post #6.
do this for each file, skipping 512 bytes only (without unpacking). replace the wildcard * with <date>_<time>
Code:
dd if=userdata*.backup of=userdata*.part0 bs=512 skip=1
dd if=userdata*.backup1 of=userdata*.part1 bs=512 skip=1
dd if=userdata*.backup2 of=userdata*.part2 bs=512 skip=1
dd if=userdata*.backup3 of=userdata*.part3 bs=512 skip=1
8. merge the files with windows
Code:
copy /v /y userdata*.part0 /b + userdata*.part1 /b + userdata*.part2 /b + userdata*.part3 /b userdata*.img /b
9. try to mount the userdata*.img with OSFMount. if not working, try extracting with 7-zip.
Besides this linux is no hacker-os and just as easily operated as windows. You can boot a live distribution from usb-stick without installing.
for further questions, please visit the main thread
[Q] How to Extract data from .Backup file ? - Created with android system recovery

zelendel said:
What do you expect. You are trying to edit a Linux based file. Not to mention a closed sourced compressed file.
Click to expand...
Click to collapse
Thank you for the answer!
Please comment next:
1) Video how to extract files from
"Linux based, Not to mention, a closed sourced compressed" backup file with metadata:
Распаковка и извлечение файлов из резервной копии - YouTube
m.youtube.com/watch?v=HNXed7RGR2g
2) quote from Android Backup Extractor's README-TXT-instruction:
"Usage:-- Windows: very easy"?
Android Backup Extractor - -- sourceforge.net/projects/adbextractor/files
-sourceforge.net/projects/adbextractor/reviews?source=navbar
I tried to use Android Backup Extractor In WinXp SP3
but windows version of this app shows cygwin&some other files missing.
Do you know why it doesn't work?
I don't need to recover backup on its own place.
My target is to extract and open few files: photos, videos, documents...
Strangely 7zip also doesn't work. Thank you in advance!

adbextractor is for adb backups *.ab only (btw open source).
cygwin is a implementation of linux utils running in windows.
the above solution uses cygwin, did you try?

aIecxs said:
adbextractor is for adb backups *.ab only (btw open source).
cygwin is a implementation of linux utils running in windows.
the above solution uses cygwin, did you try?
Click to expand...
Click to collapse
Thank you for first very detailed answer!
It seems that you have a remarkable grasp of this subject,
that's why I ask you to comment my previous post about video and Android Backup Extractor.
Your second post is not clear for me: adbextractor is only for *.ab files??
I have few files cygwin, is it a separate program??
I attempted to run file start.exe - windows version of Android Backup Extractor with those few cygwin files
but unsuccessful.
I read and test a lot, but The problem is still unsettled.

i came up two years ago with same question, how to unpack, and ended up with learning linux. i have uploaded a bash script, but i don't know if it works. i am not a developer and still learning, and nobody give feedback.
what is exactly file name of your backup? suffix *.ab or *.backup? these are completely different, don't mix up tools.
if you mean userdata_20180313_161000.backup files, please try mini tut above beginning 1.

aIecxs said:
i came up two years ago with same question, how to unpack, and ended up with learning linux. ... i am not a developer and still learning
Click to expand...
Click to collapse
It is classic aphorismic sentence!! I like it.
What's the bash script? for what goal? it is for Windows? If yes, I'll try it. give me the link.
my files are userdata_20100102_045926.backup, -=-=.backup1, 2, 3......
I'm not Englishman, "mini tut" is mini tutorial?
You have 100 posts and 25 thanks - super rate for notdeveloper.

the tool in video (adbextractor) can not unpack your backup. your backup is a splitted ext4 partition image. OSFMount can open ext4 partition images from Windows
bckp2cwm.sh script is basically doing the same steps like mini tutorial above. unfortunately it is for linux. main goal is convert userdata*.backup to cwm backup, but its paused in the middle (after unpacking, before repacking)
as windows user, all you need is a usb-stick. there is a tool unetbootin running in windows. it is a one-click-solution for downloading and copying any linux distribution to usb-stick. you can boot into linux from this usb-stick, for example ubuntu

aIecxs said:
OSFMount can open ext4 partition images from Windows
Click to expand...
Click to collapse
OSFMount can extract few files from my backup?
Thank you very much for efficient advices!!!:highfive:
I'll try them in sequence.
although Linux is not my :angel:dream.

Solution for Windows
1. Download and install free hhd hex editor neo.
2. Open each of your .backup files, in select menu select range at 0 offset size 512 decimal, delete and save.
3. Use Windows copy command to join files as described in post#11 (Thanks to the author for free education).
4. Use any program that reads Linux disk image files. I used diskinternals (mount image from the drive menu first). Openext worked too.
This won't work on tar archives if I understand correctly. This took me days to figure out and only minutes to execute. No thanks to confusing android prompts while upgrading to a new OS.

yoyohelp said:
1. Download and install free hhd hex editor neo.
2. Open each of your .backup files, in select menu select range at 0 offset size 512 decimal, delete and save.
3. Use Windows copy command to join files as described in post#11 (Thanks to the author for free education).
4. Use any program that reads Linux disk image files. I used diskinternals (mount image from the drive menu first). Openext worked too.
This won't work on tar archives if I understand correctly. This took me days to figure out and only minutes to execute. No thanks to confusing android prompts while upgrading to a new OS.
Click to expand...
Click to collapse
Hi i have a problem i have 5 files from the backup each with 2gb when i do the command from post 11 it works nice but when goes to backup part 3 it gives an error, what could it be? Im i doing something wrong?