Mission accomplished!
Thanks, sdelmont!
Here's my collection of Nexus S radio images:
http://forum.xda-developers.com/showpost.php?p=11686273&postcount=14
Here's radio.img, as produced by the command you specified.
keeps saying write failed
mazodude said:
keeps saying write failed
Click to expand...
Click to collapse
Are you flashing the file that was posted? That isn't a flashable ZIP.
I'm flashing via fastboot
Baseband from 2.3.2 (I9020XXJK8)
Here is a flashable version.
Good to flash using Clockwork Recovery.
Link: www.mediafire.com/?r6xc8wuk2b6m24f
sdelmont said:
Here's radio.img, as produced by the command you specified.
Click to expand...
Click to collapse
Thanks!
I've cut the file down to size: http://www.mediafire.com/?kwk5oi05uyh0a8l
It seems to be correct, the SHA1 on the file actually matches the SHA1 tested for in the official update zip!
Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.
What does new Android zips (full roms, but also otas) contain?
New Android flashable zips are made this way:
boot.img (kernel)
file_contexts (selinux related)
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat (for OTAs)
system.transfer.list (see explanation below)
What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:
The transfer list is a text file containing commands to transfer data from one place to another on the target partition.
Click to expand...
Click to collapse
But what each line means?:
First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x/8.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on versions >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
Fifth line and subsequent lines are all individual transfer commands.
Click to expand...
Click to collapse
Ok, but how to decompress the system.new.dat file?
All instructions are below. sdat2img, rimg2sdat and img2sdat binaries are involved. Please read carefully step by step.
NOTE for OTAs
If you are looking on decompressing system.patch.dat file or .p files, therefore reproduce the patching system on your PC, check imgpatchtools out by @erfanoabdi.
You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.
If you have questions or problems write here
Are you building from source and want to revert to the old flashing method without DAT files?
Do this (thx @Elluel):
1 - Delete this line
2 - Make sure you have this commit
Thanks
- howellzhu & luxi78 - for initial source codes
- all contributors from github
Ubuntu
Ubuntu 64-bit - Guide
Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)
We're now using sdat2img binary, the usage is very simple (make sure you have python 2.7+ installed):
./sdat2img.py <transfer_list> <system_new_file> [system_img]- <transfer_list> = input, system.transfer.list from rom zip
- <system_new_file> = input, system.new.dat from rom zip
- [system_img] = output ext4 raw image file
and a quick example:
./sdat2img.py system.transfer.list system.new.dat system.imgby running this command you will get as output the file my_new_system.img which is the raw ext4 image.
Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)
Now we need to mount or ext4 raw image into an output folder so we can see apks/jars etc.
To do this we need to type this command:
sudo mount -t ext4 -o loop system.img output/As you can see there is a new folder called output which we can edit/modify/delete your files (not able to? see here)
Now we need to compress it back to a raw ext4 image, to do this we need the make_ext4fs binary. Make sure you have the file_contexts file (taken from the Rom zip) inside the make_ext4fs path. Then type this (got issues? see here)
./make_ext4fs -T 0 -S file_contexts -l 1073741824 -a system system_new.img output/You will get the new raw ext4 image called 'system_new.img' ready for the next step.
Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)
Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
The usage is simple:
img2simg <raw_image_file> <sparse_image_file>Pretty self-explanatory, the output will be a new sparse image (.img).
Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)
Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
./img2sdat.py <system_img>- <system_img> = name of input sparse image file (from step 3)
As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.
DOWNLOADs
sdat2img.py
- github.com
make_ext4fs
- mega.co.nz
img2sdat.py
- github.com
Source - Changelog
sdat2img
Repo: https://github.com/xpirt/sdat2img
Changelog: https://github.com/xpirt/sdat2img/commits/master
Source: https://github.com/xpirt/sdat2img/blob/master/sdat2img.py
img2sdat
Repo: https://github.com/xpirt/img2sdat
Changelog: https://github.com/xpirt/img2sdat/commits/master
Source: https://github.com/xpirt/img2sdat/blob/master/img2sdat.py
Windows
Windows 32/64 bit - Guide
Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)
We're now using sdat2img.py binary, the usage is very simple (make sure you have python 3.x installed):
sdat2img.py <transfer_list> <system_new_file> <system_ext4>- <transfer_list> = input, system.transfer.list from rom zip
- <system_new_file> = input, system.new.dat from rom zip
- <system_ext4> = output ext4 raw image file
and a quick example:
sdat2img.py system.transfer.list system.new.dat system.imgby running this command you will get as output the file my_new_system.img which is the raw ext4 image.
Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)
Now you should either move to Ubuntu x64/x86 or follow the 2nd step from the Ubuntu guide on cygwin 32 bit.
Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)
Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
The usage is simple:
img2simg.exe <raw_image_file> <sparse_image_file>Pretty self-explanatory, the output will be a new sparse image (.img).
Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)
Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
img2sdat.py <system_img>- <system_img> = name of input sparse image file (from step 3)
As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.
DOWNLOADs
sdat2img.py
- github.com
make_ext4fs
- mega.co.nz
img2sdat.py
- github.com
Is there any tool to browse (and/or manipulate) these .dat files in Windows?
01 Vlatce said:
Is there any tool to browse (and/or manipulate) these .dat files in Windows?
Click to expand...
Click to collapse
i don't think so, though you can install ubuntu in a virtual machine as 64-bit and decompress there without problems.
edit: windows binaries added to OP.
xpirt said:
i don't think so, though you can install ubuntu in a virtual machine as 64-bit and decompress there without problems.
Click to expand...
Click to collapse
I found one, Ext2Explore, it can browse and extract lollipop .dat files
01 Vlatce said:
I found one, Ext2Explore, it can browse and extract lollipop .dat files
Click to expand...
Click to collapse
Can you explain how to work in more detail ? Thanks.
chihliouma said:
Can you explain how to work in more detail ? Thanks.
Click to expand...
Click to collapse
With that application you can open the system.new.dat file like a zip archive
Sent from my Nexus 5
xpirt said:
i don't think so, though you can install ubuntu in a virtual machine as 64-bit and decompress there without problems.
Click to expand...
Click to collapse
01 Vlatce said:
I found one, Ext2Explore, it can browse and extract lollipop .dat files
Click to expand...
Click to collapse
Confirmed.
Works perfectly..
Thanks a lot! This will save a lot of time theming.. (pulling the system, settings, framework without having to flash the rom first)
Oh, just for s&g's i extracted ("decompiled") the system.new file, the files (app, prive-app, framework, media, etc) all got placed on my desktop -- i deleted the system.new file from the rom -- i then placed the folders and files that got extracted and placed them in the rom -- i flashed it -- it booted! lol.. didnt think it would... but anyways, there's that.
For those who want to try:
--- > its basically self explanatory.
- once downloaded (the link above) extract the contents from the zip (i just went ahead and extracted to my desktop)
- right click the linux penguin
- run as administrator - the window will pop up
- go to file > open image > find your system.new image (should already be extracted from the rom and placed on your desktop)
- click open
- you should now see your image (or path) in the windows -- double click them (i doubled clicked both the right and left side)
- now you should be able to see all the folders
---> if you want to extract those folders
- just go up to the top
- click save and choose an extraction (i chose Desktop)
Hoped i helped -- any questions or confusion, dont be shy to ask -- i probably went too fast, idk lol
Enjoy!
Incredible_Culp said:
Confirmed.
Works perfectly..
Thanks a lot! This will save a lot of time theming.. (pulling the system, settings, framework without having to flash the rom first)
Oh, just for s&g's i extracted ("decompiled") the system.new file, the files (app, prive-app, framework, media, etc) all got placed on my desktop -- i deleted the system.new file from the rom -- i then placed the folders and files that got extracted and placed them in the rom -- i flashed it -- it booted! lol.. didnt think it would... but anyways, there's that.
For those who want to try:
--- > its basically self explanatory.
- once downloaded (the link above) extract the contents from the zip (i just went ahead and extracted to my desktop)
- right click the linux penguin
- run as administrator - the window will pop up
- go to file > open image > find your system.new image (should already be extracted from the rom and placed on your desktop)
- click open
- you should now see your image (or path) in the windows -- double click them (i doubled clicked both the right and left side)
- now you should be able to see all the folders
---> if you want to extract those folders
- just go up to the top
- click save and choose an extraction (i chose Desktop)
Hoped i helped -- any questions or confusion, dont be shy to ask -- i probably went too fast, idk lol
Enjoy!
Click to expand...
Click to collapse
I can't get this to work at all. I'm using Win 7 Pro and I've actually used this app before with stock Nexus 7 factory images and it worked fine.. What file did you actually download? I d/l 2 different versions, one was beta 2.0 and the other was a zip that said 2.2.71 but the about on the exe says 2.1, which is the version I've been using for quite a while.
With the beta 2.0, there's not even an option to open a file and with the 2.1, it errors out every time. After opening system.new.dat image and double clicking, it just hangs and the program stops responding. If I try clicking on it and selecting save, it allows me to select a folder to save it in but I just get a dialog box that says saving files and then hangs at 24% every time, finally crashing with a Microsoft C++ runtime error.
Oh btw, you didn't go too fast LOL, this is really a simple app to use but it's just not working for me with these block image files. Maybe I'm missing something but it's got me scratching my head. :laugh: If you have any ideas, I'm all ears.
Thanks,
Mike T
webdroidmt said:
I can't get this to work at all. I'm using Win 7 Pro and I've actually used this app before with stock Nexus 7 factory images and it worked fine.. What file did you actually download? I d/l 2 different versions, one was beta 2.0 and the other was a zip that said 2.2.71 but the about on the exe says 2.1, which is the version I've been using for quite a while.
With the beta 2.0, there's not even an option to open a file and with the 2.1, it errors out every time. After opening system.new.dat image and double clicking, it just hangs and the program stops responding. If I try clicking on it and selecting save, it allows me to select a folder to save it in but I just get a dialog box that says saving files and then hangs at 24% every time, finally crashing with a Microsoft C++ runtime error.
Oh btw, you didn't go too fast LOL, this is really a simple app to use but it's just not working for me with these block image files. Maybe I'm missing something but it's got me scratching my head. :laugh: If you have any ideas, I'm all ears.
Thanks,
Mike T
Click to expand...
Click to collapse
hmm interesting..
im using windows 8.1
idk how much help i can be because i just downloded the first option (provided by the link above) and went from there.
like you said, theres two.. the .exe and the .zip.. i downloaded the .exe
Incredible_Culp said:
hmm interesting..
im using windows 8.1
idk how much help i can be because i just downloded the first option (provided by the link above) and went from there.
like you said, theres two.. the .exe and the .zip.. i downloaded the .exe
Click to expand...
Click to collapse
So, did you d/l the zip that was modified 2006-8-26, which is version 2.0 beta, or did you click the link which said download latest release?
Mike T
webdroidmt said:
So, did you d/l the zip that was modified 2006-8-26, which is version 2.0 beta, or did you click the link which said download latest release?
Mike T
Click to expand...
Click to collapse
ha! didnt even see there was a latest release..
i downloaded the 2006-8-26 apparently -- didnt even look at that date either LOL
i dont want to try the latest now -- this one works fine for me lmao -- dont want to ruin it.. ah, screw it, im downloading the latest now
EDIT: Ok, i lied.. i just checked my trash.. i did download the lastest release .zip .. the 2271..
@webdroidmt @Incredible_Culp Download from here lol: http://sourceforge.net/projects/ext2read/
Maybe something with my PC setup or Win 7 because the 2006-8-26 version doesn't even give me an option for opening a file image. It actually loads up looking like an incomplete application, it's also a very small filesize. Did you just d/l the ext2explore exe, or did you also d/l ext2read2 beta zip?
Mike T
webdroidmt said:
Maybe something with my PC setup or Win 7 because the 2006-8-26 version doesn't even give me an option for opening a file image. It actually loads up looking like an incomplete application, it's also a very small filesize. Did you just d/l the ext2explore exe, or did you also d/l ext2read2 beta zip?
Mike T
Click to expand...
Click to collapse
Use the download link in the above post and click the green download button, and it should be the correct version, from 2012 rather than 2006.
Lethargy said:
@webdroidmt @Incredible_Culp Download from here lol: http://sourceforge.net/projects/ext2read/
Click to expand...
Click to collapse
That's the one I've been using for quite some time. As mentioned in my previous post, the system.new.dat file just "appears" to be saving but hangs at 24%, then eventually crashes. I'm stumped, should be so easy but it doesn't want to cooperate. LOL
Thanks anyway,
Mike T
webdroidmt said:
That's the one I've been using for quite some time. As mentioned in my previous post, the system.new.dat files just "appear" to be saving but hangs at 24%, then eventually crashes. I'm stumped, should be so easy but it doesn't want to cooperate. LOL
Thanks anyway,
Mike T
Click to expand...
Click to collapse
Weird. Try redownloading the ROM you're trying to extract?
Lethargy said:
Use the download link in the above post and click the green download button, and it should be the correct version, from 2012 rather than 2006.
Click to expand...
Click to collapse
Thanks for trying guys, that is exactly the one I've been using. I downloaded it again a few seconds ago and still getting the same result. :laugh:
Thanks.
Mike T
---------- Post added at 01:09 AM ---------- Previous post was at 01:08 AM ----------
Lethargy said:
Weird. Try redownloading the ROM you're trying to extract?
Click to expand...
Click to collapse
I've actually tried with images from 3 different roms and they all give me the same result. It's cool, I just won't be able to play around as much as I used to porting stuff.
Mike T
---------- Post added at 01:15 AM ---------- Previous post was at 01:09 AM ----------
Lethargy said:
Weird. Try redownloading the ROM you're trying to extract?
Click to expand...
Click to collapse
Just curious, you guys aren't changing the file extension to something other than .dat, right?
Mike T
I just finished an update to my flashing script, now it will work with payload.bin and unpacked payload files.
Currently, only gsi_sdm660_64 models are available for this option.
And Windows only, too.
PLEASE ENSURE YOUR PHONE HAS UNLOCKED BOOTLOADER, OTHERWISE THIS GUIDE IS USELESS.
Let me introduce you how to use the new script:
DL: https://github.com/HikariCalyx/OSTRemote/releases
source: https://github.com/HikariCalyx/OSTR...Script/generic_flash_script_gsi_sdm660_64.bat
If you use unpacked nb0 extension stock firmware as before, please follow this guide:
https://forum.xda-developers.com/nokia-7-plus/how-to/guide-workaround-flashing-sparse-t3906070
Click to expand...
Click to collapse
1. You must have Python 3 installed. It can be downloaded from https://www.python.org/downloads/
2. Extract the payload.bin from full OTA package (e.g. March 2019 update known as B2N-351F-0-00WW-B04)
3. Put the script file (and required files if you prefer original script source) and the payload.bin altogether
4. Download payload_dumper.py and update_metadata_pb2.py from the URL given by the script, and place them to the same directory of the script.
5. Type lowercase "yes" (without quotes) and press Enter, wait for the procedure complete.
6. The script will dump the payload.bin and process both system and vendor image to flashable sparse image.
7. After that, you should know what to do next.
If you already have dumped payload and converted system.img / vendor.img, the script will tell you if you want to flash the unpacked OTA packages, you can simply type "yes" and proceed.
You needn't to waste the time of processing payload.bin again, once you don't delete them.
I'm considering port this script to universal for almost every A/B Android devices, if it works great.
Nice job dude, this will be very useful to many people.
Very well Hikari, nice script
Very very much appreciated work!
Can't wait to try it out!
Thanks !
Mod edit - Translated by Google Translate:
Very supportive! !! !!
*********************
非常支持！！！
does this work for other Nokia models? Such as 3.2?
Hello!
I want to extract the apk files from the system.img file HUAWEI P30 smartphone with android 9.
But the tool that I have always used (imgextractor.exe) completes the processing of the img file with an error:
Code:
ImgExtractor version 1.3.7 <Created by And_PDA (Based on sources ext4_unpacker)>
Extractor for images in EXT2\EXT3\EXT4\YAFFS2\CRAMFS filesystem formats
support SPARSE\SIN\MOTO structure formats
Exception EOutOfMemory in module imgextractor.exe at 0001669D.
Out of memory.
Please, help me!
P.S. According to the rules of the forum, I can not yet attach an external link to my post. But if necessary, I can send it to private messages!
Did you convert SIMG to IMG before? Something is in new format SIMG or raw IMG.
adeii said:
Did you convert SIMG to IMG before? Something is in new format SIMG or raw IMG.
Click to expand...
Click to collapse
Yes i used simg2img.exe for this. I tried to process the new IMG file with imgextractor.exe, but the result was the same ((
appdroiddevelop said:
Yes i used simg2img.exe for this. I tried to process the new IMG file with imgextractor.exe, but the result was the same ((
Click to expand...
Click to collapse
Try with DiskInternals LinuxReader.
The device is a Sam Gal A11W.
I want to change one text file in the /system partition.
In order to do that, I want to start from the stock firmware I already downloaded, make the change then flash the whole modified firmware.
There are 5 .tar.md5 files (4 if we consider only one of CSC_ and HOME_CSC_) in this firmware, which can be flashed just fine in their original form.
What I want to do is to:
1 - unpack the AP_***.tar file (contains meta-data/fota.zip, boot.img.lz4, carrier.img.lz4, dtbo.img.lz4, metadata.img.lz4, recovery.img.lz4, super.img.lz4, userdata.img.lz4, vbmeta.img.lz4)
2 - unlz4 the super.img.lz4,
3 - unsparse the super.img to super.ext4.img (using simg2img)
4 - lpunpack the super.ext4.img into individual dynamic partitions (there are only 4: odm.img, product.img, system.img, vendor.img - this is NOT an A/B device !!)
5 - modify the text file on system.img
6 - lpmake the super.ext4.img back from odm.img, product.img, system.img, vendor.img
7 - re-sparse the super.ext4.img back into super.img (using img2simg)
8 - lz4 the super.img back into super.img.lz4
9 - Re-TAR the super.img.lz4 file plus all the rest of the original .img.lz4 files back into the AP_***.tar file
10 - Flash the AP_***.tar file to this mother****ing piece of Samsung crap without errors, and that is WITHOUT UNLOCKED BOOTLOADER, because the scumbags eliminated the OEM Unlock option from the Dev Options.
What I already know is:
- How to do the whole process without preserving any signatures.
This is already explained in this thread:
Editing system.img inside super.img and flashing our modifications
I'm trying to modify my system.img (/system/build.prop) to include support for multi users. After struggling a lot, I've succeeded following your guide (that's an awesome work btw) to unpack, mount, modify, umount and repack super.img. Then...
forum.xda-developers.com
but the author assumes the bootloader can be unlocked and therefore doesn't deal with any signatures / security-crypto-baloney checks.
What I (still) need to know is:
1 - How many crypto-security-baloney-whatever-signatures are there on all these files ?
2 - How can I restore them during repackaging so the flashing process does not get screwed up by the locked phone ?
If anyone here knows the answer to these 2 things ... well I guess they must be the Android Godfather Almighty !!
have you figured this out?
FugkGoocle said:
The device is a Sam Gal A11W.
I want to change one text file in the /system partition.
In order to do that, I want to start from the stock firmware I already downloaded, make the change then flash the whole modified firmware.
There are 5 .tar.md5 files (4 if we consider only one of CSC_ and HOME_CSC_) in this firmware, which can be flashed just fine in their original form.
What I want to do is to:
1 - unpack the AP_***.tar file (contains meta-data/fota.zip, boot.img.lz4, carrier.img.lz4, dtbo.img.lz4, metadata.img.lz4, recovery.img.lz4, super.img.lz4, userdata.img.lz4, vbmeta.img.lz4)
2 - unlz4 the super.img.lz4,
3 - unsparse the super.img to super.ext4.img (using simg2img)
4 - lpunpack the super.ext4.img into individual dynamic partitions (there are only 4: odm.img, product.img, system.img, vendor.img - this is NOT an A/B device !!)
5 - modify the text file on system.img
6 - lpmake the super.ext4.img back from odm.img, product.img, system.img, vendor.img
7 - re-sparse the super.ext4.img back into super.img (using img2simg)
8 - lz4 the super.img back into super.img.lz4
9 - Re-TAR the super.img.lz4 file plus all the rest of the original .img.lz4 files back into the AP_***.tar file
10 - Flash the AP_***.tar file to this mother****ing piece of Samsung crap without errors, and that is WITHOUT UNLOCKED BOOTLOADER, because the scumbags eliminated the OEM Unlock option from the Dev Options.
What I already know is:
- How to do the whole process without preserving any signatures.
This is already explained in this thread:
Editing system.img inside super.img and flashing our modifications
I'm trying to modify my system.img (/system/build.prop) to include support for multi users. After struggling a lot, I've succeeded following your guide (that's an awesome work btw) to unpack, mount, modify, umount and repack super.img. Then...
forum.xda-developers.com
but the author assumes the bootloader can be unlocked and therefore doesn't deal with any signatures / security-crypto-baloney checks.
What I (still) need to know is:
1 - How many crypto-security-baloney-whatever-signatures are there on all these files ?
2 - How can I restore them during repackaging so the flashing process does not get screwed up by the locked phone ?
If anyone here knows the answer to these 2 things ... well I guess they must be the Android Godfather Almighty !!
Click to expand...
Click to collapse
Try using modified/patched Odin. As for trying to bypass unlocking the bootloader to flash the modified firmware or trying to "mimic" or "fake" the original signature, that isn't going to work. Samsung's proprietary signature is an enigma that can't be cracked. Though you might be able to match the MD5 by adding a dummy file to the file you are modifying and filing it bit/byte by bit/byte, one step at a time, the goal is to add characters to the file until it is large enough to make your modified file match the original file's MD5. That is if your modified file is smaller than the original file, if your modified file is larger than the original file, you can delete unimportant files from the modified MD5 file until it is smaller than the original MD5 and then create the dummy file filled with dummy characters until it exactly matches the original MD5 bit for bit. Then try flashing your MD5 file once you gets it's MD5 code matching bit for bit. Try the patched version of Odin to flash your modified file. No guarantees that it will work but part of the security checks during flashing checks the MD5 for a match/mismatch.
Rab_DaJew said:
have you figured this out?
Click to expand...
Click to collapse