[TUT] Simple way to add Flash Disk (a.k.a "Internal Storage") to your own ROM - JASJAR, XDA Exec, MDA Pro ROM Development

[TUT] Simple way to add Flash Disk (a.k.a "Internal Storage") to your own ROM
It's so long time, no one answer this question "How to add Flash Disk to own-cook ROM?", so I tried to find it out myself and I had found 2 different ways to do that.
As promised, or just a way to say thanks to all this community, I will share the way which more simple than the other.
It worked with Universal devices and may work with others, too!
* What you need?
1. Your ROM file (OS.nb, a.k.a nk.nba, nk.fat), different kitchens use different file names.
2. Any HEX editor, I use XVI32 here.
3. My Flash Disk image template.
* How it work?
The maximum size for a Universal OS ROM is 63 MB (66.060.288 Bytes), normally, we use less than this size for our ROM, the remain size is useless at all, so we make it useful by partitioning it.
* Which things to modify?
- Nothing in your SYS and/or OEM.
- boot.rgu in your XIP (only if it is not XIP-ready )
- First 512 Bytes of your ROM file (called Master Boot Record) (only if it is not MBR-ready )
- First 512 Bytes of my Flash Disk template file (called Boot Sector or Boot Record), it is a must edit thing to do (determine the size of Flash Disk)
* Here we go!
1. Download my file below, extract it and your nk.nbf into the same folder.
2. Run HTC64, decode your nk.nbf into two files (nk.prj and nk.fat).
3. Run HEX editor (XVI32), open nk.fat and write down two values (or just remember if you can) RED value (from $1E6 to $1E8 [or from $1F6 to $1F8 if they are not all 00]) and BLUE value (from $1EA to $1EB [or from $1FA to $1FB if they are not all 00]).
- Note: I use luca16thebig 1.4.9 BETA1 ROM for example, values may different from your ROM or this ROM after rebuilt.
00000000 ...
...
000001E0 41 88 04 FF 41 F7 80 88 01 00 80 6F 00 00 00 00
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA
Click to expand...
Click to collapse
4. Close the file nk.fat and don't save if there is anything changed. Open VNIntFlashDisk.template (also in HEX editor).
Match two RED and BLUE values which you have writen down into two place RED (from $1C to $1E) and BLUE (from $13 to $14). Double check if the values are corrected and fixed into its places and save the template.
- Note: I have already done it with luca16thebig 1.4.9 BETA1 ROM's values, so just goto next step
00000000 ...
00000010 02 00 02 80 6F F8 0C 00 3F 00 FF 00 80 88 01 00
Click to expand...
Click to collapse
5. Run combine.bat from my package, your original ROM file will be saved as nk.fat.original. The file nk.fat now is Flash Disk ready!
6. Run HTC64, re-encode two files (nk.prj and nk.fat) into nk.nbf and we are done!
* What is RED and BLUE values mean?
All two values are in HEX and reversed order, in this example we have:
- RED value is 80 88 01, so its true value is 1.88.80h, change it into decimal number, we have 100.480. It is the number of used sectors (1 sector = 512 Bytes). And... luca16thebig 1.4.9 BETA1 ROM's original size is 51.445.760 Bytes = 100.480 sectors x 512 Bytes.
- BLUE value is 80 6F, its true value is 6F.80h, in decimal is 28.544. It is the number of free sectors, so our Flash Disk size is 28.544 sectors x 512 Bytes = 14.614.528 Bytes ~ 14 MB.
- As you known, OS ROM maximum size is 66.060.288 Bytes = 51.445.760 Bytes + 14.614.528 Bytes.
If your ROM is MBR-ready, each time you rebuilt the ROM, all two values will automatically adjusted in MBR and you just match these values into my Flash Disk template. That all!
- Note: luca16thebig 1.4.9 BETA1 ROM is XIP-ready and MBR-ready. So, if your ROM doesn't work with this instruction, post your 512 Bytes MBR and your boot.rgu file here. I will analyse and make instructions how to make them ready.
When I release this instruction, I tried to make all things clearly from basic to advance but it seem so hard to... eat! So, I did it in reversed order, let you can do it first, then explain and/or clarify. More details will be add later (only when someone needed)!
Link: VNIntFlashDisk.rar

Questions and answers
* What is XIP-ready?
- boot.rgu in your XIP already have Flash Disk related key and values, they are:
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\TRUEFFS_DOC\FATFS]
"FormatTfat"=dword:1
"EnableWriteBack"=dword:1
"MountAsROM"=dword:0
"MountHidden"=dword:0
"Folder"="Flash Disk"
Click to expand...
Click to collapse
- Note: no space in key [HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\TRUEFFS_DOC\FATFS], I don't know why there is a space before TRUEFFS_DOC, please manually delete that space when copy!
* What is MBR-ready?
MBR of your ROM already have Flash Disk partition, there are 4 partition slots in MBR.
00000000 ...
...
000001B0 __ __ __ __ __ __ __ __ __ __ __ __ __ __ 11 11
000001C0 11 11 11 11 11 11 11 11 11 11 11 11 11 11 22 22
000001D0 22 22 22 22 22 22 22 22 22 22 22 22 22 22 33 33
000001E0 33 33 33 33 33 33 33 33 33 33 33 33 33 33 44 44
000001F0 44 44 44 44 44 44 44 44 44 44 44 44 44 44 55 AA
Click to expand...
Click to collapse
- The last 2 bytes (55 AA) is a signature, it means END OF MBR.
- 5th byte of each slot is value for partition type, some mostly used values are:
+ 20 - BOOT: just a name, it isn't actually a boot partition and your MBR can be without it.
+ 23 - RAWFS: your XIP partition.
+ 25 - IMGFS: your SYS + OEM partition.
+ 00 - none: free slot, all value in this slot should be 00, too.
+ 01, 04, 06 - FATFS: our Flash Disk here! In some other devices, it's called "Internal Storage".
There are some other values but that's another matter!
So, if your MBR have a FATFS partition type in last used slot, you have a MBR-ready ROM.
- Two non-MBR-ready cases: (with and without BOOT partition slot)
00000000 ...
...
000001B0 __ __ __ __ __ __ __ __ __ __ __ __ __ __ 11 11
000001C0 11 11 20 11 11 11 11 11 11 11 11 11 11 11 22 22
000001D0 22 22 23 22 22 22 22 22 22 22 22 22 22 22 33 33
000001E0 33 33 25 33 33 33 33 33 33 33 33 33 33 33 00 00
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA
Click to expand...
Click to collapse
00000000 ...
...
000001B0 __ __ __ __ __ __ __ __ __ __ __ __ __ __ 11 11
000001C0 11 11 23 11 11 11 11 11 11 11 11 11 11 11 22 22
000001D0 22 22 25 22 22 22 22 22 22 22 22 22 22 22 00 00
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA
Click to expand...
Click to collapse
BLUE free slot must be entered manually (by HEX caculating) to make them MBR-ready! But... how to HEX caculate values? Maybe later with a MBR sample!

VNInt said:
It's so long time, no one answer this question "How to add Flash Disk to own-cook ROM?", so I tried to find it out myself and I had found 2 different ways to do that.
As promised, or just a way to say thanks to all this community, I will share the way which more simple than the other.
It worked with Universal devices and may work with others, too!
* What you need?
1. Your ROM file (OS.nb, a.k.a nk.nba, nk.fat), different kitchens use different file names.
2. Any HEX editor, I use XVI32 here.
3. My Flash Disk image template.
* How it work?
The maximum size for a Universal OS ROM is 63 MB (66.060.288 Bytes), normally, we use less than this size for our ROM, the remain size is useless at all, so we make it useful by partitioning it.
* Which things to modify?
- Nothing in your SYS and/or OEM.
- boot.rgu in your XIP (only if it is not XIP-ready )
- First 512 Bytes of your ROM file (called Master Boot Record) (only if it is not MBR-ready )
- First 512 Bytes of my Flash Disk template file (called Boot Sector or Boot Record), it is a must edit thing to do (determine the size of Flash Disk)
* Here we go!
1. Download my file below, extract it and your nk.nbf into the same folder.
2. Run HTC64, decode your nk.nbf into two files (nk.prj and nk.fat).
3. Run HEX editor (XVI32), open nk.fat and write down two values (or just remember if you can) RED value (from $1E6 to $1E8) and BLUE value (from $1EA to $1EB).
- Note: I use luca16thebig 1.4.9 BETA1 ROM for example, values may different from your ROM or this ROM after rebuilt.
4. Close the file nk.fat and don't save if there is anything changed. Open VNIntFlashDisk.template (also in HEX editor).
Match two RED and BLUE values which you have writen down into two place RED (from $1C to $1E) and BLUE (from $13 to $14). Double check if the values are corrected and fixed into its places and save the template.
- Note: I have already done it with luca16thebig 1.4.9 BETA1 ROM's values, so just goto next step
5. Run combine.bat from my package, your original ROM file will be saved as nk.fat.original. The file nk.fat now is Flash Disk ready!
6. Run HTC64, re-encode two files (nk.prj and nk.fat) into nk.nbf and we are done!
- Note: luca16thebig 1.4.9 BETA1 ROM is XIP-ready and MBR-ready. So, if your ROM doesn't work with this instruction, post your 512 Bytes MBR and your boot.rgu file here. I will analyse and make instructions how to make them ready.
More details will be add later (only when someone needed)
Link: VNIntFlashDisk.rar
Click to expand...
Click to collapse
Very Useful!!THX You!!Post More details,PLZ

very very good news thank you

WOW
guys it's just to hard to handle for me
I'll give it a try....thx for the info....
ps: VNInt, could you just add Luca's ROM with flash disk? that would be great....
Got me 13.90MB Flash disk!!!!...
Thanks VNInt.....

I can't download the file from www.4shared.com
who can apply another download link
thanks

VNInt said:
* Which things to modify?
- Nothing in your SYS and/or OEM.
- boot.rgu in your XIP (only if it is not XIP-ready )
- First 512 Bytes of your ROM file (called Master Boot Record) (only if it is not MBR-ready )
- First 512 Bytes of my Flash Disk template file (called Boot Sector or Boot Record), it is a must edit thing to do (determine the size of Flash Disk)
Click to expand...
Click to collapse
1st let me thank you for your post. it will be quite useful in the future.
2 questions just to clarify:
- how to find if rom is XIP ready or not?
- how to find if rom is MBR ready?
Maybe these are dumb questions, sory for that.
Cheers.

ultravox said:
- how to find if rom is XIP ready or not?
- how to find if rom is MBR ready?
Click to expand...
Click to collapse
Yeah! You asked must-answer questions!
When I release this instruction, I tried to make all things clearly from basic to advance but it seem so hard to... eat! So, I did it in reversed order, let you can do it first, then explain and/or clarify.
* XIP-ready: boot.rgu in your XIP already have Flash Disk related key and values, they are:
[HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\TRUEFFS_DOC\FATFS]
"FormatTfat"=dword:1
"EnableWriteBack"=dword:1
"MountAsROM"=dword:0
"MountHidden"=dword:0
"Folder"="Flash Disk"
Click to expand...
Click to collapse
* MBR-ready: MBR of your ROM already have Flash Disk partition, there are 4 partition slots in MBR.
00000000 ...
...
000001B0 __ __ __ __ __ __ __ __ __ __ __ __ __ __ 11 11
000001C0 11 11 11 11 11 11 11 11 11 11 11 11 11 11 22 22
000001D0 22 22 22 22 22 22 22 22 22 22 22 22 22 22 33 33
000001E0 33 33 33 33 33 33 33 33 33 33 33 33 33 33 44 44
000001F0 44 44 44 44 44 44 44 44 44 44 44 44 44 44 55 AA
Click to expand...
Click to collapse
- The last 2 bytes (55 AA) is a signature, it means END OF MBR.
- 5th byte of each slot is value for partition type, some mostly used values are:
+ 20 - BOOT: just a name, it isn't actually a boot partition and your MBR can be without it.
+ 23 - RAWFS: your XIP partition.
+ 25 - IMGFS: your SYS + OEM partition.
+ 00 - none: free slot, all value in this slot should be 00, too.
+ 01, 04, 06 - FATFS: our Flash Disk here! In some other devices, it's called "Internal Storage".
There are some other values but that's another matter!
So, if your MBR have a FATFS partition type in last used slot, you have a MBR-ready ROM.
- Two non-MBR-ready cases: (with and without BOOT partition slot)
00000000 ...
...
000001B0 __ __ __ __ __ __ __ __ __ __ __ __ __ __ 11 11
000001C0 11 11 20 11 11 11 11 11 11 11 11 11 11 11 22 22
000001D0 22 22 23 22 22 22 22 22 22 22 22 22 22 22 33 33
000001E0 33 33 25 33 33 33 33 33 33 33 33 33 33 33 00 00
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA
Click to expand...
Click to collapse
00000000 ...
...
000001B0 __ __ __ __ __ __ __ __ __ __ __ __ __ __ 11 11
000001C0 11 11 23 11 11 11 11 11 11 11 11 11 11 11 22 22
000001D0 22 22 25 22 22 22 22 22 22 22 22 22 22 22 00 00
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA
Click to expand...
Click to collapse
BLUE free slot must be entered manually (by HEX caculating) to make them MBR-ready!
* Updated to second #2 post!

VNInt said:
* What is XIP-ready?
- boot.rgu in your XIP already have Flash Disk related key and values, they are:
- Note: no space in key [HKEY_LOCAL_MACHINE\System\StorageManager\Profiles\TRUEFFS_DOC\FATFS], I don't know why there is a space before TRUEFFS_DOC, please manually delete that space when copy!
* What is MBR-ready?
MBR of your ROM already have Flash Disk partition, there are 4 partition slots in MBR.
- The last 2 bytes (55 AA) is a signature, it means END OF MBR.
- 5th byte of each slot is value for partition type, some mostly used values are:
+ 20 - BOOT: just a name, it isn't actually a boot partition and your MBR can be without it.
+ 23 - RAWFS: your XIP partition.
+ 25 - IMGFS: your SYS + OEM partition.
+ 00 - none: free slot, all value in this slot should be 00, too.
+ 01, 04, 06 - FATFS: our Flash Disk here! In some other devices, it's called "Internal Storage".
There are some other values but that's another matter!
So, if your MBR have a FATFS partition type in last used slot, you have a MBR-ready ROM.
- Two non-MBR-ready cases: (with and without BOOT partition slot)
BLUE free slot must be entered manually (by HEX caculating) to make them MBR-ready! But... how to HEX caculate values? Maybe later with a MBR sample!
Click to expand...
Click to collapse
how to HEX caculate values?

yanqichun9527 said:
how to HEX caculate values?
Click to expand...
Click to collapse
Do not only ask, just follow my instruction, if it do not work for you or your MBR are not ready, post your 512 bytes MBR and your boot.rgu here!
I will use them as samples to continue.

Excellent tutorial VNint..Thanks!

VNInt said:
Do not only ask, just follow my instruction, if it do not work for you or your MBR are not ready, post your 512 bytes MBR and your boot.rgu here!
I will use them as samples to continue.
Click to expand...
Click to collapse
It's Ready I think.Just Want To Know Answer

yanqichun9527 said:
It's Ready I think.Just Want To Know Answer
Click to expand...
Click to collapse
OK! As we know, values are saved in reversed order and maximum size is 1.F8.00h sectors, in decimal is 129.024 sectors = 66.060.288 Bytes (1 sector = 512 Bytes).
From now, we work with HEX only.
* Some definitions:
1. Value from $1x6 (4 Bytes) (x is C, D, E or F) is address offset of partition's first sector a.k.a number of used sectors by all previous partitions.
2. Value from $1xA (4 Bytes) (x is C, D, E or F) is number of sectors in partition a.k.a size of partition.
* Some formularies:
number of used sectors = size of last partition + address offset of last partition
size of your partition = maximum size - number of used sectors
In your sample:
number of used sectors = 1.BA.00h + 1A.80h = 1.D4.80h
size of your partition = 1.F8.00h - 1.D4.80h = 80.23h

lol how about to remove them from roms?

mr4r4n said:
lol how about to remove them from roms?
Click to expand...
Click to collapse
So easy! Just zero-fill FATFS slot in your MBR!

Nice tutorial VNInt.

Sorry
I have stupid Question....
Can we have Internal Device Storage Memory more than usually?
For example, ROM from VNInt has Internal Memory ~41MB, Storage ~9.9MB & Flashdisk ~21MB. Can we combine all of the memory (internal,storage & flashdisk) to One Place memory, so we can have Internal memory 41 + 9.9 + 21 = ~73MB? ..
Sorry 4 my bad English & my stupid Question

aladin said:
Sorry
I have stupid Question....
Can we have Internal Device Storage Memory more than usually?
For example, ROM from VNInt has Internal Memory ~41MB, Storage ~9.9MB & Flashdisk ~21MB. Can we combine all of the memory (internal,storage & flashdisk) to One Place memory, so we can have Internal memory 41 + 9.9 + 21 = ~73MB? ..
Sorry 4 my bad English & my stupid Question
Click to expand...
Click to collapse
The answer is NO, but this is an exciting idea!
I will try some hack to get it true if it can be!

aladin said:
Sorry
I have stupid Question....
Can we have Internal Device Storage Memory more than usually?
For example, ROM from VNInt has Internal Memory ~41MB, Storage ~9.9MB & Flashdisk ~21MB. Can we combine all of the memory (internal,storage & flashdisk) to One Place memory, so we can have Internal memory 41 + 9.9 + 21 = ~73MB? ..
Sorry 4 my bad English & my stupid Question
Click to expand...
Click to collapse
My Stupid Question has answered... look at this
Code:
http://forum.xda-developers.com/showthread.php?t=468776
My Question Again. How to make it?

aladin said:
My Stupid Question has answered... look at this
Code:
http://forum.xda-developers.com/showthread.php?t=468776
My Question Again. How to make it?
Click to expand...
Click to collapse
* Yeah! He got it hacked!
* How it work? This trick is made by repartitioning DiskOnChip.
- This is original DOC layout: (mtty/task 28)
Binary0 Size: 0x100000
FAT0 Size: 0x4000000 (67108864 bytes = 64 MB) (63 MB maximum ROM and Flash Disk size + 1 MB splash image)
FAT1 Size: 0xA00000 (10485760 bytes = 10 MB Extended ROM size)
FAT2 Size: 0x2C70000 (46596096 bytes = 44 MB Storage size)
All Size: 0x7770000
FAT0_ADDR=0x100000,FAT1_ADDR=0x4100000,FAT2_ADDR=0x4B00000
- And this is modified DOC layout after Cotulla's ROM:
Binary0 Size: 0x100000
FAT0 Size: 0x2A00000 (44040192 bytes = 42 MB maximum ROM size)
FAT1 Size: 0x40000 (262144 bytes = 256 KB Extended ROM resized but hidden away)
FAT2 Size: 0x4C20000 (79822848 bytes = 76 MB Storage gained size)
All Size: 0x7760000
FAT0_ADDR=0x100000,FAT1_ADDR=0x2B00000,FAT2_ADDR=0x2B40000
* How to make it?
There is a DOC repartitioning tool released by mamaich before with his source code. I'm trying to got it work unattendedly, so hard now.

Related

[Q] How to convert NDumpCE6 img to bin

Hi all,
i got a new incar navigation system. it has wince 6 installed (Blaupunkt New Nork 800 which is mostly a rebranded ADVENT ADVUV630).
Now, for troubleshooting'n'stuff i've made an Dump from that device using NDumpCE6. I what to use this dump to load it into the Windows CE Emulator form Microsoft.
is there any chance to get it to work?
i got 4 Dumps over all:
Complete Disk: (CF-Card)
DSK1.img - 249 MB (261.095.424 Bytes)
First 16 Hex Values: e9 fd ff ff ff ff ff ff ff ff ff ff ff ff ff ff
Then, i also dumped all Partitions seperatly:
Part00.img - 41,2 MB (43.251.712 Bytes)
First 16 Hex Values: fe 03 00 ea 29 2a 28 00 2a 2a 29 00 2a 2a 2a 00
Part01.img - 30,7 MB (32.243.712 Bytes)
First 16 Hex Values: eb 76 90 45 58 46 41 54 20 20 20 00 00 00 00 00
Which equals to : .vEXFAT (<<< So thats interesting...)
Part02.img - 174 MB (182.976.512 Bytes)
First 16 Hex Values: eb fe 90 4d 53 57 49 4e 34 2e 31 00 08 01 20 00
Which equals to : ...MSWIN4.1....
And little futher : 20 20 46 41 54 33 32 20 20 20 00 00 00 00 00 00
: FAT32 (<<< So thats also interesting...) NOT Anymore - its a Microsoft Windows Boot Record (http://thestarman.pcministry.com/asm/mbr/MSWin41BRinHexEd.htm )
So PART02 seems to hold the boot partition....
Update: Funny thing - i just tried to mount part02.img into DAEMON-Tools - and guess what: it works!
i found one Directory called: CE69
In that Folder i found some Files and other Folders...
arial-uni2.1.ttf
Bluetooth.dll
CE69.EXE
DriveInterfaceCTL.dll
HYDIB.DLL
INIFILE.DLL
INSTALL.INI
Memory.dll
MFCCE400.DLL
MPU.BIN
OSCtrl.exe
Protocol.dll
RunAppPath.txt
SAMPLE.DUI
SYSTEM.INI
UIContainer.dll
UIDesignerDLL.dll
UIFC.DLL
Upgrade.exe
XMRadio.dll
uifilters <FOLDER>
Resource <FOLDER>
Things i tried allready:
Use dumprom.exe to extract anything. Well i got something out of the first partition - but thats only some wince files:
binfs.dll
BINFSCheck.dll
boot.hv
busenum.dll
ceddk.dll
coredll.dll
default.hv
device.dll
devmgr.dll
filesys.dll
flashdrv.dll
fsdmgr.dll
hd.dll
i2c.dll
initdb.ini
initobj.dat
k.ceddk.dll
k.coredll.dll
k.fpcrt.dll
kernel.dll
mspart.dll
nk.exe
oalioctl.dll
osaxst0.dll
pm.dll
regenum.dll
romfsd.dll
sdbus.dll
sdhc.dll
sdmemory.dll
servicesd.exe
user.hv
utldrv.dll
wince.nls
okay - im now try to find something out about that exfat dump and if i could load it somehow...
ofcourse, any help is very welcomed ;-)
Hello,
I may be able to help you as I have the advuv630, but I am curious how do you get the NdumpCE6 to run in the first place on your unit? I see everyone saying "run it" but never how lol.... like do you name it something special and put it on sd card or something?
--thesh0ck
wow, 2012 this post. well to revive if possible, any luck. Messing with my wince head unit in my truck

Xperia J: fast+snappy stock kernel JB (11.2.A.0.21)

Xperia J: fast+snappy stock kernel JB (11.2.A.0.21)
The following guide shows how to build the stock sources for you
stock xperia j phone with JB.
When Foxcon adopted the drivers for the xperia j they left in
a huge amount of debug which slows down your phone.
When we finally build our own kernel (Chapter 7)
we optimize it by:
- optimizing kernel for speed (not size)
- remove a huge amount of debug from the drivers
- remove kernel core debug
- remove debug_fs
- build without module support
In my opinion the kernel feels much more snappier afterwards.
The system reacts much more fluent on user inputs and sound
has less hangs than before.
Let me hear how it feels for you and if you like it or not
Maybe you have further modifications. So please post it here.
The last section describes how you can build your own kernel patch files.
0. Prerequisites
1. Extracting the current boot image
2. Splitting the image into kernel, ramdisk and cmdline
3. Unpack the ramdisk
4. Build the sony kernel with the original kernel configuration (.config)
5. Build a new boot image
6. Flash the new boot image to the phone
7. Now for the FUN part: TUNE the sony kernel with the attached patch-file
A: Howto build a patch file by comparing a
fresh extracted kernel sources with your edited sources:
0. Prerequisites
==================
- Device needs to be rooted and bootloader unlocked !!!
- Device needs to be up to date with latest Jelly Bean release 11.2.A.0.21 !!!
- A linux machine as working environment
- free ARM compiler, lite version, EABI, URL:
sourcery.mentor.com/sgpp/lite/arm/portal/release2322
- latest Xperia Jlo sources, URL:
developer.sonymobile.com/downloads/xperia-open-source-archives/open-source-archive-for-build-11-2-a-0-21/
- phyton script 'mkelf.py' to re-/build parition image, URL:
dl-developer.sonymobile.com/tools/image_generation_script_for_Xperia_smartphones.zip
- another basic guide, URL:
developer.sonymobile.com/2011/05/06/how-to-build-a-linux-kernel
1. Extracting the current boot image
======================================
- Install Andrdoid SDK.
- Then add a path to your .bashrc file of your linux host:
linux-w49x:~/my_kernel # echo "export PATH=/root/adt-bundle-linux-x86-20130219/sdk/platform-tools:$PATH" >> ~/.bashrc
linux-w49x:~/my_kernel # . ~/.bashrc
- Enable "USB-Debugging" in the phone seetings
- Start the phone and connect via USB to your linux machine
- At first we copy the sony boot image to the sdcard of the device:
linux-w49x:~ # adb shell
[email protected]:/ $ su
[email protected]:/ # dd if=/dev/block/mmcblk0p3 of=/sdcard/sony_boot.img
40960+0 records in
40960+0 records out
20971520 bytes transferred in 2.078 secs (10092165 bytes/sec)
- Read kernel config of your current kernel and store it on sdcard, too:
[email protected]:/ $ su
[email protected]:/ # cat /proc/config.gz > /sdcard/sony_config.gz
130|[email protected]:/ $ exit
130|[email protected]:/ $ exit
- Transfer both to your linux PC:
linux-w49x:~ # adb pull /sdcard/sony_boot.img
linux-w49x:~ # adb pull /sdcard/sony_config.gz
2. Splitting the image into kernel, ramdisk and cmdline
========================================================
- Basically the image consists of:
* 4k singed sin header with a ?x509? certificate
* kernel
* ramdisk
* cmdline parameters for the kernel
* a lot of empty space (~15 MB)
- Hexdump the image to make it human readable:
linux-w49x:~ # hexdump -C sony_boot.img > dump
linux-w49x:~ # head dump
00000000 7f 45 4c 46 01 01 01 61 00 00 00 00 00 00 00 00 |.ELF...a........|
00000010 02 00 28 00 01 00 00 00 00 80 20 00 34 00 00 00 |..(....... .4...|
00000020 00 00 00 00 00 00 00 00 34 00 20 00 03 00 00 00 |........4. .....|
00000030 00 00 00 00 01 00 00 00 00 10 00 00 00 80 20 00 |.............. .| <== Byte 9/10/11: is kernel start (after 4k sin header)
00000040 00 80 20 00 68 70 3c 00 68 70 3c 00 00 00 00 00 |.. .hp<.hp<.....| <== Byte 9/10/11: is kernel length
00000050 00 00 00 00 01 00 00 00 68 80 3c 00 00 00 40 01 |........h.<[email protected]| <== Byte 9/10/11: is ramdisk start
00000060 00 00 40 01 c6 1b 15 00 c6 1b 15 00 00 00 00 80 |[email protected]| <== Byte 9/10/11: is ramdisk length
00000070 00 00 00 00 04 00 00 00 2e 9c 51 00 00 00 00 00 |..........Q.....| <== Byte 9/10/11: is cmdline start
00000080 00 00 00 00 00 02 00 00 00 02 00 00 00 00 00 20 |............... | <== Byte 9/10/11: is cmdline length (512 characters)
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
- Now read the addresses from behind:
00 10 00 => is 0x001000
68 70 3c => is 0x3c7068
68 80 3c => is 0x3c8068
c6 1b 15 => is 0x151bc6
2e 9c 51 => is 0x519c2e
00 02 00 => is 0x000200
- NOTE: THE NUMBERS WILL BE DIFFERENT FOR YOUR st26i DEVICE !!!
- Use 'dd' to split sony_boot.img into its single parts.
- BUT use the correct numbers from your kernel here:
linux-w49x:~ # dd skip=$((0x001000)) count=$((0x3c7068)) bs=1 if=sony_boot.img of=sony_kernel.img
3960936+0 records out
3960936 bytes (4.0 MB) copied, 66.2326 s, 59.8 kB/s
linux-w49x:~ # dd skip=$((0x3c8068)) count=$((0x151bc6)) bs=1 if=sony_boot.img of=sony_ramdisk.img.gz
1383366+0 records in
1383366+0 records out
1383366 bytes (1.4 MB) copied, 23.1965 s, 59.6 kB/s
linux-w49x:~ # dd skip=$((0x519c2e)) count=$((0x000200)) bs=1 if=sony_boot.img of=sony_cmdline.img
512+0 records in
512+0 records out
512 bytes (512 B) copied, 0.00931558 s, 55.0 kB/s
3. Unpack the ramdisk
=======================
- First unzip ramdisk (should start with 0x1f 0x8b )
linux-w49x:~ # hexdump -C sony_ramdisk.img.gz | head
00000000 1f 8b 08 00 a0 a8 50 51 00 03 ec 3d 69 73 db b8 |......PQ...=is..|
00000010 92 f9 fa f4 2b b0 72 ea cd d8 ab 83 92 cf 64 ca |....+.r.......d.|
00000020 5b 2b 5b b4 ad 7a b6 e4 91 e4 1c b5 f5 4a a1 48 |[+[..z.......J.H|
00000030 48 42 99 57 78 58 f1 ce e4 bf 6f 37 c0 03 a4 a8 |HB.WxX....o7....|
00000040 83 49 66 6a 76 ca aa 38 a6 c0 ee 46 a3 bb d1 68 |.Ifjv..8...F...h|
00000050 a0 01 58 39 55 4e 95 96 a2 28 ea 61 eb 4c 85 df |..X9UN...(.a.L..|
00000060 ca 51 4b ed 2a c5 9f 37 c7 ad a3 ab f6 71 f7 38 |.QK.*..7.....q.8|
00000070 57 7e 16 fd 6e af c1 4b ca 1b af 14 b9 be 2b 8e |W~..n..K......+.|
00000080 dc ea 1c ad c1 6b 65 bf 25 5f b7 d4 d7 4a e8 85 |.....ke.%_...J..|
00000090 f4 89 da 81 d1 98 3b a6 31 63 fe a2 e1 e9 af 5e |......;.1c.....^|
linux-w49x:~ # gunzip sony_ramdisk.img.gz
- Now again, the resulting sony_ramdisk.img should start with 0x30 0x37 0x30.
- Now extract cpio archive to a directory:
linux-w49x:~ # mkdir ramdisk
linux-w49x:~ # cd ramdisk/
linux-w49x:~/ramdisk # cpio -i < ../sony_ramdisk.img
4444 blocks
linux-w49x:~/ramdisk # ls
charger init init.qcom.ril.path.sh init.s1.rc logo.rle sys
data init.Sony.rc init.qcom.sh init.target.rc logo.rle.org system
default.prop init.goldfish.rc init.qcom.usb.rc init.trace.rc proc ueventd.Sony.rc
dev init.qcom.class_core.sh init.qcom.usb.sh init.usb.rc res ueventd.goldfish.rc
fstab.msm7627a init.qcom.class_main.sh init.rc init.usbmode.sh sbin ueventd.rc
- OPTIONAL: You can now modify the ramdisk to your needs... we will
repack it later from here.
4. Build the sony kernel with the original kernel configuration (.config)
===========================================================================
In this section we will just rebuild the sony kernel with its default config.
This step is optional. You might want to SKIP THIS STEP and continue dircetly
with building the optimized kernel (Chapter 7)
- Install the mentioned ARM compiler from Mentor (for URL, see top of page)
- Extract the kernel config we fetched from the device:
linux-w49x:~ # gunzip sony_config.gz
- Extract the sony kernel:
linux-w49x:~ # tar -xf 11.2.A.0.21.tar.bz2
- Add the config to the kernel base directory:
linux-w49x:~ # cp sony_config ./kernel/.config
linux-w49x:~ # cd kernel
linux-w49x:~/kernel # ARCH=arm CROSS_COMPILE=/root/CodeSourcery/Sourcery_CodeBench_Lite_for_ARM_EABI/bin/arm-none-eabi- make oldconfig
- OPTIONAL: reconfigure the kernel OR skip this step:
linux-w49x:~/kernel # ARCH=arm CROSS_COMPILE=/root/CodeSourcery/Sourcery_CodeBench_Lite_for_ARM_EABI/bin/arm-none-eabi- make menuconfig
- Because the ARM compiler is pretty strict, edit the kernel Makefile in "kernel/Makefile":
- Change this part....
ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
KBUILD_CFLAGS += -Os
else
KBUILD_CFLAGS += -O2
endif
- .. by appending this to the flags:
ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
KBUILD_CFLAGS += -Os $(call cc-disable-warning,maybe-uninitialized,) $(call cc-disable-warning,implicit-function-declaration,) $(call cc-disable-warning,strict-prototypes,) $(call cc-disable-warning,unused-function,) $(call cc-disable-warning,unused-variable,)
else
KBUILD_CFLAGS += -O2 $(call cc-disable-warning,maybe-uninitialized,) $(call cc-disable-warning,implicit-function-declaration,) $(call cc-disable-warning,strict-prototypes,) $(call cc-disable-warning,unused-function,) $(call cc-disable-warning,unused-variable,)
endif
- Finally we build the kernel:
linux-w49x:~/kernel # ARCH=arm CROSS_COMPILE=/root/CodeSourcery/Sourcery_CodeBench_Lite_for_ARM_EABI/bin/arm-none-eabi- make
- Time to grab a BIG 0xCOFFEE
5. Build a new boot image
===========================
- Collect the new kernel:
linux-w49x:~ # cp ~/kernel/arch/arm/boot/zImage my_kernel.img
- Pack a new ramdisk (or just take exsiting one)
linux-w49x:~ # cd ramdisk
linux-w49x:~/ramdisk # find . | cpio --quiet -H newc -o | gzip > ../my_ramdisk.img.gz
linux-w49x:~/ramdisk # cd ..
- Pack everything together using mkelf.py from sony (URL, see above):
linux-w49x:~ # python mkelf.py -o my_boot.img [email protected] [email protected],ramdisk [email protected],cmdline
6. Flash the new boot image to the phone
==========================================
- power off device
- vol up + attach usb = fastboot
linux-w49x:~ # fastboot flash boot ./my_boot.img
linux-w49x:~ # fastboot reboot
- If anything goes wrong you can always flash the extraced image using:
linux-w49x:~ # fastboot flash boot ./sony_image.img
linux-w49x:~ # fastboot reboot
7. Now for the FUN part: TUNE the sony kernel with the attached patch-file
=============================================================================
- Basically we disable "module support" as we have none
- Disable A LOT OF DEBUG: debugfs and various debug statments in MSM drivers
- Tune vibration period to be more gentle
- Optimize kernel size for speed and not for size
- Remove kernel and user space process debug infos
- Make sure you unpack the stock sony kernel sources. The sources need to be
fresh and clean!!!
linux-w49x:~ # tar -xf 11.2.A.0.21.tar.bz2
linux-w49x:~ # cd kernel/
- patch the performance tweaks to it. The perf_tweak.patch is appended to this post:
linux-w49x:~/kernel # patch -p3 < ../perf_tweak.patch
patching file ./kernel/power/earlysuspend.c
patching file ./Makefile
patching file ./arch/arm/mach-msm/smd_pkt.c
patching file ./arch/arm/mach-msm/sdio_cmux.c
patching file ./arch/arm/mach-msm/reset_modem.c
patching file ./arch/arm/mach-msm/qdsp5v2/mi2s.c
patching file ./arch/arm/mach-msm/qdsp5v2/audio_out.c
patching file ./arch/arm/mach-msm/modem_notifier.c
patching file ./arch/arm/mach-msm/msm_cpr-debug.c
patching file ./arch/arm/mach-msm/smd_rpcrouter.c
patching file ./arch/arm/mach-msm/ipc_router.c
patching file ./arch/arm/mach-msm/bam_dmux.c
patching file ./arch/arm/mach-msm/qdsp6/msm_q6vdec.c
patching file ./arch/arm/mach-msm/rmt_storage_client.c
patching file ./arch/arm/mach-msm/include/mach/debug_mm.h
patching file ./arch/arm/mach-msm/clock.c
patching file ./arch/arm/mach-msm/sdio_dmux.c
patching file ./arch/arm/mach-msm/msm_cpr.h
patching file ./arch/arm/mach-msm/qdsp5/audio_mp3.c
patching file ./arch/arm/mach-msm/qdsp5/audmgr.c
patching file ./arch/arm/mach-msm/qdsp5/audio_acdb.c
patching file ./arch/arm/mach-msm/qdsp5/audio_lpa.c
patching file ./arch/arm/mach-msm/pm2.c
patching file ./arch/arm/mach-msm/sdio_ctl.c
patching file ./arch/arm/mach-msm/clock-debug.c
patching file ./arch/arm/mach-msm/board-msm7627a-display.c
patching file ./arch/arm/mach-msm/vreg.c
patching file ./arch/arm/mach-msm/board-tamsui-jlo.c
patching file ./arch/arm/mach-msm/clock.h
patching file ./net/netfilter/xt_socket.c
patching file ./include/linux/vibrator_class.h
patching file ./include/linux/bma250.h
patching file ./drivers/media/common/tuners/xc4000.c
patching file ./drivers/tty/serial/msm_serial_hs.c
patching file ./drivers/vibrators/fih_vibrator.c
patching file ./drivers/vibrators/vibrator_class.c
patching file ./drivers/video/msm/msm_fb.c
patching file ./drivers/video/msm/mipi_orise.c
patching file ./drivers/bluetooth/bluesleep.c
patching file ./drivers/usb/otg/msm_otg.c
patching file ./drivers/usb/otg/msm72k_otg.c
patching file ./drivers/usb/gadget/f_diag.c
patching file ./drivers/usb/gadget/u_ctrl_hsuart.c
patching file ./drivers/usb/gadget/f_rmnet_smd_sdio.c
patching file ./drivers/usb/gadget/u_serial.c
patching file ./drivers/usb/gadget/u_bam.c
patching file ./drivers/usb/gadget/f_rmnet_smd.c
patching file ./drivers/input/keyboard/fih_gpio_keys.c
patching file ./drivers/input/keyboard/fih_power_key.c
patching file ./drivers/input/touchscreen/cyttsp_core.c
patching file ./drivers/input/sensor/qpdss702.c
patching file ./drivers/leds/fih_led.c
patching file ./drivers/net/wireless/bcmdhd/wl_linux_mon.c
patching file ./drivers/net/wireless/bcmdhd/Makefile
patching file ./drivers/net/wireless/bcmdhd/dhd_custom_gpio.c
patching file ./drivers/power/fih_bq27520_fuelgauger.c
patching file ./drivers/power/fih_msm_battery.c
patching file ./drivers/gpu/msm/adreno_postmortem.c
patching file ./drivers/gpu/msm/adreno.c
patching file ./drivers/rtc/rtc-msm.c
patching file ./.config
linux-w49x:~/kernel # ARCH=arm CROSS_COMPILE=/root/CodeSourcery/Sourcery_CodeBench_Lite_for_ARM_EABI/bin/arm-none-eabi- make
===> repeat steps 5) and 6) but use our new zImage.
A: Howto build a patch file by comparing a
fresh extracted kernel sources with your edited sources:
=========================================================
- Compare two kernel directories and create a patch from it:
linux-w49x:~ # export BASEDIR=$PWD
linux-w49x:~ # cd kernel/
linux-w49x:~/kernel # rm $BASEDIR/perf_tweak.patch
linux-w49x:~/kernel # find -name '*.c' -o -name '*.h' -o -name 'Makefile' -o -name '.config' | xargs [email protected] diff -upN $BASEDIR/kernel/@ $BASEDIR/my_kernel/@ >> $BASEDIR/perf_tweak.patch
Great Tutorial for Xperia J custom Kernel
Thanks .. Really helpful .
numbers were different in mine
dd skip=$((0x001000)) count=$((0x3da520)) bs=1 if=sony_boot.img of=sony_kernel.img
dd skip=$((0x3db520)) count=$((0x11a498)) bs=1 if=sony_boot.img of=sony_ramdisk.img.gz
dd skip=$((0x4f59b8)) count=$((0x000200)) bs=1 if=sony_boot.img of=sony_cmdline.img
I am with locked bootloader ( 1 week old JLo ) will do it as soon I unlock it.
I will also include swap support in kernel config and test .
omg it's so complicated...
have anyone finished it? Will it be released as flashable version ?
Lol ye makes me dizzy!
work on xperia J
this work on xperia J with bootloader locked
For simplicity's sake I like to build my kernels with CyanogenMod, but I'll check out your patch for some useful edits. :good:
Don't want to necro bump threads but....
Massive thanks to OP
I'd been messing around for the last day with trying to compile the .31 stock kernel from source.
Successfully used the above guide to dump the .31 kernel from phone add overclock to .31 source code (from Vengeance 1.42 source) compile, make boot.img and flash. :laugh:
Can I use this patch for the newest JB kernel? will it work?

[RECOVERY] [TWRP] Backup Converter Android system recovery <3e>

- for Linux only -
Stock Recovery to TWRP Backup Converter for Android system recovery <3e>
This progam is basically written for unpacking stock recovery android backup userdata_20160823_100259.backup + convert it into custom recovery nandroid backup data.ext4.win000 (but you can create your own TWRP Backups from "any" data source, too)
content and usage of bckp2win.sh is similar to bckp2cwm.sh with some slight modifications. based on previous version, it skips the checksum and unpack /data partition from userdata_00000000_000000.backup then re-pack it as TWRP Backup. optionally the screenlock pattern can be unlocked.
Requirements:
- pc with linux
- ext4 formatted hard disk
may work on ntfs, give a try (in case backup is a partition image)
Requirements (source phone):
- Android system recovery <3e> with
- "backup user data" functionality
- data must not encrypted
- external sdcard
Requirements (target device):
- root
- TWRP custom recovery
- working identical ROM pre-installed (like source phone)
before you start:
download this flashable UPDATE-sdcard.Fix.Permissions-signed.zip from osm0sis @ xda-developers to your phones memory or external sdcard - you might need it later
http://forum.xda-developers.com/showthread.php?t=2239421
TWRP and Internal Storage:
even if TWRP recovery process claims not touching /data/media, it restores files anyway. this is a great advantage side effect as we can easily restore Pictures and Files by simply including it in the backup. However, this will overwrite existing data - please don't use this option unless you know what you're doing!
if apps crashing after restoring from TWRP, this might have to do with Internal Storage - the above flashable zip will fix permissions, ownership and selinux labels for /data/media in case you manually added some files (regarding /data - of course - there is no tool in the world, which can do the same for /data partition - be warned never copy files, just always move files from one linux file system to another, and never use a windows file system)
bckp2win.sh is a linux bash script using GNU tar for creating TWRP archive files from userdata_yyyymmdd_hhmmss.backup files.
in TWRP Backup each data.ext4.win000 file represent a standalone tarball archive - this means each single archive can be unpacked for its own - without concatenating them, or having splitted files spreaded over multiple archives. unfortunately i don't know how they do it (i think TWRP use its own tar implementation), so i decided to write another bash script wich is basically doing the same thing (creating multipart standalone tarball archives):
edit: this is the main converting script (and the only file you need)
multi_tar.sh is not limited to Android system recovery <3e> userdata backup and can be used for any scope of application.
This means you can simply create TWRP Backups from "any" data source. It is summarizing files in a index file until archive size is reached and then archiving from index with GNU tar. This is a very slow procedure but it works. optionally it uses GZIP compression. (i really dont know how to check compressed file size from bash without compressing it, therefore it is compressed twice in a 2 pass way, 1-st pass is for checking size only)
edit: do not download this script, try bckp2win.sh without multi_tar.sh first (press No when asked). it is for splitting large backups only and not required in most cases
twrp_sign.sh is another bash script for creating sha2 checksums especially for TWRP Backups. But checksums can be disabled in TWRP - therefore its optional.
needs ~ 120% of free disk space and takes time about ~ 30 min, enjoy your coffee
[TUTORIAL] How to convert stock backup into TWRP backup
First of all you need to know, that userdata_yyyymmdd_hhmmss.backup files contain user data only. it is NOT a full nandroid backup like TWRP / CWM.
So we can just restore data partition from TWRP:
userdata_20160823_100259.backup -> data.ext4.win000 -> /data
The data partition contains the Internal Virtual SD Card. It is usually not included in TWRP backup. It is your decision to manually copy back to phone (recommended). But you can also restore Internal Virtual SD Card from TWRP:
/data/media/0 -> /storage/emulated/0
IMEI and WiFi MAC-Address:
/data/nvram (NOT recommended)
If there is a copy of NVRAM partition in folder /data/nvram, the script will delete it by default. However, there is a option to use IMEI and WiFi MAC-Address from backup (clone), instead from Phone.
Windows Users please click here
download UNetbootin -> https://unetbootin.github.io <- scroll this page down for tutorial
format USB flash drive FAT32install any Linux Distro to USB flash drive
Reboot your PC from USB flash drive
to access the boot menu while booting your computer:
- press the appropriate key F11 or F12 during the initial startup screen
- select the USB boot option in the BIOS boot menu
- boot Default entry
congratulations, you have
sucessfully entered your
own working Linux system!
it's easy, isn't it?
Now, come back to forum.xda-developers.com
- find Start Menu on upper left corner
- open the "Web Browser" from Favorites
- search for "bckp2win" in google
you can add your Keyboard Layout from: Settings - Keyboard
Note: performance of Firefox badly depends on USB flash drive speed
if it is too slow, try another one, or disable persistence:
- find syslinux.cfg file on USB flash drive
- open syslinux.cfg file with editor
- delete --- persistent from Try Xubuntu without installing entry
- save the changes, reboot from USB flash drive Try Xubuntu without installing entry
However, without persistence it is a read-only Live system and will lose all settings on reboot
Let's begin with preparations
- copy UPDATE-sdcard.Fix.Permissions-signed.zip to microSD card / or
- download UPDATE-sdcard.Fix.Permissions-signed.zip to target phone
- connect the source phone USB cable / or
- insert the microSD card into the PC's SD Card Reader
- copy all userdata_20160823_100259.backup files to any folder on local disk
- download all the zip files from this thread
- unpack zip files to same folder on local disk
run the shell script in Terminal
- do right-click somewhere in the backup folder, select "Open Terminal Here"
- type "sudo bash bckp2win.sh" in Terminal
- check disk space
example: when backup is 2 GB,
Avail must > 2,4 GB (120%)
when backup is 55 GB,
Avail must > 66 GB (120%)
- check file system type
Type must not vfat, fat32
all others allowed
(ntfs, ext4, fuseblk, ...)
this is very important! otherwise it will fail at the end and waste your time, and you won't know the reason. so please check carefully
- select file number to extract
- wait a long time, depends on disk speed and backup size (1~2 min / 1 GB)
Halftime break
- when message appears Press 'y' to unlock: [y/n] - Congratulations! The backup is sucessfully extracted
(if you need to edit/modify/delete files within backup this is the point for break)
- to continue with re-packing:
answer all questions with No
(or just press Enter for default):
Android sparse image (simg2img) support -> No
Flash-Friendly File System (F2FS) support -> No
unlock screenlock pattern -> No
clone old IMEI and Wifi Mac Address -> No
Restore /storage/emulated/0 -> No
use gzip compression -> No
Extract Internal Virtual SD Card to local disk -> Yes
(this will extract /data/media to TWRP folder instead, but exclude it from backup)
- wait for the script is finished
- wait for background processes
(it happens sometimes script finishes too early, if there is a data.ext4.win000 + data.ext4.win0000 file, just wait for the target size 1 GB each file)
- if checksums missing or failed, please manually run twrp_sign.sh again
(successful checksums look like this)
Finally we can restore the new backup
- copy back TWRP folder to phone
- boot into recovery mode
- create a failsafe backup (just in case...)
- move folder to the right location via MTP, or
- Advanced -> File Manager -> TWRP/BACKUPS/<serialno>/<backup folder>
- Options (blue icon on the right bottom) -> Move -> TWRP/BACKUPS/<phone name> -> Swipe to confirm
(in case converted backup is not visible in Restore list...)
- restore converted backup from TWRP
- install the flashable UPDATE-sdcard.Fix.Permissions-signed.zip
(just in case you have added files to data/media...)
- boot the phone
- move/copy all your pictures videos etc from TWRP folder back to phone
Troubleshooting
If you fail at some point please try again with more disk space or gzip compression
phone is not showing up on PC
enable Settings -> Developer options -> Select USB Configuration -> MTP:
- goto Settings -> About phone
- tab Build number seven times until you see a message
- goto Settings -> Developer options
- enable Developer options, confirm Allow development settings
- scroll down to -> Select USB Configuration
- select MTP (Media Transfer Protocol)
My source backup file is not in "userdata_YYYYMMDD_hhmmss.backup" format
This script may work with other archives, but only accept "userdata_YYYYMMDD_hhmmss.backup" pattern as input file name. But you can specify any input file or folder as parameter. the script will scan the folder for known archive types and link files into script folder:
sudo bash bckp2win.sh ~/Android/Backup/TWRP/2016-08-23--10-02-59/data.ext4.win*
mkdir: cannot create directory ‘b2wtmp’: File exists
There was a previous session left, delete the folder ‘b2wtmp’ and try again. You can keep previous session for testing purposes with parameter -f force unpack:
sudo bash bckp2win.sh -f
ERROR: something goes wrong. check disk space
The GNU tar unpacking or archiving process may fail for various reasons. However, you can suppress error messages and skip this exit point for testing purposes with parameter -f force unpack. If you run out of disk space, you can exclude folders from backup with parameter -e --exclude PATTERN, for example:
sudo bash bckp2win.sh -e */com.google.android.googlequicksearchbox*
Bugs & Known Issues
extractTarFork() process ended with ERROR: 255
probably bug in script. at the moment, only solution is manually restoring backup files. i know its annoying but i don't know the reason, yet.
- download GNU tar for android
- unpack the zip and copy the tar binary to phone
- in TWRP, copy tar binary to /cache, then wipe data
- in TWRP, go back to -> Advanced -> Terminal
- in Terminal, change directory to backup folder, then run for each file
Code:
chmod 0755 /cache/tar
cd /external_sd/TWRP/BACKUPS/<phone>/2016-08-23--10-02-59*
/cache/tar --selinux --xattrs -P -vxpf data.ext4.win000
(or for compressed files)
busybox gzip -cd data.ext4.win000 | /cache/tar --selinux --xattrs -P -vxp
Please post here for support i will answer your questions
found a multipart image, merge files
==================
... merged
try to unpack ...failed
try to mount as *...failed
skip first 512 bytes and try to mount again as ext2, as rfs, as fsfs
... failed
no files in folder "data"
exiting script
What can I do ?
My phone model is Bluboo S1 with Android 7.0, before I wiped all out I was made backup with stock backup and now I have TWRP 3.2.1. Now I have phone flashed with BLUBOO_S1_Helio_P25_L_V04_20170908 and is working but I want put my old userdata because I think there is all my 230 apps already installed.
Please check if your userdata_20180926_141645.backup is ext4 image. Each file start with a 512 byte checksum header, followed by partition image. The ext4 Super Block will start at offset 1024 bytes (from partition). The ext4 magic number 0xEF53 you can find at offset 0x38 (from the Super Block start)
We can skip (512 bytes) checksum + (1024 bytes) unused padding, because the Super Block start is at (1536 bytes) = 0x600 in this case
Please note the ext4 magic 0xEF53 at offset 0x638
encrypted files not supported
If the files look like this (no zeros within, after skip 512 bytes checksum) it is probably a raw backup of encrypted data partition
you can check with hexdump
Code:
hexdump -C -n1600 userdata_20180926_141645.backup
Unfortunately, these userdata backups are pretty useless since android 7.0 (encrypted by default), because they will not backup efs/metadata. If you wipe data from stock recovery, the metadata is wiped, too. It is impossible to decrypt data without encryption key (which is stored in metadata).
If you are really lucky maybe its not to late, do a read back of metadata partition with SP Flash Tool. Furthermore, check if userdata backup is encrypted
. . .
If the files look like this (no zeros within, after skip 512 bytes checksum) it is probably a raw backup of encrypted data partition
you can check with hexdump
Code:
hexdump -C -n1600 userdata_20180926_141645.backup
[/QUOTE]
***
there was also too little disk space and I installed the new Ubuntu on the other computer so I'll try it later
if you prefer to re-pack data by yourself, stop the script when it ask for unlock screenlock pattern (CTRL + C)
enter parent directory of data folder
run multi_tar.sh - while first arg is destination and all other args are source folders
Note you can change the output to any file or folder:
replace "data.ext4.win" with "/media/xubuntu/my-drive/Android/my-output-folder/my-file-name"
Code:
sudo bash multi_tar.sh -z -L 1048576 data.ext4.win data --transform 's,^data,/data,'
(with parameters -z for compression and -L for split size, and some string replacement within the archive - replaceing "data" with "/data" in this case)
there are still some bugs i am struggle with, restoring in TWRP fails when a single file within backup is failed.
Please check out my other solution for encrypted backup. You can restore TWRP backup from this zip instead of restoring from TWRP menu
https://forum.xda-developers.com/showthread.php?t=3899918
aIecxs said:
We can skip (512 bytes) checksum + (1024 bytes) unused padding, because the Super Block start is at (1536 bytes) = 0x600 in this case
Click to expand...
Click to collapse
Hi,
i have exact these offset you wrote here but i do not have success to mount my data. Have you an idea what could be wrong on my side?
Code:
000001d0 ab 0a 44 a3 c5 ee 69 fa 44 78 c2 ca ec 13 bb f5 |..D...i.Dx......|
000001e0 38 f4 e7 ca f2 7c 49 e3 a2 a0 d8 1e e3 f9 94 c5 |8....|I.........|
000001f0 f5 f5 3e c2 6a bc 8a 58 1c ef 0e 8a 91 29 c4 99 |..>.j..X.....)..|
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600 00 40 02 00 80 fc 08 00 00 10 00 00 77 e3 00 00 |[email protected]|
00000610 da 10 02 00 00 00 00 00 02 00 00 00 02 00 00 00 |................|
00000620 00 80 00 00 00 80 00 00 00 20 00 00 0f 8e a4 54 |......... .....T|
00000630 0f 8e a4 54 6c 02 ff ff 53 ef 01 00 02 00 00 00 |...Tl...S.......|
Code:
try to mount as ext4
...failed
skip first 512 bytes and try to mount again as ext4
...failed
try to mount as ext3
...failed
skip first 512 bytes and try to mount again as ext3
...failed
try to mount as ext2
...failed
skip first 512 bytes and try to mount again as ext2
...failed
try to mount as rfs
...failed
skip first 512 bytes and try to mount again as rfs
...failed
try to mount as f2fs
...failed
skip first 512 bytes and try to mount again as f2fs
...failed
./bckp2win.sh: Zeile 796: cd: .//b2wtmp: Datei oder Verzeichnis nicht gefunden
WARNING: no files in folder "data"
exiting script
Thanks
script looks buggy, maybe wrong mount options, or it makes a difference when called with "sudo bash bckp2win.sh"
Hello all.
Is there a way to do the inverse (system.ext4.win -> system.img) ? Does anybody know a link to instruction ?
With that system.img, I can then do: Bootloader> fastboot -S 130M flash system system.img
(if bootloader is unlocked).
Thank you everyone.
yes it is possible, check my edited reply later in 10 hours
edit: see reply in other thread
https://forum.xda-developers.com/showthread.php?t=4015725
@e5e197740b what is the error message when using this tool?
aIecxs said:
@e5e197740b what is the error message when using this tool?
Click to expand...
Click to collapse
Do I need a rooted Tablet for this? I'm not sure I understood all the instructions..
Edit:
So I ran bckpwin.sh in an Ubuntu VM, here is the output:
Code:
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#
#
# This script converts "Android system recovery <3e>" stock recovery file
# "userdata_YYYYMMDD_hhmmss.backup" into custom recovery nandroid backup file.
#
# executing file system must be ext2/3/4 otherwise app permissions will be lost.
#
#
run this script as root. type "sudo -i" or "sudo bash bckp2win.sh".
are you root?
Press 'y' to continue: [y/n] y
2,0G userdata_20200609_220537.backup
2,0G userdata_20200609_220537.backup1
2,0G userdata_20200609_220537.backup2
2,0G userdata_20200609_220537.backup3
2,0G userdata_20200609_220537.backup4
2,0G userdata_20200609_220537.backup5
110M userdata_20200609_220537.backup6
13G total
Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf
/dev/sda5 ext4 98G 45G 48G 49% /
WARNING: Make sure enough free disk space - NOT checked during process!!
1) userdata_20200609_220537.backup
select file number to extract (q to quit): 1
1) "userdata_20200609_220537.backup"
try to unpack as tar (multipart image support)
...failed
WARNING: No ext4 magic number detected. Try it anyway? [y/n] y
OPTION: install support for (EXT4) Android sparse image (simg2img)? [y/n] y
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Paket android-tools-fsutils ist nicht verfügbar, wird aber von einem anderen Paket
referenziert. Das kann heißen, dass das Paket fehlt, dass es abgelöst
wurde oder nur aus einer anderen Quelle verfügbar ist.
Doch die folgenden Pakete ersetzen es:
android-sdk-libsparse-utils android-sdk-ext4-utils
E: Für Paket »android-tools-fsutils« existiert kein Installationskandidat.
OPTION: install support for (F2FS) Flash-Friendly File System? [y/n] y
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Die folgenden zusätzlichen Pakete werden installiert:
libf2fs-format4 libf2fs5
Die folgenden NEUEN Pakete werden installiert:
f2fs-tools libf2fs-format4 libf2fs5
0 aktualisiert, 3 neu installiert, 0 zu entfernen und 3 nicht aktualisiert.
Es müssen 185 kB an Archiven heruntergeladen werden.
Nach dieser Operation werden 568 kB Plattenplatz zusätzlich benutzt.
Holen:1 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 libf2fs5 amd64 1.11.0-1.1ubuntu1 [14,1 kB]
Holen:2 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 libf2fs-format4 amd64 1.11.0-1.1ubuntu1 [16,6 kB]
Holen:3 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 f2fs-tools amd64 1.11.0-1.1ubuntu1 [154 kB]
Es wurden 185 kB in 0 s geholt (608 kB/s).
Vormals nicht ausgewähltes Paket libf2fs5:amd64 wird gewählt.
(Lese Datenbank ... 189666 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereitung zum Entpacken von .../libf2fs5_1.11.0-1.1ubuntu1_amd64.deb ...
Entpacken von libf2fs5:amd64 (1.11.0-1.1ubuntu1) ...
Vormals nicht ausgewähltes Paket libf2fs-format4:amd64 wird gewählt.
Vorbereitung zum Entpacken von .../libf2fs-format4_1.11.0-1.1ubuntu1_amd64.deb ...
Entpacken von libf2fs-format4:amd64 (1.11.0-1.1ubuntu1) ...
Vormals nicht ausgewähltes Paket f2fs-tools wird gewählt.
Vorbereitung zum Entpacken von .../f2fs-tools_1.11.0-1.1ubuntu1_amd64.deb ...
Entpacken von f2fs-tools (1.11.0-1.1ubuntu1) ...
libf2fs5:amd64 (1.11.0-1.1ubuntu1) wird eingerichtet ...
libf2fs-format4:amd64 (1.11.0-1.1ubuntu1) wird eingerichtet ...
f2fs-tools (1.11.0-1.1ubuntu1) wird eingerichtet ...
Trigger für man-db (2.9.1-1) werden verarbeitet ...
Trigger für initramfs-tools (0.136ubuntu6) werden verarbeitet ...
update-initramfs: Generating /boot/initrd.img-5.4.0-37-generic
Trigger für libc-bin (2.31-0ubuntu9) werden verarbeitet ...
found a multipart image, merge files:
../userdata_20200609_220537.backup
../userdata_20200609_220537.backup1
../userdata_20200609_220537.backup2
../userdata_20200609_220537.backup3
../userdata_20200609_220537.backup4
../userdata_20200609_220537.backup5
../userdata_20200609_220537.backup6
(please wait - up to 15 min - don't worry computer is still alive)
...merged
try to unpack as sparse ext4 image (skipping ... )
...failed
try to mount as ext4
...failed
try to mount as ext3
...failed
try to mount as ext2
...failed
try to mount as rfs
...failed
try to mount as f2fs
...failed
no files in folder "data"
caching file to disk again for second run
(please wait - up to 15 min)
try to mount as ext4
...failed
skip first 512 bytes and try to mount again as ext4
...failed
try to mount as ext3
...failed
skip first 512 bytes and try to mount again as ext3
...failed
try to mount as ext2
...failed
skip first 512 bytes and try to mount again as ext2
...failed
try to mount as rfs
...failed
skip first 512 bytes and try to mount again as rfs
...failed
try to mount as f2fs
...failed
skip first 512 bytes and try to mount again as f2fs
...failed
./bckp2win.sh: Zeile 796: cd: .//b2wtmp: Datei oder Verzeichnis nicht gefunden
WARNING: no files in folder "data"
exiting script
So it had to load some stuff of the internet (namely simg2img and F2FS and its dependencies), but it seemed to run through properly, apart from not retrieving any data.
simg2img or f2fs is not required, it's from the days i wasn't aware of backup format
try to unpack as tar ...failed
mean it doesn't extract with tar(gz) with or without 512 bytes header
WARNING: No ext4 magic number detected
mean it did not find hex 53 ef with or without 512 bytes header (f2fs/ext4 is checked both)
but it tries to mount anyway
this could mean
a) script is not working (@matrix4you claimed this, too)
b) header is not 512 bytes
c) backup is encrypted
a) and b) can be double checked with unencrypted backup. do a factory reset then right after create another backup without booting android. this should give you 12 GB backup which can be zipped into less a few MiB because it is empty ext4 image
edit: did you run the script via 'sudo ./bckp2win.sh' that would explain the bug in line 796? usage is 'sudo bash bckp2win.sh' maybe behavior is different
aIecxs said:
simg2img or f2fs is not required, it's from the days i wasn't aware of backup format
try to unpack as tar ...failed
mean it doesn't extract with tar(gz) with or without 512 bytes header
WARNING: No ext4 magic number detected
mean it did not find hex 53 ef with or without 512 bytes header (f2fs/ext4 is checked both)
but it tries to mount anyway
this could mean
a) script is not working (@matrix4you claimed this, too)
b) header is not 512 bytes
c) backup is encrypted
a) and b) can be double checked with unencrypted backup. do a factory reset then right after create another backup without booting android. this should give you 12 GB backup which can be zipped into less a few MiB because it is empty ext4 image
edit: did you run the script via 'sudo ./bckp2win.sh' that would explain the bug in line 796? usage is 'sudo bash bckp2win.sh' maybe behavior is different
Click to expand...
Click to collapse
I did go sudo bash.
I can try the reset-backup thing.
aIecxs said:
simg2img or f2fs is not required, it's from the days i wasn't aware of backup format
try to unpack as tar ...failed
mean it doesn't extract with tar(gz) with or without 512 bytes header
WARNING: No ext4 magic number detected
mean it did not find hex 53 ef with or without 512 bytes header (f2fs/ext4 is checked both)
but it tries to mount anyway
this could mean
a) script is not working (@matrix4you claimed this, too)
b) header is not 512 bytes
c) backup is encrypted
a) and b) can be double checked with unencrypted backup. do a factory reset then right after create another backup without booting android. this should give you 12 GB backup which can be zipped into less a few MiB because it is empty ext4 image
edit: did you run the script via 'sudo ./bckp2win.sh' that would explain the bug in line 796? usage is 'sudo bash bckp2win.sh' maybe behavior is different
Click to expand...
Click to collapse
So what I did now:
Reset the device, without rebooting make a backup:
I still creates 7 files 12 gigs in total.
The are, as expected, seemingly completely empty, apart from the first 512 Bytes, allthough even those are empty in files 4-7.
The first 8 bytes are always the same, both in the empty and the real backup, if the first 512 exist at all.
Here are the first 512 bytes from the first file of the empty backup:
Code:
AB EF CA 9C B0 C5 4A 0A 38 3D BE C5 5E 4F B5 ED DB 14 73 B2 5C 52 64 1B FA 57 83 A1 6A 63 BF 7A 81 4F AF 09 8D 90 07 18 D6 00 68 0B 94 80 62 62 70 4E 35 8E 28 35 2A 65 63 8D 60 6C E6 1B BF 84 22 DC 8D 22 FF BE FA D8 40 A5 37 01 14 DA 80 C0 92 4A 41 24 EA 80 F6 B3 4B 2C 45 A1 17 99 E0 2D DF F5 7E 9D 10 3B 39 DD FE 1E 48 D4 A8 24 81 B4 E1 91 55 BC 2B 80 7D 9E 1B 1C 82 C5 BD 98 59 66 FF 07 00 1E 4F 41 0B 87 4F 17 CC C4 12 2A 50 1E 70 2F 96 61 09 EF 37 B6 A3 7C 01 9C 09 49 D5 0E 98 91 EB 68 09 FE 51 6D D1 7C 4A 15 02 FF 12 5B B2 CA 31 FD 9F 07 8E 26 FA 56 72 F7 BD 55 85 BB F6 F7 59 8C 11 54 B4 E8 04 3E 10 41 B9 32 54 19 A8 BF 0C 5B 40 5D 37 72 68 BA 46 2A E5 C6 98 28 C1 79 DD F6 8E 8F AD 75 43 BA DA 1B 48 DD B9 F8 AB BC 02 6E E3 62 E5 C8 E2 20 60 43 D9 C5 47 E4 81 04 13 A9 04 8C 1E B3 0B 9D 59 B7 D4 CF A9 2F D9 96 93 BA 02 F9 A8 4E D5 FD 7A B2 74 E4 DB 9C 8B 4C 94 CF CD 61 C6 98 E7 88 B7 51 79 E9 0B FB B2 DE 63 66 A5 08 44 24 CE 76 E2 F2 7D 16 80 DA BF 3E F4 58 B3 8C 76 83 62 ED 40 CC 73 6C 28 B8 C7 78 65 8E 11 BB 8D B7 64 4B 66 B7 32 F8 88 14 BB 54 17 D0 52 3D 5A 47 B8 06 D8 00 94 7C 0B FF 7B 07 6E 4A B7 23 A9 6E 91 BB FE 6F 61 C8 0B C5 B3 E9 96 FB 9F 21 33 F8 BF 0A 9A C7 3E ED 31 B5 F5 66 BD 12 1D E9 1F D7 B2 DA 2E FC 3C 22 F4 F4 B6 10 10 BA 7E 9A 41 76 B0 43 91 DE 2E 31 64 59 DD 1E A5 5D 20 D9 5E 17 78 59 12 35 66 35 3A E6 7B 67 D5 64 D7 93 9C F4 01 47 6C 0F 92 4D 45 F9 B8 34 81 0A EA 47 1A 6C 4D 68 C1 6E A0 6C DE 7D 3E 33 06 82 D2 BF 05 82 5D 0E CF 5B E3 15 98 BB 51 98 79 A2 05 8F 7B BC 70 D1 76 35 67 A8 A0 EF
Is there any chance left?
no there is nothing you can do. what does the script say to empty backup? does it mount ext4 image?
So I had to take a few days off, but I ran the script on the empty backup.
It does mount it as a ext4 image. But I can't find the folder it mounted it to.

[TOOL][WIN,LIN,AND,DARW] Super image tools | extract or make partitions RW in super partition

Disclaimer:
Super image tools was made for testing and educational purposes, ME is not responsible for what you do on/with your device using our tools, you must agree that you using our tools on your own risk, I am not responsible for anything else!
How to use superunpack:
- First step, unpack super.sin using my tool or use @IgorEisberg unsin tool
- Step two, Superunpack. On windows just drag and drop unpacked super image onto our exe to start extraction. Also you can use it from command line, from script or from etc. On Linux use it from command line. No need to set slot like it was a case on lpunpack, our tool will auto extract all slot images for you, enjoy!
- If you need to unpack partition images in RW mode add parameter 1 at the end of command line e.g. "superunpack super.img 1", than resize partition using resize2fs, repair and unshare blocks using e2fsck. Or if you unpack without rw you no need to resize or repair it, just mount it ro.
How to manualy patch super partition in under Linux:
https://forum.xda-developers.com/t/...s-rw-in-super-partition.4120963/post-87112415
Note that, superunpack is a tool for extract all logical partitions from super image or directly from super partition.
How to use superrepack:
adb push superrepack.arm64_pie /data/local/tmp
adb shell
su
cd /data/local/tmp
mv superrepack.arm64_pie superrepack
chmod 755 superrepack
stop
./superrepack /dev/block/bootdevice/by-name/super system_a
sync
reboot
Note that, superrepack is a tool to convert logical RO partitions iside your phone super partition to RW mode without extracting anything, all things is done on the fly directly inside super partition/image! In this example system_a partiton is converted to the rw mode, if you need other partitions to rw just change system_a argument. Or if you need all partitions to rw mode do it without partition rw argumet e.g: "./superrepack /dev/block/bootdevice/by-name/super". YOU MUST RUN TOOL 4-5 TIMES UNTIL ALL ERRORS DISAPEARS!!! One of the well known errors is: "Couldn't clone file: Could not allocate block in ext2 filesystem". Look at /data/local/tmp/script.log each time and make sure it not contain any error otherwise you are not done things right and partition is not repaired yet!!! More info -> https://forum.xda-developers.com/t/...s-rw-in-super-partition.4120963/post-84966715
Platform:
- Superunpack is working on Windows, Linux, Android, Darwin11, just chose right binary.
- Superrepack is working only under android
Changelog:
- version 1 (21.Jun.2020), initial version
- version 1.1 (22.Jun.2020), dump file format detection, partition size correction in case ext4, partition group detection, have extraction progress bar, improvements
- version 2 (03.04.2021) implemented possibility to extract partition images to rw mode using Superunpack & I have made new tool called Superrepack
- version 2 (04.04.2021) implemented arguments so you would do conversion on single partition instead of doing it on all partitions
- version 3 (04.04.2021) implemented return codes and implemented output logs to be more scripting friendly
- version 4 - not released
- version 5 (08.04.2021) implemented resize and repair partitions after switching to rw mode. Implemented build script for building resize2fs, e2fsck, simg2ims, img2simg, lptools
- version 6 (08.04.2021) better loop device detection and setup
- version 7 (08.04.2021) fix selinux status detection
- version 8 (15.04.2021) fix loop device setup in superrepack
- version 9 (16.04.2021) make losetup android compatible
- version 10 - not released
- version 11 (01.05.2021) simplified, removed needs for parameter rw, implemented dm-verity disabler
- version 12 (05.05.2021) make old logs always deleted before fresh log is created, this prevent concentation with old logs
- version 13 (06.05.2021) make linux version so you should do the things on your super partition dump in linux machine
- version 14 (07.05.2021) fix compilation mess between linux and android
- version 15 (08.05.2021) use libselinux to determine and set selinux to permissive mode instead of popening getenforce-setenforce tools
Credits:
- me and me
Source code:
- source code -> https://github.com/munjeni/super_image_dumper
munjeni said:
hardcoded, no external libs, no android libs.
- my source code, later
Click to expand...
Click to collapse
Yet Another great tool!! :highfive:
Thanks!
New version is out, v11, it now detect file format, partition size correction in case ext4, partition group detection, have extraction progress bar, and it looks like:
Code:
---------------------------------------------------------
Super image dumper v_11 (by expert :) munjeni @ xda 2020)
---------------------------------------------------------
LpMetadataGeometry magic = 0x616c4467
LpMetadataGeometry struct size = 0x34
LpMetadataGeometry sha256 = 12FF55F0ABA7B506F25CB5DA5DCA09344234E8DF1D9C93AE82A499D98019467E
LpMetadataGeometry metadata_max_size = 0x10000
LpMetadataGeometry metadata_slot_count = 0x3
LpMetadataGeometry logical_block_size = 0x1000
LpMetadataHeader magic = 0x414c5030
LpMetadataHeader major_version = 10
LpMetadataHeader minor_version = 0
LpMetadataHeader header_size = 0x80
LpMetadataHeader header sha256 = CCF4F5D07842AAAE7C1B87F0E025512CF7AEA426D477B1E5175DA3D74F9B1C8C
LpMetadataHeader tables_size = 0x2e8
LpMetadataHeader tables sha256 = 52578668F89D8BCDA1BD1F748F2F69ED874C10A7062C85EF9970EE05D90161B1
LpMetadataHeader partitions offset = 0x0
LpMetadataHeader partitions num_entries = 0x8
LpMetadataHeader partitions entry_size = 0x34
LpMetadataHeader extents offset = 0x1a0
LpMetadataHeader extents num_entries = 0x5
LpMetadataHeader extents entry_size = 0x18
LpMetadataHeader groups offset = 0x218
LpMetadataHeader groups num_entries = 0x3
LpMetadataHeader groups entry_size = 0x30
LpMetadataHeader block_devices offset = 0x2a8
LpMetadataHeader block_devices num_entries = 0x1
LpMetadataHeader block_devices entry_size = 0x40
Partitions = 5 used, 3 not used, total 8
partition_1_name = system_a
attributes = 0x1
first_extent_index = 0x0
num_extents = 0x1
group_index = 0x1
partition_group = somc_dynamic_partitions_a
extent num_sectors = 0x336390 (0x66c72000 bytes total)
extent target_type = 0x0
extent target_data = 0x800 (dumping offset = 0x100000)
extent target_source = 0x0
Filetype EXT4. EXT4 size = 0x6526c000
Dumping system_a.ext4 ...
....................................................
....................................................
....................................................
..............................................
partition_2_name = system_b
attributes = 0x1
first_extent_index = 0x1
num_extents = 0x1
group_index = 0x2
partition_group = somc_dynamic_partitions_b
extent num_sectors = 0xab178 (0x1562f000 bytes total)
extent target_type = 0x0
extent target_data = 0x337000 (dumping offset = 0x66e00000)
extent target_source = 0x0
Filetype EXT4. EXT4 size = 0x150b3000
Dumping system_b.ext4 ...
..........................................
partition_3_name = product_a
attributes = 0x1
first_extent_index = 0x2
num_extents = 0x1
group_index = 0x1
partition_group = somc_dynamic_partitions_a
extent num_sectors = 0x2b62b8 (0x56c57000 bytes total)
extent target_type = 0x0
extent target_data = 0x3e2800 (dumping offset = 0x7c500000)
extent target_source = 0x0
Filetype EXT4. EXT4 size = 0x5565b000
Dumping product_a.ext4 ...
....................................................
....................................................
....................................................
..............
partition_4_name = product_b (not unused)
attributes = 0x1
first_extent_index = 0x3
num_extents = 0x0
group_index = 0x2
partition_group = somc_dynamic_partitions_b
extent num_sectors = NULL
extent target_type = NULL
extent target_data = NULL
extent target_source = NULL
Skipping dump.
partition_5_name = vendor_a
attributes = 0x1
first_extent_index = 0x3
num_extents = 0x1
group_index = 0x1
partition_group = somc_dynamic_partitions_a
extent num_sectors = 0x186d58 (0x30dab000 bytes total)
extent target_type = 0x0
extent target_data = 0x699000 (dumping offset = 0xd3200000)
extent target_source = 0x0
Filetype EXT4. EXT4 size = 0x30141000
Dumping vendor_a.ext4 ...
....................................................
............................................
partition_6_name = vendor_b (not unused)
attributes = 0x1
first_extent_index = 0x4
num_extents = 0x0
group_index = 0x2
partition_group = somc_dynamic_partitions_b
extent num_sectors = NULL
extent target_type = NULL
extent target_data = NULL
extent target_source = NULL
Skipping dump.
partition_7_name = odm_a
attributes = 0x1
first_extent_index = 0x4
num_extents = 0x1
group_index = 0x1
partition_group = somc_dynamic_partitions_a
extent num_sectors = 0xa60 (0x14c000 bytes total)
extent target_type = 0x0
extent target_data = 0x820000 (dumping offset = 0x104000000)
extent target_source = 0x0
Filetype EXT4. EXT4 size = 0x132000
Dumping odm_a.ext4 ...
partition_8_name = odm_b (not unused)
attributes = 0x1
first_extent_index = 0x5
num_extents = 0x0
group_index = 0x2
partition_group = somc_dynamic_partitions_b
extent num_sectors = NULL
extent target_type = NULL
extent target_data = NULL
extent target_source = NULL
Skipping dump.
Press any key to continue . . .
If you need to mount ext4 partition on Linux you need to mount partition RO or it will not mount!
I'm not one of those who make paid software and promote on xda, my work is always free. Even I'm always providing source code for free, source code of this tool is here -> https://github.com/munjeni/super_image_dumper , enjoy!
munjeni said:
I'm not one of those who make paid software and promote on xda, my work is always free. Even I'm always providing source code for free, source code of this tool is here -> https://github.com/munjeni/super_image_dumper , enjoy!
Click to expand...
Click to collapse
You are the best and I and many others thank you for make life so easy for us. :highfive:
PS: I will send you a PM with something you may find interesting and useful. :fingers-crossed:
Can I make this tool for remake oem.sin?
I have fw Xperia docomo bundling,and many bloatware,so I wont to remake oem.sin and deleted any bloatware, can I?
??sorry my English por????
paijoe88 said:
Can I make this tool for remake oem.sin?
I have fw Xperia docomo bundling,and many bloatware,so I wont to remake oem.sin and deleted any bloatware, can I?
?sorry my English por??
Click to expand...
Click to collapse
No you can't. Tool is unpacker only, it extracts oem from super image but doesn't reconstruct it back to super image. Even there is no known tool to make valid sin back because of signature.
Seems readonly partitions like oem, system, vendor...etc can be set to read-write mode via LP_PARTITION_ATTR overiding LpMetadataPartition.attributes 1 with 0. Anybody tried android version of this tool? E.g. dump block device e.g. /dev/block/bootdevice/super
RO flags (0x1), when it is set to 0x0 its RW mode, but header sha256 checksums of the LpMetadataGeometry and LpMetadataHeader also need to be modified in case we overvrite flags with RW mode!
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
000003080 73 79 73 74 65 6D 5F 61 00 00 00 00 00 00 00 00 system_a........
000003090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0000030A0 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 ................
0000030B0 01 00 00 00 73 79 73 74 65 6D 5F 62 00 00 00 00 ....system_b....
0000030C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0000030D0 00 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 ................
0000030E0 01 00 00 00 02 00 00 00 70 72 6F 64 75 63 74 5F ........product_
0000030F0 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a...............
000003100 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
000003110 02 00 00 00 01 00 00 00 01 00 00 00 70 72 6F 64 ............prod
000003120 75 63 74 5F 62 00 00 00 00 00 00 00 00 00 00 00 uct_b...........
000003130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000003140 01 00 00 00 03 00 00 00 00 00 00 00 02 00 00 00 ................
Click to expand...
Click to collapse
Removed fake pie binary and made new true pie 64 bit binary for android. With it you can dump your super partition on your own phone e.g.
Code:
adb push superunpack.arm64_pie /data/local/tmp
adb shell
su
cd /data/local/tmp
chmod 755 superunpack.arm64_pie
./superunpack.arm64_pie /dev/block/bootdevice/by-name/super
and whola all ext4 partitonins from your super partition is extracted to the /data/local/tmp folder.
bump. Any news regarding forcing system to be read/write in Android 10 ?
Thanks for your hard work @munjeni
See post 8.
In most case if you change those byte vbmeta protection will do a bootloop, so I think no way to set it to force rw except if you unlock bootloader and disable vbmeta protection trought fastboot (more info -> https://forum.xda-developers.com/t/tool-newflasher-xperia-command-line-flasher.3619426/post-84509179)
munjeni said:
Seems readonly partitions like oem, system, vendor...etc can be set to read-write mode via LP_PARTITION_ATTR overiding LpMetadataPartition.attributes 1 with 0. Anybody tried android version of this tool? E.g. dump block device e.g. /dev/block/bootdevice/super
RO flags (0x1), when it is set to 0x0 its RW mode, but header sha256 checksums of the LpMetadataGeometry and LpMetadataHeader also need to be modified in case we overvrite flags with RW mode!
Click to expand...
Click to collapse
I replaced all 4 occurences of these three bytes in my super.img like you suggested and flashed it to my phone.
It didn't boot probably due to the invalid sha256 checksums as you predicted.
Which bytes I must hex edit in the extracted system.ext4 in order to make it rw? I was thinking maybe the shared_blocks feature?
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
You should read this to get idea -> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout
TO be honest I never modified RO partitions. it can be probably done using parted, fsck and similar tools so you no need to do it by hexediting, try google a bit for idea I realy don't know
TO be honest I never modified RO partitions. it can be probably done using parted, fsck and similar tools so you no need to do it by hexediting, try google a bit for idea I realy don't know
Click to expand...
Click to collapse
@munjeni thank you so much for writing this amazing superunpack tool!
Thanks to your program and thanks to another tutorial I was able to create an automated bash script that transforms all read only partitions inside super.img (system, vendor , product, etc...) into read write-able partitions again and then flash to device as a brand new super.img.
HERE is the link to new universal version
munjeni said:
Disclaimer:
Superunpack tool was made for testing and educational purposes, ME is not responsible for what you do on/with your device using superunpack, you must agree that you using superunpack on your own risk, I am not responsible for anything else!
How to use:
- First step, unpack super.sin using my tool or use @IgorEisberg unsin tool
- Step two, on windows just drag and drop unpacked super image onto our exe to start extraction. Also you can use it from command line, from script or from etc. On Linux use it from command line. No need to set slot like it was a case on lpunpack, our tool will auto extract all slot images for you, enjoy!
Platform:
- Superunpack is working on Windows, Linux, Android, Darwin11, just chose right superunpack binary.
Changelog:
- version 10 (21.Jun.2020), initial version
- version 11 (22.Jun.2020), dump file format detection, partition size correction in case ext4, partition group detection, have extraction progress bar, improvements
Credits:
- not right now
Source code:
- tool is based on just this header, the rest of things is myself implemented - hardcoded, no external libs, no android libs.
- my source code -> https://github.com/munjeni/super_image_dumper
Click to expand...
Click to collapse
Wow, this is pretty impressive my friend. Big props!! Just curious, what's the difference with "this" utility(superunpack) and the standard android lpunpack?
bynarie said:
Wow, this is pretty impressive my friend. Big props!! Just curious, what's the difference with "this" utility(superunpack) and the standard android lpunpack?
Click to expand...
Click to collapse
Diferencie is that my tool not use any external dependencies so can be build easily as a static binary. Seccond diferencie is its easy to use, just drag and drop file to extract. I'm not tried lpunpack so I can't tell the exact diferencie, you tell me
munjeni said:
Diferencie is that my tool not use any external dependencies so can be build easily as a static binary. Seccond diferencie is its easy to use, just drag and drop file to extract. I'm not tried lpunpack so I can't tell the exact diferencie, you tell me
Click to expand...
Click to collapse
lpunpack:
works with sparse files
creates .img files
gives zero text output so you don't know what's happening under the hood
extracts all partitions even the empty ones
Superunpack:
works with raw files
creates .ext4 files
gives lots of text info so you know exactly what's going on
only extracts the partitions that are not empty
These are the main differences that I noticed.
Hey friend I get this error can you help me out?
[Novice alert]
lebigmac said:
lpunpack:
works with sparse files
creates .img files
gives zero text output so you don't know what's happening under the hood
extracts all partitions even the empty ones
Superunpack:
works with raw files
creates .ext4 files
gives lots of text info so you know exactly what's going on
only extracts the partitions that are not empty
These are the main differences that I noticed.
Click to expand...
Click to collapse
Nice !
munjeni said:
Diferencie is that my tool not use any external dependencies so can be build easily as a static binary. Seccond diferencie is its easy to use, just drag and drop file to extract. I'm not tried lpunpack so I can't tell the exact diferencie, you tell me
Click to expand...
Click to collapse
Good stuff, thank you!

How to find "hw_soc_version" for a QCom SOC?

I have an Android device with a QComm SDM680 SOC. The QCom part# of the SOC is SM6225.
How do I find the "hw_soc_version" and "soc_version" of the SDM680/SM6225 ?
I've found some general scripts that collate this type of info, like this one. But the SDM680 is not in any of those lists.
I've searched on the rooted device, grepped the kernel logs and the kernel opensource. fastboot getvar all doesn't expose this info either.
Does anybody know how to find these values?
Oh, that's easy. You just run an EDL client, they always ask the HWID.
You don't even need to have a loader for it.
On my EDL client just:
Code:
C:\>edl /l
Found EDL 9008
Serial: 12345678
HWID: 000cc0e100000000, QC: 000cc0e1, OEM: 0000, Model: 0000
Hash: 7be49b72f9e43372-23ccb84d6eccca4e-61ce16e3602ac200-8cb18b75babe6d09
You can also attach a UART while booting.
Code:
Format: Log Type - Time(microsec) - Message - Optional Info
Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.4-00246-S660LZB-1
S - IMAGE_VARIANT_STRING=Sdm660LA
S - OEM_IMAGE_VERSION_STRING=cibuild
S - Boot Interface: Unknown
S - Secure Boot: Off
S - Boot Config @ 0x00786070 = 0x000001c1
S - JTAG ID @ 0x00786130 = 0x000cc0e1
Note that even though this is a SDM636 the log speaks of 660, but the "JTAG ID" is the correct HWID.
Usually the certs in xbl/abl has the HW_ID in it.
Also:
Code:
Teletex string 11 3007 0000 0000 0000 0000 0000 0000 0000 0000 0000 SOC_VERS
(I've never run into this soc_version before.)
Also, AFAIK, your friendly Firehose loader repository doesn't have a loader for this.
Edit: Oh, you're not looking for the HWID?
Renate said:
Oh, that's easy. You just run an EDL client, they always ask the HWID.
You don't even need to have a loader for it.
On my EDL client just:
Code:
C:\>edl /l
Found EDL 9008
Serial: 12345678
HWID: 000cc0e100000000, QC: 000cc0e1, OEM: 0000, Model: 0000
Hash: 7be49b72f9e43372-23ccb84d6eccca4e-61ce16e3602ac200-8cb18b75babe6d09
...
Edit: Oh, you're not looking for the HWID?
Click to expand...
Click to collapse
Thanks for the tip. I checked the SAHARA output. It seems that this HWID consists of the MSM_ID+OEM+MODEL. For the SDM680 I got: HW_ID: 0x001b80e100000000 (MSM_ID=0x001b80e1 OEM_ID=0x0000 MODEL_ID=0x0000).
Looking at bkerler's qualcomm_config.py, it seems that the hw_soc_version and hwid are two different things. For example for the SDM660, the msmid entry is 0x08C0E1, with a comment that the soc_hw_version is different:
Code:
0x08C0E1: "SDM660", # 0x30060000 soc_hw_version
Renate said:
Usually the certs in xbl/abl has the HW_ID in it.
Click to expand...
Click to collapse
Even though it's about the hwid, I looked into this too. It seems that around 2016, the HWID was stored in OU fields in the certificiates in the XBL file (see pages 10-11). But after 2019, it is now stored in the metadata of the MBN image (see page 9) within the XBL file. I only mention it because I thought it might prove useful for you.
Curiously, the HWID wasn't in the certs or metadata in my stock ROM's xbl.elf. Strange.
Yahoo Mike said:
For the SDM680 I got: HW_ID: 0x001b80e100000000...
Click to expand...
Click to collapse
The good news for you is that it's not stamped OEM/model.
There's some chance that this is not SecureBoot.
Which means that any loader that's compatible with your SoC will work.
What does this say: fastboot getvar secure
What does this say: cat /proc/cpuinfo (Just the name line.)
You can also look in the DTB, either decoded or raw, it's at the beginning.
Then there's the other wrinkle that Qualcomm has SDM numbers, MSM numbers and code names for SoCs.
Maybe that cpuinfo will tell you a codename.
Renate said:
The good news for you is that it's not stamped OEM/model.
There's some chance that this is not SecureBoot.
Which means that any loader that's compatible with your SoC will work.
What does this say: fastboot getvar secure
Click to expand...
Click to collapse
I think SecureBoot is on. I've had to do a test-points recovery a few times - after I tried to run with a patched (and incorrectly signed) ABL.
In fastbootd & bootloader menus, it says SecureBoot is on. And (as you suggested) fastboot utility agrees:
Code:
C:\>fastboot getvar secure
secure: yes
Finished. Total time: 0.001s
Renate said:
What does this say: cat /proc/cpuinfo (Just the name line.)
You can also look in the DTB, either decoded or raw, it's at the beginning.
Then there's the other wrinkle that Qualcomm has SDM numbers, MSM numbers and code names for SoCs.
Maybe that cpuinfo will tell you a codename.
Click to expand...
Click to collapse
The codename is khaje.
Code:
TB128FU:/ # cat /proc/cpuinfo
Processor : AArch64 Processor rev 4 (aarch64)
...<info about 8 processors>...
Hardware : Qualcomm Technologies, Inc KHAJE
That agrees with the run-time /sys/devices/soc0/soc_id value of 518, which is "khaje" according to the stock ROM's /vendor/bin/init.qti.display_boot.sh and /vendor/bin/init.qcom.post_boot.sh.
Curiously, at the beginning of the DTB it says it's "Bengal":
Code:
00 00 00 03 00 00 00 33 00 00 00 00 51 75 61 6C .......3....Qual
63 6F 6D 6D 20 54 65 63 68 6E 6F 6C 6F 67 69 65 comm Technologie
73 2C 20 49 6E 63 2E 20 42 65 6E 67 61 6C 20 31 s, Inc. Bengal 1
47 62 20 44 44 52 20 48 44 2B 20 53 6F 43 00 00 Gb DDR HD+ SoC..
But at offset 0x2A62D0 it changes its name:
Code:
00 00 00 00 00 03 00 00 00 26 00 00 00 00 51 75 .........&....Qu
61 6C 63 6F 6D 6D 20 54 65 63 68 6E 6F 6C 6F 67 alcomm Technolog
69 65 73 2C 20 49 6E 63 2E 20 4B 68 61 6A 65 20 ies, Inc. Khaje
53 6F 43 00 00 00 00 00 00 03 00 00 00 0B 00 00 SoC.............
I can't believe how many different numbers/strings QCom has to describe a SoC: soc_id, codename, hwid, msm_id ... and the ever-elusive hw_soc_version.
Anyway, I'll load up this SoC's firehose program to bkerler's edl. I'll slip in a question about how to query the hw_soc_version. I'll post back any reply.
Yahoo Mike said:
The codename is khaje.
Click to expand...
Click to collapse
Khajeh is a city in Iran: https://en.wikipedia.org/wiki/Khajeh,_Iran
Yahoo Mike said:
Curiously, at the beginning of the DTB it says...
Click to expand...
Click to collapse
That's because you are probably looking at multiple DTBs.
You can simply grep/scan for "Qualcomm Technologies".
I don't know why they do that.
The abl scans through them and find the one that best matches.
S/N: 0x7BD1BDD5
HW ID: 0x001B80E10015006D -> HUAWEI
HASH: 0xB25DECD85D217F5D9B53DC3C42EF7846DCEF59DD3E0AF4D12606199F5099FF23D73C3AFFBE5EFBF421A81A197E41FDF5
PBL : 0x00000000
HASH TYPE: SHA384
DEV HASH: 0x0000003AC0D4
CPU : Undefined CPU: 001B80E10015006D

Categories

Resources