Whenever there's an over-the-air (OTA) update sent-out or made available (either leaked or official), there's always a large number of users that end-up reporting that their installations have failed with the infamous and dreaded "E:Error in /sdcard/xxx.zip (Status 7) installation aborted." error message. So, I decided to try to write an app that would help folks figure-out what the issues might be so that they can get their OTA update installed.
The OTA Verifier app can be used to evaluate an over-the-air (OTA) update.zip or other flashable .zip file before you attempt the install or afterwards to help you figure-out why the installation may have failed. OTA Verifier will point-out what files and conditions are being tested so that you can more easily attempt to correct these issues.
Play Market Link (free!): OTA Verifier (beta)
Do I have to be rooted to use this?
No, but rooted devices will have more conditions that the app will be able to test/evalute because it will have access to protected files that non-rooted devices don't.
Will the app actually do the installation or change anything on my device?
No. The app will only evaluate the expressions and conditions contained inside the updater-script file inside the .zip file that's trying to be installed. None of the functions or commands that try to modify your device (i.e., delete/patch/format/extract, etc.) will be evaluated or executed. The app basically operates in "read-only" mode with respect to the .zip file being evaluated.
Will this app work on older devices or flashable .zip files that use the amend update-script files?
No, this app evaluates and interprets the newer edify updater-script files (notice the "r" in "updater").
How do I use this app?
Install and launch the app
Click the "Select File" button and navigate to desired .zip file
Long-press (press and hold) the file entry for the desired .zip file
Click the "Verify OTA .zip" file pop-up
Wait for the app to process the file (should take under a minute, depending on your device)
View the displayed results (text will also be copied to the clipboard)
Miscellaneous info:
1. Savvy root users probably already know that after a failed .zip file installation, you should be able to view the /cache/recovery/log file to view the information about what might have failed. OTA Verifier will try to uncover all of the issues and not just the first one that causes the installation to fail.
2. While I wrote this app principally to evaluate OTA update.zip files, the edify updater-script files are used and written by ROM devs and others who created flashable .zip files. This app can be used to evaluate those file's updater-script files, too.
How does this all work?
Here's the basic outline/structure of what the app does:
1. the .zip file is selected by the user via the file selector
2. list of frozen system apps are identified for later reporting
3. the updater-script file is extracted from the META-INF/com/google/android folder in the .zip file
4. this updater-script file is parsed and converted into reverse polish notation (RPN) for execution
5. edify functions that might modify your device are NOT evaluated; these include apply_patch, delete, delete_recursive, format, mount, package_extract_dir/file, run_program, set_perm[_recursive], symlink, unmount, write_raw_image, etc.
6. note: the update-binary executable is not used by or involved in this app; the edify script interpreter that the app uses was written by me, in Java, specifically for this app
7. the remaining edify script functions that test conditions (such as apply_patch_check, apply_patch_space, assert, concat, file_exists, file_getprop, getprop, greater_than_int, if-then-else-endif, ifelse, is_substring, less_than_int, read_file, ui_print, and various operators (!, !=, &&, (, ), ;, ||, +, ==)) are evaluated using an operand/operator stack from the RPN expression parsed from the updater-script statements
8. expressions that fail (return a null-string) or are bypassed (usually because a resource (file/partition) is protected/secured against read-access) are reported for the user
9. after the entire script has been processed, the results are displayed in a pop-up window on the device and the text of those results are copied to the clipboard
Planned / future features:
- preferences / settings
- logging of the output to a file on the /sdcard
- test if .zip file is signed or not
- display more stats
- "explanation mode" to interpret, in English, what the edify commands are testing
Successfully tested on:
Samsung Galaxy Nexus (CDMA) 4.1.1
Samsung Galaxy Nexus (GSM) 4.1.2
Asus Nexus 7 tablet 4.1.2
Motorola Droid X 2.3.4
The following edify commands/functions/operators are supported:
Code:
[COLOR="blue"]apply_patch_check[/COLOR] // apply_patch_check ( <filepath>, <sha1-checksum> [ , <sha1-checksum> ... ] )
// apply_patch_check ( <compoundvalue> )
<compoundvalue> := [ EMMC : <filepath> : <sha1-size> : <sha1-checksum> [ <sha1-size> : <sha1-checksum> ... ]
[ MTD : <dev-block> : <sha1-size> : <sha1-checksum> [ <sha1-size> : <sha1-checksum> ... ]
[ vfat : <dev-block> : <sha1-size> : <sha1-checksum> [ <sha1-size> : <sha1-checksum> ... ]
<dev-block> := filename in /dev/block
[COLOR="Blue"]apply_patch_space[/COLOR] // apply_patch_space ( <bytes> )
[COLOR="blue"]assert[/COLOR] // assert ( <expression> )
[COLOR="blue"]concat[/COLOR] // conact ( <string> , <string> )
[COLOR="blue"]file_exists[/COLOR] // file_exists ( <filepath> )
[COLOR="blue"]file_getprop[/COLOR] // file_getprop ( <filepath>, <propertyname> )
[COLOR="blue"]getprop[/COLOR] // getprop ( <propertyname> )
[COLOR="blue"]greater_than_int[/COLOR] // greater_than_int ( <integer>, <integer> )
[COLOR="blue"]if then else endif[/COLOR] // if <condition> then <expression> [ else <expression> ] endif
[COLOR="blue"]ifelse[/COLOR] // ifelse ( <condition>, <then-expression> [ <else-expression> ] )
[COLOR="blue"]is_substring[/COLOR] // is_substring ( <string> , <string> )
[COLOR="blue"]less_than_int[/COLOR] // less_than_int ( <integer>, <integer> )
[COLOR="blue"]read_file[/COLOR] // read_file ( <filepath> )
[COLOR="blue"]ui_print[/COLOR] // ui_print ( <string> ) parsed & executed, but output not displayed
[COLOR="blue"]![/COLOR] // logical NOT operator
[COLOR="blue"]!=[/COLOR] // not equals operator
[COLOR="blue"]&&[/COLOR] // logical AND operator
[COLOR="blue"]([/COLOR] // open paren: precedence / grouping
[COLOR="blue"])[/COLOR] // close paren: precedence / grouping
[COLOR="blue"];[/COLOR] // sequence/imperative (terminates & separates statements)
[COLOR="blue"]||[/COLOR] // logical OR operator
[COLOR="blue"]+[/COLOR] // string concatenation operator
[COLOR="blue"]==[/COLOR] // equals operator
[COLOR="blue"]#[/COLOR] // comment line
The following edify functions are NOT supported (the majority of them because they are intended to modify your device):
Code:
[COLOR="blue"]apply_patch[/COLOR]
[COLOR="blue"]delete[/COLOR]
[COLOR="blue"]delete_recursive[/COLOR]
[COLOR="blue"]format[/COLOR]
[COLOR="blue"]mount[/COLOR]
[COLOR="blue"]package_extract_dir[/COLOR]
[COLOR="blue"]package_extract_file[/COLOR]
[COLOR="blue"]run_program[/COLOR]
[COLOR="blue"]set_perm[/COLOR]
[COLOR="blue"]set_perm_recursive[/COLOR]
[COLOR="blue"]symlink[/COLOR]
[COLOR="blue"]unmount[/COLOR]
[COLOR="blue"]write_raw_image[/COLOR]
[COLOR="blue"]is_mounted[/COLOR] // innocuous, but still not supported
[COLOR="blue"]stdout[/COLOR] // innocuous, but still not supported
[COLOR="blue"]show_progress[/COLOR] // innocuous, but still not supported
[COLOR="blue"]set_progress[/COLOR] // innocuous, but still not supported
Keywords:
OTA, over-the-air, updater-script, edify, amend
Screenshots:
updated 9-May-2014
updated to version 1.3
Updated app with root-enabled file browser for navigating to protected/secured directories like /cache to select OTA's update zip file.
Full change log to date:
[v1.3 - 04-Nov-2012]:
- for rooted devices, change file browser to allow navigating and selecting .zip files in protected directories (like /cache)
[v1.2 - 29-Oct-2012]:
- minor fix to handle two reports of null pointer exceptions
[v1.1 - 28-Oct-2012]:
- remove unused SD card write permission
[v1.0 - 28-Oct-2012]:
- initial Play Market release
This is pretty cool.
Any chance you could direct me to an explanation for sha1-size (for apply_patch_check) is though? I'm working on some edify scripting stuff and trying to incorporate some asserts for a specific bootloader (important to what I'm doing), and this is honestly the best reference I've found for it yet.
osm0sis said:
This is pretty cool.
Any chance you could direct me to an explanation for sha1-size (for apply_patch_check) is though? I'm working on some edify scripting stuff and trying to incorporate some asserts for a specific bootloader (important to what I'm doing), and this is honestly the best reference I've found for it yet.
Click to expand...
Click to collapse
You bet!
I'm heading home from work here soon and I'll dig-up the details for you .
(yeah, I scoured the internet for detailed information on Edify scripting as well as reviewed the actual Google/Android code--there's not a complete compendium, unfortunately).
I'll ping you back after a bit...
Cheers!
osm0sis said:
Any chance you could direct me to an explanation for sha1-size (for apply_patch_check) is though?
Click to expand...
Click to collapse
Okay, here's what I know about the apply_patch_check command:
There are basically two formats:
Code:
apply_patch_check // apply_patch_check ( <filepath>, <sha1-checksum> [ , <sha1-checksum> ... ] )
// apply_patch_check ( <compoundvalue> )
<compoundvalue> := [ EMMC : <filepath> : <sha1-size> : <sha1-checksum> [ <sha1-size> : <sha1-checksum> ... ]
[ MTD : <dev-block> : <sha1-size> : <sha1-checksum> [ <sha1-size> : <sha1-checksum> ... ]
[ vfat : <dev-block> : <sha1-size> : <sha1-checksum> [ <sha1-size> : <sha1-checksum> ... ]
The first of which is what I think you typically see in updater-script files where you have a filename, followed by one or more SHA1 digests. Here's an example:
assert(apply_patch_check("/system/app/ApplicationsProvider.apk", "41bb5aaaa2791e68b55622fcca13f0e4efa757b2", "fc845332ae7f706824de73f64ae47f93866ad245"));
The second format is what I call a compound value format, where the file or partition to be checked and the SHA1 digests to be compared, are all concatenated together in a single, colon-separated value. For example:
assert(apply_patch_check("EMMC:/dev/block/platform/omap/omap_hsmmc.0/by-name/boot:4247552:3f4d4f9549d307d152f8503983ee4ff5f46b0a43:4470784:fbd13c778b34fdb7917c1ccee6389aa9b13a92bd"));
In the above, I've colored the sizes in red of file/partition on which to compute the SHA1 checksum (colored in black). I figured this out and verified this by using dd to copy the portions of the partition in question and calculating the SHA1 checksums on it.
My app only supports parsing and evaluating two sets of lengths/SHA1s for the compound format at this time (that's all I've encountered so far in the scripts that I've viewed).
Does that help and/or answer your questions?
Cheers!
Thanks! Yeah that helps a lot, but unfortunately it doesn't seem to solve the problem I've been seeing..
Using dd to grab the partition img I can sha1sum that and that SHA1 doesn't allow it to pass the assert, or even a direct sha1_check of the partition.
Code:
run_program("/sbin/busybox", "dd", "if=/dev/block/platform/sdhci-tegra.3/by-name/USP"), "of=/tmp/bootloader.img")
ui_print(sha1_check(read_file("/tmp/bootloader.img"))); = 8c206a1a87599f532ce68675536f0b1546900d7a (also, bootloader.img is 2097152 in size)
ui_print(sha1_check(read_file("/dev/block/platform/sdhci-tegra.3/by-name/USP"))); = da39a3ee5e6b4b0d3255bfef95601890afd80709
assert(apply_patch_check("EMMC:/dev/block/platform/sdhci-tegra.3/by-name/USP:2097152:8c206a1a87599f532ce68675536f0b1546900d7a")); = Fail
They just don't match and I'm completely baffled by it. I can work around it by using dd to pull the img first and using that for my asserts but it's not as clean as I'd like it.
osm0sis said:
Yeah that helps but I guess unfortunately it doesn't entirely solve the problem I've been seeing.
Using dd to grab the partition img I can sha1sum that and that SHA1 doesn't allow it to pass the assert, or even a direct sha1_check of the partition.
They just don't match and I'm completely baffled by it. I can work around it by using dd to pull the img and using that for my asserts but it's not as clean as I'd like it.
Click to expand...
Click to collapse
Is it for the Gnex or the N7 (I have those same devices)?
I'd be happy to test something for you...just PM me or send me an email ([email protected]).
I don't know if you're using my app to test with, but you can also manually run the update-binary directly on/from your phone (i.e., you don't have to run it in recovery, but you obviously need to be careful what your updater-script does ).
I'm still looking for the exact syntax in my notes, but I'll edit this post when I find it...brb.
~ ~ ~
edit: http://wiki.opticaldelusion.org/wiki/Update-binary shows this:
update-binary <api> <pipefd> <zip>
Click to expand...
Click to collapse
I know I did this during my early testing, but I can't remember the exact syntax...I'll try it and re-re-edit .
Man thanks so much, PM sent.
Sorry for cluttering up your thread with this semi-OT stuff.
osm0sis said:
Man thanks so much, PM sent.
Sorry for cluttering up your thread with this semi-OT stuff.
Click to expand...
Click to collapse
LOL, not a problem...it's all Edify-related and updater-script related...I'm betting that's what'll lead most folks here.
Happy to help .
==================
Detailed Analysis
==================
- you have no frozen system apps (good!)
- there were 2 failed expressions; see below for the details about these tests that will cause your OTA install to fail
- there were no bypassed expressions (good!)
-------------------------------
statistics:
frozen system apps: 0
success count: 526
fail count: 2
ignore count: 1587
partition count: 2
protected count: 2
bypassed count: 0
-------------------------------
updater-script analysis details:
FAILED: line #268:
assert(apply_patch_check("/system/app/XT9IME.apk", "8aba56a4406128e78f5729753252c3d93bc21cb4", "965b437bce65018eeb31ff9a381c3687542099e0"));
----------------------
FAILED: line #1038:
assert(apply_patch_check("EMMC:/dev/block/platform/sdhci-tegra.3/by-name/LNX:5013504:c48f8e86c73fb2c2ba1794f5ec98e27c9e206ed5:5060608:319331fae14fec8a88063751475fce26bae328e0"));
So a question, could this failure above be causing my 32 GB nexus 7 to have system update issues? (Not necessarily XT9IME.apk (which I shouldn't have deleted), more the other one) Is there a fix?
Sent from my Nexus 7 using Tapatalk HD
---------- Post added at 10:44 AM ---------- Previous post was at 10:39 AM ----------
==================
Detailed Analysis
==================
- you have no frozen system apps (good!)
- there were 2 failed expressions; see below for the details about these tests that will cause your OTA install to fail
- there were no bypassed expressions (good!)
-------------------------------
statistics:
frozen system apps: 0
success count: 526
fail count: 2
ignore count: 1587
partition count: 2
protected count: 2
bypassed count: 0
-------------------------------
updater-script analysis details:
FAILED: line #268:
assert(apply_patch_check("/system/app/XT9IME.apk", "8aba56a4406128e78f5729753252c3d93bc21cb4", "965b437bce65018eeb31ff9a381c3687542099e0"));
----------------------
FAILED: line #1038:
assert(apply_patch_check("EMMC:/dev/block/platform/sdhci-tegra.3/by-name/LNX:5013504:c48f8e86c73fb2c2ba1794f5ec98e27c9e206ed5:5060608:319331fae14fec8a88063751475fce26bae328e0"));
Sent from my Nexus 7 using Tapatalk HD
Click to expand...
Click to collapse
So a question, could this failure above be causing my 32 GB nexus 7 to have system update issues? (Not necessarily XT9IME.apk (which I shouldn't have deleted), more the other one) Is there a fix?
modwilly said:
So a question, could this failure above be causing my 32 GB nexus 7 to have system update issues? (Not necessarily XT9IME.apk (which I shouldn't have deleted), more the other one) Is there a fix?
Click to expand...
Click to collapse
Looks like you've updated the boot (LNX) partition in addition to deleting (or renaming/moving) the XT9IMG.apk?
So yes, the OTA will not install until you've put both items back to their expected state.
The fix, of course, depends on exactly what you did to change them in the first place.
version 2.1 uploaded to Play Store
Major re-write of app done for version 2.0 and above.
Recent change log:
[version 2.1 - 6-May-2014]:
- fix issue w/identification of non-existent files
[version 2.0 - 5-May-2014]:
- major app update: edify parsing, RPN processing and execution engine completely re-written
- original core app behavior and functionality remain the same, but code cleanup and re-write should mean better and more robust handling of future OTA updater-script expressions
Click to expand...
Click to collapse
Thanks and let me know if you have any questions.
@scary alien could you please tell me how to get apply_patch_space bytes? i am making ota script and i got almost all, still need to generate:
apply_patch_space(bytes) || abort("Not enough free space on /cache to apply patches.");
Click to expand...
Click to collapse
i just dunno how
ZduneX25 said:
@scary alien could you please tell me how to get apply_patch_space bytes? i am making ota script and i got almost all, still need to generate:
i just dunno how
Click to expand...
Click to collapse
It's just the available space (in bytes) in the /cache partition that you want to make sure is available (i.e., free space) for any operations your updater-script file will do concerning /cache.
For example:
apply_patch_space(1000000) || abort("Not enough free space on /cache to apply patches.");
Click to expand...
Click to collapse
Would verify that there is at least 1MB (1,000,000) free bytes available in /cache.
Does that answer what you're asking?
Lemme know--happy to help if I can.
Cheers!
@scary alien not really, i mean i know how it works, just dunno how to generate proper size in updater, for example:
i create regular ota: multiple .p files and some images, zipped, signed OTA.zip size is 20mb, /patch size is 7 mb, images 10 mb and /system (new files) 3mb, how do i know how many bytes i should set to make this ota install in recovery?
should i summarize .p files size or target (patched apk, jar) files size (would be around 90mb) or what else?
I think the size would depend on when your patch files are cleaned-up...i.e., after each patching operation or at the end. If its at the end, you'll of course need to account to all of the space you might use in /cache.
I don't know of a good way to tell you what the high water mark would be other than testing and recording the output of a "df /cache" command at various points in your updater-script file.
I could do that however I don't see it universal or handy, each ota has different size so it would need more/less free space every time.
There is how Google gets this value : https://github.com/MiCode/patchrom_tools/blob/kitkat/releasetools/edify_generator.py#L131 maybe you will understand better.
ZduneX25 said:
I could do that however I don't see it universal or handy, each ota has different size so it would need more/less free space every time.
There is how Google gets this value : https://github.com/MiCode/patchrom_tools/blob/kitkat/releasetools/edify_generator.py#L131 maybe you will understand better.
Click to expand...
Click to collapse
I think I understood what you meant...there's no easy way to make this universal for each OTA except to evaluate each file that might need to use space resources from the /cache partition.
Even the edify-generator git that you referenced does this dynamically:
releasetools/ota_from_target_files:
Code:
for diff in diffs:
tf, sf, d = diff.GetPatch()
if d is None or len(d) > tf.size * OPTIONS.patch_threshold:
# patch is almost as big as the file; don't bother patching
tf.AddToZip(output_zip)
verbatim_targets.append((tf.name, tf.size))
else:
common.ZipWriteStr(output_zip, "patch/" + tf.name + ".p", d)
patch_list.append((tf.name, tf, sf, tf.size, common.sha1(d).hexdigest()))
[COLOR="Red"]largest_source_size[/COLOR] = max([COLOR="red"]largest_source_size[/COLOR], sf.size)
:
:
:
if patch_list or updating_recovery or updating_boot:
script.[COLOR="Blue"][B]CacheFreeSpaceCheck[/B][/COLOR]([COLOR="red"]largest_source_size[/COLOR])
I have seen it, but here comes largest_source_size:
Code:
largest_source_size = 0
https://github.com/MiCode/patchrom_tools/blob/kitkat/releasetools/ota_from_target_files#L532
ZduneX25 said:
I have seen it, but here comes largest_source_size:
Code:
largest_source_size = 0
https://github.com/MiCode/patchrom_tools/blob/kitkat/releasetools/ota_from_target_files#L532
Click to expand...
Click to collapse
Yeah, that's just the initialization of that variable...it gets updated in the for-loop that I included above.
(you know that's not my tool, software, right?)
Related
AnyKernel3 -- Flashable Zip Template for Kernel Releases with Ramdisk Modifications
"AnyKernel is a template for an update.zip that can apply any kernel to any ROM, regardless of ramdisk." - Koush
The concept of AnyKernel has been around for awhile, (originally by Koushik Dutta/ClockworkMod,) which allowed a device-specific kernel zImage to be flashed over device-specific ROM and use the ramdisk that came with the ROM to reduce the chance of any issues arising from the custom kernel pairing.
The drawback to this was that some kernels require modifications to the ramdisk to enable/set up kernel features, and in the old AnyKernel format there was no way to do this. Enter AnyKernel2.
AnyKernel2 pushed the format even further by allowing kernel developers to modify the underlying ramdisk for kernel feature support easily using a number of included command methods along with properties and variables to customize the installation experience to their kernel. AnyKernel3 adds the power of topjohnwu's magiskboot for wider format support by default.
A script based on Galaxy Nexus (tuna) is included for reference. An example of ramdisk-only changes can be seen in my GN Synapse Injector repo. For an example that also modifies ROM and properly injects init.d support using busybox run-parts and sepolicy-inject see CosmicDan's CosmicTweaks project. For a multi-partition example and an example of how to handle a device which only has a ramdisk when rooted see my N5X/6P BLOD Workaround Injector. Other working AK2/3 examples for more recent devices may be found on eng.stk's blu_spark device repos under Releases.
Please see the linked posts here for instructions on enabling full AVBv1 (Pixel), AVBv1, A/B slot and/or system-as-root (SAR) or 2-stage init (2SI) device support, and further guidelines for system-as-root/2-stage init (/system/system in recovery) modifications in general.
Please also see the post here for important notes about the current state of AOSP vendor_boot v4 support and AVBv2 flag options.
Magisk root is automatically detected and retained by patching the new Image.*-dtb as Magisk would!
My development work on my many projects comes out of my free time, so if you enjoy this project or anything else I've done on xda, please consider sponsoring my ongoing work using my GitHub Sponsors profile. For a one-time donation you can hit the donate link from my profile. Thank you for your support!
Source: https://github.com/osm0sis/AnyKernel3/
Download: https://github.com/osm0sis/AnyKernel3/archive/master.zip
Instructions
1) Place final kernel build product, e.g. Image.gz-dtb or zImage to name a couple, in the zip root (any separate dt, dtb, recovery_dtbo, dtbo and/or vendor_dlkm should also go here for devices that require custom ones, each will fallback to the original if not included)
2) Place any required ramdisk files in /ramdisk (/vendor_ramdisk for simple multi-partition vendor_boot v3 support) and module files in /modules (with the full path like /modules/system/lib/modules)
3) Place any required patch files (generally partial files which go with AK3 file editing commands) in /patch (/vendor_patch for simple multi-partition vendor_boot v3 support)
4) Modify the anykernel.sh to add your kernel's name, boot partition location, permissions for any added ramdisk files, and use methods for any required ramdisk modifications (optionally, also place banner and/or version files in the root to have these displayed during flash)
5) `zip -r9 UPDATE-AnyKernel3.zip * -x .git -x .github README.md *placeholder`
The LICENSE file must remain in the final zip to comply with licenses for binary redistribution and the license of the AK3 scripts.
If supporting a recovery that forces zip signature verification (like Cyanogen Recovery) then you will need to also sign your zip using the method I describe here:
[DEV][TEMPLATE] Complete Shell Script Flashable Zip Replacement + Signing [SCRIPT]
Not required, but any tweaks you can't hardcode into the source (best practice) should be added with an additional init.tweaks.rc or bootscript.sh to minimize the necessary ramdisk changes. On newer devices Magisk allows these within /overlay.d - see examples.
It is also extremely important to note that for the broadest AK3 compatibility it is always better to modify a ramdisk file rather than replace it.
If running into trouble when flashing an AK3 zip, the suffix -debugging may be added to the zip's filename to enable creation of a debug .tgz of /tmp for later examination while booted or on desktop.
Staying Up-To-Date
Now that you've got a ready zip for your device, you might be wondering how to keep it up-to-date with the latest AnyKernel commits. AnyKernel2 and AnyKernel3 have been painstakingly developed to allow you to just drop in the latest update-binary and tools directory and have everything "just work" for beginners not overly git or script savvy, but the best practice way is as follows:
1) Fork my AnyKernel3 repo on GitHub
2) `git clone https://github.com/<yourname>/AnyKernel3`
3) `git remote add upstream https://github.com/osm0sis/AnyKernel3`
4) `git checkout -b <devicename>`
5) Set it up like your <devicename> zip (i.e. remove any folders you don't use like ramdisk or patch, delete README.md, and add your anykernel.sh and optionally your Image.*-dtb if you want it up there) then commit all those changes
6) `git push --set-upstream origin <devicename>`
7) `git checkout master` then repeat steps 4-6 for any other devices you support
Then you should be able to `git pull upstream master` from your master branch and either merge or cherry-pick the new AK3 commits into your device branches as needed.
Enjoy!
Questions, comments and feedback welcome.
Credits & Thanks: All authors of the included binaries and the tools I used to port them over for their amazing work. koush for the original AnyKernel concept.
Disclaimer: Naturally, you take all the responsibility for what happens to your device when you start messing around with things.
Script Commands Reference
Everything to edit is self-contained in anykernel.sh. A quick-reference for the commands and properties included are as follows.
Properties / Variables
These are some values that will be read during the install process, allowing you to customize your installation, e.g. block= is a shell variable to specify the kernel/boot block partition that the dump_boot command method will copy and unpack.
Code:
kernel.string=KernelName by YourName @ xda-developers
do.devicecheck=1
do.modules=1
do.systemless=1
do.cleanup=1
do.cleanuponabort=0
device.name1=maguro
device.name2=toro
device.name3=toroplus
device.name4=tuna
supported.versions=6.0 - 7.1.2
supported.patchlevels=2019-07 -
block=/dev/block/platform/omap/omap_hsmmc.0/by-name/boot;
is_slot_device=0;
ramdisk_compression=auto;
patch_vbmeta_flag=auto;
do.devicecheck=1 specified requires at least device.name1 to be present. This should match ro.product.device, ro.build.product, ro.product.vendor.device or ro.vendor.product.device from the build.prop files for your device. There is support for as many device.name# properties as needed. You may remove any empty ones that aren't being used.
do.modules=1 will push the .ko contents of the modules directory to the same location relative to root (/) and apply correct permissions. On A/B devices this can only be done to the active slot.
do.systemless=1 (with do.modules=1) will instead push the full contents of the modules directory to create a simple "ak3-helper" Magisk module, allowing developers to effectively replace system files, including .ko files. If the current kernel is changed then the kernel helper module automatically removes itself to prevent conflicts.
do.cleanup=0 will keep the zip from removing its working directory in /tmp/anykernel (by default) - this can be useful if trying to debug in adb shell whether the patches worked correctly.
do.cleanuponabort=0 will keep the zip from removing its working directory in /tmp/anykernel (by default) in case of installation abort.
supported.versions= will match against ro.build.version.release from the current ROM's build.prop. It can be set to a list or range. As a list of one or more entries, e.g. 7.1.2 or 8.1.0, 9 it will look for exact matches, as a range, e.g. 7.1.2 - 9 it will check to make sure the current version falls within those limits. Whitespace optional, and supplied version values should be in the same number format they are in the build.prop value for that Android version.
supported.patchlevels= will match against ro.build.version.security_patch from the current ROM's build.prop. It can be set as a closed or open-ended range of dates in the format YYYY-MM, whitespace optional, e.g. 2019-04 - 2019-06, 2019-04 - or - 2019-06 where the last two examples show setting a minimum and maximum, respectively.
block=auto instead of a direct block filepath enables detection of the device boot partition for use with broad, device non-specific zips. Also accepts any partition filename (from by-name), e.g. boot, recovery, or vendor_boot.
is_slot_device=1 enables detection of the suffix for the active boot partition on slot-based devices and will add this to the end of the supplied block= path. Also accepts auto for use with broad, device non-specific zips.
ramdisk_compression=auto allows automatically repacking the ramdisk with the format detected during unpack. Changing auto to gz, lzo, lzma, xz, bz2, lz4, or lz4-l (for lz4 legacy) instead forces the repack as that format, and using cpio or none will (attempt to) force the repack as uncompressed.
patch_vbmeta_flag=auto allows automatically using the default AVBv2 vbmeta flag on repack, and use the Magisk configuration (Canary 23016+). Set to 0 forces keeping whatever is in the original AVBv2 flags, and set to 1 forces patching the flag (only necessary on few devices).
customdd="<arguments>" may be added to allow specifying additional dd parameters for devices that need to hack their kernel directly into a large partition like mmcblk0, or force use of dd for flashing.
slot_select=active|inactive may be added to allow specifying the target slot. If omitted the default remains active.
no_block_display=1 may be added to disable output of the detected final used partition+slot path for zips which choose to include their own custom output instead.
Command Methods
Code:
ui_print "<text>" [...]
abort ["<text>" [...]]
contains <string> <substring>
file_getprop <file> <property>
set_perm <owner> <group> <mode> <file> [<file2> ...]
set_perm_recursive <owner> <group> <dir_mode> <file_mode> <dir> [<dir2> ...]
dump_boot
split_boot
unpack_ramdisk
backup_file <file>
restore_file <file>
replace_string <file> <if search string> <original string> <replacement string> <scope>
replace_section <file> <begin search string> <end search string> <replacement string>
remove_section <file> <begin search string> <end search string>
insert_line <file> <if search string> before|after <line match string> <inserted line>
replace_line <file> <line replace string> <replacement line> <scope>
remove_line <file> <line match string> <scope>
prepend_file <file> <if search string> <patch file>
insert_file <file> <if search string> before|after <line match string> <patch file>
append_file <file> <if search string> <patch file>
replace_file <file> <permissions> <patch file>
patch_fstab <fstab file> <mount match name> <fs match type> block|mount|fstype|options|flags <original string> <replacement string>
patch_cmdline <cmdline entry name> <replacement string>
patch_prop <prop file> <prop name> <new prop value>
patch_ueventd <ueventd file> <device node> <permissions> <chown> <chgrp>
repack_ramdisk
flash_boot
flash_generic <partition name>
write_boot
reset_ak [keep]
setup_ak
"if search string" is the string it looks for to decide whether it needs to add the tweak or not, so generally something to indicate the tweak already exists. "cmdline entry name" behaves somewhat like this as a match check for the name of the cmdline entry to be changed/added by the patch_cmdline function, followed by the full entry to replace it. "prop name" also serves as a match check in patch_prop for a property in the given prop file, but is only the prop name as the prop value is specified separately.
Similarly, "line match string" and "line replace string" are the search strings that locate where the modification needs to be made for those commands, "begin search string" and "end search string" are both required to select the first and last lines of the script block to be replaced for replace_section, and "mount match name" and "fs match type" are both required to narrow the patch_fstab command down to the correct entry.
"scope" may be specified as "global" to force all instances of the string/line targeted by replace_string, replace_line or remove_line to be replaced/removed accordingly. Omitted or set to anything else and it will perform the default first-match action.
"before|after" requires you simply specify "before" or "after" for the placement of the inserted line, in relation to "line match string".
"block|mount|fstype|options|flags" requires you specify which part (listed in order) of the fstab entry you want to check and alter.
dump_boot and write_boot are the default method of unpacking/repacking, but for more granular control, or omitting ramdisk changes entirely ("OG AK" mode), these can be separated into split_boot; unpack_ramdisk and repack_ramdisk; flash_boot respectively. flash_generic can be used to flash an image to the corresponding partition. It is automatically included for dtbo and vendor_dlkm in write_boot but can be called separately if using "OG AK" mode or creating a simple partition flashing only zip.
Multi-partition zips can be created by removing the ramdisk and patch folders from the zip and including instead "-files" folders named for the partition (without slot suffix), e.g. boot-files + recovery-files, or kernel-files + ramdisk-files (on some Treble devices). These then contain Image.gz, and ramdisk, patch, etc. subfolders for each partition. To setup for the next partition, simply set block= (without slot suffix) and ramdisk_compression= for the new target partition and use the reset_ak command.
Similarly, multi-slot zips can be created with the normal zip layout for the active (current) slot, then resetting for the inactive slot by setting block= to the partition (without slot suffix) again, slot_select=inactive and ramdisk_compression= to the desired options for the target slot and using the reset_ak keep command, which will retain the patch and any added ramdisk files for the next slot.
backup_file may be used for testing to ensure ramdisk changes are made correctly, transparency for the end-user, or in a ramdisk-only "mod" zip. In the latter case restore_file could also be used to create a "restore" zip to undo the changes, but should be used with caution since the underlying patched files could be changed with ROM/kernel updates.
You may also use ui_print "<text>" to write messages back to the recovery during the modification process, abort "<text>" to abort with optional message, and file_getprop "<file>" "<property>" and contains "<string>" "<substring>" to simplify string testing logic you might want in your script.
Binary Inclusion
The AK3 repo includes current ARM builds of magiskboot, magiskpolicy, lptools_static and busybox by default to keep the basic package small. Builds for other architectures and optional binaries (see below) are available from the latest Magisk zip, or my latest AIK-mobile and Flashlt packages, respectively, here:
https://forum.xda-developers.com/t/...kernel-ramdisk-win-android-linux-mac.2073775/ (Android Image Kitchen thread)
https://forum.xda-developers.com/t/...-and-ends-multiple-devices-platforms.2239421/ (Odds and Ends thread)
Optional supported binaries which may be placed in /tools to enable built-in expanded functionality are as follows:
mkbootfs - for broken recoveries, or, booted flash support for a script/app via bind mount to /tmp (deprecated/use with caution)
flash_erase, nanddump, nandwrite - MTD block device support for devices where the dd command is not sufficient
dumpimage, mkimage - DENX U-Boot uImage format support
mboot - Intel OSIP Android image format support
unpackelf, mkbootimg - Sony ELF kernel.elf format support, repacking as AOSP standard boot.img for unlocked bootloaders
elftool (with unpackelf) - Sony ELF kernel.elf format support, repacking as ELF for older Sony devices
mkmtkhdr (with unpackelf) - MTK device boot image section headers support for Sony devices
futility + chromeos test keys directory - Google ChromeOS signature support
boot_signer-dexed.jar + avb keys directory - Google Android Verified Boot 1.0 (AVBv1) signature support
rkcrc - Rockchip KRNL ramdisk image support
Optionally moving ARM builds to tools/arm and putting x86 builds in tools/x86 will enable architecture detection for use with broad, device non-specific zips.
Boom . dibs on first :good:
You get 2 thank button presses fro me lol
Awesome work man as always
Good thing that this amazing work has it's own thread. Congrats buddy.
Sent from my Galaxy Nexus using XDA Premium 4 mobile app
Thanks guys!
I figured it would be nice to get it out there and also have it as a "Help Desk" thread for kernel devs who have questions about implementation, etc. too. Some devices might require switching it from dd to MTD-Utils, so I can help with that. So on and so forth.
Once we get a few devs who know how to use it, it should be pretty easy to help others. I'm looking at you Smitty. No pressure.
I finished my thanks ... but as always a great job.
ak said:
I finished my thanks ... but as always a great job.
Click to expand...
Click to collapse
So wait im confused. ?.. so 1) those that mean I can flash ak kerenl 4.2 with ur any kernel to my 4.4 .
2) those it have to be same kerenl for same phone manufacturer. Meaning can I be stupid enought to flash a nexus 4 kernel in my gnexus?
I understand any kernel cause I have been using smitty so thanks
milojoseph said:
So wait im confused. ?.. so 1) those that mean I can flash ak kerenl 4.2 with ur any kernel to my 4.4 .
2) those it have to be same kerenl for same phone manufacturer. Meaning can I be stupid enought to flash a nexus 4 kernel in my gnexus?
I understand any kernel cause I have been using smitty so thanks
Click to expand...
Click to collapse
Haha I wrote "device-specific" in the OP to try and avoid this very confusion.
Since I answered this same question earlier tonight in my Odds and Ends thread I'll just paste it here:
caspboy said:
so now devs can use kernels from other devices with their roms?
Click to expand...
Click to collapse
osm0sis said:
No. That's crazy talk. :laugh:
The concept of AnyKernel has been around for awhile, (originally by Koushik Dutta/ClockworkMod,) which allows device-specific kernels to be flashed over device-specific ROMs and use the ramdisk that came with the ROM to reduce the chance of any issues arising from the custom kernel pairing.
The drawback to this is that some kernels require modifications to the ramdisk to enable/set up kernel features, but in the old AnyKernel format there was no way to do this. Until now.
AnyKernel 2.0 makes it easy for kernel devs to use a number of simple command methods to automate the process of adding tweaks into a ROM's underlying ramdisk during the flashing process. :good:
Click to expand...
Click to collapse
Hopefully that helps. Basically exactly what's in the OP since that's where I edited it in from.
The only way I can explain it any further is with the very basics: that kernel boot.img files contain a zImage and a ramdisk. "AnyKernel Classic" just slaps the custom kernel zImage on top of the ROM's untouched default kernel boot.img ramdisk. AnyKernel 2.0 allows kernel devs to also modify the ramdisk to add anything required for kernel features in addition to the usual repacking it with the custom zImage and flashing it.
Great thread!! Best of lucky bro!!!
Sent from my Galaxy Nexus using xda app-developers app
AnyKernel will work on my phone now ? Thanks for enhancing awesome @osm0sis but DrRamdisk to the rest of you guys ?
wow,thats very cool,great work.
Github updated with my own forked native compiles of mkbootimg+unpackbootimg.
This should expand AnyKernel 2.0 device support a lot by using all the available offsets in mkbootimg, as exported by my drastically updated unpackbootimg. :good:
osm0sis said:
Github updated with my own forked native compiles of mkbootimg+unpackbootimg.
This should expand AnyKernel 2.0 device support a lot by using all the available offsets in mkbootimg, as exported by my drastically updated unpackbootimg. :good:
Click to expand...
Click to collapse
Hi osm0sis,
Have You update anykernel 2.0 to work with cyanogen11 roms ? Thanks for Your hard work !
Should already?
It uses the ROM kernel ramdisk like AnyKernel always has. Your custom kernel dev just needs to use it. Spread the word. :good:
osm0sis said:
Should already?
It uses the ROM kernel ramdisk like AnyKernel always has. Your custom kernel dev just needs to use it. Spread the word. :good:
Click to expand...
Click to collapse
Recently I had used Your method on Cyano11 but boot stopped on "Google".. new Cyano11 (that required ramdisk changes) had just come out and maybe anykernel 2.0 was not ready yet (I had just discovered your brillant work on It ! : Dita incrociate.
I'll try again ... if I have trouble going to ask you for help ...
I am already spreading the word : Cool:
Thanks man : Good:
What custom kernel were you trying to adapt to AnyKernel so you could flash it on CM?
osm0sis said:
What custom kernel were you trying to adapt to AnyKernel so you could flash it on CM?
Click to expand...
Click to collapse
Two kernels... My custom kernel (from cyanogenmod sources) and recently Fancy kernel (dirty-fancy)... now I want to try Fancy Kernel .. I need of a hibryd ramdisk for best final results and Your project is perfect for It !!! You're a genius !!!
Please, Can You link me Your dirty-V kernel re-pack by Anykernel 2.0 ? So I can follow It as an example. Thanks a lot...
So if I understand you, you're trying to make an AnyKernel 2.0 of Fancy Kernel so that you can flash it on any ROM for your device?
Should be doable. The DirtyV AnyKernel 2.0 is the example posted to the GitHub repo in the OP. Just follow the instructions to make your own anykernel script so that it will add the /sbin/ scripts and other ramdisk modifications (init.d, etc.) that @boype uses, instead of the DirtyV ones.
Good luck!
osm0sis said:
So if I understand you, you're trying to make an AnyKernel 2.0 of Fancy Kernel so that you can flash it on any ROM for your device?
Should be doable. The DirtyV AnyKernel 2.0 is the example posted to the GitHub repo in the OP. Just follow the instructions to make your own anykernel script so that it will add the /sbin/ scripts and other ramdisk modifications (init.d, etc.) that @boype uses, instead of the DirtyV ones.
Good luck!
Click to expand...
Click to collapse
Yes !
osm0sis ? If I want include init.rc original file by "real" ramdisk can I copy It as is into patch folder ?
It would go against the idea of AnyKernel to include the file like that. Remember, everything automatically comes from the original ramdisk, I just give you the ability to alter those files to add tweaks. :good:
The purpose of this thread is to provide a guide for users who have Proxyme preloaded in their device's firmware and want to find out how to use it effectively. Ideally, this will be a place to share experiences and ideas to further improve the tool and provide solutions to problems that people may have.
Introduction
Proxyme ( proc-zahym ) represents a system access solution comprised of the following components:
System service - provides access to privileged system environment
SSH daemon - provides secure shell (ssh) and file (scp) access (based on dropbear)
proxyme.apk - user interface module
This solution is offered as a preloaded option in firmware images and consequently cannot (should not) be installed as a regular app, either from the Play Store or being side loaded. The reason for pre-loading stems from the requirements of the system service component to be able to integrate at system level and not be bound by operating restrictions within the Android application and framework platform environment (Zygote and Dalvik sandbox). The Play Store has been enlisted as the primary and preferred source in providing updates to the user interface component; the actual app you will be interacting with.
Proxyme offers the following functionality through its user interface:
Installation/de-installation of the su binary to provide/remove root access
(useful only for other applications which require root level access)
The persistent behaviour of the su binary can be controlled by a one-shot switch
Register/de-register tag-along scripts for su enable and disable actions
(more details on this below)
Control availability and location of busybox toolbox
Start/Stop SSH daemon
Configure listening port for the SSH daemon
Configure user accounts for the SSH daemon
Submit and execute a shell script
SU Binary
The option to enable or disable the su binary switch (on/off) in the user interface is the equivalent of rooting and unrooting the device. When enabled, you are providing root access to apps which require it to perform correctly. Currently, Proxyme does not have built-in support for monitoring and 'policing' the actual access to root.
Auto Root @ Boot
This switch in the Proxyme app allows you to indicate whether the su binary should be installed or removed during a reboot or startup of the device. Setting it to the 'on' position will make the su binary persistent throughout reboot cycles and leave your phone permanently 'rooted'.
Registering Tag-along Scripts
Whenever you enable or disable the su binary with the on/off switch in the user interface, there exists an option to execute a user script just prior to and one unique to each action. This is possible by pre-registering a script for one of or both enable/disable actions. A script can virtually perform anything and is always executed within root context. Note that you must be very cautious about the scripts you are registering and be certain about their intentions, because a rogue script could cause irreparable damage to you device.
Each script has the option to override, and thus block, the intended action (enable or disable) by setting a system property named proxyme.override to anything but blank.
One purpose of having tag-along scripts would be to 'freeze' and 'unfreeze' specific root-shy apps, which do not 'like' rooted systems. This is one area where we can share the experience of pre-coded scripts for certain target apps and I do hope it will be put to good use.
To submit a script file, tap on one of the SU Enable Script or SU Disable Script text elements to start browsing for a file.
Busybox
Busybox is just that, busybox. Options are available to determine one of two hard-configured locations where it can be installed and to enable or disable it.
More to follow later...
SSH Daemon
The SSH daemon is based on dropbear. It has been modified to support logon accounts in Android, which are configured with the following parameters:
username
password
home directory
which shell to use
user ID
group ID
For whatever reasons, you can restrict access by specifying non-root user and group (0:0) IDs. The IDs you can choose from are derived from a system list which was used and known within Android at the moment of booting the device. If you have installed new apps in the meantime and would like to use their newly assigned IDs, then please reboot the phone to update this list.
Executing Shell Scripts
The ability to submit and execute a shell script from the user interface can be considered a convenient and quick way to get some tasks done. Take note however that your scripts are run in a privileged environment under the root account and that there are risks involved. A rogue or insufficiently tested script can cause major problems if/when it makes changes to key system partitions, which are normally mounted read only for obvious reasons.
Most rom images will include a sample de-bloating script,which removes ROM specific branding apps. The script. /sdcard/Proxyme/debloat.sh, shows how this is done and could serve as a base for more extensive clean-up of firmware components, if you so desire.
Operational Notes
Whenever a device boots from a factory reset condition (i.e. after wiping data), there will be no UID/GID list available in the user management screen. The reason for this is that the SuMeD setup process will complete before the app data store, the location where aforementioned list is stored. has been initialised. Restart the device in order to make this list available.
Behind The Scenes
For details regarding how Proxyme's system service components are integrated in a firmware image, please follow this trail...
Device Support
Before taking the next step to flash your phone/device, please be aware of the risks involved with performing such an operation. Prepare the device properly, i.e. sufficient battery charge, and be well informed of the correct flashing procedure(s) for your device's make and model. On Samsung devices, rooting will probably trigger 'custom' flag(s) and consequently render the warranty void. No matter how adventurous you may feel, it is always a bad idea to try to flash a firmware image which is not intended for your device. Having said all that, note that you will be flashing your phone at your own risk. You are solely responsible for anything you do to your phone/device, so make sure you are well informed and well prepared before deciding to flash or install anything on it.
The following list will be updated as soon as new firmware images are prepared for new and old devices.
Samsung Galaxy Note 10.1 2014
SM-P600 - (reference post)
Samsung Galaxy J
SC-02F (Docomo) - (reference thread)
SGH-N075T (Taiwan) - (reference thread)
Samsung Note 3
SM-N9005 - (reference post)
SM-N900A - (reference post - unconfirmed)
Samsung Galaxy S4
SHV-E330K - (reference thread)
SHV-E330L - (reference thread)
SHV-E330S - (reference thread)
SGH-I337 - (reference post - unconfirmed)
SC-04E - (reference post)
Samsung Galaxy Grand 2
SM-G710L - (reference post)
Samsung Galaxy S3
GT-I9300 - (reference post)
SC-03E - (reference thread)
SHV-E210K - (reference thread)
SHV-E210L - (reference thread)
SHV-E210S - (reference post)
SHW-M440S - (reference post)
Samsung Galaxy S2 LTE
SHV-E110S - (reference thread)
Samsung Galaxy S2
SHW-M250K - (reference post)
Planned Changes
built-in control of su access (much like what Superuser currently does)
choice of built-in simple file browser or use intents to initiate external app(s) for browsing and selecting files
...
Proxyme - Behind The Scenes
This section details how Proxyme's system service components are integrated in a firmware image.
If you are not up to speed with how a typical Android system is constructed, then I would like to suggest you at least make yourself familiar with this topic in order to fully understand what to do with the following text.
The system service components are integrated in the /system partition (mount point) in Android. In the case of changing a live system this will require mounting the appropriate partition read/write before applying the updates. If a static firmware image is to be updated, then extract the component which represents the /system partition from the package and apply the updates before re-packing the firmware image.
The following list describes the major system service components:
hijacker - this is a module you need to write, which has the role of initiating the system service in a privileged environment.
hjprepper - this module is started by the hijacker to prepare the environment prior to starting SuMeD
SuMeD - this one is what it's all about. The Proxyme app relies on this daemon to be up and running in order to perform any of its privileged functions
SSHD - the SSH daemon is represented by an updated implementation of dropbear on Android
Hijacker
The hijacker is a program you would normally have to write to replace an existing program in your rom, which is started during the boot process by for example initd. This part of the integration process requires your (creative) input, since you need to analyse the rom you are working on and figure out how and where to position the hijacker module. If you do find an existing module to hijack, make sure to always call that original module from your hijacker once it has managed to execute the hjprepper program. In some roms it suffices to start hjprepper from a shell script, which is run with root access... they exist, you just have to look for them.
This is what your hijacker could look like in C
Code:
#define PROP_HIJACK "proxyme.hijack.system"
#define HIJACKEE "/system/bin/original-program"
#define PREPPER "/system/xbin/hjprepper"
int main( int argc, char *argv[] )
{
char *lArgv[5];
char **lArgList;
int lArgCnt;
pid_t pid;
lArgList = (char **)malloc( sizeof(void *) * (argc + 1) );
for ( lArgCnt = 0; lArgCnt < argc; lArgCnt++ )
{
lArgList[ lArgCnt ] = argv[ lArgCnt ];
}
lArgList[ lArgCnt ] = NULL;
/* Fork parent process */
pid = fork();
if ( pid < 0 )
{
property_set( PROP_HIJACK, (char *)"Hijacker Startup... spawning failed, prep first before xfer" );
system( "/system/xbin/hjprepper" );
execv( HIJACKEE, lArgv );
exit( EXIT_SUCCESS );
}
else if ( pid > 0 )
{
property_set( PROP_HIJACK, (char *)"Hijacker startup... spawned, parent ascends phase 2" );
execv( HIJACKEE, lArgv );
exit( EXIT_SUCCESS );
}
if ( execl(PREPPER, PREPPER, (char *)NULL) < 0 )
{
property_set( PROP_HIJACK, (char *)"Hijacker startup... failed to call prepper" );
}
exit( EXIT_SUCCESS );
}
hjprepper
This program is responsible for setting up an operating environment for the SuMeD daemon. If you have full control over a rom's boot image, then include a call in your init process to start this module once during boot. If not, then use a hijacker program or look for existing and suitable scripts to initiate hjprepper.
hjprepper starts the SuMeD daemon once it completes the setup and configuration procedure.
SuMeD
This bad boy is responsible for the user requested actions through interaction with the Proxyme app.
Prebuilt Packages
To get you started, there are pre-built modules available,which you can download here. Currently, availability is limited to Android 4.3 and 4.4.2 only. The following zip archives are organized in a folder tree structure,which serves as a guide for where to place the modules within the /system path.
4.3 Prebuilts
4.4.2 Prebuilts
Filler 2
Filler 2
Filler 3
Filler 3
Please add support in latest SHV-E110S 4.1.2 rom(s)
Title says/asks it all...
Can You guide build pre-rooted rom by proxyme? Thank you very much.
linhbs said:
Can You guide build pre-rooted rom by proxyme? Thank you very much.
Click to expand...
Click to collapse
Behind The Scenes section has been added to the OP.
Can this method be used to prebuilts S3, S4, Note3 not Korea? Thanks so much.
linhbs said:
Can this method be used to prebuilts S3, S4, Note3 not Korea? Thanks so much.
Click to expand...
Click to collapse
Yes. You need to figure out how to get the SuMeD daemon started and that depends on the rom you want to integrate it in. The Behind The Scenes post highlights what areas to focus on when doing this.
Note that the first post includes 2 firmware images (both Android 4.3 and 4.4.2) for the international Note3 (SM-N9005). It's a no-brainer to copy the files from the appropriate directories to an equivalent and same level version firmware for another region of the same device.
Please add support N900A 4.4.2. Thank you very much.
linhbs said:
Please add support N900A 4.4.2. Thank you very much.
Click to expand...
Click to collapse
Has 4.4.2 been released on that device? If yes, a download link for the official stock firmware will help speed up the process. If not, then we wait or you could send a PM to davidcsv with the 10 or 11 digit s/n and he will monitor and download the latest release as soon as it becomes available...after that your new firmware image will be uploaded within a day.
Link: http://www.androidfilehost.com/?fid=23321874045862490. Thank you for your interest!
linhbs said:
Link: http://www.androidfilehost.com/?fid=23321874045862490. Thank you for your interest!
Click to expand...
Click to collapse
N900AUCECMLG (preloaded with Proxyme) (2014-01-04)
This rom implicitly performs a factory reset, so backup your data before flashing it. Unpack the zip archive and specify the resulting .tar.md5 filename in the PDA/AP section of the latest version of Odin.
Use Proxyme to execute the /sdcard/Proxyme/debloat.sh script to get rid of the k n o x messages.
mega.co.nz
torrent, mirror
Apparently, this firmware image is a pre-release/leaked image and not the final deal. It includes an updated bootloader and related components, meaning that it will not be straightforward to revert back to an older version of the firmware. If you encounter problems with this Proxyme preloaded image, then I'd suggest flashing the image from the original download link.
All feedback is welcome and will be appreciated. Enjoy!
Thank you very much. I ask you to add proxyme in I337 4.4.2 rom. Thank you very much.
Link: http://www.androidfilehost.com/?fid=23329332407566813
linhbs said:
Thank you very much. I ask you to add proxyme in I337 4.4.2 rom. Thank you very much.
Link: http://www.androidfilehost.com/?fid=23329332407566813
Click to expand...
Click to collapse
I337UCUFMLD (preloaded with Proxyme) (2014-01-02)
This rom implicitly performs a factory reset, so backup your data before flashing it. Unpack the zip archive and specify the resulting .tar.md5 filename in the PDA/AP section of the latest version of Odin.
Use Proxyme to execute the /sdcard/Proxyme/debloat.sh script to get rid of the k n o x messages.
mega.co.nz
torrent, mirror
Apparently, this firmware image is also a pre-release/leaked image and not the final deal. It too includes an updated bootloader and related components, meaning that it will not be straightforward to revert back to an older version of the firmware. If you encounter problems with this Proxyme preloaded image, then I'd suggest flashing the image from the original download link. A Google search shows that this image does have a few minor issues, so beware.
All feedback is welcome and will be appreciated. Enjoy!
Thank so much. I find the phone test. Will respond to you.
SC-04E Stock Firmware Proxyme Rooter images
Root Ready Stock Images
(Unfortunately, flashing these ROMs will trigger KNOX)
Kitkat 4.4
SC04EOMUFNI3 (Proxyme) (Build Date 2014-09-19)
This zip archive contains an Odin flashable file. It is not the complete stock image, so you MUST have OMUFNI3 already running on your phone or you will need to download it from the above reference sites, which carry complete stock firmware images, and flash it before continuing with this file. Instructions are included in the zip archive.
uploaded.net
mediafire
torrent, mirror2
I337:
- Before flash rom: I337UCUEMK2 version 4.3
- After flash rom I337UCUFMLD (preloaded with Proxyme) fail.
Good.
linhbs said:
I337:
- Before flash rom: I337UCUEMK2 version 4.3
- After flash rom I337UCUFMLD (preloaded with Proxyme) fail.
Click to expand...
Click to collapse
Please post the complete log from the message box in Odin. One more question, is your phone 16GB or 32GB model?
update: and also try again with newer version of Odin v3.09 instead of v3.07
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
most part of Introduction from taken sdat2img thread
if you want extract non OTA zip i suggest to use sdat2img tool
FOR AB OTA ZIPS USE THIS SCRIPT AND PAYLOAD EXTRACTOR
in this tools i made block_image_update() function from android recovery to a binary to run on PC and patch system.img with DAT files on OTA package to update system.img by OTA
also we have apply_patch() function for patching boot.img and other files like firmwares
Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.
What does new Android zips (full roms, but also OTAs) contain?
New Android flashable zips are made this way:
boot.img (kernel)
file_contexts (selinux related)
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat (for OTAs)
system.transfer.list (see explanation below)
and other patch files (.p only on OTA zip)
What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:
The transfer list is a text file containing commands to transfer data from one place to another on the target partition.
Click to expand...
Click to collapse
But what each line means?:
First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on versions >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
Fifth line and subsequent lines are all individual transfer commands.
Click to expand...
Click to collapse
all transfer commands is :
bsdiff
*erase
free
imgdiff
move
*new
stash
*zero
BlockImageUpdate is reading system.transfer.list and executing all commands
But BlockImageVerify doesn’t execute * commands, which not to make changes on system.img just verifying update can happen
original sdat2img tool only support "new" command
Ok, but how to Patch the system.img with OTA files?
All instructions are below. binaries are involved. Please read carefully step by step.
You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.
If you have questions or problems
write here
Thanks
- @xpirt , for original sdat2img tool and useful thread
XDA:DevDB Information
IMG Patch Tools, sdat2img for OTA zips, Tool/Utility for all devices (see above for details)
Contributors
erfanoabdi
Source Code: https://github.com/erfanoabdi/imgpatchtools
Version Information
Status: Testing
Created 2017-07-21
Last Updated 2020-01-23
Usage
Code:
./BlockImageUpdate <system.img> <system.transfer.list> <system.new.dat> <system.patch.dat>
args:
<system.img> = block device (or file) to modify in-place
<system.transfer.list> = transfer list (blob) from OTA/rom zip
<system.new.dat> = new data stream from OTA/rom zip
<system.patch.dat> = patch stream from OTA/rom zip
Code:
./ApplyPatch <file> <target> <tgt_sha1> <size> <init_sha1(1)> <patch(1)> [init_sha1(2)] [patch(2)]...
args:
<file> = source file from rom zip
<target> = target file (use "-" to patch source file)
<tgt_sha1> = target SHA1 Sum after patching
<size> = file size
<init_sha1> = file SHA1 sum
<patch> = patch file (.p) from OTA zip
Code:
usage: ./scriptpatcher.sh <updater-script>
args:
<updater-script> = updater-script from OTA zip to patch recovery commands
Example
for example from updater-script of OTA we have:
Code:
block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat")
apply_patch("EMMC:/dev/block/bootdevice/by-name/boot:33554432:f32a854298814c18b12d56412f6e3a31afc95e42:33554432:0041a4df844d4b14c0085921d84572f48cc79ff4",
"-", 0041a4df844d4b14c0085921d84572f48cc79ff4, 33554432,
f32a854298814c18b12d56412f6e3a31afc95e42,
package_extract_file("patch/boot.img.p"))
after getting system.img and boot.img from firmware This is equals of previous functions on PC with this tools:
Code:
~$ ./BlockImageUpdate system.img system.transfer.list system.new.dat system.patch.dat
~$ ./ApplyPatch boot.img - 0041a4df844d4b14c0085921d84572f48cc79ff4 33554432 f32a854298814c18b12d56412f6e3a31afc95e42
scriptpatcher.sh will generate all commands automatically from updater script so run it like:
Code:
~$ ./scriptpatcher.sh META-INF/com/google/android/updater-script > fullpatch.sh
check fullpatch.sh your self, you need to provide all images and files in correct name and patch as mentioned in mount and other commands of fullpatch.sh
Building Requirements
For Building this tool you need :
zlib
libbz2
openssl
It currently supports Linux x86/x64 & MacOS, Not tested on Windows.
Compile and Build command:
Code:
make
Youtube
Download links:
GitHub Release
Changelog:
GitHub Commits
Known Bugs/Issues:
Plz test and report
Nice work
Very nice
Sent from my LG-LS997 using Tapatalk
Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!
One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?
I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.
This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":
Code:
performing verification
creating cache dir cache
cache dir: cache
blockimg version is 3
maximum stash entries 236
creating stash cache/537ed49fb5f6c32dc3d205d78b6084fe54f70cd3/
The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.
daniel_m said:
Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!
One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?
I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.
This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":
The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.
Click to expand...
Click to collapse
Ah actually yes you need free space on RAM as much as both patch.dat and new.dat sizes
It's on my todo list to fix it, https://github.com/erfanoabdi/imgpatchtools/blob/master/blockimg/blockimg.cpp#L383
How to extract the file system.new.dat
evangelico793 said:
How to extract the file system.new.dat
Click to expand...
Click to collapse
You can find it in OTA zip
Motorola? See this : motorola.erfanabdi.ir
erfanoabdi said:
You can find it in OTA zip
Motorola? See this : motorola.erfanabdi.ir
Click to expand...
Click to collapse
No longer look at the picture
erfanoabdi said:
You can find it in OTA zip
Motorola? See this : motorola.erfanabdi.ir
Click to expand...
Click to collapse
Something Looks Wrong ReCheck inputs Or No Update Available
for moto g4 play 6.0.1
evangelico793 said:
No longer look at the picture
Click to expand...
Click to collapse
Please send picture again
404 not found
evangelico793 said:
Something Looks Wrong ReCheck inputs Or No Update Available
for moto g4 play 6.0.1
Click to expand...
Click to collapse
You are doing something wrong see thread for help and examples
@erfanoabdi
Thank you a ton for these tools!
They work like a charm :good:
I knew there must be a way to do this, but it's definitely above my paygrade.
@erfanoabdi
I've been trying scriptpatcher.sh on various updater-scripts and have noticed some bugs.
If you're interested in looking at the files, let me know and I'll post them.
Q9Nap said:
@erfanoabdi
I've been trying scriptpatcher.sh on various updater-scripts and have noticed some bugs.
If you're interested in looking at the files, let me know and I'll post them.
Click to expand...
Click to collapse
Yeah, there's lots of bugs in that script
I appreciate any kind of help
So far i know :
Only supporting /dev/block/boot.../by-name partitions
Can't mount ext4 in macOS
I disabled verify, but we can verify images
And i haven't tested it so much
If it gets more complicated Maybe we have to rewrite it on python.
erfanoabdi said:
Yeah, there's lots of bugs in that script
I appreciate any kind of help
So far i know :
Only supporting /dev/block/boot.../by-name partitions
Can't mount ext4 in macOS
I disabled verify, but we can verify images
And i haven't tested it so much
If it gets more complicated Maybe we have to rewrite it on python.
Click to expand...
Click to collapse
*removed*
@erfanoabdi
I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.
I tried patching to create a new recovery image and also tried patching the boot image in place.
It always returns "Unknown patch file format".
I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.
Q9Nap said:
@erfanoabdi
I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.
I tried patching to create a new recovery image:
#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
failed to stat "recovery.img": No such file or directory
Unknown patch file format
Done with error code : 1
and also tried patching the boot image in place:
#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
Unknown patch file format
Done with error code : 1
I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.
Any ideas? Is my syntax incorrect?
Click to expand...
Click to collapse
That was chainfire commit changes how can i say "no" to his PR
I didn't had time to check it
I'll check it out when I could
Q9Nap said:
@erfanoabdi
I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.
I tried patching to create a new recovery image:
#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
failed to stat "recovery.img": No such file or directory
Unknown patch file format
Done with error code : 1
and also tried patching the boot image in place:
#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
Unknown patch file format
Done with error code : 1
I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.
Any ideas? Is my syntax incorrect?
Click to expand...
Click to collapse
I've tested recovery-from-boot by chainfire and i can confirm its working
Also i fixed no such file error
Try to compile new source
*edit*
Fixed!
Poison Kitchen IDE
Development preview
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Description
A powerful IDE for android ROM development
Runs native on WINDOWS, MacOS themed.
Its powered by my XanderUI class library for the .net framework
Currently Development preview builds are only available, meaning things may be broken, behave incorrectly or other
Click to expand...
Click to collapse
Features
-Full GUI coded in C# for fast runtime
-XanderUI based controls(My C# class library)
-Based on .net 4.5
-Support for every single android version
-Unpack rom from (.zip, .tar, .md5, .img, aml_upgrade_package)
-Pull all required files from device(root needed to copy kernel block)
-Adb, fastboot and drivers installater
-ROM information
-Explorer
-Deodexing(4.4.4 and lower ATM)
-Unpacking the kernel
-Converting file_contexts.bin to standard text
-Kernel explorer
-Repacking kernel
-Logging
-Auto-generating updater-script
-Use generic symlinks if files detected
-Use generic file contets if the kernel does not contain file_contexts
-/data/app auto transition to new rom(somewhat working)
-Wipe data excluding /data/media(/sdcard path)
-Auto kernel block detection
-Auto mounting /system and /data
-Packing rom
-Signing rom
-Updater-script editor
-Extansion support
-Emulated exension scripts(applied via C#)
-Degapps extension
-Deknox extension
-Custom emoji extension
-Enable sony apps extension
Click to expand...
Click to collapse
Features not yet implemented
-root via magisk(systemless) and superSU(standard)
-init.d
-deodexing 5.0 and up
-adbd insecure
-init.d tweaks
-build.prop tweaks
-change display I.D
-add sysrw/ro binary
-logging needs tweaking
-loading rom information updated in the background
-Unpacking RUU as project(HTC)
-Unpacking TFT as project(SONY)
-Unpacking system.dat(sparse)
-Setting your actual device as the project and apply changes in realtime
-/data/app auto transition to new rom
-Kernel may not unpack
-Convert line endings in explorer context menu
-Inbuilt file editor(with EOL auto-detection)
-Boot animation player/changer
-Auto flash rom with ORS(TWRP)
-Bluestacks as rom
Click to expand...
Click to collapse
Settings
Load settings()
-Set default startup project or startup menu
-Enable/disable logging
-Stop logger from detecting files or folders added, deleted, changed or moved
Installation settings()
-Change romname
-Change installation type(autodetect on unpack, User generated, Tool generated)
-Change file contexts method(Auto, extracted from kernel, assumed)
-Enable/disable safewipe
-Enable/disable data/app auto transition
-Enable/disable autodetect ernel blockpath(add path below)
Pack settings()
-Change compression level
-Add signing method(pack into presigned, sign on zip, none)
-Change java heapsize
-Exclude files and folders from being packed/detected by IDE
Default program settings()
Change default program to open image files
Change default program to open video files
Change default program to open audio files
Change default program to open prop files
Change default program to open archive files
Change default program to open jar/apk files
Change default program to open other type files
Current version and updates()
Cleanup settings()
Click to expand...
Click to collapse
License
GNU GPL V3
Downloads
Downloads page
XDA:DevDB Information
Poison Kitchen IDE, Tool/Utility for the Android General
Contributors
Ricky Divjakovski
Version Information
Status: Testing
Created 2018-04-19
Last Updated 2018-04-19
Creating extensions and documentation
Code:
Description -
As extensions are a great adittion to the IDE, whats the use if you cant make your own for automated building?
Information you need to know -
The tool looks for the file "extension.info".
In the extension.info file you will specify the extension name, description and the poison shell script file(.psh).
Package the folder containing the extension as a zip archive
------------------------------------------------------------------
Entension index
------------------------------------------------------------------
Sample extension -
https://github.com/Ricky310711/Poison-Kitchen-Extension-Example
extension.info -
Lines will only be read if starting with "Name", "Description" or "Run"
-------------------- code exmpla
# this sets the extension name
Name:Fake optimizer
# this sets the extension description
Description:Do not apply this to rom, its a fake extension to show an example
# this is the poison script to run(must be in same directory)
Run:FakeOpt.psh
------------------------------------------------------------------
Scripting language(.psh file)
------------------------------------------------------------------
Information
-Must be linux EOL(\n)
-Anything but recognised 1st args are ignored
-Any errors will be ignore by the shell
-Use full paths asif you are in the root(/)
Extracting content to rom
---------- code example
EXTRACT|myFolder
1st arg states we are extracting(copying) a folder to the rom
2nd arg is the folder to extract(must be in same directory as extansion)
NOTE: This will extract the folder to the root of the rom
Changing a line in a file
---------- code example
CHANGE|START|/system/build.prop|ro.product.device=|THIS IS STARTS TEST
1st arg specifies we are changing a file
2nd arg specifies where on a line to search for the string(arg4)
3rd arg specifies the file to change
4th arg specifies the string to look for
5th arg is what you would like to replace the line with
The second arg can be START, CONTAINS or END
Appending a file
---------- code example
APPEND|TOP|/system/build.prop|# A TEST FOR APPENDING TOP
1st arg specifies we are appending a file
2nd arg specifies if we are appending at the TOP or BOTTOM
3rd are is the file to append
4th arg is the content to append
Remove a line from a file
---------- code example
REMOVE|/system/build.prop|# end fota properties
1st arg states we are removing a line from a file
2nd arg specifies the file
3rd arg is the string to look for in the line
Delete a file or folder
---------- code example
DELETE|/system/preinstall
Pretty straight forward, will delete a file or folder called "preinstall" from /system/
Create a file or folder
---------- code example
CREATE|DIRECTORY|/preinstall
arg 1 states we are creating something
arg 2 specifies if a FILE or FOLDER
arg 3 is the file or folder to create
Rename a file or folder
---------- code example
RENAME|/system/bin/am|amRenamed
1st arg specifies we are renaming a file or folder
2nd arg is the file or folder
3rd arg is the new name
Changelog
Development preview 2
-Added pull rom from device in setup
-Smoother nvigation
-XanderUI 1.6.0 integration
-Message boxes are now themed like the app
-Extensions now show progress upon running
-Deodexing up to Android 4.4.4 implemented
Development preview 1
-initial release
reserved3
supports all versions of android and all devices, no manual input needed.
dev preview 3 possibly tommorow
Yay now I can at least try to build my own g6 ROMs. Nice work.
development on hold until the 30th as im on holiday, i have been dedicating little time to this as internet here is extremely slow
Magisk and SU support added aswell as deodexing 5.x > 6.0
Also working on disabling signature verification, adding sysrw(as binary)
hi, im trying out the program, it never completes the extraction, and the progress icon starts at 53% do i need to do anything when trying to start the program.
i haVE TRIed running as administartor and without running as admin.
thanks
Twisted714 said:
hi, im trying out the program, it never completes the extraction, and the progress icon starts at 53% do i need to do anything when trying to start the program.
i haVE TRIed running as administartor and without running as admin.
thanks
Click to expand...
Click to collapse
A dialog will more then likely poppup requesting permission to run imgextractor.exe, be sure to accept that to complete the process
Ricky Divjakovski said:
A dialog will more then likely poppup requesting permission to run imgextractor.exe, be sure to accept that to complete the process
Click to expand...
Click to collapse
i am attaching some pics to what happens.
please advise. thanks
there are a couple more that are insignificant
Twisted714 said:
i am attaching some pics to what happens.
please advise. thanks
there are a couple more that are insignificant
Click to expand...
Click to collapse
Could you please PM me with pics of the firmware your selecting or even a link?
I've gone ahead and tried this. I pointed the app to my system.img (not the fastboot zip) and ended up with "error extracting". Tool looks promising. ROM is available for download here:
http://en.miui.com/download-333.html
I extracted it and used the images/system.img
oreo27 said:
I've gone ahead and tried this. I pointed the app to my system.img (not the fastboot zip) and ended up with "error extracting". Tool looks promising. ROM is available for download here:
http://en.miui.com/download-333.html
I extracted it and used the images/system.img
Click to expand...
Click to collapse
Im going to test now, if the firmware has system.transfer.list, and system.dat, it will not be able to be unpacked until i make a native library or extension to perform the operation as i think it would be extremely stupid for the need to have python installed to run.
The only reason i havent rebuilt smali/baksmali etc is 1. Would take me months, 2. we are modifying a system that relies on java to operate.
in future, hefty operations like unpacking etc, will be coded in ASM/C code for faster operation, Im planning on making the whole project open source to allow changes and fixes submitted by other developers, So if anyones interested let me know and ill upload the source code(Written in C# for the .net 4.5 framework)
Ricky Divjakovski said:
Im going to test now, if the firmware has system.transfer.list, and system.dat, it will not be able to be unpacked until i make a native library or extension to perform the operation as i think it would be extremely stupid for the need to have python installed to run.
The only reason i havent rebuilt smali/baksmali etc is 1. Would take me months, 2. we are modifying a system that relies on java to operate.
in future, hefty operations like unpacking etc, will be coded in ASM/C code for faster operation, Im planning on making the whole project open source to allow changes and fixes submitted by other developers, So if anyones interested let me know and ill upload the source code(Written in C# for the .net 4.5 framework)
Click to expand...
Click to collapse
Not sure if this helps but I've run Imgextractor directly on my system.img with this result:
Code:
Mi-A1-Repository>Imgextractor Mi-A1-Repository\Roms\Fastboot\tissot_images_V.9.5.10.0.ODHMIFA_8.0\images\system.img
ImgExtractor version 1.3.6 <Created by And_PDA (Based on sources ext4_unpacker)>
Extractor for images in EXT2\EXT3\EXT4\YAFFS2\CRAMFS filesystem formats
support SPARSE\SIN\MOTO structure formats
Open image file Mi-A1-Repository\Roms\Fastboot\tissot_images_V.9.5.10.0.ODHMIFA_8.0\images\system.img (size 3221225472 bytes) successfull...
Analize format of file. Please wait...
Found SPARSE FORMAT
Found EXT4 FORMAT
free space in image 188895232 bytes
Extract started. Please wait...
Extract 750 folders and 6306 files successfull
Found 386 symlink files
File stats (uid, gid, permission) save to Mi-A1-Repository\Roms\Fastboot\tissot_images_V.9.5.10.0.ODHMIFA_8.0\images\system__statfile.txt
Extract finish success
Press Enter to continue...
oreo27 said:
Not sure if this helps but I've run Imgextractor directly on my system.img with this result:
Code:
Mi-A1-Repository>Imgextractor Mi-A1-Repository\Roms\Fastboot\tissot_images_V.9.5.10.0.ODHMIFA_8.0\images\system.img
ImgExtractor version 1.3.6 <Created by And_PDA (Based on sources ext4_unpacker)>
Extractor for images in EXT2\EXT3\EXT4\YAFFS2\CRAMFS filesystem formats
support SPARSE\SIN\MOTO structure formats
Open image file Mi-A1-Repository\Roms\Fastboot\tissot_images_V.9.5.10.0.ODHMIFA_8.0\images\system.img (size 3221225472 bytes) successfull...
Analize format of file. Please wait...
Found SPARSE FORMAT
Found EXT4 FORMAT
free space in image 188895232 bytes
Extract started. Please wait...
Extract 750 folders and 6306 files successfull
Found 386 symlink files
File stats (uid, gid, permission) save to Mi-A1-Repository\Roms\Fastboot\tissot_images_V.9.5.10.0.ODHMIFA_8.0\images\system__statfile.txt
Extract finish success
Press Enter to continue...
Click to expand...
Click to collapse
Found the error, the img file is of a wierd format, as its named system.img, it actually contains files from the root, so within the system.img the the root of the image is actually the root of the device rather then the root of the system partition, its extremely odd but none the less extremely simple to fix
Issue is already fixed and will be included in dev preview 3, that is expected for public release in a day or so with much more additions
Ricky Divjakovski said:
Found the error, the img file is of a wierd format, as its named system.img, it actually contains files from the root, so within the system.img the the root of the image is actually the root of the device rather then the root of the system partition, its extremely odd but none the less extremely simple to fix
Issue is already fixed and will be included in dev preview 3, that is expected for public release in a day or so with much more additions
Click to expand...
Click to collapse
Yeah. I found it odd that the system.img had a /system directory in it. Awesome! Can't wait
Ricky Divjakovski said:
Could you please PM me with pics of the firmware your selecting or even a link?
Click to expand...
Click to collapse
its on mega.
https://mega.nz/#!dCA2mAYS!-GrKWuuTNODaYEbt3LWiw4LJzxkrz5wI3T94mQ4PU90
it is a android 6 to android 7 full update. its installed, but i am trying to learn cobble together a rom. this img is for a zoomtak upro, i have found today an image for the uplus/vplus. it has much more stuff in it.
thanks
Twisted714 said:
its on mega.
https://mega.nz/#!dCA2mAYS!-GrKWuuTNODaYEbt3LWiw4LJzxkrz5wI3T94mQ4PU90
it is a android 6 to android 7 full update. its installed, but i am trying to learn cobble together a rom. this img is for a zoomtak upro, i have found today an image for the uplus/vplus. it has much more stuff in it.
thanks
Click to expand...
Click to collapse
I will check it when i get home, if its an upgrade package, i cannot ad support for it..
Hello, recently I looked into LG backup and restore application using static analysis.
Note: this is based on version lg bnr 8.0.26 from 2020-02-13, tested with 7.0.20 from 2008-12-31
Here is the extract tool:
https://github.com/Mysak0CZ/LBFtool
And here is the technical part (you don't need to read this, if you just want to extract your data):
The file contains 2 headers (one of them is referred to as "new")
These are XML documents encrypted using AES/ECB/PKCS5Padding + SHA256 and null IV with static password
Code:
cd562f61-5399-3978-ac76-7c54b0508010
And they look like this:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<BackUp>
<Common>
<Name>LGBackup_200217_1</Name> <!-- Name of the backup file -->
<Id>Randomly generated UUID</Id> <!-- Id of the backup file -->
<IsNew>True</IsNew> <!-- IsNew is same in new and old header -->
<Version>1.00</Version>
<Attr Name="product_model">LM-X525</Attr>
<Attr Name="os_version">9</Attr>
<!-- And other attributes including google email -->
</Common>
<BackupItem Category="settings_DB" StartOffset="59392" DataSize="136626" />
<BackupItem Category="MEDIA_DB" StartOffset="901169" DataSize="78393344" FileName="external.db" />
<!-- Other backuped items -->
</BackUp>
The "new" header has more information than original header, but same structure.
The attribute "IsNew" signals the presence of the new header. (and is True in both files)
All numbers inside file are in Big-endian.
And this is the structure of the file (decimal position):
Code:
0: Unknown - 0x00 for me
1: Int32 - size of the new header
5: Int64 - offset to the new header (usually 13)
13 ... EOF-33: data
EOF-33: 20 bytes - SHA1 hash of the password (if present, otherwise still data)
EOF-13: Int64 - offset to the (old) header
EOF-5: Int32 - size of the (old) header
EOF-1: Bool - Lock byte - the file is (0x01) or isn't (0x00) protected by password
The password is only checked during import, not used to encrypt data (so to remove password, just set the last byte to 0)
Every "BackupItem" entry has a "Category".
Special case is Category="APPLICATION" with this structure:
Code:
<BackupItem Category="APPLICATION">
<APPLICATION Name="com.lge.ime" StartOffset="19221586" DataSize="2385983" Package="com.lge.ime">
<Attr Name="com.lge.ime">com.lge.ime</Attr>
</APPLICATION>
<APPLICATION Name="org.exmple.app" StartOffset="1429652009" DataSize="3420160" FileName="org.exmple.app.tar" Package="org.exmple.app">
<Attr Name="org.exmple.app">org.exmple.app</Attr>
</APPLICATION>
<!-- ... -->
</BackupItem>
All applications are tar files that contain 2 folders:
app - contains the apk file
data - application's internal data
All other items have:
StartOffset - Offset inside file
DataSize - Self explanatory
FileName - Not all have this, but most do; it is the original name of the backuped file.
The files are not compressed nor encrypted.
Footnote: This is my first post on XDA, so sorry for any mistakes I made (including my english) and feel free to correct me/give me advice.
Mysak0CZ said:
Hello, recently I looked into LG backup and restore application using static analysis.
Note: this is based on version lg bnr 8.0.26 from 2020-02-13, tested with 7.0.20 from 2008-12-31
Here is the extract tool:
And here is the technical part (you don't need to read this, if you just want to extract your data):
The file contains 2 headers (one of them is referred to as "new")
These are XML documents encrypted using AES/ECB/PKCS5Padding + SHA256 and null IV with static password
And they look like this:
The "new" header has more information than original header, but same structure.
The attribute "IsNew" signals the presence of the new header. (and is True in both files)
All numbers inside file are in Big-endian.
And this is the structure of the file (decimal position):
The password is only checked during import, not used to encrypt data (so to remove password, just set the last byte to 0)
Every "BackupItem" entry has a "Category".
Special case is Category="APPLICATION" with this structure:
All applications are tar files that contain 2 folders:
app - contains the apk file
data - application's internal data
All other items have:
StartOffset - Offset inside file
DataSize - Self explanatory
FileName - Not all have this, but most do; it is the original name of the backuped file.
The files are not compressed nor encrypted.
Footnote: This is my first post on XDA, so sorry for any mistakes I made (including my english) and feel free to correct me/give me advice.
Click to expand...
Click to collapse
Nice job dude! just a the quick recommendation: I would suggest that you put your code on gitlab or GitHub so it will become easier for others to contribute
Sent from my OnePlus 6T using XDA Labs
Hi. I'm trying to use this tool but I don't know how to use it. I followed the instructions and downloaded Python. but none of the other steps are working for me. I have my backup on my PC and am trying to extract my files into normal formats as I wish to move away from the LG platform. Thank you.
---------- Post added at 07:20 AM ---------- Previous post was at 06:57 AM ----------
Nevermind, I found a work around.
For anyone else in the future wondering: If you select backup with the app on the phone and backup to the internal memory, it will create .pdf files of all text/images in internal storage>>Quickmemo+. This doesn't help to retrieve the audio or video recordings, but at least that way you can retrieve your text files/images.
Tried script on old backup, failed with error.
Mysak0CZ said:
Hello, recently I looked into LG backup and restore application using static analysis.
Note: this is based on version lg bnr 8.0.26 from 2020-02-13, tested with 7.0.20 from 2008-12-31
Here is the extract tool:
And here is the technical part (you don't need to read this, if you just want to extract your data):
The file contains 2 headers (one of them is referred to as "new")
These are XML documents encrypted using AES/ECB/PKCS5Padding + SHA256 and null IV with static password
Code:
cd562f61-5399-3978-ac76-7c54b0508010
...
[/CODE]
All applications are tar files that contain 2 folders:
app - contains the apk file
data - application's internal data
All other items have:
StartOffset - Offset inside file
DataSize - Self explanatory
FileName - Not all have this, but most do; it is the original name of the backuped file.
The files are not compressed nor encrypted.
Footnote: This is my first post on XDA, so sorry for any mistakes I made (including my english) and feel free to correct me/give me advice.
Click to expand...
Click to collapse
My first post as well. Thanks for the script. I ran it on an old backup I was hoping to extract contacts from. See excerpt with error below.
C:\Temp\LG G4\Backup>python extract.py LGBackup_190218.lbf
[+] Backup size: 597864605
Parsing...
Reading header
[*] Header start offset: 597792768
[*] Header length: 71824
[+] Header read OK
Reading header (new)
[*] Header start offset: 7021235167904623918
Traceback (most recent call last):
File "extract.py", line 125, in <module>
root = readHeader(True)
File "extract.py", line 99, in readHeader
size = getBackupInfoSize(isNew)
File "extract.py", line 88, in getBackupInfoSize
raise Exception("Invalid header length.")
Exception: Invalid header length.
C:\Temp\LG G4\Backup>
Click to expand...
Click to collapse
I'm a complete newbie with Python, haven't done any coding in over a decade. Where do I go from here?
Ninja_Toad said:
My first post as well. Thanks for the script. I ran it on an old backup I was hoping to extract contacts from. See excerpt with error below.
I'm a complete newbie with Python, haven't done any coding in over a decade. Where do I go from here?
Click to expand...
Click to collapse
Hi! Your backup seem to be at least partially corrupted. (The new header position seems to start at 7021235167904623918, however the file is only 597864605 bytes long). Fortunatly there are two headers in the backup and the old one seems to be ok (You got "[+] Header read OK" for the old header). So you can try ignoring the "new" header.
To ignore the faulty header just set the relevant "FORCE_*_HEADER" to True in the script header. In your case just set "FORCE_OLD_HEADER = True".
Also note, that extracting contacts in readable format is beyond this script's current scope. This can however be done manually. After you get your data successfully extracted, look for export/data/contacts_DB file. This is a TAR file in which you need to find file "data/data/user/0/com.android.providers.contacts/databases/contacts2.db". This is the contact database as your phone stores it. To actually extract the contacts, you will need to find a way to convert the database to some other format (I have't looked into how the contacts are stored, you will need to do your own googling)
Files extracted but...
hello guys,
I also managed to extract my files using the hack mentioned above (force old headers to TRUE) since by using the default script i was also getting a similar error as above.
However now all of my files (pictures in particular) are corrupted and i cannot seem to be able to open them using several editors.
Anyone else had the same issue?
Thanks in advance!
serlal said:
hello guys,
I also managed to extract my files using the hack mentioned above (force old headers to TRUE) since by using the default script i was also getting a similar error as above.
However now all of my files (pictures in particular) are corrupted and i cannot seem to be able to open them using several editors.
Anyone else had the same issue?
Thanks in advance!
Click to expand...
Click to collapse
Yes I have exactly same issue as you have. All exported files cannot be read.
Hello @tking2000 and @serlal, this script was written based on versions that are available to me. There are two possibilities of what is happening:
1) The file was badly corruped, however the old header stayed itact. (This is the likely case, if you are trying this tool because your phone refused to restore the backup)
2) The file was made by version that does something differently and the script doesn't account for it.
In the first case there is nothing anyone can do (short of trying some corrupted file recovery software hoping the corruption is only minor)
In the second case, you can try sending me a PM with following information, so I can have a look and confirm whether this is truly the case (and improve the script)
1) The script's output
2a) The <Version> and <Attr Name="settingVersionName"> from the <Common> part of the header
2b) If you aren't concerned about privacy or it is just easier for you: just send the whole header file
3) The version of the com.lge.bnr app (LG Mobile Switch); this is not necessary, but may help me
Feel free to censor anything you are sending with [CENSORED] (like your email or filenames)
I have no experience with python, but I have one of these lbf files and I'm trying to figure out how to make it work and all I get is couldn't get library pycryptodome. How do I get Python 3.8.3 to get this library?
Multipart backups
Hello everyone,
I just finished an update, that should enable the tool to extract backups that have multiple files.
If you received "Exception: Invalid header length." in the past and your backup has multiple files, try the script now; your backups are likely to extract correctly now.
This is the likely case for: @Ninja_Toad @serlal and @tking2000
This was made possible thanks to @cheesyzombie who sent me the data I asked for.
@kamikazkid, You don't need to know Python to use this tool, the easiest way is to enter the commands in quotes on the github page into the console (To open the console Shift+Right click in the folder with backups and choose "Open command window here").
Just:
Download the zip on the github page (Under "Code" button)
Make sure the extract.py and your backup are in the same folder
Follow the usage, skip the 4th step and replace xxxxxx to match your backup name.
I figured it out, it was something with windows 10 and the version of python installation I used. Microsoft garbaging things up as usual, thanks for the help though. Thanks, I recovered my files from my old phone successfully.
I have no experience withc++, but I have one of these lbf files and I'm trying to figure out how to make it work, it is a great this forum
Is there a way to import the _DB files (contacts, messages, call logs etc.) back to a phone?
I asked before:
https://forum.xda-developers.com/lg-g3/general/restore-backup-lbf-lineage-17-t4172483#post83644281
Thank you so much for this! I was able to extract the Messaging data from an LG G8 ThinQ. Does anybody know the easiest way to view that data on my PC?
Hi, you can follow these steps to extract data fron _db files:
1) Send the _db files like contacts_db, messages_db to any android phone
2) Double tap on the file and open it using File Manager (which is by default on Mi phones)
3) The file will now open as a folder. Continue opening the folders until you reach to the last file.
4) The last file is the database file. It won't open directly. Select the file and extract the file.
5) Now transfer the file to computer and you can now view it using SQLite DB Browser if its a contacts file or you can use any VMG reader if its a messages file.
Hope this helps you.
Hi @Mysak0CZ, just follow your instructions on GitHub but when I run the command on Command Prompt, nothing happens. Just like this:
F:\>python -m pip install pycryptodome
F:\>python extract.py LGBackup-201031.lbf
When I enter nothing happened. Kindly help to advise for this.
Thank you
Problem in powershell
Mysak0CZ said:
Hi! Your backup seem to be at least partially corrupted. (The new header position seems to start at 7021235167904623918, however the file is only 597864605 bytes long). Fortunatly there are two headers in the backup and the old one seems to be ok (You got "[+] Header read OK" for the old header). So you can try ignoring the "new" header.
To ignore the faulty header just set the relevant "FORCE_*_HEADER" to True in the script header. In your case just set "FORCE_OLD_HEADER = True".
Also note, that extracting contacts in readable format is beyond this script's current scope. This can however be done manually. After you get your data successfully extracted, look for export/data/contacts_DB file. This is a TAR file in which you need to find file "data/data/user/0/com.android.providers.contacts/databases/contacts2.db". This is the contact database as your phone stores it. To actually extract the contacts, you will need to find a way to convert the database to some other format (I have't looked into how the contacts are stored, you will need to do your own googling)
Click to expand...
Click to collapse
Hi, I am trying to extract via your method but in the power shell in sending the command "python extract.py LGBackup_1x0x07.lbf" it does not execute and brings me back the initial path line. Do you have solutions in this regard? Thanks in advance
Ps. I use Win10
Worked so well. I only needed to recover images and audio recordings after realizing that LG backup tool won't restore on other brand phones.
Thank you!!
Mysak0CZ said:
Hello, recently I looked into LG backup and restore application using static analysis.
Note: this is based on version lg bnr 8.0.26 from 2020-02-13, tested with 7.0.20 from 2008-12-31
Here is the extract tool:
https://github.com/Mysak0CZ/LBFtool
And here is the technical part (you don't need to read this, if you just want to extract your data):
The file contains 2 headers (one of them is referred to as "new")
These are XML documents encrypted using AES/ECB/PKCS5Padding + SHA256 and null IV with static password
Code:
cd562f61-5399-3978-ac76-7c54b0508010
And they look like this:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<BackUp>
<Common>
<Name>LGBackup_200217_1</Name> <!-- Name of the backup file -->
<Id>Randomly generated UUID</Id> <!-- Id of the backup file -->
<IsNew>True</IsNew> <!-- IsNew is same in new and old header -->
<Version>1.00</Version>
<Attr Name="product_model">LM-X525</Attr>
<Attr Name="os_version">9</Attr>
<!-- And other attributes including google email -->
</Common>
<BackupItem Category="settings_DB" StartOffset="59392" DataSize="136626" />
<BackupItem Category="MEDIA_DB" StartOffset="901169" DataSize="78393344" FileName="external.db" />
<!-- Other backuped items -->
</BackUp>
The "new" header has more information than original header, but same structure.
The attribute "IsNew" signals the presence of the new header. (and is True in both files)
All numbers inside file are in Big-endian.
And this is the structure of the file (decimal position):
Code:
0: Unknown - 0x00 for me
1: Int32 - size of the new header
5: Int64 - offset to the new header (usually 13)
13 ... EOF-33: data
EOF-33: 20 bytes - SHA1 hash of the password (if present, otherwise still data)
EOF-13: Int64 - offset to the (old) header
EOF-5: Int32 - size of the (old) header
EOF-1: Bool - Lock byte - the file is (0x01) or isn't (0x00) protected by password
The password is only checked during import, not used to encrypt data (so to remove password, just set the last byte to 0)
Every "BackupItem" entry has a "Category".
Special case is Category="APPLICATION" with this structure:
Code:
<BackupItem Category="APPLICATION">
<APPLICATION Name="com.lge.ime" StartOffset="19221586" DataSize="2385983" Package="com.lge.ime">
<Attr Name="com.lge.ime">com.lge.ime</Attr>
</APPLICATION>
<APPLICATION Name="org.exmple.app" StartOffset="1429652009" DataSize="3420160" FileName="org.exmple.app.tar" Package="org.exmple.app">
<Attr Name="org.exmple.app">org.exmple.app</Attr>
</APPLICATION>
<!-- ... -->
</BackupItem>
All applications are tar files that contain 2 folders:
app - contains the apk file
data - application's internal data
All other items have:
StartOffset - Offset inside file
DataSize - Self explanatory
FileName - Not all have this, but most do; it is the original name of the backuped file.
The files are not compressed nor encrypted.
Footnote: This is my first post on XDA, so sorry for any mistakes I made (including my english) and feel free to correct me/give me advice.
Click to expand...
Click to collapse
ok I am so lost. And here is the technical part (you don't need to read this, if you just want to extract your data):
that statement would make it appear easy. well the link only takes you to the code, there is no tool and nothing with "new" as header. so ive tried deciphering what to do for hours now and i cant figure npthing out. rooted and flashed my lgv20 and did a backup with lg bridge beforehand. trying to restore and phone connects to computer fine but doesnt regster in lg bridge so that is why i need to extract .lbf file. can someone make this easier?
hello..it ended up successful.but there is no jpeg or mp4 file on DCIM folder..how can i restore them?should i do somthing with the script? i don't know pyton