Related
I'm looking for a way to apply the patch files to the TF system image on a PC. This is not about how to get the ota to flash- I got that covered. This is about repacking roms or making flashable updates that don't rely on stock firmware.
I found a few ways, but none of them seem "easy."
The dirty way- apply the OTA and grab the system image.
From the PC command line - Linux has diff/patch, but that doesn't work for some of the patches in "IMGDIFF2" format. Besides, I'm not sure I ever got the regular BSDIFF40 patches to work.
The git option- There is the apply_patch source code in github. This works great, but I have to run it on the TF. Is it possible to compile this to work on my ubuntu box?
Maybe there is some magic script out there called apply_ota?
Thanks in advance
gee one said:
I'm looking for a way to apply the patch files to the TF system image on a PC. This is not about how to get the ota to flash- I got that covered. This is about repacking roms or making flashable updates that don't rely on stock firmware.
I found a few ways, but none of them seem "easy."
The dirty way- apply the OTA and grab the system image.
From the PC command line - Linux has diff/patch, but that doesn't work for some of the patches in "IMGDIFF2" format. Besides, I'm not sure I ever got the regular BSDIFF40 patches to work.
The git option- There is the apply_patch source code in github. This works great, but I have to run it on the TF. Is it possible to compile this to work on my ubuntu box?
Maybe there is some magic script out there called apply_ota?
Thanks in advance
Click to expand...
Click to collapse
I recently wrote some code for a ARM basestation (basically like a Freescale cell-phone board), and had to be able to supply some updates over time. Before the project ended, I'd also found bsdiff and bspatch and tested them on linux, where they worked fine (as long as you'd created either an ARM bsdiff/patch version or somehow took the big/little endianness of the hardware into account).
At first I thought it wasn't working but it seemed to do OK on standard cross-compiled-for-ARM files. (well, they still ran and performed the same stunt..).
The thing that worried me about this method is that they'd never have the same md5sum before as after, likely because they had a different amount of bytes. (e.g. you make your own before and after and build each, then create the patch file using bsdiff, then patch the old file using the patch file, and create a 'newtestfile', but look at the number of bytes (or md5sum it) difference between the original stage 2 file and the stage 1 + the patch file via bspatch).
Sorry that is pretty confusing the way I just tried to explain it.. I'll look this over again tomorrow and figure out if I said anything worthwhile
For inputs, the applypatch binary takes the file names, sha1 checksums of the before and after, and the final target size. Chance are high that the patch would only work if everything went in and came out properly.
Code:
apply_patch("/system/framework/framework-res.apk", "-",
99b382b5886e505508fa3f730d778a1bed4bc04e, 9761707,
a4798fb8d61d08b6f067e522e67107f6dd556148, package_extract_file("patch/system/framework/framework-res.apk.p"));
I think that bsdiff/patch will generally work on most of the OTA (BSDIFF40 files), but some of the files are IMGDIFF2, which targets binary/data files.
The applypatch binary works on both and as far as I can tell, is the same binary that the TF uses. Here is a snippet from the AOSP recovery... TF binary included Use the binary on your TF, not applypatch.sh! I suppose you can put it in system/xbin/
It would resolve the issue if I could compile this to run on my desktop from ubuntu. Short of that, I'm working on a script for the TF to parse the updater script.
gee one said:
For inputs, the applypatch binary takes the file names, sha1 checksums of the before and after, and the final target size. Chance are high that the patch would only work if everything went in and came out properly.
Code:
apply_patch("/system/framework/framework-res.apk", "-",
99b382b5886e505508fa3f730d778a1bed4bc04e, 9761707,
a4798fb8d61d08b6f067e522e67107f6dd556148, package_extract_file("patch/system/framework/framework-res.apk.p"));
I think that bsdiff/patch will generally work on most of the OTA (BSDIFF40 files), but some of the files are IMGDIFF2, which targets binary/data files.
The applypatch binary works on both and as far as I can tell, is the same binary that the TF uses. Here is a snippet from the AOSP recovery... TF binary included Use the binary on your TF, not applypatch.sh! I suppose you can put it in system/xbin/
It would resolve the issue if I could compile this to run on my desktop from ubuntu. Short of that, I'm working on a script for the TF to parse the updater script.
Click to expand...
Click to collapse
I'm not sure I understand the intent. I've just awoken and that's probably part of the problem..
'applypatch' is part of the standard (say 10.04 ubuntu) repo and is in a package called makepatch. Did you want to be able to run it "from" a standard x86 Ubuntu dist but have it targeting "armle" files or just be able to run it both on Ubuntu & Android correctly?
The files I tested were a) both binaries in different stages of evolution, and b) both built for armle(droids) and x86_64 and didn't behave differently, worked, and the checksum could be pre-calculated.
I'm going to mess around with applypatch when I'm out of my morning coma and get back to this in a bit.
Hah -- I do wonder if there's already a script (maybe it's a binary though) that lives in the dm* app on the tf101 that just does this already. You may have been right about that. Something has to apply those OTA patches. There's a few mechanisms for detecting them but I hadn't thought about how they're applied.
I'm trying to think if there's any scenario where the market applies 'patches' rather than always downloads the next version in entirety. Not sure about that.
Anyway, I'll see if I can get anywhere with applypatch.
Edit:: After a little messing around :
I finally see what you are trying to do and why what I'm doing is totally useless for that. I'm going to spend a little time looking over the imgdiff2 (apk) deal, and probably end up now figuring out a thing.
I'll look into the makepatch package later today. Maybe that might be what I'm looking for.
Just to be clear, the intent is to be able to patch a "stock" system without it actually running on the TF. For example, to take the stock 8.6.5.9 image, decompress system/ to the PC, apply the patches from the OTA, and then end up with a stock 8.6.5.13 image. This would all happen on the PC, not the TF. It would streamline the ROM cooking process and make it easier and faster to spit out a complete, flashable update that won't fail sha1 checks because of mods.
sent from my cyanogen(mod) vision
Zombie Thread Alert!!!
You have been warned....
OK, so my C-fu isn't strong enough to figure out how to cross compile the apply patch binary to run on a x86 computer. However, I did notice that the applypatch binary gets built in the aosp/generic_x86 build, specifically in the system/bin folder.
This will be an exectuble file that will run on a x86 linux based system and do the same thing that the applypatch binary does to the ota package during the ota update. In other words, you can apply the ota patch to a system image from the comfort of the linux command line without having to actually flash the ota. This will allow you to be the first one on your block to have the new system image without having to mess up whatever rom you are currently running.
Of course, this might all be moot if we don't see another ota for our tf. Come on ASUS, just one more so I can test this out on a live update.
OK, so if you read this far, repo sync your aosp source and build the generic_x86 branch. The binary is aosp/target/product/generic_x86/system/bin/applypatch_static
edit: then after you've done that, use a script to parse the updater-script to extract the patching commands and then she-bang! I'll post mine a little later- I've got to get some flowers for tonight
Any luck with that? I'm also searching for a way to apply an OTA update to a ROM using Ubuntu.
bgcngm said:
Any luck with that? I'm also searching for a way to apply an OTA update to a ROM using Ubuntu.
Click to expand...
Click to collapse
Yeah, it works like a charm. The binary is compiled in the aosp/generic_x86 build. There is no need to flash, you can just pull it out of the system/bin folder after compiling the source. You can probably use an "mmm" command to make just files you want, but I'm not sure which package it's in. Then, I use a script that converts the OTA patch commands in the updater-script to a script file that runs on my linuxbox that calls the binary and patches the ROM under linux/ubuntu.
Do you mind sharing your applypatch binary? I don't have AOSP source on my Ubuntu installation right now, so if you don't mind sharing I wouldn't need to build it from sources...
bgcngm said:
Do you mind sharing your applypatch binary? I don't have AOSP source on my Ubuntu installation right now, so if you don't mind sharing I wouldn't need to build it from sources...
Click to expand...
Click to collapse
I attached the applypatch binary- it should work on most x86 linux distros, maybe.
As I stated in my earlier post, the binary is built with the x86 build. I'm not so good with the build system, but I think the code is from the bootable/recovery/applypatch. I included the relevant notices for that code.
In the OTA updater-script, there will be a bunch of applypatch commands- you might have to change the name/permissions of the binary, but that syntax should be what you are looking for.
Enjoy!
Thank you very much for your time. It seems we are almost the only one trying this. The thing is that I really need to get this done because OTA update is failing for me by running it the usual way.
I've tried your binary and it works under my Ubuntu installation, however I was trying to manually patch just one file and got this:
Code:
[email protected]:~/Desktop/system/app$ applypatch_static ./ApplicationsProvider.apk - d9023cd58fd055e1ca3c8f8492b2c36aba923c6a 6184 69bea5d5a62980c611b903de8243d980f30e5fb5:../../patch/system/app/ApplicationsProvider.apk.p
applying patch to ./ApplicationsProvider.apk
target 6184 bytes; free space 1080520704 bytes; retry 1; enough 1
chown of "./ApplicationsProvider.apk.patch" failed: Operation not permitted
I got that chown error message although ApplicationsProvider.apk.patch was created. Is this the patched file?
EDIT: Nevermind, I forgot to run the command as superuser.
EDIT2: Would it be too much if I ask you also to share the script that converts OTA commands from updater-script into the correct syntax to run on Ubuntu?
There is no guarantee that this will work on your machine. It's a work in progress and will probably require some editing. This just creates the scripts that extract the commands from the OTA updater-script.
I haven't used it in a while since ASUS hasn't put out any updates lately, but I would copy the updater-script and rename it delete_source. Then I would edit out everything in delete_source except the delete commands. There might be an issue with a trailing ; or similar character. The script runs on the copy of the updater-script to pull out the applypatch commands and the delete_source to pull out the files that need to be deleted from the rom.
Check the syntax, but it will be something like ./tf_file <head of stock rom> <head of patch source>
If you have the donor stock system folder and the patch folder from the OTA at the same dir level, it would be ./tf_file system patch
The script also assumes that the applypatch binary is in your path as apply_patch. You can adjust as needed.
Lastly, my bash skills are probably pretty weak, so if you see ways to improve this script or add error/path checking, feel free to post improvements, etc. There is probably a really slick way to do this with just one script and not a script that makes three other scripts. The historical reason for this is that I used to transfer the files to my transformer and run the script on there. Running in Ubuntu is much faster!
Good luck!
Code:
#!/bin/bash
# extract_source is used to pull files to be patched from source tree
# delete_source is ued to delete unneeded files from the source tree
# tf_file applies the patches to the extracted source files
# don't forget to copy the direct updates in the OTA
# usage is apply_ota
# assumes that updater-script and delete-script are at same directory level
# updater-script is from META-INF folder, delete-script is from updater-script
# add error checking to count number of patches and number of patched files
# parse trailing / in folder names
work_dir=`pwd`
tf_file=$work_dir"/tf_file"
extract_source=$work_dir"/extract_source"
delete_file=$work_dir"/delete_source"
work_file=$work_dir"/work_file"
echo "#!/bin/bash" > $tf_file
sed ':a;N;$!ba;s/\n/\ /g' updater-script | sed -e 's/;/\n/g' -e 's/\ \ */\ /g' | grep "apply_patch(" | sed -e 's/"//g' -e 's/,\ p/\:p/g' -e 's/,//g' -e 's/package_extract_file(/$2\//g' -e 's/(/\ $1/g' -e 's/))//g' >> $tf_file
chmod 755 $tf_file
echo "#!/bin/bash" > $extract_source
#sed ':a;N;$!ba;s/\n/\ /g' updater-script | sed -e 's/;/\n/g' -e 's/\ \ */\ /g' | grep "apply_patch(" | cut -d \" -f 2 |sed -e s'/^/cp\ --parents\ $1/g' -e s'/$/\ \$2"\/\"/g' >> $extract_source
sed ':a;N;$!ba;s/\n/\ /g' updater-script | sed -e 's/;/\n/g' -e 's/\ \ */\ /g' | grep "apply_patch(" | cut -d \" -f 2 |sed -e 's/^/cp\ --parents\ $1/g' -e 's/$/\ \$2"\/\"/g' -e 's:/system::g' >> $extract_source
chmod 755 $extract_source
sed -e 's:",:\n:g' -e 's:\ ::g' delete-script > $delete_file
sort -r $delete_file > $work_file
echo "#!/bin/bash" > $delete_file
grep / $work_file | grep -v /$ | sed -e 's:"/:rm\ \$1/:g' >> $delete_file
grep /$ $work_file | sed -e 's:"/:rmdir\ \$1/:g' >> $delete_file
chmod 755 $delete_file
exit 0
IMGDIFF Tool Needed...
Hello Sir, i am looking for a tool to generate IMGDIFF patches on windows... i already have a BSDIFF tool, but coudn't find IMGDIFF... Thank You
I have no idea about windows, but the source is here.
https://android.googlesource.com/platform/bootable/recovery/+/master
You could install linux and build it from there. Follow these directions http://source.android.com/source/building.html
There is also lots of great info here http://wiki.cyanogenmod.org/w/Development#Learning_To_Build_CM
Good luck!
gee one said:
Good luck!
Click to expand...
Click to collapse
I am very thankful to you for replying me sir... these guides are awesome, but i dont think my slow internet connection and PC with 1GB RAM will be allowing me to download or compile
actully i was making a tool to apply OTA Updates in PC, it is semi functional for now, it is able to read updater script and apply bsdiff patches, but can't handle imgdiff patches and now i am stuck at this point because of no imgdiff port for windows... :crying:
i will be very thankful if you or @bgcngm can compile imgdiff at least for cygwin? else i will have to postpond this project until someone else compile and port imgdiff for win/cygwin
Sorry- I'm clueless about cygwin. You could try dual booting to linux to compile from there.
gee one said:
You could try dual booting to linux to compile from there.
Click to expand...
Click to collapse
Thanks alot for taking your time to reply me sir now it seems like linux is the only option left
as you are a linux user will it be too much if i ask you to find a right guide to compile ota tools only? cause i don't know much about linux, and it will be difficult for me to find right guide to compile ota tools only...
jawad6233 said:
Hello Sir, i am looking for a tool to generate IMGDIFF patches on windows... i already have a BSDIFF tool, but coudn't find IMGDIFF... Thank You
Click to expand...
Click to collapse
Answer is here - [Tool]imgdiff executable for windows. BTW, anybody have applypatch for Win32?
Hoping this works for lollipop.
About Busybox: busybox.net/about.html
More on Busybox: busybox.net
This is just for anyone who wants to try, and especially those without access to a PC.
:
: --------------- BUILD STEPS --------------
:
Things we'll need besides your phone:
- "Android Terminal Emulator" app
- "Complete Linux Installer" app , I also recommend trying "linux deploy" for more advanced usage
- internet/wifi
- latest "busybox" source
1) We need to get Ubuntu or Debian booted for a sufficient build environment (kali linux works well too). I've used them all on Android but I like the better stocked terminal in the Ubuntu images. I used the app "Complete Linux Installer" which is free and works beautifully, very simple and easy. In the app you want to follow the short instructions to download an Ubuntu image, rename it to ubuntu.img, and place it in a folder named ubuntu in /sdcard. Then hit menu in the app and click the terminal screen icon that says "Launch". An Ubuntu terminal will now open in Android Terminal Emulator. Super quick and easy.
2) Let's download some crucial build environment tools.
Code:
apt-get install -y gcc build-essential libncurses5-dev libpam0g-dev libsepol1-dev libselinux1-dev
--EDIT-(30AUG2014)--
For Selinux compatibility and loginutils, we need to also download a few extra packages. Already included in the code above.
3) Now the cool thing about this chroot Ubuntu environment is that we still have access to the sdcard to transfer files between Android environment and the chroot jail. Extract your downloaded busybox source to your Ubuntu home with something like:
Code:
cd
tar -xf /sdcard/Download/busybox*bz2
cd busybox*
4) Now we can build busybox statically. The first thing we do is generate a Makefile by running "make" with a "defconfig" (default configuration file) Usually you will run "./configure" with other programs, but busybox compiles more like a kernel, so it uses a config which has a huge checklist of options.
(After successfully compiling busybox, we can go back and customize the .config; this entails that for each "CONFIG ..." line we see, we can uncomment it and mark it "y" or "n" to configure some option... This can be more easily done from a terminal busybox menu interface with "make menuconfig". You just need to crank font down to 7 or use telnet/ssh)
Skip "make defconfig" if you use a customized ".config" file such as one I've attached.
Code:
make defconfig
If all goes well, we now have a Makefile and are ready to compile:
Code:
make clean && make LDFLAGS=-static
Let "make" crank out the binary for a couple minutes. The extra variable we set after make is to compile statically. When compiling is complete we'll have a few different busybox binaries at the root of the source directory. We use the one named "busybox" since we're not debugging.
5) Now let's copy it to /system/usr/bin to install for test usage.
Code:
cp ./busybox /android/data/media/0
(Open a new terminal tab to get into Android Environment)
mount -o remount,rw /system
mkdir -p /system/usr/bin
cp -f /sdcard/busybox /system/usr/bin
chmod 0555 /system/usr/bin/busybox
/system/usr/bin/busybox --install -s /system/usr/bin
mount -o remount,ro /system
PATH+=:/system/usr/bin
.. and done. Run some scripts and enjoy your static busybox!
:
: Extra steps for SELinux-enabled busybox
:
Here are the extra steps you need to take to compile busybox with SELinux features. Sorry it took so long to get this added to this first post.
First we need to download the source for libselinux and libsepol and compile it. (This is for use with the standard glibc toolchain.)
Code:
cd
apt-get source libselinux libsepol
cd libselinux*
make
cd
cd libsepol*
make
Now that we have those libraries compiled, we can proceed to the busybox compilation.
Code:
cd
cd busybox*
make clean && make LDFLAGS='-static -L ../libselinux*/src -L ../libsepol*/src' CFLAGS='-Os -I ../libselinux*/include -I ../libsepol*/include'
That's pretty much it. It initially seems more complicated than it actually is, but all we're really doing is including the libraries for libselinux and libsepol into the busybox compilation.
edit:
**Commands to run if you have compile errors:
Code:
apt-get build-dep busybox
apt-get install -y build-essential
apt-get -f update
dpkg --configure -a
:
: --------------- DOWNLOADS --------------
:
***** Attached are flash installers for busybox (v1.23.1 stable, non-SELinux, 374 applets included!, ~1.1MB size) or busybox (v1.23.1 stable, SELinux, 386 applets included!, ~1.6MB size) *****
Since it's up-to-date it has some nice extras most people haven't seen like a "-I" option for xargs! Yes, that is correct, busybox xargs has its testicles back.
Code:
e.g.
$ echo Hello | xargs -I{} echo {} world!
> Hello world!
: ---------- UPDATES ----------
-------------------EDIT-2-(30AUG2014)----------------------
Got a Selinux-enabled busybox attached now. This means Selinux flags are integrated into applets like ls, id, ps, etc, and there are now 12 extra Selinux applets to give a total of 386 applets, ~1.6MB in size. The previous one is more portable, but this one can completely replace toolbox and gives you Selinux control for Android 4.4+. Plus it's pure maxed-out awesomeness.
***I've also attached the .config files for each busybox I've compiled, so anybody can remake them (from their phone!) as I've posted. You just need to download and extract the .config file to the root of your busybox source directory and type "make".***
-------------------EDIT-3----------------------
YashdSaraf has made some very useful flash zips to install either the non-selinux- or selinux-enabled busybox 1.23.1 via recovery. Installation replaces the stock busybox in /system/xbin. I've attached the zips he made to the end of this OP.
(**Note: Thought I'd mention that there will be a handful of applets that don't work in "Android" environment such as su(don't worry this isn't linked with the installer) Part of this is because of the way Android's default file structure is an amputated modified version of linux. With almost all of them, slight modifications to environment and file structure can be made to get them to work. This is just normal behaviour of busybox in android. The su and init applets shouldn't be used in Android though. I keep them compiled into the binary anyway for completeness of the build and because they work and are needed for a root.gz initrd or some chroot environments. It also doesn't hurt keeping them compiled in. You just have to remember not to link them when installing busybox.
-------------------EDIT-4-(06SEPT2014)----------------------
:
: How to compile against(using) uclibc for a smaller binary!!
:
Download the attached arm-linux-uclibcgnueabi toolchain package that I pre-compiled. Extract to /data/media:
Code:
cd /data/media
zip='/sdcard/Download/2014-09-06__arm-buildroot-linux-uclibcgnueabi.tar.lz.zip'
unzip -op "$zip" | lzip -d | tar -xf -
Then let's open up the "Complete Linux Installer" or "Linux Deploy" terminal.
To use the toolchain with a busybox build, we just need to specify the parameter CROSS_COMPILE which is a prefix to the compiler tools. So if you extracted the toolchain to /data/media, you will use:
Code:
make clean && make LDFLAGS=-static CROSS_COMPILE=/android/data/media/arm-buildroot-linux-uclibcgnueabi/bin/arm-buildroot-linux-uclibcgnueabi-
When you're done you should have a busybox binary with 374 functions with size around 1.1MB. That's a 20% decrease in size from using the standard glibc toolchain!
**IMPORTANT Notes
- The toolchain can't be used with lollipop since it's not compiled with -fPIC. I'll fix this later. Busybox is fine since it's static, it's just the toolchain I uploaded.
- Selinux-enabled busybox .config errors out when building using the uclibc toolchain; I think this is a lack of support issue. In the "Complete Linux Installer" app you'll need to add the mount "/data/media" in options. This gives you access to it as "/android/data/media", very very useful for extra space needs.
Difference between SELinux and non-SELinux busybox
The SELinux (NSA security enhanced Linux) binary comes with the following extra utilities: chcon, getenforce, getsebool, load_policy, matchpathcon, restorecon, runcon, selinuxenabled, setenforce, setfiles, setsebool, and sestatus. There are also some selinux flags enabled for applets such as "ps" and "ls", e.g. "ps -Z" and "ls -Z" to show the context for processes or files. If you are using Android 4.3 or newer, then you probably want to use the SELinux-enabled busybox since Android 4.3 is when SELinux was introduced to Android. Using the SELinux busybox on older version of Android without SELinux file structure should probably work besides the SELinux applets, but I haven't tested this. The non-SELinux binary can be used on any version of Android. When it comes down to it, the system actually uses "/system/bin/toolbox" SELinux applets for SELinux operations, so unless you specifically want to use busybox's SELinux tools for personal use, the safest option is to go with the non-SELinux busybox. I use Android 4.3.1 and 5.x, so I use busybox's better featured SELinux tools.
Latest updates see post 2
Busybox 1.23.1 (2015-02-06) below
Busybox compilation on Linux
reserved
Great Info here!
But I would be interested to know how well this method works on Samsung Stock devices running AOS 4.2 and above? Any experience?
Awesome info, this thread came up #1 while googling busybox 1.23
I made a flashable zip of the attached binary in the op to clean the old one(if any) and install the new busybox in xbin, just in case if anyone needs it. :good:
Is it work on xperia sp on 4.3 fw yes ?
YashdSaraf said:
Awesome info, this thread came up #1 while googling busybox 1.23
I made a flashable zip of the attached binary in the op to clean the old one(if any) and install the new busybox in xbin, just in case if anyone needs it. :good:
Click to expand...
Click to collapse
Thanx,worx fine with Carbon 4.4.4 on my LG.
GREETZ FROM TEAM-OPTIMA!!!
E:V:A said:
Great Info here!
But I would be interested to know how well this method works on Samsung Stock devices running AOS 4.2 and above? Any experience?
Click to expand...
Click to collapse
Thanks man. I've been compiling tons of stuff with Debian and Ubuntu chroot no problem on top of 4.3.1 Vanir and also 4.4.4 Carbon, both are my daily drivers. "Complete Linux Installer" is pretty fast compared to some other chroot apps like GNUroot (no offense to GNUroot, it works but is way too slow). It runs real-time compared to non-chroot. When compared to my dual-core 2007 Pentium M laptop, it's about 2-3 times as slow which isn't too bad for compiling something like mksh or even busybox which takes up to 5 mins I'd say.
In terms of binary size, compiling natively is better than cross-compiling it seems. I used gcc with no size optimizations here, so 1.37MB is pretty nice compared to some others around 2MB with full configs. With this method and klcc (gcc wrapper) I got mksh compiled to 192KB. I'm currently trying to build a uclibc toolchain on my laptop that will give me a mksh binary under 300KB..
YashdSaraf said:
Awesome info, this thread came up #1 while googling busybox 1.23
I made a flashable zip of the attached binary in the op to clean the old one(if any) and install the new busybox in xbin, just in case if anyone needs it. :good:
Click to expand...
Click to collapse
Cool thanks man! That is really useful, glad to hear from CALIBAN that it works. Could I add this to the OP with credit to you?
Hamidreza2010 said:
Is it work on xperia sp on 4.3 fw yes ?
Click to expand...
Click to collapse
Yes, xperia sp uses armv7 so you should be good to go.
7175 said:
Cool thanks man! That is really useful, glad to hear from CALIBAN that it works. Could I add this to the OP with credit to you?
Click to expand...
Click to collapse
Sure bro go ahead
Edit: Went ahead and made one for selinux enabled busybox :silly: , you can add this one in the op too.
Hey guys I was able to get an entire uClibc toolchain built the other day (using buildroot). I tested it and it makes some nice small binaries with about 20%+ smaller size than the standard glibc. Man that took hours to compile but was well worth it. It really put the stability of Android OS to the test as well. Kitkat absolutely couldn't finish compiling with multiple oom's and oops's, but Vanir 4.3.1 stuck it out real nice. Once I had the huge amount of required buildroot packages downloaded, I was able to compile in TWRP as well with good stability. (I have the "Complete Linux Installer" startup chroot script aliased in my mkshrc so I can pull up an ubuntu terminal without starting the app. )
So I got 3 new attachments to OP:
- arm-linux-uclibc toolchain for anyone who wants to compile stuff with it (host=arm AND target=arm)
- busybox (374 fcns, non-selinux) 1116KB
- lzip binary (in case you need it to unzip the toolchain, which is highly compressed from 64MB to 14MB with lzip's lzma)
**As I mentioned in the updated OP, I wasn't able to get selinux-enabled busybox compiled with uclibc. This may be something unsupported, or there may be a patch fix. I'll check it out. I'll try with musl libc and musl-gcc as well.
I have another approach, I try aboriginal cross compiler toolchain in archLinux it produced small binary, but I can't compile busybox for android. For Linux it work. Maybe need bionic lib?
ndrancs said:
I have another approach, I try aboriginal cross compiler toolchain in archLinux it produced small binary, but I can't compile busybox for android. For Linux it work. Maybe need bionic lib?
Click to expand...
Click to collapse
Sounds interesting. I honestly haven't given this a try yet, but I'm very interested in taking a look at it. At this point I'm pretty much addicted to making the smallest binaries I can and testing out different toolchains. I'll give it a good search on duckduckgo, and if you have any insightful links that would be great.
Edit: Alright cool I found the source for Aboriginal Linux at landley.net/aboriginal and am building on Android now. I'm also trying this on my x86_64 laptop so that I can compare the differences like I have with glibc, uclibc, musl, klibc binary builds in a native environment and a cross-compile environment.
I see from my laptop's build that a busybox was generated, but it was dynamic and has a libc.so.6 dependency. @ndrancs : this might be what you were talking about. Did you try compiling static? Also see if "make allnoconfig && make clean && make LDFLAGS=-static" works for compiling busybox with Aboriginal Linux.
7175 said:
Edit: Alright cool I found the source for Aboriginal Linux at landley.net/aboriginal and am building on Android now. I'm also trying this on my x86_64 laptop so that I can compare the differences like I have with glibc, uclibc, musl, klibc binary builds in a native environment and a cross-compile environment.
I see from my laptop's build that a busybox was generated, but it was dynamic and has a libc.so.6 dependency. @ndrancs : this might be what you were talking about. Did you try compiling static? Also see if "make allnoconfig && make clean && make LDFLAGS=-static" works for compiling busybox with Aboriginal Linux.
Click to expand...
Click to collapse
I preferred to use uclibc I think it easy to setup and produced small binary.. Aboriginal cross-compiler use uclibc as default. Btw I don't use cmd : LDFLAGS=-static instead I set it in .config.. Maybe I try this later..
ndrancs said:
I preferred to use uclibc mk it easy to setup and produced small binary.. Aboriginal cross-compiler use uclibc as default. Btw I don't use cmd : LDFLAGS=-static instead I set it in .config.. Maybe I try this later..
Click to expand...
Click to collapse
Ok yeah I like how aboriginal set up with uclibc, and it has scripts for each build stage, so you can stop at the toolchain. I'll be interested to see their future releases with the musl libc as well.
Also for anyone interested, I figured out how to run dynamic binaries in android:
- make the directories "/lib" and "/lib/arm-linux-gnueabihf"
Code:
mkdir -p /lib/arm-linux-gnueabihf
- copy the linker "ld-linux-armhf.so.3" to "/lib"
- find a specific binary's dependencies: e.g. for dynamic mksh do:
Code:
strings mksh | grep \\.so
- copy the listed libs to "/lib/arm-linux-gnueabihf": e.g. for mksh that would be libc.so.6. The libs/linker you copy over will come from the mounted ubuntu/debian/... image you have mounted like with "Complete Linux Installer".
- adjust your LD_LIBRARY_PATH:
Code:
LD_LIBRARY_PATH=/lib:/lib/arm-linux-gnueabihf:$LD_LIBRARY_PATH
Any plan to update the busybox to current version. Thanks.
@7175 can you update flashable zip to 1.23.0 stable ?
@ndrancs @exodius48 : Thanks for notifying me guys, I needed to get around to updating to 1.23.0 stable. I updated the original post with no-edify installers for busybox 1.23.0 stable. There's a non-SELinux uclibc compiled version and a full 386-applet SELinux glibc compiled version. They're included in this post too for ease.
7175 said:
@ndrancs @exodius48 : Thanks for notifying me guys, I needed to get around to updating to 1.23.0 stable. I updated the original post with no-edify installers for busybox 1.23.0 stable. There's a non-SELinux uclibc compiled version and a full 386-applet SELinux glibc compiled version. They're included in this post too for ease.
Click to expand...
Click to collapse
Great..been waiting for this release.. :good:
Btw, can i use busybox_full_selinux.zip on android 4.2.2 MIUI rom?
exodius48 said:
Great..been waiting for this release.. :good:
Btw, can i use busybox_full_selinux.zip on android 4.2.2 MIUI rom?
Click to expand...
Click to collapse
Yeah that should work just fine. I'm pretty sure any SELinux tools or applet flags should work since 4.2 introduced SELinux to its filesystem. Let me know if there are any issues.
7175 said:
Yeah that should work just fine. I'm pretty sure any SELinux tools or applet flags should work since 4.2 introduced SELinux to its filesystem. Let me know if there are any issues.
Click to expand...
Click to collapse
Great release..busybox_full_selinux.zip working fine so far on MIUI rom V5 android 4.2.2. :victory:
7175 said:
@ndrancs @exodius48 : Thanks for notifying me guys, I needed to get around to updating to 1.23.0 stable. I updated the original post with no-edify installers for busybox 1.23.0 stable. There's a non-SELinux uclibc compiled version and a full 386-applet SELinux glibc compiled version. They're included in this post too for ease.
Click to expand...
Click to collapse
Hey @7175
Great guide. I am able to compile just fine on my device using your guide. However, is there any way to compile the selinux applets support using a Linux PC (or NDK)? I am not able to find a selinux supported toolchain. May be you can help.
(Check out https://www.python.org for information on what python is.)
Yes you read that correctly, static python for Android! It took a while to figure out how to get this compiled, but I finally did it. I present you with an installer and a little bit of testing. The second post contains information on compilation and the sources. Btw, any cool python scripts are welcomed!
INSTALLATION
Recovery flash installers are attached that will install python 2.7.9 and/or 3.4.2 to /system/pythonX.X.X. You can install both if you want. Installation size is about 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
After installation, the python director(ies) in /system will contain the static python binary, some scripts, and a bunch of modules and documents. Separate scripted executables will be installed to /system/bin/python or /system/bin/python3 depending on which one is installed. These basically just set the PYTHONHOME environment variable and execute the python binary.
TESTING
When you boot up to Android after installation, you should be able to just open up a terminal and fire off some python commands.
Test using python 2.7.9:
Code:
python -c 'print "Hello World!"'
Test using python 3.4.2:
Code:
python3 -c 'print("Hello World!")'
You can also write scripts shelled with python:
Code:
#!/system/bin/python
print "Hello World!"
Make sure to set them as executable with "chmod +x".
Adjust your screen brightness via /sys (root required, tested on Galaxy Nexus):
Code:
python -c 'f=open("/sys/devices/omapdss/display0/backlight/s6e8aa0/brightness","w"); f.write("40"); f.close()'
A cool script I made to tweak file system I/O like rq_affinity, rotational, etc for I/O blocks:
Code:
#!/system/bin/python
import os,re,sys
list=[]
# find all directories containing rq_affinity
for roots, dirs, files in os.walk('/sys'):
for file in files:
match=re.search(r'\S+/rq_affinity',os.path.join(roots,file))
if match:
list.append(match.group().replace('rq_affinity',''))
# write specific values to files in each directory found before
for dir in list:
for name in 'rq_affinity', 'rotational', 'read_ahead_kb', 'nr_requests', 'iostats', 'nomerges', 'add_random':
try:
f=open(dir+name,'w')
if name is 'rq_affinity': f.write('1')
elif name is 'read_ahead_kb': f.write('512')
elif name is 'nr_requests': f.write('512')
else: f.write('0')
f.close()
except IOError:
sys.stderr.write('Problem writing to ' + dir+name + '\n')
Or execute "python" or "python3" without any parameters to open up the interpreter and go from there. When you start for example "python" (python 2.7.8), you should see something like:
Code:
Python 2.7.8 (default, Dec 2 2014, 05:15:18)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
***I haven't fully tested this yet, so let me know how it goes. I know basic commands work, but there are still some complexities in the compilation that may need to be figured out.***
(update) -- Install Extra Packages/Modules via "easy_install" -- (root required)
1) In a shell, remount /system read-writable with:
Code:
mount -o remount,rw /system
2) Change directory to /system/pythonX.X.X, i.e.,
Code:
cd /system/pythonX.X.X
3) Execute easy_install followed by a package name, i.e.,
Code:
./easy_install [i]package_name[/i]
4) Let it download and install! Then test it out.
5) Remount /system read-only with:
Code:
mount -o remount,ro /system
Also, to remove a package, execute:
Code:
./easy_install -m [i]package_name[/i]
rm -r ../lib/pythonX.X/site-packages/[i]package_name[/i]*.egg
*** Flashing the resolv.conf patch may be required to make an internet connection to download modules.
Also, a few things need to be done to get easy_install working:
1) For python 2.7.8 or 2.7.9 you need to link python2.7 to python like this:
Code:
ln -s /system/python2.7.9/bin/python2.7 /system/python2.7.9/bin/python
I'll make sure this is automatically done in future installs.
2) SSL certificates need to be added to "/etc/pki/tls/certs/" to use SSL with easy_install.
Code:
mkdir -p /etc/pki/tls/certs
curl http://curl.haxx.se/ca/cacert.pem -o /etc/pki/tls/certs/ca-bundle.crt
ISSUES
The python installations have the following modules compiled in with external dependencies such as libraries not included at this point:
Code:
--------------PYTHON 2.7.8-2.7.9----------------------
_hashlib _multiprocessing _ssl
_testcapi bz2 crypt
dbm
Code:
--------------PYTHON 3.4.2-----------------------------
_crypt _dbm _decimal
_hashlib _multiprocessing _ssl
_testcapi
I will look into this more.
As @bubbleguuum points out, name resolution with python 3.4.2's urllib wasn't working. I found a workaround by adding the line
Code:
options single-request-reopen
to /system/etc/resolv.conf. This still needs some testing but seemed to work for me. I've included a recovery flash zip to patch /system/etc/resolv.conf if the line is not there since this is a root operation.
UPDATES
* 2014-12-02 * Compiled in more modules for each installation. This makes the installation a bit bigger, but it's worth it. You get a more complete python! To get stuff like help functions and math and readline modules, you need the larger installations linked below under "DOWNLOADS".
* 2014-12-03 * Thanks @cybojenix for pointing out some incapatibility issues with the original edify installer I was using. I have now updated the installers to use SuperSU's great non-edify sh-script installer.
* 2014-12-05 * Some updates to the installers to include more modules, plus size reduction for python 3.4.2. Modules added to python 2.7.8 : _bsddb _ctypes, _ctypes_test, _hotshot, _json, _lsprof, _sqlite3, future_builtins, and ossaudiodev, easy_install, pip(broken for now). Modules added to python 3.4.2: _bz2, _ctypes, _ctypes_test, _gdbm, _json, _lsprof, _opcode, _sqlite3, _testbuffer, _testimportmultiple, ossaudiodev, xxlimited
* 2014-12-05 * Some big updates to the installer to include almost the rest of the modules that wouldn't compile. Also easy_install is working on my end with these new experimental builds. With a little effort, should be able to get pip working as well. I had to use some hacky flags to get it to compile, ignore unresolved-symbols and such, so it definitely needs some testing. Another thing I did was tweak the install script to write over previous installations from these installations. I recommend downloading the experimental builds if you read this. The worst thing that might happen is one of the new modules I've included won't work completely.
* 2014-12-14 * Added python 2.7.9. Trimmed the installation sizes down to the much more reasonable 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
* 2014-12-24 * Attached a recovery flash.zip to patch resolv.conf to try and fix a urllib name resolution error.
* 2014-12-26 * A couple updates: Tweaked python installations so that modules looks for "/system/bin/sh" instead of "/bin/sh" when needed. Also a few "#!" corrections for scripts in the python bin directory. This shouldn't break anything from what I've tested, only make Android python more capable, but I'll keep the old installations attached for now just in case since the modifications involved using "sed". The second update is nice. I've added a little section on installing extra modules with easy_install(included and working with all installations).
Compiling
The environment I compiled in was a debian "wheezy" image mounted as a loop device on Android. The reason I compile this way is so I have full access to the arm environment tools, gcc compiler, etc. without having to go through the trouble of using a cross-compiler (which doesn't work in this case from my testing with uClibc, or eglibc. Code sourcery's might work but I didn't feel like booting an x86 linux installation to try). You can recreate the same environment pretty easily by using either Complete Linux Installer or Linux Deploy apps. I use both.
To get a static python compiled, I first downloaded the following packages with APT:
Code:
[i]apt-get build-dep python python3[/i]
build-essential gcc make # building tools
binutils-gold # awesome gold linker
zlib1g-dev # zlib
libreadline-dev, libncurses5-dev # readline, curses
libbz2-dev # bz2
libsqlite3-dev # sqlite3
python-bsddb3, python3-bsddb3
libgdbm-dev # gdb
libssl-dev # ssl
python-tk, python3-tk # tkinter
libdb-dev # db
python-gdbm python-bsddb3
libffi-dev # _ctypes
tcl8.6-dev # tkinter
libx11-dev # tkinter
libmpdec-dev # decimal
Then I manually installed binutils-gold by unpacking the rpm for armv7h (this isn't available with APT yet). You can get it here though. (Update: this should be available in the debian repositories now.)
For static compilation, you need to make some changes to Modules/Setup from the python source directory. Specifically, you need to add
Code:
*static*
to the top of the file. Then you need to uncomment any commented modules you would like compiled in. When you compile and see a list of failed modules, search for each of these in Modules/Setup and uncomment them. For example, change
Code:
#math mathmodule.c _math.c
to
Code:
math mathmodule.c _math.c
. Then recompile. There may be some shared dependencies and packages as well you need to figure out to get them to compile. Take a look at the attached "Setup" files for more detail.
The actually compile steps look like this:
(1)
Code:
./configure --build=arm --prefix="$PWD"/out LDFLAGS="-static -static-libgcc -Wl,--unresolved-symbols=ignore-all -Wl,--export-dynamic" CPPFLAGS=-static CXXFLAGS=-static CFLAGS="-Os -static" LDFLAGS=-static LD=ld.gold
(2) Modify the Modules/Setup file.
(3)
Code:
make clean; make install
After compiling, there are some things you can do to make your installation in "./out" smaller. Strip the large binaries in "out/bin", i.e.
Code:
strip -s ./out/bin/python2.7
Also remove the python archive library, *.o's, etc. with:
Code:
make clean
This might seem kind of weird, but it works and cleans out unnecessary files from your installation.
**Make sure to backup your Modules/Setup file if you run "make distclean". I lost my 3.4.2 Modules/Setup file after doing this. :silly: Now I gotta recreate it, dammit.
**I should also mention that compile time is very fast given the amount of data generated. It takes about 10 minutes to get python and all it's modules compiled on my Galaxy Nexus.**
:EDITS:
: Updated some configure parameters. Removed unnecessary, extra "./configure". Also added some updates to the included "Setup" files.
: Hacky update to "./conifgure" to ignore unresolved symbols, export dynamic. This allows certain modules to be compiled which have shared dependencies.
: Update to trimming down the installation size after "make". Use another "make clean".
Downloads - Extract in the source directory
Setup (python 2.7.8)
Setup (python 3.4.2)
Setup (Experimental) (python 2.7.8)
Setup (Experimental) (python 3.4.2)
Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference
cybojenix said:
Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference
Click to expand...
Click to collapse
Thanks for your reply! I've changed the installer to use SuperSU's no-edify sh-scripted updater. I tested on my device, but let me know if there are any issues though. I kept it pretty basic.
I added some stuff on the source modifications and compilation to the 2nd post. The only file I've actually modified so far in the source is the generated Modules/Setup file, and I've included the one I used for python 2.7.8. Unfortunately, I over-cleaned the python 3.4.2 directory, which deleted my modified Modules/Setup there, so I'll have to add it later when I re-edit it.
That's a great idea on pip/easy_install. I did download the pip_installer and tried out installing on the static python I made. Got some errors though pointing to a few modules I haven't gotten compiled into the static python installation (listed in the OP near the bottom, specifically _ctypes). I'll see what I can do about that. The line for compiling the module might just need to be added to Modules/Setup. Hopefully that's the case.
@7175
Thank you very much for these binaries.
There is however a problem: name resolution (DNS) doesn't seem to work at all (with both python 2 and 3 downloads).
All attempts to use urllib.request.urlopen('http://somehost.com') fail with "<urlopen error [Errno -2] Name or service not known>" (running python as root but it doesn't matter, and a rooted Nexus 4 running 4.4.4 stock ROM):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 455, in open
response = self._open(req, data)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 473, in _open
'_open', req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1202, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1176, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
Click to expand...
Click to collapse
it works if replacing somehost.com by its ip address.
To reproduce above stack trace with python 3:
import urllib.request
urllib.request.urlopen('http://www.google.com')
while this work:
urllib.request.urlopen('http://173.194.45.229')
nslookup and general name resolution work anywhere else:
[email protected]:/ # nslookup www.google.com
nslookup wwwgoogle.com
Server: 8.8.4.4
Address 1: 8.8.4.4 google-public-dns-b.google.com
Name: wwwgoogle.com
Address 1: 2a00:1450:4007:80c::1014 par03s15-in-x14.1e100.net
Address 2: 173.194.45.242 par03s15-in-f18.1e100.net
Address 3: 173.194.45.240 par03s15-in-f16.1e100.net
Address 4: 173.194.45.244 par03s15-in-f20.1e100.net
Address 5: 173.194.45.241 par03s15-in-f17.1e100.net
Address 6: 173.194.45.243 par03s15-in-f19.1e100.net
Click to expand...
Click to collapse
This issue has probably something to do with how python was compiled to do name resolution. I remember vaguely a similar issue
on Ubuntu 12.04, with a ffmpeg compile that would always fail to resolve hostnames in URLs. No sure what the exact fix was but it had something to do with the libc and its name resolution mechanism. Or maybe something ipv6 related ?
@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
EDIT: Hmm, looks like re-compiling with "--disable-ipv6" didn't fix the problem. Will have to look more into this issue and hold off on uploading ipv4-only python3 since it didn't fix it. In the meantime, I've attached a recovery flash.zip to the OP to add the above mentioned line to /system/etc/resolv.conf if needed, which seems to fix the issue for me.
7175 said:
@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
Click to expand...
Click to collapse
Thank you for looking into this so fast.
My intended usage is for running the python binary from a regular non-root app (which works great!), so editing resolv.conf is not an option.
I've been researching that DNS resolving issue and could not find a clear explanation on why it fails on Android and not on other systems, and why exactly the added line in resolv.conf fixes (or rather workaround) it.
All seems to point to socket.getaddrinfo() failing for an unknown reason.
Is Python compiled with HAVE_GETADDRINFO defined ? If that's the case it uses the libc definition of getaddrinfo, otherwise
it uses an emulation function found in getaddrinfo.c.
My guess is that it is the latter, and what is causing this issue on Android.
It's probable this issue has already been solved on Android since other Python binaries exists, but Googling around do not give
much answer.
I get this error when trying to install anything with easy_install (using /system/python3.4.2/bin/easy_install-3.4 because /system/python3.4.2/easy_install doesn't seem to exist like in the OP):
Code:
# ./easy_install-3.4 feedparser
Searching for feedparser
Reading https://pypi.python.org/simple/feedparser/
Download error on https://pypi.python.org/simple/feedparser/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'feedparser' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or download links found for feedparser
error: Could not find suitable distribution for Requirement.parse('feedparser')
i got curl and the certs installed in /etc/pki/tls/certs/ as per instrucions in the OP, any help?
Thoughts on PIL?
Hmm... new problems here. Android Lollipop have no /system/etc/resolv.conf by-default and urllib patch is no more works.
How it can be fixed?
http://qpython.net/index.php
Where I can find source code and related documents for project
Pratik Raj said:
Where I can find source code and related documents for project
Click to expand...
Click to collapse
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?
hackintosh5 said:
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?
Click to expand...
Click to collapse
Idk about OP, but I can surely post my source for python 2.7 on arm/arm64. I compiled it like a year ago, but it wasn't too bad, just required a bunch of patches and code for dlopen
I just want to say thank you so much.
I wrote this guide to make it easy for everyone to be able to cross compile c/c++ for android, the easy way that I know. I admit, I am a total n00b to c/c++. I started learning to cross compile when I got my new nexus 9 for christmas. I, for some reason, couldnt get busybox installed on it. So, I had to manually compile and load it on there. After reading a hundred tutorials and none of them working, I got frustrated. Finally, I compiled a working version. And it was SO EASY. Anyway, Im specifically giving instructions for Ubuntu 14.04/14.10. First thing you need to do is download a package called binutils-arm-linux-gnueabi.
- sudo apt-get install binutils-arm-linux-gnueabi
- sudo apt-get install binutils-aarch64-linux-gnu (for aarch64, or arm 64bit, or armv8)
- sudo apt-get install binutils-arm-linux-gnueabihf (for armhf)
To compile busybox for your platform, you must download the source package you want to compile from http://busybox.net/downloads/. Then extract it
- tar xvf busybox.tar.gz
Then, cd into your root busybox folder.
- export ARCH=arm
- export CROSS_COMPILE=arm-linux-gnueabi- (dont forget the trailing dash(-))
This will setup your variables for compilation. Then
- make menuconfig (if you dont get a config menu, you need to "sudo apt-get install libncurses5-dev")
Now, go into the busybox settings -> Build options. Select the option "build busybox as static executable". You can select/deselect options by hitting the space bar. [ESC] key will take you back. Make sure to save your new configuration. Its almost time to compile! One last thing. If you want to add your name to it, go into the Makefile in the root busybox folder. At the top of the file, where it says EXTRAVERSION = .git, you can add something like -bynarie or whatever you want. That way when you run busybox on the terminal emulator on android, it will print out something like "BusyBox v1.24.0-bynarie". OK! Time to compile. Really simple:
- make install
This will compile every applet, and put everything in a folder called _Install. You will have busybox and busybox_unstripped in the root directory. Please, be aware that this busybox puts an applet named "su" in the bin folder in the _install folder. So, DO NOT COPY THE SU APPLET INTO YOUR ANDROID BIN FOLDER, OR YOU WILL LOSE ROOT!!! Now, to verify it compiled to the right architecture, do "file busybox" at the term and it should spit out something like "ELF 32bit ARM executable". If this is the case, congrats. We have successfully compiled busybox for android arm!! Copy the needed files over to your device and set permissions and you are done. The suggested way to move the busybox binary to your device is as follows:
- adb push busybox /data/local/tmp/busybox
- open adb shell and do "chmod 755 /data/local/tmp/busybox" or chmod it on your linux box before pushing.
- open file manager on device and move busybox to /system/xbin
- If properly done, should work.
For other nix programs like tar, the procedure is as follows:
- Open term, cd into root source folder
- ./configure CC="arm-linux-gnueabi-gcc" CPP="arm-linux-gnueabi-cpp" --host=arm-linux-gnueabi
- make
The CC variable is your C compiler command and the CPP variable is the C preprocessor, if you need to add a C++ compiler just add CXX="arm-linux-gnueabi-g++" but TAR is specifically C only. You can add all three variables I would assume to be safe. Most of the time, the readme or install documents will give you some guidance on cross compiling. But, this is how I successfully compiled TAR for arm and aarch64.
If you want to compile small/single source file c/c++, you will be using the same tools, but in a different way. Cd into your source file directory, and depending on which type of source it is (c or cpp), you will do the following:
- arm-linux-gnueabi-gcc helloWorld.c -static -o helloworld.out (for C)
- arm-linux-gnueabi-g++ helloWorld.cpp -static -o helloworld.out (C++)
Yep, its that easy. I cant guarantee 100% this will work on everything, but its a good starting point. And you dont even have to fool with the Android NDK. I find that the android ndk would mostly benefit "Apps" that need to integrate c/c++ code into them, not little console apps run from the terminal emulator. I hope someone finds this guide useful and if anyone has any problems trying to get somethin to work, you can respond and I will do my best to help.
thanks
# FYT-Firmware-Workbench
PLEASE DO NOT USE WITH T'EYES CC3 FIRMWARE
Scripts intended to mount the FYT UIS7862 firmware, make modifications and reassemble it back
They can retrieve the firmware from specified location and after modification, they can put the packed and signed firmware into another specified location
Files:
1-) config.ini = A file to setup parameters, paths for the original firmware and patched firmware, temp directory to work with firmware files and mount point to edit the firmware
2-) import_original.sh = Retrieves the ZIP file from the location specified in config.ini
3-) import_patched.sh = Retrieves the same files that 'import_original.sh' but from the patched location specified in the config.ini (normally a firmware that you've already patched before)
4-) unpack.sh = (autorun after import*) Unpack the imported firmware and create the mount moints (as specified in the config.ini file) to work with the firmware
5-) tweaker.sh = Allows change Locale, Language, Country and allows enabling ADB and OTG (if hardware supports it)
6-) inject_mods = Copies all contents of "_mods" folder into system partition, like gps.conf or even bootanimation, up to you to creat the same folder structure with your own files to inject
7-) repack.sh = Unmount the modified firmware from the mount points, delete the mount points, assembly the new ZIP file, sign it and move it to the patched directory as specified in config.ini
clean_local.sh = This script will unmount the firmware and delete it, it will also delete temporary files. as its name says, it cleans the local workbench !
N'joy it !
[CHANGELOG]
2.0 Introduces a different ZIP signature method
1.0 Initial release
Did you already try to flash it? I seed that you sign with the test keys but does that now work on the 7862?
On the previous architectures the recovery did not allow self signed zips.
Yes it works well, of course I flashed a modified 6315_1 befor make this post. You can trust the scripts, take a look at the tweaker.sh who allow you to modify some settings, Locale,Region, Coutry, and more to come !
Hi,
I tried your tools. Nice work.
I first did the "import_original.sh" and then the "unpack.sh". I did use this on a RPi4 running Debian Linux. All my Linux machines are currently RPi boxes and I use my headless server RPi4 for most stuff I do myself on images. (Next to that I have a Chromebook running Linux as well, but that one doesn't allow me to mount images in the linux sandbox.)
In the unpack.sh I get errors after brotli on the resizing of all the images:
Bash:
**********************************************
Processing 'system' image ...
BROTLI -- Handling system.new.dat.br...
Converting system.new.dat file into system.img file
When tring the "Converting .." is where it goes wrong without error message. The sdat2img is using "#!/usr/bin/env python" which, on an python3 only system gives "/usr/bin/env: ‘python’: No such file or directory"
When I change the call to "python3 sdat2img.py <etcetera>" it works. It might be better to use something like
Code:
if [ `which python` ]; then
./tools/sdat2img.py
elif [ `which python3`]; then
python3 ./tools/sdat2img.py
else
printf "\n\nNo python found!!\n\n"
exit 1
fi
Otherwise your created dat file is 0 bytes (simply an echo of 0 bytes to a new file), then giving corrupt empty images of 128 MB when trying to resize them.
And about the resizing/shrinking of the images: Why do you do that?
Simply leave the img as is and mount it read-write, not read-only.
Please use
Code:
mount -t ext4 -o loop,rw $unzippedPath$i.img $MOUNTPOINT/$i
and then simply copy (as root) your stuff in or make the modifications directly (as root). After unmounting you can immediately repack and brotli-compress it again. No extra intermediate steps necessary.
****
And some other remarks (actually a bit of nit-picking, please don't feel offended. I think you did a great job):
You use and pack a 64bit jdk with it, which is of course very convenient in most cases but doesn't work on my ARM raspberry pi.
I would put something in the shell script like
Bash:
echo "Singning '${FIRMWAREZIPFILENAME}' file..."
if [ -n `which java` ]; then
# Not found. Use packed version and hope it is a 64bit system
tools/jdk64/bin/java -jar tools/SignApkv2.jar <etcetera>
else
# Use system version
java -jar <etcetera>
fi
You also use
Bash:
echo " "
echo "READY !!!."
echo " "
Why not use
Code:
printf "\n\nREADY !!!\n\n"
Don't feel offended at all fellow, that's normal you're improving the scripts so suit your needs, that's normal, and your code (above) seems much more adapted than mine, I packed a 64bit java 8 cause java 11 does not sign the ZIP, (or at least I didn't reach the sign correctly)
Y have a giltab at home with that projects (FYT, TS10 etc....) y you want to participate and pull-request your changes, os help me improving that scripts, you're welcome and I'll send you the link via PM
mariodantas said:
I packed a 64bit java 8 cause java 11 does not sign the ZIP, (or at least I didn't reach the sign correctly)
Click to expand...
Click to collapse
Yes, you are right. the signapkv2 is already very old. I forgot that. I just tested and java V11 comes without the sun packages.
Will add and arm64 java 8 and detect the proc architecture by scripts to select the right java
@surfer63 I forgot to tell you that I released FYT Firmware Workbench 2.0 which includes another 6315_1 signature method (no more signapkV2), (as required with lsec6315update after Jul/2/22) thos signs well the new 6315_1 files, so we can still modify their contents and repack again and flash without the message: "Please use legal system"
I finally got my images to mount. (had to sudo)
I used the modifications @surfer63 suggested. (python3 was the trick here)
edit: are the mountpoints supposed to be RO? I'm getting "no space.." error running the tweaker.
j-5 said:
I finally got my images to mount. (had to sudo)
I used the modifications @surfer63 suggested. (python3 was the trick here)
edit: are the mountpoints supposed to be RO? I'm getting "no space.." error running the tweaker.
Click to expand...
Click to collapse
No, they shoud have 128Mb + each uhntil repack
NOTE: before running FYT Firmware Workbench do a "sudo su" (you must run as ROOT) that's why it didn't work flawlessly, I'll modify the scripts to ask for sudo su if needed !
@mariodantas : Your FYT workbench was still in the Android head units section. I had it moved to the FYT forum.
mariodantas said:
No, they shoud have 128Mb + each uhntil repack
NOTE: before running FYT Firmware Workbench do a "sudo su" (you must run as ROOT) that's why it didn't work flawlessly, I'll modify the scripts to ask for sudo su if needed !
Click to expand...
Click to collapse
OK, got it.
I had commented out the "expand' part, my bad .
Is there a github repo for the latest scripts? I'd love to try this out.
No Github, I'll release some fixes soon !
mariodantas said:
I'll release some fixes soon !
Click to expand...
Click to collapse
this is very good, thanks for your hard work!
I am also having trouble parsing the system.new.dat.br file (I only chose to edit it in config). After changing to python3, file began to be mounted and repack. But I get an error on the device in the middle of installation related to "dynamic_partitions_op_list" even though I didn't change anything
Where are the 2.0 tools?
Andy826 said:
Where are the 2.0 tools?
Click to expand...
Click to collapse
In the OP
surfer63 said:
And about the resizing/shrinking of the images: Why do you do that?
Click to expand...
Click to collapse
@surfer63 I do that to allow me to add content inside, if not, the free size is 0Kb after mount
@mariodantas The CC3 Firmware Tools don't seem to work with the January firmware. The error when using them on the January firmware is:
Code:
ext2fs_open2: Bad magic number in super-block
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open _tmp/_firmware/elable.img
Any chance of this being fixed please?
ahmed123 said:
Any chance of this being fixed please?
Click to expand...
Click to collapse
Not compatible with CC3 anymore