Database Info

[HOW-TO] ROM-HACKING: init.rc ext2-auto-mount / ROM Signing / ROM Kitchen

AS MENTIONED IN THE INTRODUCTION TEXT THIS HAS ONLY BEEN TESTED ON AMON RA ROM 1.6.2 BUT SHOULD REALLY WORK ON ANY ROM THAT HAS NO EXT2 AUTO-MOUNT. AND YEAH THIS WHOLE PROCESS HAS BEEN DONE ON A 32a BOARD. FOR THOSE THAT TRY THIS ON OTHER ROMS LET ME KNOW HOW IT GOES.
I've searched and shuffled through the entire forum and made inquiries to ROM authors without much light being shed on this issue. I doubt I am the only one who has been looking for a way of doing this so I decided to do a small HOW-TO. Here I will explain step by step as to how you can implement a script to be part of your ROM that will auto mount an ext2 partition on boot up if such partition is present. I have included all the tools I've used in order to pull this off, and as the title suggests this has only been done on Amon Ra's latest 1.6.2 ROM. In order to follow these instructions you are expected to allready have set up an adb enviroment on your linux box and for the signing process to work you must have sun-java present, the gnu java wont work. And of course a microSD card with an ext2 partition
1. Download install.sh to your home directory
Code:
wget http://www.grindhouse.no/androidtools/install.sh
chmod a+x install.sh
2. Now execute the install.sh script which will create a directory to work in and download a tool and script package and unpack it.
Code:
./install.sh
When the install.sh script is done you need to move the mkbootimg preferebly to your tools directory of your SDK.
Code:
mv toolstomove/mkbootimg <path/to/sdk/tools/mkbootimg>
3. Unpack the RA1.6.2 ROM into a directory in your home dir. In this HOW-TO we will use directory name "ra1.6.2" as an example through out the entire process.
4. Copy the boot.img from ra1.6.2 to the ROM-cooker dir
Code:
cp $HOME/ra1.6.2/boot.img $HOME/ROM-cooker/boot.img
cd $HOME/ROM-cooker
5. Use unpack.pl to extract the ramdisk from the boot image. I've modified the script a little so it automates the entire process and decompresses the ramdisk to a directory
Code:
./unpack boot.img
6. Now you can either replace the init.rc file here with the one I've included in this package or you can add these lines by yourself. In wich case do the following
Code:
cd boot.img-ramdisk
pico init.rc
Press CTRL+w and then CTRL+t and input 27. hit enter. This will take you to line 27 of init.rc so you can add a line right before the init process remounts the rootfs in read-only mode. Add following line:
Code:
mkdir /sdext2 0771 system system
Now scroll down to the end of the init.rc file and add the following:
Code:
service mountsdext2 /system/bin/mountsd
user root
group root
oneshot
7. You have now edited (or replaced) your init.rc file and prepared it to execute a script on boot that will detect an ext2 partition and boot it if there is one to be found. Now you have to make the mountsd script a part of the ROM. Do the following:
Code:
cd $HOME/ROM-cooker
mv toolstomove/mountsd $HOME/ra1.6.2/system/bin/mountsd
rm -rf toolstomove
8. Now that the init.rc file is sorted out and mountsd has been placed in /system/bin of the ROM so it is time to re-pack the boot.img:
Code:
cd $HOME/ROM-cooker
./repack boot.img-kernel boot.img-ramdisk boot.img
rm $HOME/ra1.6.2/boot.img
mv boot.img $HOME/ra1.6.2/boot.img
9. Your ROM now has a new boot image with an updated init.rc and the /system/bin dir has the script needed to auto-mount the microsd ext2. Now you must re-zip the ROM and sign it. Do the following:
Code:
cd $HOME/ra1.6.2
zip -r update.zip *
mv update.zip $HOME/ROM-cooker/update.zip
cd $HOME/ROM-cooker
./sign.pl update.zip
10. The ROM is now signed and you now have a file called update-signed.zip. Connect the phone to your computer and execute thus:
Code:
./push update-signed.zip
11. Now you are ready to flash the modified ROM which will auto-mount an ext2 partition on your microSD. There is no need to wipe before flashing. If you have no prior experience with ROM flashing or whatever just backup your current install. If you're using OpenHOME or anything similar, nothing will be changed or damaged but if you're using MontAlbert's themes with the ROM you will have to flash them again after flashing this modified ROM.
Code:
adb reboot recovery
12. Flash from choose zip and of course choose update-signed.zip. Reboot. After the system boots up again you can now check whats what with either one of the commands:
Code:
[email protected]:~$ adb shell mount | grep sdext2
/dev/block/mmcblk0p2 on /sdext2 type ext2 (rw,noatime,nodiratime,errors=continue)
[email protected]:~/boot$ adb shell busybox df -h | grep sdext2
/dev/block/mmcblk0p2 893.7M 13.0K 846.0M 0% /sdext2
13. Voila! Your RA 1.6.2 ROM now detects and mounts your microSD ext2 partition on boot. Woohoo?
I hope the HOW-TO was easy reading and that you have succeeded in hacking up your ROM. I know that certain ROMs have this as a built-in function but Amon Ra's does not. But since alot of people including myself use his ROM because of the high speed and stability I thought I should contribute to his project and add a cool (and missed?) function to it.
Mind you that you can use the ROM-cooker set to further adjust and hack up the ROM as you see fit. Happy learning!

Very nice!
Now the question many people will ask : why would you automount ext2 if you don't use apps2sd ?

I personally have ubuntu on my ext2 And besides this approach can be used for a number of things, people who have had the need, or wanted to experiment with init.rc doing things on boot, the mountsd script can easily be altered to do what ever needed.
For me its been a learning curve finding these things out, so by sharing it I may spare some people breaking their backs over this whole init.rc thing. people may want to modify init.rc for whatever reason, so I'm sure people wont have a problem finding a way of putting this to use, and its a subject that isnt all that covered on the forum .. and hey .. at least they get a rom kitchen out of the whole shabang

Very interesting! Thank you.
I used your unpack-program to unpack a recovery-image. It seems to work fine. What I am trying to do is change the state the recovery-image returns the phone to. Would it be possible to just replace your mountsd-script with, for example, a script that installs apps? Or is there a better way to do what Im trying to achieve?
Cheers,
edit: I noticed that on the emulator it is sufficient to just place an apk-file in "data/app" to get it installed. Could it be possible that this is all I need a script to do? :O or could I hurt my poor phone by doing so you think?

sandis84 said:
edit: I noticed that on the emulator it is sufficient to just place an apk-file in "data/app" to get it installed. Could it be possible that this is all I need a script to do? :O or could I hurt my poor phone by doing so you think?
Click to expand...
Click to collapse
That's indeed all you need to do.

Hi!
So I tried to create a signed update.zip, but it failed. It didnt create a "update-script"-file, so my device refused to install it. I wrote my own "update-script"-file, but then it complained "no digest" for the file. How do I solve this?

post the contents of your script people might see whats up

so is this all on linux?
also where are the script files for your tutorial
thanks for the time to put together

sitimber said:
so is this all on linux?
also where are the script files for your tutorial
thanks for the time to put together
Click to expand...
Click to collapse
Says where its at in the first line : )
Code:
wget http://www.grindhouse.no/androidtools/install.sh
But now that I checked, I have to apologize, I see I have a missed payment with my hosting, I'll fix that within the day. Also sorry I havent been answering the few questions here I've been afk cause of surgery.

sitimber said:
post the contents of your script people might see whats up
Click to expand...
Click to collapse
well, I looked in another "update-script" file and found this:
assert compatible_with("0.2") == "true"
assert getprop("ro.product.device") == "dream" || getprop("ro.build.product") == "dream"
show_progress 0.5 0
write_radio_image PACKAGE:radio.img
show_progress 0.5 10
Click to expand...
Click to collapse
So I figured that nothing was essential other then the line "write_radio_image PACKAGE:radio.img". Also ofcourse I made sure it contained the name of my image-file instead of "radio.img". This gave me the "no digest" message, so now I feel unsure on how to create a working update.zip.
edit:
SOLVED! How silly of me. When you sign the update, a hash of each file is put in manifest.mf. Since I added the update-script after signing the file, ofcourse the digest(hash) was missing. Now everything works alot better and I can proceed... until I get stuck again
Cheers,
edit2:
Just to get a better understanding, what exactly does each line do here? Or where can I read about this?
Code:
service mountsdext2 /system/bin/mountsd
user root
group root
oneshot
edit3:
Ok, so I have experimentet, but I still dont manage to solve those last steps. I tried to edit init.rc and just add "mkdir /testdir 0000 system system" where the other directories were created. I then repacked it, zipped it, signed it, put it on my sdcard, started up a custom recovery, installed the update and rebooted. Everything seems to work fine. But when I start adb and check around, I dont see the "testdir"-directory. Also when I check in init.rc my line is gone. Do you guys have an idea of where I went wrong?

sitimber said:
so is this all on linux?
also where are the script files for your tutorial
thanks for the time to put together
Click to expand...
Click to collapse
it doesnot necesarily have to be linux ...you can also do it in windows using cygwin and dsxda's android rom kitchen

Mount Transformer to Linux Mint - automated.

Hey TF owners,
For anyone looking for a way to mount their Transformer to Linux Mint, I wrote a script to automate the process of configuring your machine to allow you to browse/modify files on the tablet's SSD. This will probably also work on Ubuntu, but I have only so far tested it on Linux Mint.
Download the attached file, and run it.
Edit: Connect your Transformer to your PC before running.
Edit 2: You only need to run this script once - not every time you connect the tablet to your PC.
Hit the thanks button/comment if this worked for you. If not, also comment and I/someone will probably help you out. Also, if anyone tests it on Ubuntu successfully, be sure to comment and let people know.
Note: All configuration files are backed up in the script before changes are made. You can always revert safely.
I'll throw in the ever so popular "use at your own risk" for safe measure.
Cheers.
June 29: Updated script uploaded.
-Cleaner
-Removes an unnecessary comment pushed into fstab in the previous script
-Already executable. Just run it. [Thanks MagnusRagnarok for the idea]
-Packed in 7z from Linux Mint this time as opposed to in zip format from the TF, for [hopefully] universal accessibility.
-md5sum 615f2bfa820ee9ac8ea2c967149a2f03 TransformerMount2.7z

ok so maybe a stupid question, but does my tf need to be rooted to try this? also if you can create an automated script to install a linux distro you think it would be possible for you to create a script to root?? btw any idea on what actually works? can the tegra 2 run with compiz??

0lddirtypaki said:
ok so maybe a stupid question, but does my tf need to be rooted to try this? also if you can create an automated script to install a linux distro you think it would be possible for you to create a script to root?? btw any idea on what actually works? can the tegra 2 run with compiz??
Click to expand...
Click to collapse
I thin you've misunderstood the intention of the OP.
It was to be able to mount your Transformer under Linux and be able to read/write the files stored on the Transformer's internal storage. Not to actually install a full blown Linux Distro on your Transformer.

Wow, talk about reading before posting.
Sent from my Transformer TF101 using XDA Premium App

only on linux mint--
does it work on ubuntu?

Yes, this works perfectly on ubuntu

I have a problem extracting the zip file with Ubuntu 10.10
Do I need an alternate unzip program? If so please specify. I do not run MS products.
Error:
7-Zip 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_CA.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
Processing archive: /tmp/TransformerMount.zip
Extracting TransformerMount.bash Unsupported Method
Sub items Errors: 1

I can extract it with 7zip on Windows on the work computer, and with both Explorer and AndroZip on the TF without issue.
Sent from my Milestone using XDA App

Acer Mod of the script
I'm trying to edit the file for use with an Acer Iconia. So far everything works but the modifying of the android.rules file. You see, it doesn't exist on my laptop (mint 11) in the directory the script looks at nor is there an android.rules file in lib/udev/rules.d. While I could probably write a rules file and make it work, I'd like to know where it comes from and maybe discover why it does not exist on my system. Anyone have any ideas? Also - rootblock - if I can get this to work would you object to my posting it in the Iconia forum? Thanks!
UPDATE:
Creating the file worked fine. Any reason to edit the script to check if the file exists and create it if it doe not or am I the exception to the rule? i.e. Would most users have it there in the first place?

3DSammy said:
I have a problem extracting the zip file with Ubuntu 10.10
Do I need an alternate unzip program? If so please specify. I do not run MS products.
Error:
7-Zip 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_CA.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
Processing archive: /tmp/TransformerMount.zip
Extracting TransformerMount.bash Unsupported Method
Sub items Errors: 1
Click to expand...
Click to collapse
Probably a bad download. And you could also try file-roller to open it (I think its installed by default, it is on Debian/Gentoo)

Tortel1210 said:
Probably a bad download
Click to expand...
Click to collapse
I installed the "p7zip" deb, re-downloaded a few times with Firefox and even tried wget. Each time I get the same error result. I was more interested in examining the bash file as I do not have a TF yet. Oh well not a big deal.

3DSammy said:
I have a problem extracting the zip file with Ubuntu 10.10
Do I need an alternate unzip program? If so please specify. I do not run MS products.
Error:
7-Zip 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_CA.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
Processing archive: /tmp/TransformerMount.zip
Extracting TransformerMount.bash Unsupported Method
Sub items Errors: 1
Click to expand...
Click to collapse
Had the same issue with Archive manager - Xarchiver worked, however.

ubuntu & Asus transformer
If anyone having a problem with mtpfs on ubuntu especially version 10.4 ( segfault problem which prevents from mounting the TF), use newer version of mtpfs in the link below.
http://pkgs.org/download/debian-wheezy/debian-main-i386/mtpfs_0.9-3_i386.deb.html.
1. install the package
ubuntu> sudo dpkg -i mtpfs_0.9-3_i386.deb
2. Then run rootblock script if you did not do it.

just wondering, would this have any effect for the adb connections ? My ubuntu cannot find the transformer from adb but could mount it just find

Try This....
anyone having issues try one of the following extensions.. tar.gz would be my first suggestion for those using ubuntu like myself. i also took the liberty of changing the perms to exe on the script before i compressed all these so now all you need to do is
1.download the extension pack i upped
(find the one that works for you)
2.open file
3.execute
*i also tossed the edited script in the pack for anyone who cannot get it to open.
fyi; ark also worked for me to access the file from the op, while my 7z kept erroring out.
​

ari197 said:
just wondering, would this have any effect for the adb connections ? My ubuntu cannot find the transformer from adb but could mount it just find
Click to expand...
Click to collapse
I can access the TF with adb just fine. Just cd to the android-sdk-linux_x86/platform-tools folder containing adb and run:
Code:
./adb devices
or
Code:
./adb shell
if that folder isn't already configured into your path. Works for me anyway.

I've been looking at this and trying to do it with udev instead, since I want the dynamic creation of the folder in /media. Mounting works great, but removing doesn't. So anyone with knowledge of udev rules is more than welcome to give me some hints. Here's the rules so far:
Code:
See below...
The command run when removing works if I enter it as root in a terminal, so it seems to me that it's either not run when removing or it's failing somehow when run from udev. Edit: I've figured out that it's not run at all. Looking into it...
Also, hint for those who follow the manual instructions you could also get the necessary permissions through Users and Groups instead of editing the groups file. You still have to edit /etc/fuse.conf to allow other though.

These are instructions for automatically mounting and unmounting your pad (not keyboard!). This will create (when plugged in and detected) and remove (when unplugged) the Transformer directory under /media, so that the experience is more in line with other USB devices.
For those of you who already have run the script in the first post, you need to do some cleanup. Edit /etc/fstab and /etc/udev/rules.d/51-android.rules and remove the lines which the script created. Which lines in fstab should be straight forward, and in 51-android.rules look for rows that either have no idVendor set or has it set to 0b05. If unsure, post here FIRST before making changes!
This is the content of /etc/udev/rules.d/51-android.rules:
Code:
ACTION=="add", SUBSYSTEM=="usb", ENV{ID_MODEL}=="MTP", ENV{ID_MODEL_ID}=="4e0f", MODE="0666", RUN+="/bin/bash -c 'mkdir -p /media/Transformer && mtpfs /media/Transformer -o allow_other'"
ACTION=="remove", SUBSYSTEM=="usb", ENV{ID_MODEL}=="MTP", ENV{ID_MODEL_ID}=="4e0f", RUN+="/bin/bash -c 'fusermount -u /media/Transformer && rmdir /media/Transformer'"
If you have other Android devices you may or may not have other lines here too. If so, just add these two lines to the end of the file. My Transformer is bought in Sweden so if things don't work for you, maybe there's another idProduct/ID_MODEL_ID out there? Check with lsusb when your pad is plugged in, the vendor (xxxx) and product (yyyy) ID is [xxxx:yyyy] on the line that says ASUSTek Computer Inc. For my pad, the productID/ID_MODEL_ID is 4e0f.
You need to uncomment allow_other in /etc/fuse.conf and add your user to the fuse group (preferably via Users and Groups) apart from adding the above two lines to your 51-android.rules:
Start Users and Groups, choose your user and press Advanced settings. Under the tab User Privileges you should mark Mount user-space filesystems (FUSE).
Open and edit /etc/fuse.conf (see below on how to use nano). Uncomment user_allow_other in the end (remove the # at the beginning of the line).
After this you should reboot your computer. This will have two effects: The udev rules will be reloaded so the changes apply, and your user will have the necessary privileges granted.
Now you can try to plug in your pad and see if it shows up under /media. Give it some time, it takes a while. (Approx. 30 seconds for me.)
A note on editing files from the terminal window
When editing files from the terminal I always use nano. The command is (if opening a file that requires super user rights):
sudo nano [path-to-file]
The editing is straightforward, works just like any other text editor. When you're finished, press Ctrl+X (Exit) and answer the question with Y (if your system is in english that is) if you want to overwrite the earlier file.
Problems?
If you have ADB activated you could try rmbell's udev rules. This removes the match ID_MODEL == MTP and changes the productID to 4e1f. The productID changes depending on what functions are activated, as discussed in this post by ritterkeks. If you want the pad to be mounted with and without ADB activated, you can add all four rules.
If the udev rules doesn't work, run this command in the terminal and see what happens when you plug the pad in:
udevadm monitor --property --udev
Exit with Ctrl+C. Are the values the same as in the udev rules you're using? If not, change accordingly or deactivate some function which has made the productID change.

"unable to mount location" error
I'm getting an error that reads "Unable to mount location"
Then:
fusermount: failed to open /etc/fuse.conf: Permission denied
fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf
I'm a monkey in a spaceship when it comes to some of this stuff Any idea what I should do (in layman's terms)?
All help appreciated!

I'm not sure if your problem is from the script in the first post or my instructions. But as the message says you need to edit fuse.conf. Also, you need to be a member of the FUSE group.
Open and edit fuse.conf with this command in a terminal:
sudo nano /etc/fuse.conf
Uncomment (remove the #) user_allow_other in the end, exit with Ctrl+X and save with Y.
Start Users and Groups, choose your user and press Advanced settings. Under the tab User Privileges you should mark Mount user-space filesystems (FUSE). I believe that you need to reboot after this change, otherwise the necessary privileges hasn't kicked in.
Try it and let me know

[Q] Is there an easy way to apply the patch files from the OTA on a PC?

I'm looking for a way to apply the patch files to the TF system image on a PC. This is not about how to get the ota to flash- I got that covered. This is about repacking roms or making flashable updates that don't rely on stock firmware.
I found a few ways, but none of them seem "easy."
The dirty way- apply the OTA and grab the system image.
From the PC command line - Linux has diff/patch, but that doesn't work for some of the patches in "IMGDIFF2" format. Besides, I'm not sure I ever got the regular BSDIFF40 patches to work.
The git option- There is the apply_patch source code in github. This works great, but I have to run it on the TF. Is it possible to compile this to work on my ubuntu box?
Maybe there is some magic script out there called apply_ota?
Thanks in advance

gee one said:
I'm looking for a way to apply the patch files to the TF system image on a PC. This is not about how to get the ota to flash- I got that covered. This is about repacking roms or making flashable updates that don't rely on stock firmware.
I found a few ways, but none of them seem "easy."
The dirty way- apply the OTA and grab the system image.
From the PC command line - Linux has diff/patch, but that doesn't work for some of the patches in "IMGDIFF2" format. Besides, I'm not sure I ever got the regular BSDIFF40 patches to work.
The git option- There is the apply_patch source code in github. This works great, but I have to run it on the TF. Is it possible to compile this to work on my ubuntu box?
Maybe there is some magic script out there called apply_ota?
Thanks in advance
Click to expand...
Click to collapse
I recently wrote some code for a ARM basestation (basically like a Freescale cell-phone board), and had to be able to supply some updates over time. Before the project ended, I'd also found bsdiff and bspatch and tested them on linux, where they worked fine (as long as you'd created either an ARM bsdiff/patch version or somehow took the big/little endianness of the hardware into account).
At first I thought it wasn't working but it seemed to do OK on standard cross-compiled-for-ARM files. (well, they still ran and performed the same stunt..).
The thing that worried me about this method is that they'd never have the same md5sum before as after, likely because they had a different amount of bytes. (e.g. you make your own before and after and build each, then create the patch file using bsdiff, then patch the old file using the patch file, and create a 'newtestfile', but look at the number of bytes (or md5sum it) difference between the original stage 2 file and the stage 1 + the patch file via bspatch).
Sorry that is pretty confusing the way I just tried to explain it.. I'll look this over again tomorrow and figure out if I said anything worthwhile

For inputs, the applypatch binary takes the file names, sha1 checksums of the before and after, and the final target size. Chance are high that the patch would only work if everything went in and came out properly.
Code:
apply_patch("/system/framework/framework-res.apk", "-",
99b382b5886e505508fa3f730d778a1bed4bc04e, 9761707,
a4798fb8d61d08b6f067e522e67107f6dd556148, package_extract_file("patch/system/framework/framework-res.apk.p"));
I think that bsdiff/patch will generally work on most of the OTA (BSDIFF40 files), but some of the files are IMGDIFF2, which targets binary/data files.
The applypatch binary works on both and as far as I can tell, is the same binary that the TF uses. Here is a snippet from the AOSP recovery... TF binary included Use the binary on your TF, not applypatch.sh! I suppose you can put it in system/xbin/
It would resolve the issue if I could compile this to run on my desktop from ubuntu. Short of that, I'm working on a script for the TF to parse the updater script.

gee one said:
For inputs, the applypatch binary takes the file names, sha1 checksums of the before and after, and the final target size. Chance are high that the patch would only work if everything went in and came out properly.
Code:
apply_patch("/system/framework/framework-res.apk", "-",
99b382b5886e505508fa3f730d778a1bed4bc04e, 9761707,
a4798fb8d61d08b6f067e522e67107f6dd556148, package_extract_file("patch/system/framework/framework-res.apk.p"));
I think that bsdiff/patch will generally work on most of the OTA (BSDIFF40 files), but some of the files are IMGDIFF2, which targets binary/data files.
The applypatch binary works on both and as far as I can tell, is the same binary that the TF uses. Here is a snippet from the AOSP recovery... TF binary included Use the binary on your TF, not applypatch.sh! I suppose you can put it in system/xbin/
It would resolve the issue if I could compile this to run on my desktop from ubuntu. Short of that, I'm working on a script for the TF to parse the updater script.
Click to expand...
Click to collapse
I'm not sure I understand the intent. I've just awoken and that's probably part of the problem..
'applypatch' is part of the standard (say 10.04 ubuntu) repo and is in a package called makepatch. Did you want to be able to run it "from" a standard x86 Ubuntu dist but have it targeting "armle" files or just be able to run it both on Ubuntu & Android correctly?
The files I tested were a) both binaries in different stages of evolution, and b) both built for armle(droids) and x86_64 and didn't behave differently, worked, and the checksum could be pre-calculated.
I'm going to mess around with applypatch when I'm out of my morning coma and get back to this in a bit.
Hah -- I do wonder if there's already a script (maybe it's a binary though) that lives in the dm* app on the tf101 that just does this already. You may have been right about that. Something has to apply those OTA patches. There's a few mechanisms for detecting them but I hadn't thought about how they're applied.
I'm trying to think if there's any scenario where the market applies 'patches' rather than always downloads the next version in entirety. Not sure about that.
Anyway, I'll see if I can get anywhere with applypatch.
Edit:: After a little messing around :
I finally see what you are trying to do and why what I'm doing is totally useless for that. I'm going to spend a little time looking over the imgdiff2 (apk) deal, and probably end up now figuring out a thing.

I'll look into the makepatch package later today. Maybe that might be what I'm looking for.
Just to be clear, the intent is to be able to patch a "stock" system without it actually running on the TF. For example, to take the stock 8.6.5.9 image, decompress system/ to the PC, apply the patches from the OTA, and then end up with a stock 8.6.5.13 image. This would all happen on the PC, not the TF. It would streamline the ROM cooking process and make it easier and faster to spit out a complete, flashable update that won't fail sha1 checks because of mods.
sent from my cyanogen(mod) vision

Zombie Thread Alert!!!
You have been warned....
OK, so my C-fu isn't strong enough to figure out how to cross compile the apply patch binary to run on a x86 computer. However, I did notice that the applypatch binary gets built in the aosp/generic_x86 build, specifically in the system/bin folder.
This will be an exectuble file that will run on a x86 linux based system and do the same thing that the applypatch binary does to the ota package during the ota update. In other words, you can apply the ota patch to a system image from the comfort of the linux command line without having to actually flash the ota. This will allow you to be the first one on your block to have the new system image without having to mess up whatever rom you are currently running.
Of course, this might all be moot if we don't see another ota for our tf. Come on ASUS, just one more so I can test this out on a live update.
OK, so if you read this far, repo sync your aosp source and build the generic_x86 branch. The binary is aosp/target/product/generic_x86/system/bin/applypatch_static
edit: then after you've done that, use a script to parse the updater-script to extract the patching commands and then she-bang! I'll post mine a little later- I've got to get some flowers for tonight

Any luck with that? I'm also searching for a way to apply an OTA update to a ROM using Ubuntu.

bgcngm said:
Any luck with that? I'm also searching for a way to apply an OTA update to a ROM using Ubuntu.
Click to expand...
Click to collapse
Yeah, it works like a charm. The binary is compiled in the aosp/generic_x86 build. There is no need to flash, you can just pull it out of the system/bin folder after compiling the source. You can probably use an "mmm" command to make just files you want, but I'm not sure which package it's in. Then, I use a script that converts the OTA patch commands in the updater-script to a script file that runs on my linuxbox that calls the binary and patches the ROM under linux/ubuntu.

Do you mind sharing your applypatch binary? I don't have AOSP source on my Ubuntu installation right now, so if you don't mind sharing I wouldn't need to build it from sources...

bgcngm said:
Do you mind sharing your applypatch binary? I don't have AOSP source on my Ubuntu installation right now, so if you don't mind sharing I wouldn't need to build it from sources...
Click to expand...
Click to collapse
I attached the applypatch binary- it should work on most x86 linux distros, maybe.
As I stated in my earlier post, the binary is built with the x86 build. I'm not so good with the build system, but I think the code is from the bootable/recovery/applypatch. I included the relevant notices for that code.
In the OTA updater-script, there will be a bunch of applypatch commands- you might have to change the name/permissions of the binary, but that syntax should be what you are looking for.
Enjoy!

Thank you very much for your time. It seems we are almost the only one trying this. The thing is that I really need to get this done because OTA update is failing for me by running it the usual way.
I've tried your binary and it works under my Ubuntu installation, however I was trying to manually patch just one file and got this:
Code:
[email protected]:~/Desktop/system/app$ applypatch_static ./ApplicationsProvider.apk - d9023cd58fd055e1ca3c8f8492b2c36aba923c6a 6184 69bea5d5a62980c611b903de8243d980f30e5fb5:../../patch/system/app/ApplicationsProvider.apk.p
applying patch to ./ApplicationsProvider.apk
target 6184 bytes; free space 1080520704 bytes; retry 1; enough 1
chown of "./ApplicationsProvider.apk.patch" failed: Operation not permitted
I got that chown error message although ApplicationsProvider.apk.patch was created. Is this the patched file?
EDIT: Nevermind, I forgot to run the command as superuser.
EDIT2: Would it be too much if I ask you also to share the script that converts OTA commands from updater-script into the correct syntax to run on Ubuntu?

There is no guarantee that this will work on your machine. It's a work in progress and will probably require some editing. This just creates the scripts that extract the commands from the OTA updater-script.
I haven't used it in a while since ASUS hasn't put out any updates lately, but I would copy the updater-script and rename it delete_source. Then I would edit out everything in delete_source except the delete commands. There might be an issue with a trailing ; or similar character. The script runs on the copy of the updater-script to pull out the applypatch commands and the delete_source to pull out the files that need to be deleted from the rom.
Check the syntax, but it will be something like ./tf_file <head of stock rom> <head of patch source>
If you have the donor stock system folder and the patch folder from the OTA at the same dir level, it would be ./tf_file system patch
The script also assumes that the applypatch binary is in your path as apply_patch. You can adjust as needed.
Lastly, my bash skills are probably pretty weak, so if you see ways to improve this script or add error/path checking, feel free to post improvements, etc. There is probably a really slick way to do this with just one script and not a script that makes three other scripts. The historical reason for this is that I used to transfer the files to my transformer and run the script on there. Running in Ubuntu is much faster!
Good luck!
Code:
#!/bin/bash
# extract_source is used to pull files to be patched from source tree
# delete_source is ued to delete unneeded files from the source tree
# tf_file applies the patches to the extracted source files
# don't forget to copy the direct updates in the OTA
# usage is apply_ota
# assumes that updater-script and delete-script are at same directory level
# updater-script is from META-INF folder, delete-script is from updater-script
# add error checking to count number of patches and number of patched files
# parse trailing / in folder names
work_dir=`pwd`
tf_file=$work_dir"/tf_file"
extract_source=$work_dir"/extract_source"
delete_file=$work_dir"/delete_source"
work_file=$work_dir"/work_file"
echo "#!/bin/bash" > $tf_file
sed ':a;N;$!ba;s/\n/\ /g' updater-script | sed -e 's/;/\n/g' -e 's/\ \ */\ /g' | grep "apply_patch(" | sed -e 's/"//g' -e 's/,\ p/\:p/g' -e 's/,//g' -e 's/package_extract_file(/$2\//g' -e 's/(/\ $1/g' -e 's/))//g' >> $tf_file
chmod 755 $tf_file
echo "#!/bin/bash" > $extract_source
#sed ':a;N;$!ba;s/\n/\ /g' updater-script | sed -e 's/;/\n/g' -e 's/\ \ */\ /g' | grep "apply_patch(" | cut -d \" -f 2 |sed -e s'/^/cp\ --parents\ $1/g' -e s'/$/\ \$2"\/\"/g' >> $extract_source
sed ':a;N;$!ba;s/\n/\ /g' updater-script | sed -e 's/;/\n/g' -e 's/\ \ */\ /g' | grep "apply_patch(" | cut -d \" -f 2 |sed -e 's/^/cp\ --parents\ $1/g' -e 's/$/\ \$2"\/\"/g' -e 's:/system::g' >> $extract_source
chmod 755 $extract_source
sed -e 's:",:\n:g' -e 's:\ ::g' delete-script > $delete_file
sort -r $delete_file > $work_file
echo "#!/bin/bash" > $delete_file
grep / $work_file | grep -v /$ | sed -e 's:"/:rm\ \$1/:g' >> $delete_file
grep /$ $work_file | sed -e 's:"/:rmdir\ \$1/:g' >> $delete_file
chmod 755 $delete_file
exit 0

IMGDIFF Tool Needed...
Hello Sir, i am looking for a tool to generate IMGDIFF patches on windows... i already have a BSDIFF tool, but coudn't find IMGDIFF... Thank You

I have no idea about windows, but the source is here.
https://android.googlesource.com/platform/bootable/recovery/+/master
You could install linux and build it from there. Follow these directions http://source.android.com/source/building.html
There is also lots of great info here http://wiki.cyanogenmod.org/w/Development#Learning_To_Build_CM
Good luck!

gee one said:
Good luck!
Click to expand...
Click to collapse
I am very thankful to you for replying me sir... these guides are awesome, but i dont think my slow internet connection and PC with 1GB RAM will be allowing me to download or compile
actully i was making a tool to apply OTA Updates in PC, it is semi functional for now, it is able to read updater script and apply bsdiff patches, but can't handle imgdiff patches and now i am stuck at this point because of no imgdiff port for windows... :crying:
i will be very thankful if you or @bgcngm can compile imgdiff at least for cygwin? else i will have to postpond this project until someone else compile and port imgdiff for win/cygwin

Sorry- I'm clueless about cygwin. You could try dual booting to linux to compile from there.

gee one said:
You could try dual booting to linux to compile from there.
Click to expand...
Click to collapse
Thanks alot for taking your time to reply me sir now it seems like linux is the only option left
as you are a linux user will it be too much if i ask you to find a right guide to compile ota tools only? cause i don't know much about linux, and it will be difficult for me to find right guide to compile ota tools only...

jawad6233 said:
Hello Sir, i am looking for a tool to generate IMGDIFF patches on windows... i already have a BSDIFF tool, but coudn't find IMGDIFF... Thank You
Click to expand...
Click to collapse
Answer is here - [Tool]imgdiff executable for windows. BTW, anybody have applypatch for Win32?

Hoping this works for lollipop.

[Tool]File(1) command for Android (ver 5.11)

After wandering around looking for an android build of the file command, I finally went ahead and compiled it myself.
The attached package contains the file binary (v5.11), the libmagic.so shared library, the associated magic files and the build specific files that I needed (over and above the source) to build it. The paths of the files are relative to '/'.
Refs:
source: file website (www_darwinsys_com/file/)
compiling: Native Vim on Android (gdr_geekhood_net/gdrwpl/vim-android.php)
Note: Add the location of libmagic.so to LD_LIBRARY_PATH to get rid of the "CANNOT LINK EXECUTABLE" error. (I use /data/local/lib)

Compiled a dynamically linked library and associated binary instead of the monolithic binary built earlier.

samveen said:
After wandering around looking for an android build of the file command, I finally went ahead and compiled it myself.
The attached package contains the file binary (v5.11), the libmagic.so shared library, the associated magic files and the build specific files that I needed (over and above the source) to build it. The paths of the files are relative to '/'.
Refs:
source: file website (www_darwinsys_com/file/)
compiling: Native Vim on Android (gdr_geekhood_net/gdrwpl/vim-android.php)
Note: Add the location of libmagic.so to LD_LIBRARY_PATH to get rid of the "CANNOT LINK EXECUTABLE" error. (I use /data/local/lib)
Click to expand...
Click to collapse
I followed your instructions and it works perfectly.
Code:
$ echo $LD_LIBRARY_PATH
/data/local/lib:/data/local/lib:/data/local/lib:/system/lib:/vendor/lib
$ file
Usage: file [-bchikLlNnprsvz0] [--apple] [--mime-encoding] [--mime-type]
[-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
file -C [-m magicfiles]
file [--help]
$ getprop | grep device
[ro.cm.device]: [jordan]
[ro.product.device]: [umts_jordan]
I have been looking for this for a while. Man!! This post needs some serious attention. Great work.
Sent from my MB525 usiIng XDA

@samveen Any chance you could compile it with libmagic statically linked again to make it portable? Thanks for your time.

osm0sis said:
@samveen Any chance you could compile it with libmagic statically linked again to make it portable? Thanks for your time.
Click to expand...
Click to collapse
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).

samveen said:
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).
Click to expand...
Click to collapse
I was just looking for this. Great tool.
Thanks for your work.

samveen said:
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).
Click to expand...
Click to collapse
Any luck? I think it only needs libmagic statically linked to be portable; the other NDK/Bionic stuff can probably stay dynamic to keep the filesize down. I see file 5.17 source is available now too if you wanted to update your builds to the latest.

osm0sis said:
Any luck? I think it only needs libmagic statically linked to be portable; the other NDK/Bionic stuff can probably stay dynamic to keep the filesize down. I see file 5.17 source is available now too if you wanted to update your builds to the latest.
Click to expand...
Click to collapse
If you want only the system file type, the workaround i use is this (eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE)
In a script just add a var like this
TYPE=$(eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE) and you will have the FS type in $TYPE.
Phone: Samsung Galaxy SII - GT-I9100
Kernel: Dorimanx kernel v10.43v99-mv3
1st ROM : SlimSaber 4.4.2 Maliv3 by fusionjack build of 20140320 (Online)
2nd ROM :
MODEM: UHMS1
MODS: Partition Fix, Hue Blue v4.4 by Kroz :good:

Computoncio said:
If you want only the system file type, the workaround i use is this (eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE)
In a script just add a var like this
TYPE=$(eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE) and you will have the FS type in $TYPE.
Click to expand...
Click to collapse
Interesting but nope; I need file to determine a filetype. Compressed archives via a script, to be specific.

@osm0sis I got the code compiled (both with a dynamic libmagic.so, and libmagic compiled into file) but I need a little more time to compile the magic data (it requires running file on the target device to compile the magic data into a binary datafile). Once I have that figured out (either by compiling it or by creating a script to allow the users to do it themselves, I'll post it all in one go (need one more day).

samveen said:
@osm0sis I got the code compiled (both with a dynamic libmagic.so, and libmagic compiled into file) but I need a little more time to compile the magic data (it requires running file on the target device to compile the magic data into a binary datafile). Once I have that figured out (either by compiling it or by creating a script to allow the users to do it themselves, I'll post it all in one go (need one more day).
Click to expand...
Click to collapse
Awesome! Thanks so much. The magic file isn't terribly important since you can grab one a lot of places and specify it with the -m parameter; keeps the file size down too if you were thinking of including it in the binary somehow!

Alright. I just wanted to thank you again for your work but I managed to make a static native compile (~440kb) of file-5.17 using my N7 and a kbox build environment I set up. It's available in my ARM Android Image Kitchen linked in my sig. Cheers!

osm0sis said:
Alright. I just wanted to thank you again for your work but I managed to make a static native compile (~440kb) of file-5.17 using my N7 and a kbox build environment I set up. It's available in my ARM Android Image Kitchen linked in my sig. Cheers!
Click to expand...
Click to collapse
@osm0sis - Can you please share how did you static compile this? I have setup the Ubuntu 13.10 image on Android using Complete Linux Installer (with all the necessary build tools setup).
I am able to compile this, but it's linking the shared library "libmagic.la".

Need to pass --static to ./configure and have a cross compiler set up, for ARMv7.

ericlnu said:
Need to pass --static to ./configure and have a cross compiler set up, for ARMv7.
Click to expand...
Click to collapse
I don't think that's a valid option.
Code:
$ ./configure --static
configure: error: unrecognized option: `--static'
Try `./configure --help' for more information

amit.bagaria said:
I don't think that's a valid option.
Code:
$ ./configure --static
configure: error: unrecognized option: `--static'
Try `./configure --help' for more information
Click to expand...
Click to collapse
Nope, seems not ^_^. Sorry, it usually is with autoconf derived configure scripts.

amit.bagaria said:
@osm0sis - Can you please share how did you static compile this? I have setup the Ubuntu 13.10 image on Android using Complete Linux Installer (with all the necessary build tools setup).
I am able to compile this, but it's linking the shared library "libmagic.la".
Click to expand...
Click to collapse
Haven't tried it using a full Linux image on Android, just made it native compile in KBox2 on my N7'13 for awhile by editing in the gettext functions it was complaining about, and have since moved on to cross-compiling on my Windows 7 desktop in Cygwin using the NDK. There were still a few tricks to getting it to be a true static compile. It's all linked in my post here:
http://forum.xda-developers.com/showthread.php?p=54510825
My latest compile included in my AIK-mobile package is file 5.20. :good:

I don't suppose you could do a quick compile with the -fPIE flag? Trying to use this on 5.0 + and don't want to patch my linker.
Tried to compile it myself (it sort-of worked, file --version is operational) but when I try to actually use it for something my terminal borks out and switches to some weird font after about 20 warnings. (configure also seems to be using the -rpath flag apparently, linker complains about it) hmm
EDIT: I've got it working after some trial and error. If anyone requests I'll post it
EDIT 2: android 6.0's linker seems less forgiving than 5.0/5.1, need to compile without -rpath (so no just using ./configure) will keep posted

Can you please post the file binary you built with -pie ?

[Scripting][ARM][Installer][Port] Python (static) 2.7.9 and 3.4.2

(Check out https://www.python.org for information on what python is.)
Yes you read that correctly, static python for Android! It took a while to figure out how to get this compiled, but I finally did it. I present you with an installer and a little bit of testing. The second post contains information on compilation and the sources. Btw, any cool python scripts are welcomed!
INSTALLATION
Recovery flash installers are attached that will install python 2.7.9 and/or 3.4.2 to /system/pythonX.X.X. You can install both if you want. Installation size is about 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
After installation, the python director(ies) in /system will contain the static python binary, some scripts, and a bunch of modules and documents. Separate scripted executables will be installed to /system/bin/python or /system/bin/python3 depending on which one is installed. These basically just set the PYTHONHOME environment variable and execute the python binary.
TESTING
When you boot up to Android after installation, you should be able to just open up a terminal and fire off some python commands.
Test using python 2.7.9:
Code:
python -c 'print "Hello World!"'
Test using python 3.4.2:
Code:
python3 -c 'print("Hello World!")'
You can also write scripts shelled with python:
Code:
#!/system/bin/python
print "Hello World!"
Make sure to set them as executable with "chmod +x".
Adjust your screen brightness via /sys (root required, tested on Galaxy Nexus):
Code:
python -c 'f=open("/sys/devices/omapdss/display0/backlight/s6e8aa0/brightness","w"); f.write("40"); f.close()'
A cool script I made to tweak file system I/O like rq_affinity, rotational, etc for I/O blocks:
Code:
#!/system/bin/python
import os,re,sys
list=[]
# find all directories containing rq_affinity
for roots, dirs, files in os.walk('/sys'):
for file in files:
match=re.search(r'\S+/rq_affinity',os.path.join(roots,file))
if match:
list.append(match.group().replace('rq_affinity',''))
# write specific values to files in each directory found before
for dir in list:
for name in 'rq_affinity', 'rotational', 'read_ahead_kb', 'nr_requests', 'iostats', 'nomerges', 'add_random':
try:
f=open(dir+name,'w')
if name is 'rq_affinity': f.write('1')
elif name is 'read_ahead_kb': f.write('512')
elif name is 'nr_requests': f.write('512')
else: f.write('0')
f.close()
except IOError:
sys.stderr.write('Problem writing to ' + dir+name + '\n')
Or execute "python" or "python3" without any parameters to open up the interpreter and go from there. When you start for example "python" (python 2.7.8), you should see something like:
Code:
Python 2.7.8 (default, Dec 2 2014, 05:15:18)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
***I haven't fully tested this yet, so let me know how it goes. I know basic commands work, but there are still some complexities in the compilation that may need to be figured out.***
(update) -- Install Extra Packages/Modules via "easy_install" -- (root required)
1) In a shell, remount /system read-writable with:
Code:
mount -o remount,rw /system
2) Change directory to /system/pythonX.X.X, i.e.,
Code:
cd /system/pythonX.X.X
3) Execute easy_install followed by a package name, i.e.,
Code:
./easy_install [i]package_name[/i]
4) Let it download and install! Then test it out.
5) Remount /system read-only with:
Code:
mount -o remount,ro /system
Also, to remove a package, execute:
Code:
./easy_install -m [i]package_name[/i]
rm -r ../lib/pythonX.X/site-packages/[i]package_name[/i]*.egg
*** Flashing the resolv.conf patch may be required to make an internet connection to download modules.
Also, a few things need to be done to get easy_install working:
1) For python 2.7.8 or 2.7.9 you need to link python2.7 to python like this:
Code:
ln -s /system/python2.7.9/bin/python2.7 /system/python2.7.9/bin/python
I'll make sure this is automatically done in future installs.
2) SSL certificates need to be added to "/etc/pki/tls/certs/" to use SSL with easy_install.
Code:
mkdir -p /etc/pki/tls/certs
curl http://curl.haxx.se/ca/cacert.pem -o /etc/pki/tls/certs/ca-bundle.crt
ISSUES
The python installations have the following modules compiled in with external dependencies such as libraries not included at this point:
Code:
--------------PYTHON 2.7.8-2.7.9----------------------
_hashlib _multiprocessing _ssl
_testcapi bz2 crypt
dbm
Code:
--------------PYTHON 3.4.2-----------------------------
_crypt _dbm _decimal
_hashlib _multiprocessing _ssl
_testcapi
I will look into this more.
As @bubbleguuum points out, name resolution with python 3.4.2's urllib wasn't working. I found a workaround by adding the line
Code:
options single-request-reopen
to /system/etc/resolv.conf. This still needs some testing but seemed to work for me. I've included a recovery flash zip to patch /system/etc/resolv.conf if the line is not there since this is a root operation.
UPDATES
* 2014-12-02 * Compiled in more modules for each installation. This makes the installation a bit bigger, but it's worth it. You get a more complete python! To get stuff like help functions and math and readline modules, you need the larger installations linked below under "DOWNLOADS".
* 2014-12-03 * Thanks @cybojenix for pointing out some incapatibility issues with the original edify installer I was using. I have now updated the installers to use SuperSU's great non-edify sh-script installer.
* 2014-12-05 * Some updates to the installers to include more modules, plus size reduction for python 3.4.2. Modules added to python 2.7.8 : _bsddb _ctypes, _ctypes_test, _hotshot, _json, _lsprof, _sqlite3, future_builtins, and ossaudiodev, easy_install, pip(broken for now). Modules added to python 3.4.2: _bz2, _ctypes, _ctypes_test, _gdbm, _json, _lsprof, _opcode, _sqlite3, _testbuffer, _testimportmultiple, ossaudiodev, xxlimited
* 2014-12-05 * Some big updates to the installer to include almost the rest of the modules that wouldn't compile. Also easy_install is working on my end with these new experimental builds. With a little effort, should be able to get pip working as well. I had to use some hacky flags to get it to compile, ignore unresolved-symbols and such, so it definitely needs some testing. Another thing I did was tweak the install script to write over previous installations from these installations. I recommend downloading the experimental builds if you read this. The worst thing that might happen is one of the new modules I've included won't work completely.
* 2014-12-14 * Added python 2.7.9. Trimmed the installation sizes down to the much more reasonable 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
* 2014-12-24 * Attached a recovery flash.zip to patch resolv.conf to try and fix a urllib name resolution error.
* 2014-12-26 * A couple updates: Tweaked python installations so that modules looks for "/system/bin/sh" instead of "/bin/sh" when needed. Also a few "#!" corrections for scripts in the python bin directory. This shouldn't break anything from what I've tested, only make Android python more capable, but I'll keep the old installations attached for now just in case since the modifications involved using "sed". The second update is nice. I've added a little section on installing extra modules with easy_install(included and working with all installations).

Compiling
The environment I compiled in was a debian "wheezy" image mounted as a loop device on Android. The reason I compile this way is so I have full access to the arm environment tools, gcc compiler, etc. without having to go through the trouble of using a cross-compiler (which doesn't work in this case from my testing with uClibc, or eglibc. Code sourcery's might work but I didn't feel like booting an x86 linux installation to try). You can recreate the same environment pretty easily by using either Complete Linux Installer or Linux Deploy apps. I use both.
To get a static python compiled, I first downloaded the following packages with APT:
Code:
[i]apt-get build-dep python python3[/i]
build-essential gcc make # building tools
binutils-gold # awesome gold linker
zlib1g-dev # zlib
libreadline-dev, libncurses5-dev # readline, curses
libbz2-dev # bz2
libsqlite3-dev # sqlite3
python-bsddb3, python3-bsddb3
libgdbm-dev # gdb
libssl-dev # ssl
python-tk, python3-tk # tkinter
libdb-dev # db
python-gdbm python-bsddb3
libffi-dev # _ctypes
tcl8.6-dev # tkinter
libx11-dev # tkinter
libmpdec-dev # decimal
Then I manually installed binutils-gold by unpacking the rpm for armv7h (this isn't available with APT yet). You can get it here though. (Update: this should be available in the debian repositories now.)
For static compilation, you need to make some changes to Modules/Setup from the python source directory. Specifically, you need to add
Code:
*static*
to the top of the file. Then you need to uncomment any commented modules you would like compiled in. When you compile and see a list of failed modules, search for each of these in Modules/Setup and uncomment them. For example, change
Code:
#math mathmodule.c _math.c
to
Code:
math mathmodule.c _math.c
. Then recompile. There may be some shared dependencies and packages as well you need to figure out to get them to compile. Take a look at the attached "Setup" files for more detail.
The actually compile steps look like this:
(1)
Code:
./configure --build=arm --prefix="$PWD"/out LDFLAGS="-static -static-libgcc -Wl,--unresolved-symbols=ignore-all -Wl,--export-dynamic" CPPFLAGS=-static CXXFLAGS=-static CFLAGS="-Os -static" LDFLAGS=-static LD=ld.gold
(2) Modify the Modules/Setup file.
(3)
Code:
make clean; make install
After compiling, there are some things you can do to make your installation in "./out" smaller. Strip the large binaries in "out/bin", i.e.
Code:
strip -s ./out/bin/python2.7
Also remove the python archive library, *.o's, etc. with:
Code:
make clean
This might seem kind of weird, but it works and cleans out unnecessary files from your installation.
**Make sure to backup your Modules/Setup file if you run "make distclean". I lost my 3.4.2 Modules/Setup file after doing this. :silly: Now I gotta recreate it, dammit.
**I should also mention that compile time is very fast given the amount of data generated. It takes about 10 minutes to get python and all it's modules compiled on my Galaxy Nexus.**
:EDITS:
: Updated some configure parameters. Removed unnecessary, extra "./configure". Also added some updates to the included "Setup" files.
: Hacky update to "./conifgure" to ignore unresolved symbols, export dynamic. This allows certain modules to be compiled which have shared dependencies.
: Update to trimming down the installation size after "make". Use another "make clean".
Downloads - Extract in the source directory
Setup (python 2.7.8)
Setup (python 3.4.2)
Setup (Experimental) (python 2.7.8)
Setup (Experimental) (python 3.4.2)

Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference

cybojenix said:
Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference
Click to expand...
Click to collapse
Thanks for your reply! I've changed the installer to use SuperSU's no-edify sh-scripted updater. I tested on my device, but let me know if there are any issues though. I kept it pretty basic.
I added some stuff on the source modifications and compilation to the 2nd post. The only file I've actually modified so far in the source is the generated Modules/Setup file, and I've included the one I used for python 2.7.8. Unfortunately, I over-cleaned the python 3.4.2 directory, which deleted my modified Modules/Setup there, so I'll have to add it later when I re-edit it.
That's a great idea on pip/easy_install. I did download the pip_installer and tried out installing on the static python I made. Got some errors though pointing to a few modules I haven't gotten compiled into the static python installation (listed in the OP near the bottom, specifically _ctypes). I'll see what I can do about that. The line for compiling the module might just need to be added to Modules/Setup. Hopefully that's the case.

@7175
Thank you very much for these binaries.
There is however a problem: name resolution (DNS) doesn't seem to work at all (with both python 2 and 3 downloads).
All attempts to use urllib.request.urlopen('http://somehost.com') fail with "<urlopen error [Errno -2] Name or service not known>" (running python as root but it doesn't matter, and a rooted Nexus 4 running 4.4.4 stock ROM):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 455, in open
response = self._open(req, data)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 473, in _open
'_open', req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1202, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1176, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
Click to expand...
Click to collapse
it works if replacing somehost.com by its ip address.
To reproduce above stack trace with python 3:
import urllib.request
urllib.request.urlopen('http://www.google.com')
while this work:
urllib.request.urlopen('http://173.194.45.229')
nslookup and general name resolution work anywhere else:
[email protected]:/ # nslookup www.google.com
nslookup wwwgoogle.com
Server: 8.8.4.4
Address 1: 8.8.4.4 google-public-dns-b.google.com
Name: wwwgoogle.com
Address 1: 2a00:1450:4007:80c::1014 par03s15-in-x14.1e100.net
Address 2: 173.194.45.242 par03s15-in-f18.1e100.net
Address 3: 173.194.45.240 par03s15-in-f16.1e100.net
Address 4: 173.194.45.244 par03s15-in-f20.1e100.net
Address 5: 173.194.45.241 par03s15-in-f17.1e100.net
Address 6: 173.194.45.243 par03s15-in-f19.1e100.net
Click to expand...
Click to collapse
This issue has probably something to do with how python was compiled to do name resolution. I remember vaguely a similar issue
on Ubuntu 12.04, with a ffmpeg compile that would always fail to resolve hostnames in URLs. No sure what the exact fix was but it had something to do with the libc and its name resolution mechanism. Or maybe something ipv6 related ?

@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
EDIT: Hmm, looks like re-compiling with "--disable-ipv6" didn't fix the problem. Will have to look more into this issue and hold off on uploading ipv4-only python3 since it didn't fix it. In the meantime, I've attached a recovery flash.zip to the OP to add the above mentioned line to /system/etc/resolv.conf if needed, which seems to fix the issue for me.

7175 said:
@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
Click to expand...
Click to collapse
Thank you for looking into this so fast.
My intended usage is for running the python binary from a regular non-root app (which works great!), so editing resolv.conf is not an option.

I've been researching that DNS resolving issue and could not find a clear explanation on why it fails on Android and not on other systems, and why exactly the added line in resolv.conf fixes (or rather workaround) it.
All seems to point to socket.getaddrinfo() failing for an unknown reason.
Is Python compiled with HAVE_GETADDRINFO defined ? If that's the case it uses the libc definition of getaddrinfo, otherwise
it uses an emulation function found in getaddrinfo.c.
My guess is that it is the latter, and what is causing this issue on Android.
It's probable this issue has already been solved on Android since other Python binaries exists, but Googling around do not give
much answer.

I get this error when trying to install anything with easy_install (using /system/python3.4.2/bin/easy_install-3.4 because /system/python3.4.2/easy_install doesn't seem to exist like in the OP):
Code:
# ./easy_install-3.4 feedparser
Searching for feedparser
Reading https://pypi.python.org/simple/feedparser/
Download error on https://pypi.python.org/simple/feedparser/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'feedparser' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or download links found for feedparser
error: Could not find suitable distribution for Requirement.parse('feedparser')
i got curl and the certs installed in /etc/pki/tls/certs/ as per instrucions in the OP, any help?

Thoughts on PIL?

Hmm... new problems here. Android Lollipop have no /system/etc/resolv.conf by-default and urllib patch is no more works.
How it can be fixed?

http://qpython.net/index.php

Where I can find source code and related documents for project

Pratik Raj said:
Where I can find source code and related documents for project
Click to expand...
Click to collapse
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?

hackintosh5 said:
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?
Click to expand...
Click to collapse
Idk about OP, but I can surely post my source for python 2.7 on arm/arm64. I compiled it like a year ago, but it wasn't too bad, just required a bunch of patches and code for dlopen

I just want to say thank you so much.