Database Info

[Tool]File(1) command for Android (ver 5.11)

After wandering around looking for an android build of the file command, I finally went ahead and compiled it myself.
The attached package contains the file binary (v5.11), the libmagic.so shared library, the associated magic files and the build specific files that I needed (over and above the source) to build it. The paths of the files are relative to '/'.
Refs:
source: file website (www_darwinsys_com/file/)
compiling: Native Vim on Android (gdr_geekhood_net/gdrwpl/vim-android.php)
Note: Add the location of libmagic.so to LD_LIBRARY_PATH to get rid of the "CANNOT LINK EXECUTABLE" error. (I use /data/local/lib)

Compiled a dynamically linked library and associated binary instead of the monolithic binary built earlier.

samveen said:
After wandering around looking for an android build of the file command, I finally went ahead and compiled it myself.
The attached package contains the file binary (v5.11), the libmagic.so shared library, the associated magic files and the build specific files that I needed (over and above the source) to build it. The paths of the files are relative to '/'.
Refs:
source: file website (www_darwinsys_com/file/)
compiling: Native Vim on Android (gdr_geekhood_net/gdrwpl/vim-android.php)
Note: Add the location of libmagic.so to LD_LIBRARY_PATH to get rid of the "CANNOT LINK EXECUTABLE" error. (I use /data/local/lib)
Click to expand...
Click to collapse
I followed your instructions and it works perfectly.
Code:
$ echo $LD_LIBRARY_PATH
/data/local/lib:/data/local/lib:/data/local/lib:/system/lib:/vendor/lib
$ file
Usage: file [-bchikLlNnprsvz0] [--apple] [--mime-encoding] [--mime-type]
[-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
file -C [-m magicfiles]
file [--help]
$ getprop | grep device
[ro.cm.device]: [jordan]
[ro.product.device]: [umts_jordan]
I have been looking for this for a while. Man!! This post needs some serious attention. Great work.
Sent from my MB525 usiIng XDA

@samveen Any chance you could compile it with libmagic statically linked again to make it portable? Thanks for your time.

osm0sis said:
@samveen Any chance you could compile it with libmagic statically linked again to make it portable? Thanks for your time.
Click to expand...
Click to collapse
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).

samveen said:
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).
Click to expand...
Click to collapse
I was just looking for this. Great tool.
Thanks for your work.

samveen said:
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).
Click to expand...
Click to collapse
Any luck? I think it only needs libmagic statically linked to be portable; the other NDK/Bionic stuff can probably stay dynamic to keep the filesize down. I see file 5.17 source is available now too if you wanted to update your builds to the latest.

osm0sis said:
Any luck? I think it only needs libmagic statically linked to be portable; the other NDK/Bionic stuff can probably stay dynamic to keep the filesize down. I see file 5.17 source is available now too if you wanted to update your builds to the latest.
Click to expand...
Click to collapse
If you want only the system file type, the workaround i use is this (eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE)
In a script just add a var like this
TYPE=$(eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE) and you will have the FS type in $TYPE.
Phone: Samsung Galaxy SII - GT-I9100
Kernel: Dorimanx kernel v10.43v99-mv3
1st ROM : SlimSaber 4.4.2 Maliv3 by fusionjack build of 20140320 (Online)
2nd ROM :
MODEM: UHMS1
MODS: Partition Fix, Hue Blue v4.4 by Kroz :good:

Computoncio said:
If you want only the system file type, the workaround i use is this (eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE)
In a script just add a var like this
TYPE=$(eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE) and you will have the FS type in $TYPE.
Click to expand...
Click to collapse
Interesting but nope; I need file to determine a filetype. Compressed archives via a script, to be specific.

@osm0sis I got the code compiled (both with a dynamic libmagic.so, and libmagic compiled into file) but I need a little more time to compile the magic data (it requires running file on the target device to compile the magic data into a binary datafile). Once I have that figured out (either by compiling it or by creating a script to allow the users to do it themselves, I'll post it all in one go (need one more day).

samveen said:
@osm0sis I got the code compiled (both with a dynamic libmagic.so, and libmagic compiled into file) but I need a little more time to compile the magic data (it requires running file on the target device to compile the magic data into a binary datafile). Once I have that figured out (either by compiling it or by creating a script to allow the users to do it themselves, I'll post it all in one go (need one more day).
Click to expand...
Click to collapse
Awesome! Thanks so much. The magic file isn't terribly important since you can grab one a lot of places and specify it with the -m parameter; keeps the file size down too if you were thinking of including it in the binary somehow!

Alright. I just wanted to thank you again for your work but I managed to make a static native compile (~440kb) of file-5.17 using my N7 and a kbox build environment I set up. It's available in my ARM Android Image Kitchen linked in my sig. Cheers!

osm0sis said:
Alright. I just wanted to thank you again for your work but I managed to make a static native compile (~440kb) of file-5.17 using my N7 and a kbox build environment I set up. It's available in my ARM Android Image Kitchen linked in my sig. Cheers!
Click to expand...
Click to collapse
@osm0sis - Can you please share how did you static compile this? I have setup the Ubuntu 13.10 image on Android using Complete Linux Installer (with all the necessary build tools setup).
I am able to compile this, but it's linking the shared library "libmagic.la".

Need to pass --static to ./configure and have a cross compiler set up, for ARMv7.

ericlnu said:
Need to pass --static to ./configure and have a cross compiler set up, for ARMv7.
Click to expand...
Click to collapse
I don't think that's a valid option.
Code:
$ ./configure --static
configure: error: unrecognized option: `--static'
Try `./configure --help' for more information

amit.bagaria said:
I don't think that's a valid option.
Code:
$ ./configure --static
configure: error: unrecognized option: `--static'
Try `./configure --help' for more information
Click to expand...
Click to collapse
Nope, seems not ^_^. Sorry, it usually is with autoconf derived configure scripts.

amit.bagaria said:
@osm0sis - Can you please share how did you static compile this? I have setup the Ubuntu 13.10 image on Android using Complete Linux Installer (with all the necessary build tools setup).
I am able to compile this, but it's linking the shared library "libmagic.la".
Click to expand...
Click to collapse
Haven't tried it using a full Linux image on Android, just made it native compile in KBox2 on my N7'13 for awhile by editing in the gettext functions it was complaining about, and have since moved on to cross-compiling on my Windows 7 desktop in Cygwin using the NDK. There were still a few tricks to getting it to be a true static compile. It's all linked in my post here:
http://forum.xda-developers.com/showthread.php?p=54510825
My latest compile included in my AIK-mobile package is file 5.20. :good:

I don't suppose you could do a quick compile with the -fPIE flag? Trying to use this on 5.0 + and don't want to patch my linker.
Tried to compile it myself (it sort-of worked, file --version is operational) but when I try to actually use it for something my terminal borks out and switches to some weird font after about 20 warnings. (configure also seems to be using the -rpath flag apparently, linker complains about it) hmm
EDIT: I've got it working after some trial and error. If anyone requests I'll post it
EDIT 2: android 6.0's linker seems less forgiving than 5.0/5.1, need to compile without -rpath (so no just using ./configure) will keep posted

Can you please post the file binary you built with -pie ?

[Script]Automated (Unix) Build Environment Setup v1.1

Automated Build Environment Setup
Version: 1.1​
Download:
- When your in Unix, follow the installation instructions
- When your in Windows, bootup your Unix VM or reboot in a Unix environment
Description:
I made a unix script that sets up an kernel developing environment fully automated. The script includes the good version of java (since the packages are not available trough apt-get) and the i9001 Kernel source (with adjusted makefile, ariesve_defconfig and the wifi drivers). This script may save you a lot of time, after the script is finished you will have a ready-to-mod kernel source. This script should be only used on a x64 machine (x86 will result in a corrupt environment)
(total process make take upto 30 min, depending on your connection speed)
(Please remind that I'm a unix noob, so If you find an error or missing package or w.e. please let me know!)
Made it on Mint Linux, but it should work for Ubuntu also.
What actions does it perform?
- Installing Java
- Getting required packages
- Installing Repo
- Cloning Prebuilt Repository (compiler)
- Extracting I9001 Kernel Source
- Copying ARIESVE_DEFCONFIG to source root (you can also pull config.gz from your phone "adb pull /proc/config.gz" then unzip it and copy the config file to .config (cp config .config)
- Run xconfig from kernel source after setup (optionally)
Installation Instructions:
1. Open Terminal Emulator, enter the following commands:
-- "mkdir -p ~/envsetup"
-- "cd ~/envsetup"
-- "wget http://broodplank.net/files/auto_build_env_setup.tar.gz"
-- "tar xvfz auto_build_env_setup.tar.gz"
-- "chmod +x setup.sh"
2. Now enter this command to start the script:
-- "./setup.sh"
3. After the script is done you will have a working android kernel development environment
4. You can find additional patches and scripts in the 'Patches & Addons' folder, use it!
What to do with the kernel afterwards?
- To configure: chmod +x xconfig && ./xconfig
- To clean the source: make clean
- To compile (quick): make -j8
- To compile (extended): chmod +x compile && ./compile
the zImage will be placed in /arch/arm/boot/
Adding things such as governors/io schedulers can be learned by browsing commits of other kernel developers (on github)

Thanks Brood! Awesome. Going to try it as soon as i have internet again :silly:

Sot0 said:
Thanks Brood! Awesome. Going to try it as soon as i have internet again :silly:
Click to expand...
Click to collapse
You're welcome, I hope more people will attempt to build a kernel now

Nice work brood. I see you are an "Recognized Contributor"
Congratz!!

Amazing as usual

mrjraider said:
Nice work brood. I see you are an "Recognized Contributor"
Congratz!!
Click to expand...
Click to collapse
Thanks, and yes idd got promoted!
YMYA said:
Amazing as usual
Click to expand...
Click to collapse
hehe thnx

congrats on your promotion brood..your contribution and support is highly appreciated by many (including myself). ^_^
btw, thanks for this awesome tutorial..

m0e_03 said:
congrats on your promotion brood..your contribution and support is highly appreciated by many (including myself). ^_^
btw, thanks for this awesome tutorial..
Click to expand...
Click to collapse
Thank you , I do my best to keep my roms/tuts and apps up to date ^^
And you're welcome, Hope you can build your own kernel if you would like to.
(If someone tests it, please let me know the results, so far 50 downloads but no feedback, so I really don't know if it works the way it should)

Nice Im gonna try it keep up the good work

Released v1.1,
changelog:
Added HKTW Patch and Localversion hack (else wifi doesn't work by default when compiling) and some additional shell scripts for easier use.
get it from the hub

broodplank1337 said:
Released v1.1,
changelog:
Added HKTW Patch and Localversion hack (else wifi doesn't work by default when compiling) and some additional shell scripts for easier use.
get it from the hub
Click to expand...
Click to collapse
Still no internet access at home
Really Want to try it and will report as soon as possible

[GUIDE][HACK]Cross Compiling for OSX on Linux with AOSP

Hi Folks
I wasn't sure where this should belong but as it is a bit of an Hack this forum is probably the most appropriate
Introduction
This short tutorial will show you how to patch the Android Build System to allow you to cross-compile Android AOSP host tools ( adb, fastboot etc ) for OSX using a linux based machine. This is something Google said was impossible or at the very least unsupported.
Assumptions
You have a linux based machine and working copy of the AOSP source tree.
You can/have successfully compile(d) a full Android Release from this tree.
A basic idea of how the Android Build System works is beneficial.
Getting Started
I've set-up a git repository which contains a binary copy of the OSX SDK 10.6 and the apple-darwin10-gcc cross compiler. So first things first. open a terminal and set the root of the AOSP sources tree to the current directory.
STAGE 1: Copy the OSX SDK
Step 1.
Clone the repo with the SDK and toolchain
Code:
git clone https://github.com/trevd/android_platform_build2.git build2
Step 2.
Create /Developer directory at your filesystem root, this is a known location for the SDKs
Code:
sudo mkdir /Developer
sudo chown $USER.$USER /Developer
Step 3.
Copy and unpack the SDK package
Code:
cp build2/osxsdks10.6.tar.gz /Developer
cd /Developer
tar -zxvf osxsdks10.6.tar.gz
rm osxsdks10.6.tar.gz
cd - # back to aosp root
STAGE 2 : Swapping the Toolchain
This is where the fun begins :laugh:
The Android Build system has the majority of the infrastructure in place already to build for OSX, the only problem is that you need OSX to build for OSX. However we can remedy that with a couple of dirty hacks :laugh:.
The prebuilts/gcc/darwin-x86 directory contains a toolchain compatible with osx ( mach-o binaries ). We are going to swap this for a linux compatible ( elf ) executables.
Step 4:
Copy and unpack the elf compatible darwin cross toolchain
Code:
cp build2/i686-apple-darwin-4.2.1.tar.gz prebuilts/gcc/linux-x86/host
cd prebuilts/gcc/linux-x86/host
tar -zxvf i686-apple-darwin-4.2.1.tar.gz
cd - # back to aosp root
Step 5:
Remove the mach-o binaries and symlink the elf binaries in it's place
Code:
cd prebuilts/gcc
rm -rf darwin-x86
ln -s linux-x86 darwin-x86
cd - # back to aosp root
Step 6:
We also need to replace the mach-o version of the ccache executable which live in the prebuilt/misc directory
Code:
cd prebuilts/misc
rm -rf darwin-x86
ln -s linux-x86 darwin-x86
cd - # back to aosp root
STAGE 3: Patching the build system .mk files
We need to patch a couple of files in the build directory namely the build/core/combo/HOST_darwin-x86.mk the main crux of this is swapping the ar tool for libtool so static libraries can be created without error.
Code:
patch -p1 < build2/build.patch
If the patch has been applied successfully you should see the following
Code:
patching file system/core/adb/Android.mk
patching file build/core/combo/HOST_darwin-x86.mk
patching file build/core/combo/select.mk
patching file build/core/envsetup.mk
patching file build/envsetup.sh
You are now ready to cross compile!! :good: ..... well not quite, but nearly.... here's why!
The Android Build System will attempt to build both the Target and Host files for most modules so I'd advise using a lunch option which already has a full target built for it or alternatively you can build the generic sdk using the following commands at the AOSP source tree root.
Code:
. build/envsetup.sh
lunch sdk-eng
make sdk
This will stop target dependency errors occurring when you build individual modules.
NOW we're ready to cross compile.
STAGE 4: Building Modules
At present module build is very much a piecemeal process. To build adb for example we need to build the dependencies first. This is not too onerous as most host modules have very few dependencies.
Building adb
adb has dependencies on the following libraries
Code:
external/zlib
external/openssl
system/core/liblog
system/core/libcutils
system/core/libzipfile
I've found the easiest way to compile the dependencies is to navigate to each directory in turn an use to "mm" build system command to compile the individual module. the commands I run to compile adb are as follows.
From AOSP Source Root
Code:
cd external/zlib
USE_DARWIN=true mm -j8
cd ../openssl
USE_DARWIN=true mm -j8
croot # go back to the AOSP root
cd system/core/liblog
USE_DARWIN=true mm -j8
cd ../libcutils
USE_DARWIN=true mm -j8
cd ../libzipfile/
USE_DARWIN=true mm -j8
cd ../adb
USE_DARWIN=true mm -j8
All being well you should now have and adb binary located at out/host/darwin-x86/bin/adb. running the file command on this binary should produce the following output
Code:
adb: Mach-O executable i386
Conclusion
Although this method is a little rough and ready, it should produce the desired results if you need to cross compile for OSX. The eventual goal would be to compile a full OSX Android SDK on linux in a similar manner to the way the windows-sdk is currently compiled. This requires more investigation as compiling the windows sdk on linux employs a little bit of trickery on the part of the build system.
Final Notes and FAQs:
Why can't I just type make <module> from the root?
Doing this triggers building of additional modules such as LLVM and clang which are to deployed out/host/darwin-x86/bin the build system then attempts to use binary later on. These are obviously built for the Mach-o architecture and as such are incompatible with the linux. This results in a build error which can and would be resolved by the above mentioned trickery ( see conclusion )

I use OSX binaries (along with Windows and my native Linux) in one of my projects. Thanks, I have always relied on finding compiled binaries elsewhere. Lack of an OSX aapt held up an update at one point.
One of those things that you don't really use until you need it, but I will try to remember to give it a shot. I don't have any doubt that it works.

mateorod said:
I use OSX binaries (along with Windows and my native Linux) in one of my projects. Thanks, I have always relied on finding compiled binaries elsewhere. Lack of an OSX aapt held up an update at one point.
One of those things that you don't really use until you need it, but I will try to remember to give it a shot. I don't have any doubt that it works.
Click to expand...
Click to collapse
Thanks. Yes this really is an edge case. Hopefully It will help some folks out.
Regarding aapt in particular.... It's perfectly possible to build aapt, however, we need to do some slight of hand with the clang and clang++ executables as libpng on which aapt depends uses these 2 binaries as part of it's build process.
Here's the build list and the clang trick if you want to try it some time.
Code:
build/libs/host
external/expat
external/zlib
system/core/liblog
system/core/libcutils
mkdir out/host/darwin-x86/bin
cp out/host/linux-x86/bin/clang out/host/darwin-x86/bin
cp out/host/linux-x86/bin/clang++ out/host/darwin-x86/bin
external/libpng
frameworks/base/libs/androidfw
frameworks/native/libs/utils
frameworks/base/tools/aapt
I started off with a clean out/host/darwin-x86 directory so I didn't miss any dependencies.
like I mentioned the clang "swap out" is something the make win_sdk option does automatically so with it a little more research I should be able to get the mac build to do the same but you'll have to "fill yer boots" with the ghetto method for now
For reference here's a link to the sdk building instructions http://tools.android.com/build which describes how to cross compile the windows sdk on linux ( in case anyone was wondering what the hell i'm on about)

My use case has come up
I will be cross-compiling for OSX today...specifically with aapt in mind. I will report back, but I fully expect it to work as described.

mateorod said:
My use case has come up
I will be cross-compiling for OSX today...specifically with aapt in mind. I will report back, but I fully expect it to work as described.
Click to expand...
Click to collapse
Cheers Man!
Hopefully no bitrot has crept in since april and now. I know I've changed my OS version since to Lubuntu 13.04, not like the OS version really matters any.
mateorod said:
but I fully expect it to work as described.
Click to expand...
Click to collapse
Then you Sir, are either Drunk or a Fool! LOL Keep expectations Quantum and only decided when the result is observed a'la Schrodinger Cat

okay...So I was trying to compile SlimRom (so as to get an OSX aapt binary with the SlimRom changes) and things did not necessarily go as planned. There were enough changes to the SlimRom android_build that your build/build.patch does not apply cleanly. I spent some time and tried to modify the patch so that it would work for both SlimRom, AOSP and probably others, but each android_build repo has some differences in surrounding the HOST_AR line, so commenting that just was not portable between flavors. Not cool.
Anyway, turns out that this method does not quite work out of the box for non-AOSP versions (not that you claimed that it did). I got some unfamiliar errors related to (I believe) some OSX toolchains. But in both times I tried this, I actually had to pretty immediately swap out of that flavor and so I was unable to do much debugging. (I keep all the flavors I build {CM, AOKP, SlimRom, PAC, PA, OpenPDroid, etc, etc, etc} all layered in one android/system/jellybean directory. It saves a ton of space, but only allows me to do one thing at a time.)
So the only feedback I have is nothing...I even formatted my hard drive in-between and forgot to put up a paste, so the errors are currently lost to history.
Things that I noticed, for better or worse
You recommend putting the SDKs in the root dir. I believe the documentation is recommending the Developer be placed in home (as per the SDK/ADT bundle docs).
You might want a
Code:
mv android_platform_build2 build2
line. I normally wouldn't bother, but it looks like you are trying to post a line-by-line guide.
I would put the recommendation that a full build be available to the out folder (or a built generic sdk) right at the top, since it is a preliminary step. I had to revert my handwritten changes, then build, then reapply the changes and rebuild since I thought it needed a clean out dir.
Did you have any trouble with git reverting the toolchain swap? On two separate machines, I had to go so far as to delete .repo/projects/prebuilts/gcc/* and prebuilts/gcc/darwin-x86/arm/arm-eabi-4.6. It kept complaining of that the project in the .repo folder was a bad match. No amount of git trickery (which I am not terrible at) let me back out more easily.
I am willing to try again...but I have some other small things to attend to first. It is an admirable hack you have here sir. I will return to it soon and report back once more.

mateorod said:
okay...So I was trying to compile SlimRom (so as to get an OSX aapt binary with the SlimRom changes) and things did not necessarily go as planned. There were enough changes to the SlimRom android_build that your build/build.patch does not apply cleanly. I spent some time and tried to modify the patch so that it would work for both SlimRom, AOSP and probably others, but each android_build repo has some differences in surrounding the HOST_AR line, so commenting that just was not portable between flavors. Not cool.
Anyway, turns out that this method does not quite work out of the box for non-AOSP versions (not that you claimed that it did). I got some unfamiliar errors related to (I believe) some OSX toolchains. But in both times I tried this, I actually had to pretty immediately swap out of that flavor and so I was unable to do much debugging. (I keep all the flavors I build {CM, AOKP, SlimRom, PAC, PA, OpenPDroid, etc, etc, etc} all layered in one android/system/jellybean directory. It saves a ton of space, but only allows me to do one thing at a time.)
So the only feedback I have is nothing...I even formatted my hard drive in-between and forgot to put up a paste, so the errors are currently lost to history.
Things that I noticed, for better or worse
You recommend putting the SDKs in the root dir. I believe the documentation is recommending the Developer be placed in home (as per the SDK/ADT bundle docs).
You might want a
Code:
mv android_platform_build2 build2
line. I normally wouldn't bother, but it looks like you are trying to post a line-by-line guide.
I would put the recommendation that a full build be available to the out folder (or a built generic sdk) right at the top, since it is a preliminary step. I had to revert my handwritten changes, then build, then reapply the changes and rebuild since I thought it needed a clean out dir.
Did you have any trouble with git reverting the toolchain swap? On two separate machines, I had to go so far as to delete .repo/projects/prebuilts/gcc/* and prebuilts/gcc/darwin-x86/arm/arm-eabi-4.6. It kept complaining of that the project in the .repo folder was a bad match. No amount of git trickery (which I am not terrible at) let me back out more easily.
I am willing to try again...but I have some other small things to attend to first. It is an admirable hack you have here sir. I will return to it soon and report back once more.
Click to expand...
Click to collapse
Hi
Thanks for this, It sounds like you've suffered an exercise in frustration there. I wasn't aware that "SlimRom" had a different aapt ( just out of general ignorance and not having paid any attention )
SDK - My Tree last time I used this was /Developer directory in the root - I think It comes from what the toolchain is expecting, I just gave it what it wants
mv android_platform_build2 build2 - Yep I did mean that it's the git clone line which wants changing
Code:
git clone https://github.com/trevd/android_platform_build2.git build2
SDK Recommendation - I shall move that to the top, even though it is already in there, It should probably be highlighted better and possible it's own "Stage"
Reverting the toolchain - Ahh , It appears I work slightly different from most in this respect. I have a general mistrust of SCM's ( I lost too much code on too many different SCM's, Probably through my own inability to use them correctly but ) what I do to revert to change is
Code:
cd prebuilts/gcc/darwin-x86/host/
rm -rf i686-apple-darwin-4.2.1
repo sync i686-apple-darwin-4.2.1
You can do this "trick" on any project in the source tree it's only on rare occasions where I screwed up badly that I have to delete anything in .repo/projects but I also have my distro in their own individual directories with there own full git trees, which is a massive waste of space and has a ton of redundancy due to the AOSP repositories being mirror by every single one but switching between them is a lot easier
If SlimRom's changes are localized to aapt, I'd be more inclined to drop it into the AOSP build and try that... If you have a link to slimrom's frameworks/base repository I'll grab it and try it myself.
On a final note there's a "full version" of the HOST_darwin make file in the build2/core/combo directory the changes to envsetup.mk and select.mk are minimal and can easily be applied manually. You don't need to patch the adb makefile if your not building it.
Again Thanks for the feedback

[Scripting][ARM][Installer][Port] Python (static) 2.7.9 and 3.4.2

(Check out https://www.python.org for information on what python is.)
Yes you read that correctly, static python for Android! It took a while to figure out how to get this compiled, but I finally did it. I present you with an installer and a little bit of testing. The second post contains information on compilation and the sources. Btw, any cool python scripts are welcomed!
INSTALLATION
Recovery flash installers are attached that will install python 2.7.9 and/or 3.4.2 to /system/pythonX.X.X. You can install both if you want. Installation size is about 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
After installation, the python director(ies) in /system will contain the static python binary, some scripts, and a bunch of modules and documents. Separate scripted executables will be installed to /system/bin/python or /system/bin/python3 depending on which one is installed. These basically just set the PYTHONHOME environment variable and execute the python binary.
TESTING
When you boot up to Android after installation, you should be able to just open up a terminal and fire off some python commands.
Test using python 2.7.9:
Code:
python -c 'print "Hello World!"'
Test using python 3.4.2:
Code:
python3 -c 'print("Hello World!")'
You can also write scripts shelled with python:
Code:
#!/system/bin/python
print "Hello World!"
Make sure to set them as executable with "chmod +x".
Adjust your screen brightness via /sys (root required, tested on Galaxy Nexus):
Code:
python -c 'f=open("/sys/devices/omapdss/display0/backlight/s6e8aa0/brightness","w"); f.write("40"); f.close()'
A cool script I made to tweak file system I/O like rq_affinity, rotational, etc for I/O blocks:
Code:
#!/system/bin/python
import os,re,sys
list=[]
# find all directories containing rq_affinity
for roots, dirs, files in os.walk('/sys'):
for file in files:
match=re.search(r'\S+/rq_affinity',os.path.join(roots,file))
if match:
list.append(match.group().replace('rq_affinity',''))
# write specific values to files in each directory found before
for dir in list:
for name in 'rq_affinity', 'rotational', 'read_ahead_kb', 'nr_requests', 'iostats', 'nomerges', 'add_random':
try:
f=open(dir+name,'w')
if name is 'rq_affinity': f.write('1')
elif name is 'read_ahead_kb': f.write('512')
elif name is 'nr_requests': f.write('512')
else: f.write('0')
f.close()
except IOError:
sys.stderr.write('Problem writing to ' + dir+name + '\n')
Or execute "python" or "python3" without any parameters to open up the interpreter and go from there. When you start for example "python" (python 2.7.8), you should see something like:
Code:
Python 2.7.8 (default, Dec 2 2014, 05:15:18)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
***I haven't fully tested this yet, so let me know how it goes. I know basic commands work, but there are still some complexities in the compilation that may need to be figured out.***
(update) -- Install Extra Packages/Modules via "easy_install" -- (root required)
1) In a shell, remount /system read-writable with:
Code:
mount -o remount,rw /system
2) Change directory to /system/pythonX.X.X, i.e.,
Code:
cd /system/pythonX.X.X
3) Execute easy_install followed by a package name, i.e.,
Code:
./easy_install [i]package_name[/i]
4) Let it download and install! Then test it out.
5) Remount /system read-only with:
Code:
mount -o remount,ro /system
Also, to remove a package, execute:
Code:
./easy_install -m [i]package_name[/i]
rm -r ../lib/pythonX.X/site-packages/[i]package_name[/i]*.egg
*** Flashing the resolv.conf patch may be required to make an internet connection to download modules.
Also, a few things need to be done to get easy_install working:
1) For python 2.7.8 or 2.7.9 you need to link python2.7 to python like this:
Code:
ln -s /system/python2.7.9/bin/python2.7 /system/python2.7.9/bin/python
I'll make sure this is automatically done in future installs.
2) SSL certificates need to be added to "/etc/pki/tls/certs/" to use SSL with easy_install.
Code:
mkdir -p /etc/pki/tls/certs
curl http://curl.haxx.se/ca/cacert.pem -o /etc/pki/tls/certs/ca-bundle.crt
ISSUES
The python installations have the following modules compiled in with external dependencies such as libraries not included at this point:
Code:
--------------PYTHON 2.7.8-2.7.9----------------------
_hashlib _multiprocessing _ssl
_testcapi bz2 crypt
dbm
Code:
--------------PYTHON 3.4.2-----------------------------
_crypt _dbm _decimal
_hashlib _multiprocessing _ssl
_testcapi
I will look into this more.
As @bubbleguuum points out, name resolution with python 3.4.2's urllib wasn't working. I found a workaround by adding the line
Code:
options single-request-reopen
to /system/etc/resolv.conf. This still needs some testing but seemed to work for me. I've included a recovery flash zip to patch /system/etc/resolv.conf if the line is not there since this is a root operation.
UPDATES
* 2014-12-02 * Compiled in more modules for each installation. This makes the installation a bit bigger, but it's worth it. You get a more complete python! To get stuff like help functions and math and readline modules, you need the larger installations linked below under "DOWNLOADS".
* 2014-12-03 * Thanks @cybojenix for pointing out some incapatibility issues with the original edify installer I was using. I have now updated the installers to use SuperSU's great non-edify sh-script installer.
* 2014-12-05 * Some updates to the installers to include more modules, plus size reduction for python 3.4.2. Modules added to python 2.7.8 : _bsddb _ctypes, _ctypes_test, _hotshot, _json, _lsprof, _sqlite3, future_builtins, and ossaudiodev, easy_install, pip(broken for now). Modules added to python 3.4.2: _bz2, _ctypes, _ctypes_test, _gdbm, _json, _lsprof, _opcode, _sqlite3, _testbuffer, _testimportmultiple, ossaudiodev, xxlimited
* 2014-12-05 * Some big updates to the installer to include almost the rest of the modules that wouldn't compile. Also easy_install is working on my end with these new experimental builds. With a little effort, should be able to get pip working as well. I had to use some hacky flags to get it to compile, ignore unresolved-symbols and such, so it definitely needs some testing. Another thing I did was tweak the install script to write over previous installations from these installations. I recommend downloading the experimental builds if you read this. The worst thing that might happen is one of the new modules I've included won't work completely.
* 2014-12-14 * Added python 2.7.9. Trimmed the installation sizes down to the much more reasonable 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
* 2014-12-24 * Attached a recovery flash.zip to patch resolv.conf to try and fix a urllib name resolution error.
* 2014-12-26 * A couple updates: Tweaked python installations so that modules looks for "/system/bin/sh" instead of "/bin/sh" when needed. Also a few "#!" corrections for scripts in the python bin directory. This shouldn't break anything from what I've tested, only make Android python more capable, but I'll keep the old installations attached for now just in case since the modifications involved using "sed". The second update is nice. I've added a little section on installing extra modules with easy_install(included and working with all installations).

Compiling
The environment I compiled in was a debian "wheezy" image mounted as a loop device on Android. The reason I compile this way is so I have full access to the arm environment tools, gcc compiler, etc. without having to go through the trouble of using a cross-compiler (which doesn't work in this case from my testing with uClibc, or eglibc. Code sourcery's might work but I didn't feel like booting an x86 linux installation to try). You can recreate the same environment pretty easily by using either Complete Linux Installer or Linux Deploy apps. I use both.
To get a static python compiled, I first downloaded the following packages with APT:
Code:
[i]apt-get build-dep python python3[/i]
build-essential gcc make # building tools
binutils-gold # awesome gold linker
zlib1g-dev # zlib
libreadline-dev, libncurses5-dev # readline, curses
libbz2-dev # bz2
libsqlite3-dev # sqlite3
python-bsddb3, python3-bsddb3
libgdbm-dev # gdb
libssl-dev # ssl
python-tk, python3-tk # tkinter
libdb-dev # db
python-gdbm python-bsddb3
libffi-dev # _ctypes
tcl8.6-dev # tkinter
libx11-dev # tkinter
libmpdec-dev # decimal
Then I manually installed binutils-gold by unpacking the rpm for armv7h (this isn't available with APT yet). You can get it here though. (Update: this should be available in the debian repositories now.)
For static compilation, you need to make some changes to Modules/Setup from the python source directory. Specifically, you need to add
Code:
*static*
to the top of the file. Then you need to uncomment any commented modules you would like compiled in. When you compile and see a list of failed modules, search for each of these in Modules/Setup and uncomment them. For example, change
Code:
#math mathmodule.c _math.c
to
Code:
math mathmodule.c _math.c
. Then recompile. There may be some shared dependencies and packages as well you need to figure out to get them to compile. Take a look at the attached "Setup" files for more detail.
The actually compile steps look like this:
(1)
Code:
./configure --build=arm --prefix="$PWD"/out LDFLAGS="-static -static-libgcc -Wl,--unresolved-symbols=ignore-all -Wl,--export-dynamic" CPPFLAGS=-static CXXFLAGS=-static CFLAGS="-Os -static" LDFLAGS=-static LD=ld.gold
(2) Modify the Modules/Setup file.
(3)
Code:
make clean; make install
After compiling, there are some things you can do to make your installation in "./out" smaller. Strip the large binaries in "out/bin", i.e.
Code:
strip -s ./out/bin/python2.7
Also remove the python archive library, *.o's, etc. with:
Code:
make clean
This might seem kind of weird, but it works and cleans out unnecessary files from your installation.
**Make sure to backup your Modules/Setup file if you run "make distclean". I lost my 3.4.2 Modules/Setup file after doing this. :silly: Now I gotta recreate it, dammit.
**I should also mention that compile time is very fast given the amount of data generated. It takes about 10 minutes to get python and all it's modules compiled on my Galaxy Nexus.**
:EDITS:
: Updated some configure parameters. Removed unnecessary, extra "./configure". Also added some updates to the included "Setup" files.
: Hacky update to "./conifgure" to ignore unresolved symbols, export dynamic. This allows certain modules to be compiled which have shared dependencies.
: Update to trimming down the installation size after "make". Use another "make clean".
Downloads - Extract in the source directory
Setup (python 2.7.8)
Setup (python 3.4.2)
Setup (Experimental) (python 2.7.8)
Setup (Experimental) (python 3.4.2)

Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference

cybojenix said:
Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference
Click to expand...
Click to collapse
Thanks for your reply! I've changed the installer to use SuperSU's no-edify sh-scripted updater. I tested on my device, but let me know if there are any issues though. I kept it pretty basic.
I added some stuff on the source modifications and compilation to the 2nd post. The only file I've actually modified so far in the source is the generated Modules/Setup file, and I've included the one I used for python 2.7.8. Unfortunately, I over-cleaned the python 3.4.2 directory, which deleted my modified Modules/Setup there, so I'll have to add it later when I re-edit it.
That's a great idea on pip/easy_install. I did download the pip_installer and tried out installing on the static python I made. Got some errors though pointing to a few modules I haven't gotten compiled into the static python installation (listed in the OP near the bottom, specifically _ctypes). I'll see what I can do about that. The line for compiling the module might just need to be added to Modules/Setup. Hopefully that's the case.

@7175
Thank you very much for these binaries.
There is however a problem: name resolution (DNS) doesn't seem to work at all (with both python 2 and 3 downloads).
All attempts to use urllib.request.urlopen('http://somehost.com') fail with "<urlopen error [Errno -2] Name or service not known>" (running python as root but it doesn't matter, and a rooted Nexus 4 running 4.4.4 stock ROM):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 455, in open
response = self._open(req, data)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 473, in _open
'_open', req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1202, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1176, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
Click to expand...
Click to collapse
it works if replacing somehost.com by its ip address.
To reproduce above stack trace with python 3:
import urllib.request
urllib.request.urlopen('http://www.google.com')
while this work:
urllib.request.urlopen('http://173.194.45.229')
nslookup and general name resolution work anywhere else:
[email protected]:/ # nslookup www.google.com
nslookup wwwgoogle.com
Server: 8.8.4.4
Address 1: 8.8.4.4 google-public-dns-b.google.com
Name: wwwgoogle.com
Address 1: 2a00:1450:4007:80c::1014 par03s15-in-x14.1e100.net
Address 2: 173.194.45.242 par03s15-in-f18.1e100.net
Address 3: 173.194.45.240 par03s15-in-f16.1e100.net
Address 4: 173.194.45.244 par03s15-in-f20.1e100.net
Address 5: 173.194.45.241 par03s15-in-f17.1e100.net
Address 6: 173.194.45.243 par03s15-in-f19.1e100.net
Click to expand...
Click to collapse
This issue has probably something to do with how python was compiled to do name resolution. I remember vaguely a similar issue
on Ubuntu 12.04, with a ffmpeg compile that would always fail to resolve hostnames in URLs. No sure what the exact fix was but it had something to do with the libc and its name resolution mechanism. Or maybe something ipv6 related ?

@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
EDIT: Hmm, looks like re-compiling with "--disable-ipv6" didn't fix the problem. Will have to look more into this issue and hold off on uploading ipv4-only python3 since it didn't fix it. In the meantime, I've attached a recovery flash.zip to the OP to add the above mentioned line to /system/etc/resolv.conf if needed, which seems to fix the issue for me.

7175 said:
@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
Click to expand...
Click to collapse
Thank you for looking into this so fast.
My intended usage is for running the python binary from a regular non-root app (which works great!), so editing resolv.conf is not an option.

I've been researching that DNS resolving issue and could not find a clear explanation on why it fails on Android and not on other systems, and why exactly the added line in resolv.conf fixes (or rather workaround) it.
All seems to point to socket.getaddrinfo() failing for an unknown reason.
Is Python compiled with HAVE_GETADDRINFO defined ? If that's the case it uses the libc definition of getaddrinfo, otherwise
it uses an emulation function found in getaddrinfo.c.
My guess is that it is the latter, and what is causing this issue on Android.
It's probable this issue has already been solved on Android since other Python binaries exists, but Googling around do not give
much answer.

I get this error when trying to install anything with easy_install (using /system/python3.4.2/bin/easy_install-3.4 because /system/python3.4.2/easy_install doesn't seem to exist like in the OP):
Code:
# ./easy_install-3.4 feedparser
Searching for feedparser
Reading https://pypi.python.org/simple/feedparser/
Download error on https://pypi.python.org/simple/feedparser/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'feedparser' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or download links found for feedparser
error: Could not find suitable distribution for Requirement.parse('feedparser')
i got curl and the certs installed in /etc/pki/tls/certs/ as per instrucions in the OP, any help?

Thoughts on PIL?

Hmm... new problems here. Android Lollipop have no /system/etc/resolv.conf by-default and urllib patch is no more works.
How it can be fixed?

http://qpython.net/index.php

Where I can find source code and related documents for project

Pratik Raj said:
Where I can find source code and related documents for project
Click to expand...
Click to collapse
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?

hackintosh5 said:
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?
Click to expand...
Click to collapse
Idk about OP, but I can surely post my source for python 2.7 on arm/arm64. I compiled it like a year ago, but it wasn't too bad, just required a bunch of patches and code for dlopen

I just want to say thank you so much.

[GUIDE]Cross Compiling C/C++ for ARM on Linux

I wrote this guide to make it easy for everyone to be able to cross compile c/c++ for android, the easy way that I know. I admit, I am a total n00b to c/c++. I started learning to cross compile when I got my new nexus 9 for christmas. I, for some reason, couldnt get busybox installed on it. So, I had to manually compile and load it on there. After reading a hundred tutorials and none of them working, I got frustrated. Finally, I compiled a working version. And it was SO EASY. Anyway, Im specifically giving instructions for Ubuntu 14.04/14.10. First thing you need to do is download a package called binutils-arm-linux-gnueabi.
- sudo apt-get install binutils-arm-linux-gnueabi
- sudo apt-get install binutils-aarch64-linux-gnu (for aarch64, or arm 64bit, or armv8)
- sudo apt-get install binutils-arm-linux-gnueabihf (for armhf)
To compile busybox for your platform, you must download the source package you want to compile from http://busybox.net/downloads/. Then extract it
- tar xvf busybox.tar.gz
Then, cd into your root busybox folder.
- export ARCH=arm
- export CROSS_COMPILE=arm-linux-gnueabi- (dont forget the trailing dash(-))
This will setup your variables for compilation. Then
- make menuconfig (if you dont get a config menu, you need to "sudo apt-get install libncurses5-dev")
Now, go into the busybox settings -> Build options. Select the option "build busybox as static executable". You can select/deselect options by hitting the space bar. [ESC] key will take you back. Make sure to save your new configuration. Its almost time to compile! One last thing. If you want to add your name to it, go into the Makefile in the root busybox folder. At the top of the file, where it says EXTRAVERSION = .git, you can add something like -bynarie or whatever you want. That way when you run busybox on the terminal emulator on android, it will print out something like "BusyBox v1.24.0-bynarie". OK! Time to compile. Really simple:
- make install
This will compile every applet, and put everything in a folder called _Install. You will have busybox and busybox_unstripped in the root directory. Please, be aware that this busybox puts an applet named "su" in the bin folder in the _install folder. So, DO NOT COPY THE SU APPLET INTO YOUR ANDROID BIN FOLDER, OR YOU WILL LOSE ROOT!!! Now, to verify it compiled to the right architecture, do "file busybox" at the term and it should spit out something like "ELF 32bit ARM executable". If this is the case, congrats. We have successfully compiled busybox for android arm!! Copy the needed files over to your device and set permissions and you are done. The suggested way to move the busybox binary to your device is as follows:
- adb push busybox /data/local/tmp/busybox
- open adb shell and do "chmod 755 /data/local/tmp/busybox" or chmod it on your linux box before pushing.
- open file manager on device and move busybox to /system/xbin
- If properly done, should work.
For other nix programs like tar, the procedure is as follows:
- Open term, cd into root source folder
- ./configure CC="arm-linux-gnueabi-gcc" CPP="arm-linux-gnueabi-cpp" --host=arm-linux-gnueabi
- make
The CC variable is your C compiler command and the CPP variable is the C preprocessor, if you need to add a C++ compiler just add CXX="arm-linux-gnueabi-g++" but TAR is specifically C only. You can add all three variables I would assume to be safe. Most of the time, the readme or install documents will give you some guidance on cross compiling. But, this is how I successfully compiled TAR for arm and aarch64.
If you want to compile small/single source file c/c++, you will be using the same tools, but in a different way. Cd into your source file directory, and depending on which type of source it is (c or cpp), you will do the following:
- arm-linux-gnueabi-gcc helloWorld.c -static -o helloworld.out (for C)
- arm-linux-gnueabi-g++ helloWorld.cpp -static -o helloworld.out (C++)
Yep, its that easy. I cant guarantee 100% this will work on everything, but its a good starting point. And you dont even have to fool with the Android NDK. I find that the android ndk would mostly benefit "Apps" that need to integrate c/c++ code into them, not little console apps run from the terminal emulator. I hope someone finds this guide useful and if anyone has any problems trying to get somethin to work, you can respond and I will do my best to help.

thanks