Database Info

[DEV] Kernel development HOWTO and Interactive menu

I havent yet found a simple guide for compiling kernels. Some of them assume too much, and some are just outdated. So I thought I'd write my own for devs/budding devs. Here you go!
Note:
This is not a guide for newbies. It's a dev guide for devs.
Research before asking questions, please
For The Menu driven interactive kernel build script, see Post #31
I will be developing this guide as I go, so it will be incomplete initially, or lacking in detailed explanations.
Essentials:
Ubuntu Box (By this I mean a PC with a Ubuntu installation, not a live CD)
A toolchain-Either the Android NDK, or your own toolchain
HTC Desire GB/Froyo source from htcdevs.com, or sources from github
Familiarity with the linux shell and basic linux commands.
The will to learn
First things first,
1. Getting the sources
The HTC Desire source is available from two kinds of resources-you can either get it from htcdevs.com (official HTC Dev site), or from source code uploaded from someone else. For the purpose of this tutorial, I'll assume we're working on the official HTC GB source code. So download bravo_2.6.35_gb-mr.tar.gz from htcdevs.com.
2. Setting up the compilation box and preparing source code
2.1 Install some essential linux packages from the Linux terminal:
Code:
sudo apt-get install libncurses5-dev
2.2 Extract the source code
The file you downloaded is a tar archive (like a zip file), so you need to extract it to a convenient location. Let's hit the linux shell-open a terminal window in linux (Accessories->Terminal)
Type:
Let's go to our home directory:
Code:
cd ~/
Now, create the directories for our kernel compilation box.
Code:
mkdir -p ~/android/kernel
Now you need to copy the tar.gz file from wherever you downloaded it to, to this dir.
Extract the archive:
Code:
tar -xvf ~/android/kernel/bravo_2.6.35_gb-mr.tar.gz
cd ~/android/kernel/bravo_2.6.35_gb-mr
Now we can view the extracted files within the directory ~/android/kernel/bravo_2.6.35_gb-mr/
2.3 Set up the toolchain
A toolchain is a set of programs which allow you to compile source code (any source code, not just kernels). The toolchain is specific for the processor and hardware, so we need a toolchain specific for Android and especially the Desire. If you're a semiadvanced-pro user, you may consider compiling your own toolchain (See theGanymedes' guide for doing so). If compilation of kernels is all that you require, fortunately for you, there is an easy way-the Android NDK - v7 (latest as of now) is available here
Get the NDK for Linux - android-ndk-r7-linux-x86.tar.bz2
Code:
mkdir -p ~/android/ndk
Now copy the NDK file to ~/android/ndk
Whenever I say copy, you have to manually copy the file with any file manager. Nautilus comes with Ubuntu, and Dolphin with Kubuntu. You may also use the shell of course with
Code:
cp [sourcefile] [destination]
Extract it:
Code:
tar -jvxf android-ndk-r7-linux-x86.tar.bz2
Now add the path for your toolchain to the env variable:
Code:
gedit ~/.bashrc
At the end of the file, add this line:
Code:
PATH=$PATH:~/android/ndk/android-ndk-r7-linux-x86/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin
3. Setting up kernel parameters
Kernels are compiled with a program called gnu make, and use a set of configuration options specified within a file called Makefile.
A vital point to note is that kernels are compiled with a program called gcc (basically the gnu C compiler), and our NDK itself has its own optimized version of gcc. While compiling, we're actually cross compiling it (meaning compiling a binary package on a system which is different from the actual system which is meant to run it- you're compiling it on your PC while it's actually meant to run on your Desire)
This means that when you compile it, you have to make sure that you compile it with the NDK's version of gcc instead of the system version. Otherwise you end up with a kernel meant to run on your pc, duh! Specifying which gcc to use is by the CROSS_COMPILE variable. You can set it up with this command:
Code:
CROSS_COMPILE=arm-linux-androideabi-
Note the hyphen (-) at the end, and do not forget to include it! At compilation time, system will actually use this variable to find all the programs it needs. Eg: The path for gcc will become arm-linux-androideabi-gcc
We can compile kernels with many different options, like with ext4 support, or without; ext4 support as part of the kernel zImage (in which case it makes the kernel larger), or as a loadable module (of the form somename.ko, which is loaded at init.d/init.rc with the command insmod modulename.ko)
We specify the exact options we require with the help of a useful configuration program called menuconfig (which as the name suggests, is a menu for configuration of make options).
An important thing to note is that as far as kernel compilation is concerned, there are a vast amount of options to setup, and unless you're thorough with kernel compilation, you wont be able to set up the options correctly and get your kernel to boot. Fortunately for us, the kernel source already comes with a default set of parameters which can be easily set up.
Note that all make commands must be executed within the directory bravo_2.6.35_gb-mr. Let's go there now:
Code:
cd ~/android/kernel/bravo_2.6.35_gb-mr
make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- bravo_defconfig
This produces a .config file (used by the menuconfig) containing essential parameters to produce a booting kernel for the Desire.
Note: There is a simpler way to get the basic .config file, and this is to get it from a running kernel built by someone else. You can extract the .config from a running kernel with these commands:
Code:
cd ~/android/kernel/bravo_2.6.35_gb-mr
adb pull /proc/config.gz
zcat config.gz > .config
Now we can open menuconfig and add anything we need in addition.
Code:
make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- menuconfig
You can view the huge amount of options available in menuconfig.
You can add ext4 support for example (See image above)
Once you're done choosing options, you can exit menuconfig.
4. Compiling it
This is simple. The basic command is:
make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- -j10
The -j10 specifies the number of jobs to execute per operation. I can usually go upto 50 on my Quad core CPU. Beware, this can bring a slow CPU to a crawl and freeze up linux itself.
During compilation, you will see all sorts of messages, which may include warnings too. In most cases, its safe to ignore warnings. If there are errors, the compilation will stop, and you will have to fix the issues.
5. Distributing your kernel to users
At the end of compilation, it generates files named zImage, and various .ko files.
You have to copy them from their default location to a zip file. The best way is to use my variant of koush's Anykernel, and copy the files to it. Then, you can zip the whole folder and lo and behold-you have your flashable kernel zip which you can distribute to others.
You can also remove the zImage and the modules from /system/lib/modules of any kernel zip available with you, and copy over your files to it, at the correct location.
So, let's say that you have extracted an existing kernel zip to the location ~/flashable
The file structure should be like this:
Code:
|-- kernel
| |-- dump_image
| |-- mkbootimg
| |-- mkbootimg.sh
| |-- unpackbootimg
| `-- zImage
|-- META-INF
| |-- CERT.RSA
| |-- CERT.SF
| |-- com
| | `-- google
| | `-- android
| | |-- update-binary
| | `-- updater-script
| `-- MANIFEST.MF
`-- system
`-- lib
`-- modules
`-- bcm4329.ko
8 directories, 11 files
I've included my flashable zip directory along with this post. Download file kernel_flashable.tar.bz2.zip to ~/
Code:
cd ~/
tar -jvxf kernel_flashable.tar.bz2.zip
This will create the directory structure outlined above.
Now after every compilation of the kernel, execute these commands from where you executed make:
Code:
cp arch/arm/boot/zImage ~/kernel_flashable
find . -name '*ko' -exec cp '{}' ~/kernel_flashable/system/lib/modules/ \;
cd ~/kernel_flashable
zip -r mykernel ./
This will create mykernel.zip at ~/kernel_flashable. You can distribute this to your users to flash. Make sure you edit updater-script before though

Common errors and other stuff
Ok, post #1 was simple stuff. Now, supposing you get errors while compiling. Post #2 is about that, and ups the level of knowledge a bit..
Some kernel compilation errors:
Treat warnings as errors-Solved by removing the string "-Werror" from all Makefiles of the file which failed to compile. Some people had said that the real error (Array out of bounds warning) was because of gcc optimizations. But putting -O2 to -O0 didnt do a thing.
No of jobs - ought not to exceed 50.
"warning: variable set but not used [-Wunused-but-set-variable]"-Look at KBUILD_CFLAGS in the main Makefile. Add -Wno-error=unused-but-set-variable to the existing set of flags.
Note the following from gcc manual:
-WerrorMake all warnings into hard errors. Source code which triggers warnings will be rejected.
-w Inhibit all warning messages. If you're familiar with C code and like to fix stuff, rather than ignoring potential bugs, use this only as a last resort- A 'brahmastram' (most powerful weapon in your time of gravest need) as the epics would say
-WerrorMake all warnings into errors.
-Werror=Make the specified warning into an error. The specifier for a warning is appended, for example -Werror=switch turns the warnings controlled by -Wswitch into errors. This switch takes a negative form, to be used to negate -Werror for specific warnings, for example -Wno-error=switch makes -Wswitch warnings not be errors, even when -Werror is in effect. You can use the -fdiagnostics-show-option option to have each controllable warning amended with the option which controls it, to determine what to use with this option.
So what I did to suppress errors was to add:
Code:
KBUILD_CFLAGS += -w
KBUILD_CFLAGS += -Wno-error=unused-but-set-variable
Though the -Wunused-but-set-variable is not a real issue in itself, it generates so much "noise" that you may miss actual make errors.
This is the error what I was talking about..
Code:
drivers/net/wireless/bcm4329_204/wl_iw.c: In function 'wl_iw_set_pmksa':
drivers/net/wireless/bcm4329_204/wl_iw.c:5075:5: error: array subscript is above array bounds [-Werror=array-bounds]
drivers/net/wireless/bcm4329_204/wl_iw.c:5078:5: error: array subscript is above array bounds [-Werror=array-bounds]
Solution:
Edit drivers/net/wireless/bcm4329_204/Makefile
Locate -Werror within DHDCFLAGS, and delete it.
Code:
DHDCFLAGS = -DLINUX -DBCMDRIVER -DBCMDONGLEHOST -DDHDTHREAD -DBCMWPA2 \
-DUNRELEASEDCHIP -Dlinux -DDHD_SDALIGN=64 -DMAX_HDR_READ=64 \
-DDHD_FIRSTREAD=64 -DDHD_GPL -DDHD_SCHED -DBDC -DTOE -DDHD_BCMEVENTS \
-DSHOW_EVENTS -DBCMSDIO -DDHD_GPL -DBCMLXSDMMC -DBCMPLATFORM_BUS \
-Wall -Wstrict-prototypes -Werror -DOOB_INTR_ONLY -DCUSTOMER_HW2 \
-DDHD_USE_STATIC_BUF -DMMC_SDIO_ABORT -DWLAN_PFN -DWLAN_PROTECT \
-DBCMWAPI_WPI \
This will prevent gcc from treating mere warnings as errors.

How to modify kernels by applying mods - Applying Kernel Patches
Ok, you have compiled a simple stock kernel. Now what? Would you like to add fixes/mods developed by other kernel devs? This post explains patches and how exactly to do this.
Patches to the kernel are applied via patch files. Patch files are simple text files generated by the linux diff program which takes two text files, compares them and writes the differences (hence called diff) to another text file which by convention has the extension .patch
Attached to this post is a patch containing my "Extended battery" fix with Sibere's battfix. I'll explain patching with this. Let's understand the patch file. Open it up in any text editor.
Code:
diff -rupN -X /home/droidzone/android/kernel/exclude.opts bravo_2.6.35_gb-mr/drivers/power/ds2784_battery.c bravo_2.6.35_gb-mr.main//drivers/power/ds2784_battery.c
--- bravo_2.6.35_gb-mr/drivers/power/ds2784_battery.c 2011-08-25 13:16:53.000000000 +0530
+++ bravo_2.6.35_gb-mr.main//drivers/power/ds2784_battery.c 2011-11-06 16:43:21.544317342 +0530
@@ -118,8 +118,11 @@ PS. 0 or other battery ID use the same p
/* Battery ID = 1: HT-E/Formosa 1400mAh */
#define BATT_ID_A 1
#define BATT_FULL_MAH_A 1400
-
#define BATT_FULL_MAH_DEFAULT 1500
+#define BATT_FULL_MAH_CAMERONSINO 2400
+#define BATT_ID_CAMERONSINO
+#define BATT_TYPE 0
+
Note the first line:
Code:
diff -rupN -X /home/droidzone/android/kernel/exclude.opts bravo_2.6.35_gb-mr/drivers/power/ds2784_battery.c bravo_2.6.35_gb-mr.main//drivers/power/ds2784_battery.c
diff -rupN basically describes the command that was used to generate this patch. The -u means that the patch file is something called a universal patch
bravo_2.6.35_gb-mr/drivers/power/ds2784_battery.c was the original file, and bravo_2.6.35_gb-mr.main//drivers/power/ds2784_battery.c was the target file or file which contains the mod..
How to apply patch files?
The command depends on where your current directory is. If you're in ~/android/kernel/bravo_2.6.35_gb-mr/ and your current directory contains the directory 'drivers', you can apply this patch with this command:
Code:
patch -p1<extended_battfix.patch
If you're within drivers, then you have to modify the command like this:
Code:
patch -p2<extended_battfix.patch
Hope you get the gist. Basically, as you move into the source tree, you have to increment the patch level by the number of directories you've moved down into. Very simple, isnt it?

Sharing and Collaborating - Using Github and Commits
Kernel compilation is a group effort (at least it ought to be). When different devs work on different parts of the code and create their own mods, development progresses. For this purpose, it is important that you share your code with other devs. The best way to do this to upload your sources to github.
First, create a github account.
Next you can view other devs' github sources and examine their commits. Commits are basically patches applies to the previous source uploaded. Github commits use the universal patch format and can be viewed directly, downloaded as patch files, and applied to your code. You can also choose to download the whole source tree uploaded by another dev and examine it.

Kernel Build Interactive Menu system
This saves quite a lot of time if you make kernels a lot..
See post #22

Ok, the basic guide is done, guys... If you have doubts, I'll try to clear them

yeah....yeah....yeah...so nice...big thx...will try this as soon as possible..
that is what i searchd so long
edit: rated with 5 stars
with kind regards

Thank you very much droidzone.
I was waiting for a n00b guide.
Tapatalking

good job droidzone
[+1] [ i like]

Added a Howto on how to apply kernel source patch files, to post #3

lol now i understand how patching works.. i write all this **** by myself.. lol

Midian666 said:
lol now i understand how patching works.. i write all this **** by myself.. lol
Click to expand...
Click to collapse
Ha ha.. that would not have been so easy

Droidzone said:
Added a Howto on how to apply kernel source patch files, to post #3
Click to expand...
Click to collapse
sorry for offtopic but nice again and you see many people thought like me with the how to..
with kind regards...Alex

Alex-V said:
sorry for offtopic but nice again and you see many people thought like me with the how to..
with kind regards...Alex
Click to expand...
Click to collapse
I like explaining stuff and sharing..
This guide was written specifically because of your request, and I have never forgotten how you helped when I was a newbie to development.. I wouldnt probably have started developing if not for good responses from Firerat and you.

Droidzone said:
I like explaining stuff and sharing..
This guide was written specifically because of your request, and I have never forgotten how you helped when I was a newbie to development.. I wouldnt probably have started developing if not for good responses from Firerat and you.
Click to expand...
Click to collapse
and now i learn from you lol thx
with kind regards..Alex

Fantastic guide!!!!!!!

Did some more work on the first post. It now includes a flashable zip template and instructions on how to easily create your own flashable zip after compilation is over.

maybe some improvments to your making a flashable zip.
i did this with my kernels.. it took the version infos from the config files.. and put it into a folder... after this u can make zip.
ive stolen this from manus source
Code:
localVersion=`cat .your-config | fgrep CONFIG_LOCALVERSION= | cut -f 2 -d= | sed s/\"//g`
linuxVersion=`cat .your-config | fgrep "Linux kernel version:" | cut -d: -f 2 | cut -c2-`
VERSION=$linuxVersion$localVersion
echo "Kernel version=$VERSION"
rm -rf flash/system/lib/modules/*
mkdir flash/system/lib/modules/$VERSION
mkdir flash/system/lib/modules/$VERSION/kernel
tar czf modules.tgz `find . -name '*.ko'`
cd flash/system/lib/modules/$VERSION/kernel
tar xzf ../../../../../../modules.tgz
cd - > /dev/null
rm modules.tgz

This is good..Actually when I generate kernels I test too many versions that I dont usually change the local version number in the menuconfig. Instead I use the date and time (including second) to name the kernel dir and kernel zip name...
Like this..
Code:
date_str=`date '+%d%m%y_%H%M%S'`
dirname="kernel_"$nameflag"_"$date_str
pckdir="$packagedir/$dirname"
mkdir $pckdir
lastfolder=$pckdir
cd $outdir/
echo
zipnoname="kbase_"$nameflag"_"$date_str
zipaddnoname="kmods_"$nameflag"_"$date_str
zipname=$zipnoname".zip"
zipaddname=$zipaddnoname".zip"
zip -r $zipnoname ./
mv $zipname $pckdir/
As you can see, its part of my script which does a lot of things..
But getting the localversion too is a good thing..I'd put it into a textfile in the zip which users can read..

Great guide. Thanks a lot
Sent from my HTC Desire using Tapatalk

[Q] Own kernel or one with kexec

Hi,
I've been trying to get my own kernel with few modifications running on my ASUS Transformer. I've followed few guides around with no luck. What I did:
Tried two source trees:
1) Official from ASUS
2) Roach2010s tree from github (https://github.com/Roach2010/android_kernel_TF101.git)
Used .config from my current kernel which is running fine (Prime kernel) without any changes.
Compiled kernel.
So far looks good, with few modifications to config I got new modules that works so crosscompiler is not misscompiling. Now the part where I'm doing something wrong and can't figure out what.
I started with Prime Kernel from http://forum.xda-developers.com/showthread.php?t=1251044
* Unziped the archive
* blobunpack blob
* created blob.LNX in several ways described bellow
* blobpack blob.HEADER blob LNX blob.LNX
* dd if=blob of=/dev/block/mmcblk0p4
* reboot
How I created blob.LNX
A) Use extracted blob.LNX and use abootimg to replace kernel
* abootimg -u blob.LNX -k zImage
B) Extracted all parts and recreated image using abootimg
* abootimg -x blob.LNX
* abootimg --create blob.LNX -f bootimg.cfg -k zImage -r initrd.img
C) Extracted all parts and recreated image using bootunpack and mkbootimg
* bootunpack blob.LNX
* mkbootimg --kernel zImage --ramdisk ramdisk.gz -o blob.LNX
In addition I tried few modifications:
* enlarging bootsize in bootimg.cfg to make sure everything fits
* passing cmdline my current kernel booted up with as default in .config, as cmdline in bootimg.cfg and both
All my efforts ended up on ASUS boot up screen, no matter what I try. So my question is, am I missing something? Did I skipped some important part? Have I done something wrong? Any ideas appreciated.
If nobody has any idea, can somebody please create kernel with enabled kexec for my ASUS Transformer? That was the ultimate goal of trying to get my own kernel, but if I can't get working just recompiled one...

-miska- said:
Hi,
I've been trying to get my own kernel with few modifications running on my ASUS Transformer. I've followed few guides around with no luck. What I did:
Tried two source trees:
1) Official from ASUS
2) Roach2010s tree from github (https://github.com/Roach2010/android_kernel_TF101.git)
Used .config from my current kernel which is running fine (Prime kernel) without any changes.
Compiled kernel.
So far looks good, with few modifications to config I got new modules that works so crosscompiler is not misscompiling. Now the part where I'm doing something wrong and can't figure out what.
I started with Prime Kernel from http://forum.xda-developers.com/showthread.php?t=1251044
* Unziped the archive
* blobunpack blob
* created blob.LNX in several ways described bellow
* blobpack blob.HEADER blob LNX blob.LNX
* dd if=blob of=/dev/block/mmcblk0p4
* reboot
How I created blob.LNX
A) Use extracted blob.LNX and use abootimg to replace kernel
* abootimg -u blob.LNX -k zImage
B) Extracted all parts and recreated image using abootimg
* abootimg -x blob.LNX
* abootimg --create blob.LNX -f bootimg.cfg -k zImage -r initrd.img
C) Extracted all parts and recreated image using bootunpack and mkbootimg
* bootunpack blob.LNX
* mkbootimg --kernel zImage --ramdisk ramdisk.gz -o blob.LNX
In addition I tried few modifications:
* enlarging bootsize in bootimg.cfg to make sure everything fits
* passing cmdline my current kernel booted up with as default in .config, as cmdline in bootimg.cfg and both
All my efforts ended up on ASUS boot up screen, no matter what I try. So my question is, am I missing something? Did I skipped some important part? Have I done something wrong? Any ideas appreciated.
If nobody has any idea, can somebody please create kernel with enabled kexec for my ASUS Transformer? That was the ultimate goal of trying to get my own kernel, but if I can't get working just recompiled one...
Click to expand...
Click to collapse
Here is what I've done. If you have successfully built a kernel with the resulting zImage, then you are part way there, I believe there is a kernel config option to enable kexec support but I haven't tried that. Next, you can take some other kernel's .zip file (CWM flashable) and unzip it. You may need to download a zip utility. You'll have 2 folders and a kernel blob. If you bootunpack this kernel blob, you'll end up with the kernel blob and a some *.LNX file. This *.LNX file is the same as a boot.img file. You can use dsixda's Android kitchen to split this into the initrd and the kernel (zImage) parts. Replace the zImage with your own and move any modules you may have also built if necessary into the initrd part, join them back together into a boot.img in the kitchen.
copy this boot.img back to where you unzipped the kernel.zip, delete the original *.LNX file, rename the boot.img to the same name as the previous *.LNX file and then bootpack it together and flash it through CWM. Zip the 2 folders and the kernel blob you just made back together with whatever name you want. You can edit the text in the updater script before you zip it all up, but whether you do or not it should boot.

Yes, there is kexec config option, but I haven't suceeded even without changing anything so enabling it doesn't make kernel boot :-D I tried Android Kitche to split boot image and I ended up with the same files (compared the content to check) as with abootimg -x. Tried recreating update.zip and sign it using Android Kitchen, just to be sure, that something in android is not in the way to the actualization from running system. Still no luck :-(

-miska- said:
Yes, there is kexec config option, but I haven't suceeded even without changing anything so enabling it doesn't make kernel boot d:-D I tried Android Kitche to split boot image and I ended up with the same files (compared the content to check) as with abootimg -x. Tried recreating update.zip and sign it using Android Kitchen, just to be sure, that something in android is not in the way to the actualization from running system. Still no luck :-(
Click to expand...
Click to collapse
I didn't even sign mine as I have signature verification turned off in CWM recovery. Didn't change the text either as I was mostly experimenting and learning. I know kexec works under linux, but I think it requires separate package(s) and configuration to do so. I got a bit confused with blobpack, blobunpack info, but figured out that with just the kernel you don't seem to need the mentioned header file, just the .LNX which is the same as boot.img which is the combined kernel zImage and initramfs. If the kernel blob is still there and you use the same name for the output file then it doesn't matter because it will get overwritten anyway. Worked for me at least using source of kernel I've booted before and my running .config.

sidneyk said:
I didn't even sign mine as I have signature verification turned off in CWM recovery. Didn't change the text either as I was mostly experimenting and learning. I know kexec works under linux, but I think it requires separate package(s) and configuration to do so. I got a bit confused with blobpack, blobunpack info, but figured out that with just the kernel you don't seem to need the mentioned header file, just the .LNX which is the same as boot.img which is the combined kernel zImage and initramfs. If the kernel blob is still there and you use the same name for the output file then it doesn't matter because it will get overwritten anyway. Worked for me at least using source of kernel I've booted before and my running .config.
Click to expand...
Click to collapse
hmmm, zip file I had as an example was using blobed boot image going through mmcblk0p4. Do you have some link to .zip file that does it differently?
kexec is a way how to boot something else from linux directly without need to fiddle with bootloader. To use it, two parts are needed - kernel that supports it (that's what I can't get) and tool to actually use it/call it. Tool is not a problem, got that one hopefully ready, but without the kernel...

-miska- said:
hmmm, zip file I had as an example was using blobed boot image going through mmcblk0p4. Do you have some link to .zip file that does it differently?
kexec is a way how to boot something else from linux directly without need to fiddle with bootloader. To use it, two parts are needed - kernel that supports it (that's what I can't get) and tool to actually use it/call it. Tool is not a problem, got that one hopefully ready, but without the kernel...
Click to expand...
Click to collapse
Have you tried Koush's "anykernel.zip" code (probably requires a few mods)? It appears to be trying to replace the blob based updater-scripts that are all over the place.
I've used it successfully, but it has mostly been on other devices, and it is very easy to use. I think some of the templates are too generic and maybe confusing, and without figuring out how edify scripting actually works, it is mysterious, but I'd look at this code, git it and try to use it:
I'll try to provide a working example since I just added a few modules to one of the kernels 2.6.36-4 that're out there for the tf101, but I need to be sure it's working first. I think there's perhaps one difference at least between what Koush shows for the xoom and the tf101 so am working on it.
https://github.com/koush/AnyKernel
Good luck -

-miska- said:
hmmm, zip file I had as an example was using blobed boot image going through mmcblk0p4. Do you have some link to .zip file that does it differently?
kexec is a way how to boot something else from linux directly without need to fiddle with bootloader. To use it, two parts are needed - kernel that supports it (that's what I can't get) and tool to actually use it/call it. Tool is not a problem, got that one hopefully ready, but without the kernel...
Click to expand...
Click to collapse
I was using clemsyn-blades_kernel_ver22a zip file. I don't know if it was doing it different or not, haven't checked that far into it.

sidneyk said:
I was using clemsyn-blades_kernel_ver22a zip file. I don't know if it was doing it different or not, haven't checked that far into it.
Click to expand...
Click to collapse
hmmm, checked that one, uses blobed image and 'dd if=/tmp/blob of=/dev/block/mmcblk0p4' as well. Maybe I'll try different crosscompiler anyway, that's the one thing I haven't altered yet :-/

hachamacha said:
Have you tried Koush's "anykernel.zip" code (probably requires a few mods)? It appears to be trying to replace the blob based updater-scripts that are all over the place.
I've used it successfully, but it has mostly been on other devices, and it is very easy to use. I think some of the templates are too generic and maybe confusing, and without figuring out how edify scripting actually works, it is mysterious, but I'd look at this code, git it and try to use it:
I'll try to provide a working example since I just added a few modules to one of the kernels 2.6.36-4 that're out there for the tf101, but I need to be sure it's working first. I think there's perhaps one difference at least between what Koush shows for the xoom and the tf101 so am working on it.
Click to expand...
Click to collapse
Haven't tried that one, looks interesting... This one doesn't use blobed update and wites image directly somewhere. Just would require to check that that somewhere is the right place :-D Thanks, will take a look at that and what other edify commands are availble in updater, sounds like interesting alternative approach...

-miska- said:
Haven't tried that one, looks interesting... This one doesn't use blobed update and wites image directly somewhere. Just would require to check that that somewhere is the right place :-D Thanks, will take a look at that and what other edify commands are availble in updater, sounds like interesting alternative approach...
Click to expand...
Click to collapse
I'm modifying the script I've seen passed around (not quite Koush's git repo version) passed around to see if I can get it to work on the tf101. The 'write it somewhere' edify command is the question mark, but I think it is going on it's (the device's) internal partition table and vectored to 'boot', which is either a terrific generic idea, or terrible depending upon what edify does. I can't really find a heck of a lot explaining anything about the individual edify commands. I'm just getting rid of the 'showstoppers' where partition names like mmc0p* are used that are clearly wrong for the tf101. I made the mistake of trying one that I only later realized thought that partition 1 was data, when it is actually partition 7. Good thing I can make nvflash backups on my 'old' transformer.
I'll post back later today with any results I get. I'm not concerned about whether my kernel worked since it is completely experimental , just that it got written there, so I might use a working version with a different kernel name (in Makefile) just so I can get 'proof of concept' .
On a slightly different note but having to do with what you're doing, I tried the blob route this week, and for some reason, blobunpack/pack right from Rayman's git repo do not unpack the blobs correctly for say 'clemsyms' or 'Prime's' blobs, which has me wondering about some change that maybe took place. In any case, it forces me down this other path anyway.
If they are working OK for you, could you tell me a couple things?
1) Your linux distro and architecture (x86/x86_64)
2) did you build them from Rayman's repo? Did you get binaries from somewhere, if so where?
3) parameters? I don't think mine take any but the blob name.
4) Output suffixes. I only get .LNX from any of the above blobs which is useless.
EDIT: I was recalling that 'edify' in CWM came into being somewhere (maybe) past the only version that works with the tf101 (we're on ~v3.x and edify ~v4/5+). If that's the case, then we're all stuck with blobs because that one write command is edifi(ed) most likely. I'll stare at the git CWM source today too to figure out if it used the edify stuff in this version. I think Solarnz had it in his git hub.

hachamacha said:
I'm modifying the script I've seen passed around (not quite Koush's git repo version) passed around to see if I can get it to work on the tf101. The 'write it somewhere' edify command is the question mark, but I think it is going on it's (the device's) internal partition table and vectored to 'boot', which is either a terrific generic idea, or terrible depending upon what edify does. I can't really find a heck of a lot explaining anything about the individual edify commands. I'm just getting rid of the 'showstoppers' where partition names like mmc0p* are used that are clearly wrong for the tf101. I made the mistake of trying one that I only later realized thought that partition 1 was data, when it is actually partition 7. Good thing I can make nvflash backups on my 'old' transformer.
I'll post back later today with any results I get. I'm not concerned about whether my kernel worked since it is completely experimental , just that it got written there, so I might use a working version with a different kernel name (in Makefile) just so I can get 'proof of concept' .
On a slightly different note but having to do with what you're doing, I tried the blob route this week, and for some reason, blobunpack/pack right from Rayman's git repo do not unpack the blobs correctly for say 'clemsyms' or 'Prime's' blobs, which has me wondering about some change that maybe took place. In any case, it forces me down this other path anyway.
If they are working OK for you, could you tell me a couple things?
1) Your linux distro and architecture (x86/x86_64)
2) did you build them from Rayman's repo? Did you get binaries from somewhere, if so where?
3) parameters? I don't think mine take any but the blob name.
4) Output suffixes. I only get .LNX from any of the above blobs which is useless.
EDIT: I was recalling that 'edify' in CWM came into being somewhere (maybe) past the only version that works with the tf101 (we're on ~v3.x and edify ~v4/5+). If that's the case, then we're all stuck with blobs because that one write command is edifi(ed) most likely. I'll stare at the git CWM source today too to figure out if it used the edify stuff in this version. I think Solarnz had it in his git hub.
Click to expand...
Click to collapse
Blobs are used on the tf101 because they are the ONLY way of flashing boot/recovery, there is no block device mapping of them on our device

lilstevie said:
Blobs are used on the tf101 because they are the ONLY way of flashing boot/recovery, there is no block device mapping of them on our device
Click to expand...
Click to collapse
OK: Thanks lilstevie,
That takes care of that. Time for me to make peace with blobs.

After steve's reply, I just went to using blobs. I've got my own kernel running fine on the tf101 using that method.
For the best reference I've seen on using blobs and boottools , try this post:
http://forum.xda-developers.com/showthread.php?t=1193737
---

Just got back from work, will ply with it some more, but I'll start with answering the questions...
hachamacha said:
1) Your linux distro and architecture (x86/x86_64)
Click to expand...
Click to collapse
Gentoo x86-64
hachamacha said:
2) did you build them from Rayman's repo? Did you get binaries from somewhere, if so where?
Click to expand...
Click to collapse
Compiled from git repo. I always tried to find the most upstream repo for each tool and then compiled it by myself.
hachamacha said:
3) parameters? I don't think mine take any but the blob name.
4) Output suffixes. I only get .LNX from any of the above blobs which is useless.
Click to expand...
Click to collapse
These two comes together:
'blobunpack blob' - takes a blob as input and ouptuts blob.HEADER and blob.LNX
'bootunpack blob.LNX' - takes blob.LNX as input and outputs blob.LNX-kernel.gz, blob.LNX-ramdisk.cpio.gz and blob.LNX-config
'abootimg -x blob.LNX' - takes blob.LNX as input and outputs zImage, initrd.img and bootimg.cfg
Resulting files from bootunpack and abootimg are almost same, only difference is the configuration file
To repack:
'abootimg --create newblob/blob.LNX -f bootimg.cfg -k zImage -r initrd.img'
or
'mkbootimg --kernel zImage --ramdisk blob.LNX-ramdisk.cpio.gz -o newblob/blob.LNX'
and then
'blobpack blob.HEADER newblob/blob LNX newblob/blob.LNX'
Unless I change kernel, everything works just fine :-D

-miska- said:
Just got back from work, will ply with it some more, but I'll start with answering the questions...
Gentoo x86-64
Compiled from git repo. I always tried to find the most upstream repo for each tool and then compiled it by myself.
These two comes together:
'blobunpack blob' - takes a blob as input and ouptuts blob.HEADER and blob.LNX
'bootunpack blob.LNX' - takes blob.LNX as input and outputs blob.LNX-kernel.gz, blob.LNX-ramdisk.cpio.gz and blob.LNX-config
'abootimg -x blob.LNX' - takes blob.LNX as input and outputs zImage, initrd.img and bootimg.cfg
Resulting files from bootunpack and abootimg are almost same, only difference is the configuration file
To repack:
'abootimg --create newblob/blob.LNX -f bootimg.cfg -k zImage -r initrd.img'
or
'mkbootimg --kernel zImage --ramdisk blob.LNX-ramdisk.cpio.gz -o newblob/blob.LNX'
and then
'blobpack blob.HEADER newblob/blob LNX newblob/blob.LNX'
Unless I change kernel, everything works just fine :-D
Click to expand...
Click to collapse
Pretty similar, although the kernel zImage itself is always a mystery unless you've not changed anything, but even then, getting it built with the right toolchain, etc isn't guaranteed. So lets assume that just works for now since it'll become obvious as it goes along.
I guess I have not heard of 'abootimg' as a tool for this, so I've been using the more manual way of dissecting the initrd as follows:
Code:
gunzip -dc ../blob.LNX-ramdisk.cpio.gz | cpio -i
If you need to change something , for example, in default.prop like ro.secure=0, then you'd do it there.
Then repack into a new ramdisk:
Code:
find . | cpio -o -H newc | gzip > ../newramdisk.cpio.gz
Finally I just had a somewhat heavily modified zImage from my build, so did this to make the blob (I'd copied zImage to blob.LNK-zImage.gz below):
Code:
./mkbootimg --kernel blob.LNX-zImage.gz --ramdisk newramdisk.cpio.gz -o boot.img
./blobpack blob.HEADER newblob LNX boot.img
zip -r imagename.zip blob MET* system // whatever the syntax was.
NOTE: I did this on a native 64 bit ubuntu LTS 10.04 box.
Unless I typo'd up there, that 'should' work. If it does boot, then first thing, take a look at settings, and kernel info so you can verify that you're running the kernel you desired (hopefully you renamed it in Makefile the first 4-5 lines).

Solved
Ok, got it working!!! Problem was bad crosscompiler :-( Modules I crosscompiled worked fine, so I ruled crosscompiler out :-/ Looks like I was too quick in judgement :-( Now I have kernel recompiled with original settings and evne the modified one and it still works and boot. Now I'm going to play with new features I got! Thanks a lot for all help!!!
Just for the record, crosscompiler I was originally using was codesourcery 2011.03 and to make it work I switched to official crosscompiler from NDK. Rest of the commands was Ok, I was just suspecting wrong step as I was quite familiar with kernel building and quite unfamiliar with the blob stuff :-(

Congrats!
For some reason I avoid the codesourcery stuff and stick with either the prebuilt toolchains or else just build my own from gnu source.
Anyway, glad you figured it out.

I have been following a few different instructions for the tools and was concentrated on just learning to rebuild a kernel on my own setup - Ubuntu 11.10. I only installed Ubuntu since it was the distro mostly referenced in the tutorials. I've also tried a couple different tool chains, some work, some don't.
I then find an existing *.zip CWM flashable kernel to work with, usually trying to use one I've successfully ran before, and unzip it. This gives 2 folders and a blob file. Whenever I run bootunpack on the blob I only get a resultant blob.LNX file and, so far never any blob.HEADER file. I understood that the blob.LNX was the same as boot.img from reading through and use dsixda's kitchen to split up the .LNX file I've renamed to boot.img. I then replace the zImage with the one I've just built and repack to boot.img in the kitchen. Then I move that boot.img back to unzipped kernel directory and rename to blob.LNX and run bootpack with blob as output and just ignore the .HEADER part. I then rezip the 2 folders (after replacing any modules in there) and blob into a new zip file and reflash in CWM. If it was based on a kernel I've booted before then it usually works without any problems. I can replace text in the updater-script, if I want and am just reusing the initramfs from the original zip. I have signature verification turned off in CWM, so that doesn't choke it. I need to read more about building initramfs before I do it. So far, this works for me, but I haven't really done any modification to the source, other than rebuilding it with my running config.

sidneyk said:
I have been following a few different instructions for the tools and was concentrated on just learning to rebuild a kernel on my own setup - Ubuntu 11.10. I only installed Ubuntu since it was the distro mostly referenced in the tutorials. I've also tried a couple different tool chains, some work, some don't.
I then find an existing *.zip CWM flashable kernel to work with, usually trying to use one I've successfully ran before, and unzip it. This gives 2 folders and a blob file. Whenever I run bootunpack on the blob I only get a resultant blob.LNX file and, so far never any blob.HEADER file. I understood that the blob.LNX was the same as boot.img from reading through and use dsixda's kitchen to split up the .LNX file I've renamed to boot.img. I then replace the zImage with the one I've just built and repack to boot.img in the kitchen. Then I move that boot.img back to unzipped kernel directory and rename to blob.LNX and run bootpack with blob as output and just ignore the .HEADER part. I then rezip the 2 folders (after replacing any modules in there) and blob into a new zip file and reflash in CWM. If it was based on a kernel I've booted before then it usually works without any problems. I can replace text in the updater-script, if I want and am just reusing the initramfs from the original zip. I have signature verification turned off in CWM, so that doesn't choke it. I need to read more about building initramfs before I do it. So far, this works for me, but I haven't really done any modification to the source, other than rebuilding it with my running config.
Click to expand...
Click to collapse
The architecture really seems to make a big difference in some configurations.
I have one native linux box with 64 bit 10.04 LTS on it, and it always behaves as well as possible, so I did this blob/boot/tools work on it, and it went as it should (creating HEADER and LNX) files, etc.
Then in addition I use several linux distros in VMs, one of them being more like yours, an 11.10 distro with just the androidSDK and all the build tools, prebuilt chains, etc. That will do exactly as you said. I actually built those blobtools/boottools from Koush's git, and they don't work correctly in that one environment. What is different to make that happen? I'm just guessing that something important like the native x86_64 gcc world is different enough to foul things up. It really doesn't matter. Once I got the tools working on the native box, I just transferred them to the other boxes including 11.10 and they work fine.
If you're using 64 bit and would like them I can probably stick them into a .tar.bz2 or whatever and stick up a link to them, or maybe if you can find working binaries to download, you might get those working. Once the blobunpack is returning only the .LNX file, you've pretty well had it as far as progress.
Good luck

hachamacha said:
The architecture really seems to make a big difference in some configurations.
I have one native linux box with 64 bit 10.04 LTS on it, and it always behaves as well as possible, so I did this blob/boot/tools work on it, and it went as it should (creating HEADER and LNX) files, etc.
Then in addition I use several linux distros in VMs, one of them being more like yours, an 11.10 distro with just the androidSDK and all the build tools, prebuilt chains, etc. That will do exactly as you said. I actually built those blobtools/boottools from Koush's git, and they don't work correctly in that one environment. What is different to make that happen? I'm just guessing that something important like the native x86_64 gcc world is different enough to foul things up. It really doesn't matter. Once I got the tools working on the native box, I just transferred them to the other boxes including 11.10 and they work fine.
If you're using 64 bit and would like them I can probably stick them into a .tar.bz2 or whatever and stick up a link to them, or maybe if you can find working binaries to download, you might get those working. Once the blobunpack is returning only the .LNX file, you've pretty well had it as far as progress.
Good luck
Click to expand...
Click to collapse
If by 'native' you mean a hard disk install as opposed to a VM install, then that's where I'm at. I have Ubuntu 11.10 x86_64 installed to a separate partition. I have the recommended stuff installed including the ia32 libs, but I never see a blob.HEADER file with either kernel.zips or ROM zips. I can unpack and repack kernels without the HEADER though and they boot just fine.
But, yes, if you don't mind posting a link with your files I'll give them a try sometime. Thanks.

[Tool]File(1) command for Android (ver 5.11)

After wandering around looking for an android build of the file command, I finally went ahead and compiled it myself.
The attached package contains the file binary (v5.11), the libmagic.so shared library, the associated magic files and the build specific files that I needed (over and above the source) to build it. The paths of the files are relative to '/'.
Refs:
source: file website (www_darwinsys_com/file/)
compiling: Native Vim on Android (gdr_geekhood_net/gdrwpl/vim-android.php)
Note: Add the location of libmagic.so to LD_LIBRARY_PATH to get rid of the "CANNOT LINK EXECUTABLE" error. (I use /data/local/lib)

Compiled a dynamically linked library and associated binary instead of the monolithic binary built earlier.

samveen said:
After wandering around looking for an android build of the file command, I finally went ahead and compiled it myself.
The attached package contains the file binary (v5.11), the libmagic.so shared library, the associated magic files and the build specific files that I needed (over and above the source) to build it. The paths of the files are relative to '/'.
Refs:
source: file website (www_darwinsys_com/file/)
compiling: Native Vim on Android (gdr_geekhood_net/gdrwpl/vim-android.php)
Note: Add the location of libmagic.so to LD_LIBRARY_PATH to get rid of the "CANNOT LINK EXECUTABLE" error. (I use /data/local/lib)
Click to expand...
Click to collapse
I followed your instructions and it works perfectly.
Code:
$ echo $LD_LIBRARY_PATH
/data/local/lib:/data/local/lib:/data/local/lib:/system/lib:/vendor/lib
$ file
Usage: file [-bchikLlNnprsvz0] [--apple] [--mime-encoding] [--mime-type]
[-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
file -C [-m magicfiles]
file [--help]
$ getprop | grep device
[ro.cm.device]: [jordan]
[ro.product.device]: [umts_jordan]
I have been looking for this for a while. Man!! This post needs some serious attention. Great work.
Sent from my MB525 usiIng XDA

@samveen Any chance you could compile it with libmagic statically linked again to make it portable? Thanks for your time.

osm0sis said:
@samveen Any chance you could compile it with libmagic statically linked again to make it portable? Thanks for your time.
Click to expand...
Click to collapse
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).

samveen said:
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).
Click to expand...
Click to collapse
I was just looking for this. Great tool.
Thanks for your work.

samveen said:
@osm0sis Sure. I'll add in a statically linked binary, but it'll take a little time (2-3 days).
Click to expand...
Click to collapse
Any luck? I think it only needs libmagic statically linked to be portable; the other NDK/Bionic stuff can probably stay dynamic to keep the filesize down. I see file 5.17 source is available now too if you wanted to update your builds to the latest.

osm0sis said:
Any luck? I think it only needs libmagic statically linked to be portable; the other NDK/Bionic stuff can probably stay dynamic to keep the filesize down. I see file 5.17 source is available now too if you wanted to update your builds to the latest.
Click to expand...
Click to collapse
If you want only the system file type, the workaround i use is this (eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE)
In a script just add a var like this
TYPE=$(eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE) and you will have the FS type in $TYPE.
Phone: Samsung Galaxy SII - GT-I9100
Kernel: Dorimanx kernel v10.43v99-mv3
1st ROM : SlimSaber 4.4.2 Maliv3 by fusionjack build of 20140320 (Online)
2nd ROM :
MODEM: UHMS1
MODS: Partition Fix, Hue Blue v4.4 by Kroz :good:

Computoncio said:
If you want only the system file type, the workaround i use is this (eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE)
In a script just add a var like this
TYPE=$(eval $(/sbin/blkid /dev/block/mmcblk0p11 | /sbin/awk ' { print $3 } '); /sbin/busybox echo $TYPE) and you will have the FS type in $TYPE.
Click to expand...
Click to collapse
Interesting but nope; I need file to determine a filetype. Compressed archives via a script, to be specific.

@osm0sis I got the code compiled (both with a dynamic libmagic.so, and libmagic compiled into file) but I need a little more time to compile the magic data (it requires running file on the target device to compile the magic data into a binary datafile). Once I have that figured out (either by compiling it or by creating a script to allow the users to do it themselves, I'll post it all in one go (need one more day).

samveen said:
@osm0sis I got the code compiled (both with a dynamic libmagic.so, and libmagic compiled into file) but I need a little more time to compile the magic data (it requires running file on the target device to compile the magic data into a binary datafile). Once I have that figured out (either by compiling it or by creating a script to allow the users to do it themselves, I'll post it all in one go (need one more day).
Click to expand...
Click to collapse
Awesome! Thanks so much. The magic file isn't terribly important since you can grab one a lot of places and specify it with the -m parameter; keeps the file size down too if you were thinking of including it in the binary somehow!

Alright. I just wanted to thank you again for your work but I managed to make a static native compile (~440kb) of file-5.17 using my N7 and a kbox build environment I set up. It's available in my ARM Android Image Kitchen linked in my sig. Cheers!

osm0sis said:
Alright. I just wanted to thank you again for your work but I managed to make a static native compile (~440kb) of file-5.17 using my N7 and a kbox build environment I set up. It's available in my ARM Android Image Kitchen linked in my sig. Cheers!
Click to expand...
Click to collapse
@osm0sis - Can you please share how did you static compile this? I have setup the Ubuntu 13.10 image on Android using Complete Linux Installer (with all the necessary build tools setup).
I am able to compile this, but it's linking the shared library "libmagic.la".

Need to pass --static to ./configure and have a cross compiler set up, for ARMv7.

ericlnu said:
Need to pass --static to ./configure and have a cross compiler set up, for ARMv7.
Click to expand...
Click to collapse
I don't think that's a valid option.
Code:
$ ./configure --static
configure: error: unrecognized option: `--static'
Try `./configure --help' for more information

amit.bagaria said:
I don't think that's a valid option.
Code:
$ ./configure --static
configure: error: unrecognized option: `--static'
Try `./configure --help' for more information
Click to expand...
Click to collapse
Nope, seems not ^_^. Sorry, it usually is with autoconf derived configure scripts.

amit.bagaria said:
@osm0sis - Can you please share how did you static compile this? I have setup the Ubuntu 13.10 image on Android using Complete Linux Installer (with all the necessary build tools setup).
I am able to compile this, but it's linking the shared library "libmagic.la".
Click to expand...
Click to collapse
Haven't tried it using a full Linux image on Android, just made it native compile in KBox2 on my N7'13 for awhile by editing in the gettext functions it was complaining about, and have since moved on to cross-compiling on my Windows 7 desktop in Cygwin using the NDK. There were still a few tricks to getting it to be a true static compile. It's all linked in my post here:
http://forum.xda-developers.com/showthread.php?p=54510825
My latest compile included in my AIK-mobile package is file 5.20. :good:

I don't suppose you could do a quick compile with the -fPIE flag? Trying to use this on 5.0 + and don't want to patch my linker.
Tried to compile it myself (it sort-of worked, file --version is operational) but when I try to actually use it for something my terminal borks out and switches to some weird font after about 20 warnings. (configure also seems to be using the -rpath flag apparently, linker complains about it) hmm
EDIT: I've got it working after some trial and error. If anyone requests I'll post it
EDIT 2: android 6.0's linker seems less forgiving than 5.0/5.1, need to compile without -rpath (so no just using ./configure) will keep posted

Can you please post the file binary you built with -pie ?

[KERNEL] Stock Kernel (no write protection)

This is the stock kernel without system write protection. Completely stock configuration, completely stock ramdisk. The only difference is it allows write to system
Uses Telstra 3.17.841.2 source from htcdev.com:
http://dl4.htc.com/RomCode/Source_and_Binaries/evitaul-jb-crc-3.4.10-240d4d5.zip
Devs who need a vanilla kernel are free to modify or include with ROMs
installer: http://goo.gl/64HJc
boot.img: http://goo.gl/IRZ9m

Does this also uninstall any old modules?
Sent from my VENOMized HoxL

area51avenger said:
Does this also uninstall any old modules?
Sent from my VENOMized HoxL
Click to expand...
Click to collapse
The installer deletes /system/lib/modules/wp_mod.ko and /system/lib/wp_mod.ko
There is no init.d support either, so it won't load any init scripts.

Nice job, this should be useful! :good:

On my side, I had some problems. I have tried to compile the kernel, but I went a little bit further with the modifications:
disabled the /system write protection.
integrated the prima_wlan source tree in drivers/staging. The sources were downloaded from the codeaurora.org site and the driver version was 3.1.7.16 (I have specifically chosen this version because it was the closest to the 3.1.7.0 version that exists in the stock ROM).
compressed the vmlinux image with LZMA instead of gzip (this is an option in the configuration) to obtain the zImage file.
compressed the ramdisk image with LZMA instead of gzip (in the configuration file, I have seen 3 enabled options: gzip, LZMA and XZ and I thought recompressing the ramdisk might not hurt).
The toolchain I have used was downloaded from android.googlesource.com. Then I have unpacked the original boot.img like this: trimmed the first 128 bytes from the file (up to the ANDROID magic label), then used the unpackbootimg tool (this was a tool I have found in a Gingerbread source tree). Then, I have used the new zImg, the LZMA compressed ramdisk and mkbootimg (this was also picked from the same Gingebread source tree) to build a new boot.img. I have booted in the recovery, flashed the new modules, then booted in fastboot mode and flashed the boot.img. The result was an annoying boot loop and it wasn't very easy to put the phone in fastboot mode again.
I have some questions about what I have done so far. Could you, please, tell me if there was a wrong manoeuvre/setting/thingie? I suspect the unpacking process has not produced the proper data (command line, base address ...). What tool have you used to unpack the original boot.img file? I also suspect the packing process. Is a new mkbootimg required?
And some questions about your work. What prima_wlan driver have you used in your boot.img? Have you modified the suffix added to the 3.4.10 version when you have configured the kernel in order to make it work?

AlxMAX said:
On my side, I had some problems. I have tried to compile the kernel, but I went a little bit further with the modifications:
disabled the /system write protection.
integrated the prima_wlan source tree in drivers/staging. The sources were downloaded from the codeaurora.org site and the driver version was 3.1.7.16 (I have specifically chosen this version because it was the closest to the 3.1.7.0 version that exists in the stock ROM).
compressed the vmlinux image with LZMA instead of gzip (this is an option in the configuration) to obtain the zImage file.
compressed the ramdisk image with LZMA instead of gzip (in the configuration file, I have seen 3 enabled options: gzip, LZMA and XZ and I thought recompressing the ramdisk might not hurt).
The toolchain I have used was downloaded from android.googlesource.com. Then I have unpacked the original boot.img like this: trimmed the first 128 bytes from the file (up to the ANDROID magic label), then used the unpackbootimg tool (this was a tool I have found in a Gingerbread source tree). Then, I have used the new zImg, the LZMA compressed ramdisk and mkbootimg (this was also picked from the same Gingebread source tree) to build a new boot.img. I have booted in the recovery, flashed the new modules, then booted in fastboot mode and flashed the boot.img. The result was an annoying boot loop and it wasn't very easy to put the phone in fastboot mode again.
I have some questions about what I have done so far. Could you, please, tell me if there was a wrong manoeuvre/setting/thingie? I suspect the unpacking process has not produced the proper data (command line, base address ...). What tool have you used to unpack the original boot.img file? I also suspect the packing process. Is a new mkbootimg required?
And some questions about your work. What prima_wlan driver have you used in your boot.img? Have you modified the suffix added to the 3.4.10 version when you have configured the kernel in order to make it work?
Click to expand...
Click to collapse
use dsi xda kitchen to pack the boot.img for easy use
or
use this
unpackbootimg -i boot.img -o /directoryfordump
mkbootimg --cmdline 'console=ttyHSL0,115200,n8' --kernel zImage --ramdisk ramdisknew.gz --base 80400000 --ramdiskaddr 81900000 -o newboot.img

Where have you got that ramdisk address from? unpackbootimg doesn't report it.
Zarboz said:
mkbootimg --cmdline 'console=ttyHSL0,115200,n8' --kernel zImage --ramdisk ramdisknew.gz --base 80400000 --ramdiskaddr 81900000 -o newboot.img
Click to expand...
Click to collapse
Edit: I have done some research on mkbootimg and unpackbootimg and their usage is a little bit confusing. The boot image header contains a kernel address, a ramdisk address and some other data (including the command line provided to the kernel by the bootloader). The mkbootimg tool like it is in the Android repository uses more measures: a base address, a kernel offset, a ramdisk offset etc. The kernel address is base + kernel offset and so on. When unpacking the boot image with unpackbootimg, one must suppose one of the parameters fixed, usually the kernel offset (0x00008000). Consequently, the base address is computed as the kernel address - the kernel offset. To conclude, the command line to use should be:
Code:
mkbootimg --cmdline 'console=ttyHSL0,115200,n8' --base 80400000 --ramdisk_offset 1408000 --kernel boot.img-zImage --ramdisk boot.img-ramdisk.gz -o boot-2.img
This produces the same boot.img as the one posted by flar2, except for the command line.
Edit 2: I have recompiled the kernel, disabled the /system protection, used the mkbootimg command line right above and now my phone boots and runs well. The prima_wlan.ko that I have compiled together with the kernel (3.1.7.16 from codeaurora source repository) doesn't work. It might be a matter of wi-fi firmware. Fortunately, the stock prima_wlan.ko (proprietary 3.1.7.0) module works well.

AlxMAX said:
Where have you got that ramdisk address from? unpackbootimg doesn't report it.
Edit: I have done some research on mkbootimg and unpackbootimg and their usage is a little bit confusing. The boot image header contains a kernel address, a ramdisk address and some other data (including the command line provided to the kernel by the bootloader). The mkbootimg tool like it is in the Android repository uses more measures: a base address, a kernel offset, a ramdisk offset etc. The kernel address is base + kernel offset and so on. When unpacking the boot image with unpackbootimg, one must suppose one of the parameters fixed, usually the kernel offset (0x00008000). Consequently, the base address is computed as the kernel address - the kernel offset. To conclude, the command line to use should be:
Code:
mkbootimg --cmdline 'console=ttyHSL0,115200,n8' --base 80400000 --ramdisk_offset 1408000 --kernel boot.img-zImage --ramdisk boot.img-ramdisk.gz -o boot-2.img
This produces the same boot.img as the one posted by flar2, except for the command line.
Edit 2: I have recompiled the kernel, disabled the /system protection, used the mkbootimg command line right above and now my phone boots and runs well. The prima_wlan.ko that I have compiled together with the kernel (3.1.7.16 from codeaurora source repository) doesn't work. It might be a matter of wi-fi firmware. Fortunately, the stock prima_wlan.ko (proprietary 3.1.7.0) module works well.
Click to expand...
Click to collapse
you have to do some haxxing to get the CAF prima to load up

Zarboz said:
you have to do some haxxing to get the CAF prima to load up
Click to expand...
Click to collapse
I am interested of that hack. Could you, please, send me a link to some documentation about it?

AlxMAX said:
I am interested of that hack. Could you, please, send me a link to some documentation about it?
Click to expand...
Click to collapse
There isn't any I have found just trial and error
Sent from my HTC One X using xda app-developers app

Zarboz said:
There isn't any I have found just trial and error
Click to expand...
Click to collapse
If you are referring to the code enclosed by #ifdef CONFIG_HTC_WIFI_NVS ... #endif in wlan_hdd_main.c, then it is not simple trial and error. Have you picked the code from somewhere else and integrated into the prima sources or have you found the prima sources containing already that piece of code?

[SCRIPTS] ROM, kernel, anykernel updater and Environment-setup scripts

Hey guys
I've been working on some scripts to automate my work for quite some time now and thought, why not share it?
So that's what I'm gonna do now
For now, we have three scripts: one to build a ROM, one to compile a kernel, to make an anykernel updater zip and to set up and entire build environment on Arch and Ubuntu-based distros (2nd post). The kernel and anykernel scripts, however, are linked, which means the kernel script automatically executes the anykernel script so as soon as you compile a kernel, you get a recovery flashable zip.
A small note: These scripts were made for the LG Optimus 4x HD, but can easily be modified for any other device.
So, you may ask yourself now 'nice, but what do these script do exactly?' well, that's what I'm gonna tell you now
ROM
The ROM build script can be found HERE
This script is made for CM 11, but can be easily modded to be used with other ROMs. I will also add some more scripts to support more ROMs in the future.
In order to make it compatible with your machine and your device, we have several configurable parameters in the beginning of the script:
Code:
# Configurable parameters
ccache_dir=/home/laufersteppenwolf/ccache/CM11
ccache_log=/home/laufersteppenwolf/ccache/CM11/ccache.log
jobs_sync=30
jobs_build=20
rom=cm
rom_version=11
device_codename=p880
If you want to use a special ccache dir, you can define it using the first parameter. Same for ccache's log.
The "jobs_sync" variable defines how many jobs will be running at the same time while running "repo sync" (---> the -j value)
The "jobs_build" variable defines how many jobs will be used for the make command
"rom" defines the ROM you are trying to compile (used in the lunch command, but also for the zip upload command)
"rom_version" defines the version of the ROM. You can change it to 10.2 or any other version depending on the CM version you are compiling.
Last but not least, the devices codename. In my case it's "p880", but you can just change it to your device (mako, hammerhead, maguro,...) and the lunch as well as the upload command will be changed to your device.
About the upload, if you have a goo.im account and also configured it in your ssh_config, you can just leave it the way it is. However, if you use a different file hoster, please edit the last command to your needs
Furthermore, we have some more features. These, however, don't need to be defined inside the script, but added to the execution command.
You can execute the script with the command:
Code:
. build.sh [-h -n -r -c -ncc -d -j #]
But what do these flags do?
Code:
-h | --help
shows a help message and exits the script
Code:
-n | --nosync
skips the "repo sync" command
Code:
-r | --release
uploads the build when it's done
Code:
-c | --clean
executes "make clean" instead of "make installclean"
Code:
-ncc | --no_ccache
disables ccache
Code:
-d | --debug
shows some debug stuff (not really needed for the "normal" user )
Code:
-j #
sets a custom number of jobs for the "make" command
Example:
Code:
. build.sh -c -r
This command syncs the repos, executes "make clean" instead of "make installclean" and uploads the build to the defined file hoster as soon as the build is done.
Kernel
The kernel build script can be found HERE
This script is made for the p880 (aka x3), but can be easily modded to work with your device.
We have 2 configurable parameters in the beginning of the script:
Code:
defconfig=cyanogenmod_x3_defconfig
jobs=32
"defconfig" defines the name of your defconfig to be used
And "jobs" defines the -j value to be used for make
All modules are being copied to ./out/modules/ and the zImage is being copied to ./out/zImage. Which means no more searching for modules inside the various folders, but having everything in one place
If you take a look into the script, the created out dir is after the above command copied to "~/smb/kernel/out". This is where my network drive is mounted. Please change it to a path of your choice.
This folder now needs to have the files from the anykernel folder inside, because the anykernel build script is being executed next. What this script exactly does will be explained later.
When the anykernel updater zip is done, the script cd's back to the place it was initially executed and will then be done.
If you don't want to make an anykernel zip, just remove the commands after
Code:
find -name '*.ko' | xargs -I {} cp {} ./out/modules/
(well, maybe except for the commands that tell you the script it done )
Anykernel updater zip
Wanna know what an anykernel updater zip is?
The anykernel updater zip only replaces the actual kernel (zImage) and not the whole boot.img. This means the ramdisk stays the same, which makes the kernel compatible with every ROM and Android version.
The script and it's needed tools can be found HERE
This script creates an anykernel updater zip from a given zImage and modules. The zImage has to be in the same folder as the script itself. The modules have to be in a folder called "modules", which also should be located in the same folder as the script. In the beginning of the script, you can define the name of the zip:
Code:
zipname="WWJB_v009_anykernel.zip"
IMPORTANT:
Please make sure to edit the updater-script so it is compatible with your device! Otherwise you can (hard) brick your device!
I am not responsible for possibly bricked devices!
Another feature of the anykernel updater is, you can mod the ramdisk while flashing (repacking the boot.img). This can be done inside THIS file. By default it creates an unsecured boot image. But you can edit way, way more using this method. For an example what can be done, you might wanna take a look at THIS file
The signapk.jar and the sign keys were taken from the latest CM11 sources.
You have a question, suggestion, bug report? Please feel free to leave a reply in this thread
Many thanks go to @GermainZ and @doixanh for helping me fighting with bash
Thanks also go to Koush, for "inventing" the anykernel updater

Environment setup script
Who is bored of setting up an entire build environment for Android? I certainly am, which is why I wrote this "tiny" script
This script sets up a complete build environment, so you can directly start to repo sync a ROM after the script is done. And the best thing is, it is extremely customizable without even having to edit the script itself. All done by appending the appropriate flags to the executing command.
A further big advantage of this script is, that it is compatible with the two most common Linux distros Arch Linux and Ubuntu (also including all distros that are based on these 2, like Xubuntu, Lubuntu,....). This compatibility is achieved by detecting the distro in the beginning of the script and then using the appropriate commands (apt-get or pacman).
The script is located HERE and called "setup.sh"
To run the script, cd to the folder the script is located and enter ". setup.sh" and append all desired flags. If you need help, just append the "-h" or "--help" flag and some help will be shown.
So, you may ask yourself now "That's nice, but what the hell does this script install/set up?" and the answer is quite simple: All you need, and (if desired) even more
The "base package" includes:
Ccache
Java
GIT
All needed build tools
Python 2
The Android SDK
Furthermore, you can also choose the "extended package" by appending the flag "-e" or "--extended", which includes:
SSH
iostat
bmon
htop
geany
The dev-host commandline tool by GermainZ
udev-rules (on Arch only)
If this still isn't enough for you, you can also add your own packages. This can be done by either editing the script and entering the packages inside the quotes of the following line:
Code:
extra="" # Add here some extra packages to install
OR
by appending the flag "-s" or "--special" and then listing the packages within quotes, seperated by spaces.
Example:
Code:
. setup.sh -s "package1 package2 package3"
This script is made to perform all actions on its own, however, it will not do so by default. To have it automated, you need to append either "-a" or "--automated". The script will then install the "base package" with all default packages (such as Oracle's JDK 6).
But you can still also append the other flags, like "-e", "-s", "-j",...
Now, let's talk about Java. This script can install 3 Java versions: Oracle's JDK 6, openJDK 6 and openJDK 7. The default version is Oracle's JDK 6. You can change the version by either appending the "-j" or "--java" flag followed by the desired version (1, 2 or 3), or if the -a flag isn't triggered, the script will ask you about it.
What? You don't want to install/change java at all? No problem! Just append "--no-java" and Java will get completely skipped.
The same thing also happens to the SDK when appending "--no-sdk".
The script has been tested on (X)ubuntu 13.10 and 14.04 by @nilse and me, and on Arch Linux by @SMillerNL and me. However, I have only tested Oracle's JDK 6, no other java versions but according to the wiki pages, it should work just fine
You have suggestions, feedback, improvements? Shoot! Just let me know and I'll do my best to include it
A big thanks goes to my testers @nilse and @SMillerNL who also made some nice suggestions. But also to @GermainZ for his dev-host commandline tool
Furthermore to @eagleeyetom for giving me the idea

Thanks awesome work it will be helpfull i was looking for such thing yesterday XD you're best @laufersteppenwolf

The anykernel script doesn't work on newer devices with zImage-dtb right?
Doesn't look like it based on the script, but I've had trouble with the M8 DTB so figured I'd ask.

u saved my life thanks buddy i will try it !!!

Hi
Fits ics?

xboxfanj said:
The anykernel script doesn't work on newer devices with zImage-dtb right?
Doesn't look like it based on the script, but I've had trouble with the M8 DTB so figured I'd ask.
Click to expand...
Click to collapse
To be honest, I have no idea
I have only tested it on the 4x HD myself, but if you use all the same commands as the the ROM build process does (maybe also grab the needed files) it should work. I mean, it does exactly the same thing as when the ROM build packs the boot image, it puts the zImage and ramdisk into one file. Only anykernel does this on the device itself, rather than on your PC.
If you want, we can take a look into it together
Marília de Oliveira said:
Fits ics?
Click to expand...
Click to collapse
Sure, should work with all ROMs, just edit some stuff (like rom_version=11 to rom_version=9 when compiling CM) and it should work just fine

laufersteppenwolf said:
To be honest, I have no idea
I have only tested it on the 4x HD myself, but if you use all the same commands as the the ROM build process does (maybe also grab the needed files) it should work. I mean, it does exactly the same thing as when the ROM build packs the boot image, it puts the zImage and ramdisk into one file. Only anykernel does this on the device itself, rather than on your PC.
If you want, we can take a look into it together
Sure, should work with all ROMs, just edit some stuff (like rom_version=11 to rom_version=9 when compiling CM) and it should work just fine
Click to expand...
Click to collapse
How do I edit? Can you explain me better .. Thanks

Thanks a lot @lauferstppenwolf
Really takes the stress out of executing commands during my kernel compiles.

Marília de Oliveira said:
How do I edit? Can you explain me better .. Thanks
Click to expand...
Click to collapse
open in notepad or gedit or whatever and edit what u want
BTW does -j number of jobs should be for example "-j6" or "-j 6"?

Marília de Oliveira said:
How do I edit? Can you explain me better .. Thanks
Click to expand...
Click to collapse
It's quite simple, open the script with an editor (I personally like Geany) and edit the following lines:
Code:
ccache_dir=/home/laufersteppenwolf/ccache/CM11
ccache_log=/home/laufersteppenwolf/ccache/CM11/ccache.log
jobs_sync=30
jobs_build=20
rom=cm
rom_version=11
device_codename=p880
example:
You want to compile CM9 for the Nexus 5 (codename hammerhead) on a dual core CPU and your username (on your PC) is username:
Code:
ccache_dir=/home/[COLOR="Red"]username[/COLOR]/ccache/CM11
ccache_log=/home/[COLOR="red"]username[/COLOR]/ccache/CM11/ccache.log
jobs_sync=[COLOR="red"]5[/COLOR]
jobs_build=[COLOR="red"]3[/COLOR]
rom=cm
rom_version=[COLOR="red"]9[/COLOR]
device_codename=[COLOR="red"]hammerhead[/COLOR]
and you're already done Your script can now compile CM9 for the Nexus 5

laufersteppenwolf said:
It's quite simple, open the script with an editor (I personally like Geany) and edit the following lines:
Code:
ccache_dir=/home/laufersteppenwolf/ccache/CM11
ccache_log=/home/laufersteppenwolf/ccache/CM11/ccache.log
jobs_sync=30
jobs_build=20
rom=cm
rom_version=11
device_codename=p880
example:
You want to compile CM9 for the Nexus 5 (codename hammerhead) on a dual core CPU and your username (on your PC) is username:
Code:
ccache_dir=/home/[COLOR="Red"]username[/COLOR]/ccache/CM11
ccache_log=/home/[COLOR="red"]username[/COLOR]/ccache/CM11/ccache.log
jobs_sync=[COLOR="red"]5[/COLOR]
jobs_build=[COLOR="red"]3[/COLOR]
rom=cm
rom_version=[COLOR="red"]9[/COLOR]
device_codename=[COLOR="red"]hammerhead[/COLOR]
and you're already done Your script can now compile CM9 for the Nexus 5
Click to expand...
Click to collapse
I have done everything ... where else do I put the script?

Marília de Oliveira said:
I have done everything ... where else do I put the script?
Click to expand...
Click to collapse
just put it inside the root of your sources (the place where you have initiated the repo) and then run it using ". build.sh [flags]"

laufersteppenwolf said:
just put it inside the root of your sources (the place where you have initiated the repo) and then run it using ". build.sh [flags]"
Click to expand...
Click to collapse
Thanks ... Worked in xperia mini pro with ics :good:

gerciolisz said:
open in notepad or gedit or whatever and edit what u want
BTW does -j number of jobs should be for example "-j6" or "-j 6"?
Click to expand...
Click to collapse
Sry for the late reply, haven't seen it at all
it's "-j x", as it only listens to the things behind the actual flag

laufersteppenwolf said:
Sry for the late reply, haven't seen it at all
it's "-j x", as it only listens to the things behind the actual flag
Click to expand...
Click to collapse
thx after reading that script i know now i thought it is same like brunch/lunch -jx

Alright guys, a big update just found its way to github: a script to set up a complete build environment on Ubuntu and Arch-based distros.
If you want me to add support for another distro, please contact me either via PM or via IRC (#TeamFun or #p880-dev) and give me details on what must be changed (install command, packages, ...)
Also the kernel script got some updates, to abort the script if the kernel didn't compile and to copy the anykernel zip directly to dropbox to share it with your testers
Cheers

laufersteppenwolf said:
You want to compile CM9 for the Nexus 5 (codename hammerhead) on a dual core CPU and your username (on your PC) is username:
Code:
ccache_dir=/home/[COLOR="Red"]username[/COLOR]/ccache/CM11
ccache_log=/home/[COLOR="red"]username[/COLOR]/ccache/CM11/ccache.log
jobs_sync=[COLOR="red"]5[/COLOR]
jobs_build=[COLOR="red"]3[/COLOR]
rom=cm
rom_version=[COLOR="red"]9[/COLOR]
device_codename=[COLOR="red"]hammerhead[/COLOR]
Click to expand...
Click to collapse
couldnt you make the script universal by using $HOME?

Wow! Great job and well written... I'll be trying this on my old archserver box that I thought I would have to convert to Ubuntu because I couldn't get all the required packages configured correctly. Looking over your scripts you've touched it all!

Thanks a bunch for the scripts, especially the env-setup one. I'll test it one of these days in Manjaro.
However, env-setup/setup.sh has a small bug: after running the script in LM16 (build environment already set up) it wants to reinstall Java no matter which flag is given.
Also, it doesn't detect installed SDK for some reason.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}