[HOWTO] chroot Ubuntu 11.10 under CM7 - Android Software/Hacking General [Developers Only]

INTRODUCTION
This document describes a procedure for installing Ubuntu 11.10 Oneiric Ocelot onto your Android device to run alongside an existing Android rom. My instructions assume you are running CyanogenMod 7. I will try to provide workarounds for other roms.
Ubuntu on a mobile device, running alongside Android, is somewhat difficult to use and is useless for most users. You should not attempt this procedure unless you have a need to run arbitrary Linux software from a mobile device. Many mobile computing needs can be met by installing remote access software on a traditional desktop computer. Conversely if you find the idea of having a general purpose Linux system in your pocket to be freaking awesome, whether you need it or not, please read on.
CAUTION
The following procedure assumes familiarity with Linux fundamentals, and basic familiarity with Android's Linux underpinnings. Data loss is possible, especially for the data on your SD card, so please make backups. Some commercial Android software is recommended, specifically Tasker and Better Terminal Emulator Pro. A large, fast SD card is also recommended. (I <3 my PNY 32 GB class 10 card)
Some optional steps are proposed that may interfere with how your Android device manages its SD card. You may be required to manually run a script before certain Android features, like mounting your device's SD card on a desktop PC via USB cable, will function properly.
Each step will have, whenever possible, a recommended procedure and one or more alternative procedures. If the recommended procedure doesn't work for you, the alternative procedure might require some research, manual-page reading, or other outside assistance to accomplish.
Shell commands should be executed on a root shell on your Android device. Use the built-in Terminal Emulator, ConnectBot in local mode, Better Terminal Emulator Pro, "adb shell" from a connected PC, or your favorite terminal emulator.
OVERVIEW
Too long; didn't read? Here's a quick summary of the steps involved:
Partition your SD card, creating a 2 GB or 4 GB Linux partition
Confirm the new partition is mounted at /sd-ext
Unpack http://cdimage.ubuntu.com/ubuntu-core/daily/current/oneiric-core-armel.tar.gz to /sd-ext
Create an empty file /sd-ext/.nomedia
Edit /system/etc/init.d/05mountsd to bind proc, dev, and sys to /sd-ext
Optionally edit /system/etc/init.d/05mountsd to bind /data, /system, and /cache to /sd-ext/mnt
Prepare a script to open a chroot shell with correct environment variables
Update /etc/apt/sources.list and /etc/resolv.conf according to your preference
Overwrite /sbin/start with /sbin/true
Copy or link important files into /etc and /lib
Initialize the package manager and begin downloading packages
(optional) Integrate Ubuntu services with Tasker profiles
STEP 1: Partition your SD Card
While a basic Ubuntu system can be installed in as little as 200 MB of disk space, many software packages require more space. Also many Linux tasks require unusual file attributes, permissions, or types, which cannot be placed on your SD card's regular FAT32 / VFAT file system.
Recommended: Use an existing Ubuntu system, or even an Ubuntu live CD, and a USB card reader to repartition your SD card without losing any data. Be sure the first primary partition remains FAT32 / VFAT, and create a second primary partition of type ext3 or ext4.
Alternative 1: Use any desktop system to back up all of the data on your SD card, either with a card reader or using your Android device. Use your phone's recovery partition to destructively repartition your SD card with a 2 or 4 GB ext3 or ext4 partition, and NO swap partition. Then copy your SD card contents back to your SD card.
Alternative 2: If you don't want to partition your SD card, you can create large empty files on your SD card and mount those. Some of the later steps will work differently and you may have to improvise.
Create large empty files on your SD card to contain your Ubuntu filesystem: dd if=/dev/zero of=/mnt/sdcard/UBUNTU.IMG bs=32768 count=65536
Identify a free loopback device number that isn't currently in use and won't likely be occupied by Android Apps2SD in the future: ls /dev/block
Create a new loop device for your file. Note the first number is the loop filename number you observed wasn't in use (at least a few dozen higher than the highest number you see), then a b and a 7, and then a number one less than the filename number. For example: mknod /dev/block/loop200 b 7 199
Associate the loop device with your new empty file: losetup /mnt/block/loop200 /mnt/sdcard/UBUNTU.IMG
Find which versions of mkfs are on your system: ls /system/xbin/mkfs*
Using whatever version you have (ext4 or ext3 preferred, but ext2 is acceptable) run: mkfs.ext# /dev/block/loop200
Save these commands in a script for later use. The dd and mkfs commands were only used this once, but the mknod and losetup commands will have to be run manually from a script each time you boot your phone.
STEP 2: Mount your Linux Partition
To avoid causing problems for your Android system, we shouldn't tamper with any of Android's critical files or directories. We will mount our Linux partition in a new directory that Android isn't using. With any luck, our rom already contains scripts that will automatically mount this for us.
Recommended: If you have partitioned your SD card and you are running CyanogenMod 7, use a root shell or file manager to confirm /sd-ext contains at least a directory called lost+found. If lost+found is present in /sd-ext then this step is done and you should continue to step 3.
Alternative 1: Use a root shell to navigate to /etc/init.d and look for a file called 05mountsd. If that file doesn't exist, you may need to create it or use another alternative method. If it does exist, run it with ./05mountsd and observe any error messages. If fsck or mount are failing because the filesystem type doesn't match what was expected, you can edit the file and change the filesystem type in the script.
Hint: you can do a quick search-and-replace using sed and a regular expression:
Code:
mv 05mountsd 05mountsd-old
cat 05mountsd-old | sed -e s/ext3/ext4/ > 05mountsd
diff 05mountsd-old 05mountsd
#changes look good to you? then proceed:
rm 05mountsd-old
ls -al
chown root.shell 05mountsd
chmod 750 05mountsd
#adapt owner (root), group (shell) and permissions (-rwxr-x---) to match the other files in that directory
Alternative 2: If your rom doesn't create, or attempt to mount, anything called /sd-ext you might have to create your own mount point.
Code:
mount -o remount,rw /system
mkdir /system/sd-ext
mount -o remount,ro /system
Then, in the following steps, replace any reference to /sd-ext with /system/sd-ext.
STEP 3: Install the Base System
Recommended: Download this file: http://cdimage.ubuntu.com/ubuntu-core/daily/current/oneiric-core-armel.tar.gz (Should be under 32 MB, and is the only potentially-insecure download required because the file is unsigned.)
Store it on your SD card somewhere, like in /mnt/sdcard/download/. (If you use your Android device's browser, it will probably land there automatically.)
If you're nervous about making a mistake, double-check your system partition is read-only:
Code:
mount | grep -v block/dm- | grep \(rw
#If anything there looks like it shouldn't be read-write, make it read-only:
mount -o remount,ro /system
Open a root shell and navigate to /sd-ext. Use pwd (present working directory) to confirm you really are in /sd-ext. Then go ahead and unpack:
Code:
tar zxvf /mnt/sdcard/download/oneiric-core-armel.tar.gz
STEP 4: Create a .nomedia file
Android may try to scan your Linux partition for images, videos, and pictures. The Linux partition contains many files and directories and this scan can take a long time. To avoid these lengthy scans, we will add a file to the Linux partition that blocks well-behaved Android services from scanning.
Recommended: Use a root shell to navigate to /sd-ext. Create the file and set its permissions:
Code:
echo > .nomedia
chown root.root .nomedia
chmod 644 .nomedia
STEP 5: Bind /proc, /dev, and /sys to the Linux partition
Recent versions of Linux provide a way to clone a partition, so when changes are made to one copy of the partition the changes take effect immediately in the other copy. We will use this to make the /proc, /dev, and /sys partitions used by Android available in our Linux partition as well.
This is necessary because when we run Ubuntu we will be using chroot (change root) to make /sd-ext the only filesystem Ubuntu can see. Ubuntu will need to be able to see /proc, /dev, and /sys, as well as some of the files in /etc, /vendor/lib, and /system/lib. We will meet some of these needs with mount --bind
Additionally we will need these bindings to be recreated every time the /sd-ext partition is mounted. So we will add some commands to the run-parts script that mounts /sd-ext, located at /etc/init.d/05mountsd.
Recommended: Open a root shell and navigate to /etc/init.d, and use a text editor to open 05mountsd.
Near the end of the file you should see some commands that operate on /sd-ext:
Code:
$BB chown 1000:1000 $SD_EXT_DIRECTORY;
$BB chmod 770 $SD_EXT_DIRECTORY;
Edit the chmod command and change the permission mask number from 770 to 775, which grants non-root users the ability to read the contents of that directory. Then add the following commands immediately after the chown and chmod commands already present:
Code:
$BB mount --rbind /dev /sd-ext/dev
$BB mount --rbind /proc /sd-ext/proc
$BB mount --rbind /sys /sd-ext/sys
STEP 6: (OPTIONAL) Bind /system, /data, and /cache to /sd-ext
This step is optional, and provides a way to operate on Android system files from within Ubuntu. If you're concerned about a stray command destroying your Ubuntu system and you want to make sure it doesn't take any Android system files with it, you should skip this step.
Recommended: Open a root shell and navigate to /sd-ext/mnt, and create some directories which will serve as mount points:
Code:
pwd
# you should be in /sd-ext/mnt
mkdir -m 700 system data cache
chown root.root system data cache
ls -al
# Each of system, data, and cache should show drwx------ and root for both owner and group
Then navigate to /etc/init.d and edit 05mountsd. Add the following under the existing mount --rbind commands you added earlier:
Code:
$BB mount --rbind /system /sd-ext/mnt/system
$BB mount --rbind /data /sd-ext/mnt/data
$BB mount --rbind /cache /sd-ext/mnt/cache
STEP 7: Prepare a script to open a chroot shell with proper environment variables
Now you will need an easy way to create a first process, a shell, in the new environment you've created. This shell will start life in a different world than regular Android shell processes live in: it will use a different root filesystem and will be able to see different files and directories, and it will have access to a vast number of full-featured programs and services.
We will be using the Bourne Again Shell, bash. We will launch bash with a program called env, which sets some environment variables and then launches a program with that new environment. Finally, we will be launching env with a program called chroot, which changes the root filesystem seen by a program before it launches it.
Option 1: Better Terminal Emulator Pro: I don't want to say "recommended" because it involves buying software that's not really necessary. I'm partial to Better Terminal Emulator Pro because it seems to have more available options (compared to the built-in Terminal Emulator) and it doesn't kill your shell unexpectedly when network connectivity changes (unlike ConnectBot). I'm not affiliated with the author of that software. Also, after I bought that I stopped looking for alternative shells, so there may exist other alternative shells that work better for this.
Use Better Terminal Emulator Pro and add the commands necessary for starting your chroot shell to BTEP's options. Once you do this, BTEP will essentially be a dedicated Ubuntu console for you, and may be difficult to use for other tasks.
My settings are:
Shell: Android Shell
Command Line: /system/bin/sh
Initial Command:
Code:
/system/xbin/chroot /sd-ext /usr/bin/env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=linux /bin/bash -l; exit
clear
Start as root: yes
Option 2: Terminal Emulator, and a shell script: Open a root shell and create a file:
Code:
#!/system/bin/sh
/system/xbin/chroot /sd-ext /usr/bin/env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=linux /bin/bash -l
exit
Save the file with whatever name you like, perhaps "ubuntu", and put it somewhere in your path. You could add it to /system/bin: mount -o remount,rw /system; mv ubuntu /system/bin/ubuntu; mount -o remount,ro /system
Option 3: ConnectBot: I don't recommend ConnectBot for this because, while it does offer "local" connection types, if your connectivity changes (even wifi to cell) it will close your "connection" immediately.
That said: create a new connection with type local, and give it the following initial command:
Code:
su -
/system/xbin/chroot /sd-ext /usr/bin/env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=linux /bin/bash -l; exit
clear
STEP 8: Update /etc/apt/sources.list and /etc/resolv.conf
Ubuntu uses a package manager which automatically finds, downloads, verifies, and installs software for you. The package manager requires a list of trusted repositories which contain binaries built for ARM CPUs, like the one in your Android device.
FIXME: Later I'll discuss the pros and cons of each of these selections. For now I'll just provide my sources.list file.
Open your Ubuntu shell, using the method from step 7, and issue the following commands:
Code:
echo nameserver 8.8.8.8 > /etc/resolv.conf
# you can use a nameserver of your choice if you like.
echo nameserver 8.8.4.4 >> /etc/resolv.conf
# notice the >> instead of > which means we're appending, not overwriting.
apt-get install nano
# follow the prompts - say yes if it asks if you want to proceed.
nano -w /etc/apt/sources.list
And here are the contents of my sources.list file, containing binary sources for ARM CPUs:
Code:
deb http://ports.ubuntu.com/ubuntu-ports/ oneiric main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ oneiric-security main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ oneiric-updates main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ oneiric-backports main restricted universe multiverse
deb http://archive.canonical.com/ oneiric partner
deb http://archive.canonical.com/ oneiric-security partner
deb http://archive.canonical.com/ oneiric-updates partner
deb http://archive.canonical.com/ oneiric-backports partner
STEP 9: Overwrite /sbin/start with /sbin/true
Nothing too complicated here. Some Ubuntu packages install services, which the system will usually want to automatically start and end when the system boots and shuts down.
However, on an Android device Android is responsible for booting and shutting down the system, and it doesn't know much about Ubuntu services.
Later (FIXME: when?) we'll discuss how to use an Android program like Tasker to more intelligently manage Ubuntu services. But for now, we need to trick Ubuntu's package installation scripts into believing it has successfully started a service. We will use a useless program called true, which does nothing and reports that it was successful, in place of a program called start, which will be unable to start a service because it has no access to Android's init system.
Open your Ubuntu shell and do this: cp /sbin/true /sbin/start
STEP 10: Copy or link important files into /etc and /lib
FIXME: I haven't finished this section yet. Some of my software remains broken until I fix this.
STEP 11: Initialize the package manager and begin downloading packages
Now we can begin using our Ubuntu system's package manager to install any other packages we might need.
Open your Ubuntu shell and do the following:
Code:
apt-get update
# This will take a short while, and will download updated software catalogs from each of the update servers
apt-get upgrade
# This looks for updates to any currently-installed packages. There aren't many packages installed so this should be quick.
apt-get install PACKAGE NAMES GO HERE
Then keep using apt-get install to install whatever you need.
Here are some common packages and package sets:
ubuntu-dev-tools: development tools including a compiler and linker
tightvncserver: simple X server you can connect to with androidVNC
FIXME: I need to put a lot more here.

Sorry for bumping old threat, but I am kind of stuck. Everything goes fine, but when I try to create the script in BTEP (and android terminal) I am getting segmentation fault in both of them.
Checked for syntax errors 100 of times. I am using CM6 froyo 2.2 on Motorola milestone. SD card has 2 gb of ext3 partition mounted in /sd-ext.
*EDIT*
Reflashed CM 7 latest and same thing happens. Segmentation Fault.

Related

Debian on Haykuro ADP 1.5

Debian on Haykuro ADP 1.5
I have seen people running into issues getting Debian running on cupcake builds so I decided to post a guide on how I did it. It's important to note that I got this information from http://www.saurik.com/id/10 and http://android-dls.com/wiki/index.php?title=Haykuro_Apps_to_SD. I have simply made some changes for it to applicable to the current setup and to use the native ext2 partition instead of mounting an image like people have been doing with Debian. I would also like to say thanks to everyone in the community for their continued hard work putting out this builds etc.
1.) Get Haykuro ADP 1.5 Apps2SD up and running.
A. Check if your phone is recognised.
adb devices
B. Mount the phone into ADB.
adb remount
C. Bring up the shell.
adb shell
D. Check if /system/sd is visible.
df
E. Skip this step if you already have your apps on SD.
cp -a /data/app /system/sd
F. Skip this step if you already have your app-private on SD.
cp -a /data/app-private /system/sd
G. Remove the current apps folder on device.
rm -r /data/app
H. Remove the current app-private folder on device.
rm -r /data/app-private
I. Link the app folder on sd to the one on device.
ln -s /system/sd/app /data/app
J. Link the app-private folder on sd to the one on device.
ln -s /system/sd/app-private /data/app-private
K. Finally Reboot.
reboot
*You should now have a fully functioning apps2sd build running. We are now going to start getting the files for Debian.
2.) We need to get debbootstrap on the system you are working from. Hopefully you are running a debian based distribution.
apt-get install debootstrap
3.) Next we need to create a folder to mount the ext2 partition of the card to. This is fine being on for instance your desktop.
mkdir sd
4.) Plug in your phone and make sure you choose the mount option.
5.) If your system auto-mounts the drive you will need to unmount it as it is probably mounted with noexec. (This is where mine was mounted for example.)
umount /media/disk1
6.) Now lets mount the ext2 partition on your sdcard. (Note your drive and partition may differ from mine.)
mount /dev/sdb2 sd
7.) At this point I created a folder as I didnt want debian cluttering my partition but this is up to you.
cd sd
mkdir debian
8.) Next lets actually get some files on there.
debootstrap --verbose --arch armel --foreign lenny debian http://ftp.de.debian.org/debian
9.) Now we can unmount the partition.
umount debian
10.) For the next steps we need to get into adb shell again. (See above for getting into shell.)
11.) Now we need to set a couple variables. (Note the mnt variable is set to where you placed your debian files Haykuro Apps2SD mounts the ext2 partition at /system/sd from there we have my debian folder I created.)
export mnt=/system/sd/debian
export PATH=/usr/bin:/usr/sbin:/bin:$PATH
export TERM=linux
export HOME=/root
12.) Now we just have to have debbootstrap extract and configure everything. (This is gonna take 10-15 minutes.)
chroot $mnt /debootstrap/debootstrap --second-stage
13.) From here we have a couple fixes to make.
A.) We need to fix the sources.list for apt.
echo 'deb http://ftp.de.debian.org/debian lenny main' >$mnt/etc/apt/sources.list
B.) We need to fix the resolv.conf
echo 'nameserver 4.2.2.2' >$mnt/etc/resolv.conf
14.) Now we are ready to go into Debian.
chroot $mnt /bin/bash
15.) Let's tidy up mtab.
rm -f /etc/mtab
ln -s /proc/mounts /etc/mtab
15.) And get everything mounted.
mount -t devpts devpts /dev/pts
mount -t proc proc /proc
mount -t sysfs sysfs /sys
16.) Don't forget to set a password for root.
passwd root
You know have a fully functioning Debian in a chrooted environment on a cupcake build. Now you can install OpenSSH or whatever else from here using apt-get. I haven't covered unionfs or various scrip
When you restart your phone will need to resetup some items. You can create a shell script with the commands listed below.
-------------------------------------------------------------------------------------------
export mnt=/system/sd/debian
export PATH=/usr/bin:/usr/sbin:/bin:$PATH
export TERM=linux
export HOME=/root
mount -t devpts devpts $mnt/dev/pts
mount -t proc proc $mnt/proc
mount -t sysfs sysfs $mnt/sys
chroot $mnt /bin/bash
-------------------------------------------------------------------------------------------
Here is the script for those that would prefer to download it.
Debian Boot Script
How much space does this take in the ext2 partition?
What would you need to do to add a GUI such as Gnome?
Nice work.
Wow. THANKS!!!! I've been trying to figure out how to do this FOREVER.
How do you completely remove debian off your phone, not just your sdcard, so you can reinstall.
could you place a better explanation on step 2?
2.) We need to get debbootstrap on the system you are working from. Hopefully you are running a debian based distribution.
apt-get install debootstrap
I dont know what is debootstrap or where to run this commands
Thanks
P.S I am running a Mac
My ext2 partition currently has 163 MB used between the debian install and some apps I have on it. The size of the partition all depends on what you want to do with it. I gave myself 500 MB.
juangil said:
could you place a better explanation on step 2?
2.) We need to get debbootstrap on the system you are working from. Hopefully you are running a debian based distribution.
apt-get install debootstrap
I dont know what is debootstrap or where to run this commands
Thanks
P.S I am running a Mac
Click to expand...
Click to collapse
What I mean is for this to work you need to be running Debian Linux or a derivative thereof. I for instance did this from Kubuntu 9.04. These instructions wouldn't work from a Mac as you wouldn't have apt-get etc.
whoaaa
first post, omg.
great guide, dude.
kronarq said:
What I mean is for this to work you need to be running Debian Linux or a derivative thereof. I for instance did this from Kubuntu 9.04. These instructions wouldn't work from a Mac as you wouldn't have apt-get etc.
Click to expand...
Click to collapse
Thanks I appreciated your explanation, I have ubuntu installed but no experience at all with it, I'll try to follow your instructions
one more question, on step 12..
10.) For the next steps we need to get into adb shell again. (See above for getting into shell.)
11.) Now we need to set a couple variables. (Note the mnt variable is set to where you placed your debian files Haykuro Apps2SD mounts the ext2 partition at /system/sd from there we have my debian folder I created.)
export mnt=/system/sd/debian
export PATH=/usr/bin:/usr/sbin:/bin:$PATH
export TERM=linux
export HOME=/root
12.) Now we just have to have debbootstrap extract and configure everything. (This is gonna take 10-15 minutes.)
chroot $mnt /debootstrap/debootstrap --second-stage
do I continue in adb or where do I go to do step 12? and, after I type whats in step 12 should I expect something to happen?
Thanks
juangil said:
one more question, on step 12..
12.) Now we just have to have debbootstrap extract and configure everything. (This is gonna take 10-15 minutes.)
chroot $mnt /debootstrap/debootstrap --second-stage
do I continue in adb or where do I go to do step 12? and, after I type whats in step 12 should I expect something to happen?
Thanks
Click to expand...
Click to collapse
This step should be done in adb shell. This is what unpacks everything and sets it up on the ext2 partition. Before this the files are all packed up and nothing will run. You should see quite a few things happening at this stage. What are you getting?
Can I use debootstrap on pc and then copy the debian folder into the phone sd card? Don't have an usb cable with me, but wanna try this...
Just did it - it works. Downloaded files from pc then continue with terminal application.
Nice guide - thank you!
Question. Does it make the phone run slower or decrease battery life?
kronarq said:
This step should be done in adb shell. This is what unpacks everything and sets it up on the ext2 partition. Before this the files are all packed up and nothing will run. You should see quite a few things happening at this stage. What are you getting?
Click to expand...
Click to collapse
I'm getting this:
Code:
chroot $mnt /debootstrap/debootstrap --second-stage
chroot: cannot execute /debootstrap/debootstrap: No such file or directory
MorphWVUtuba said:
I'm getting this:
Code:
chroot $mnt /debootstrap/debootstrap --second-stage
chroot: cannot execute /debootstrap/debootstrap: No such file or directory
Click to expand...
Click to collapse
Did you set the variables?
Code:
export mnt=/system/sd/debian
export PATH=/usr/bin:/usr/sbin:/bin:$PATH
export TERM=linux
export HOME=/root
You can check it:
Code:
# echo $mnt
/system/sd/debian
Okay, I must've typo'd. I ran those commands again & this time it's installing/unpacking stuff. Thanks for the help.
how do i get those files (debootstrap, verbose, arch armel, and foreign lenny?
I go to the site linked there, but I do not see them listed.
Hi. I was just install debian, but i have problem with apt-get.
I want use
Code:
apt-get install icewm
but it results is
Code:
apt-get install icewm
Reading package lists... Done
Building dependency tree... Done
E: Couldn't find package icewm
Any ideas
You probably need to add repo's to /etc/apt/sources.list
I think I'm missing something obvious.
I got through every step of this tutorial, and I can't boot Debian.
If I go to a term and $bootdeb, I get 'permission denied.' So I $su and #bootdeb and I get 'not found.' I've tried cd'ing to /system/sd/debian and get the same results. Any ideas?
I followed all of the steps very carefully and had no troubles. I'm running the latest JF, 1.43 w/ 1.5 ADP. But I was using Haykuro 1.5 before this - so no flash. Is that what's preventing me from running?
Thanks

A Real Linux Android System

Before reading.
This article requires that your phone is Rooted and has Busybox installed.
HTC owners, S-off your phone as well.
Android is a great operating system for SmartPhones. Only problem with it (From a more technical users point of view) is the small useless bad excuse of a linux system running behind the UI. Some of the smaller issues can be fixed by installing Busybox that provides a better basic set of tools to work with, but it's not enough. What I would like is to be able to use almost any real linux cli application, daemon, library etc. Also some compilers would be great. I'm using an HTC desire which has more than enough power to do things like compiling (I know, I have done it.), and so has a lot of other SmartPhones. Why would one need this when there is cross compilers you can use on regular and faster machines? Well, because It's fun, why else. I did not spend all that money on a SmartPhones just to make a few phone calls and write a few text messages. My old Sony Ericsson did that quite well. I spend that money to get a phone with many possibilities that I could play with.
I tried to search the web for some helping instructions on how to port a real linux system to this phone. What I mostly got, was people who ported linux systems that would run as an extra OS on the phone, some even with an Xserver accessible via VNC. Sounds fun, but not what I was after. I did not want an extra OS using CPU and unnecessary RAM. I just wanted some extra tools, libs etc to work with and the regular linux user management. I managed to find an old article from someone using Debians armel version on an old G1 phone. I thought that if it worked on that phone, it would be possible to do so on mine. So I started playing around with that.
The way this works is by installing a bare minimum debian on an second partition. This will not boot anything up since it will just work as extra tools and lib using the already booted android linux system.. You will still only have one OS but more tools (Including the nice APT package system) to work with.
Now there is two ways of using debian on android.
The first way is to setup the Debian in an chroot environment. This has some flaws like I have not yet been able to make the chroot start second jobs, which means that you will be able to start applications and daemons running in the current shell session. But you will not be able to make them run in the background. But as an development environment it will still work great, as you will be able to use the Debian libs and compilers.
The second way (The way I was after) is to setup Debian using the UAFS kernel module. This is used to merge directories into one with different rules for each. For an example you would be able to merge /debian_root/etc and /system/etc into a shared /etc with RW on Debians etc and RO on Androids forcing any changes to be made in /debian_root/etc. Using this option you will use Debian to extend the Android Linux part. Android UI with Debian Shell.
This article is split into 3 parts. The first part is the setup of Debian. Must be done whether you want chroot or a merged system. In the second part we will cover the setup of the chroot environment.
The third part is the setup of the merged system. This will require one thing in order to work.
You will need a kernel with the uafs module which is not default in most android kernels. You will need to compile your own kernel. People using the Thalamus kernel, can download a kernel with this module kernel-2.6.37.2_AUFS.zip. A big thanks to Thalamus for providing this. Note that this kernel was a special request from me that he was kind enough to provide. It comes with absolutely no guaranties. Do not email spam him about errors you might encounter using it as this is not an official release.
Note that once you mount the Debian system, you will not be able to mount the SD Card when connecting it to your PC. You can download the App “Multi Mount SD-Card” from Market which will be able to mount your SD Card on your PC while mounted on your phone. This app should also speed up transfers on 2.6.37 kernels, or so I have heard.
Part 1: Building Debian
We are going to run debian from a file containing a ext2 filesystem. If your SD Card is big enough you might be able to create an extra partition and force it to mount as something else other than ext-sd, but in this article we are going to use a partitioned file.
Also, this will require that you have the tool “debootstrap” on your PC. This is found in Ubuntu and Debian, dont know about other Linux distro's or Operating systems. If you don't use Linux or an distro with debootstrap you can download a Ubuntu Live CD from the Ubuntu website. This can be run on CD, DVD and USB disks without installing it. For this article I have used Ubuntu 10.10.
Open a terminal on your Ubuntu/Debian/LiveCD PC and cd to the folder where you want to temp. store your debian. I will call this path “debian_dir”
The first thing we will do is to create the partitioned file.
I'm creating a file with 1.5GB of space. You can create the size you want.
Don't use 0 in the size, dd don't like that.
Code:
cd debian_dir
dd if=/dev/zero of=debian.img seek=1499999999 bs=1 count=1
mke2fs -F debian.img
mkdir debian
mount -o loop debian.img debian/
Now that we have created the file and mounted it, we should download and place debian inside of it. First we will install debootstrap.
Code:
apt-get install debootstrap
debootstrap --arch armel --foreign squeeze debian http://ftp.us.debian.org/debian
umount debian/
Debian Squeeze is Debian 6.0 which replaces the old Lenny.
Copy the debian.img file to your SD Card on your phone. I placed mine in the root of the SD Card.
Now open a terminal on your phone. You can download one from market. There are some free once, but the paid “Better Terminal” is the best to work with in my opinion.
Note: All shell work is done on the phone from now on.
First we need some temp. exports.
Type the fallowing:
Code:
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
export HOME=/root
export TERM=linux
export DM=/data/local/debian
export DI=/sdcard/debian.img
Now we will create the Debian dir and then create a loop device which we will use to mount the Debian image to the Debian dir.
Code:
mkdir -d $DM
mknod /dev/loop0 b 7 0
mount -o loop,noatime $DI $DM
Now that we have debian mounted in our debian folder, we need to finalize the installation. Not all files as been installed yet. Don't forget to activate Mobile Network or Wifi.
chroot $DM /debootstrap/debootstrap –second-stage
We also need to insert the debian repository and a DNS (I use OpenDNS).
You can use whatever you want for the DNS, you routers IP for an example. It should have some default Ips configured.
Code:
echo “deb http://ftp.us.debian.org/debian squeeze main contrib non-free” > $DM/etc/apt/sources.list
echo “208.67.222.222” > $DM/etc/resolv.conf
Now change the root to the debian dir and set a root password and make sure that certain folders already exists.
Code:
chroot $DM /bin/bash
passwd root
mkdir root
exit
Now we need to make sure that certain variables exists and that the values is as they should be. So we will alter the /system/bin/sh that android uses as login shell.
Code:
cd /system/bin
mv sh sh0
Create a new /system/bin/sh file with the fallowing content.
This will ensure that these variables is created and/or changed every time you enter a new shell session. You can call another shell like bash or ash instead /system/xbin/sh, just make sure to use ash or sh at the top, since bash does not seam to understand [email protected]
Warning: It is very important that the /system/bin/sh file is made correct. Android will not boot if anything is wrong with this file. It should be exactly like the one I posted, and you cannot make it on your PC and push it to the phone. Don't know why, it works fine with other shell scripts, but not that one.
Enter an adb shell from your PC and use vi or nano (You can download nano here here) to copy and paste the content directly into the ADB Shell.
Note: ADB from the recovery will always work. It does not use /system/bin/sh
Code:
#!/system/xbin/sh
/system/xbin/sh --login [email protected]
Make sure that it has execute rights. Then type sh in the terminal to make sure that it works before you close your ADB Shell session.
Code:
chmod a+x /system/bin/sh
sh
sh0 will not be used, it's up to you wether you want to save it or not. Just make sure that /system/xbin/sh exists, otherwise
copy /system/bin/sh0 to /system/xbin/sh.
Now we need the profile file that is needed by this method. This file will make sure that we have all our variables and the data that they should contain.
Android does not keep very good track of these things. Place this data in the file /system/etc/profile
Code:
export TERM=linux
export TERMINFO=/etc/terminfo
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/vendor/bin:/system/bin:/system/xbin
if [ "$EUID" == "0" ]; then
if [ -z "$HOME" ] || [ -z "$USER" ] || [ -z "$GROUPS" ]; then
export HOME=/root
export USER=root
export GROUPS=0
fi
else
# Make sure that this user id variable exists.
if [ -z $EUID ]; then
export EUID=$(echo $(id | cut -d "(" -f 1) | cut -d "=" -f 2)
fi
# If we have an /etc/passwd file available,
# we should also make sure to get name, group and home info from that file and
# store in the default linux variables.
if [ -f /etc/passwd ]; then
# Create the user info variables based on user id and /etc/passwd
export USER=$(grep "x:${EUID}:" /etc/passwd | cut -d : -f 1)
export GROUPS=$(grep "x:${EUID}:" /etc/passwd | cut -d : -f 4)
export HOME=$(grep "x:${EUID}:" /etc/passwd | cut -d : -f 6)
fi
fi
export ADID=$EUID
if [ "$PS1" ]; then
if [ "$BASH" ]; then
# The file bash.bashrc already sets the default PS1.
if [ ! -z $HOME ] && [ -f $HOME/.bashrc ]; then
. $HOME/.bashrc
elif [ -f /etc/bash.bashrc ]; then
. /etc/bash.bashrc
else
export PS1='\h:\W\$ '
fi
else
if [ "$EUID" == "0" ]; then
PS1='\w # '
else
PS1='\w $ '
fi
fi
else
PS1='\h:\W\$ '
fi
if [ -d /etc/profile.d ]; then
for i in /etc/profile.d/*.sh; do
if [ -r $i ]; then
. $i
fi
done
unset i
fi
We will also need an /etc/bash.bashrc which is needed because /etc/profile not always gets executed.
Code:
if [ "$ADID" != "$EUID" ]; then
# Some times /etc/profile is not executed (For an example when using Androids su command).
# /etc/bash.bashrc however it used instead. So we just execute it manually in these cases.
# If you create a local bashrc ($HOME/.bashrc), remember to include this check.
# /etc/bash.bachrc is not used when a local file exists.
if [ -f /etc/profile ]; then
. /etc/profile
# Since /etc/profile links back to this file,
# we will make sure not to start an unending loop.
# the profile file should update this, but it's better to be on the safe side.
if [ "$ADID" != "$EUID" ]; then
export ADID=$EUID
fi
fi
fi
PS1='\h:\W\$ '
And last we have some user and group configurations in order to make Debian better work with Android. Open your Android terminal emulator and make sure it opens as user and not root. Now type "/system/xbin/id".
What we need here is the uid, gid and the ids of each accosiated groups along with the names of it all. In my terminal I have "uid=10048(app_48) gid=10048(app_48) groups=1015(sdcard_rw),3003(inet)".
What we will do it create this user along with the groups in our Debian inveroment. However, I use the uid name and gid name "bterm" instead of app_48, but still with the original IDs. The inet and sdcard_rw groups will be created as system groups. Do the same for your app user and group if you don't want any login possibilities with this user. I however have assigned home and password to this user.
This needs to be doen in chroot
Code:
chroot $DM /bin/bash
groupadd --system --gid 1015 sdcard_rw
groupadd --system --gid 3003 inet
groupadd --gid 10048 bterm
adduser --home /home/bterm --gid 10048 --uid 10048 bterm
gpasswd -a bterm sdcard_rw
gpasswd -a bterm inet
exit
Now your ready for the next step. Remember that the user you have created, is the user used by your terminal emulator when you are not logged on as root. This ID is only assigned to that specific app, if you install another terminal, it will get another ID.
Part 2 – Setting up a chroot environment
The best way of doing this is to make two small scripts and place them in /system/bin or /system/xbin. One script for doing the mount after phone boot. And one script for entering the chroot when open a new shell session.
Here is the mounting script that I call mkdebian.
This needs to be executed once every time the phone has rebooted. If your ROM supports init scripts, you can add it there for auto-mount on boot.
Code:
#!/system/bin/sh
export DM=/data/local/debian
export DI=/sdcard/debian.img
# Create the loop device and mount the debian image
mknod /dev/loop0 b 7 0
mount -o loop,noatime $DI $DM
# Let's make sure that we can access our sd card from within the chroot
if [ ! -d "$DM/mnt/sdcard" ]; then
mkdir $DM/mnt/sdcard
fi
mount --bind /mnt/sdcard $DM/mnt/sdcard
# Now we make sure that the chroot has access to all the device maps
if [ ! -d "$DM/dev" ]; then
mkdir $DM/dev
fi
mount --bind /dev $DM/dev
# We also need to make proc, sys etc available from within the chroot
chroot $DM mount -t proc proc /proc
chroot $DM mount -t sysfs sysfs /sys
# And last fix the mtab
chroot $DM rm -f /etc/mtab
chroot $DM ln -s /proc/mounts /etc/mtab
echo “Debian was mounted in $DM/”
Now for the script that I called “debian”.
This script will make you enter the chroot environment using one single command. Every time you exist the debian chroot, you can enter back using the command “debian”.
Code:
#!/system/bin/sh
chroot /data/local/debian/ /bin/bash
And that is it. Open a terminal, type debian and start using your debian environment. It might be a good idea to run aptitude update or apt-get update before you start using the package manager.
Part 3 – Settings up a merged Android/Debian environment
This is the fun part to play with. A merged Android/Debian environment without the usage of chroot. This only requires a script to be executed at boot, or after boot.
Remember that this will not work unless your kernel has the aufs module.
Code:
#!/system/xbin/sh
export DM=/data/local/debian
export DI=/mnt/sdcard/debian.img
# Now we mount the debian image to our debian folder.
mknod /dev/loop0 b 7 0
mount -o loop,noatime $DI $DM
# These two directories exists in both android and debian,
# so these are going to be merged together.
mount -t aufs -o dirs=$DM/etc=rw:/system/etc=ro aufs /system/etc
# For some reason, mounting /sbin with aufs does not work.
# So we make a little cheat with symbolic links instead.
mount -o remount,rw /
mv /sbin /sbin0
ln -s $DM/sbin /
rm -rf /sbin/adbd
rm -rf /sbin/ueventd
ln -s /sbin0/adbd /sbin/
ln -s /sbin0/ueventd /sbin/
# Let's get all the debian root directories out to the real root dir.
rm -rf /root
for x in tmp root lib bin boot home media opt selinux srv usr var
do
ln -s $DM/$x /
done
if [ -f /system/xbin/bash ]; then
mount --bind /bin/bash /system/xbin/bash
mount --bin /system/xbin/bash /system/xbin/sh
else
mount --bind /bin/bash /system/xbin/sh
fi
echo "Debian has been merged with Android..."
Run this at startup and start using all of debians possibilities (Don't run it yet).
The / dir is left with RW permissions. If this is set to RO then apt-get will not work. It writes tmp log files to this dir when running. It does not matter since all changes to this dir is restored to default on every boot.
Also dont forget to run apt-get update or aptitude update to update the package list.
You are now able to install and run almost anything (CLI Only) that runs on debian systems and other linux dists. Personally I'm currently running OpenSSH Server and Lighttpd with PHP. A perfect small transportable web server.
You can use either /etc/init.d/[your service] start
or
service [your service] start
to start daemons you have installed.
Now you can use the "mkdebian" command to merge your new Debian with Android.
Note: The command “su” from debian works differently than the android “su” command.
The debian command uses the regular linux management to provide access which means that it will ask for your root password that you created in the first part. The android command will just provide you with su rights without a password. Both commands provides su rights to both android and debian, so it is possible to remove the android command and create a new Android/Java based SU app for the Android UI that will require root password.
Note: There is a small problem with the Android "su" command. It does not recheck the /etc/profile.
That means that if you log on using the regular user, and then uses the "su" command, user variables does not get updated with su paths.
Using the Debian "su" command will work fine, just note that this command will promp for password. Also logging in using the root account
directly works fine.
Good luck
[Edit: 2011-03-09 11:06]
- Changed the /etc/profile script in Part 1
- Added /etc/bash.bashrc script in Part 1
- Changed User and Group setup and moved it from Part 3 to Part 1
- Fixed error in /system/bin/sh
All of this will fix some permission problems with Debian
[Edit: 2011-03-08 11:21]
- Fixed problems with user management by adding an /etc/profile and changing /system/bin/sh and the debian merge script. Using the "login" command is no longer needed.
[Edit: 2011-03-07 08:49]
- Fixed bug in Part 1 sh script.
[Edit: 2011-03-07 00:20]
- Added a new /system/bin/sh script to Part 1
- Added the way we create missing users and groups in Part 3
[Edit: 2011-03-06 19:36]
- Inserted a warning text in part 1 about avoiding problems when creating a new /system/bin/sh.
Applying Host name
If you want to set a custom hostname for your phone, you can create an init script (If your ROM supports it).
Code:
#!/system/bin/sh
# Get the hostname if it exists, or set it to localhost
if [ -f /system/etc/hostname ]; then
_HOSTNAME=$(cat /system/etc/hostname)
if [ -z $_HOSTNAME ]; then
_HOSTNAME=localhost
fi
else
_HOSTNAME=localhost
fi
# Update the Hostname
echo "$_HOSTNAME" > /proc/sys/kernel/hostname
# Update the hosts file
echo "127.0.0.1 localhost.localdomain localhost" > /etc/hosts
# If the hostname is not localhost, we add one more line to the hosts file
if [ "$_HOSTNAME" != "localhost" ]; then
echo "127.0.0.1 $_HOSTNAME" >> /etc/hosts
fi
This will look in /system/etc/ for the regular linux hostname file. If it exists, it will take the hostname from that file and set as the global hostname.
If it does not exist, "localhost" is used instead. Also it creates the /system/etc/hosts file.
Add or Edit your hostname like this.
Code:
echo "myhostname" > /system/etc/hostname
Note: Remember to delete the hosts and hostname files in your debian /etc/ dir when using merged environment. Otherwise they will overwrite the once in /system/etc/ as debian has first priority.
Manipulating the Android Permissions
Androids permissions works by providing a User ID for each application. This user (Application) can then get a range of different permissions by being added to groups with the permissions required by the app. For an example an app that needs to write to the SD Card, should be a member of Androids sdcard_rw group which translates into 1015 in a Linux group id. Using the numeric values of the Android groups, we can assign these permissions to shell users, daemons etc.
An example:
I had problems with my Http Server because I wanted my www folder on my sdcard. But, the www-data user and group could not be allowed to write to the SD Card, which I needed it to do, because Android controls what permissions the SD Card should have, not even root can change that.
What I did was creating a system group that I named sdcard_rw (Same name as the android, just to better keep track of the groups) with the same Id as the Android sdcard_rw (1015). I then assigned that group to my www-data user (gpasswd -a www-data sdcard_rw) and now my lighttpd had Read/Write access to my sdcard.
You can use the same example for creating a regular SSH User by using the Android shell group or for any other type of user that needs some Android permissions to work properly.
[Edit: 2011-03-08 21:13]
- Added an permission example
Great...
I was just looking into this myself for the last few days.
strange how there isn't more of this being talked about.
at least I have been unable to find anything.
Omg this sounds awesome,
Will try this when u get home and will report back
Sent from my Nexus One using XDA App
I'm still trying to work out some problems with User Rights on SD Card. Can't get lighttpd to be allowed to create files in my SD Card www dir (Does not matter what user and group I set it to use). I can't even get the root account to be allowed to change permissions on SD Card files and folders. It's not the biggest problem, but will report back when I figure it out.
Nice! I've been constantly disappointed for the past year at how un-Linuxy Android is under the hood and unimpressed with the old chroot tricks. This looks much better and I'll try it soon.
I have added a new /system/bin/sh script that can be found in Part 1. This script will automatically look for bash and, if that exists, try to find bash.bashrc, .profile or .bashrc in /etc or the users home dir. If bash does not exist, it looks for sh instead. If you have both one of the Android specific compiled bash, then Debians bash is used when available as this is made to work in a real linux environment. Android bash does not seam to work that way. Also debians packages gets updates via APT.
Also I changed the way groups and users is created in Part 3. Manually inserting them to /etc/passwd, /etc/shadow and /etc/group did not seam to be enough. The users and groups where only half registered. For an example the command "login" gave me a lot of errors, and i had problems with user rights. By using adduser and addgroup in a chroot before the first merge, the user management seams to work as it should now.
I was playing with Debian on my phone just a few days ago too. I've found that same G1 article you mentioned, and installed Lenny in a chroot environment. Background daemons work fine for me, I've installed OpenSSH and I'm using ConnectBot connecting to 127.0.0.1 to get a terminal, so I don't have to chroot every time. I even have X with tightvnc server + android-vnc-viewer. Running the desktop Firefox on my phone is fun Although not very useful. I'm still thinking about what awesome things we could do with this that we couldn't otherwise.
Anyway, great howto.
Do you have any details on the Thalamus kernel? What phone is it for? Overclocked, audio hack, etc? Or is it just a vanilla kernel with the extra module?
What applications can we dream up?
The first thing that springs to mind is could this be used to create a low power server running bittorrent and the like?
Would a cheap device like the Vodafone 845 be under-powered for this?
Can you connect to a usb hub to allow multiple devices to be connected to it? I'm thinking usb network adapters, external storage, etc.
How much additional effort on top of this guide would be needed?
Just curious...
will not... - -!
Well I learned one thing. Don't make Shell scripts when you are tired. I started a shell loop on phone bootup with my /system/bin/sh fix (Which had been changed). When using the merged environment, you need (for now) to enter /bin/login when you open a new shell and login as user with the password you created in Part 3. This will make Debian take over the user management which among other things will make sure that the right variables etc. exists. I have tried to locate where android controls new shell sessions, but I cant seam to find it.
When you make Debian take over, you can use your Linux shell as any other. .bashrc, .profile, /etc/bash.bashrc, /etc/profile files, change default shell (bash, sh, ash) in /etc/passwd and so on.
#9 Well the kernel is the same that is found in the Oxygen 2.0.1, so it's an gingerbread rom (2.3.3). If it works only in Oxygen, I don't know but I don't think so.
#10 He he, Could not say. For now I'm just trying to make it work as I want it to work. After that I might start playing around with different things.
The "login" command is no longer needed. I finally found a way to successfully control user variables and management, making Android and Debian work together as one when merging the two.
This is amazing. Totally going to do this later tonight when I get home.
If you merge using AUFS instead of making a chroot, you should perhaps use the system user (Id: 1000, Group: 1000) as the regular user.
There are only two static users in Android, 0 and 1000.
Android just isn't made to work as an multi-user system. This is how the Java UI is made, and we really need to Linux part to be compatible with that part of the OS. Instead of multi-users, android uses it's apps as users. Each app works as one user with different id. So the 10048 I used as regular user is just the ID of mine "Better Terminal". Switching to another terminal app changed the EUID.
Use the root and only root to work with in the shell. And then delete Debians "su" command and stick with the one from android. Debians need all users that execute "su" to be written to the /etc/passwd.
All GNU tool and library, that's Great. thanks a lot.
I have added an example of how to apply Android rules to shell users, daemons etc. Some daemons will need this in order to function.
I have added this in Thread 2 below the hostname example.
I also changed the main Thread. Found a fix for the rest of the permission problems, including the Android "su" command issue.
I have tested this Android/Debian merge using several methods (ADB, SSH, different local terminals) and it worked on most. I had one problem with one local terminal app that did not execute /etc/profile and some Debian users in /etc/passwd that didn't either.
To fix this I changed /system/bin/sh from using "/system/xbin/sh --login [email protected]" to instead use "/system/xbin/sh --rcfile /system/etc/profile [email protected]". I then moved the Debians /bin/bash to /usr/bin/bash and made a similar script file in /bin/bash > "/usr/bin/bash --rcfile /etc/profile [email protected]". You can make the same for /bin/sh if you don't like using bash for your /etc/passwd users.
Remember to change the bash bind mount in the mkdebian script from /bin/bash to /usr/bin/bash
Use whatever method works on your phone.
This looks like a ton of fun, and could be very useful. I'd like to try this on my NookColor (N2Acard, CM7.1), probably using The Guardian Project's lilDebi image as a base. They go the chroot route with their installer, and I'm going to have to do some hacking on their scripts to make them work with the NookColor. I figure if I'm going to have to hack on it, I might as well put my efforts toward a more ideal solution.
Question: It doesn't look like the CM7.1 kernel supports aufs. That is,
Code:
cat /proc/filesystems
doesn't return aufs as one of the supported filesystems.
Aside from a lack of elegance, is there any reason we can't apply your /sbin workaround to "merging" the whole filesystem? [I recognize there's a certain amount of ignorance behind the question; I figure there's only one way to cure that: Ask.]
OR, how difficult might it be to take a complete CM7.1 kernel and add aufs support to it?
'preciate any help I can get!
mailman1175 said:
This looks like a ton of fun, and could be very useful. I'd like to try this on my NookColor (N2Acard, CM7.1), probably using The Guardian Project's lilDebi image as a base. They go the chroot route with their installer, and I'm going to have to do some hacking on their scripts to make them work with the NookColor. I figure if I'm going to have to hack on it, I might as well put my efforts toward a more ideal solution.
Question: It doesn't look like the CM7.1 kernel supports aufs. That is,
Code:
cat /proc/filesystems
doesn't return aufs as one of the supported filesystems.
Aside from a lack of elegance, is there any reason we can't apply your /sbin workaround to "merging" the whole filesystem? [I recognize there's a certain amount of ignorance behind the question; I figure there's only one way to cure that: Ask.]
OR, how difficult might it be to take a complete CM7.1 kernel and add aufs support to it?
'preciate any help I can get!
Click to expand...
Click to collapse
here's where you get aufs patches.
you'll have to build the kernel yourself to make it work.
to the OP:
using the aufs mount method instead of a chroot...
I have a small problem, I don't know if it shows up with your .img filesystem method, but with an sd-ext partition used intead of the .img file, apt-get fouls massively when trying to update/upgrade libc6.
it complains about the libc6*.so existing in two places at once, and requests the existing one be removed. If it's removed, then nothing in the Linux install works, because just about everything is compiled against libc. I can use busybox to manipulate files with the libc6 removed or hidden away, but can't seem to get past the error since dpkg and apt don't work without some version of libc installed.
Any hints/suggestions? I like your scripts a lot better than the chroot ones I cobbled together from everyone else's work last year for similar purposes before I got aufs working in a kernel.

[DISCUSSION]All about boot.img's and kernels!!!

So, I've seen many a people talk about boot.imgs, and kernels, and mostly spamming dev threads. So, why not create a new thread for it?
Here, ask your questions related to kernels and boot.imgs, and feel free to post any *improvements* that you've made to an existing boot.img/kernel, and so...
Also, feel free to join the discussion, feature requests, whether or not possible, etc, etc.
This thread might be one whole lot of junk, but still if it helps from keeping people from spamming dev threads, why not
So, start!
Lets start from the basics...
Compiling a kernel is easy, IMO :fingers-crossed:
But, the unpacking repacking stuff is a bit difficult...
Anyways, here's the basics:
Packaging kernel for flashing on target device (by sakindia123): http://forum.xda-developers.com/showpost.php?p=31656992&postcount=3
Packaging kernel for flashing on target device (by #define): http://forum.xda-developers.com/showthread.php?t=2114594 (start from step 6)
New CM11 Ramdisk changes...
Well, this was taking up a lot of space on the main thread... Read it here :fingers-crossed:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
edit: some more thoughts...
If you're going to boot this ROM from your SD Card, then, better beware of the new changes in the mounting process... A new fstab.pico is handling the mounting processes..
And, its moved from init.rc to init.pico.rc :|
So, If you're gonna do the editing stuff, you'd find that init.pico.rc has:
Code:
on fs
# mount mtd partitions
# Mount /system rw first to give the filesystem a chance to save a checkpoint
mount yaffs2 [email protected] /system
mount yaffs2 [email protected] /system ro remount
mount yaffs2 [email protected] /data nosuid nodev
mount yaffs2 [email protected] /cache nosuid nodev
# mount partitions
mount_all /fstab.pico
So, if you're gonna just edit those lines, and just change the [email protected], or the [email protected] lines, then, you'd likely be pretty much booting the ROM again from your internal nand. the main reason for this being the line below: mount_all /fstab.pico, which is the fstab.pico I was talking about...
Now, if you'd open up fstab.pico, its pretty much like the /etc/fstab that you'd find on any linux system. the typical device, mountpoint, type and all... If you'd open it, you'll find this:
Code:
#<src> <mnt_point> <type> <mnt_flags and options> <fs_mgr_flags>
/dev/block/mtdblock3 /system yaffs2 ro,barrier=1 wait
/dev/block/mtdblock4 /cache yaffs2 nosuid,nodev,barrier=1 wait,check
/dev/block/mtdblock5 /data yaffs2 nosuid,nodev,noauto_da_alloc,barrier=1 wait,check
/devices/platform/msm_sdcc.1/mmc_host/mmc0 auto vfat defaults voldmanaged=sdcard0:auto
Well, those were the lines from the CM11 preview, not exactly the same, the last line was derped by me in my vain attempts to get the sdcard running... Oh, and the mounting of the SD Card's completely changed too :silly:
BTW, getting back to mounting the ROM from the sdcard, you'd need to edit this fstab.pico file, to something similar to this:
Code:
#<src> <mnt_point> <type> <mnt_flags and options> <fs_mgr_flags>
/dev/block/mmcblk0p2 /system ext4 ro,barrier=1 wait
/dev/block/mmcblk0p4 /cache ext4 nosuid,nodev,barrier=1 wait,check
/dev/block/mmcblk0p3 /data ext4 nosuid,nodev,noauto_da_alloc,barrier=1 wait,check
/devices/platform/msm_sdcc.1/mmc_host/mmc0 auto vfat defaults voldmanaged=sdcard0:auto
Well this is the first post, asking for help... Hope you get the hint *wink *wink
In case you can't get something done, ask for help here!
I tried porting the touchscreen gestures from Siyah Kernel with reference from this commit: https://github.com/gokhanmoral/siyahkernel3/commit/9f57d9efc7458c1a9f540cd04bc5cb14e08fb342
Well, that more or less turned out to be like this: https://github.com/vineethraj49/android_kernel_htc_pico/tree/gestures (check the last few commits...)
And, turns out it works :laugh: with a small bug
1. only single finger gestures work, i think....
2. no way to get the infinite while loop started at init.d
Here's how I did it... Reference thread: http://forum.xda-developers.com/showthread.php?t=1831254
the init.d script is as belows, doesn't work...
Code:
#!/system/bin/sh
echo "
# Gesture 1 - swipe 1 finger near the top
1:1:(0|150,0|150)
1:1:(210|320,0|150)
" > /sys/devices/virtual/misc/touch_gestures/gesture_patterns
while [ 1 ]
do
GESTURE=`cat /sys/devices/virtual/misc/touch_gestures/wait_for_gesture`
if [ "$GESTURE" == "1" ]; then
screencap > /sdcard/`date +%H%M%S.png`
fi;
done
So, inputted those commands using adb, in adb shell, and it works... Got a hell lot of screenshots... but, any way to fix the bugs I mentioned?
Hey, one question
I'd like to know about one thing, I've seen people talking about memory increasing kernels and kernels that mount certain partitions(like cache, data etc. etc. ) . i want to know how this works? I mean, what all things are to be done in both things for them to work?
I'll be glad if you tell this to me in a noob-friendly way. :victory:
Thank You! :fingers-crossed:
#Superuser said:
Hey, one question
I'd like to know about one thing, I've seen people talking about memory increasing kernels and kernels that mount certain partitions(like cache, data etc. etc. ) . i want to know how this works? I mean, what all things are to be done in both things for them to work?
I'll be glad if you tell this to me in a noob-friendly way. :victory:
Thank You! :fingers-crossed:
Click to expand...
Click to collapse
Note: memory increasing kernels and, kernels that mount certain partitions(like cache, data etc. etc. ) both do the exact same thing.
Clarifications:
1. There's no such thing as a memory increasing kernel, and there can't be. No!? Why not? Because, however much memory's present, so, remains. This particular *myth* comes from the slang "memory increasing scripts", i.e. scripts that mount an external SD Card's partition as the internal /data partition.
2. The kernel doesn't increase the memory (check clarification 1). The increasing in memory is done by reverse-mounting (yes, reverse-mounting partitions, is a fairly popular slang for this practice) partitions from the SDCard as the internal partitions, and thus, the phone thinks that it has a more storage, than its own internal storage.
So, how does the memory increasing get done?
Ramdisks! These are the files that go on form the root file system. The kernel itself is packed with the ramdisk. A typical android bootup sequence is this: (thanks to the writers of this article here: http://elinux.org/Android_Booting)
1. The first program which runs on any Android system is the bootloader. Technically, the bootloader is outside the realm of Android itself, and is used to do very low-level system initialization, before loading the Linux kernel. The kernel then does the bulk of hardware, driver and file system initialization, before starting up the user-space programs and applications that make up Android.
2. 'init'
A key component of the Android bootup sequence is the program 'init', which is a specialized program for initializing elements of the Android system. Unlike other Linux systems (embedded or otherwise), Android uses its own initialization program. (Linux desktop systems have historically used some combination of /etc/inittab and sysV init levels - e.g. /etc/rc.d/init.d with symlinks in /etc/rc.d/rc.). Some embedded Linux systems use simplified forms of these -- such as the init program included in busybox, which processes a limited form of /etc/inittab, or a direct invocation of a shell script or small program to do fixed initialization steps.
The Android 'init' program processes two files, executing the commands it finds in them, called 'init.rc' and 'init.<machine_name>.rc', where <machine_name> is the name of the hardware that Android is running on. (Usually, this is a code word. The name of the HTC1 hardware for the ADP1 is 'trout', and the name of the emulator is 'goldfish'.
The 'init.rc' file is intended to provide the generic initialization instructions, while the 'init.<machine_name>.rc' file is intended to provide the machine-specific initialization instructions.
====================================================================================================
Now, that's the general booting process. Now, lets look into our phone's booting process. The bootloader is "HBOOT" specialised for our phones, made by HTC.
This boots up the hardware, loads the kernel, and, the "init" process starts running. If you unpack any kernel you'd find these two files:
Code:
init.rc
init.pico.rc
As you'd have guessed by now, the init.rc is general instructions, and init.pico.rc is the hardware device specific parts.
The init process is what will set up all native services and this is similar to a regular Linux system boot.
So, the init process is also the reason why the filesystems are mounted. If you'd open up any init.rc file, you'd find these lines:
Code:
on fs
# mount mtd partitions
# Mount /system rw first to give the filesystem a chance to save a checkpoint
mount yaffs2 [email protected] /system
mount yaffs2 [email protected] /system ro remount
mount yaffs2 [email protected] /data nosuid nodev
mount yaffs2 [email protected] /cache nosuid nodev
This language is commonly known as the "Android Init Language", and you can look up for help here: https://android.googlesource.com/platform/system/core/+/master/init/readme.txt
So, this is where the filesystems get mounted.
Now, to make a reverse-mounting boot.img, we'd need to modify these lines. Lets mount the second partition in the SD Card in the data partition. So, we'd simply replace this line:
Code:
mount yaffs2 [email protected] /data nosuid nodev
with a modified line, like this:
Code:
mount ext4 /dev/block/mmcblk0p2 /data nosuid nodev
If you'd see, this follows this configuration:
mount <type> <device> <dir> [ <mountoption> ]*
Now, the nosuid, nodev, etc are mountoptions. There are a variety of mount options, and the right choice of them, is likely to make your mounted partition accessible faster. Example: You can disable journaling, or use a different method of journaling, so that, you get access to the partitions faster, and as a direct result, your phone *might* become faster. The general mount options can be found here: http://linux.die.net/man/8/mount
So, that's about how you do the "reverse-mounting" thing.
p.s. here's the mount config that I usually use. Also posting the reasons
p.p.s. I use an ext4 partition
Code:
mount ext4 /dev/block/mmcblk0p2 /data nosuid nodev noatime nodiratime nouser norelatime nostrictatime noiversion nobarrier noauto_da_alloc nouser_xattr data=writeback commit=30 inode_readahead_blks=64 errors=continue
Code:
# [*]nosuid - Do not allow set-user-identifier or set-group-identifier bits to take effect.
# [*]nodev - Do not interpret character or block special devices on the file system.
# [*]noatime - Do not update inode access times on this filesystem.
# [*]nodiratime - Do not update directory inode access times on this filesystem.
# [*]nouser - Forbid an ordinary (i.e., non-root) user to mount the filesystem. This is the default.
# [*]norelatime - Do not use relatime feature. See also the strictatime mount option.
# [*]nostrictatime - Use the kernel's default behaviour for inode access time updates.
# [*]noiversion - Do not increment the i_version inode field.
# [**]nobarrier - This enables/disables barriers. nobarrier disables it, barrier enables it. Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance penalty.
# data=writeback - Data ordering is not preserved - data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest-throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.
# commit=30 - Sync all data and metadata every 30 seconds. The default value is 5 seconds. Zero means default. (Setting it to very large values will improve performance.)
# noauto_da_alloc - http://forum.xda-developers.com/showthread.php?t=876069
# nouser_xattr - Support "user." extended attributes (or not).
# errors=continue - errors={continue|remount-ro|panic} Define the behaviour when an error is encountered. (Either ignore errors and just mark the filesystem erroneous and continue,
# inode_readahead_blks=64 -set to 64 from default 32
thewisenerd said:
Note: memory increasing kernels and, kernels that mount certain partitions(like cache, data etc. etc. ) both do the exact same thing.
Clarifications:
1. There's no such thing as a memory increasing kernel, and there can't be. No!? Why not? Because, however much memory's present, so, remains. This particular *myth* comes from the slang "memory increasing scripts", i.e. scripts that mount an external SD Card's partition as the internal /data partition.
2. The kernel doesn't increase the memory (check clarification 1). The increasing in memory is done by reverse-mounting (yes, reverse-mounting partitions, is a fairly popular slang for this practice) partitions from the SDCard as the internal partitions, and thus, the phone thinks that it has a more storage, than its own internal storage.
So, how does the memory increasing get done?
Ramdisks! These are the files that go on form the root file system. The kernel itself is packed with the ramdisk. A typical android bootup sequence is this: (thanks to the writers of this article here: http://elinux.org/Android_Booting)
1. The first program which runs on any Android system is the bootloader. Technically, the bootloader is outside the realm of Android itself, and is used to do very low-level system initialization, before loading the Linux kernel. The kernel then does the bulk of hardware, driver and file system initialization, before starting up the user-space programs and applications that make up Android.
2. 'init'
A key component of the Android bootup sequence is the program 'init', which is a specialized program for initializing elements of the Android system. Unlike other Linux systems (embedded or otherwise), Android uses its own initialization program. (Linux desktop systems have historically used some combination of /etc/inittab and sysV init levels - e.g. /etc/rc.d/init.d with symlinks in /etc/rc.d/rc.). Some embedded Linux systems use simplified forms of these -- such as the init program included in busybox, which processes a limited form of /etc/inittab, or a direct invocation of a shell script or small program to do fixed initialization steps.
The Android 'init' program processes two files, executing the commands it finds in them, called 'init.rc' and 'init.<machine_name>.rc', where <machine_name> is the name of the hardware that Android is running on. (Usually, this is a code word. The name of the HTC1 hardware for the ADP1 is 'trout', and the name of the emulator is 'goldfish'.
The 'init.rc' file is intended to provide the generic initialization instructions, while the 'init.<machine_name>.rc' file is intended to provide the machine-specific initialization instructions.
====================================================================================================
Now, that's the general booting process. Now, lets look into our phone's booting process. The bootloader is "HBOOT" specialised for our phones, made by HTC.
This boots up the hardware, loads the kernel, and, the "init" process starts running. If you unpack any kernel you'd find these two files:
Code:
init.rc
init.pico.rc
As you'd have guessed by now, the init.rc is general instructions, and init.pico.rc is the hardware device specific parts.
The init process is what will set up all native services and this is similar to a regular Linux system boot.
So, the init process is also the reason why the filesystems are mounted. If you'd open up any init.rc file, you'd find these lines:
Code:
on fs
# mount mtd partitions
# Mount /system rw first to give the filesystem a chance to save a checkpoint
mount yaffs2 [email protected] /system
mount yaffs2 [email protected] /system ro remount
mount yaffs2 [email protected] /data nosuid nodev
mount yaffs2 [email protected] /cache nosuid nodev
This language is commonly known as the "Android Init Language", and you can look up for help here: https://android.googlesource.com/platform/system/core/+/master/init/readme.txt
So, this is where the filesystems get mounted.
Now, to make a reverse-mounting boot.img, we'd need to modify these lines. Lets mount the second partition in the SD Card in the data partition. So, we'd simply replace this line:
Code:
mount yaffs2 [email protected] /data nosuid nodev
with a modified line, like this:
Code:
mount ext4 /dev/block/mmcblk0p2 /data nosuid nodev
If you'd see, this follows this configuration:
mount <type> <device> <dir> [ <mountoption> ]*
Now, the nosuid, nodev, etc are mountoptions. There are a variety of mount options, and the right choice of them, is likely to make your mounted partition accessible faster. Example: You can disable journaling, or use a different method of journaling, so that, you get access to the partitions faster, and as a direct result, your phone *might* become faster. The general mount options can be found here: http://linux.die.net/man/8/mount
So, that's about how you do the "reverse-mounting" thing.
p.s. here's the mount config that I usually use. Also posting the reasons
p.p.s. I use an ext4 partition
Code:
mount ext4 /dev/block/mmcblk0p2 /data nosuid nodev noatime nodiratime nouser norelatime nostrictatime noiversion nobarrier noauto_da_alloc nouser_xattr data=writeback commit=30 inode_readahead_blks=64 errors=continue
Code:
# [*]nosuid - Do not allow set-user-identifier or set-group-identifier bits to take effect.
# [*]nodev - Do not interpret character or block special devices on the file system.
# [*]noatime - Do not update inode access times on this filesystem.
# [*]nodiratime - Do not update directory inode access times on this filesystem.
# [*]nouser - Forbid an ordinary (i.e., non-root) user to mount the filesystem. This is the default.
# [*]norelatime - Do not use relatime feature. See also the strictatime mount option.
# [*]nostrictatime - Use the kernel's default behaviour for inode access time updates.
# [*]noiversion - Do not increment the i_version inode field.
# [**]nobarrier - This enables/disables barriers. nobarrier disables it, barrier enables it. Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance penalty.
# data=writeback - Data ordering is not preserved - data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest-throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.
# commit=30 - Sync all data and metadata every 30 seconds. The default value is 5 seconds. Zero means default. (Setting it to very large values will improve performance.)
# noauto_da_alloc - http://forum.xda-developers.com/showthread.php?t=876069
# nouser_xattr - Support "user." extended attributes (or not).
# errors=continue - errors={continue|remount-ro|panic} Define the behaviour when an error is encountered. (Either ignore errors and just mark the filesystem erroneous and continue,
# inode_readahead_blks=64 -set to 64 from default 32
Click to expand...
Click to collapse
Three words - YOU ARE GREAT! I mean, in the whole explanation you did, I just knew that there is some reverse mounting in init.rc in which few partitions are mounted as data/cache and you cleared my concepts. One more thing related to this I'd like to ask, in sense based roms(For our device) is it possible to do changes in init.rc(which I guess would be required) so that apps are directly installed in sd-ext and the symlink also gets done.
Once again,
THANKS
thewisenerd said:
Note: memory increasing kernels and, kernels that mount certain partitions(like cache, data etc. etc. ) both do the exact same thing.
Clarifications:
Click to expand...
Click to collapse
Lol thats a huge explanation! :good: :highfive:
#Superuser said:
Three words - YOU ARE GREAT! I mean, in the whole explanation you did, I just knew that there is some reverse mounting in init.rc in which few partitions are mounted as data/cache and you cleared my concepts. One more thing related to this I'd like to ask, in sense based roms(For our device) is it possible to do changes in init.rc(which I guess would be required) so that apps are directly installed in sd-ext and the symlink also gets done.
Once again,
THANKS
Click to expand...
Click to collapse
Yes, it is possible. Open your init.rc and find the following line:
Code:
mkdir /data/app 0771 system system
mkdir /data/app-private 0771 system system
mkdir /data/app-asec 0700 root root
mkdir /data/app-lib 0771 system system
Now change it like this:
Code:
# For installing apps directly to /sd-ext:
mkdir /sd-ext/app 0771 system system
mkdir /sd-ext/app-private 0771 system system
mkdir /sd-ext/app-asec 0700 root root
mkdir /sd-ext/app-lib 0771 system system
# Now the symlinking part:
symlink /sd-ext/app /data/app
symlink /sd-ext/app-private /data/app-private
symlink /sd-ext/app-asec /data/app-asec
symlink /sd-ext/app-lib /data/app-lib
NOTE: Before doing this make sure that you have mounted /sd-ext partition.
@thewisenerd....excellent post.....cleared so many doubts....You are really great.
I have a question, I am no dev but just trying to learn some basics about android. As u said that ramdisk does the job of mounting sd partitions using the reverse mounting thing. What is the job of the scripts like int2ext or ungaze or mount2sd.
Is it like when the reverse mounting is not done in ramdisk we need to use these scripts. If yes, then how do these scripts talk 2 ramdisk or kernel to tell them to mount these sd partitions.
I know this may sound noob to you. But m just trying to learn some basics.
@cute_prince Thanks. Now, I'm gonna post all my doubt related to kernels and ramdisks! Thanks to @thewisenerd as well!
cuteitsme said:
@thewisenerd....excellent post.....cleared so many doubts....You are really great.
I have a question, I am no dev but just trying to learn some basics about android. As u said that ramdisk does the job of mounting sd partitions using the reverse mounting thing. What is the job of the scripts like int2ext or ungaze or mount2sd.
Is it like when the reverse mounting is not done in ramdisk we need to use these scripts. If yes, then how do these scripts talk 2 ramdisk or kernel to tell them to mount these sd partitions.
I know this may sound noob to you. But m just trying to learn some basics.
Click to expand...
Click to collapse
Everyone's a newbie (unless they remain to stay a n00b). Anyways, let's get back to on-topic.
So... How does these files in
Code:
/system/init.d/<insert-script-name>
cause the reverse-mount?
For this, we look back, into the init.rc and init.pico.rc files
If you'd read this, https://android.googlesource.com/platform/system/core/+/master/init/readme.txt, you'd find that the Android Init Language gives you an "exec" command for running apps/scripts. So, you should have gotten the hint by now
So, open up init.rc and search for any "exec" command. I can just make it easier, but I want you to find any "exec" call that runs processes from /system/*.
OFC, I'd be telling the answer below, but I want you to find it too
So, here's the key. You're likely to find lines as the same as, or similar to below lines:
Code:
# Run sysinit
exec /system/bin/sysinit
in any one of the *.rc files (mostly init.rc or init.pico.rc).
So, lets take a look at this file.
Below is the above file (/system/bin/sysinit) from CM10.2, weekly 9
Code:
#!/system/bin/sh
export PATH=/sbin:/system/sbin:/system/bin:/system/xbin
/system/bin/logwrapper /system/xbin/run-parts /system/etc/init.d
So, what's this, exactly?
Linux users would be familiar with the PATH variable name
And, logwrapper? Here's standard help:
Code:
Usage: logwrapper [-d] BINARY [ARGS ...]
Forks and executes BINARY ARGS, redirecting stdout and stderr to
the Android logging system. Tag is set to BINARY, priority is
always LOG_INFO.
-d: Causes logwrapper to SIGSEGV when BINARY terminates
fault address is set to the status of wait()
So, its going to execute a binary file, but which?
The next "argument" reads "/system/xbin/run-parts"
Again, here's standard help:
Code:
BusyBox v1.20.2-cm9 bionic (2012-11-18 13:31 +0100) multi-call binary.
Usage: run-parts [-t] [-l] [-a ARG] [-u MASK] DIRECTORY
Run a bunch of scripts in DIRECTORY
-t Print what would be run, but don't actually run anything
-a ARG Pass ARG as argument for every program
-u MASK Set the umask to MASK before running every program
-l Print names of all matching files even if they are not executable
So, run a bunch of scripts in a directory?
That pretty much explains why the next "argument" follows as "/system/etc/init.d ".
So, that's how init.d works :cyclops:
============================================================================================================
Now, moving on to memory increasing scripts (oh! I hate that slang)
Anyways, so, we found out that during the boot, the init.rc file is calling the /system/bin/sysinit file. If you'd notice, a few lines "above",
Here's something from http://www.kpbird.com/2012/11/in-depth-android-boot-sequence-process.html
Below is the sequence of android booting. Note that until all these processes are completed, you still see the boot logo. So, if your phone's struck at green htc screen, then, any one of this processes is.. hung.
on early-init:
Set init and its forked children's oom_adj.
Set the security context for the init process.
on init
setup the global environment
Create cgroup mount point for cpu accounting and many other things...
on fs
mount mtd partitions
on post-fs
change permissions of system directories
on post-fs-data
change permission of /data folders and sub folders
on boot
basic network init ,Memory Management ,etc
service servicemanager
start system manager to manage all native services like location, audio, shared preference etc..
service zygote
start zygote as app_process
So, the "/system/bin/sysinit" runs at the "boot" service. Also, it is run by the bootloader. So, it has full access to the root file system that the ramdisk creates. Now, to be noted: the "boot" service runs after the "fs" service, evidently, because else, you wouldn't be able to access the "/system/bin/sysinit" otherwise.
So, the "sysinit" script runs, running all the files from /system/etc/init.d with the help of busybox
That's about it
But, how does the reverse-mounting take place!?
Let me take the example of the simplest reverse mounting script I've ever found: int2ext. I've seen the mounts2sd script, and think its bloated, IMO, because why have a 1000+ lines script, when a script with >40 lines can do it :angel: ) (no offense). For me, just placing script, setting permissions, rebooting should increase the memory of my device. No roundabout stuff.
So, I'd be explaining how int2ext works, below:
Note: before you proceed, you'd have noticed that the sysinit file set the PATH variable. Its the location for all the accessible binary files. (if you don't understand this, using linux might help you understand this better ) This is similar to the PATH variable in windows too.. This just tells the system where to look for executable binary files/programs.
So, here's a very minimal int2ext script (modified to make this post smaller):
The purple lines ("Royal Blue" according to XDA), are the lines of code. Rest are my comments.
Credits go to original file, got from here: https://github.com/croniccorey/cronmod-scripts/blob/master/int2ext%20scripts/INT2EXT
## Only continue if mmcblk0p2 exists
if [ ! -e /dev/block/mmcblk0p2 ] //If /dev/block/mmcblk0p2 doesn't exist
then //then
exit //this script exists. this code's here for safety.
fi;
## Set SD cache size
SD_CACHE=/sys/devices/virtual/bdi/179:0/read_ahead_kb //read_ahead_kb is the amount of kb that the kernel reads, beforehand.
if [ -e $SD_CACHE ] //checking the existence of the file, just in case...
then
busybox echo "2048" > $SD_CACHE; //2048 is found to be an optimum value for sdcards, class 4 and better.
fi;
## Make /sd-ext directory if needed and unmount /sd-ext if it already mounted
##why? because the writer croniccorey, has done some thinking here
##will explain side by side... If the directory /sd-ext doesn't exist, its created here.
##note: the commands run by the init.rc have full access to the root file system created.
if [ ! -e /sd-ext ] //if doesn't exist sd-ext foler in root file system
then
busybox mount -o remount,rw /; //mount the root file system "/" as rewritable
busybox mkdir /sd-ext; //create a directory named sd-ext
busybox mount -o remount,ro /; //mount the file system as read-only again
else
busybox umount /sd-ext; //else, unmount the sd-ext partition, i.e. partition in the sd-card.
fi;
## Move /data mount point to /sd-ext
INT_DATA=$(busybox mountpoint -n /data | cut -d ' ' -f1) //this gets the mountpoint of the data partition.
// this command "busybox mountpoint -n /data | cut -d ' ' -f1" is actually runnable, and you'd get the mount point
// of the /data partition in the internal memory (which is this: /dev/block/mtdblock5)
busybox umount /data; //unmount the /data partition too/
// the data partition is unmounted because we're going to do some stuff
busybox mount $INT_DATA /sd-ext;
// mount the internal data partition in the sd-ext folder
## Mount mmcblk0p2 to /data
busybox mount -o noatime,nodiratime,nosuid,nodev /dev/block/mmcblk0p2 /data; //well, this is self-explanatory. mount the partition in sdcard into internal memory.
busybox chown 1000:1000 /data; //chown the folder by root (get more help by running "man chown" in a shell)
busybox chmod 771 /data; //chmod the folder (get more help by running "man chmod" in a shell)
## Move existing files
if [ ! -e /data/app ] //why /data/app? simply, it can also be anything else like /data/data...
then
busybox mv /sd-ext/* /data;
// the sd-ext folder has the files of the internal /data partition
// those are moved to the sd-card's partition, which is mounted in /data now.
fi;
## Unmount /sd-ext
//unmount the internal data partition.
// we have the partition from the sd-card mounted in the /data partition currently.
busybox umount /sd-ext;
sync;
//sync changes with file system
Click to expand...
Click to collapse
the comments should explain almost everything...
This script, is almost flawless. Couldn't find any bugs in it. Does what would have been done by changing the mount point in the "on fs" part, where it mounts the file system.
An added advantage is that this can be put into use any time you want... Example, you use your phone, internal memory gets filled up, just put this script in /system/etc/init.d, and set permissions, and reboot! voila! memory increased!
Note: reversing this can't be done by just deleting the script. an appropriate script that moves back user data to internal partitions may be needed.
Also note:
You can always tweak this command:
busybox mount -o noatime,nodiratime,nosuid,nodev /dev/block/mmcblk0p2 /data;
for better performance. Check my config, in post 8 or so... 1'st page..
You should've also guessed why we set permissions to executable, by now... Else, the file wouldn't be able to "execute"
P.S. Adding a few lines to init.rc should make it possible to have no permission change, but, that can wait for another day...
Simply awesome.....what an explanation sirji....
Still trying to understand some part but i will have to do more reading for that first....that i will do.....but must say U rock man.....thanks a lot for this....
thewisenerd said:
Everyone's a newbie (unless they remain to stay a n00b). Anyways, let's get back to on-topic.
So... How does these files in
Code:
/system/init.d/<insert-script-name>
cause the reverse-mount?
For this, we look back, into the init.rc and init.pico.rc files
If you'd read this, https://android.googlesource.com/platform/system/core/+/master/init/readme.txt, you'd find that the Android Init Language gives you an "exec" command for running apps/scripts. So, you should have gotten the hint by now
So, open up init.rc and search for any "exec" command. I can just make it easier, but I want you to find any "exec" call that runs processes from /system/*.
OFC, I'd be telling the answer below, but I want you to find it too
So, here's the key. You're likely to find lines as the same as, or similar to below lines:
Code:
# Run sysinit
exec /system/bin/sysinit
in any one of the *.rc files (mostly init.rc or init.pico.rc).
So, lets take a look at this file.
Below is the above file (/system/bin/sysinit) from CM10.2, weekly 9
Code:
#!/system/bin/sh
export PATH=/sbin:/system/sbin:/system/bin:/system/xbin
/system/bin/logwrapper /system/xbin/run-parts /system/etc/init.d
So, what's this, exactly?
Linux users would be familiar with the PATH variable name
And, logwrapper? Here's standard help:
Code:
Usage: logwrapper [-d] BINARY [ARGS ...]
Forks and executes BINARY ARGS, redirecting stdout and stderr to
the Android logging system. Tag is set to BINARY, priority is
always LOG_INFO.
-d: Causes logwrapper to SIGSEGV when BINARY terminates
fault address is set to the status of wait()
So, its going to execute a binary file, but which?
The next "argument" reads "/system/xbin/run-parts"
Again, here's standard help:
Code:
BusyBox v1.20.2-cm9 bionic (2012-11-18 13:31 +0100) multi-call binary.
Usage: run-parts [-t] [-l] [-a ARG] [-u MASK] DIRECTORY
Run a bunch of scripts in DIRECTORY
-t Print what would be run, but don't actually run anything
-a ARG Pass ARG as argument for every program
-u MASK Set the umask to MASK before running every program
-l Print names of all matching files even if they are not executable
So, run a bunch of scripts in a directory?
That pretty much explains why the next "argument" follows as "/system/etc/init.d ".
So, that's how init.d works :cyclops:
============================================================================================================
Now, moving on to memory increasing scripts (oh! I hate that slang)
Anyways, so, we found out that during the boot, the init.rc file is calling the /system/bin/sysinit file. If you'd notice, a few lines "above",
Here's something from http://www.kpbird.com/2012/11/in-depth-android-boot-sequence-process.html
Below is the sequence of android booting. Note that until all these processes are completed, you still see the boot logo. So, if your phone's struck at green htc screen, then, any one of this processes is.. hung.
on early-init:
Set init and its forked children's oom_adj.
Set the security context for the init process.
on init
setup the global environment
Create cgroup mount point for cpu accounting and many other things...
on fs
mount mtd partitions
on post-fs
change permissions of system directories
on post-fs-data
change permission of /data folders and sub folders
on boot
basic network init ,Memory Management ,etc
service servicemanager
start system manager to manage all native services like location, audio, shared preference etc..
service zygote
start zygote as app_process
So, the "/system/bin/sysinit" runs at the "boot" service. Also, it is run by the bootloader. So, it has full access to the root file system that the ramdisk creates. Now, to be noted: the "boot" service runs after the "fs" service, evidently, because else, you wouldn't be able to access the "/system/bin/sysinit" otherwise.
So, the "sysinit" script runs, running all the files from /system/etc/init.d with the help of busybox
That's about it
But, how does the reverse-mounting take place!?
Let me take the example of the simplest reverse mounting script I've ever found: int2ext. I've seen the mounts2sd script, and think its bloated, IMO, because why have a 1000+ lines script, when a script with >40 lines can do it :angel: ) (no offense). For me, just placing script, setting permissions, rebooting should increase the memory of my device. No roundabout stuff.
So, I'd be explaining how int2ext works, below:
Note: before you proceed, you'd have noticed that the sysinit file set the PATH variable. Its the location for all the accessible binary files. (if you don't understand this, using linux might help you understand this better ) This is similar to the PATH variable in windows too.. This just tells the system where to look for executable binary files/programs.
So, here's a very minimal int2ext script (modified to make this post smaller):
The purple lines ("Royal Blue" according to XDA), are the lines of code. Rest are my comments.
Credits go to original file, got from here: https://github.com/croniccorey/cronmod-scripts/blob/master/int2ext%20scripts/INT2EXT
the comments should explain almost everything...
This script, is almost flawless. Couldn't find any bugs in it. Does what would have been done by changing the mount point in the "on fs" part, where it mounts the file system.
An added advantage is that this can be put into use any time you want... Example, you use your phone, internal memory gets filled up, just put this script in /system/etc/init.d, and set permissions, and reboot! voila! memory increased!
Note: reversing this can't be done by just deleting the script. an appropriate script that moves back user data to internal partitions may be needed.
Also note:
You can always tweak this command:
busybox mount -o noatime,nodiratime,nosuid,nodev /dev/block/mmcblk0p2 /data;
for better performance. Check my config, in post 8 or so... 1'st page..
You should've also guessed why we set permissions to executable, by now... Else, the file wouldn't be able to "execute"
P.S. Adding a few lines to init.rc should make it possible to have no permission change, but, that can wait for another day...
Click to expand...
Click to collapse
I read somewhere, that init.d scripts run alphabetically, maybe this is the reason int2ext is named as 40int2ext so that it starts first?
#Superuser said:
I read somewhere, that init.d scripts run alphabetically, maybe this is the reason int2ext is named as 40int2ext so that it starts first?
Click to expand...
Click to collapse
Yeah....I guess that's the linux thing. I think the higher the no. the late the script will be called. This is basically to run the more important scripts before the other scripts.
hi i unpacked the boot.img files of cm11 beta 1 and 2 in order to understand this better and understood many things which u have explained.
I did found those lines starting with exec in some init. files.
Thanks a lot for this. Since u must be aware that the sd-ext is not mounting in buid 2 and thus int2ext is not working. So, out of curiosity I was trying to understand the difference between these two files from different builds(beta 1 and 2) but not able to find any.
Can u give any hint as to why even after having the same lines in fstab.pico of build 1 and build 2 int2ext was working in buld 1 and not in build 2. Of course by doing reverse mounting we can overcome this issue. But I am just trying to understand the difference between the two builds at ramdisk level.
cuteitsme said:
hi i unpacked the boot.img files of cm11 beta 1 and 2 in order to understand this better and understood many things which u have explained.
I did found those lines starting with exec in some init. files.
Thanks a lot for this. Since u must be aware that the sd-ext is not mounting in buid 2 and thus int2ext is not working. So, out of curiosity I was trying to understand the difference between these two files from different builds(beta 1 and 2) but not able to find any.
Can u give any hint as to why even after having the same lines in fstab.pico of build 1 and build 2 int2ext was working in buld 1 and not in build 2. Of course by doing reverse mounting we can overcome this issue. But I am just trying to understand the difference between the two builds at ramdisk level.
Click to expand...
Click to collapse
Hmm... Could you post the two boot.img's here? I'd like to have a look
thewisenerd said:
Hmm... Could you post the two boot.img's here? I'd like to have a look
Click to expand...
Click to collapse
Sure y not.....here u go
@thewisenerd: does the ION Kernel used in cyanogenmod 11 support swap
PiCo HackR said:
@thewisenerd: does the ION Kernel used in cyanogenmod 11 support swap
Click to expand...
Click to collapse
The question is: Why do you need swap?

[MOD] Encrypt sdcard with EncFS

Introduction
While Android allows to encrypt the /data partition this is not the case for the sdcard with many ROMs. So there are many files on the sdcard that remain accessible for wicked persons who have physical access to it. This script is an attempt to solve this problem by encrypting the content of the sdcard with the help of EncFS.
This thread is for people who have an interest to encrypt their sdcard. If you don't need it of if you want to discuss the necessity of encrypting the sdcard or encryption in general, please stay away and open a new thread if necessary.
Why EncFS?
When I was looking for a proper way to encrypt my sdcard, I wanted the following points to be fulfilled:
well-known, strong encryption method - the encrypted data should be accessible with other devices and systems (e.g. with Windows, MacOS, Linux)
conventional plain data and encrypted data on one sdcard - some file possibly should stay unencrypted (e.g. files for osmand, video files,...)
shared space for plain and encrypted data - especially with small sdcards it is more comfortable to stay with one big partition instead of splitting the sdcard
EncFS fulfills these points. Furthermore, there are working binaries and build instructions for Android thanks to Cryptionite.
Is it stable? Will it slow down my device?
I'm running my phone with encrypted sdcard for about a year now without any serious problems. I tested it with Sultanxda's CM11 KitKat ROM and ivanich's CM12 Lollipop ROM so I expect it to work at least on every CM11 and CM12 ROM. Please let me know if it works / doesn't work for you.
Using encryption will obviously slow down the read/write speed from/to sdcard. However, for me it is fast enough. As encryption and decryption is taking place in the CPU I expect the deceleration to be less on slow sdcards. But this is only an assumption.
With a SanDisk Ultra Class10 and AES 128 Bit encryption I tested the speed with the dd method on Android Lollipop:
Without encryption:
Write: 9 Mb/s
Read: 21,3 Mb/s
With EncFS encryption:
Write: 4,5 Mb/s
Read: 19,4 MB/s
Again, I hope to hear about the speed you measure.
Installation Instructions
Please backup your device before you try this! I don't know if this is working on your device and ROM. Do it at your own risk! I'm not responsible for any damage or data loss!
encrypt your /data partition. Your password for the sdcard-encryption will be stored on /data in plain text, so this is essential
install the attached zip-file via recovery
create a folder named ".encrypted" in the root of your sdcard
create an EncFS repository in the ".encrypted" folder. This is not difficult, search for howtos if you don't know what to do. You can do this on your device via adb or terminal (as root):
Code:
/system/bin/mount.encfs /sdcard/.encrypted /sdcard/decrypted
or with Linux, Windows or MacOS (via MacPorts), for example:
Code:
encfs /path-to-sdcard/.encrypted /only-temporary
save the password you entered in /data/local/passwd on your phone (adb or terminal as root)
Code:
echo "your-encfs-password-here" > /data/local/passwd
reboot
if everything works, you have the decrypted files of the ".encrypted" folder in "/storage/sdcard0" and ordinary plain files on your sdcard in "/storage/sdcard1"
now you can move the data of your apps, photos, ringtones, music,... from "/storage/sdcard1" to "/storage/sdcard0"
After flashing a new ROM, you only have to flash the zip-file again.
How does it work?
The script is executed every time you mount your sdcard (at boot and when inserting the sdcard). To achieve this, the sdcard daemon at /system/bin/sdcard is replaced with a script which performs the mounting stuff. It is looking for the ".encrypted" folder on the sdcard. If the folder exists, it decrypts it with the password stored in "/data/local/passwd". You will have the following sdcard structure dependent of the presence of the ".encrypted" folder:
".encrypted" folder is present (and "/data/local/passwd" containing the right password):
/storage/sdcard0 -> the decrypted data of .encrypted (the EncFS folder)
/storage/sdcard1 -> all the data of your sdcard
".encrypted" folder is not present:
/storage/sdcard0 -> all the data of your sdcard
So if you rename or delete the ".encrypted" folder on your sdcard, the script should react as if it doesn't exist.
There are some apps which can be configured to use another folder as "sdcard0". These can make use of "sdcard1", especially if they must access big files very fast (like Osmand or some podcast-apps).
Where is the EncFS binary from?
The EncFS binary is an ARMv7-version taken from Cryptonite. You can also build it yourself and replace the one in the zip file.
[ Reserved ]
support!!
Info
Can i encrypt/decrypt a single folder in internal sdcard?
ki1120 said:
support!!
Click to expand...
Click to collapse
Yeah.. so... I don't know what to say...
ultrasound1991 said:
Can i encrypt/decrypt a single folder in internal sdcard?
Click to expand...
Click to collapse
This is exactly what cryptonite does. You can mount single encrypted folders via a comfortable GUI.
If you want to mount the folder automatically together with the sdcard my script can also be adjusted. However, umnounting the sdcard (from settings or when connecting USB) could be prevented. This sould be tested. You can ask me if you need help.
Info
Cryptonite is no longer supported.
I have smartphone with Android L, root, and only internal sdcard.
I want to use your script to mount an encrypted folder, that contain encfs6.xml file.
Can you help me? I am not a programmer.
EncFS (FUSE) UID, GID not set, always "0" ("root")
It's old though, I was experimenting on a KitKat. All sets up fine, just the permissions part. Whatever uid and gid is set in command, folder is always mounted as uid=0 and gid=0 i.e. root when checked by 'mount' command. Mount details are as under:
Code:
[email protected]:/ # mount | grep "storage/sdcard"
encfs /storage/sdcard0 fuse.encfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
/dev/fuse /storage/sdcard1 fuse rw,nosuid,nodev,relatime,user_id=1023,group_id=1023,default_permissions,allow_other 0 0
[email protected]:/ #
It's because sdcard daemon is run as root?
But apps need uid and gid 1023 (media_rw) or perhaps 1028 (sdcard_r). I'm not sure about this. As the init.qcom.rc lines say:
Code:
# create virtual SD card at /mnt/sdcard, based on the /data/media directory
# daemon will drop to user/group system/media_rw after initializing
# underlying files in /data/media wil be created with user and group media_rw (1023)
service sdcard /system/bin/sdcard -u 1023 -g 1023 -l /data/media /mnt/shell/emulated
class late_start
service fuse_sdcard1 /system/bin/sdcard -u 1023 -g 1023 /mnt/media_rw/sdcard1 /storage/sdcard1
.
Now, if I run command as user 1023 or 1028, encfs can't read /storage or /mnt directories due to restricted permissions:
Code:
[email protected]:/ # su 1023 /system/bin/mount.encfs -s -o "rw,nosuid,nodev,uid=1023,gid=1023,umask=006,allow_other" /mnt/media_rw/sdcard0/.encrypted /storage/sdcard0
The directory "/storage/sdcard0" does not exist. Should it be created? (y,n) y
Unable to create directory: : Permission denied
06:06:32 (main.cpp:435) Unable to locate mount point, aborting.
How to achieve this drop part?:
Code:
#daemon will drop to user/group system/media_rw after initializing
I think I'm missing something very simple.
And encfs latest static binary for arm can be made available somehow other than compiling? That's a real tough part.
Does EncFS work with Android 6+? If no, any similar alternatives i.e. command-line solution, not apps.
mirfatif said:
I think I'm missing something very simple.
Click to expand...
Click to collapse
Do you use the script I provided? It should create the sdcard as root before running encfs:
Code:
mkdir /storage/sdcard0 0700 root root
After that it runs encfs as root – the "drop part" is actually "uid=0,gid=01028" with which encfs mounts the encrypted folder:
Code:
/system/bin/mount.encfs -s -o "rw,nosuid,nodev,uid=0,gid=01028,umask=6,allow_other" /mnt/media_rw/sdcard0/.encrypted /storage/sdcard0
mirfatif said:
And encfs latest static binary for arm can be made available somehow other than compiling? That's a real tough part.
Click to expand...
Click to collapse
Development of Cryptonite seems quite inactive unfortunately. But there is a fork that seems to have some fixes for building, a build script and build instructions.
mirfatif said:
Does EncFS work with Android 6+? If no, any similar alternatives i.e. command-line solution, not apps.
Click to expand...
Click to collapse
I didn't test it with Android 6 as from there on adoptable storage is available. Adoptable storage fulfils most of the things I wanted for sdcard encryption (see my first post: strong encryption; conventional plain data and encrypted data). Thus, it basically makes my encfs-solution obsolete for Android 6+.
bastei said:
Do you use the script I provided? It should create the sdcard as root before running encfs:
Code:
mkdir /storage/sdcard0 0700 root root
After that it runs encfs as root – the "drop part" is actually "uid=0,gid=01028" with which encfs mounts the encrypted folder:
Code:
/system/bin/mount.encfs -s -o "rw,nosuid,nodev,uid=0,gid=01028,umask=6,allow_other" /mnt/media_rw/sdcard0/.encrypted /storage/sdcard0
Click to expand...
Click to collapse
I used these scripts without any problems on:
Android 4.4:
Code:
#!/system/bin/sh
#####################################################################################################
# DISABLE INTERNAL MEMORY (/data/media/0) AND USE EXTERNAL SD CARD AS INTERNAL STORAGE #
# ENCRYPT FILES ON EXTERNAL SD CARD FOR SECURITY #
# #
# STEPS: #
# 1. DIM i.e disable internal memory (KitKat, Lollipop), not explained here #
# #
# 2. change in boot.img > ramdisk > init.qcom.rc: #
# service fuse_sdcard0 /system/bin/sdcard -u 1023 -g 1023 -d /mnt/media_rw/sdcard0 /storage/sdcard0 #
# to #
# service fuse_sdcard1 /system/bin/sdcard -u 1023 -g 1023 -d /mnt/media_rw/sdcard0 /storage/sdcard1 #
# #
# ensure both lines exist in init.qcom.rc: #
# "mkdir /storage/sdcard1 0700 root root" and "mkdir /storage/sdcard0 0700 root root" #
# #
# 3. rename /system/bin/sdcard to sdcard_orig #
# 4. replace /system/bin/sdcard with this script (named "sdcard") #
# 5. copy encfs bianry to /system/bin #
# 6. set chmod +x permissions on all 3 files #
# #
# 7. encrypt internal storage (from settings) and save encfs password: #
# echo <your_password> | tee /data/local/pswd #
# 8. run this as root to mount encrypted directory: #
# /system/bin/encfs -o allow_other,umask=0 /mnt/media_rw/sdcard0/.encfs /data/local/SDCardTemp #
# copy all data on internal storage (/data/media/0/) to /data/local/SDCardTemp #
# reboot #
#####################################################################################################
export PATH=/system/bin:/system/xbin
# save log
mkdir -p /data/local/tmp
exec &>/data/local/tmp/encfs.log
date
echo "sdcard $*"
# mount encfs if not mounted already
if ! mountpoint /storage/sdcard0
then
cat /data/local/pswd | encfs -S /mnt/media_rw/sdcard0/.encfs /storage/sdcard0 -- -o allow_other,umask=0
# or may use uid=0,gid=1028,umask=006
fi
# pass original init command to sdcard daemon
sdcard_orig "[email protected]"
Android 5.0:
Code:
#!/system/bin/sh
# add following lines to boot.img > ramdisk > init.qcom.rc before sdcard service:
## service encfs /system/bin/encfs.sh
## oneshot
## class late_start
# save this script as /system/bin/encfs.sh
# copy encfs bianry to /system/bin
# chmod a+x /system/bin/encfs*
# echo <password> | tee /data/local/pswd
# /system/bin/encfs -o allow_other,umask=0 /data/media/encfs_crypted /data/local/SDCardTemp
# rsync -a --progress --exclude=encfs_crypted /data/media/ /data/local/SDCardTemp/
export PATH=/system/bin
# wait to avoid "PROTO_EXEC|PROTO_WRITE failed" error
sleep 5
# save log
exec &>/data/local/encfs.log
date
# mount encfs
exec cat /data/local/pswd | encfs -S /data/media_encfs_crypted /data/media -- -o uid=1023,gid=1023,umask=006,allow_other,nonempty &&
echo Mounted.
Mount options "nosuid" and "nodev" are set by default as a security measure. Mostly applications making use of FUSE (in kernel space) and optionally libfuse (in userspace) also set (or give a commandline parameters to set) "allow_other" and "default_permissions" options by default.
UMASK, UID, GID, ALLOW_OTHER
"umask" (or "dmask" and "fmask") needs to be provided explicitly to make files world accessible. Default mode set by "encfs" is 0600 (umask=177) and only mount owner (root) can access files unless every newly created file is manually chmod-ed.
Similarly, "uid" and "gid" options are needed to set desired owner/group. Default owner:group is the mount owner (0:0) unless every newly created file is manually chown-ed.
Actually mount owner and files/directories owner can be different, which was the source of my confusion. That's how FUSE is implemented to let non-privileged mounts work. Have a look at this:
user_id=N
group_id=N Specifies the numeric uid/gid of the mount owner.
This option should not be specified by the filesystem owner. It is set by libfuse (or, if libfuse is not used, must be set by the filesystem itself).​
. . .
uid=N
Override the st_uid field set by the filesystem (N is numeric).​
Click to expand...
Click to collapse
Since all "init" processes (including "sdcard" daemon) run as root (uid=0 and gid=0), 'mount' command always show "user_id=0" and "group_id=0". However it should be noted that provided "uid" and "gid" appear when we view files' ownership:
Code:
[B]/~#[/B] encfs -o allow_other,[COLOR="Red"]umask=7,uid=1023,gid=0[/COLOR] /.enc /enc
EncFS Password:
[B]/~#[/B] mount | grep enc
encfs on /encfs type fuse.encfs (rw,nosuid,nodev,relatime,[COLOR="red"]user_id=0,group_id=0[/COLOR],default_permissions,allow_other)
[B]/~#[/B] touch /enc/test
[B]/~#[/B] ls -l /.enc /enc
/.enc:
[COLOR="red"]-rw-rw-rw-[/COLOR] 1 [COLOR="Red"]0 0[/COLOR] 0 Mar 3 17:58 CoXOLL0fc0RaxZjq2HaSj9en
/enc:
[COLOR="YellowGreen"]-rwxrwx---[/COLOR] 1 [COLOR="YellowGreen"]1023 0[/COLOR] 0 Mar 3 17:58 test
[B]/~#[/B] chmod 0777 /enc/test
[B]/~#[/B] chown 0:1023 /enc/test
[B]/~#[/B] ls -l /.enc /enc
/.enc:
[COLOR="red"]-rwxrwxrwx[/COLOR] 1 [COLOR="red"]0 1023[/COLOR] 0 Mar 3 17:58 CoXOLL0fc0RaxZjq2HaSj9en
/enc:
[COLOR="YellowGreen"]-rwxrwx---[/COLOR] 1 [COLOR="YellowGreen"]1023 0[/COLOR] 0 Mar 3 17:58 test
[B]/~#[/B]
Setting "uid"/"gid" isn't necessarily required if universal access is granted using "umask=0" and "allow_other" mount options. Since every app is treated as a user on Android, so even the file explorer (same user who mounted filesystem by getting root privileges) can't access EncFS mounted directory without "allow_other" (if not running as root). So "allow_other" is necessarily required.
Mounting with "gid=9997,dmask=0007,fmask=0117,allow_other" at "/mnt/runtime/write/emulated/0/<mount_point>" on Android 6+ grants write access to apps with android.permission.WRITE_EXTERNAL_STORAGE granted. See this answer for details.
MOUNT OWNER
On Linux non-root users who are member of "fuse" group can mount and un-mount FUSE filesystems using "fusermount(3)" which is always run as root by set-UID-root or "setcap cap_sys_admin+ep". "fuse" group is the GID of "/dev/fuse". But non-privileged users can't set "allow_other" unless "user_allow_other" is set in FUSE configuration file /etc/fuse.conf. "/etc/fuse.conf" is used by "libfuse", and hence by all program depending on or linked to this library like "bindfs", "encfs", "sshfs", "rclone" and so on. So a non-root user can be the mount owner but other users (including root) can't access the mounted filesystem.
However on Android non-privileged user can't (u)mount FUSE filesystem because:
FUSE support is fully enabled from kernel side (on Linux it's usually a loadable module), can be checked by:
Code:
grep -E 'fuse|fuseblk' /proc/filesystems
But there are limitations in userspace.
There's no "fusermount" binary available on Android and the libfuse implementation doesn't use FUSE configuration because it's meant only for root users. But if we try to use binaries statically linked to standard Linux "libfuse" (like EncFS), they do read "/etc/fuse.conf" and may depend on "fusermount".
Apps on Android are run with all privileges dropped, so they can't make "mount" syscall even if "fusermount" is present with SUID bit set.
"/dev/fuse" is readable and writable only by root (0600, 0:0).
Non-root users on Android may face problem accessing filesystems (e.g. executing EncFS binary and reading "/dev/fuse") due to DAC or MAC permission checks. SELinux might need to be set "permissive" for the same reason, or the policy needs to be patched. SELinux also restricts Linux capabilities of processes.
So in short, EncFS and all other programs that make use of FUSE, can only be mounted by root on Android.
MOUNT NAMESPACE
Next important thing to be considered is "mount namespace" where even "allow_other" and root access won't help you. Mount namespaces enable apps to be exposed to an individual view of mounted filesystems. It means that "/proc/mounts" (which actually is "/proc/self/mounts") may look different to different processes. FUSE was implemented since KitKat to impose permissions derived from directory structure by emulating internal and/or external SD cards instead of direct mounting them. Mount namespaces are implemented since Marshmallow to serve the purpose of apps isolation in better way enforced by "zygote" at runtime. On rooted devices when an app requests a root shell, SU manager starts the shell in a new isolated namespace. While FUSE is implemented through UID's, namespaces are implemented through PID's. Super user rights with namespace separation are granted through
Code:
unshare -m sh
i.e. providing the app (process id) an independent view of mount points, not visible to the rest of apps.
Namespaces only affect (1) emulated SD card mounts handled by zygote or (2) any other filesystem mounted from a shell which was initiated by SuperSU with mount NS separation. All root processes started by Android ("init" on boot) run in global namespace. For example all mounts from adb shell will have no namespace separation because "adbd" is run in global namespace. Similarly, all shared mounts done at boot by init are propagated as a single unit from default/global/root namespace to all new zygote-forked per process namespaces. If we un-mount a filesystem from an app with root access, it will not affect the system and other apps. However if mount namespace separation is disabled, un-mounting a filesystem from any app will un-mount it globally. So isolated namespace feature provided by SU solutions is a security measure.
Say, you execute the following command from Termux (or any other terminal emulator app):
Code:
/system/bin/encfs -o allow_other /data/media/0/.encfs /data/media/0/encfs
Now, if mount namespace separation is enabled (by default), you will be able to open /data/media/0/encfs only from Termux app and from no other file explorer etc. For any other app, EncFS is not mounted at all. However Android su binaries provide a mount master option to always mount in global namespace:
Code:
[COLOR="Red"][B]su -mm[/B][/COLOR] /system/bin/encfs -o allow_other /data/media/0/.encfs /data/media/0/encfs
nsenter command can also be used to enter an existing isolated mount namespace. But note that namespaces are killed as soon as they have zero process running.
Other option is to disable "Mount Namespace Separation" in SuperSU app settings. However, do not disable this if you want to use apps like Titanium Backup or "Hide Root" feature as in Magisk etc. In addition to Isolated and Global, Magisk also provides a third option: Inherited i.e. run the root shell in same isolated namespace in which app is already running.
Related: Partition gets unmounted automatically in Android Oreo
DROP DAEMON PRIVILEGES
What I quoted earlier:
#daemon will drop to user/group system/media_rw after initializing
Click to expand...
Click to collapse
This "drop daemon privilege" thing is specific for "sdcard" daemon only as stated here and neither true for FUSE in general, nor related to filesystem "user_id" or "uid" options:
It must be run as root, but will drop to requested UID/GID as soon as it mounts a filesystem.
Click to expand...
Click to collapse
Code:
snprintf(opts, sizeof(opts),
"fd=%i,rootmode=40000,default_permissions,allow_other,user_id=%d,group_id=%d",
fd, uid, gid);
res = mount("/dev/fuse", dest_path, "fuse", MS_NOSUID | MS_NODEV, opts);
if (res < 0) {
ERROR("cannot mount fuse filesystem: %s\n", strerror(errno));
goto error;
}
/* Drop privs */
res = setgroups(sizeof(kGroups) / sizeof(kGroups[0]), kGroups);
if (res < 0) {
ERROR("cannot setgroups: %s\n", strerror(errno));
goto error;
}
res = setgid(gid);
if (res < 0) {
ERROR("cannot setgid: %s\n", strerror(errno));
goto error;
}
res = setuid(uid);
if (res < 0) {
ERROR("cannot setuid: %s\n", strerror(errno));
goto error;
}
fuse_init(&fuse, fd, source_path, write_gid, derive, split_perms);
umask(0);
res = ignite_fuse(&fuse, num_threads);
/* we do not attempt to umount the file system here because we are no longer
* running as the root user */
Process should call "setuid" itself to drop/elevate privileges which isn't the case for EncFS/FUSE by default. We can't use "su" or "setuidgid" or "chpst" commands to drop privileges later on but this isn't needed either as explained earlier. Only "uid=" and "gid=" options are enough.
Same source code file also explains somewhat how "legacy" and "derived permissions" work particularly on Android 4.4, 5 and 6 using FUSE filesystems for emulating SD card.
SUMMARY
On Android, only root can mount EncFS (FUSE) filesystem
Do mention "allow_other" option if you want to access EncFS mounted directory from a non-root app
Do mention "umask=" option if you want to read/write encrypted files from a non-root app
"uid" and "gid" need to be set if you want to give some specific non-root app write access to the filesystem. Otherwise "allow_other" and "umask" can suffice.
Disable "Mount Namespace Separation" option in SuperSU app if:
EncFS is to be mounted feom CLI (Terminal Emulator) app and not by some boot script or from adb shell, and
You want to access EncFS mounted directory from some app other than the root shell in CLI app
bastei said:
Development of Cryptonite seems quite inactive unfortunately. But there is a fork that seems to have some fixes for building, a build script and build instructions.
Click to expand...
Click to collapse
Latest EncFS binary is available here. Thanks to @DualJoe. It works with Android Oreo as well. However, overlapping with already mounted FUSE or sdcardfs (a FUSE-like filesystem implemented in Android 8+ that works in kernel space) should be avoided.
This is my understanding of EncFS and FUSE. Correct me if I'm wrong.
You could use /mnt/encfs. I think it might not be affected by namespace isolation.
DualJoe said:
You could use /mnt/encfs. I think it might not be affected by namespace isolation.
Click to expand...
Click to collapse
It didn't work for me. What I have understood is that mount namespace isolation implemented by SuperSU app is totally independent of that implemented by AOSP on emulated storage. SuperSU grants root access on the basis of app, not path. When an app requests root rights from SuperSU, a new namespace is created in which all global mounts are cloned. Now it's up to that app what it does with the mountpoints, not affecting any other process. So the path where filesystem is mounted doesn't matter. Also whatever location we choose, mount call won't be successful without root. I have added a few lines about this in my previous post.
Are you sure you're invoking 'mount' with 'su -mm'?
I can only speak for my system (LOS13, Magisk) but i can mount cifs or ecryptfs on any place (/data/media/0, /mnt/*) without problems. Magisk is still set to default namespace option (inherited). The '-mm' switch is always 'global namespace' though as it's primarily intended for public mounting (as described in 'su --help').
DualJoe said:
Are you sure you're invoking 'mount' with 'su -mm'?
I can only speak for my system (LOS13, Magisk) but i can mount cifs or ecryptfs on any place (/data/media/0, /mnt/*) without problems. Magisk is still set to default namespace option (inherited). The '-mm' switch is always 'global namespace' though as it's primarily intended for public mounting (as described in 'su --help').
Click to expand...
Click to collapse
Yes definitely 'su -mm' mounts in global namespace. I was talking about simple mount command from a su shell, without '-mm'.
After putting password in terminal after encfs command, does nothing. Means decrypted directory always remain empty. any idea how to resolve this ? Android 13
Edit1: now this working but new issue came which ia mentioned in next post.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
i using below command:
su -mm -c "encfs --public -o nonempty,allow_other,rw,noatime,noexec,nodev,nosuid,gid=9997,umask=006 -S /mnt/runtime/write/3B86-9400/.encrypted /mnt/runtime/write/emulated/0/decrypted"
I can see content in the file manager only but not gallery.
After Android media store refresh only content of decrypted folder showing in gallery but content of sub folder of decrypted folder is not showing in the gallery.
Any idea how to resolve this issue.

[Tutorial] How to emulate Second Partition on SD-Card without formatting

[TUTORIAL] SCROLL DOWN BELOW TO NEXT POST
Seems i have accidentially written the nearly same Init.d Enabler like term-init.sh by Ryuinferno @ XDA.
the small different, no busybox run-parts required, and install-recovery-2.sh is used (because SuperSU is running from install-recovery.sh)
However, main intent was to increase internal memory with Bulent Akpinar's Link2SD, but without using a 2. partition.
First have installed Androguide.fr's Universal Init.d, but the scripts were executed delayed. a quick view in SuperSU daemon install-recovery.sh proposed me to write my own script, so i decided to uninstall the App. besides this i highly recommend this app! always give it a try first (i just later realized that this is another version of term-init.sh - which i found on XDA after i was finished and decided to share...)
Once the init.d job was running, it turned out the delay wasn't Universal Init.d App's fault. instead the reason is the Android OS is mounting sdcard delayed. this means the provisional watchdog mount script 00_mount_sdext2.sh becames unusable for me. anyway i keep it attached for using with 2. partition. the final Link2sd mount script 4.3+ 00_mount_sdcardtmp.sh is pre-mounting the sdcard first, then mounting the ext4 image file. both files are just examples and will not work for you without modifying the paths.
Stock kernel does support init.d, but SuperSU is required for this. for better understanding please read Pandages' short description how it works.
For Link2SD i have installed a ext4 image file located on external microSD Card. Due the limitation of FAT32 file systems, the maximum file size is 4 GB. However, some busybox binaries are compiled without large file support and cannot create files bigger then 2 GB - thats why this download comes with busybox. if you don't have arm cpu, there is a copy of busybox /system/xbin/busybox.bak which you must manually rename.
initial create empty ext4 image file
Code:
busybox dd if=/dev/zero of=/sdcard/.data.sdext2.img bs=4k count=1048576
busybox mke2fs -T ext4 -F -q /sdcard/.data.sdext2.img
run the commands from adb shell or terminal emulator
the dd command creates a new file on sdcard. the file size is specified with count.
the mke2fs command is formatting the image file with ext4 file system.
the (.) dot in front makes a filename hidden.
helpful commands for initial gathering
Code:
busybox df
busybox cat /proc/partitions
busybox mount
all mount points with same size usually have identical partition (group by block size). query for matching major + minor in /proc/partitions and find the name (for example /dev/block/mmcblk1p1). make sure the mount point exists and is empty, or create a new folder (for example /storage/sdcardtmp). mount shows the file system type (for example vfat).
modify the paths in 00_mount_sdcardtmp. you have to define two different mounts. the mount point for Link2SD is /data/sdext2.
After this, Link2SD now detects 2. partition on boot. but instead of a 2. partition you just have a single image file on sdcard (which can easily copied for backup)
the download includes a small install shell script which just copy the files into /system. provision for making files executable is unpacking the downloaded zip file init_d.zip to /data/local/tmp. don't forget to set permissions after unpacking.
[TUTORIAL] PLEASE CLICK HERE TO EXPAND PICTURES
(rotate the page to landscape for best view)
1. UNPACK THE FILES2. SET PERMISSIONS
extract file init_d.zip tomake busybox and
/data/local/tmpinstall.sh executable
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
3. INSTALL init.d
execute the shell scriptmake sure script was
install.sh with su permissionsfinished successfully
(or terminal command: "su -c sh install.sh)"(or you already have init.d support?)
4. GATHER SDCARD PARTITION PARAMETERS
Read Driver and Device Numbers
open a root shell. make sure you have su permissions (type "su").
type "busybox df"
identify the external microSD Card folder. its usually "/storage/sdcard"
(or something, and symlinked to "/sdcard")
group all mount points with same size (group by block size)
find the related filesystem, starting with "/dev/block/vold/puplic:179,97"
(or something)
Find the Block Partition
open a shell
type "busybox cat /proc/partitions"
identify Driver and Device Numbers, for example major = 179, minor = 97
find the name. complete the path, starting with "/dev/block/mmcblk1p1" (or something)
Copy the Mount Options
open a shell
type "busybox mount"
identify the related filesystem, starting with "/dev/block/vold/puplic:179,97" (or something)
find the filesystem type, for example "vfat" (depends on microSD Card)
copy the mount options. usually from "rw" to "uid/gid"
5. CREATE THE MOUNT SCRIPT
use Mount-Button to make system partition read-write able
(or terminal command: "su -c busybox mount -o remount,rw /system")
navigate to folder "/system/etc/init.d"
delete the example file "00_mount_sdext2" (not required)
open the example file "00_mount_sdcardtmp" in text editor
MODIFYING THE init.d MOUNT SCRIPT EXAMPLE
Pre-mounting external microSD Card
edit the first mount line. usage: mount -t <type> -o <options> <source> <target>
Source: "/dev/block/mmcblk1p1" (Block Partition, see above 4.)
Target: "/storage/sdcardtmp"
Filesystem type: "vfat" (see above 4.)
Options: "rw, ..." (see above 4.)​example: "mount -t vfat -o rw,dirsync,nosuid,nodev,noexec,relatime,uid=1000,gid=1015 /dev/block/mmcblk1p1 /storage/sdcardtmp"​Mounting the ext4 image file
the second mount line is like the first mount, except the source is a file. thus requires the additional mount option "loop". further use same mount options as used for "/data" (usually from "rw" to "uid/gid"). these might be different from above.
Source: "/storage/sdcardtmp/.data.sdext2.img" (Image File, see below 7.)
Target: "/data/sdext2" (see below 6.)
Filesystem type: "ext4" (see below 7.)
Options: "loop,rw, ..."​
6. PREPARING MOUNT POINTS
Make sure the target folders exists and folders are empty. consider that file names are case-sensitive.
You can create a new mount point asFor Link2SD a fixed mount point
well. the folder name for microSD Cardis defined, the folder name is
is an example, vary it as you like.detected from app.
use Mount-Button to make root directory read-write able
(or terminal command: "su -c busybox mount -o remount,rw /")
navigate to folder "/storage"
create new folder "/storage/sdcardtmp"
navigate to folder "/data"
make sure the identical path is specified
in mount script "00_mount_sdcardtmp"
(see above)
create new folder "/data/sdext2"
7. CREATE THE EXT4 IMAGE FILE
Finally, create the Image File "/storage/sdcardtmp/.data.sdext2.img". the file name is an example, vary it as you like. choose a location anywhere on external microSD Card. the (.) dot in front of filename makes a file hidden.
identify the external microSD Card folder.
its usually "/storage/sdcard" symlinked to "/sdcard"
calculate the number of blocks.
the maximum file size is 4096 bytes x 1048576 = 4294967296 bytes (4 GiB)
open a shell
create a new file with busybox "dd" command
Code:
busybox dd if=/dev/zero of=/sdcard/.data.sdext2.img bs=4k count=1048576
wait 10 minutes
format the file with busybox "mke2fs" command. the filesystem type is "ext4"
Code:
busybox mke2fs -T ext4 -F -q /sdcard/.data.sdext2.img
make sure the identical file name is specified in mount script "00_mount_sdcardtmp"
(see above)
8. MOUNT THE (EMULATED) SECOND PARTITION
Test the mount script
open a root shell. make sure you have su permissions (type "su")
type "sh /system/etc/init.d/00_mount_sdcardtmp"
type "busybox df"
if successful, there are two new mounts "/storage/sdcardtmp" and "/data/sdext2"​
Check the mount options
type "busybox mount"
identify the new mount points "/storage/sdcardtmp" and "/data/sdext2"
compare the mount options. they should be similar:
"/dev/block/vold/puplic:179,97" = "/storage/sdcardtmp"
"/data" = "/data/sdext2"
(or something)
FINISH. REBOOT THE PHONE. CHECK IF SUCCESSFUL
NOTE: IF YOU WON'T GO THROUGHT THIS MANUALLY CODING TORTURE
there is an app from google play called Mount2SD the solution as it has most of the mount scripts needed. It won't work with encryption. I personally prefer the loop file method descriped here (and never tested it), but looks like a good alternative for Link2SD
NOTE: The following apps are used:
For older versions (screenshots) please see Attached Files below
I found the tar.gz but its so difficult for me i dont understand
for unpacking, busybox or any unpacking app is needed. for example root explorer can browse tar.gz files and extract it.
for manually unpacking the tar.gz
Code:
busybox gunzip -c init_d.tar.gz | busybox tar -vxC /data/local/tmp
This is the screenshot command u ask me for
But i just download your file it is still in download folder .. should i unpack init.d.tar.gz first ? And then send u the command ?
Btw the screenshot is from last to first start with busybox df
Sorry i never seen newer android, knox is a new thing to me. seems like the /data partition is emulated, maybe it is not available during boot?
you can try to pre-mount sdcard. in your case it is /dev/block/vold/puplic:179,33 - (major = 179, minor = 33) - which is equal to /dev/block/mmcblk1p1. the mount point is /storage/31D1-1308 - but its not empty, so you have to create a new folder for example /storage/31D1-1308tmp (which must re-created from your init.d script on every boot). the file system type is sdfat. the mount options i am not sure. try rw,dirsync,nosuid,nodev,noexec,noatime,nodiratime,uid=1023,gid=1023 (taken from your screenshots)
this is for the first mount. check if it works, you should see files in /storage/31D1-1308tmp (and new mount is listed in busybox df). you can check the mount command on adb shell or terminal emulator, it works during runtime.
for the second mount then create a ext4 image file first. then mount /storage/31D1-1308tmp/.data.sdext2.img to /data/sdext2 type ext4 with options loop,rw,nosuid,nodev,noexec,relatime,low_uid=1023,low_gid=1023,gid=9997 if success /data/sdext2 is listed in busybox df.
if mount works on adb shell, but not in mount script, increase the timer busybox sleep 4

Categories

Resources