howto apps2sd as ext4 - myTouch 3G, Magic Android Development

howto apps2sd as ext4 - myTouch 3G, Magic Android Development

Ok. So lots of talk about ext for apps2sd and how it's being mounted as ext2 even if you formatted to ext4 etc.
Also talk about how journalling hurts performance and such. We'll tackle all those points here.
1. Howto fix the mounting of your ext partition in the meantime till your mod updates their software.
A. Pre-requisites
You must have already partitioned your sd card with an ext4 fs for apps2sd. You can do this in recovery or some other app.
You should also get the program file expert. It's awesome. But any filemanager that has root ability will work. You can do all this in terminal too so whatever floats your boat. I'll use file expert below.
B. Remount your system fs as read/write (not sure if it's actually needed since /system is mounted rw by most mods already... but it can't hurt). You can do this in file expert by hitting menu and selecting More, then the top option labeled "Mount", select read write. It should say success.
You can also do this in terminal but like i said above, i dont think this is absolutely needed but i always assumed it was so never tried this not doing it.
C. go back down directories using the back key till you hit /. Then goto /system/etc/init.d/
D. select S05mountsd (the S0# isn't important here so if yours is different dont worry). Once selected, hit menu and select file operations. Select permissions and change permissions to read/write all.
E. Open the file with a text editor of choice. Scroll down the file to the part that has the command with -t auto in it. It should be pretty obvious, and it's near the end.
Change -t auto to -t ext4. It doesn't matter if your fs is actually ext2 or ext3, mounting as ext4 is always better. The ext4 block allocator and general code is more efficient even if you dont use ext4 features.
You may also want to add an option to the options list data=writeback. This disables journal caching of actual data and only caches metadata. This is really integral to performance on such a slow medium like sd cards and we'll discuss that later.
Save your file and exit.
F. Reboot your phone. You should now have your sd-ext mounted as ext4.
2. data=writeback and why?
there are multiple modes of journalling available to ext4. The default is ordered. This means that there is a specific order that information is written to the fs to ensure that nothing is lost that is said to be written. This means metadata about the fs and file data is both cached in the journal. Safe but slow.
data=writeback still allows crash protection but means you may lose actual newly written data. Your fs wont be corrupt, but your file may not have gotten those last updates you were writing and had not synced yet. It's a pretty decent compromise.
There are other forms that offer better performance but basically remove journal integrity and we're not concerned with those.
There are also options that determine the interval of journal syncing and you can find that information on ext4 man pages.
Why do we care so much about performance here? because sd flash is slow and finite. A journal file is like swap in that it is a static file that's contents get written to anytime you write to your fs. On a normal hdd, it's not noticeable really but on a class 6 SD card, you are going to notice having to write everything twice to your sd card. So we choose writeback mode to remove that double writing and only cache fs metadata updates so that the integrity of the filesystem isn't hurt, and only that particular file that we crashed prior to syncing gets effected. 99% of the time this is fine and perfectly safe and way better than nothing.
3. So if journalling can hurt performance by using up write bandwidth to a medium we are writing other stuff on, wouldn't swap be bad too?
Yes. Swap should only be put on sd cards if you absolutely need it or your phone wont work. Aside from stressing the card, the bandwidth to read/write is extremely small for something you basically have to be interactive with and move to and from ram. Additionally android complicates matters by managing memory at a higher level than the kernel and this means that swap is ignored as a portion of memory. So while technically userspace apps deal with swap transparently, the low memory manager of android will operate as if you didn't have swap...which defeats the purpose.
Use swap as a last choice.
Also, the ability to make swap partitions and swap files is a common issue in linux-land. It's best to use a partition as this removes fs overhead but it becomes even more of a concern when you put it on an ext4 fs with journalling active. With default settings, a swap file will be considered just another file and the journal will cache the data and metadata updates (very bad for performance). if you use swap on an ext4 fs that's actually mounted as ext4 you _NEED_ to make the data option writeback. You will be very disappointed if you dont.
4. Other important notes:
If you use any ext4 specific options like extents, you will no longer be able to read the files written by recovery programs that have not been updated to mount fs's as ext4. Also, just enabling extents doesn't convert data on the fs to use them. You have to write the files while the option is enabled to see their benefit. This is more easily done by putting the card in an actual linux computer and mounting with the correct options and copying the data from the filesystem and then back again ....not for the novice. Though any new files written will automatically have extents even if you dont redo all current data. So best to bother with that after a fresh install prior to restoring apps.
You may also be thinking... Why do i even need a fat fs, i can make the whole thing ext4 and it will be awesome! You would be right, it would be awesome. But it's not so easy. A couple things have to be done in order to make everything written world rewritable and accessible on such a partition. you would have to setgid to a group like "sdcard" to a directory in this new fs and give it rwx group permissions and then have all users current and future become part of that group and mount bind the directory to /sdcard. Needless to say, recovery apps would have to also be updated. Then you could make your main sd partition any fs your kernel supports.

Nice post, cm7 and diet gb I build already have -t ext3 set, do you think ext4 would make much difference?

ext4 has a better block allocator and other features both backwards compatible and ext4 specific. There's really no reason to not mount ext2/3 and 4 as ext4

Thanks for the answer, This is also in the kernel I compile,
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_XATTR=y

Yes, i began my work on the kernel tonight and noticed that in the latest kernels at least that ext4 is being masqueraded as ext2 and as i thought early on as might be the case, that the reporting of the partition as being mounted as ext2 is a bug that has been squashed in the newest vanilla kernels.
This is still an issue, but not a serious one. FS repair tools and such may bug out when trying to found out what fs they're dealing with as the kernel reports the wrong ext version but the functionality of the fs is not affected ( you still get journalling and all that jazz).
So this is less of a performance issue (cept the adding of data=writeback) and more of an issue with other programs needing to see the correct fs.

Ur Diet 09.05 still did not see ext4 partition by Titanium.
carz12 said:
Thanks for the answer, This is also in the kernel I compile,
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_XATTR=y
Click to expand...
Click to collapse
Sent from my Gingerbread on Sapphire using XDA Premium App

If the kernel you're using is one of the newer kernels (probably anything after 2.6.32...but you should be using something much newer) then it likely doesn't even have ext2 or ext3 compiled in. It's all just ext4 like it should be. The issue you still may have however is that the kernel reports the lower revision extfs because ext4's driver accepts the ext4 fs but the kernel is checking for ext2....
Basically, we are asking if this is an ext2fs. The ext4fs driver answers cuz it's all there is and it's masquerading as ext2 and ext3 and ext4. So it looks and sees ext4 and says, hey, i can read this and returns YES. Now the answer yes was to the question of ext2, so that's what gets reported everywhere, but internally, ext4 knows it's got an ext4 fs. mount options that normally would error out with -t ext2 are accepted because the fs driver isn't really ext2 and it knows the fs isn't either so the mount options pass.
This is a kernel bug in an aesthetic and practical sense. Functionally it's all working as intended but other apps and users will get confused as to what exactly is the filesystem, ext2, ext3 or ext4. It can't know without probing the fs itself. Which is stupid for many applications.
Supposedly the latest kernels have a patch to fix this to correctly report the correct fs revision of ext. Though perhaps that hasn't been pushed to linus's branch yet.
In any case, you still should be modifying the mount options to use data=writeback so you're not clogging up the little SD bandwidth you have with data caching in the journal. You really only need fs metadata caching so you dont bork the entire partition on a crash. data=writeback does that, but default is data=ordered.

thank you for such a good post!

Related

apps on sd card better way

Is there better way to get applications on sd card. Why are we using ext2 ? We can format sd card and partition it without using pc . We can format it into native supported file system of fat. why are need to make dev when in /dev/block device exist ? we need just to modifiee mounted.conf and /etc/fstab

I think because it's easier to say then to do But if you can make it so easy i love to use youre methode! (newest way is pretty easy, flash modded JF rom and exec a script and you are done: http://forum.xda-developers.com/showthread.php?t=480582)

I would think that you could run into issues with permissions/etc if you tried to use a FAT filesystem to store applications.
About the only advantge of FAT is its ubiquity. In every other regard there is a better solution out there. Ext2 is by far the most widely supported linux filesystem that supports the full POSIX feature set.

simply put FAT32/VFAT is a horid horid filesystem, and does not support unix permissions, symlinks, etc.
EXT2 is a native linux filesystem, supports all of the above and has been time tested as stable.
That said it would be slick to get EXT3 (or 4!) support, or even reiserFS support.... journaling FTW

Well, you could just compile the kernel module, but your flash would probably survive a week with journaling... theres a reason you don't use it on flash.
Flash gets worn down by use, and journaling writes every second... So, journaling FTL.

tritron777 said:
Is there better way to get applications on sd card. Why are we using ext2 ? We can format sd card and partition it without using pc . We can format it into native supported file system of fat. why are need to make dev when in /dev/block device exist ? we need just to modifiee mounted.conf and /etc/fstab
Click to expand...
Click to collapse
the phone is linux .. EXT2 is linux .. FAT does not support proper file permissions and if you wanted to try setting something up feel free .. however .. you mention "native" supported file system and that is linux .. storage on the SD comes standard with FAT file system simply because the phone only uses it to store extra files like MP3 and JPG etc .. applications need a higher level of communication that FAT just cannot handle .. that's why Microsoft gave up on it for windows

"Stale NFS file handle"

Hi all
I have all my apps and data moved to the sdcard a while back but my phone crashed and I decided to move only the apps to the sdcard.
I'm trying remove the /system/sd/data folder on the sdcard and keeps getting that error message. How to fix it?
Thanks

This should only happen if an NFS export is mounted to a linux box of some sort, and that export has changed or otherwise become inaccessible.

any idea how to fix it?

i have this problem, i'm not able to delete the data folder beacause of that error. :/

your ext2 partition is corrupted. this happens quite a lot if data is moved to the sdcard. boot up linux and run fsck on the sdcard and that should fix it.

cool. I'll give that a try.
Thanks

I'm having this problem too. Can you provide some basic instructions for how to mount and fix under linux? I have an ubuntu vmware image I can boot to on my windows pc and a usb card reader. Will that suffice?

When you plug the card into an ubuntu box it should automatically mount it as the next available drive. You'll have to figure out what device node the card shows up as, unmount it (umount /dev/<insert device name here>), and then run a filesystem check (fsck /dev/<insert device name here>) on the unmounted card. The utility will report various problems about "inodes" which you will want to say yes to fix. Once it has run through the file system should be in a consistent state and ready for use again.
You run the risk of losing stuff written to the card (which is probably corrupt anyways) when you run the fsck so you may want to take a copy of the data first.
On a side note: I am not sure what the default mount options are listed for moving the stuff onto the sd card in the faqs but I suspect it may help prevent corruption to mount the card on android with the sync option. Though, this will definitely slow writes to the card. It would definitely be a bad idea to remove the card while your G1 is running either way.

equid0x said:
On a side note: I am not sure what the default mount options are listed for moving the stuff onto the sd card in the faqs but I suspect it may help prevent corruption to mount the card on android with the sync option. Though, this will definitely slow writes to the card. It would definitely be a bad idea to remove the card while your G1 is running either way.
Click to expand...
Click to collapse
I was thinking of using the sync option, but then I read this at http://linux.die.net/man/8/mount:
sync
All I/O to the file system should be done synchronously. In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.
Click to expand...
Click to collapse

Busybox seems to have an fsck command built in, but I don't think all the supporting stuff is there. I'd like to have a way to fsck my ext2 partition while on the go and not near my linux box.
I know that you can't fsck without unmounting the partition and of course it would be bad to unmount the partition with apps on it while the phone is running, but I was thinking it would be nice to be able to boot into the recovery console.
I tried this and attempted to do a fsck on /dev/mmcblk0p2 with the fsck in busybox as follows:
Code:
busybox fsck -t ext2 /dev/mmcblk0p2
But the error I got was that fsck.ext2, which is the actual executable that should be used, isn't there.
What would it take to get this onto the system so that I could boot into recovery and do a quick fsck and then reboot back into phone mode?

I was thinking of using the sync option, but then I read this at http://linux.die.net/man/8/mount:
Click to expand...
Click to collapse
Where flash lifetime is concerned... I agree, this will certainly have some impact. However, the amount of wear concerned really depends on the number of write cycles the particular flash you are writing to can handle, and how good any wear leveling in the memory controller is. Modern flash memory will likely last on the order of years even with tons of writing going on. If all you are moving to the card are the apps, that data will likely be written once (or maybe a few times over the life of an app) and only re-read from that point forward. The caching will eventually commit any data in the buffer to "disk" regardless of how much is actually there. The idea is to line up all the writes so they can be done efficiently. Where ext3 is concerned, the commit interval is 5 seconds by default, I am not sure what it was in ext2 but I imagine it is similar. Ext2 is not really a flash optimized filesystem, but it is readily available on basically any linux distro, and is supported on Android. A better fs for flash drives where write cycles are an issue might be something like jffs2 or yaffs.
At any rate, sd cards are cheap. Why not just throw it away when it starts to die?
But the error I got was that fsck.ext2, which is the actual executable that should be used, isn't there.
What would it take to get this onto the system so that I could boot into recovery and do a quick fsck and then reboot back into phone mode?
Click to expand...
Click to collapse
You would have to compile an ARM6 compatible version of fsck and get it onto the recovery partition so you could run it.

just turn off your phone, pull out the sdcard, boot on a linux os and
then in console type :
fsck -p /dev/your_ext2_partition

Is there any way to clear this error message on a windows xp computer?
Maybe use pargon partion manager, but where do I go to fix it in pargon?

equid0x said:
Ext2 is not really a flash optimized filesystem, but it is readily available on basically any linux distro, and is supported on Android. A better fs for flash drives where write cycles are an issue might be something like jffs2 or yaffs.
Click to expand...
Click to collapse
Hmm... Well, we know that yaffs is supported on Android because that's what the onboard storage uses. So I wonder why the tutorial for apps-on-sdcard suggests creating an ext2 partition? Couldn't we create a yaffs partition on the sdcard and use that instead?
Maybe because yaffs isn't as commonly supported in non-Android partitioning tools (which you would have to use to set up the card initially)?
In any case, if there's no real downside to having the partition be yaffs, how can I go about formatting it as such? I don't recall seeing such an option on gParted or anything, but then again I didn't look that carefully.

So can anyone clue me into how I might add yaffs/yaffs2 support to my desktop linux box? I'd like to try using a yaffs2 partition rather than ext2 to see if I get better reliability, but I need to add support for that filesystem first. Can't seem to figure out how to do that quite yet...

You will likely need to create the partition from the terminal using something like fdisk or cfdisk which will allow you to select the proper filesystem ID. The command for creating the filesystem is mkyaffs. The fstab in android will need to be modified to enable mount of this file system at boot. You will need to install all of the yaffs support tools on a linux desktop to get access to the mkyaffs command.
Yaffs is designed to be used directly on NAND or NOR flash memory (not abstracted through the controller built into an SD card) but it may work anyways. I am not very familiar with the specifics of this particular FS. Most of these flash filesystems are designed to provide a bootable root filesystem for an embedded device.
Yaffs kernel support can be built into a custom kernel with the instructions here:
http://yaffs.net/howto-incorporate-yaffs
Its not likely you will find pre-made packages for any of this in a common distro like Ubuntu. So, you will need to know how to compile it all by hand. A good starting point for a lot of linux info is The Linux Documentation Project at:
http://www.tldp.org/
FWIW I have built homebrew linux based routers for dual ISPs, IPSEC VPN and the like using a lightly modified version of CentOS and 4GB CF cards plugged into an ATA adapter. I used EXT2 on these and they were in production use at a small 13 server farm for a couple of years before being replaced with newer equipment without any failures whatsoever.
I have also used CF cards in small 200Mhz cube PCs as basic web kiosks for extended periods of time without any failures as well.
Under normal usage patterns on a mobile device probably does not require a large amount of writes in the grand scheme of things. I'd say it is fairly likely that your card will outlive the device you are using it in regardless of the filesystem in use.
If you are seeing lots of corruption I would suspect a flakey/failing SD card or some other hardware related problem. It definitely pays to buy high quality flash media. I would also suggest not allowing the phone to constantly run dead if you know things are being written to the SD card, since random power failures during a write to flash can permanently damage the media.

[CONCEPT] Single Partition No-Format Apps2SD

So I was using Slax. Great LiveCD/USB linux, extremely customizable, modular, fast, and small, and has the capability of either:
- saving changes to its rootfs onto an AUFS mounted on a non-linux FS (FAT32, NTFS) using posixovl (POSIX Overlay FS) with metadata (permissions, etc.) being held in files
- saving changes to a fixed-size loop mount image.
This got me thinking.
If we could insert all the necessary modules, code, etc. for posixovl into the Android linux, and make a modified a2sd script that takes advantage of posixovl, we could effectively do away with the requirement for crazy partitioning.
It should be simple enough for ROM devs to implement, assuming it's ready and installed:
1. Create folder on main partition if it doesn't exist, something like /sdcard/system/[app, app-private, dalvik-cache, app_s]
2. In the init scripts, before the a2sd stuff, mount /sdcard/system with posixovl on /system/sd
3. Run a2sd as normal, it should automatically just work.
I'll hopefully test this once I get my phone to a stable development/testing stage, and I don't need to make phone calls for a while. Anyone else is welcome to try to implement this idea.
My current test environment:
- HTC Dream (T-mo G1) with the deadly SPL of doom
- Cyanogen Experimental, latest build
- Amon_RA's modded recovery
- Wipe /data, move all existing apps to backup, remove a2sd partition, Backup for Root Users to restore some settings and data
Anyone with ideas or improvements, please let me know.
To be tested:
- Feasibility (can it work?)
- Functionality (does it work?)
- Portability (Can it work on other ROMs and devices like Hero, Pulse, Blur etc.? If so, will likely be moved to XDA's new Android board)
- Stability (Will everything Force Close on boot? Does it run fast enough? Does anything get corrupted over time?)
Links:
- http://sourceforge.net/projects/posixovl - Sourceforge page for posixovl

In desktop linux you can create a file with the touch command, and mount the file to a mountpoint after formatting it to ext4 for example.
Maybe this is the easier way?
I have done this about 5 years ago, but I will try it today and report if it worked.
edit: ok done already:
1. create a file of the desired size, eg: dd if=/dev/zero of=filename bs=filesize count=1
2. use mke2fs to format the file
3. create a mountpoint and mount the file
thats all. Now I have a 128MB file on my PC, mountable and usable like a partition.

Archont said:
In desktop linux you can create a file with the touch command, and mount the file to a mountpoint after formatting it to ext4 for example.
Maybe this is the easier way?
I have done this about 5 years ago, but I will try it today and report if it worked.
Click to expand...
Click to collapse
As far as I know, you can do that on the mobile Android, too, and that does work in theory. This technique involves mounting a loop filesystem, and it too will allow one-partition apps2sd, but it's less flexible, and I would think slower, than the overlay method.
For a 512MB apps image:
Create empty 512MB file
# dd if=/dev/zero of=/sdcard/apps.img bs=1024k count=512
Format it to Ext2
# mke2fs -L Apps2SD /sdcard/apps.img
Unmount existing a2sd
# umount /system/sd
mount new a2sd image
# mount -t ext2 -o loop /sdcard/apps.img /system/sd
Make the usual directories, and a mountpoint for the old a2sd partition
# mkdir /system/sd/app; mkdir /system/sd/app-private; mkdir /system/sd/dalvik-cache; mkdir /system/sd/apps-tmp
Mount the old a2sd partition
# mount -t ext2 /dev/mmcblk0p2 /system/sd/apps-tmp/
Move all files from the old partition to the image file
# mv /system/sd/apps-tmp/* /system/sd/
Unmount and remove the mountpoint, we don't need it anymore
# umount /system/sd/apps-tmp
# rmdir /system/sd/apps-tmp
Finally, you add the following line to the init script where the a2sd auto mount happens, and comment out the old line.
Code:
[...]
mount -t ext2 -o loop /sdcard/apps.img /system/sd
#mount -t ext2 /dev/mmcblk0p2 /system/sd/
[...]
This should do what you described, in theory. I can't say whether it will work or not. I can't tell whether it will or won't screw up your phone, I can't be held responsible if you screw something up or overlook the details. Either of us might have made a typo somewhere; apply common sense before doing anything.

This sounds great! I think this would also be usefull for someone like me, who has a sd card that doesn't want to be partitioned anymore (cross-linked files??). Only thing possible is fat32 or ntfs.
Am I correct with my assumption?

This sounds a lot harder and more complicated then partioning. Your also talking about a lot of work needing to be done just so people can avoid doing a simple thing like partioning a sd card. I would say it would be very difficult alone to get the os to run a virtual mounting service especially since that will take up resources and slow down the phone. There is a reason they only use this technique on live cds is it works but its slow. most of the computers they are running on have 1gig of ram and 2ghz cpu's. i really dont think the g1 can push this.

I do not think that this is great, it definitely is slower than a ext4 partition on a good class 6 microSD card. And it is more vulnerable to data loss since 2 different filesystems including a 20 year old non-journaling fs at the base of this construct are involved.
Another problem that came to my mind: when you mount your SD card as external USB device to a PC, the file containing your apps will no longer be accessible, or Android will make using the phone as external data storage impossible.

Interesting.
There is, however, a major problem: What happens when you unmount the fat partition on the phone in order to connect with a computer using UMS? Answer: everything on the phone will crash and burn since the apps filesystem will suddenly disappear = BAD.
posixovl is a nice find though...
Note that aufs, loopmount linux filesystems, etc., wouldn't be needed with this since posixovl appears to be vfat with posix extensions, so you should be able to just use posixovl directly on the sdcard.
There are several problems with that though... i.e. how reliable is posixovl regarding users tampering with it?
In any case, a prerequisite for use of it would be certain other changes being planned...
You might want to contribute to this thread:
http://forum.xda-developers.com/showthread.php?t=577941
(note: the thread links to a thread at android-platform, the one here has, as expected, gone off on a tangent... just ignore the junk.)
TylTru said:
So I was using Slax. Great LiveCD/USB linux, extremely customizable, modular, fast, and small, and has the capability of either:
- saving changes to its rootfs onto an AUFS mounted on a non-linux FS (FAT32, NTFS) using posixovl (POSIX Overlay FS) with metadata (permissions, etc.) being held in files
- saving changes to a fixed-size loop mount image.
This got me thinking.
If we could insert all the necessary modules, code, etc. for posixovl into the Android linux, and make a modified a2sd script that takes advantage of posixovl, we could effectively do away with the requirement for crazy partitioning.
It should be simple enough for ROM devs to implement, assuming it's ready and installed:
1. Create folder on main partition if it doesn't exist, something like /sdcard/system/[app, app-private, dalvik-cache, app_s]
2. In the init scripts, before the a2sd stuff, mount /sdcard/system with posixovl on /system/sd
3. Run a2sd as normal, it should automatically just work.
I'll hopefully test this once I get my phone to a stable development/testing stage, and I don't need to make phone calls for a while. Anyone else is welcome to try to implement this idea.
My current test environment:
- HTC Dream (T-mo G1) with the deadly SPL of doom
- Cyanogen Experimental, latest build
- Amon_RA's modded recovery
- Wipe /data, move all existing apps to backup, remove a2sd partition, Backup for Root Users to restore some settings and data
Anyone with ideas or improvements, please let me know.
To be tested:
- Feasibility (can it work?)
- Functionality (does it work?)
- Portability (Can it work on other ROMs and devices like Hero, Pulse, Blur etc.? If so, will likely be moved to XDA's new Android board)
- Stability (Will everything Force Close on boot? Does it run fast enough? Does anything get corrupted over time?)
Links:
- http://sourceforge.net/projects/posixovl - Sourceforge page for posixovl
Click to expand...
Click to collapse

lbcoder said:
Interesting.
There is, however, a major problem: What happens when you unmount the fat partition on the phone in order to connect with a computer using UMS? Answer: everything on the phone will crash and burn since the apps filesystem will suddenly disappear = BAD.
Click to expand...
Click to collapse
I kinda overlooked that point. Oops.
Though this same problem does exist on regular apps2sd when you remove the card without dismounting it, killing all apps and their processes, and freezing Dalvik's method of autostarting some apps.
I do tend to swap cards every now and then, but only after a reboot. Dalvik re-enumerates and caches dex, which makes for a slow boot, but it seems to just work in most cases that the apk install doesn't drop the app's functional payload (helper Linux/shell utils, libraries, NDK .so's) in /data/data (like some emulators, the Android Scripting Environment)
lbcoder said:
There are several problems with that though... i.e. how reliable is posixovl regarding users tampering with it?
Click to expand...
Click to collapse
As far as I know, the metadata files are marked as hidden and system files, and begin with a '.'. And I haven't tried this, but I think modifying the actual files under Windows has no negative effects, but moving, deleting, or copying files would likely be a no-no.
Also, I don't think there's a 'fsck' for posixovl, meaning that if any metadata files were screwed with the wrong way, the entire overlay FS would be trashed.
lbcoder said:
In any case, a prerequisite for use of it would be certain other changes being planned...
You might want to contribute to this thread:
http://forum.xda-developers.com/showthread.php?t=577941
(note: the thread links to a thread at android-platform, the one here has, as expected, gone off on a tangent... just ignore the junk.)
Click to expand...
Click to collapse
I checked that out. It was actually a small inspiration for what I was thinking of.
In any case, Android's package management system needs an overhaul. The package storage needs to be de-Linuxified, as all it is is a bunch of .apk files and .dex/.odex files, the UIDs of apps are in the AndroidManifest.xml, right?
In a somewhat unrelated note, app data needs to be moved to a specified folder structure on the sdcard. My card is full of folders in the root directory with random names.

If I'm understanding you correctly, you're talking about storing an image file on the normal SD card partition(which has to be FAT32 as far as I've seen) and then mounting it, correct? This idea has been talked about at length before on at least 3 separate occasions(2 of which were on this very forum) and found to be a bad idea due primarily to massive security risks since FAT32 has no permissions.
Also, I believe cyanogen ended up dumping unionfs/aufs due to rampant memory issues.
If you are talking about mounting an image from the FAT32 partition, please don't endorse this. We don't want to be throwing in security bugs into android, especially ones such as this which can't be plugged up.
As a modification to what I said: If you're suggesting doing this(or something similar) on a separate filesystem, after that project to change the AOSP to support one with permissions is finished, then I'm in full support.

If you want to go for a single partition on the sd card, why don't you just make the entire card use ext4? Your linux desktop reads it anyway, it uses journaling and so on, I guess it would be faster compared to fat32 and it is definitely safer to use.
And i guess it is not too complicated to mount this partition and use it for pictures, music and so on.
I have not tried this (yet) and I go to bed in 20 minutes, but maybe I will start testing something in that direction tomorrow.

[email protected] said:
If I'm understanding you correctly, you're talking about storing an image file on the normal SD card partition(which has to be FAT32 as far as I've seen) and then mounting it, correct? This idea has been talked about at length before on at least 3 separate occasions(2 of which were on this very forum) and found to be a bad idea due primarily to massive security risks since FAT32 has no permissions.
Also, I believe cyanogen ended up dumping unionfs/aufs due to rampant memory issues.
If you are talking about mounting an image from the FAT32 partition, please don't endorse this. We don't want to be throwing in security bugs into android, especially ones such as this which can't be plugged up.
As a modification to what I said: If you're suggesting doing this(or something similar) on a separate filesystem, after that project to change the AOSP to support one with permissions is finished, then I'm in full support.
Click to expand...
Click to collapse
UNIX permissions don't do anything in the way of "security" unless you have no access to the actual storage device from another computer (as is the case with the unrooted Dream's internal memory), or unless encryption is used. The posixovl driver OVERLAYS Unix permissions over Fat32 filesystems. But even still, with Unix permissions, nothing's stopping someone else from mounting the Ext2 partition and using chown and chmod.
And the image file on the SD card's Fat32 partition is a complete Ext2 partition complete with Permissions. Nothing is lost.
Archont said:
If you want to go for a single partition on the sd card, why don't you just make the entire card use ext4? Your linux desktop reads it anyway, it uses journaling and so on, I guess it would be faster compared to fat32 and it is definitely safer to use.
And i guess it is not too complicated to mount this partition and use it for pictures, music and so on.
I have not tried this (yet) and I go to bed in 20 minutes, but maybe I will start testing something in that direction tomorrow.
Click to expand...
Click to collapse
We'd just have to find the part in the Android that mounts /sdcard/, and change 'vfat' to 'ext2'. The only reason I wouldn't do this, is because it would immediately make it incompatible with Windows and Mac's default FS drivers. As far as I know, the only FS's that are supported universally within Linux, Mac, and Windows, are FAT and NTFS. And NTFS can be made to have crude support for permissions through security descriptors. Although, the Dream SPL, the Recovery images, and most of Android only uses FAT32.

This is discussed in android-platform Group :
http://groups.google.com/group/andr...read/thread/bf0709c157451cd9/f6aee1830c84620f
The goal is to be able to integrate this in android.
And not having to partition the SDCard is one of the requirements so far...

Unix permissions are not stored using fat or vfat, and ntfs is not really supported in desktop linux and i guess it cannot be used in android linux.
I would not use windows anyway so this is no problem to me, and there are drivers around to mount ext systems in windows. As Mac OS is based on unix there will be a solution for this too.
Access usind adb push and pull, via ftp and so on is not touched by using ext4 on the entire sd card I guess.
And if you don't go the easy way using gparted on a live cd or usb device to create 2 partitions, you will have to live with some disadvantages anyway.
Finally I want to say that my ideas are far from being perfect or usable at all, I see this thread as a kind of brainstorming.

im not as linux or android savvy as probably any of you but before the current method of creating a swap partition became the "standard", people used a swap file on the sdcard and linked that. seems similar to what you are suggesting here.
ofcourse when mounting the fat partition elsewhere (ums in windows for example) that swap file could no longer be used within android. i dont see a way to get passed the same issue, but worse here, due to android not having crucial apps when the fat partition is mounted.
then again, i am pretty much over my head in this conversation and could be over looking something...

I'm kind of fascinated by the FUSE + posixovl method of doing this. In the long run I have a feeling that it's going to perform like ****, but I think it's worth testing.
I managed to get both libfuse and mount.posixovl built and running on Android.
posix-overlay(/sdcard/fuse) on /sdcard/fuse type fuse.posixovl (rw,nosuid,user_id=0,group_id=0,default_permissions)
Giving this a little testing now, it definitely works.
Code:
/sdcard/fuse # ls -l
drwxr-xr-x 2 1000 1000 4096 Nov 5 17:17 test

TylTru said:
UNIX permissions don't do anything in the way of "security" unless you have no access to the actual storage device from another computer (as is the case with the unrooted Dream's internal memory), or unless encryption is used. The posixovl driver OVERLAYS Unix permissions over Fat32 filesystems. But even still, with Unix permissions, nothing's stopping someone else from mounting the Ext2 partition and using chown and chmod.
And the image file on the SD card's Fat32 partition is a complete Ext2 partition complete with Permissions. Nothing is lost.
We'd just have to find the part in the Android that mounts /sdcard/, and change 'vfat' to 'ext2'. The only reason I wouldn't do this, is because it would immediately make it incompatible with Windows and Mac's default FS drivers. As far as I know, the only FS's that are supported universally within Linux, Mac, and Windows, are FAT and NTFS. And NTFS can be made to have crude support for permissions through security descriptors. Although, the Dream SPL, the Recovery images, and most of Android only uses FAT32.
Click to expand...
Click to collapse
Ya, I meant more from the standpoint of a rogue app. Since FAT32 has no permissions, what would prevent such an app from modifying the stored image file to, say, change a trusted app with superuser permissions to some new code of its own making to, for example, watch for credit card numbers and send them back to the person who made the original rogue app? I'm always hesitant with any ideas that suggest storing an image file on the sdcard for appstosd for this reason.

Forget it, it's useless.
An overlay filesystem prevents you from enabling USB storage.

If you want to play around with FUSE on Android, here's a repository for my port of libfuse..
http://github.com/cyanogen/android_external_fuse

Hi,
I have an idea. I used symbian S60 of Nokia, Symbian can install app to sdcard. I see that when I mount sdcard to PC, my phone immediately hold all activations of all applications on my phone. And they have a PC sync software that help us access sdcard but not mount sdcard (like that we copy file from computer to sdcard via debug mode on android).
I think we should find out how symbian can do it and we will use their way .
I'm not a developer, I'm just an user.

I talked to a few people about this, and some deep kernel voodoo is going to be needed for this to really happen without partitioning.
Another idea is to forge ahead with this, and ditch the "unmount fs for usb storage" and use RNDIS + Samba or something like that instead to access files on SD. I kind of like this idea.

[Q] ext2/ext3/ex4 and lynux-swap with incubus RLS14/15

Hi, I have been testing several NAND set-ups
-system to NAND and data to resized data,img
-system to resized system.img and data to resized data.img
-system to NAND and data to ext2
...
I am currently using NAND and data to ext2 partition, so all apps are installed there. After lots of reading I can not understand posts that are obsolete and posts that are not so I would appreciate if someone can explain me the basics and optimal setup for incubus superfroyo RLS14/15 roms:
I have now 4 partitions:
1) fat32,
2) 1,5G ext2 (app, app-private etc),
3) 1,5G ext2 (lost+found folder) and
4) 96M lynux-swap.
- How can I turn ext2 to ext3? do I need to do so? I have also read about ext4. Upgrading to ext3/ext4 applies to my 2) partition, to my 3) partition or to both? will I loose data stored? "there is no ext3/ext4 fs because there isn't the support on the compiled kernel. Ext3/ex4 are journal file system so are bad for limited writing device like nand or sd and thre is no good rease to use it" answered by l1qd1.
- How can I check if my lynux-swap is enabled and used by the rom?
- Do partitions 2) and 3) really need to be <1,5 G? why so?
- Can I take apk from apps folder in partition 2) in case I want to keep them for future fresh installs?
- What is lynux-swap optimal size? a 1,5G partition would be worse than 96 M? Why?
- Which tools are you using with incubus roms to manage installaion location and similar functionalities?
Another big mark I have is about deoxeding installed aps (not belonging to the ROM). Should I do it? why? how?
Pretty basic stuff I guess for experienced users & programmers, but I am new to lynux and android. Have read lots of things but it is difficult to learn by myself what would work with our Polaris and incubus ROMS and dzo/kk kernels,

Stick to ext2 man, you won't turn your old engine to 240mph on a Ford Model T.
You're not getting some major boost if the hardware dosn't use it, so I'd stick with ext2.
And you also want Super FroYo on your ext2 partition, yes? I'm here to help.

- How can I check if my lynux-swap is enabled and used by the rom?
adb shell
Code:
free
if on swap you have 0 it isn't enabled.
Code:
swapon /dev/device
to enable it or i suggest you tu use compcache.
- Do partitions 2) and 3) really need to be <1,5 G? No
- Can I take apk from apps folder in partition 2) in case I want to keep them for future fresh installs? Yes but not all apk support this
- What is lynux-swap optimal size? a 1,5G partition would be worse than 96 M? Why? Too Big = more sparse but it's a problem only for real disk so you can create a swap big as you want but it's unuseful (100 M is enaught)
i suggest you to switch to my kernels because are more updated and kk move to another device.

Yah l1q1d I love ur kernels. Eager for more !

l1q1d, I am testing your kernel as you advised me. I could apply the wifi patch without problems using your androiupdate18-09-10, as you told me some time ago they are both 2.6.25.
However I do not manage to turn linux-swap on: that's what I do: I install ubuntu, I use G-Parted to create partitions on the sd card. When I do so, the third partition is always shown as linux-swap in G-Parted but when I test adb shell -> parted -> print, it always shows as ext2. I can not dismount and can not swap on it.
The other way round, if I deleted all partitions with G-Parted and create the partitions with mkpartsfs or if I use mkfs to change the third partition from ext2 to linux-swap, then sometimes I can swap on it (directly on adb shell or with G-Parted) but when I unplug the device will tell me the card is empty and will not be able to mount it, access it from ubuntu or from the device. If I reboot then, the system will hang until I install both data and sysetm to nand. This happens no matter if I activate or not from the polaris or even if I switch on or off usb mode. The card at that point is useless. I guess the partition table or some other basic file is messed.
Could you please explain a little more how can I come with a 13.5G fat32 partition, a 2,4 ext2 partiions and a 0,1 lynux swap swapped on partition??
I really need to test this set up, since with the standard set-up (system to nand, fat32 for files and ext2 for installed apps I can not install all apps I use since performance drops significantly).
regards

well, ... dumb me, maybe with compcache you are referring to launch a script?
where do I get this compcache.sh? how do I launch it? from adb? from the terminal? from the sd card?...

[Q&A] 2 GB External Data -- Hefe Hook Kernel and others

The Hefe Hook kernel allows you to mount a partition of your microSD as /data, getting 2 GB (or more) for your apps and their data.
Please ask your questions here about installation, use, or general approach.

This is great @jeffsf and can u show me how to re-partition the "real internal" storage? I mean expanding the /system since u put the /data out of it. Thanks man
Sent from my GT-N7000 using Tapatalk 2

daothanhduy1996 said:
This is great @jeffsf can u show me how to re-partition the "real internal" storage? I mean expanding the /system since u put the /data out of it.
Click to expand...
Click to collapse
Yes, you've hit on another of the reasons I've been exploring using the microSD for "live" storage. It's one thing to use potentially slow storage for photos that you basically write once or "app to SD" where you read the APK at boot time, make sure your ODEX in the Davlik cache is good, then generally read from the internal-flash cache. It is another when that memory is being read and written "constantly" when your phone is running.
The good thing is that once /data is not part of the internal flash filesystem, you don't have to worry about one ROM (kernel) thinking it begins at one place and another saying it starts at another. Previously, if one ROM had one /system size and another and a different one, the next partition, /data, would look to be corrupt when you swapped ROMs.
As a warning, not all users have fast microSD cards. Some that say "Class 10" on them really are dogs, especially for small reads/writes. The "Class" ratings are for sustained writes, as you would have with a camera recording video. If your ROM is "external /data" only, or even defaults to that, be prepared for a slew of "Your ROMs sucks. It is so slow." complaints.
You'll also need a way to automate formatting the card. It can be done on the phone, as long as you aren't trying to preserve any data.
As I recall, the layout of the MTD partitions is done in drivers/mtd/onenand/samsung_galaxys4g.h I would be careful not to move the partition boundary for efs, as you'd have to move the data it contains in your updater script. Repeating the warning about not moving the boot and recovery partitions is probably a good idea as well!
Your build tree may need some of these values, or at least think it needs some of these values. For example, device/samsung/aries-common from the CyanogenMod/cm-11.0 (KitKat) branch calls out NAND page sizes, partition sizes, and flash block sizes. I haven't looked in detail at your build tree so I can't comment on how it might handle things differently than the CyanogenMod one.

itzik2sh said:
Hi
I hope I don't ask anything silly, but please let me know if any of my assumptions is wrong :
1. I take FBi's251's AOKP milestone 6 (ICS 4.0.4)
2. 8GB SDCard was formatted to FAT32 (4GB) and EXT3 (4GB) using TWRP kernel
(Beastmode's proton kernel to be exact).
3. I would flash this kernel and it would move apps and their data to the sd-ext
without any special additions.
Thanks. I read the thread, but unfortunately 8GB SD is what i have and I think it should be enough.
Thanks again.
P.S - it's for 2 guys I already sold them my SGS4Gs. I want them to be happy...
Click to expand...
Click to collapse
AOKP should be fine. I haven't tried it recently, but it was the tree in which I did the early Hefe Kernel development.
As I understand it, you have a microSD with
partition 1: 4 GB FAT32
partition 2: 4 GB ext3
So that can work, but will need some tweaking of the init-on-fs.sh script. I intentionally didn't use the second partition as so many scripts gobble that up as sd-ext and do who knows what to it.
My first preference would be to reformat the cards, perhaps:
6 GB FAT32
1 MB ext2/3/4 (Yes, 1 MB, a sliver, choice of ext2, ext3, ext4 up to you)
2 GB ext4
since then the script will work without modification and if they install a third-party script that uses the second partition, it won't corrupt their data.
If you were to keep the formatting the way that it is now, you'd need to edit the mount commands in the script to look something like:
Code:
/system/xbin/busybox umount /data
/system/xbin/busybox mount -t ext3 -o noatime /dev/block/mmcblk0p2 /data
(removing the sd-ext mount)
I'm not sure what your expectations are, but all that the kernel and that script will do is mount a different disk partition on /data -- you need to manually move the data over (or restore from something like Titanium Backup). There may be some trickery in renaming that could be used with TWRP backups to restore from data.yaffs2.win to the new /data partition, but I haven't tried that at all.

Hi Jeff
Thanks for your quick reply, and sorry again for not seeing the Q&A thread.
I think making it :
partition 1: 4 GB FAT32 (sdcard)
partition 2: 2 GB ext3 (sd-ext)
partition 3: 2 GB ext3 (data)
partition 4: 1 MB (spare)
would be better and handle data as well. don't you think ?
Would it be worth doing with a "Team" micro-SD card (class 6 I believe) ?
Thanks.

I haven't tried a Class 6 card, but my gut feeling is that it will be dicey. I didn't "commit" to using /data on microSD until I had tried it for several days using Titanium Backup's ability to move both apps and app data to the external card. I would try that first, especially as the phones in question aren't going to be in your hands (I consider you an expert user, able to manage things outside the UI with ease).
I've attached some testing I did a while ago with Transcend and SanDisk cards. When you look at them, realize that the speed scale changes between them. I have a feeling that the real "performance" on a device is going to be related to relatively small reads and writes, not the ability to stream video to the card. I also don't know much about the Team brand, but I found that even some well-known brands didn't have the performance of the Transcend or SanDisk in the same category.
However you configure your cards, I would definitely recommend a journaling filesystem of some sort. I've had my microSD come loose inside the phone. The journal will at least help to reduce any filesystem corruption should that happen.
You don't need the fourth partition -- I have it there to be able to keep rsync backups for fast ROM swapping.
.

Regarding the apps data, have you tried exploring the Mount2SD script ?
sent from me

I've tried a couple of the scripts out there in the past. Since backing up my data is very important to me, I trust the scripts in Titanium Backup to work well with its backup/restore strategies.
Mounts2SD looks like it has gotten a lot more sophisticated than it was when I tried it in the past. It sounds like something worth trying in its current state. At a quick glance (and not looking at the code), I'd personally make some different choices about features; enabling journaling, and being concerned about why lost+found was filling up (things should only appear there if the file system is found to be corrupt).

Database Info

welcome