So I was using Slax. Great LiveCD/USB linux, extremely customizable, modular, fast, and small, and has the capability of either:
- saving changes to its rootfs onto an AUFS mounted on a non-linux FS (FAT32, NTFS) using posixovl (POSIX Overlay FS) with metadata (permissions, etc.) being held in files
- saving changes to a fixed-size loop mount image.
This got me thinking.
If we could insert all the necessary modules, code, etc. for posixovl into the Android linux, and make a modified a2sd script that takes advantage of posixovl, we could effectively do away with the requirement for crazy partitioning.
It should be simple enough for ROM devs to implement, assuming it's ready and installed:
1. Create folder on main partition if it doesn't exist, something like /sdcard/system/[app, app-private, dalvik-cache, app_s]
2. In the init scripts, before the a2sd stuff, mount /sdcard/system with posixovl on /system/sd
3. Run a2sd as normal, it should automatically just work.
I'll hopefully test this once I get my phone to a stable development/testing stage, and I don't need to make phone calls for a while. Anyone else is welcome to try to implement this idea.
My current test environment:
- HTC Dream (T-mo G1) with the deadly SPL of doom
- Cyanogen Experimental, latest build
- Amon_RA's modded recovery
- Wipe /data, move all existing apps to backup, remove a2sd partition, Backup for Root Users to restore some settings and data
Anyone with ideas or improvements, please let me know.
To be tested:
- Feasibility (can it work?)
- Functionality (does it work?)
- Portability (Can it work on other ROMs and devices like Hero, Pulse, Blur etc.? If so, will likely be moved to XDA's new Android board)
- Stability (Will everything Force Close on boot? Does it run fast enough? Does anything get corrupted over time?)
Links:
- http://sourceforge.net/projects/posixovl - Sourceforge page for posixovl
In desktop linux you can create a file with the touch command, and mount the file to a mountpoint after formatting it to ext4 for example.
Maybe this is the easier way?
I have done this about 5 years ago, but I will try it today and report if it worked.
edit: ok done already:
1. create a file of the desired size, eg: dd if=/dev/zero of=filename bs=filesize count=1
2. use mke2fs to format the file
3. create a mountpoint and mount the file
thats all. Now I have a 128MB file on my PC, mountable and usable like a partition.
Archont said:
In desktop linux you can create a file with the touch command, and mount the file to a mountpoint after formatting it to ext4 for example.
Maybe this is the easier way?
I have done this about 5 years ago, but I will try it today and report if it worked.
Click to expand...
Click to collapse
As far as I know, you can do that on the mobile Android, too, and that does work in theory. This technique involves mounting a loop filesystem, and it too will allow one-partition apps2sd, but it's less flexible, and I would think slower, than the overlay method.
For a 512MB apps image:
Create empty 512MB file
# dd if=/dev/zero of=/sdcard/apps.img bs=1024k count=512
Format it to Ext2
# mke2fs -L Apps2SD /sdcard/apps.img
Unmount existing a2sd
# umount /system/sd
mount new a2sd image
# mount -t ext2 -o loop /sdcard/apps.img /system/sd
Make the usual directories, and a mountpoint for the old a2sd partition
# mkdir /system/sd/app; mkdir /system/sd/app-private; mkdir /system/sd/dalvik-cache; mkdir /system/sd/apps-tmp
Mount the old a2sd partition
# mount -t ext2 /dev/mmcblk0p2 /system/sd/apps-tmp/
Move all files from the old partition to the image file
# mv /system/sd/apps-tmp/* /system/sd/
Unmount and remove the mountpoint, we don't need it anymore
# umount /system/sd/apps-tmp
# rmdir /system/sd/apps-tmp
Finally, you add the following line to the init script where the a2sd auto mount happens, and comment out the old line.
Code:
[...]
mount -t ext2 -o loop /sdcard/apps.img /system/sd
#mount -t ext2 /dev/mmcblk0p2 /system/sd/
[...]
This should do what you described, in theory. I can't say whether it will work or not. I can't tell whether it will or won't screw up your phone, I can't be held responsible if you screw something up or overlook the details. Either of us might have made a typo somewhere; apply common sense before doing anything.
This sounds great! I think this would also be usefull for someone like me, who has a sd card that doesn't want to be partitioned anymore (cross-linked files??). Only thing possible is fat32 or ntfs.
Am I correct with my assumption?
This sounds a lot harder and more complicated then partioning. Your also talking about a lot of work needing to be done just so people can avoid doing a simple thing like partioning a sd card. I would say it would be very difficult alone to get the os to run a virtual mounting service especially since that will take up resources and slow down the phone. There is a reason they only use this technique on live cds is it works but its slow. most of the computers they are running on have 1gig of ram and 2ghz cpu's. i really dont think the g1 can push this.
I do not think that this is great, it definitely is slower than a ext4 partition on a good class 6 microSD card. And it is more vulnerable to data loss since 2 different filesystems including a 20 year old non-journaling fs at the base of this construct are involved.
Another problem that came to my mind: when you mount your SD card as external USB device to a PC, the file containing your apps will no longer be accessible, or Android will make using the phone as external data storage impossible.
Interesting.
There is, however, a major problem: What happens when you unmount the fat partition on the phone in order to connect with a computer using UMS? Answer: everything on the phone will crash and burn since the apps filesystem will suddenly disappear = BAD.
posixovl is a nice find though...
Note that aufs, loopmount linux filesystems, etc., wouldn't be needed with this since posixovl appears to be vfat with posix extensions, so you should be able to just use posixovl directly on the sdcard.
There are several problems with that though... i.e. how reliable is posixovl regarding users tampering with it?
In any case, a prerequisite for use of it would be certain other changes being planned...
You might want to contribute to this thread:
http://forum.xda-developers.com/showthread.php?t=577941
(note: the thread links to a thread at android-platform, the one here has, as expected, gone off on a tangent... just ignore the junk.)
TylTru said:
So I was using Slax. Great LiveCD/USB linux, extremely customizable, modular, fast, and small, and has the capability of either:
- saving changes to its rootfs onto an AUFS mounted on a non-linux FS (FAT32, NTFS) using posixovl (POSIX Overlay FS) with metadata (permissions, etc.) being held in files
- saving changes to a fixed-size loop mount image.
This got me thinking.
If we could insert all the necessary modules, code, etc. for posixovl into the Android linux, and make a modified a2sd script that takes advantage of posixovl, we could effectively do away with the requirement for crazy partitioning.
It should be simple enough for ROM devs to implement, assuming it's ready and installed:
1. Create folder on main partition if it doesn't exist, something like /sdcard/system/[app, app-private, dalvik-cache, app_s]
2. In the init scripts, before the a2sd stuff, mount /sdcard/system with posixovl on /system/sd
3. Run a2sd as normal, it should automatically just work.
I'll hopefully test this once I get my phone to a stable development/testing stage, and I don't need to make phone calls for a while. Anyone else is welcome to try to implement this idea.
My current test environment:
- HTC Dream (T-mo G1) with the deadly SPL of doom
- Cyanogen Experimental, latest build
- Amon_RA's modded recovery
- Wipe /data, move all existing apps to backup, remove a2sd partition, Backup for Root Users to restore some settings and data
Anyone with ideas or improvements, please let me know.
To be tested:
- Feasibility (can it work?)
- Functionality (does it work?)
- Portability (Can it work on other ROMs and devices like Hero, Pulse, Blur etc.? If so, will likely be moved to XDA's new Android board)
- Stability (Will everything Force Close on boot? Does it run fast enough? Does anything get corrupted over time?)
Links:
- http://sourceforge.net/projects/posixovl - Sourceforge page for posixovl
Click to expand...
Click to collapse
lbcoder said:
Interesting.
There is, however, a major problem: What happens when you unmount the fat partition on the phone in order to connect with a computer using UMS? Answer: everything on the phone will crash and burn since the apps filesystem will suddenly disappear = BAD.
Click to expand...
Click to collapse
I kinda overlooked that point. Oops.
Though this same problem does exist on regular apps2sd when you remove the card without dismounting it, killing all apps and their processes, and freezing Dalvik's method of autostarting some apps.
I do tend to swap cards every now and then, but only after a reboot. Dalvik re-enumerates and caches dex, which makes for a slow boot, but it seems to just work in most cases that the apk install doesn't drop the app's functional payload (helper Linux/shell utils, libraries, NDK .so's) in /data/data (like some emulators, the Android Scripting Environment)
lbcoder said:
There are several problems with that though... i.e. how reliable is posixovl regarding users tampering with it?
Click to expand...
Click to collapse
As far as I know, the metadata files are marked as hidden and system files, and begin with a '.'. And I haven't tried this, but I think modifying the actual files under Windows has no negative effects, but moving, deleting, or copying files would likely be a no-no.
Also, I don't think there's a 'fsck' for posixovl, meaning that if any metadata files were screwed with the wrong way, the entire overlay FS would be trashed.
lbcoder said:
In any case, a prerequisite for use of it would be certain other changes being planned...
You might want to contribute to this thread:
http://forum.xda-developers.com/showthread.php?t=577941
(note: the thread links to a thread at android-platform, the one here has, as expected, gone off on a tangent... just ignore the junk.)
Click to expand...
Click to collapse
I checked that out. It was actually a small inspiration for what I was thinking of.
In any case, Android's package management system needs an overhaul. The package storage needs to be de-Linuxified, as all it is is a bunch of .apk files and .dex/.odex files, the UIDs of apps are in the AndroidManifest.xml, right?
In a somewhat unrelated note, app data needs to be moved to a specified folder structure on the sdcard. My card is full of folders in the root directory with random names.
If I'm understanding you correctly, you're talking about storing an image file on the normal SD card partition(which has to be FAT32 as far as I've seen) and then mounting it, correct? This idea has been talked about at length before on at least 3 separate occasions(2 of which were on this very forum) and found to be a bad idea due primarily to massive security risks since FAT32 has no permissions.
Also, I believe cyanogen ended up dumping unionfs/aufs due to rampant memory issues.
If you are talking about mounting an image from the FAT32 partition, please don't endorse this. We don't want to be throwing in security bugs into android, especially ones such as this which can't be plugged up.
As a modification to what I said: If you're suggesting doing this(or something similar) on a separate filesystem, after that project to change the AOSP to support one with permissions is finished, then I'm in full support.
If you want to go for a single partition on the sd card, why don't you just make the entire card use ext4? Your linux desktop reads it anyway, it uses journaling and so on, I guess it would be faster compared to fat32 and it is definitely safer to use.
And i guess it is not too complicated to mount this partition and use it for pictures, music and so on.
I have not tried this (yet) and I go to bed in 20 minutes, but maybe I will start testing something in that direction tomorrow.
[email protected] said:
If I'm understanding you correctly, you're talking about storing an image file on the normal SD card partition(which has to be FAT32 as far as I've seen) and then mounting it, correct? This idea has been talked about at length before on at least 3 separate occasions(2 of which were on this very forum) and found to be a bad idea due primarily to massive security risks since FAT32 has no permissions.
Also, I believe cyanogen ended up dumping unionfs/aufs due to rampant memory issues.
If you are talking about mounting an image from the FAT32 partition, please don't endorse this. We don't want to be throwing in security bugs into android, especially ones such as this which can't be plugged up.
As a modification to what I said: If you're suggesting doing this(or something similar) on a separate filesystem, after that project to change the AOSP to support one with permissions is finished, then I'm in full support.
Click to expand...
Click to collapse
UNIX permissions don't do anything in the way of "security" unless you have no access to the actual storage device from another computer (as is the case with the unrooted Dream's internal memory), or unless encryption is used. The posixovl driver OVERLAYS Unix permissions over Fat32 filesystems. But even still, with Unix permissions, nothing's stopping someone else from mounting the Ext2 partition and using chown and chmod.
And the image file on the SD card's Fat32 partition is a complete Ext2 partition complete with Permissions. Nothing is lost.
Archont said:
If you want to go for a single partition on the sd card, why don't you just make the entire card use ext4? Your linux desktop reads it anyway, it uses journaling and so on, I guess it would be faster compared to fat32 and it is definitely safer to use.
And i guess it is not too complicated to mount this partition and use it for pictures, music and so on.
I have not tried this (yet) and I go to bed in 20 minutes, but maybe I will start testing something in that direction tomorrow.
Click to expand...
Click to collapse
We'd just have to find the part in the Android that mounts /sdcard/, and change 'vfat' to 'ext2'. The only reason I wouldn't do this, is because it would immediately make it incompatible with Windows and Mac's default FS drivers. As far as I know, the only FS's that are supported universally within Linux, Mac, and Windows, are FAT and NTFS. And NTFS can be made to have crude support for permissions through security descriptors. Although, the Dream SPL, the Recovery images, and most of Android only uses FAT32.
This is discussed in android-platform Group :
http://groups.google.com/group/andr...read/thread/bf0709c157451cd9/f6aee1830c84620f
The goal is to be able to integrate this in android.
And not having to partition the SDCard is one of the requirements so far...
Unix permissions are not stored using fat or vfat, and ntfs is not really supported in desktop linux and i guess it cannot be used in android linux.
I would not use windows anyway so this is no problem to me, and there are drivers around to mount ext systems in windows. As Mac OS is based on unix there will be a solution for this too.
Access usind adb push and pull, via ftp and so on is not touched by using ext4 on the entire sd card I guess.
And if you don't go the easy way using gparted on a live cd or usb device to create 2 partitions, you will have to live with some disadvantages anyway.
Finally I want to say that my ideas are far from being perfect or usable at all, I see this thread as a kind of brainstorming.
im not as linux or android savvy as probably any of you but before the current method of creating a swap partition became the "standard", people used a swap file on the sdcard and linked that. seems similar to what you are suggesting here.
ofcourse when mounting the fat partition elsewhere (ums in windows for example) that swap file could no longer be used within android. i dont see a way to get passed the same issue, but worse here, due to android not having crucial apps when the fat partition is mounted.
then again, i am pretty much over my head in this conversation and could be over looking something...
I'm kind of fascinated by the FUSE + posixovl method of doing this. In the long run I have a feeling that it's going to perform like ****, but I think it's worth testing.
I managed to get both libfuse and mount.posixovl built and running on Android.
posix-overlay(/sdcard/fuse) on /sdcard/fuse type fuse.posixovl (rw,nosuid,user_id=0,group_id=0,default_permissions)
Giving this a little testing now, it definitely works.
Code:
/sdcard/fuse # ls -l
drwxr-xr-x 2 1000 1000 4096 Nov 5 17:17 test
TylTru said:
UNIX permissions don't do anything in the way of "security" unless you have no access to the actual storage device from another computer (as is the case with the unrooted Dream's internal memory), or unless encryption is used. The posixovl driver OVERLAYS Unix permissions over Fat32 filesystems. But even still, with Unix permissions, nothing's stopping someone else from mounting the Ext2 partition and using chown and chmod.
And the image file on the SD card's Fat32 partition is a complete Ext2 partition complete with Permissions. Nothing is lost.
We'd just have to find the part in the Android that mounts /sdcard/, and change 'vfat' to 'ext2'. The only reason I wouldn't do this, is because it would immediately make it incompatible with Windows and Mac's default FS drivers. As far as I know, the only FS's that are supported universally within Linux, Mac, and Windows, are FAT and NTFS. And NTFS can be made to have crude support for permissions through security descriptors. Although, the Dream SPL, the Recovery images, and most of Android only uses FAT32.
Click to expand...
Click to collapse
Ya, I meant more from the standpoint of a rogue app. Since FAT32 has no permissions, what would prevent such an app from modifying the stored image file to, say, change a trusted app with superuser permissions to some new code of its own making to, for example, watch for credit card numbers and send them back to the person who made the original rogue app? I'm always hesitant with any ideas that suggest storing an image file on the sdcard for appstosd for this reason.
Forget it, it's useless.
An overlay filesystem prevents you from enabling USB storage.
If you want to play around with FUSE on Android, here's a repository for my port of libfuse..
http://github.com/cyanogen/android_external_fuse
Hi,
I have an idea. I used symbian S60 of Nokia, Symbian can install app to sdcard. I see that when I mount sdcard to PC, my phone immediately hold all activations of all applications on my phone. And they have a PC sync software that help us access sdcard but not mount sdcard (like that we copy file from computer to sdcard via debug mode on android).
I think we should find out how symbian can do it and we will use their way .
I'm not a developer, I'm just an user.
I talked to a few people about this, and some deep kernel voodoo is going to be needed for this to really happen without partitioning.
Another idea is to forge ahead with this, and ditch the "unmount fs for usb storage" and use RNDIS + Samba or something like that instead to access files on SD. I kind of like this idea.
I just installed a new ROM and the SDCARD and SDCard-ext are in the wrong location. They are listed under the root director ( / ) and /storage instead of under /mnt/sdcard
Does it work? You could create a symlink. If that fixed it, put it in a script on bootup.
post-mortem said:
Does it work? You could create a symlink. If that fixed it, put it in a script on bootup.
Click to expand...
Click to collapse
Does it work is a relative question. (wasn't meant to sound wise-a$$). I can access them, but apps do not see them at all. They are looking for the cards in /mnt therefor not finding them.
I changed them in /system/etc/vold.fstab but now it won't boot. The not booting is not an issue, I can fix that, but I don't know how to fix these incorrect paths/
No one else who's using this ROM is having these problems? I say either reflash, create symlinks, or choose another ROM.
post-mortem said:
No one else who's using this ROM is having these problems? I say either reflash, create symlinks, or choose another ROM.
Click to expand...
Click to collapse
Not sure if anyone else is running it. Haven't been able to find a thread on here for it. How do you do symlinks?
mikemikemikexxx said:
Not sure if anyone else is running it. Haven't been able to find a thread on here for it. How do you do symlinks?
Click to expand...
Click to collapse
ln -s target shortcut
See a bit more here.
Make a script in your /system/etc/init.d directory, so these links will be created on every boot.
post-mortem said:
ln -s target shortcut
See a bit more here.
Make a script in your /system/etc/init.d directory, so these links will be created on every boot.
Click to expand...
Click to collapse
From what I have read you cannot use them on FAT.
Why does editing this file not work? I fail to see how Android could do anything other than read the file, provided permissions are the same.
mikemikemikexxx said:
From what I have read you cannot use them on FAT.
Why does editing this file not work? I fail to see how Android could do anything other than read the file, provided permissions are the same.
Click to expand...
Click to collapse
The internal memory is not FAT; typically, only the 1st partition on the SD card is FAT, everything else should be EXT. So, executing ln -s /storage /sd-ext should work (or wherever it's currently being mounted).
post-mortem said:
The internal memory is not FAT; typically, only the 1st partition on the SD card is FAT, everything else should be EXT. So, executing ln -s /storage /sd-ext should work (or wherever it's currently being mounted).
Click to expand...
Click to collapse
I meant the card but ln is not working.
One more thing I am going to try is an ADB pull and push. If that doesn't work I really need some community support.
bump to the front.
Are my questions that complex or that dumb that I get so few replays? /bump
Sent from my XT875 using xda app-developers app
Is there a good reason that you don't want to use another ROM?
Mainly speed and compatibility. I need the offline commands only available in JB. Stock ROM is dead slow and is only ICS. Other custom ROMs were also on the slow side and/or battery hogs.
Sent from my XT875 using xda app-developers app
So, is this right?
FAT partition is currently being mounted as /storage
EXT partition is currently being mounted as /mnt/sdcard
post-mortem said:
So, is this right?
FAT partition is currently being mounted as /storage
EXT partition is currently being mounted as /mnt/sdcard
Click to expand...
Click to collapse
Well the way my phone is laid out is
System (I assume too large so they split it in two which resulted in the below SD-Card)
SD-Card
SD-Card EXT (The actual SD Card) - FAT32
I assume the System and SD-Card are standard Android partition format.
Do you use a file manager on the phone, such as ES File Explorer? Are you familiar with using adb?
Edit: Basically, I'm trying to figure out the exact path of the mounted partitions, and where they're supposed to be mounted.
post-mortem said:
Do you use a file manager on the phone, such as ES File Explorer? Are you familiar with using adb?
Edit: Basically, I'm trying to figure out the exact path of the mounted partitions, and where they're supposed to be mounted.
Click to expand...
Click to collapse
They are 100% supposed to be mounted under: /mnt/sdcard and /mnt/sdcard-ext
Right now they are mounting to: /storage/sdcard and /storage/sdcard-ext
There is also an entry in under root ( /sdcard and /sdcard-ext --- For these two I am booting into stock ROM to see if they are suppose to be directly in root as well, will post up when I know)
Check the scripts in your init.d to see if one of them is mounting the sd partitions under /storage.
I suppose you could also try booting without the sd card in place, and see if those /storage directories still exist. Not sure how much that will tell you, though.
post-mortem said:
Check the scripts in your init.d to see if one of them is mounting the sd partitions under /storage.
I suppose you could also try booting without the sd card in place, and see if those /storage directories still exist. Not sure how much that will tell you, though.
Click to expand...
Click to collapse
The init.d for sure says to load sdcards to /storage . (on this particular ROM only)
I will try without the sdcard as you suggested and let you know. Battery died on me, phone won't boot even when plugged in until at least 10%.
Which script is mounting the sd card to /storage? Is it part of the ROM, or is it something which the dev included in this ROM? I ask in case the ROM expects the sd card to be mounted there.
Regardless, try changing the script to link to /sdcard.
I was having some trouble after removing the internal SD using the method explained at Bruno's ROM FAQ found at:
http://forum.xda-developers.com/showthread.php?p=30786931#post30786931
The problem I found is similar to that described by other users at
http://forum.xda-developers.com/showthread.php?p=40634583
I wanted to post there but I'm a newbie here, so I'm not allowed
I'll describe how I got to this point, I flashed latest BM JB version, then I repartitioned to remove the internal SD and recovered /data from nandroid backup like the FAQ says.
At that point everything looked fine except that the SD was not mounted. Looked at /etc/vold.fstab and it was like this:
dev_mount sdcard /storage/sdcard0 [email protected] /devices/platform/goldfish_mmc.0 /devices/platform/mtk-sd.0/mmc_host
dev_mount sdcard2 /storage/sdcard1 auto /devices/platform/goldfish_mmc.1 /devices/platform/mtk-sd.1/mmc_host
Looks like as the first one was failing, the second one was not being mounted.
What I did was several tests but I ended up commenting the first of those lines and everything looks fine except for the legacy /sdcard link which still points to /storage/sdcard0 which is not mounted. Other than that first tests show everything fine, photos are found on the external card, storage shows it as a external card and allows it to be umounted for extraction, ...
Hope this helps the guys in trouble even tough to achieve that it would be good to have this posted on the original thread.
If more info is needed to help debug this please ask.
Thanks to Bruno for these great ROMs
The problem with my solution
The problem I find with my solution is that there are a lot of apps that don't find the sdcard.
If /sdcard is not ok these apps tell you they don't find the sd or that the sd is full or whatever.
The result is that a lot of apps malfunction
I've read new posts on Bruno's development thread and there is no new reply on this.
Please, help!
Still no solution for this
As my old solution ended up giving a lot of problems with programs needing the legacy /sdcard link and I still don't have a real solution for this problem I ended up having only this line on the vold.fstab file:
dev_mount sdcard /storage/sdcard0 auto /devices/platform/goldfish_mmc.1 /devices/platform/mtk-sd.1/mmc_host
This makes the device identify the external sd as an internal one, but at least the /sdcard points to it and thus programs seem to work ok.
This had been my first solution but then I thought that having it as external one would be better but it resulted that it wasn't
I really hope somebody knowledgeable on this gives me a hand.
Regards.
Problems never end
I still hadn't found a clean solution for this and now, after powering on the phone to change battery, when we where booting it up with an almost empty battery that was to be recharged we got an error similar to the one shown here:
http://forum.xda-developers.com/showthread.php?t=2068332
saying Encryption unsuccessfull and after rebooting with more battery and a cable plugged the problem seemed to remain (the problem, not the error). I mean that after reboot the partition where apps data is stored seemed to be lost, or at least the apps didn't get to their data and they were all unlogged and no signs of their setups, ...
slight esnnype
Well, in the end it was not so bad.
I decided after this last problem that I could install version 1.2 which is more recent than the one I had installed, so I went to CWM and I formatted cache and data partitions and then installed version 1.2 which indeed recognized (or so it said) that I had the partitions extended and thus no internal SD and it solved things on vold, so no the external flash is seen as the only flash on the system and everything works ok out of the box.
End of the story? We'll see.