Related
Is there anyone out there who has simply created a image of an ext2 filesystem rather than physically partitioning the SD card?
I was thinking of creating an appstorage.img file and mounting via the loopback device in order to store apps on the SD card. This would allow me to keep the whole card as FAT32.
So on a linux computer, I'd do:
dd if=/dev/zero of=appstorage.img seek=500999999 bs=1 count=1
mke2fs -F appstorage.img
Then I'd copy the appstorage.img onto the SD card.
Finally I'd have to mount the new filesystem image on the phone....so using the terminal emulator app on the phone itself, I'd do something like:
mkdir /appstorage
mount -o loop appstorage.img /appstorage
mv /data/app/* /appstorage
ln -s /appstorage /data/app/
Not sure if I'd need to load an ext2 module first. If so, I'd want to insert that on bootup of the phone. Would I insert that in one of the rc.init or rc.local files to be run at boot?
Same for the mount command?
The problem is that at boot time, /data/app needs to be available the same time the ext2.ko module of a non-rooted phone becomes available.
JesusFreke added ext2.ko support into somewhere before that point. (I just checked, it's in the system.img flash file; found it in JFs build environment)
In order to use your solution, which many people including me have contemplated, you need to have loop.ko available at that very point in time. That means building your own system.img image file and flashing it with fastboot.
Hope this helps!
Thanks...it does help, although that probably puts it beyond my capabilities. I don't think I'm up to building a custom system.img....maybe it isn't as difficult as I think.
This is similar to what LucidREM did, building a custom mod to the JF build?
Wonder if this would be included as part of the next official Android release from Google/TMobile? Obviously they can't/won't want to partition people's SD cards to expand app storage....seems the filesystem image is the only way to go.
Autarkis said:
The problem is that at boot time, /data/app needs to be available the same time the ext2.ko module of a non-rooted phone becomes available.
JesusFreke added ext2.ko support into somewhere before that point. (I just checked, it's in the system.img flash file; found it in JFs build environment)
In order to use your solution, which many people including me have contemplated, you need to have loop.ko available at that very point in time. That means building your own system.img image file and flashing it with fastboot.
Click to expand...
Click to collapse
Hmm...wait a second here....maybe I don't fully understand.
But looking at the modified init.rc from a thread about moving apps to the SD card, it appears that it is mounting files via loopback and then later directly doing the insmod of the ext2.ko module prior to mounting the ext2 partition on the SD card. Take a look:
Code:
on system
mount cramfs [email protected]/system/modules/modules.cramfs /system/modules ro
mount cramfs [email protected]/system/xbin/xbin.cramfs /system/xbin ro
insmod /system/modules/2.6.25-01843-gfea26b0/kernel/fs/ext2/ext2.ko
mknod /dev/mmcblk0p2 b 179 2 1000
mount ext2 /dev/mmcblk0p2 /system/sd noatime nodiratime
Am I missing something? Are the first two lines utilizing the loopback device already, meaning it would have been loaded previously?
bump. (very curious about this thread)
A bump from me too - last night I had to format my 8Gb SD card back to Fat32 and put myself back to the standard JF RC9 build - my G1 stopped reading my SD card's Fat32 partition and no matter how many times I formatted it or re-created the partition, nothing would work.
Only after FULLY formatting and loosing my my Ext2 partition on it did it start reading it again, causing my phone to fail to boot until I put the standard JF build back on and do a full factory reset.
This is a pretty fundamental flaw of the phone - is there really a technical reason why app's can't be run off a Fat32 SD card?
I am curious to this topic as well...
wont u need to edit a file? so it would auto remount if you reboot the phone?
For those that have installed the Debian distro on their phone, you are doing almost the exact same thing. You create a debian.img file and format it as an ext2 filesystem.
Then you mount it via the loopback device as an ext2 filesystem. But that is after your phone is already booted up in Android and running.
So the question is when the loopback module for the kernel gets loaded. From Autarkis post, he indicates the loading of the loop.ko module is a problem and would require a custom boot image.
But it looks to me like the loopback module is already loaded when init.rc is called...so I'm confused as to why this won't work.
I briefly tried it on the 1GB card that came with the phone, but gave up after a few minutes. Didn't seem to be working. Dunno...maybe I'll give it another go. But I have to believe that someone else has already tried this rather than go through the pain of partitioning.
Wouldn't you then have the same disadvantage as debian.img on FAT? i.e. that you couldn't unmount the FAT (on connection to a PC) as the ext2 partition image was still mounted.
I was thinking about doing the opposite; moving my debian.img over to within the (ext2) app partition to get around this very issue, I can't really work out why you are trying to introduce this problem!
digitalspaghetti said:
A bump from me too - last night I had to format my 8Gb SD card back to Fat32 and put myself back to the standard JF RC9 build - my G1 stopped reading my SD card's Fat32 partition and no matter how many times I formatted it or re-created the partition, nothing would work.
Only after FULLY formatting and loosing my my Ext2 partition on it did it start reading it again, causing my phone to fail to boot until I put the standard JF build back on and do a full factory reset.
This is a pretty fundamental flaw of the phone - is there really a technical reason why app's can't be run off a Fat32 SD card?
Click to expand...
Click to collapse
I think there's a couple reasons the phone can't install apps on a fat32 sd card. The biggest one is that you can't make symlinks to/from a FAT32 file system, meaning even if you were google and wanted to make it possible, it'd likely have to be an all or nothing deal and they didn't want to make the phone unusable/crash without an SD. For the average hacker here, it'd just be too difficult to make the operating system read apps from FAT32.
I was able to install and run apps from the sd card (ext2), but somehow I still uninstall some apps that i don't use frequently. Is there anyway I can remove the apps from the sd card that i uninstalled? I know the fact that they don't occupy that much spaces, but still I'm thirsty for the knowledge on that issue
uuummm you can go to your terminal and
1- su
2- cd /system/sd/app or app-private
3- ls
4- rm "filename you wanna remove" without double quotes
5- Thats it !!!
samysam05 said:
uuummm you can go to your terminal and
1- su
2- cd /system/sd/app or app-private
3- ls
4- rm "filename you wanna remove" without double quotes
5- Thats it !!!
Click to expand...
Click to collapse
and because of the "ln -s" command you could also cd /data/data/app .. it all points to the same place now
issues partitioning SD card for moving apps
I am having ALOT of trouble trying to format my 4GB SD card to the fat32/ext2 partitions.
I have tried using partition manager 9, setting the fat32 (2.8GB) as a primary, and also setting the SECOND partition ext2 as a primary (1GB). I also tried using gparted and did the same EXACT partition, size and settings in all the different programs. I also tried a mixture of FAT16/FAT32 and ext2 for all the settings. I then tried using ext2fsx on my mac using the same exact settings as before. No matter what, it doesn't mount /system/sd, it always is mounted to /system/sdcard.
(the below are estimated because I was entering all commands in the terminal on my G1)
I'm suppose to see something similar to the following:
dev/mmcblk0p2 1.2G 9.1M 1.1G 1% /system/sd
I always get something to similar to:
dev/mmcblk0p2 1.2G 9.1M 1.1G 1% /system/sdcard
I always see the fat32 partition but never the ext2. I know that alot of people on this forum seem to be able to do this without issue. I unfortunately have been working it on all day.
Can someone PLEASE help me out? Thanks!!!
have you tried moving the apps to sd? see if you actually move all your apps and look at that line again to see if there is an adjustment in size
Which directory does your mountd.conf and init.rc files specify in it if you open them up? Does it say /system/sd or system/sdcard? I believe tempo had his saying system/sdcard and used different files than the ones in the post by dwang that does basically the same thing. I recall one of them using /system/sdcard in their post and the other doing just /system/sd. Use the same file from the same post and don't mix the 2 up because they are not the same.
I used dwang's instructions and files and got my 2GB card partitioned and working. I have had a problem getting my 8gb card working for some reason but that is another story. It seems my 8gb isn't readable or writeable after I partition the ext2 section although the fat32 is ok. I'm familiar with linux but don't know what I need to do to make that partition writeable when I can't get it to show up at all except in gparted and partition manager under windows. I tried busybox df -h without it showing so no luck there...
Hopefully what I metioned helps you and maybe somebody has an idea why mine didn't work either although I wasn't very specific.
I tried both files and mounting on both /system/sd and /sd neither of which worked . I also tried various programs to partition the sd card. I have no other ideas...
Hi,
This is how it should look.
I use an adb shell through the computer (windows xp).
The drive with 1.5g available is the sd card drive.
Non-market apps on SD card issue
Okay... Getting this to work has sucked horribly for me and after rooting through the other forums that would apply to my issue i've found no solution. I have RC33 JFv1.42 LucidREM mod, with /app, /app-private, and /data all copied to the EXT2 partition of my SD card. I am trying to install the AndroidVNC viewer using the AppsInstaller app, I have AndroidVNC saved onto the root of the FAT32 partition of my SD card. The Appsinstaller sees it but when I go to install the .apk itself it's giving me the following error:
"androidVNC could not be viewed. Free up some space on your phone and try again." Giving me a 'cancel,' and 'manage applications' button under.
The error doesn't make any sense because when I login to my terminal as root and # busybox df -h, the only parts of my system that are full are the usual suspects:
/system
/system/modules
/system/xbin
all of which are full, but they always have been for me, so I'm not sure if the Appsinstaller is trying to use one of those blocks to extract to temporarily or something like that. what am i missing here!?!?
bump
Partial App migration to SD card...possible?
Has anyone tried to move only *some* of the apps to the SD card?
I'm stuck with a class 4 mSD card, so as I understand it, moving the apps en masse to the SD card isn't a recommended option.
But I was wondering if perhaps I might move *some* apps to the SD card...maybe just ones that I don't use often, or aren't particularly complicated, or maybe just ones that seem to run perfectly fine from the card.
Some of the "high performance" apps, or ones with a lot of I/O dependency I'd leave on the phone internal storage.
As it stands right now, I'm at total app saturation. I've uninstalled infrequently used apps and have them backed up for ready restoration anytime/anyplace. I've moved every cache possible. Yet even at 13MB free, the phone still complains I don't have enough room to install the latest build of AndNav2.
Should I even attempt a partial migration? I have no paid apps (and apparently couldn't install any even if I wanted to) so I'd just be moving the *.apk file and maybe the *.odex files? Is that all I need to move and then symlink to?
Bump for day crew.....
I'm using this setup. I use my own bash script to move apps:
Code:
#!/bin/bash
# mvappndata.sh 1.1 by Autarkis
# GPL OpenSource
adb remount
adb shell cp -rp /data/app/$1.apk /system/sd/app/$1.apk
adb shell rm /data/app/$1.apk
adb shell ln -s /system/sd/app/$1.apk /data/app/$1.apk
adb shell cp -rp /data/data/$1 /system/sd/data/$1
adb shell rm -r /data/data/$1
adb shell ln -s /system/sd/data/$1 /data/data/$1
You call it with the application's name without the .apk suffix.
Code:
mvappndata.sh com.mobisystems.msdict.embedded.wireless.pons.ssd
Known Bugs:
* apps moved this way don't show as Installed or Updated in Market, but as Free
* sometimes, when you are running out of memory, it won't be possible to install new apps from market - the downloading process will hang right at the beginning. Making even more room does the trick
I leave all apps that are being started at boot time or once started should run as a service in the internal memory. I move all the big apps (like dictionaries) to the SD. It mostly works, but since many apps are buggy themselves it's difficult to tell where a certain Force Close is coming from.
Cheers from Switzerland,
Autarkis
Ah...precisely what I was looking to do. Thanks....that helps a lot.
I'm guessing I can modify the script to run directly on the phone....it appears you have constructed to move apps while connected to a PC via the USB cable. Is this correct?
f4phantomii said:
Ah...precisely what I was looking to do. Thanks....that helps a lot.
I'm guessing I can modify the script to run directly on the phone....it appears you have constructed to move apps while connected to a PC via the USB cable. Is this correct?
Click to expand...
Click to collapse
Yes on both counts. I also use ls -laS to display installed apps sorted by size.
looping on android, home button & silent not working on 1.42
Hi,
Had a weird experience the other day. Phone was fine (rooted RC33/JF1.41). Got some AK Notepad reminders but was busy with something so I ignored them (but heard the alert go off). A few minutes later, the phone spontaneously rebooted and from that point on just looped on the android screen.
I figured I was due for an upgrade, so I popped out the sdcard and upgraded to JF1.42. The phone did come back to life, but at that point the home button won't work (wasn't a hardware issue as it worked to get me to the home+power loader) and when I held down power to shut down, I got only the "power off phone" option but no "silent mode" option. Very strange. This was obviously no good, so I went back to 1.41.
Long story short, the phone still looped, so I took this as a sign that a wipe was unavoidable (I didn't have a recent nandroid backup), but since I had WAY too many apps I never used (~200) and my phone was getting unstable (things were constantly force-quitting), I figured I may as well do some spring cleaning and start fresh.
I learned my lesson and finally got the important apps reinstalled and the phone reconfigured the way I want it -- and then did a nandroid backup so I can avoid this in the future.
But what the heck happened that made the buttons not work? That one freaked me out a bit...
And of course I still wonder what caused me to loop in the first place, totally randomly and spontaneously. Are there specific things that can cause that and if it happens again, is there an easy way to fix it?
Thanks
Gah! This is driving me nuts! I absolutely can't make this work.
I can transfer the whole /data/app and /data/data directories to /system/sd, but as I have a class 4 SD card, I get tons of force closes on apps.
So then I try just moving single apps. For example, moving the OI Flashlight app to the sd card.
Code:
# busybox -a /data/app/org.openintents.flashlight.apk /system/sd/app
# rm -r /data/app/org.openintents.flashlight.apk
# ln -s /system/sd/app/org.openintents.flashlight.apk /data/app/org.openintents.flashlight.apk
I do the same thing with the appropriate data directory for the app. I check and the files are moved. The symlinks are present.
But when I try to run the app, it just tells me that the application is not installed on my phone!
Grr! Then if I delete the symlinks and cp the files back to their original locations, same result. I end up having to re-install from backup.
Very frustrating. Any advice?
I don't know what busybox -a does.
It probably shoud say "cp -rp" instead.
It's important that you preserve access rights.
It's also very important that /system/sd is an ext2 partition. vfat won't do.
My script also assumes that as a preparation, /system/sd/app and /system/sd/data have been created. Like that:
Code:
mkdir /system/sd/app
mkdir /system/sd/data
I assumed that was self-explanatory, but it probably wasn't.
unionfs works for this:
have cake and eat it micro-howto:
(Use a recent jf dist)
/system/bin/insmod /system/modules/2.6.25-01843-gfea26b0/kernel/fs/ext2/ext2.ko /system/modules/2.6.25-01843-gfea26b0/kernel/fs/unionfs/unionfs.ko
(partition sd card as described in the aps-on-sd howto)
(mount to /data/local/ext for this example)
mkdir /data/local/ext/app
busybox mount -v -t unionfs -o dirs=/data/local/ext/app=rw:/data/app=ro unionfs /data/app
This has been tested; downloaded apps go to sd card but the system sees apps in /data as well.
The only issue with this is you can't umount the unionfs mount. In theory, you can have writes go to the real /data/app with:
busybox mount -o remount,dirs=/data/local/ext/app=ro:/data/app=rw /data/app
But I have not tested this.
losing storage after moving cache and apps.
Hi all,
I have moved apps to Sd Card.. I have moved cache to sd card and when all is said and done I had 47mb of 74mb free.. YES... but wait..
Now in using my phone I keep losing free space I am now at 45 of 74mb free. How can I found out what is taking my free space up .?
Please help. Thanks in advance for your time.
I was trying to install my apps to my sd card today and now my phone wont reboot. This is what went down. I partitioned my 4g micro sd card, or so i thought. I used Paragon and it gave me an error after reboot;
invalid partition data- partition control blocks information incompatible"
So I restarted my computer. I checked on my phone and it showed i only had 3 gigs available. The partition i made was for 1, so i figured what the hell. It did work. I checked in Paragon and it showed the two partitions. Anyway, i proceeded to input the prompts in adb and it connected, but the second cmd wouldnt work $ mkdir /backup. I went ahead and followed the rest of the commands. At $ adb shell reboot the phone of course rebooted.
I didnt finish the last two commands but accidently restarted my phone. Now it hangs at the G1 screen. rooted g1 by the way.
I have wiped the data and cache, but of course that will do nothing in this case. So it does get me to that screen. I can also hold the camera and power and see my skating androids. But how do i get my phone to boot back up?
At the recovery utility using Alt X to get to console gives me a # prompt.
Help would be greatly appreciated guys, i dont have another phone haha!
Thanks
I'm guessing what you means by "I didnt finish the last two commands but accidently restarted my phone." is that you weren't able to do the "ln -s /system/sd/app /data/app"? If so then at that # prompt, you need to type those very same commands,
Code:
ln -s /system/sd/app /data/app
reboot
hopefully will work after that
Nope. Thats a no go. However, i decided to throw the update back onto my sdcard via sdcard reader. I put the sd card back in and its about done updating back to rc33. I'll report. Seems good so far.
Its all good!
Thanks
you wiped data and that symbolic link is still there?
I wiped my phone and want to redo the app to sd process, but the apps are still on my sd card ext2 partition is there a way I can restore those apps after a wipe or do I have to wipe the ext2 partition and just start over, if so how would I go about doing so.
Joeriginal said:
I wiped my phone and want to redo the app to sd process, but the apps are still on my sd card ext2 partition is there a way I can restore those apps after a wipe or do I have to wipe the ext2 partition and just start over, if so how would I go about doing so.
Click to expand...
Click to collapse
You've done accidentally what I did intentionally.
I moved apps and data over to the SD ext2 partition, then intentionally wiped the phone to start as fresh as possible.
I thought all I needed to do was recreate the symlinks with:
Code:
ln -s /system/sd/app /data/app
ln -s /system/sd/data /data/data
reboot
That should have worked, but didn't. Maybe you'll have better luck than I did.
I was using the LucidREM mod of the JF1.42 firmware which comes with scripts for moving the data and apps.
If you aren't using this version, you'll need repeat the whole manual process of modifying the mount.d and init.rc files so you can see the /system/sd partition on the card.
If you are, you can use the copy.sh undoapps and copy.sh undodata commands and that will put everything back on your phone. At that point it should be exactly as it was before.
If successful, you can try moving everything back either with the Market app, or one of the other scripts floating around on here....
For what it's worth, I never could get things to work when I moved /data/data.....I've had to keep that on my phone and just move the apps. Others seem to have more success moving both without issue.
Joeriginal said:
I wiped my phone and want to redo the app to sd process, but the apps are still on my sd card ext2 partition is there a way I can restore those apps after a wipe or do I have to wipe the ext2 partition and just start over, if so how would I go about doing so.
Click to expand...
Click to collapse
I can give you the code to do it, or you can just use my app that runs the same commands I'd give you. If you use the app just uncheck the checkbox before pressing the move apps to sd button. If you want to go the code route, follow my tutorial starting from the recovery terminal part.
EDIT: oh and 4phantomii: did you just do that code or did you rm -r /data/data and rm -r /data/app first? If you didn't remove the directories first I have no idea what that would do but it seems like it'd make a symlink INSIDE the /data/app and /data/data rather than making them the symlinks.
Hi All,
Here is my first contribution to the community, hope it is useful!
I've rezzed up a construct to produce a new /data on sd. It makes use of a second partition formatted ext2 or ext3 mounted to /system/sd with /system/sd/data then mounted over /data.
I've included a lengthy readme file which should explain things in greater detail.
Read the readme first to get a list of dirs & files this construct adds to the /data, then copy the files as explained and off you go!
currently my /data "volume" is 2GB and all is lovely!
Things to note:
I'm rom JF1.51 ADP1 CRB43, the sdcard is 8gb class6
The sd /data contains several scripts of my own design which if not needed can be deleted. I include my modified bootdeb.sh script as it mounts to /data/local/mnt/debian vs /data/local/mnt (that is a script I modified for my own needs which you may find useful)
Feel free to adjust any of the scripts according to your needs. Just note that if I update the construct, that your changes may be undone if you use it, so back them up in that scenario.
Also, I include my mount.ak.sh script which make use of a loopfs to accomplish the roughly the same thing, but in a single partition environment (intended as proof of concept).
**There is NO booting into recovery and wiping of the phone for this mod!**
Thanks to JesusFreak for the roms and saurik for the Debian info!
Cheers!
Wow! Nice, Going to try it out. Thanks
is that mean after running data2sd the phone's completely running on sd? and so much faster?
Hi,
To followup on xnycen's question, why would we do this? Is the benefit only in providing more space, or is there also a performance increase (or decrease)?
hate to see the sdcard fail with this change
well...
Yes, after the install you are running complete on sd,
but because of the method used, if you reboot, and take out the sdcard before the boot starts, you will be running as you were before the data2sd aka normal; put the sdcard back in, reboot and you are in data2sd mode again.
This effectively gives you two android run modes: "normal" and data2sd. If the SDCore can not be located the Android will boot in normal mode. You can of course try to keep them sync'd, but if you do the data2sd from a clean slate you can effectively have a "safemode" and if you have an issue with some app, go into this "safemode" and do some investigations or adjustments as the SDCore will mount if you put the sdcard in after the system is in "safemode" but the apps and such will not be loaded until you reboot.
The size expansion is the primary benefit of the construct. As far as speed goes, I haven't checked to see if there are any speed gains in moving /data to the sd... Can someone who's interested check that out and let us know?
Darkstrumn said:
Yes, after the install you are running complete on sd,
but because of the method used, if you reboot, and take out the sdcard before the boot starts, you will be running as you were before the data2sd aka normal; put the sdcard back in, reboot and you are in data2sd mode again.
This effectively gives you two android run modes: "normal" and data2sd. If the SDCore can not be located the Android will boot in normal mode. You can of course try to keep them sync'd, but if you do the data2sd from a clean slate you can effectively have a "safemode" and if you have an issue with some app, go into this "safemode" and do some investigations or adjustments as the SDCore will mount if you put the sdcard in after the system is in "safemode" but the apps and such will not be loaded until you reboot.
The size expansion is the primary benefit of the construct. As far as speed goes, I haven't checked to see if there are any speed gains in moving /data to the sd... Can someone who's interested check that out and let us know?
Click to expand...
Click to collapse
Speed is all I'm interested in If it speeds the device up, why not?
Anyone have anything to report about the speed? I am very interested in this
wow having this "dual boot" seems like an incredible idea... can anyone report on this?
can someone please give a more noob friendly instructions? from what I understand as of now is:
1) boot into recovery and wipe the phone (I'm assuming we can keep whatever ROM we had before)
2) place data2sd.img in root of SDcard (fat32)
3) and this is where I get lost... how exactly do I move the sh file to that location? how do I chmod it? how do I run it? what are the adb commands? thanks!
Hmmnm I thought we weren't doing this because of inheriant security risks?
Not to mention what happens when you mount your fat32 partition....are the apps no longer (as well as your data) available?
NO booting into recovery and wiping of the phone!
Hold up people!
**There is NO booting into recovery and wiping of the phone!**
My bad for not being clear on the wipe instructions!
1) (optional) If you are to wipe it (your choice it is not required), then press menu->settings->SD card & phone storage->scroll to bottom of screen and select "Factory data reset". This will wipe the /data partition only removing all apps and settings. BACKUP YOUR /data dir to sd, First or you will need to down load all your apps again to include paid apps!!
2) place data2sd.img in root of SDcard (fat32)
3) terminal in or adb shell in.
4) cp /sdcard/data2sd.sh to /data/local/bin - to copy the script to your user-space
5) chmod 0750 /data/local/bin/data2sd.sh - to make it executable
6) /data/local/bin/data2sd.sh or data2sd.sh may work to run the script
7) once the install is done it should tell you to reboot. If you had the ddms debugger running, you can observer the log as it is working.
8) done. It may boot a little longer if you clean-slate installed and then restored your apps by copying them back into /data/app and /data/app-private which has the affect of reinstalling everything, and thus the boot will take a while if you have alot of apps like I do. Again you can observe this as the Android boots if you have the debugger running.
thanks a lot man, will give this a shot in a few hours! not at my comp right now
security risks and (u)mounting /sdcard
TheDudeOfLife said:
Hmmnm I thought we weren't doing this because of inheriant security risks?
Not to mention what happens when you mount your fat32 partition....are the apps no longer (as well as your data) available?
Click to expand...
Click to collapse
Not sure why there would be any security issues, the Android is already rooted. If someone can steal the sdcard, they can steal the entire Android, and if the sdcard is left lying about, then yes it can be strolen then too.
The construction of the SDCore assumes all the original permissions, so nothing has access to things it shouldn't save the scripts that use the cache as a backup mount point, but it is given the same permissions as /data, so no worries.
With this construct in place, you can't physically remove the sdcard without first powering off the Android, or rebooting it and removing the sdcard first thing. But while the system is operational, you can mount the sdcard to usb and umount it from usb without issue as long as any mounts to the sdcard are released (loopfs, etc) prior to trying. vold will give 10 tries to umount /sdcard to include attempting to kill the processes with file locks on /sdcard when it tries to umount the volume to attach it to usb.
A a matter of fact, if you observe via teh ddms debugger during boot, the vold service is busy checking the sdcard and mounting it well after the system has already mounted the sdcp2 and begun initializing apps from the sd /data.
So, as long as "sdcard partition2" (sdcp2) remains mounted and all, you can mount the sdcard to the connected Host and back and still have access to your apps and data all you want. I reckon this is how the app2sd and dalvik2sd constructs also perform.
The only thing to really note is that the first partition must be vfat (fat32) and the second either ext2 or ext3 (currently as they support file permissions). vold only cares about the first partition which it expects to be a vfat volume. So as long as p1 is vfat and p2 is posix compliant, we're golden!
The only real question I don't have a good answer for is the speed gains question. My card is a 8GB class6 and things seem fairly responsive. I don't know how much different it would be with a class 4 or 2, or the real speed difference with the built in storage. Honestly it feels a little more responsive, but I haven't really done things that make it feel sluggish.
Oh, I saw the .img and assumed you just mounted the .img from fat32. My bad for not reading all the instructions.
Any problems with apps crashing? I have had a lot of quirky issues in the past with merging the entire partition.
Thanks for the response. I like what you've done here. I'll have to try it out for sure.
TheDudeOfLife said:
Oh, I saw the .img and assumed you just mounted the .img from fat32. My bad for not reading all the instructions.
Any problems with apps crashing? I have had a lot of quirky issues in the past with merging the entire partition.
Thanks for the response. I like what you've done here. I'll have to try it out for sure.
Click to expand...
Click to collapse
No worries; the .img file can be thought of as an install cdrom it is only used the one time during install.
As far as apps crashing, no problems!
During my experimentation building the construct, permission mangling happens if apps are moved out of /data/app, /data/app-private but /data/data and maybe /data/dalvik-cache are not cleaned up prior to copying apps back into their respective app dirs.
What happens is the app dirs are monitored. if you move an apk into the dirs android will install the app. Likewise if you delete an app from those dirs PackageManager will uninstall the app. If /data/data already has the dir for the app, it will use it. if the /data/system/packages.xml and /data/data don't have the same id assigned to that app, you get a permissions\\id conflict; the PackageManager will not manipulate the dir if the id's don't match.
The app /data/local/bin/archiveApps.sh, /data/local/bin/softWIPE.sh and /data/local/bin/restoreApps.sh scripts allow one to backup thier apps to the new /data/app-archive and /data/app-private-archive dirs, clean out /data/data/ /data/dalvik-cache, and restore the apps to their app dirs. As the apps are copied into their dirs, the PackageManager will update /data/system/packages.xml proper and rebuild /data/data and the dalvikVM will build the /dalvik-cahce proper.
Someone made a seemingly nice script (I've not used it yet) called fix_permissions.sh that will parse the /data/system/packages.xml and update the ownership id of the apps /data/data sub dir proper and thus is more scalpel like in operation, but that process doesn't make a backup of the apps; so using both may be the most effective replacing the softWIPE.sh part of the process with the fix_permissions.sh instead.
The only time you should run into issues is if say you have 10 apps installed in teh SDCore /data and 6 in Android /data and then attempt to sync them (because the 6 are different than the 10) by copying\\merging Android /data with SDCore /data the SDCore /data/data /data/dalvik-cache and /data/system/packages.xml will now be mangled.
/data/system/packages.xml will now no longer know about the 10 apps that were installed prior, and on boot PackageManager will "install" them, but /data/data will have dirs inside with now different id's from what was newly assigned them in /data/system/packages.xml, and the 10 apps will now not work correctly but the 6 will as dirs with matching id's were created when they were "installed" new.
To fix, either the softWIPE.sh process outlined above, or the fix_permissions.sh process also mentioned above.
So did anyone try this at all?
Can this be done after doing the apps2sd by marcusmaximus? I tried it but when trying to chmod 0750 /data/local/bin/data2sd.sh it said "No such file or directory" so i tried mkdir /data/local/bin and got "File Exists" so i proceeded with copying data2sd.sh to /data/local/bin again which went without showing any errors and when trying to chmod 0750 /data/local/bin/data2sd.sh again i got the same error so i tried to ls /data/local/bin and just got # what am i doing wrong?
looks interesting! might try this during the weekends ill give an update if i encounter any issues or not
interesting. I'll be looking into this!
So I was using Slax. Great LiveCD/USB linux, extremely customizable, modular, fast, and small, and has the capability of either:
- saving changes to its rootfs onto an AUFS mounted on a non-linux FS (FAT32, NTFS) using posixovl (POSIX Overlay FS) with metadata (permissions, etc.) being held in files
- saving changes to a fixed-size loop mount image.
This got me thinking.
If we could insert all the necessary modules, code, etc. for posixovl into the Android linux, and make a modified a2sd script that takes advantage of posixovl, we could effectively do away with the requirement for crazy partitioning.
It should be simple enough for ROM devs to implement, assuming it's ready and installed:
1. Create folder on main partition if it doesn't exist, something like /sdcard/system/[app, app-private, dalvik-cache, app_s]
2. In the init scripts, before the a2sd stuff, mount /sdcard/system with posixovl on /system/sd
3. Run a2sd as normal, it should automatically just work.
I'll hopefully test this once I get my phone to a stable development/testing stage, and I don't need to make phone calls for a while. Anyone else is welcome to try to implement this idea.
My current test environment:
- HTC Dream (T-mo G1) with the deadly SPL of doom
- Cyanogen Experimental, latest build
- Amon_RA's modded recovery
- Wipe /data, move all existing apps to backup, remove a2sd partition, Backup for Root Users to restore some settings and data
Anyone with ideas or improvements, please let me know.
To be tested:
- Feasibility (can it work?)
- Functionality (does it work?)
- Portability (Can it work on other ROMs and devices like Hero, Pulse, Blur etc.? If so, will likely be moved to XDA's new Android board)
- Stability (Will everything Force Close on boot? Does it run fast enough? Does anything get corrupted over time?)
Links:
- http://sourceforge.net/projects/posixovl - Sourceforge page for posixovl
In desktop linux you can create a file with the touch command, and mount the file to a mountpoint after formatting it to ext4 for example.
Maybe this is the easier way?
I have done this about 5 years ago, but I will try it today and report if it worked.
edit: ok done already:
1. create a file of the desired size, eg: dd if=/dev/zero of=filename bs=filesize count=1
2. use mke2fs to format the file
3. create a mountpoint and mount the file
thats all. Now I have a 128MB file on my PC, mountable and usable like a partition.
Archont said:
In desktop linux you can create a file with the touch command, and mount the file to a mountpoint after formatting it to ext4 for example.
Maybe this is the easier way?
I have done this about 5 years ago, but I will try it today and report if it worked.
Click to expand...
Click to collapse
As far as I know, you can do that on the mobile Android, too, and that does work in theory. This technique involves mounting a loop filesystem, and it too will allow one-partition apps2sd, but it's less flexible, and I would think slower, than the overlay method.
For a 512MB apps image:
Create empty 512MB file
# dd if=/dev/zero of=/sdcard/apps.img bs=1024k count=512
Format it to Ext2
# mke2fs -L Apps2SD /sdcard/apps.img
Unmount existing a2sd
# umount /system/sd
mount new a2sd image
# mount -t ext2 -o loop /sdcard/apps.img /system/sd
Make the usual directories, and a mountpoint for the old a2sd partition
# mkdir /system/sd/app; mkdir /system/sd/app-private; mkdir /system/sd/dalvik-cache; mkdir /system/sd/apps-tmp
Mount the old a2sd partition
# mount -t ext2 /dev/mmcblk0p2 /system/sd/apps-tmp/
Move all files from the old partition to the image file
# mv /system/sd/apps-tmp/* /system/sd/
Unmount and remove the mountpoint, we don't need it anymore
# umount /system/sd/apps-tmp
# rmdir /system/sd/apps-tmp
Finally, you add the following line to the init script where the a2sd auto mount happens, and comment out the old line.
Code:
[...]
mount -t ext2 -o loop /sdcard/apps.img /system/sd
#mount -t ext2 /dev/mmcblk0p2 /system/sd/
[...]
This should do what you described, in theory. I can't say whether it will work or not. I can't tell whether it will or won't screw up your phone, I can't be held responsible if you screw something up or overlook the details. Either of us might have made a typo somewhere; apply common sense before doing anything.
This sounds great! I think this would also be usefull for someone like me, who has a sd card that doesn't want to be partitioned anymore (cross-linked files??). Only thing possible is fat32 or ntfs.
Am I correct with my assumption?
This sounds a lot harder and more complicated then partioning. Your also talking about a lot of work needing to be done just so people can avoid doing a simple thing like partioning a sd card. I would say it would be very difficult alone to get the os to run a virtual mounting service especially since that will take up resources and slow down the phone. There is a reason they only use this technique on live cds is it works but its slow. most of the computers they are running on have 1gig of ram and 2ghz cpu's. i really dont think the g1 can push this.
I do not think that this is great, it definitely is slower than a ext4 partition on a good class 6 microSD card. And it is more vulnerable to data loss since 2 different filesystems including a 20 year old non-journaling fs at the base of this construct are involved.
Another problem that came to my mind: when you mount your SD card as external USB device to a PC, the file containing your apps will no longer be accessible, or Android will make using the phone as external data storage impossible.
Interesting.
There is, however, a major problem: What happens when you unmount the fat partition on the phone in order to connect with a computer using UMS? Answer: everything on the phone will crash and burn since the apps filesystem will suddenly disappear = BAD.
posixovl is a nice find though...
Note that aufs, loopmount linux filesystems, etc., wouldn't be needed with this since posixovl appears to be vfat with posix extensions, so you should be able to just use posixovl directly on the sdcard.
There are several problems with that though... i.e. how reliable is posixovl regarding users tampering with it?
In any case, a prerequisite for use of it would be certain other changes being planned...
You might want to contribute to this thread:
http://forum.xda-developers.com/showthread.php?t=577941
(note: the thread links to a thread at android-platform, the one here has, as expected, gone off on a tangent... just ignore the junk.)
TylTru said:
So I was using Slax. Great LiveCD/USB linux, extremely customizable, modular, fast, and small, and has the capability of either:
- saving changes to its rootfs onto an AUFS mounted on a non-linux FS (FAT32, NTFS) using posixovl (POSIX Overlay FS) with metadata (permissions, etc.) being held in files
- saving changes to a fixed-size loop mount image.
This got me thinking.
If we could insert all the necessary modules, code, etc. for posixovl into the Android linux, and make a modified a2sd script that takes advantage of posixovl, we could effectively do away with the requirement for crazy partitioning.
It should be simple enough for ROM devs to implement, assuming it's ready and installed:
1. Create folder on main partition if it doesn't exist, something like /sdcard/system/[app, app-private, dalvik-cache, app_s]
2. In the init scripts, before the a2sd stuff, mount /sdcard/system with posixovl on /system/sd
3. Run a2sd as normal, it should automatically just work.
I'll hopefully test this once I get my phone to a stable development/testing stage, and I don't need to make phone calls for a while. Anyone else is welcome to try to implement this idea.
My current test environment:
- HTC Dream (T-mo G1) with the deadly SPL of doom
- Cyanogen Experimental, latest build
- Amon_RA's modded recovery
- Wipe /data, move all existing apps to backup, remove a2sd partition, Backup for Root Users to restore some settings and data
Anyone with ideas or improvements, please let me know.
To be tested:
- Feasibility (can it work?)
- Functionality (does it work?)
- Portability (Can it work on other ROMs and devices like Hero, Pulse, Blur etc.? If so, will likely be moved to XDA's new Android board)
- Stability (Will everything Force Close on boot? Does it run fast enough? Does anything get corrupted over time?)
Links:
- http://sourceforge.net/projects/posixovl - Sourceforge page for posixovl
Click to expand...
Click to collapse
lbcoder said:
Interesting.
There is, however, a major problem: What happens when you unmount the fat partition on the phone in order to connect with a computer using UMS? Answer: everything on the phone will crash and burn since the apps filesystem will suddenly disappear = BAD.
Click to expand...
Click to collapse
I kinda overlooked that point. Oops.
Though this same problem does exist on regular apps2sd when you remove the card without dismounting it, killing all apps and their processes, and freezing Dalvik's method of autostarting some apps.
I do tend to swap cards every now and then, but only after a reboot. Dalvik re-enumerates and caches dex, which makes for a slow boot, but it seems to just work in most cases that the apk install doesn't drop the app's functional payload (helper Linux/shell utils, libraries, NDK .so's) in /data/data (like some emulators, the Android Scripting Environment)
lbcoder said:
There are several problems with that though... i.e. how reliable is posixovl regarding users tampering with it?
Click to expand...
Click to collapse
As far as I know, the metadata files are marked as hidden and system files, and begin with a '.'. And I haven't tried this, but I think modifying the actual files under Windows has no negative effects, but moving, deleting, or copying files would likely be a no-no.
Also, I don't think there's a 'fsck' for posixovl, meaning that if any metadata files were screwed with the wrong way, the entire overlay FS would be trashed.
lbcoder said:
In any case, a prerequisite for use of it would be certain other changes being planned...
You might want to contribute to this thread:
http://forum.xda-developers.com/showthread.php?t=577941
(note: the thread links to a thread at android-platform, the one here has, as expected, gone off on a tangent... just ignore the junk.)
Click to expand...
Click to collapse
I checked that out. It was actually a small inspiration for what I was thinking of.
In any case, Android's package management system needs an overhaul. The package storage needs to be de-Linuxified, as all it is is a bunch of .apk files and .dex/.odex files, the UIDs of apps are in the AndroidManifest.xml, right?
In a somewhat unrelated note, app data needs to be moved to a specified folder structure on the sdcard. My card is full of folders in the root directory with random names.
If I'm understanding you correctly, you're talking about storing an image file on the normal SD card partition(which has to be FAT32 as far as I've seen) and then mounting it, correct? This idea has been talked about at length before on at least 3 separate occasions(2 of which were on this very forum) and found to be a bad idea due primarily to massive security risks since FAT32 has no permissions.
Also, I believe cyanogen ended up dumping unionfs/aufs due to rampant memory issues.
If you are talking about mounting an image from the FAT32 partition, please don't endorse this. We don't want to be throwing in security bugs into android, especially ones such as this which can't be plugged up.
As a modification to what I said: If you're suggesting doing this(or something similar) on a separate filesystem, after that project to change the AOSP to support one with permissions is finished, then I'm in full support.
If you want to go for a single partition on the sd card, why don't you just make the entire card use ext4? Your linux desktop reads it anyway, it uses journaling and so on, I guess it would be faster compared to fat32 and it is definitely safer to use.
And i guess it is not too complicated to mount this partition and use it for pictures, music and so on.
I have not tried this (yet) and I go to bed in 20 minutes, but maybe I will start testing something in that direction tomorrow.
[email protected] said:
If I'm understanding you correctly, you're talking about storing an image file on the normal SD card partition(which has to be FAT32 as far as I've seen) and then mounting it, correct? This idea has been talked about at length before on at least 3 separate occasions(2 of which were on this very forum) and found to be a bad idea due primarily to massive security risks since FAT32 has no permissions.
Also, I believe cyanogen ended up dumping unionfs/aufs due to rampant memory issues.
If you are talking about mounting an image from the FAT32 partition, please don't endorse this. We don't want to be throwing in security bugs into android, especially ones such as this which can't be plugged up.
As a modification to what I said: If you're suggesting doing this(or something similar) on a separate filesystem, after that project to change the AOSP to support one with permissions is finished, then I'm in full support.
Click to expand...
Click to collapse
UNIX permissions don't do anything in the way of "security" unless you have no access to the actual storage device from another computer (as is the case with the unrooted Dream's internal memory), or unless encryption is used. The posixovl driver OVERLAYS Unix permissions over Fat32 filesystems. But even still, with Unix permissions, nothing's stopping someone else from mounting the Ext2 partition and using chown and chmod.
And the image file on the SD card's Fat32 partition is a complete Ext2 partition complete with Permissions. Nothing is lost.
Archont said:
If you want to go for a single partition on the sd card, why don't you just make the entire card use ext4? Your linux desktop reads it anyway, it uses journaling and so on, I guess it would be faster compared to fat32 and it is definitely safer to use.
And i guess it is not too complicated to mount this partition and use it for pictures, music and so on.
I have not tried this (yet) and I go to bed in 20 minutes, but maybe I will start testing something in that direction tomorrow.
Click to expand...
Click to collapse
We'd just have to find the part in the Android that mounts /sdcard/, and change 'vfat' to 'ext2'. The only reason I wouldn't do this, is because it would immediately make it incompatible with Windows and Mac's default FS drivers. As far as I know, the only FS's that are supported universally within Linux, Mac, and Windows, are FAT and NTFS. And NTFS can be made to have crude support for permissions through security descriptors. Although, the Dream SPL, the Recovery images, and most of Android only uses FAT32.
This is discussed in android-platform Group :
http://groups.google.com/group/andr...read/thread/bf0709c157451cd9/f6aee1830c84620f
The goal is to be able to integrate this in android.
And not having to partition the SDCard is one of the requirements so far...
Unix permissions are not stored using fat or vfat, and ntfs is not really supported in desktop linux and i guess it cannot be used in android linux.
I would not use windows anyway so this is no problem to me, and there are drivers around to mount ext systems in windows. As Mac OS is based on unix there will be a solution for this too.
Access usind adb push and pull, via ftp and so on is not touched by using ext4 on the entire sd card I guess.
And if you don't go the easy way using gparted on a live cd or usb device to create 2 partitions, you will have to live with some disadvantages anyway.
Finally I want to say that my ideas are far from being perfect or usable at all, I see this thread as a kind of brainstorming.
im not as linux or android savvy as probably any of you but before the current method of creating a swap partition became the "standard", people used a swap file on the sdcard and linked that. seems similar to what you are suggesting here.
ofcourse when mounting the fat partition elsewhere (ums in windows for example) that swap file could no longer be used within android. i dont see a way to get passed the same issue, but worse here, due to android not having crucial apps when the fat partition is mounted.
then again, i am pretty much over my head in this conversation and could be over looking something...
I'm kind of fascinated by the FUSE + posixovl method of doing this. In the long run I have a feeling that it's going to perform like ****, but I think it's worth testing.
I managed to get both libfuse and mount.posixovl built and running on Android.
posix-overlay(/sdcard/fuse) on /sdcard/fuse type fuse.posixovl (rw,nosuid,user_id=0,group_id=0,default_permissions)
Giving this a little testing now, it definitely works.
Code:
/sdcard/fuse # ls -l
drwxr-xr-x 2 1000 1000 4096 Nov 5 17:17 test
TylTru said:
UNIX permissions don't do anything in the way of "security" unless you have no access to the actual storage device from another computer (as is the case with the unrooted Dream's internal memory), or unless encryption is used. The posixovl driver OVERLAYS Unix permissions over Fat32 filesystems. But even still, with Unix permissions, nothing's stopping someone else from mounting the Ext2 partition and using chown and chmod.
And the image file on the SD card's Fat32 partition is a complete Ext2 partition complete with Permissions. Nothing is lost.
We'd just have to find the part in the Android that mounts /sdcard/, and change 'vfat' to 'ext2'. The only reason I wouldn't do this, is because it would immediately make it incompatible with Windows and Mac's default FS drivers. As far as I know, the only FS's that are supported universally within Linux, Mac, and Windows, are FAT and NTFS. And NTFS can be made to have crude support for permissions through security descriptors. Although, the Dream SPL, the Recovery images, and most of Android only uses FAT32.
Click to expand...
Click to collapse
Ya, I meant more from the standpoint of a rogue app. Since FAT32 has no permissions, what would prevent such an app from modifying the stored image file to, say, change a trusted app with superuser permissions to some new code of its own making to, for example, watch for credit card numbers and send them back to the person who made the original rogue app? I'm always hesitant with any ideas that suggest storing an image file on the sdcard for appstosd for this reason.
Forget it, it's useless.
An overlay filesystem prevents you from enabling USB storage.
If you want to play around with FUSE on Android, here's a repository for my port of libfuse..
http://github.com/cyanogen/android_external_fuse
Hi,
I have an idea. I used symbian S60 of Nokia, Symbian can install app to sdcard. I see that when I mount sdcard to PC, my phone immediately hold all activations of all applications on my phone. And they have a PC sync software that help us access sdcard but not mount sdcard (like that we copy file from computer to sdcard via debug mode on android).
I think we should find out how symbian can do it and we will use their way .
I'm not a developer, I'm just an user.
I talked to a few people about this, and some deep kernel voodoo is going to be needed for this to really happen without partitioning.
Another idea is to forge ahead with this, and ditch the "unmount fs for usb storage" and use RNDIS + Samba or something like that instead to access files on SD. I kind of like this idea.