Hi All,
Here is my first contribution to the community, hope it is useful!
I've rezzed up a construct to produce a new /data on sd. It makes use of a second partition formatted ext2 or ext3 mounted to /system/sd with /system/sd/data then mounted over /data.
I've included a lengthy readme file which should explain things in greater detail.
Read the readme first to get a list of dirs & files this construct adds to the /data, then copy the files as explained and off you go!
currently my /data "volume" is 2GB and all is lovely!
Things to note:
I'm rom JF1.51 ADP1 CRB43, the sdcard is 8gb class6
The sd /data contains several scripts of my own design which if not needed can be deleted. I include my modified bootdeb.sh script as it mounts to /data/local/mnt/debian vs /data/local/mnt (that is a script I modified for my own needs which you may find useful)
Feel free to adjust any of the scripts according to your needs. Just note that if I update the construct, that your changes may be undone if you use it, so back them up in that scenario.
Also, I include my mount.ak.sh script which make use of a loopfs to accomplish the roughly the same thing, but in a single partition environment (intended as proof of concept).
**There is NO booting into recovery and wiping of the phone for this mod!**
Thanks to JesusFreak for the roms and saurik for the Debian info!
Cheers!
Wow! Nice, Going to try it out. Thanks
is that mean after running data2sd the phone's completely running on sd? and so much faster?
Hi,
To followup on xnycen's question, why would we do this? Is the benefit only in providing more space, or is there also a performance increase (or decrease)?
hate to see the sdcard fail with this change
well...
Yes, after the install you are running complete on sd,
but because of the method used, if you reboot, and take out the sdcard before the boot starts, you will be running as you were before the data2sd aka normal; put the sdcard back in, reboot and you are in data2sd mode again.
This effectively gives you two android run modes: "normal" and data2sd. If the SDCore can not be located the Android will boot in normal mode. You can of course try to keep them sync'd, but if you do the data2sd from a clean slate you can effectively have a "safemode" and if you have an issue with some app, go into this "safemode" and do some investigations or adjustments as the SDCore will mount if you put the sdcard in after the system is in "safemode" but the apps and such will not be loaded until you reboot.
The size expansion is the primary benefit of the construct. As far as speed goes, I haven't checked to see if there are any speed gains in moving /data to the sd... Can someone who's interested check that out and let us know?
Darkstrumn said:
Yes, after the install you are running complete on sd,
but because of the method used, if you reboot, and take out the sdcard before the boot starts, you will be running as you were before the data2sd aka normal; put the sdcard back in, reboot and you are in data2sd mode again.
This effectively gives you two android run modes: "normal" and data2sd. If the SDCore can not be located the Android will boot in normal mode. You can of course try to keep them sync'd, but if you do the data2sd from a clean slate you can effectively have a "safemode" and if you have an issue with some app, go into this "safemode" and do some investigations or adjustments as the SDCore will mount if you put the sdcard in after the system is in "safemode" but the apps and such will not be loaded until you reboot.
The size expansion is the primary benefit of the construct. As far as speed goes, I haven't checked to see if there are any speed gains in moving /data to the sd... Can someone who's interested check that out and let us know?
Click to expand...
Click to collapse
Speed is all I'm interested in If it speeds the device up, why not?
Anyone have anything to report about the speed? I am very interested in this
wow having this "dual boot" seems like an incredible idea... can anyone report on this?
can someone please give a more noob friendly instructions? from what I understand as of now is:
1) boot into recovery and wipe the phone (I'm assuming we can keep whatever ROM we had before)
2) place data2sd.img in root of SDcard (fat32)
3) and this is where I get lost... how exactly do I move the sh file to that location? how do I chmod it? how do I run it? what are the adb commands? thanks!
Hmmnm I thought we weren't doing this because of inheriant security risks?
Not to mention what happens when you mount your fat32 partition....are the apps no longer (as well as your data) available?
NO booting into recovery and wiping of the phone!
Hold up people!
**There is NO booting into recovery and wiping of the phone!**
My bad for not being clear on the wipe instructions!
1) (optional) If you are to wipe it (your choice it is not required), then press menu->settings->SD card & phone storage->scroll to bottom of screen and select "Factory data reset". This will wipe the /data partition only removing all apps and settings. BACKUP YOUR /data dir to sd, First or you will need to down load all your apps again to include paid apps!!
2) place data2sd.img in root of SDcard (fat32)
3) terminal in or adb shell in.
4) cp /sdcard/data2sd.sh to /data/local/bin - to copy the script to your user-space
5) chmod 0750 /data/local/bin/data2sd.sh - to make it executable
6) /data/local/bin/data2sd.sh or data2sd.sh may work to run the script
7) once the install is done it should tell you to reboot. If you had the ddms debugger running, you can observer the log as it is working.
8) done. It may boot a little longer if you clean-slate installed and then restored your apps by copying them back into /data/app and /data/app-private which has the affect of reinstalling everything, and thus the boot will take a while if you have alot of apps like I do. Again you can observe this as the Android boots if you have the debugger running.
thanks a lot man, will give this a shot in a few hours! not at my comp right now
security risks and (u)mounting /sdcard
TheDudeOfLife said:
Hmmnm I thought we weren't doing this because of inheriant security risks?
Not to mention what happens when you mount your fat32 partition....are the apps no longer (as well as your data) available?
Click to expand...
Click to collapse
Not sure why there would be any security issues, the Android is already rooted. If someone can steal the sdcard, they can steal the entire Android, and if the sdcard is left lying about, then yes it can be strolen then too.
The construction of the SDCore assumes all the original permissions, so nothing has access to things it shouldn't save the scripts that use the cache as a backup mount point, but it is given the same permissions as /data, so no worries.
With this construct in place, you can't physically remove the sdcard without first powering off the Android, or rebooting it and removing the sdcard first thing. But while the system is operational, you can mount the sdcard to usb and umount it from usb without issue as long as any mounts to the sdcard are released (loopfs, etc) prior to trying. vold will give 10 tries to umount /sdcard to include attempting to kill the processes with file locks on /sdcard when it tries to umount the volume to attach it to usb.
A a matter of fact, if you observe via teh ddms debugger during boot, the vold service is busy checking the sdcard and mounting it well after the system has already mounted the sdcp2 and begun initializing apps from the sd /data.
So, as long as "sdcard partition2" (sdcp2) remains mounted and all, you can mount the sdcard to the connected Host and back and still have access to your apps and data all you want. I reckon this is how the app2sd and dalvik2sd constructs also perform.
The only thing to really note is that the first partition must be vfat (fat32) and the second either ext2 or ext3 (currently as they support file permissions). vold only cares about the first partition which it expects to be a vfat volume. So as long as p1 is vfat and p2 is posix compliant, we're golden!
The only real question I don't have a good answer for is the speed gains question. My card is a 8GB class6 and things seem fairly responsive. I don't know how much different it would be with a class 4 or 2, or the real speed difference with the built in storage. Honestly it feels a little more responsive, but I haven't really done things that make it feel sluggish.
Oh, I saw the .img and assumed you just mounted the .img from fat32. My bad for not reading all the instructions.
Any problems with apps crashing? I have had a lot of quirky issues in the past with merging the entire partition.
Thanks for the response. I like what you've done here. I'll have to try it out for sure.
TheDudeOfLife said:
Oh, I saw the .img and assumed you just mounted the .img from fat32. My bad for not reading all the instructions.
Any problems with apps crashing? I have had a lot of quirky issues in the past with merging the entire partition.
Thanks for the response. I like what you've done here. I'll have to try it out for sure.
Click to expand...
Click to collapse
No worries; the .img file can be thought of as an install cdrom it is only used the one time during install.
As far as apps crashing, no problems!
During my experimentation building the construct, permission mangling happens if apps are moved out of /data/app, /data/app-private but /data/data and maybe /data/dalvik-cache are not cleaned up prior to copying apps back into their respective app dirs.
What happens is the app dirs are monitored. if you move an apk into the dirs android will install the app. Likewise if you delete an app from those dirs PackageManager will uninstall the app. If /data/data already has the dir for the app, it will use it. if the /data/system/packages.xml and /data/data don't have the same id assigned to that app, you get a permissions\\id conflict; the PackageManager will not manipulate the dir if the id's don't match.
The app /data/local/bin/archiveApps.sh, /data/local/bin/softWIPE.sh and /data/local/bin/restoreApps.sh scripts allow one to backup thier apps to the new /data/app-archive and /data/app-private-archive dirs, clean out /data/data/ /data/dalvik-cache, and restore the apps to their app dirs. As the apps are copied into their dirs, the PackageManager will update /data/system/packages.xml proper and rebuild /data/data and the dalvikVM will build the /dalvik-cahce proper.
Someone made a seemingly nice script (I've not used it yet) called fix_permissions.sh that will parse the /data/system/packages.xml and update the ownership id of the apps /data/data sub dir proper and thus is more scalpel like in operation, but that process doesn't make a backup of the apps; so using both may be the most effective replacing the softWIPE.sh part of the process with the fix_permissions.sh instead.
The only time you should run into issues is if say you have 10 apps installed in teh SDCore /data and 6 in Android /data and then attempt to sync them (because the 6 are different than the 10) by copying\\merging Android /data with SDCore /data the SDCore /data/data /data/dalvik-cache and /data/system/packages.xml will now be mangled.
/data/system/packages.xml will now no longer know about the 10 apps that were installed prior, and on boot PackageManager will "install" them, but /data/data will have dirs inside with now different id's from what was newly assigned them in /data/system/packages.xml, and the 10 apps will now not work correctly but the 6 will as dirs with matching id's were created when they were "installed" new.
To fix, either the softWIPE.sh process outlined above, or the fix_permissions.sh process also mentioned above.
So did anyone try this at all?
Can this be done after doing the apps2sd by marcusmaximus? I tried it but when trying to chmod 0750 /data/local/bin/data2sd.sh it said "No such file or directory" so i tried mkdir /data/local/bin and got "File Exists" so i proceeded with copying data2sd.sh to /data/local/bin again which went without showing any errors and when trying to chmod 0750 /data/local/bin/data2sd.sh again i got the same error so i tried to ls /data/local/bin and just got # what am i doing wrong?
looks interesting! might try this during the weekends ill give an update if i encounter any issues or not
interesting. I'll be looking into this!
Does anyone know how to update the mount points so that when I insert a microsd card it is mounted to the mnt/sdcard folder? I still would like to be able to use the internal storage though. Maybe it can be mounted someplace else.
I ask because I would like to backup my apps to the SD card using Titanium backup and Astro. Backing up my apps to the internal memory makes no sense. Especially when you are trying to move Angry Bird saved games between devices.
I did the opposite when I was running the CM7 on the GTablet: all the ROMs mapped sdcard to the internal memory while CM7 had it the way you want it. I didn't need an SD card for the GTab, so suffice to say, I updated it to change it back so all my apps would work again.
Anyway, you need root, and all you have to do is find /etc/fstab and change the mount points to where ever you want them. On the GTab, I just switched emmc and sdcard around. This worked on Android 2.3, so I assume its the same for 3.x. I don't have my Transformer with me here at work or I'd look to make sure.
IMPORTANT: Make sure you make a system backup before you mess with it, cause you could mess up the file and then nothing will mount on start up, so proceed at your own risk.
http://forum.xda-developers.com/showthread.php?t=1122710
Last page has a script you can use or you can use the app method, I think it is towards the end of the 2nd page, its a 1.99 app but works fine and survives reboots
EDIT: Post 38 and 41
I am a programmer with experience working on Linux, brand new to Android. Recently I got my first Android phone, a Moto G 4G (XT1045) and installed CM11-20141009-NIGHTLY-peregrine on it. Today I bought a 32GB SDCard and formatted it to ext4 filesystem with AParted.
So it seems like cyanogenmod kernel should have SDCard fix for kitkat built in. Now I have a specific problem. I am trying to switch Titanium Backup to use sdcard1 as backup the location. So I have to browse to that folder.
My sdcard seems to be mounted in two places:
/mnt/media_rw/sdcard1
/storage/sdcard1
but both of them have problems. /storage/sdcard1 is a fuse mount and is read-only, even to root. I can browse there with TitaniumBackup, but then TitaniumBackup can't write any backups. /mnt/media_rw/sdcard1 is writable, and I can make directories on it from adb. But TitaniumBackup cannot browse inside of it, because media_rw is only accessible to user 1023(AID_MEDIA_RW). So I can't point TitaniumBackup to my sdcard!
How to fix this?
colinw1 said:
I am a programmer with experience working on Linux, brand new to Android. Recently I got my first Android phone, a Moto G 4G (XT1045) and installed CM11-20141009-NIGHTLY-peregrine on it. Today I bought a 32GB SDCard and formatted it to ext4 filesystem with AParted.
So it seems like cyanogenmod kernel should have SDCard fix for kitkat built in. Now I have a specific problem. I am trying to switch Titanium Backup to use sdcard1 as backup the location. So I have to browse to that folder.
My sdcard seems to be mounted in two places:
/mnt/media_rw/sdcard1
/storage/sdcard1
but both of them have problems. /storage/sdcard1 is a fuse mount and is read-only, even to root. I can browse there with TitaniumBackup, but then TitaniumBackup can't write any backups. /mnt/media_rw/sdcard1 is writable, and I can make directories on it from adb. But TitaniumBackup cannot browse inside of it, because media_rw is only accessible to user 1023(AID_MEDIA_RW). So I can't point TitaniumBackup to my sdcard!
How to fix this?
Click to expand...
Click to collapse
Hmmm. There have been problems with ext sd cards in cyanogenmod nightlies in the past and it is bound to happen. Not sure.
But try a CM11 snapshot instead of the nightlies as it is more stable with fewer bugs or wait for nightly updates which are very frequent.
Oh and make sure you do a nandroid backup before trying the CM11 Snapshot just in case.
https://download.cyanogenmod.org/?device=falcon&type=snapshot
Cheers!
CuckinFunt said:
Hmmm. There have been problems with ext sd cards in cyanogenmod nightlies in the past and it is bound to happen. Not sure.
But try a CM11 snapshot instead of the nightlies as it is more stable with fewer bugs or wait for nightly updates which are very frequent.
Oh and make sure you do a nandroid backup before trying the CM11 Snapshot just in case.
Cheers!
Click to expand...
Click to collapse
Thanks for the response - it seems it was not a problem with the ROM.
I've been working on this for about 8 hrs since the last post(!!), and I've made a lot of progress. I guess I'll just post it here in case someone comes across this question, though maybe not in scope of my original question.
It turns out my custom rom cyanogenmod (CM) has a fix for the KitKat modification built in. From the command line, as root, as long as I use /mnt/media_rw/sdcard1 to access the external secondary storage (my sdcard), everything works fine. Apps don't seem to mind that /storage/sdcard1 is mostly readonly.
I tested several non-rooted apps (e.g. doubleTwist), and noticed that they were able to write to various locations on my sdcard, always as user:group 1023:1023, ie MEDIA_RW. So the CM fix works. Note that you DO NOT have to install SDFix Apps or anything like that, or modify /etc/permissions/platform.xml. My original goal had been to get TitaniumBackup to start using my sdcard1. What I had to do was, as root, creat /mnt/media_rw/sdcard1/Backup/TitaniumBackup, set the owner and permissions correctly, and then I was able to switch to that directory in Titanium Backup (note, you have to use the Android system back button to go up a level when navigating the directories in Titanium Backup).
Also, It's important if you are using ext4 file system for your sdcard1, that it actually be ext4, not ext2, or CM doesn't recognize it. I used AParted to format it to ext4.
Finally, seems like FolderMount is great app for actually USING your sdcard1, once it's writable. I used FolderMount to redirect sdcard0/DCIM to sdcard1/DCIM, and now my camera is saving to the sdcard.
Sorry for maybe stating so much obvious info, but thought it might help some newcomer like myself - also seems like there's not a really good guide out there for users on how to work with the KitKat restrictions. Seems like with a Mod'd ROM like CM11, maybe nothing has changed(??) I am too new to Android to know the answer.
I've been searching both this forum and the net generally and not found an answer to my specific problem so I'm hoping someone can help.
I'm trying to swap out my ROM, I'm currently running CM 11 nightly but I'm getting endless FCs with apps so thought I'd try something different.
When I try to run the nandroid backup with TWRP v2.7.0.1 the backup fails every time with the error E: unable to mount '/sd-ext'
Can anyone advise? Thanks
tht looks like external sd card.
Try changing location for your backup to internal memory or make sure you got SD card plugged in to SD slot.
you may also try command on terminal emulator (or via adb shell):
ln -s /storage/emulated/0/ /sd-ext/
it will create symbolic link named sd-ext in root of your system and it will point to internal storage (if I am not mistaken, from now on it should not matter if you select internal or external storage as target for your backup - it will end up as internal anyways)
Sent from my TF300T using XDA Free mobile app
Thanks, the backup was to be created on the internal SD card, but you are right it is almost certainly a problem with the external card. I'm unable to mount either the dock SD or micro SD cards in TWRP - the options are just greyed out and can't be selected.
I can't get either card to mount when I boot to system ether I have noticed. I've tried formatting them as FAT32, EXFAT and NTFS and no dice so I'm wondering if this is a KK issue - they both worked flawlessly when I was running PA and Omnirom JB ROMs.
I've managed to find a workaround - I've flashed CWM and have been able to do a nandroid backup with that - but I still can't mount either the micro SD or dock SD cards with CWM or in system when I flashed KatKISS ROM - and of course now I can't use any old TWRP nandroids I have from JB ROMs without reflashing TWRP. I'd also rather be running TWRP ultimately if I can so it would be good to get to the bottom of what is causing this issue.
Not quite sure what the command line you suggested does, but I'm guessing it won't be useful with the problem as it stands now?
sorry for late reply as I was out of internet lately.
that command creates /sd-ext/ folder that acts similar to shortcut to intertnal sdcard
any call for /sd-ext (for example: cd /sd-ext) will end up on internal sdcard.
That wont help with accessing your external sdcard (as this was main issue here) - it would merely fool installer to 'think' it is working on external sd while it was still on internal.
Sent from my TF300T using XDA Free mobile app
Thanks for your help. I've had endless troubles which ultimately ended in a hard brick during a failed flash of the stock ROM. I'm waiting for a replacement motherboard - something wasn't right with the old board anyway, it doesn't seem that it was just ROM issue. Hopefully when it comes I'll find that it wasn't a hardware issue associated with the SD cards as well.
I have a Sprint Galaxy Note $ Running Stock og5 Rom Rooted via Beast Mode TWRP then Play update, then flashed to Plazma Kernel. I have modified the /etc/permissions/platfom.xml file to include Group Id media_rw in the write to external storage permission and mounted the /mnt/media_rw/estSDcard as r/w in the root explorer portion of ES File Manager and it seemed to work fine for as long as I can remember but now if I try to copy or paste or send something to /storage/extSDcard it is created asychrously and ends up in the LOST.dir folder after a simple unmount remount or reboot or SDcard pull. Other apps are having the same problems like app backup restore and Btitorrent and others when trying to write directly to extSDcard I even went ass far as adding sdcard_all to the aforementioned gid permission set. It seems like such an obvious thing that has been overlooked by develpoers of the rom. I can create masting folders "no asychronously' if I do so buy navigating to /mnt/media_rw/extSDcard rather than /storage/extSDcard and they'll last but not all programs will allow that directory in their settings. Furthermore ES download Manager won't function with either directory and will not allow their app backup file to be saved the /mnt directory. only /0 or /storage/extSDcard.
I have exhausted my abilities with this and have no answer also It seems to be a new problem, and I haven't done anything I can think of to cause this. Partition table is 55000mb Fat32 17009mb ext4 journal partition on the extSDcard. Any Ideas?
Below you can see The before and after of unmounting and remounting my sd card resulting in Populated Lost.dir and loss of created folders also you can see the way that all nav methods resultu in same permission set when checked with a root explorer "total Commander"
wELL i DON'T KNOW WHY BUT REFORMATTING THE SD CARD TO A SINGLE FAT 32 VOLUME SOLVED THE LEAK. i FIRST TRIED AN ALTERNATE SD CARD WITH A SIMILAR PATITION TABLE (EXT2/FAT32) BUT HAD THE SAME PROBLEMS EXCEPT THE DATA WOULD SIMPLY BE GONE INSTEAD OF IN THE LOST.DIR FOLDER. i HAVE YET TO IMPLEMENT THIS TO MY PRIMARY USE SD CARD because of the fact that I have backups and files that I need on that card from before the leak. I anticipate the same results, fingers crossed.
If ANYONE has any incite into why the partition table set up as it was would affect the retention of data in this manner please let me know.
PLEASE PLEASE let me know
Hello,
Try posting your question/issue in the ROM/kernel thread where you got them or their dedicated Q&A thread(if it has one) or even PM the developer in case its something he isn't aware of, he may want to try and solve it so his ROM has no issues.
I hope this helps. Good luck.
Droidriven said:
Hello,
Try posting your question/issue in the ROM/kernel thread where you got them or their dedicated Q&A thread(if it has one) or even PM the developer in case its something he isn't aware of, he may want to try and solve it so his ROM has no issues.
I hope this helps. Good luck.
Click to expand...
Click to collapse
Thanks, I was using stock rom though. And the caps lock was an accident, at any rate i have changed roms and the issue appeared to be a faulty sd card to begin with. I wasn't able to retain data from my reader writer to phone on my primary sd ince reformatted but was when I used a different one. Anyway thanks for the advice and I will use it if I have different questions