Related
I've seen a lot of people complaining about not being able to boot after using A2SD for a while and I think it's definitely necessary to make clear the danger in using A2SD and some ways to prevents them.
Do not move /data/data to SD.
You can safely move /data/app and /data/app-private to SD. Be cautions with /data/dalvik-cache. (See below)
Make sure the ext2 partition are mounted with noatime and nodiratime (rom maker's job)
Do regular file system checks (using Linux)
Back up your ext2 partition and redo them regularly
Here're the theories:
(NAND) Flash memory has two serious limitations when used to store frequently changing data: you cannot do random write unless you erase a whole page first and any bit can only be written a limited number of times (typically a couple millions). When one bit in a page is detected to be faulty, you lose the whole page. (More details: http://en.wikipedia.org/wiki/Flash_memory#Limitations )
Most mobile os vendors overcome this problem by using special file systems, namely yaffs and jffs, which arrange files according to page size and do write operations only when necessary. They also provide journals so hardware faults could be reliably detected and corrected.
EXT2, which is the file system used in A2SD, DO NOT have these features. It may stupidly allow several pages to be erased hundreds of times of just to write some small files into it. Things are made even worse because some fixed part of the file system (inode table and bitmap) has to be written EVERYTIME a file operation is done. The will accelerate the wearing of those pages and when they become inaccessible, you lose the whole file system.
A2SD would work fine if you only move /data/app to it, because the application files are never modified. It is a completely different story for /data/data because the sqlite databases in there are modified almost every other second!! (And I suspect the OS commit them to disk very often to ensure data integrity.)
In addition, android does not have system check tools for ext2, so it will not be able to detect any problem with the file system until it’s too late.
Edit:
I am not sure how frequent the system updates files in dalvik-cache, but I would say you only move it when you are running out of space in /data.
(Also changed title)
billc.cn said:
I've seen a lot of people complaining about not being able to boot after using A2SD for a while and I think it's definitely necessary to make clear the danger in using A2SD and some ways to prevents them.
Do not move /data/data to SD.
Do regular file system checks (using Linux)
Back up your ext2 parition and redo them regularly
Here're the theories:
(NAND) Flash memory has two serious limitations when used to store frequently changing data: you cannot do random write unless you erase a whole page first and any bit can only be written a limited number of times (typically a couple millions). When one bit in a page is detected to be faulty, you lose the whole page. (More details: http://en.wikipedia.org/wiki/Flash_memory#Limitations )
Most mobile os vendors overcome this problem by using special file systems, namely yaffs and jffs, which arrange files according to page size and do write operations only when necessary. They also provide journals so hardware faults could be reliably detected and corrected.
EXT2, which is the file system used in A2SD, DO NOT have these features. It may stupidly allow several pages to be erased hundreds of times of just to write some small files into it. Things are made even worse because some fixed part of the file system (inode table and bitmap) has to be written EVERYTIME a file operation is done. The will accelerate the wearing of those pages and when they become inaccessible, you lose the whole file system.
A2SD would work fine if you only move /data/app to it, because the application files are never modified. It is a completely different story for /data/data because the sqlite databases in there are modified almost every other second!! (And I suspect the OS commit them to disk very often to ensure data integrity.)
In addition, android does not have system check tools for ext2, so it will not be able to detect any problem with the file system until it’s too late.
Click to expand...
Click to collapse
these are very good points. Actually come to think of it we better mount the ext2 partition with noatime. Because right now every read will wear the flash down. flash storage is really not meant for ext2 filesystem.
knaries2000 said:
these are very good points. Actually come to think of it we better mount the ext2 partition with noatime. Because right now every read will wear the flash down. flash storage is really not meant for ext2 filesystem.
Click to expand...
Click to collapse
In JF1.5 build it is mounted with noatime, i believe:
#mount
/dev/mmcblk0p2 on /system/sd type ext2 (rw,noatime,nodiratime,errors=continue)
But I totally agree with the point of the thread /data/data should not be moved to sd. Not that it's only dangerous (as described), I even don't see any advantages of it.
Dimath said:
In JF1.5 build it is mounted with noatime, i believe:
#mount
/dev/mmcblk0p2 on /system/sd type ext2 (rw,noatime,nodiratime,errors=continue)
But I totally agree with the point of the thread /data/data should not be moved to sd. Not that it's only dangerous (as described), I even don't see any advantages of it.
Click to expand...
Click to collapse
really. that's good, but I am running haykuro's build right now and it is not mounted with noatime. I will have to change the init script.
I'm using JF's 1.5 A2SD build, and I'm pretty certain I moved over /data/data. I didn't really ask myself the question when doing it, but what exactly is stored in /data/data? Is there a command I can run to move it back off the SD card?
Are ext2 and fat the only supported file systems in the android kernel? If not maybe it would be best to move to a wiser file system.
Rekna said:
Are ext2 and fat the only supported file systems in the android kernel? If not maybe it would be best to move to a wiser file system.
Click to expand...
Click to collapse
Yeah, why can't we use yaffs etc?
Dimath said:
Yeah, why can't we use yaffs etc?
Click to expand...
Click to collapse
i don't think most partitioners support yaffs
i know the partition manager on ubuntu 8.10 doesn't
tubaking182 said:
i don't think most partitioners support yaffs
i know the partition manager on ubuntu 8.10 doesn't
Click to expand...
Click to collapse
Indeed, and I'm fairly sure that's why it hasn't been done. But keep in mind several million write cycles is a heck of a lot and apps only really write their data caches occasionally, so it'll likely be a while before anything bad happens(on the scale of years) so this is slight sensationalism. Actually the term "cache" is a slight misnomer here since it's really just storage the apps use for temporary data. If it were used as some kind of extended RAM or a real cache then I could see problems but with what it's used for it should be a non-issue.
billc.cn said:
Here're the theories:
Click to expand...
Click to collapse
SD cards are a form of removable flash that have their own write controllers. In most cases, the write controllers also perform wear levelling. This means even if a program writes to the same file on the sd repeatedly, each time it writes, it is not writing the same physical location on the flash. The reason for this is because due to the way flash works, entire blocks have to be erased before they can be rewritten. To make writing faster, the memory controller keeps a list of empty blocks that are ready to use. When a file is changed, the entire file with the new changes is written to a new memory block and then the old block with obsolete data is then reset to zeros (data deleted).
That said, I still think A2SD is a red herring that only contributes to newbies spamming these forums.
jashsu said:
SD cards are a form of removable flash that have their own write controllers. In most cases, the write controllers also perform wear levelling. This means even if a program writes to the same file on the sd repeatedly, each time it writes, it is not writing the same physical location on the flash. The reason for this is because due to the way flash works, entire blocks have to be erased before they can be rewritten. To make writing faster, the memory controller keeps a list of empty blocks that are ready to use. When a file is changed, the entire file with the new changes is written to a new memory block and then the old block with obsolete data is then reset to zeros (data deleted).
That said, I still think A2SD is a red herring that only contributes to newbies spamming these forums.
Click to expand...
Click to collapse
Which is exactly why I made my new method that's able to deal with a lot of user mistakes and can be incorporated into ROMs to make it take almost 0 user effort
Dimath said:
Yeah, why can't we use yaffs etc?
Click to expand...
Click to collapse
yaffs on SD cards can in some common cases invalidate the wear leveling in hardware that SD cards do, as they so the wear leveling in software.
You are simply understimating ROM makers, SD cards are different from the internal flash in that they do auto wear leveling, that's why you can put common filesystems like FAT which have statically placed allocation tables and writes to the same logical sector will always land on very different places in the card every time.
Yes, noatime will help a lot as it will _reduce_ writes to your SD card.
ext2 is not journalled so it will have less writes too than ext3 or any other journalled filesystem.
So ROM makers are already doing a good job, don't understimate them please.
Help setting up userinit.sh & user.conf for "Backing Swap" on CyanogenMod
I am trying to set up Backing swap, which combines Compcache and traditional swap, on the phone/rom in my sig. I have been unable to do so for two main reasons.
1. I used Amon RAs recovery to make the partitions so I'm not sure what order they are in (need to know so I can put the right partition # in the useritit.sh file).
2. I am unsure how to create these files once I have all the proper info to plug into them, or where to acquire a base-line version of these files to edit.
I have been looking for hours but only seem to find posts of people suggesting I use these files. Even the Q&A wiki doesn't contain anything on the subject either (unless I was entering the wrong search critiria). I only have the terminal on my phone to work with or my pc which is win XP 64bit, so if I need any 3rd software please provide the windows flavor.
TIA everyone,
~psguardian
You can have one or the other one not both.
Swap always slows down the phone after a couple of hours and compache is compressed into the ram.If you want a fast rom use Super-D or Wesgarner.
Ace42 said:
You can have one or the other one not both.
Swap always slows down the phone after a couple of hours and compache is compressed into the ram.If you want a fast rom use Super-D or Wesgarner.
Click to expand...
Click to collapse
According to the Cyanogenmod wiki it does work with both (atleast you can back a shrunk compcache with the swap partition), I just don't want to try putting the file together until I can cipher out which part. # my swap partition is, as this may turn my ext3 partition into a 512mb swap partition & I don't want that.
~psguardian
Think i was thinking about backswap..thats not important though.
Swap partition is Block3
EXT is block 2
Ace42 said:
Think i was thinking about backswap..thats not important though.
Swap partition is Block3
EXT is block 2
Click to expand...
Click to collapse
I can see how backswap & standard swap wouldn't work together because they will try to use the same space. Is the linuxswap partition always mounted as Block3 ? when going through the format process in Amon RA recovery fat 32 is the last partition defined, which is why I hesitated to use the example given on the wiki (since back when that was put together partitioning was all manually handled).
~psguardian
still not smooth
I managed to get user.conf maker app on the market but it doesn't work properly in 4.2.13 (it sets the files up with its own defaults, but never accepts user input for value changes).
Had to
cp /system/sd/userinit.sh /sdcard/userinitsh.txt &
cp /system/sd/user.conf /sdcard/userconf.txt
so that I could use a 'notepad' type app to edit them propperly & then push them back to /system/sd now my phone is lagging worse then before so I'm not sure what I've done wrong... for backingswap am I suppose to enable compcache AND backingswap or just backingswap? I think its option A because that is the only time my swap partition shows active under 'free' command.
So now a new set of questions.
How much RAM is good to allocate to compcache
How large a swap partition should be used
What swappiness level is good for my setup
I know these answer are subject to the situation on my phone so let me give some stats, I am running a theme w/advanced launcher (5 screens) 10 widgets & 3 background apps that update every 15-30min (1 of them is a craigslist scanner that pulls search data from 10-15 regions so it probly should count as 3 by itself) so lets say 15ish self-updating apps.
Please any constructive input is very very welcome
Thanks In Advance Q&A guys-n-gals
~psguardian
Are there any plans to implement a Button into the App-Manager, to move the Data-Part of an app to the /ext2 Partition ?
Especially on low memory devices (like the G1) this will give us the chance to install a lot of apps without runnung to "low space on device".
I have written a very small bash-script to manually do this job, but it would be more comfortable with a Button.
move.sh
Code:
#!/system/bin/sh
#
#data_to_move=com.alk.copilot
#data_to_move=com.camelgames.blowup
#data_to_move=com.drodin.tuxrider
#data_to_move=com.gameloft.android.AMEU.GloftAsphalt5.asphalt5
#data_to_move=com.navigon.navigator
#data_to_move=com.polarbit.ironsightlite
#data_to_move=com.polarbit.ragingthunder
#data_to_move=com.polarbit.rthunder2
#data_to_move=com.polarbit.waveblazerlite
#data_to_move=com.estrongs.android.pop
su
mount -o remount,rw /data
mkdir /sd-ext/data
cd /data/data
echo About to Move ${data_to_move}
cp -r -p ${data_to_move} /sd-ext/data
rm -r -f ${data_to_move}
ln -s /sd-ext/data/${data_to_move} ${data_to_move}
unmove.sh
Code:
#!/system/bin/sh
#
#data_to_move=com.alk.copilot
#data_to_move=com.camelgames.blowup
#data_to_move=com.drodin.tuxrider
#data_to_move=com.gameloft.android.AMEU.GloftAsphalt5.asphalt5
#data_to_move=com.navigon.navigator
#data_to_move=com.polarbit.ironsightlite
#data_to_move=com.polarbit.ragingthunder
#data_to_move=com.polarbit.rthunder2
#data_to_move=com.polarbit.waveblazerlite
#data_to_move=com.estrongs.android.pop
su
mount -o remount,rw /data
mkdir /sd-ext/data
cd /data/data
echo About to unMove ${data_to_move}
rm ${data_to_move}
mkdir ${data_to_move}
cp -r -p /sd-ext/data/${data_to_move} ${data_to_move}
Apps that need lots of space should use the fat32 partition gracefully, using this hack would considerably slow down your system(even class6). Why double the reads and writes to the slow mmc?
Because not all apps using the fat32-Partition.
Navigon for example uses 14 MB of internal storage.
If you install such apps on a G1, you can quickly run out of space.
Low on cache space will considerably slow down the overall system performance.
I would not suggest to move the data of any installed apps, but from the big ones.
Apps like Navigation and 3D-Games are writing not frequently their data and if you own a G1 you are already swapping when using an Eclaid based ROM. So this wouldn't make a big difference in the livetime of your sdcard.
Perhaps someone could implement a partial move (e.g. only libraries) to avoid massive writes to the card ... and/or setting a threshold value, so apps with small data could not be moved.
For me, moving the apps in the Script above has given me additional 35 MB internal space and everything is still runnung "fast".
/data/data
TheGenesis said:
Because not all apps using the fat32-Partition.
Navigon for example uses 14 MB of internal storage.
If you install such apps on a G1, you can quickly run out of space.
Low on cache space will considerably slow down the overall system performance.
I would not suggest to move the data of any installed apps, but from the big ones.
Apps like Navigation and 3D-Games are writing not frequently their data and if you own a G1 you are already swapping when using an Eclaid based ROM. So this wouldn't make a big difference in the livetime of your sdcard.
Perhaps someone could implement a partial move (e.g. only libraries) to avoid massive writes to the card ... and/or setting a threshold value, so apps with small data could not be moved.
For me, moving the apps in the Script above has given me additional 35 MB internal space and everything is still runnung "fast".
Click to expand...
Click to collapse
there is already a script in existance to move things like that:
# lucid -d -sd
would move app data to the sdcard and symlink .. this (however) does not move individual pieces .. i would be interested to know the speed difference on these apps that you moved .. also .. this will create extra difficulty when attempting any nandroid backup/restore .. i have seen people putting up comments because their phone crashed due to the excessive number of symlinks across the phone .. if you are not careful with them you could lose all your data
The Space allocated by the "big apps" is mainly used by their ./lib dir. Because of this, it would be enough to move and symlink only those dirs.
With Games the "rest" goes to settings and scores.
On a G1 there is absolutely no difference in speed when moving data to sd.
Perhaps its because the G1 is not the top performer at all
I have played those games in the list with data in internal and on sd-ext and there is no difference in speed ... loading time is also the same.
I'm satisfied with the results, but these scripts didn't remove the data when apps where uninstalled or re-installed and thats the reason of my request.
Take a look on your storage and see how much space (libraries) should be on the sdcard.
Code:
du -sk /data/data/* | sort -rn | head
btw ... did you ever enabled JIT on a G1 and played ExZeus or Armageddon Squadron ?
Its amazing what is possible on this "outdated" Hardware!
P.S. Nandroid Backup runs perfectly with this symlinks (no recursive/double Backups)
Update:
Nandroid Backup only saves app and app-private ... could you edit this to save everything excluding "crap-dirs" ?
If nandbackup uses standard tar calls, you can use the following command:
Code:
tar pcvf /sdcard/nandroid/sd-ext.tar . -C /sd-ext --exclude dalvik-cache --exclude lost+found
... it would save everything the user place on the partition including userinit.sh
Restoring such a tarball works perfectly with nandbackup.
Thx in advance
Thom
Int. mem for SWAP ?
well, after the moving some heavy apps data
I have 54mb free Int. mem (out of 90)
is that possible to use 24-32mb of this (fast?)memory
for SWAP ? instead of linux swap
and does it make any sense?
...just tried to enable /swapfile.swp via Swapper2
though it says
-creating swap - ok
-changing permission - ok
-formatting swap - ok
but
- enabling swap(file) - FAIL
sorry if it's just another stupid question
G1, stock cm5.0.8 test4, 32mb linux swap
TheGenesis said:
Update:
Nandroid Backup only saves app and app-private ... could you edit this to save everything excluding "crap-dirs" ?
If nandbackup uses standard tar calls, you can use the following command:
Code:
tar pcvf /sdcard/nandroid/sd-ext.tar . -C /sd-ext --exclude dalvik-cache --exclude lost+found
... it would save everything the user place on the partition including userinit.sh
Restoring such a tarball works perfectly with nandbackup.
Thx in advance
Thom
Click to expand...
Click to collapse
Have you tried BART? http://forum.xda-developers.com/showthread.php?t=562292
It's included in the recovery, and lets you backup/not_backup whatever you want.
zelipukin said:
well, after the moving some heavy apps data
I have 54mb free Int. mem (out of 90)
is that possible to use 24-32mb of this (fast?)memory
for SWAP ? instead of linux swap
and does it make any sense?
...just tried to enable /swapfile.swp via Swapper2
though it says
-creating swap - ok
-changing permission - ok
-formatting swap - ok
but
- enabling swap(file) - FAIL
sorry if it's just another stupid question
G1, stock cm5.0.8 test4, 32mb linux swap
Click to expand...
Click to collapse
It's not a stupid question ... I have had the same Idea yesterday ...
I have googled about life spawn of the internal flash memory, but I haven't found any satisfactory answer yet.
Anywhere here who know how fast the internal flash is ?
What about write cycles and wear levelling ?
If it has no integrated wear levelling, swapping will kill the phone in a few days.
I think your enable swap has failed due to wrong permissions ... try to enable with a defered call in your userinit.
Update: I have checked my filesystem ... /cache has actually 29 MB free ... is it correct, that /cache is only used by OTA updates ? Probably we can create a priorized swap there in addition to ext.
Keep up the good work dude...this sounds great
TheGenesis said:
Navigon for example uses 14 MB of internal storage.
If you install such apps on a G1, you can quickly run out of space.
Low on cache space will considerably slow down the overall system performance.
I would not suggest to move the data of any installed apps, but from the big ones.
Apps like Navigation and 3D-Games are writing not frequently their data and if you own a G1 you are already swapping when using an Eclaid based ROM. So this wouldn't make a big difference in the livetime of your sdcard.
Perhaps someone could implement a partial move (e.g. only libraries) to avoid massive writes to the card ... and/or setting a threshold value, so apps with small data could not be moved.
For me, moving the apps in the Script above has given me additional 35 MB internal space and everything is still runnung "fast".
Click to expand...
Click to collapse
If navigon is using that much internal storage it's a very poorly written application. Low cache doesn't slow the system down, just apps that require such huge amounts of it.
I'm already swapping when using Eclair? Wrong, using swap on the sdcard is horrible, I'd never recommend it to anyone and I personally don't use it. I have 114 apps installed and my cache is 9% used, it seems to me like your apps aren't clearing their cache correctly, or they're just poorly written. The argument that 'you are already swapping so this wouldn't make a big difference in the livetime of your sdcard' is untrue, you are effectively doubling the amount of read/writes to the mmc, if not more, so the lifetime could potentially be cut in half, of course depending on use.
I'd say nice try, but this really just working around crap apps.
I use a 16 GB class 6 SD-Card with static wear-levelling.
Assuming that a standard-flash-nand-cell lasts about 10.000 write cycles, and my swap-write-turnover is currently about 1,6 GB per day, my SD-Card will last about 273 years (minus regular writes).
So I don't care about livetime.
Besides Navigon, there are many apps, that store their huge libraries to /data .... Games in most cases ... If you aren't using "bad written apps" its fine for YOU ... everyone else has to do some tweaks when installing some of them to the limited internal storage.
I have 278 apps installed and the only limit for me, is currently the free space on my sdcard.
If you haven't enabled swap since you have flashed your first Eclair ROM, you have probably never felt what is "speed" or you never need more than 1 app running simultaniously .... or you are using a different phone instead of a G1
You say "swapping to sd is horrible" ... I think you have used the wrong parameters ... when I diable swapping my system is lagging ... even when I work with one app the same time.
Did you enabled compcache while swapping ? Did you use a swapfile on FAT32 ? Is your swappinness levor 50 or above 60 ? Are you using al class 4 or slower sd-card ? Are you running heavy memory consuming apps without killing them from time to time ?
All these can turn a fast swapping system into an unusable phone.
You cannot enable swap and use the system like before.
Update:
I have copied 18 MB from cache to data and it tooks round about 18 seconds.
Same file from sdcard (FAT32) to sdcard tooks 6 seconds ...
I will use the sdcard
I should have listened to internal voice telling me not to argue with a fool cause people might not know the difference..
Something strange
I did some stupid (owing to absence of linux knowledge)
experiments regarding to swap_2_/
I believe if it possible it should be done through userinit/config
or smth during boot to enable r/w. give necc permissions etc.
I just used Swapper and RootManager
If I create .swp (it creates but does not work) in any place but /cache
it (just existence of this file) does system unstable, slow and unresponsive
in /cache or any existing or newly created folders inside /cache/ it's OK
before reboot when those new folders/files disappear
=
After a wile something happened with my phone (not a first or last time)
many apps caused FC, settings were lost etc
tried "fix uid missmatches" - dots filled out numerous screens
and after ~20min I decided to reboot
tried nandroid - same endless ....................................................
after the reboot I found no FCs but still missing settings for some apps
(sim_linked_data apps like CoPilot were OK) so I Titaniumed non-working apps
data (5-6 apps) and evrthng seems fine
=
BUT when I look at internal memory available
I find 73Mb (out of ~90) FREE
there was 53Mb free before the accident
Is that normal? And whats the limit?
I have my laps and brain scratched to find some application of this
=
just my experience
-compcache always gives me horrible slow phone - not using
-linux swap - best results compare to no_swap - allways use
Some apps are storing many data to /data and sometimes to /cache.
I think your restores have cleared some of them.
Try CacheMate instead of such manouvers
I have checked the write throughput using dd:
Internal Storage: 3,5 MB/s
Class 6 SD Card: 7 MB/s
... annoying ... USB to SD-Card is 3,5 MB/s and SD-Card via Card-Reader (PC) is about 9 MB/s.
Hey man, i had the same problem and decided to go a head and write a small tool that does exactly this, this is a UI tool that shows all the folders in /data/data (excluding system folders) and let you move your apps to your sd, you should have an APP2SD ROM installed with root (of-course) and sd card partitioned to EXT and FAT32.
Contact me if you want to check it out, i never found the time to publish it ([email protected])
hi,
I have tried that and it works, but...it works until reboot...
After reboot I don't see directory /sd-ext/data....
I don't know why it always been deleted....
Information
JIT Compiler
Going Deeper With Android 2.2′s JIT Compiler
by Quentyn Kennemer on May 26th, 2010 at 3:09 am
Before the official Froyo announcement at I/O, we’d learned that an Adobe employee’s phone was running at ridiculously high speeds. Most outlets quickly chalked this up to the possibility that Google implemented a Just-In-Time compiler for their Dalvik Virtual Machine.
Sure enough, they confirmed our suspicions at the keynote in San Francisco, and we even got a taste of real-world performance (there’s no “theoretical” performance hikes here, folks). We know what JIT does, but I’m sure there are a lot of non-developers out there that can’t understand how it’s able to provide the huge bump in performance for Dalvik.
Technical lead for Android’s Dalvik team – Dan Bornstein – signed in with a blog post over at the Android developers site going into a bit more details about what’s really going on beneath the hood to significantly improve performance of Android (in certain cases) without needing to touch the hardware. Thankfully, he puts it in plain English so you can get – at the least – a pretty basic understanding of what’s going on. Head over there now if you want to learn more about what makes your Froyo so sweet.
Linux-Swap
What is SWAP?
Swap space is an auxiliary storage, such as a portion of a hard-disk, which can be used as memory by the operating system when system RAM is insufficient. This is especially useful on systems with very little system RAM, such as most DD-WRT compatible routers, as it helps prevent the system from running out of memory when multiple background processes are installed.
The difference between Froyo A2SD, A2SD and A2SD+
One of the glories of using Android is having an SD card for storage rather than having internal-only memory. Thanks to Android’s Linux blood, you can even harness the power of the SD card beyond its usual file storage capabilities. On Android, your internal memory is precious and you do not want to install games and applications – especially those which are up to 50MB in sizes – on your internal memory just to make your phone run slower. Instead, you can install the applications on SD card and let your phone take care of the system apps instead, saving more internal memory and speeding up your phone.
The Partitions
Android natively supports fat32 partition. However, thanks to the Android community, support for swap and ext partitions can be enabled too. Depending on the ROM, some can support up to ext4, while others support up to ext3. The explanation about the differences between these partitions is indeed very lengthy and not part of our chapter today. What is important to know, is that Android has support for fat32, swap, ext2, ext3, and ext4 partition support.
Your SD card is by default formatted to fat32. In order to use swap and ext partitions, you need to repartition your SD card. There are several ways to do this, but the most common way is by using a custom recovery installed on your phone (ClockworkMod or AmonRA recovery). Swap is virtual memory which uses extra space on your SD card for virtual memory. However, since Android already has DalvikVM, swap is not really needed. I myself don’t use swap space on my SD partitions. Ext partition is extended partition which was the first ever type of partition created specifically for Linux. It is based of the standard UNIX file system and was designed to overcome the limitations of Minix file systems. Ext 2 is second extended partition, ext3 is third extended partition and ext4 is the fourth extended partition respectively.
Dalvik Virtual Machine
One of the best functions of Android has to be the Dalvik cache. Dalvik cache is a wonder from the point your Android starts up, runs, hibernates and all the way till you device shuts down. Dalvik cache collects the information about the installed applications and frameworks, and organizes them into a writeable cache. Under this writeable cache, it stores the “optimized” bytecode of the applications which is used by the applications themselves later for a smoother operation. This dalvik cache can grow immensely huge as more applications are installed on your phone. It is safe to wipe dalvik-cache. It will be rebuilt again when the phone boots. This also explains why your phone takes ages to start up for the first time. As for my Nexus One, having about 145 applications installed, it takes about 13 minutes to build the cache.
If you ever extract an APK installer file, you will always find a file named classes.dex. This is the file Dalvik finds to build the cache. What makes the process slow? APK is an archive (which is why you can open it up with an unarchiver such as WinRAR or 7-Zip). Being an archive, it provides limited write access to the files contained within and the fact that archives are compressed. Not to forget, APKs are encrypted archives too. Therefore, DalvikVM has to extract the classes.dex files and build the Dalvik table accordingly which makes it easier to write data on it too. With this collective set of data, the Android OS no longer needs to index the applications and find their classes.dex when the phone is already running. Instead, it will just look into one place, and will know what to do next. Nifty huh?
To know what is going on inside the Dalvik VM, you can read about it here.
Froyo A2SD (F-A2SD)
When Froyo was released to Android community, one of its new features was the A2SD implementation. F-A2SD uses fat32 partition natively for application storage. This means, all you have to do is just slot in your SD card and its all ready to go. There is no need to partition the SD card whatsoever. This was a great effort from Google to include A2SD to the Android OS as it gives you an option to choose which applications you want to move to SD card, and which you want to leave on internal memory – BUT – with a condition! If the application developer decides to protect his application and not include A2SD support, you would not be able to move it to SD card. One major problem that F-A2SD has is that it only uses fat32 partition. This way, when the SD card is mounted to the computer, the applications become inaccessible. Not only that, if the applications have widget support, the widgets are removed too when the SD card is mounted. This can become rather a hassle especially if you have to mount your computer several times in a day.
A2SD
The A2SD method is much more interesting. It harnesses the glory of ext partitions. This way, the applications (protected or non-protected), will all be installed on the SD card ext partition. The good thing about ext partition is that when you mount your SD card, the ext partition is NOT mounted together. This said, when your SD card is mounted, the applications will still be accessible and separated from the files and folders on your fat32 partition. On A2SD, the dalvik cache resides on the phone memory.
A2SD+
A2SD+ takes the A2SD one step further. Its pretty useless to have a 512MB A2SD capacity if your dalvik cache is still on phone memory and you have lots of applications installed. This is because the dalvik cache can become pretty huge and just by using HALF of your A2SD’s ext partition, your internal memory can become FULL because of dalvik. Therefore, in A2SD+, the dalvik cache is also moved to SD card. This way, your internal memory is free as a highway. However, remember that Dalvik cache is accessed very frequently. If you have a slow SD card, the overall performance might be affected. I recommend using a class 6 or class 10 SD card for the purpose.
So make your pick guys. In Android’s world, you always have choices. Depending on your needs, use the partition that satisfies you. Hope this article clears any misunderstanding that anyone might be having. Cheers~
Filesystems (post is for galaxy s but all android is ~ same in filesystems)
Reality behind RFS Lag
Background
All data is stored on an 8gb or 16gb MoviNAND chip, of which 2GB is ‘system data’, and the rest is for user storage. The MoviNAND is one of the first mobile ‘smart SSD’ chips. That means that the MoviNAND handles all operations such as data wear leveling, physical data lookup, as well as having it’s own internal buffers. This cleverness is both good… and very bad.
RFS
RFS has a fairly badly written driver, that will call an fsync on file close.
Basically, RFS runs in ‘ultra secure’ mode by default. This security may not be really needed – I personally don’t want it if it means enormous slow downs. It also doesn’t help data security if the system/app is holding a file open, only if it closes the file. The MoviNAND is also fairly smart, and appears to write it’s cache to disk before turning off, and also appears to have capacitors to keep it alive for a little bit of time in the event of a power cut.
SQLite
Most Android apps use SQLite – a fairly simple database that is easy to embed. Sqlite has ‘transactions’ – not real transactions, but a transaction in sqlite is where the database is locked for the duration of a database write, and multiple databases writes can be included in one transaction. At the end of a transaction, sqlite will call FSYNC on the database file, causing a possibly long wait while the MoviNAND does it’s thing. Certain applications will not bunch up writes into a single transaction, and will do all of their writes in new transactions. This means that fsync will be called again and again. This isn’t really a problem on most devices, as fsync is a very fast operation. This is a problem on the SGS, because MoviNAND fsync is very slow.
The various fixes and why they work
Native EXT4 to replace RFS (Voodoo)
By replacing RFS with EXT4, the ‘sync on fileclose’ problem is removed. The EXT series of filesystems is also more efficient at allocating information into blocks than RFS/FAT32 is. This means less real writes to MoviNAND, which means that the MoviNAND buffer should be smaller, and when a sync is called, fewer commands have to be run. When a sync is called on EXT4, it will still be very slow, as the MoviNAND’s sync is still slow.
Basically, EXT4 improves filesystem grouping which leads to less commands, and does not have the broken ‘sync on file close’ that RFS does. It will not heavily improve sqlite database access in certain apps, as the full fsync on transaction end will still have to go through MoviNAND, and will be slow.
When pulling out the battery, there is a chance to lose data that has been written to a file but has not yet been told to sync to disk. This means that EXT4 is less secure than RFS. However, I believe the performance to be worth the risk.
Loopback EXT2 on top of RFS (OCLF)
By creating a loopback filesystem of EXT2, the ‘sync on fileclose’ problem is removed as well. Since the Loopback File is never closed until the EXT2 is unmounted, RFS will not call fsync when a file in the EXT2 loopback is closed. Since a single large file is created on RFS instead of multiple small files, RFS is unable to mis-allocate the file, or fragment it. The actual allocation of filesystem blocks is handled by EXT2. As a note, care should be taken in making the large file on RFS – it MUST align correctly with the MoviNAND boundries, or operations will be slowed down due to double-disk accesses for files, etc. It is unknown whether OCLF is aligning this correctly (how to determine this? 4KB block size gives double the performance of 2KB block size, so it might be aligning it correctly already).
Loopback also has the benefit of speeding up Sqlite databases (at the expense of a transaction being lost in power outage, as it could still be in ram). As always, this is a performance tradeoff between data security when the battery is pulled out, and performance. When pulling a battery out while using the loopback filesystem, there is a chance to lose the last few seconds of database writes. In practice, this isn’t a huge deal for a mobile phone – most lost data will be resynced when the phone reboots. In my opinion, the performance is worth it because of the very slow speed of a sync on MoviNAND.
Loopback EXT2 on top of EXT4
All of the above for normal loopback EXT2 applies. In addition, when the loopback flushes data, it will be flushed to EXT4 instead of RFS. This will probably be better than flushing to RFS, as the RFS driver is not as well written as the EXT4 driver. The difference should not be very large, though.
Journaling
Journaling on an SSD is not required. Your data will not be lost, your puppy will not die. Here is a post made by Theodore Tso -http://marc.info/?l=linux-ext4&m=125803982214652&w=2
But there will be some distinct tradeoffs with
omitting the journal, including possibility that sometimes on an
unclean shutdown you will need to do a manual e2fsck pass.
Not using a journal is not a big deal, as long as you take care to do a full e2fsck pass when an unclear shutdown has occurred. This is the main reason for a journal – to prevent the need to do a full disk check, and instead the journal can be easily read, and the full disk check avoided.
EXT2 vs EXT4
EXT2 appears to work better on the SGS than EXT4. This is because EXT4 has more CPU overhead than EXT2. Journaling is also very bad on MoviNAND. Why? It appears to be the command buffer in the MoviNAND controller. A call to update the journal will use a command slot in the MoviNANDs buffer, that could otherwise have been used for a real disk write. This means that journaling on MoviNAND is a VERY expensive operation compared to journaling on a ‘dumb’ disk.
Well, you could technically use EXT4 and simply disable the high cpu and other features until you are left with EXT2, since EXT4 and EXT2 are basically the same thing.
At any rate, the difference between EXT4 and EXT2 is not very large, and there’s no need for flamewars over it – it comes down to a choice of ‘running’ performance vs ‘startup’ performance, with EXT2 edging out EXT4 for everyday speed, while EXT4 not required a long disk check at boot.
Future Work
Rewrite the firmware for the MoviNAND’s flash to handle fsyncs properly and not bring the system to it’s knees. I joke, but this is really the true solution.
Other solutions include hacking EXT’s fsync method to return instantly, and ensuring that the real fsync is called when the system shuts down. Or doing nothing, fsync is there for a reason, I guess, and would be fine if MoviNAND’s fsync wasn’t so very slow.
There is probably a lot of small details missing from this writeup. They’ll be updated when we learn more. Thanks for all the useful discussions and arguments, everyone!
Many Thanks to Dennis for this awesome information (officially posted here)
Nice cp, but you could take out everything about movienand, which is not present in g3 we only have onenand...
Ah, and don't forget to give credits or link to op...
FadeFx said:
Nice cp, but you could take out everything about movienand, which is not present in g3 we only have onenand...
Ah, and don't forget to give credits or link to op...
Click to expand...
Click to collapse
Of course Done.
kyrillos13 said:
Of course Done.
Click to expand...
Click to collapse
hi
very useful info for the members very nice of u added to the roll up thread
I don t know why I never read this. Very nice and interesting info.
thank you very much for this!!
What about the link2SDapp? Is this a good choice? I'm using it but think it has probs with the second partition.
Godyn said:
What about the link2SDapp? Is this a good choice? I'm using it but think it has probs with the second partition.
Click to expand...
Click to collapse
Better use app2sd scripts
Thx,
I used it, but it filled my ROM.
What app do you suggest?
I used app 2 SD https://market.android.com/details?id=com.a0soft.gphone.app2sd&feature=search_result
But that didn't solve it.
Hi everyone.
As you know, TRIM support in Android was introduced in version 4.3. But it was there already in the kernel. What Android developers did is just included the fstrim utility and made it run periodically in background.
I was a hard time thinking why my phone is so slow. Then I narrowed the problem down to internal storage slowness. It was causing the whole OS to crawl when writing something to /data partition. Android is a complex system, so it always writes something to /data even if you think it should not. This was causing lags and freezes, sometimes up to 10 seconds of unresponsiveness.
So I started reading about LagFix, Samsung's and Nexus's issues in the past. But those issues were of different nature. If you're using 4.4 ROM, you already have all the needed support in the kernel and in the OS.
My problem was that the /data partition had ext4 block size set to 4096. Everything else like /preinstall and /system had block size of 1024.
I manually ran fstrim -v on these partitions and it successfully trimmed big amounts of bytes. On /data partition it trimmed 0 always. So I started searching and found out that only the block size is different between /system and /data.
I had backed up /data, reformatted it to to 1024 block size, restored and checked fstrim again. Now it successfully trimmed A LOT of bytes. It started working. After several minutes the phone ran much quicker - it was flying actually, no lags at all
So check your /data block size by running 'df' command in terminal. If it is NOT 1024, then you might be affected by fstrim doing nothing! I am not going to tell how important the trimming is for SSDs and EMMCs.
You can also check by running this under root:
fstrim -v /data
If it tells it trimmed 0 bytes, the TRIM is not working either.
I try this on my very old Atrix 2, my /system already 1024 except /data and /cache
When I ran the command, it said that "fstrim operation not supported on transport endpoint" (Mokee 4.4.4 on system 2)
Then I wiped all and installed a fresh Mokee 4.4.4 on system 1, now I can trim my /data but it's still show 2048 when I execute df command and /cache is still not supported
Hello all,
I look after a fleet of about 70 of these phones....Mostly on JB. Over the last 2-3 weeks most seem to be slowing down considerably. I struggle to see why/if this trim issue would affect a lot of phones in such a narrow window of time after over 4 years, but will look into this more.
John
'fstrim -v /data' should tell how much has it trimmed - if it tells 0 bytes - then TRIM is not working by some reason.