Related
I've seen a lot of people complaining about not being able to boot after using A2SD for a while and I think it's definitely necessary to make clear the danger in using A2SD and some ways to prevents them.
Do not move /data/data to SD.
You can safely move /data/app and /data/app-private to SD. Be cautions with /data/dalvik-cache. (See below)
Make sure the ext2 partition are mounted with noatime and nodiratime (rom maker's job)
Do regular file system checks (using Linux)
Back up your ext2 partition and redo them regularly
Here're the theories:
(NAND) Flash memory has two serious limitations when used to store frequently changing data: you cannot do random write unless you erase a whole page first and any bit can only be written a limited number of times (typically a couple millions). When one bit in a page is detected to be faulty, you lose the whole page. (More details: http://en.wikipedia.org/wiki/Flash_memory#Limitations )
Most mobile os vendors overcome this problem by using special file systems, namely yaffs and jffs, which arrange files according to page size and do write operations only when necessary. They also provide journals so hardware faults could be reliably detected and corrected.
EXT2, which is the file system used in A2SD, DO NOT have these features. It may stupidly allow several pages to be erased hundreds of times of just to write some small files into it. Things are made even worse because some fixed part of the file system (inode table and bitmap) has to be written EVERYTIME a file operation is done. The will accelerate the wearing of those pages and when they become inaccessible, you lose the whole file system.
A2SD would work fine if you only move /data/app to it, because the application files are never modified. It is a completely different story for /data/data because the sqlite databases in there are modified almost every other second!! (And I suspect the OS commit them to disk very often to ensure data integrity.)
In addition, android does not have system check tools for ext2, so it will not be able to detect any problem with the file system until it’s too late.
Edit:
I am not sure how frequent the system updates files in dalvik-cache, but I would say you only move it when you are running out of space in /data.
(Also changed title)
billc.cn said:
I've seen a lot of people complaining about not being able to boot after using A2SD for a while and I think it's definitely necessary to make clear the danger in using A2SD and some ways to prevents them.
Do not move /data/data to SD.
Do regular file system checks (using Linux)
Back up your ext2 parition and redo them regularly
Here're the theories:
(NAND) Flash memory has two serious limitations when used to store frequently changing data: you cannot do random write unless you erase a whole page first and any bit can only be written a limited number of times (typically a couple millions). When one bit in a page is detected to be faulty, you lose the whole page. (More details: http://en.wikipedia.org/wiki/Flash_memory#Limitations )
Most mobile os vendors overcome this problem by using special file systems, namely yaffs and jffs, which arrange files according to page size and do write operations only when necessary. They also provide journals so hardware faults could be reliably detected and corrected.
EXT2, which is the file system used in A2SD, DO NOT have these features. It may stupidly allow several pages to be erased hundreds of times of just to write some small files into it. Things are made even worse because some fixed part of the file system (inode table and bitmap) has to be written EVERYTIME a file operation is done. The will accelerate the wearing of those pages and when they become inaccessible, you lose the whole file system.
A2SD would work fine if you only move /data/app to it, because the application files are never modified. It is a completely different story for /data/data because the sqlite databases in there are modified almost every other second!! (And I suspect the OS commit them to disk very often to ensure data integrity.)
In addition, android does not have system check tools for ext2, so it will not be able to detect any problem with the file system until it’s too late.
Click to expand...
Click to collapse
these are very good points. Actually come to think of it we better mount the ext2 partition with noatime. Because right now every read will wear the flash down. flash storage is really not meant for ext2 filesystem.
knaries2000 said:
these are very good points. Actually come to think of it we better mount the ext2 partition with noatime. Because right now every read will wear the flash down. flash storage is really not meant for ext2 filesystem.
Click to expand...
Click to collapse
In JF1.5 build it is mounted with noatime, i believe:
#mount
/dev/mmcblk0p2 on /system/sd type ext2 (rw,noatime,nodiratime,errors=continue)
But I totally agree with the point of the thread /data/data should not be moved to sd. Not that it's only dangerous (as described), I even don't see any advantages of it.
Dimath said:
In JF1.5 build it is mounted with noatime, i believe:
#mount
/dev/mmcblk0p2 on /system/sd type ext2 (rw,noatime,nodiratime,errors=continue)
But I totally agree with the point of the thread /data/data should not be moved to sd. Not that it's only dangerous (as described), I even don't see any advantages of it.
Click to expand...
Click to collapse
really. that's good, but I am running haykuro's build right now and it is not mounted with noatime. I will have to change the init script.
I'm using JF's 1.5 A2SD build, and I'm pretty certain I moved over /data/data. I didn't really ask myself the question when doing it, but what exactly is stored in /data/data? Is there a command I can run to move it back off the SD card?
Are ext2 and fat the only supported file systems in the android kernel? If not maybe it would be best to move to a wiser file system.
Rekna said:
Are ext2 and fat the only supported file systems in the android kernel? If not maybe it would be best to move to a wiser file system.
Click to expand...
Click to collapse
Yeah, why can't we use yaffs etc?
Dimath said:
Yeah, why can't we use yaffs etc?
Click to expand...
Click to collapse
i don't think most partitioners support yaffs
i know the partition manager on ubuntu 8.10 doesn't
tubaking182 said:
i don't think most partitioners support yaffs
i know the partition manager on ubuntu 8.10 doesn't
Click to expand...
Click to collapse
Indeed, and I'm fairly sure that's why it hasn't been done. But keep in mind several million write cycles is a heck of a lot and apps only really write their data caches occasionally, so it'll likely be a while before anything bad happens(on the scale of years) so this is slight sensationalism. Actually the term "cache" is a slight misnomer here since it's really just storage the apps use for temporary data. If it were used as some kind of extended RAM or a real cache then I could see problems but with what it's used for it should be a non-issue.
billc.cn said:
Here're the theories:
Click to expand...
Click to collapse
SD cards are a form of removable flash that have their own write controllers. In most cases, the write controllers also perform wear levelling. This means even if a program writes to the same file on the sd repeatedly, each time it writes, it is not writing the same physical location on the flash. The reason for this is because due to the way flash works, entire blocks have to be erased before they can be rewritten. To make writing faster, the memory controller keeps a list of empty blocks that are ready to use. When a file is changed, the entire file with the new changes is written to a new memory block and then the old block with obsolete data is then reset to zeros (data deleted).
That said, I still think A2SD is a red herring that only contributes to newbies spamming these forums.
jashsu said:
SD cards are a form of removable flash that have their own write controllers. In most cases, the write controllers also perform wear levelling. This means even if a program writes to the same file on the sd repeatedly, each time it writes, it is not writing the same physical location on the flash. The reason for this is because due to the way flash works, entire blocks have to be erased before they can be rewritten. To make writing faster, the memory controller keeps a list of empty blocks that are ready to use. When a file is changed, the entire file with the new changes is written to a new memory block and then the old block with obsolete data is then reset to zeros (data deleted).
That said, I still think A2SD is a red herring that only contributes to newbies spamming these forums.
Click to expand...
Click to collapse
Which is exactly why I made my new method that's able to deal with a lot of user mistakes and can be incorporated into ROMs to make it take almost 0 user effort
Dimath said:
Yeah, why can't we use yaffs etc?
Click to expand...
Click to collapse
yaffs on SD cards can in some common cases invalidate the wear leveling in hardware that SD cards do, as they so the wear leveling in software.
You are simply understimating ROM makers, SD cards are different from the internal flash in that they do auto wear leveling, that's why you can put common filesystems like FAT which have statically placed allocation tables and writes to the same logical sector will always land on very different places in the card every time.
Yes, noatime will help a lot as it will _reduce_ writes to your SD card.
ext2 is not journalled so it will have less writes too than ext3 or any other journalled filesystem.
So ROM makers are already doing a good job, don't understimate them please.
Anyone have a working compcache module for 1.5 builds? The previous JF builds used to have it, but I don't see one for 1.5.
Thanks.
I just tried compcache. Epic fail. I'm not sure if it's just broken horribly on ARM or what, but soon as I do "swapon /dev/ramzswap0", everything starts crashing. No kernel panics, just apps dying. I tried with and without a disk-based swap as a backing store (I have 3 partitions on my SD card).
I tried using the loop method earlier to create a swap on hero on my sdcard. it was successful but it's still painfully slow paging, unfortunately.
afflaq said:
I tried using the loop method earlier to create a swap on hero on my sdcard. it was successful but it's still painfully slow paging, unfortunately.
Click to expand...
Click to collapse
I have swap on actual SD card partition right now, and it's actually pretty snappy. Tons of **** running
cyanogen said:
I have swap on actual SD card partition right now, and it's actually pretty snappy. Tons of **** running
Click to expand...
Click to collapse
ugh mine bogged down. are you overclocking? It seems like I notice a *LOT* of problems when things start dipping into the swap. Ultimately I'll probably be back on one of your builds tomorrow..just get the goddamned lockscreen from 6.0r1 or help me port the framework over so i can have it on my personal build >=)
I tried it few days ago, with the same result. BTW, file-backed swap should be just as fast as a swap partition.
Hey guys, could you post some details about what you are doing so that we can get an idea of what works and what doesn't?
* What build are you on?
* Where did you get the compcache module? Did you compile it? If not, what kernel was it compiled for?
* What version of compcache did you use?
* What commands (options) did you use to install the module?
* What size device did you use?
* Did you use a file backing store?
* Did you mess with the swapiness setting? If so, what did you set it to?
I run the Dude's 1.2a build. I compiled compcache 5.3 against the android-msm-2.6.27 kernel using the prebuilt gcc 4.3. I tried many combinations of the settings, but it's always the same, processes get bus errors or segfaults... The modules I used are attached.
I can't wait until we figure this out, I think any performance boost or memory increase is definately needed with newer features and roms being released weekly
So how do we put a usable pagefile swap partition on Hero? I've been looking all day how to do it, but I'm not finding much.
What did you use to do this? Would love to set it up myself.
cyanogen said:
I have swap on actual SD card partition right now, and it's actually pretty snappy. Tons of **** running
Click to expand...
Click to collapse
http://www.cyrket.com/package/lv.n3o.swapper
I'd rather use squashfs with unionfs to compress /bin /and lib to save space and optimize speed, then use more device memory to create a swapfile instead of using an sd which would be slower. Although someone needs to set up a build environment and build a working compcache module and use that with my ideas above, once done android should run how it should have since october
Defcon
defconoi said:
I'd rather use squashfs with unionfs to compress /bin /and lib to save space and optimize speed, then use more device memory to create a swapfile instead of using an sd which would be slower. Although someone needs to set up a build environment and build a working compcache module and use that with my ideas above, once done android should run how it should have since october
Defcon
Click to expand...
Click to collapse
I nominate you.
But I have my app2sd and I have a bit of free space left on the internal. How much space would a swap file need? And also, what would be faster, the dalvik cache, or a swap file on internal?
From my understanding swapper doesn't work on 1.5.
dixxa said:
http://www.cyrket.com/package/lv.n3o.swapper
Click to expand...
Click to collapse
bad understanding since it works and I'm using it
Sure it runs and installs fine but after looking at it for an hour it never actually used the swap. It always stays at 0 bytes used.
dixxa said:
bad understanding since it works and I'm using it
Click to expand...
Click to collapse
currently using cyanogen rom and I'm sure it use it !
40Mo used on 75 available...
dixxa said:
currently using cyanogen rom and I'm sure it use it !
40Mo used on 75 available...
Click to expand...
Click to collapse
Let us know if you do not experience too many force closures eventually. I do whenever I use swap with JF1.5. I keep trying it every now and then just to eventually be disappointed.
defconoi said:
I'd rather use squashfs with unionfs to compress /bin /and lib to save space and optimize speed, then use more device memory to create a swapfile instead of using an sd which would be slower. Although someone needs to set up a build environment and build a working compcache module and use that with my ideas above, once done android should run how it should have since october
Defcon
Click to expand...
Click to collapse
Could you please explain what you are suggesting again, I can't understand it. For starters, there is no /bin on android, which lib are you referring to? And what kind of space are you trying to save by doing this (which partition? RAM?)? How would this optimize speed?
How does unionfs come into play in your scenario?
If you use the device memory (I suppose you mean flash, not RAM) for swap you will be wearing out your internal flash fairly quickly which, unlike an sdcard, cannot be replaced easily.
Compcache cannot use a file as backing store yet, only partitions.
If you have a great idea which I do not understand, please accept my apologies, but what you wrote sounds like you just chained together a bunch of buzzwords without understanding them. If you have a potentially viable idea, please enlighten us so that we can implement it.
Im trying out a theory. I running JAC Hero 2.3 and wanna try running the swapfile on the /data partition. I moved app_s over to /system/sd and used swapper app to create the swapfile but android doesnt seem to use it. when i run free It says Total/used/free are all 0 but the swap file is 34 meg. I tried putting it in a sub folder and chown root.root on the folder before setting up and still no luck. Anyone have any ideas why it wont work on /data or how to get it to work on /data?
Once you run swapper go into the setting and change the location of your swap file from /sdcard/swapfile.swp to /system/sd/swap.swp ( see sxfx post[url]
you don't want to do that. Swapping involves a lot of writing and erasing, it'll wear out your internal chip and you'll start experiencing reduced capacity, write cycles for nand are even less than for flash memory!
jubeh said:
you don't want to do that. Swapping involves a lot of writing and erasing, it'll wear out your internal chip and you'll start experiencing reduced capacity, write cycles for nand are even less than for flash memory!
Click to expand...
Click to collapse
not to mention, swapon as implemented (on every build i've tried) doesn't work on a swapfile stored on a yaffs2 partition
jubeh said:
you don't want to do that. Swapping involves a lot of writing and erasing, it'll wear out your internal chip and you'll start experiencing reduced capacity, write cycles for nand are even less than for flash memory!
Click to expand...
Click to collapse
Actually /sdcard/ is your fat32 portion of your sd card, /system/sd/ is the ext2/3 of your sd card.
Also I have done a bunch of research on this write/read fiasco just because of android.
And even if you set a swap file to your SD sure it will shorten your life of the card but it will still last you at least 2 years.
I have been using USB devices on linux as swap locations forever now and I still have thumb drives that have been used and abused for months and months as a swap place and they are still pulling strong.
As for the internal chip, Im not sure what you mean by that?
Even if you could put swap on the internal flash, its not going to be faster.
Putting swap on the internal flash will make things slower as the internal flash is about 3 to 5 times slower than a class 6 sdcard.
you're comparing using a flash drive for swap in a full blown computer that probably has around 1-4gb of ram, the swap file is hardly ever touched, unless you're running a lot of applications at a time. Dream has only 90 mb available to dalvik, and rosie is a big fat... lady... plus linux/dalvik manage memory in a different way, so files are often dropped to swap and they dont stay there for long (maybe in a 256 swap, but not in a 32 mb one).
the OP also is talking about moving his swap to his internal storage (chip, nand, whatever), as he thinks it's having no effect working from the sd card. Personally, i think that both a2sd and swapper are flawed. They're overcompensating for an os that was not meant to run on that device, and the real work should be in porting (as in developing, not just file-swapping as most "devs" do here) a launcher app that we can feel comfortable with running on a stock android system with stock (or slightly improved) libraries and that we can call comparable to rosie. Just look at ahome or dxtop or openhome, they're good, solid, great looking home replacements that work as well or better than rosie, but they run out of the stock libraries. I wonder why nobody has made a free, open source home replacement app yet
jubeh said:
you're comparing using a flash drive for swap in a full blown computer that probably has around 1-4gb of ram, the swap file is hardly ever touched, unless you're running a lot of applications at a time. Dream has only 90 mb available to dalvik, and rosie is a big fat... lady... plus linux/dalvik manage memory in a different way, so files are often dropped to swap and they dont stay there for long (maybe in a 256 swap, but not in a 32 mb one).
the OP also is talking about moving his swap to his internal storage (chip, nand, whatever), as he thinks it's having no effect working from the sd card. Personally, i think that both a2sd and swapper are flawed. They're overcompensating for an os that was not meant to run on that device, and the real work should be in porting (as in developing, not just file-swapping as most "devs" do here) a launcher app that we can feel comfortable with running on a stock android system with stock (or slightly improved) libraries and that we can call comparable to rosie. Just look at ahome or dxtop or openhome, they're good, solid, great looking home replacements that work as well or better than rosie, but they run out of the stock libraries. I wonder why nobody has made a free, open source home replacement app yet
Click to expand...
Click to collapse
You make a very good point about computer having 1-4gigs of ram and not needing a swap file.
Honestly I almost never run a swap file on a computer that has more then 1gig of ram.
Plus90% of the time when I do run a swap file is when Im running a live distro of linux of a cd, but also 90% of the time I run the distro from the flash drive instead of cd cause its much faster.
So now imagine how many reads/writes I abuse that flash drive with by running a full OS on it.
Also I don't understand how a2sd or swapper is flawed? Just because they do their job? I mean it's not really our fault that they made the G1 with a little less memory spaces then we would like it to have.
But that's exactly why we have tools like swapper and a2sd. Plus no one is really forced to run these roms on the G1 phone and those of us that do realize that we have to take extra steps in creating tools to help it.
And that's not only true for the G1 but anywhere in the computer world these days.
dwang said:
Even if you could put swap on the internal flash, its not going to be faster.
Putting swap on the internal flash will make things slower as the internal flash is about 3 to 5 times slower than a class 6 sdcard.
Click to expand...
Click to collapse
I'd like to spread swap over the sd and internal storage if possible -- should make paging a lot less evident if priorities are set up properly.
dwang said:
Even if you could put swap on the internal flash, its not going to be faster.
Putting swap on the internal flash will make things slower as the internal flash is about 3 to 5 times slower than a class 6 sdcard.
Click to expand...
Click to collapse
internal memory is faster. class 6 guarantees 6mbs read/write times but doesnt mean the bus can support those times. if you run a test copying something to /system/sd and to /data you will see /data is faster. as for those saying it will degrade the internal memory, that is not the case being the flash memory inside is designed to have much much much more read/write cycles. Think about it in a stock G1, this is where dalvik-cache is writen to as well as email, sms, user settings, cache for browser and uTube. Do not confuse internal flash memory to sd card flash memory
Are you sure about that? This guy has some test results and it indicates that a class 6 sdcard is much faster than the internal flash.
http://forum.xda-developers.com/showpost.php?p=4059520&postcount=15
MonkySlap said:
internal memory is faster. class 6 guarantees 6mbs read/write times but doesnt mean the bus can support those times. if you run a test copying something to /system/sd and to /data you will see /data is faster. as for those saying it will degrade the internal memory, that is not the case being the flash memory inside is designed to have much much much more read/write cycles. Think about it in a stock G1, this is where dalvik-cache is writen to as well as email, sms, user settings, cache for browser and uTube. Do not confuse internal flash memory to sd card flash memory
Click to expand...
Click to collapse
dwang said:
Are you sure about that? This guy has some test results and it indicates that a class 6 sdcard is much faster than the internal flash.
http://forum.xda-developers.com/showpost.php?p=4059520&postcount=15
Click to expand...
Click to collapse
That's interesting. Going to have to try that test out. I just noticed when xfering stuff to /data it was faster then to /system/sd. If it is then touche my friend.....touche
I know we are talking about a mobile OS here, but i remember the rule of thumb for Windows pagefile being 1.5 times the amount of ram...I am assuming the pagefile memory is similiar to the linux-swap. I could be wrong. So i did some searching and i found this on linux-swap.
The key question is how much? Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.
[quote/]
So now we have 97,876 kb of memory showing in free. So should the linux swap be 48,938 kb or 48 mb instead of 32? Or because we have less then 128mb of memory should we follow the rule of giving it 1.5 (192 mb) times the amount of memory?
I am about to repartition my SDCard following the 1.5x rule. I have to research which commands to use to monitor swap...anyone know them off hand?
Edit:
I found this about making a swapfile instead of swap partition:
Swap file
As well as the swap partition, Linux also supports a swap file that you can create, prepare, and mount in a fashion similar to that of a swap partition. The advantage of swap files is that you don't need to find an empty partition or repartition a disk to add additional swap space.
To create a swap file, use the dd command to create an empty file. To create a 1GB file, type:
Code:
dd if=/dev/zero of=/swapfile bs=1024 count=1048576
I think the code needs to be changed. i am testing a few things now. I believe if=/dev/zero needs to be changed to something more amoung the lines of /dev/block/mmcblk0. Everything i try is getting a "dd: can't open '/swapfile': Read-only file system" I am assuming i have to mount it as rw but im not 100% on how to do that. Any help is appreciated.
Click to expand...
Click to collapse
/swapfile is the name of the swap file, and the count of 1048576 is the size in kilobytes (i.e. 1GB).
Prepare the swap file using mkswap just as you would a partition, but this time use the name of the swap file:
Code:
mkswap /swapfile
And similarly, mount it using the swapon command: swapon /swapfile.
The /etc/fstab entry for a swap file would look like this:
/swapfile none swap sw 0 0
Click to expand...
Click to collapse
Interesting, let me know if this helps with speed, I will definitely be using it.
Thats the plan =) I am going to repartition the sdcard now with the larger swap partition and then maybe down the line...we can switch from using a 3rd partition to using the swapfile instead.
I'm considering doing the same thing. What you posted makes sense. A bigger swap partition would free up ram for more active applications. Since I'm using a Hero Rom that would probably help haha.
I've already test this theory at swap partitions beyond 32MB including 48, 64, 192, up to 1gb at varying swap partition sizes. Also I tested multiple swappiness settings (20,30,40,60,80,100). The conclusion, bigger is not always better. The larger the swap partition became, the more the system had to swap out then back into memory, which took a hell of a lot of time (upwards of 5 minutes just to go from browser to home). General rule of thumb for swap files, just make a swap file the size that is enough, not one that is huge.
Good luck with your method of swap (which already exists, through methods like the app swapper), I know there are threads floating around where people use swap files, and end up switching to compcache or a swap partition because of higher stability.
andonnguyen said:
I've already test this theory at swap partitions beyond 32MB including 48, 64, 192, up to 1gb at varying swap partition sizes. Also I tested multiple swappiness settings (20,30,40,60,80,100). The conclusion, bigger is not always better. The larger the swap partition became, the more the system had to swap out then back into memory, which took a hell of a lot of time (upwards of 5 minutes just to go from browser to home). General rule of thumb for swap files, just make a swap file the size that is enough, not one that is huge.
Good luck with your method of swap (which already exists, through methods like the app swapper), I know there are threads floating around where people use swap files, and end up switching to compcache or a swap partition because of higher stability.
Click to expand...
Click to collapse
Ahh good to know, I tried searching for linux swap and not much came up so i didn't see any threads on this. The setup seemed to run faster, but that is part of Drizzy's v2.7 updates.
What commands did you use to watch the swap partition?
I was actually one of the first people I knew to put swap on the phone but I happen to know that the swapper application does that plus more! and even back in the day when I did it I couldn't use to much or my phone would lag! I recommend either 32mb or 64 mb of swap and just default swappiness but if you use swap I also recommend to underclock your phone to 383!
kickfliprock13 said:
I was actually one of the first people I knew to put swap on the phone but I happen to know that the swapper application does that plus more! and even back in the day when I did it I couldn't use to much or my phone would lag! I recommend either 32mb or 64 mb of swap and just default swappiness but if you use swap I also recommend to underclock your phone to 383!
Click to expand...
Click to collapse
I would only underclock if ur running hero...this seems to be the only rom underclocking really benefits you
damnitpud said:
Ahh good to know, I tried searching for linux swap and not much came up so i didn't see any threads on this. The setup seemed to run faster, but that is part of Drizzy's v2.7 updates.
What commands did you use to watch the swap partition?
Click to expand...
Click to collapse
in terminal type:
$free
it will show a section for swap. if it reads all zeros then the swap isn't being used.
Here's an example of my phone:
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
You can see the difference of the swap being used, and the swap not being used.
damnitpud said:
I know we are talking about a mobile OS here, but i remember the rule of thumb for Windows pagefile being 1.5 times the amount of ram...I am assuming the pagefile memory is similiar to the linux-swap. I could be wrong. So i did some searching and i found this on linux-swap.
The key question is how much? Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.
[quote/]
So now we have 97,876 kb of memory showing in free. So should the linux swap be 48,938 kb or 48 mb instead of 32? Or because we have less then 128mb of memory should we follow the rule of giving it 1.5 (192 mb) times the amount of memory?
I am about to repartition my SDCard following the 1.5x rule. I have to research which commands to use to monitor swap...anyone know them off hand?
Edit:
I found this about making a swapfile instead of swap partition:
Swap file
As well as the swap partition, Linux also supports a swap file that you can create, prepare, and mount in a fashion similar to that of a swap partition. The advantage of swap files is that you don't need to find an empty partition or repartition a disk to add additional swap space.
To create a swap file, use the dd command to create an empty file. To create a 1GB file, type:
Code:
dd if=/dev/zero of=/swapfile bs=1024 count=1048576
/swapfile is the name of the swap file, and the count of 1048576 is the size in kilobytes (i.e. 1GB).
Prepare the swap file using mkswap just as you would a partition, but this time use the name of the swap file:
Code:
mkswap /swapfile
And similarly, mount it using the swapon command: swapon /swapfile.
The /etc/fstab entry for a swap file would look like this:
/swapfile none swap sw 0 0
Click to expand...
Click to collapse
We switched from the swap file to the swap partition because it turned out that the swap partition was more stable wand was faster too. Also, we didn't have to set it up each time we flashed a new ROM and we could just have it activated by default. The problem with the swap file is that if you place it on the fat32 portion of your sdcard and then mount it to the computer, all hell breaks loose on your phone. I corrupted my card and was forced to re-partition everything because of that. As it is, I'm happy with my 32MB of swap with default swap of whatever the ROM dev chooses.
Click to expand...
Click to collapse
All this info is very good. Thank you everyone who replied. A lot of this info must be stashed away in an older thread somewhere no?
But after all this im going to give up on the swapfile and swap size. I do have a question for some of you that know more about this then me.
I want to run Hero after getting used to it. but after a few hours of running it, it slows down. i have followed every step the ROM devs have said and it still slows down. So i am wondering now, i noticed that Drizzy's ROM has an app and app_s folder in data. I am assuming that app_s is all the apps that go to the sdcard. But even a class 6 card is slower then the internal storage. So maybe moving apks like Rosie and some system apks to internal storage could speed up the way it Hero runs...am i wrong here or going down a path others have already tried?
damnitpud said:
All this info is very good. Thank you everyone who replied. A lot of this info must be stashed away in an older thread somewhere no?
But after all this im going to give up on the swapfile and swap size. I do have a question for some of you that know more about this then me.
I want to run Hero after getting used to it. but after a few hours of running it, it slows down. i have followed every step the ROM devs have said and it still slows down. So i am wondering now, i noticed that Drizzy's ROM has an app and app_s folder in data. I am assuming that app_s is all the apps that go to the sdcard. But even a class 6 card is slower then the internal storage. So maybe moving apks like Rosie and some system apks to internal storage could speed up the way it Hero runs...am i wrong here or going down a path others have already tried?
Click to expand...
Click to collapse
Actually class 6 is faster than internal storage, someone benchmarked the two in another thread that I can't currently find. Other people have tried already, it's probably best to read the first post of each hero thread, and make connections as to what dev's are doing, and what they're not doing. Their latest builds tend to be improvements upon previous builds, so if rosie and all the apk's are on the sdcard, its probably there for a reason whether it be speed or stability.
andonnguyen said:
Actually class 6 is faster than internal storage, someone benchmarked the two in another thread that I can't currently find. Other people have tried already, it's probably best to read the first post of each hero thread, and make connections as to what dev's are doing, and what they're not doing. Their latest builds tend to be improvements upon previous builds, so if rosie and all the apk's are on the sdcard, its probably there for a reason whether it be speed or stability.
Click to expand...
Click to collapse
I am going to research class 6 speeds, I can't see a memory card being faster then internal storage. but I've been wrong in the past...many times heh =)
Yeah the Drizzy ROM is a huge improvement in speed then the others...But there is still quite a bit of lag. I hope they can get Hero running as smooth as JF or Cyanogen's ROMs.
andonnguyen said:
in terminal type:
$free
it will show a section for swap. if it reads all zeros then the swap isn't being used.
Here's an example of my phone:
Click to expand...
Click to collapse
your free shows 97860 total ram, mine with cyanogen 3.9.1 shows 97880 totaly ram, but the phone is supposed to have 192megs of ram. what gives?
Sirus20x6 said:
your free shows 97860 total ram, mine with cyanogen 3.9.1 shows 97880 totaly ram, but the phone is supposed to have 192megs of ram. what gives?
Click to expand...
Click to collapse
Mine in TouchFlo 3D 2.7 shows 98328 for ram. The phone has 192mb of memory but not all of it is user accessible. Not sure why.
Edit:
Just found this on a site:
free will report slightly less memory as being in a computer than the actual total. This is mostly because the kernel (i.e., the core of the operating system) cannot be swapped out (i.e., the kernel always remains in main memory while the computer is in operation), and thus the memory that it occupies can never be freed. There can also be regions of memory that are reserved for other purposes, according to the specific system architecture.
Sirus20x6 said:
your free shows 97860 total ram, mine with cyanogen 3.9.1 shows 97880 totaly ram, but the phone is supposed to have 192megs of ram. what gives?
Click to expand...
Click to collapse
Some of the ram has been allocated for the function of phone and OS running
a 100 meg kernel? my computers kernel is less than 3 megs lol. I imagine that a few megs are saved for a frame buffer, and the os takes up most of the rest but still that's almost exactly half the phones ram. it seems like 100 megs is rather large for a phone OS
I remember windows XP taking half of the basic prebuilt computer's RAM when it first got released. it's not impossible. This IS the first hardcore phone os we've seen like android
So now we have 97,876 kb of memory showing in free. So should the linux swap be 48,938 kb or 48 mb instead of 32? Or because we have less then 128mb of memory should we follow the rule of giving it 1.5 (192 mb) times the amount of memory?
Click to expand...
Click to collapse
In all of the linux systems I have built over the years I have always used 1.5x as the rule... possibly more if I expected that I would upgrade RAM later on. I have always used a swap partition(keeps things nice and clean), but the ability to swap to a file has been around for years. All of the material I've read on the subject maintains that the file is just as fast as a swap partition, though I am willing to bet its lsightly slower due to FS overhead, though this out to be minimal - possibly a bigger deal on a slow embedded system like the G1..
I am going to research class 6 speeds, I can't see a memory card being faster then internal storage. but I've been wrong in the past...many times heh =)
Click to expand...
Click to collapse
Its all in the type of flash chip used and its interface... the interfaces are likely fairly equal(if not exactly the same). The speed of the actual media is entirely dependent on what you spend on it... in all cases for use as swap the speed is SLOW.
So maybe moving apks like Rosie and some system apks to internal storage could speed up the way it Hero runs...am i wrong here or going down a path others have already tried?
Click to expand...
Click to collapse
Sounds reasonable if the internal flash is faster than the flash on the sd card.
What commands did you use to watch the swap partition?
Click to expand...
Click to collapse
You might be interested in the output of the vmstat command(not sure if its available on android) and the file /proc/meminfo (similar to the free command).
You can also watch disk(flash) io in real time with the iostat command....
I did some testing on a random 1GB un-classed sd card I have here and I got 9MB/s writes and 12MB/s reads dding the entire card in Ubuntu.... I used this on the G1 with a swap partition and it lead to lots of force closes and long delays. Flash media will not be an ideal media for a swap file until read/write speeds approach 40MB/s or better and even then will still cause slow operation while swapping(just like a PC).
Based on the many factors with the speed of the platform.... the G1 either needs more actual RAM(find out whats up with the piece we can't get to), or less apps running simultaneously to maintain snappy speed.... and maybe some less heavy apps... 12 MB's of allocated memory is a LOT on a platform with only 96MB available to begin with....
IIRC someone ran benchmarks and found the internal flash mem was almost equivalent to a class 4 card (approx 4 MBps), this particular class 6 card im running right now is almost 8 MBps nearly double the speed of the internal flash memory
Information
JIT Compiler
Going Deeper With Android 2.2′s JIT Compiler
by Quentyn Kennemer on May 26th, 2010 at 3:09 am
Before the official Froyo announcement at I/O, we’d learned that an Adobe employee’s phone was running at ridiculously high speeds. Most outlets quickly chalked this up to the possibility that Google implemented a Just-In-Time compiler for their Dalvik Virtual Machine.
Sure enough, they confirmed our suspicions at the keynote in San Francisco, and we even got a taste of real-world performance (there’s no “theoretical” performance hikes here, folks). We know what JIT does, but I’m sure there are a lot of non-developers out there that can’t understand how it’s able to provide the huge bump in performance for Dalvik.
Technical lead for Android’s Dalvik team – Dan Bornstein – signed in with a blog post over at the Android developers site going into a bit more details about what’s really going on beneath the hood to significantly improve performance of Android (in certain cases) without needing to touch the hardware. Thankfully, he puts it in plain English so you can get – at the least – a pretty basic understanding of what’s going on. Head over there now if you want to learn more about what makes your Froyo so sweet.
Linux-Swap
What is SWAP?
Swap space is an auxiliary storage, such as a portion of a hard-disk, which can be used as memory by the operating system when system RAM is insufficient. This is especially useful on systems with very little system RAM, such as most DD-WRT compatible routers, as it helps prevent the system from running out of memory when multiple background processes are installed.
The difference between Froyo A2SD, A2SD and A2SD+
One of the glories of using Android is having an SD card for storage rather than having internal-only memory. Thanks to Android’s Linux blood, you can even harness the power of the SD card beyond its usual file storage capabilities. On Android, your internal memory is precious and you do not want to install games and applications – especially those which are up to 50MB in sizes – on your internal memory just to make your phone run slower. Instead, you can install the applications on SD card and let your phone take care of the system apps instead, saving more internal memory and speeding up your phone.
The Partitions
Android natively supports fat32 partition. However, thanks to the Android community, support for swap and ext partitions can be enabled too. Depending on the ROM, some can support up to ext4, while others support up to ext3. The explanation about the differences between these partitions is indeed very lengthy and not part of our chapter today. What is important to know, is that Android has support for fat32, swap, ext2, ext3, and ext4 partition support.
Your SD card is by default formatted to fat32. In order to use swap and ext partitions, you need to repartition your SD card. There are several ways to do this, but the most common way is by using a custom recovery installed on your phone (ClockworkMod or AmonRA recovery). Swap is virtual memory which uses extra space on your SD card for virtual memory. However, since Android already has DalvikVM, swap is not really needed. I myself don’t use swap space on my SD partitions. Ext partition is extended partition which was the first ever type of partition created specifically for Linux. It is based of the standard UNIX file system and was designed to overcome the limitations of Minix file systems. Ext 2 is second extended partition, ext3 is third extended partition and ext4 is the fourth extended partition respectively.
Dalvik Virtual Machine
One of the best functions of Android has to be the Dalvik cache. Dalvik cache is a wonder from the point your Android starts up, runs, hibernates and all the way till you device shuts down. Dalvik cache collects the information about the installed applications and frameworks, and organizes them into a writeable cache. Under this writeable cache, it stores the “optimized” bytecode of the applications which is used by the applications themselves later for a smoother operation. This dalvik cache can grow immensely huge as more applications are installed on your phone. It is safe to wipe dalvik-cache. It will be rebuilt again when the phone boots. This also explains why your phone takes ages to start up for the first time. As for my Nexus One, having about 145 applications installed, it takes about 13 minutes to build the cache.
If you ever extract an APK installer file, you will always find a file named classes.dex. This is the file Dalvik finds to build the cache. What makes the process slow? APK is an archive (which is why you can open it up with an unarchiver such as WinRAR or 7-Zip). Being an archive, it provides limited write access to the files contained within and the fact that archives are compressed. Not to forget, APKs are encrypted archives too. Therefore, DalvikVM has to extract the classes.dex files and build the Dalvik table accordingly which makes it easier to write data on it too. With this collective set of data, the Android OS no longer needs to index the applications and find their classes.dex when the phone is already running. Instead, it will just look into one place, and will know what to do next. Nifty huh?
To know what is going on inside the Dalvik VM, you can read about it here.
Froyo A2SD (F-A2SD)
When Froyo was released to Android community, one of its new features was the A2SD implementation. F-A2SD uses fat32 partition natively for application storage. This means, all you have to do is just slot in your SD card and its all ready to go. There is no need to partition the SD card whatsoever. This was a great effort from Google to include A2SD to the Android OS as it gives you an option to choose which applications you want to move to SD card, and which you want to leave on internal memory – BUT – with a condition! If the application developer decides to protect his application and not include A2SD support, you would not be able to move it to SD card. One major problem that F-A2SD has is that it only uses fat32 partition. This way, when the SD card is mounted to the computer, the applications become inaccessible. Not only that, if the applications have widget support, the widgets are removed too when the SD card is mounted. This can become rather a hassle especially if you have to mount your computer several times in a day.
A2SD
The A2SD method is much more interesting. It harnesses the glory of ext partitions. This way, the applications (protected or non-protected), will all be installed on the SD card ext partition. The good thing about ext partition is that when you mount your SD card, the ext partition is NOT mounted together. This said, when your SD card is mounted, the applications will still be accessible and separated from the files and folders on your fat32 partition. On A2SD, the dalvik cache resides on the phone memory.
A2SD+
A2SD+ takes the A2SD one step further. Its pretty useless to have a 512MB A2SD capacity if your dalvik cache is still on phone memory and you have lots of applications installed. This is because the dalvik cache can become pretty huge and just by using HALF of your A2SD’s ext partition, your internal memory can become FULL because of dalvik. Therefore, in A2SD+, the dalvik cache is also moved to SD card. This way, your internal memory is free as a highway. However, remember that Dalvik cache is accessed very frequently. If you have a slow SD card, the overall performance might be affected. I recommend using a class 6 or class 10 SD card for the purpose.
So make your pick guys. In Android’s world, you always have choices. Depending on your needs, use the partition that satisfies you. Hope this article clears any misunderstanding that anyone might be having. Cheers~
Filesystems (post is for galaxy s but all android is ~ same in filesystems)
Reality behind RFS Lag
Background
All data is stored on an 8gb or 16gb MoviNAND chip, of which 2GB is ‘system data’, and the rest is for user storage. The MoviNAND is one of the first mobile ‘smart SSD’ chips. That means that the MoviNAND handles all operations such as data wear leveling, physical data lookup, as well as having it’s own internal buffers. This cleverness is both good… and very bad.
RFS
RFS has a fairly badly written driver, that will call an fsync on file close.
Basically, RFS runs in ‘ultra secure’ mode by default. This security may not be really needed – I personally don’t want it if it means enormous slow downs. It also doesn’t help data security if the system/app is holding a file open, only if it closes the file. The MoviNAND is also fairly smart, and appears to write it’s cache to disk before turning off, and also appears to have capacitors to keep it alive for a little bit of time in the event of a power cut.
SQLite
Most Android apps use SQLite – a fairly simple database that is easy to embed. Sqlite has ‘transactions’ – not real transactions, but a transaction in sqlite is where the database is locked for the duration of a database write, and multiple databases writes can be included in one transaction. At the end of a transaction, sqlite will call FSYNC on the database file, causing a possibly long wait while the MoviNAND does it’s thing. Certain applications will not bunch up writes into a single transaction, and will do all of their writes in new transactions. This means that fsync will be called again and again. This isn’t really a problem on most devices, as fsync is a very fast operation. This is a problem on the SGS, because MoviNAND fsync is very slow.
The various fixes and why they work
Native EXT4 to replace RFS (Voodoo)
By replacing RFS with EXT4, the ‘sync on fileclose’ problem is removed. The EXT series of filesystems is also more efficient at allocating information into blocks than RFS/FAT32 is. This means less real writes to MoviNAND, which means that the MoviNAND buffer should be smaller, and when a sync is called, fewer commands have to be run. When a sync is called on EXT4, it will still be very slow, as the MoviNAND’s sync is still slow.
Basically, EXT4 improves filesystem grouping which leads to less commands, and does not have the broken ‘sync on file close’ that RFS does. It will not heavily improve sqlite database access in certain apps, as the full fsync on transaction end will still have to go through MoviNAND, and will be slow.
When pulling out the battery, there is a chance to lose data that has been written to a file but has not yet been told to sync to disk. This means that EXT4 is less secure than RFS. However, I believe the performance to be worth the risk.
Loopback EXT2 on top of RFS (OCLF)
By creating a loopback filesystem of EXT2, the ‘sync on fileclose’ problem is removed as well. Since the Loopback File is never closed until the EXT2 is unmounted, RFS will not call fsync when a file in the EXT2 loopback is closed. Since a single large file is created on RFS instead of multiple small files, RFS is unable to mis-allocate the file, or fragment it. The actual allocation of filesystem blocks is handled by EXT2. As a note, care should be taken in making the large file on RFS – it MUST align correctly with the MoviNAND boundries, or operations will be slowed down due to double-disk accesses for files, etc. It is unknown whether OCLF is aligning this correctly (how to determine this? 4KB block size gives double the performance of 2KB block size, so it might be aligning it correctly already).
Loopback also has the benefit of speeding up Sqlite databases (at the expense of a transaction being lost in power outage, as it could still be in ram). As always, this is a performance tradeoff between data security when the battery is pulled out, and performance. When pulling a battery out while using the loopback filesystem, there is a chance to lose the last few seconds of database writes. In practice, this isn’t a huge deal for a mobile phone – most lost data will be resynced when the phone reboots. In my opinion, the performance is worth it because of the very slow speed of a sync on MoviNAND.
Loopback EXT2 on top of EXT4
All of the above for normal loopback EXT2 applies. In addition, when the loopback flushes data, it will be flushed to EXT4 instead of RFS. This will probably be better than flushing to RFS, as the RFS driver is not as well written as the EXT4 driver. The difference should not be very large, though.
Journaling
Journaling on an SSD is not required. Your data will not be lost, your puppy will not die. Here is a post made by Theodore Tso -http://marc.info/?l=linux-ext4&m=125803982214652&w=2
But there will be some distinct tradeoffs with
omitting the journal, including possibility that sometimes on an
unclean shutdown you will need to do a manual e2fsck pass.
Not using a journal is not a big deal, as long as you take care to do a full e2fsck pass when an unclear shutdown has occurred. This is the main reason for a journal – to prevent the need to do a full disk check, and instead the journal can be easily read, and the full disk check avoided.
EXT2 vs EXT4
EXT2 appears to work better on the SGS than EXT4. This is because EXT4 has more CPU overhead than EXT2. Journaling is also very bad on MoviNAND. Why? It appears to be the command buffer in the MoviNAND controller. A call to update the journal will use a command slot in the MoviNANDs buffer, that could otherwise have been used for a real disk write. This means that journaling on MoviNAND is a VERY expensive operation compared to journaling on a ‘dumb’ disk.
Well, you could technically use EXT4 and simply disable the high cpu and other features until you are left with EXT2, since EXT4 and EXT2 are basically the same thing.
At any rate, the difference between EXT4 and EXT2 is not very large, and there’s no need for flamewars over it – it comes down to a choice of ‘running’ performance vs ‘startup’ performance, with EXT2 edging out EXT4 for everyday speed, while EXT4 not required a long disk check at boot.
Future Work
Rewrite the firmware for the MoviNAND’s flash to handle fsyncs properly and not bring the system to it’s knees. I joke, but this is really the true solution.
Other solutions include hacking EXT’s fsync method to return instantly, and ensuring that the real fsync is called when the system shuts down. Or doing nothing, fsync is there for a reason, I guess, and would be fine if MoviNAND’s fsync wasn’t so very slow.
There is probably a lot of small details missing from this writeup. They’ll be updated when we learn more. Thanks for all the useful discussions and arguments, everyone!
Many Thanks to Dennis for this awesome information (officially posted here)
Nice cp, but you could take out everything about movienand, which is not present in g3 we only have onenand...
Ah, and don't forget to give credits or link to op...
FadeFx said:
Nice cp, but you could take out everything about movienand, which is not present in g3 we only have onenand...
Ah, and don't forget to give credits or link to op...
Click to expand...
Click to collapse
Of course Done.
kyrillos13 said:
Of course Done.
Click to expand...
Click to collapse
hi
very useful info for the members very nice of u added to the roll up thread
I don t know why I never read this. Very nice and interesting info.
thank you very much for this!!
What about the link2SDapp? Is this a good choice? I'm using it but think it has probs with the second partition.
Godyn said:
What about the link2SDapp? Is this a good choice? I'm using it but think it has probs with the second partition.
Click to expand...
Click to collapse
Better use app2sd scripts
Thx,
I used it, but it filled my ROM.
What app do you suggest?
I used app 2 SD https://market.android.com/details?id=com.a0soft.gphone.app2sd&feature=search_result
But that didn't solve it.