Database Info

[DEV] /cache converted to swap

So as we all know /cache is used for OTA updates which, when you have your phone rooted is useless.
What I'm wondering is why we can't use /dev/block/mtdblock4 (aka /cache) as a permanent onboard swap partition. Granted it will only give us an additional 30MB but the speed of this swap will be higher than that of our sdcard.
In order to test this we'll have to:
convert this partition to a linux-swap partition.
mkswap /dev/block/mtdblock4
update init.rc to no longer mount this as yaffs2
add this new swap space to the swap script
The issue is of course being able to run a partition utility on the phone. Second, if this doesn't work we don't have a way to format /cache back to yaffs2 (that I know of).
If you have experience with any of the issues I listed above please comment/post. If you don't have a clue please don't post so we can keep this thread clean. I would appreciate only experienced devs responding.
If this is possible it would benefit Hero/MotoBlur ROMS which are memory intensive.

For your second concern, why would we need to change it back to /cache? Just make it a disclaimer when people attempt this operation.
More importantly, I think the primary issue is that this is rom space is it not? Doesn't this portion of the memory have a limited amount of times that it could be written to?

Hmm why not trying "parted /dev/block/mtdblock4" in recovery? ;-)
rest would be easy.

Couldn't we use mkyaffs2img (in recovery /sbin) to turn it back into yaffs?

persiansown said:
For your second concern, why would we need to change it back to /cache? Just make it a disclaimer when people attempt this operation.
More importantly, I think the primary issue is that this is rom space is it not? Doesn't this portion of the memory have a limited amount of times that it could be written to?
Click to expand...
Click to collapse
Well my concern was if there was some adverse effect of removing /cache from the system we'd need a way to revert back.
I am not aware of any MAX amount of times that /cache can be written to.
*Edited*
Due to unintelligent comments and moronism prevalent in this thread, I removed a line from this post to keep us on track of what this thread was meant to discuss.

There's another thing:
Recovery can't be used without /cache !
So we need to convert back each shutdown and convert to linux-swap each startup..

dumfuq said:
Couldn't we use mkyaffs2img (in recovery /sbin) to turn it back into yaffs?
Click to expand...
Click to collapse
I hadn't checked in the recovery img for any yaffs2 utilities. But if that indeed does exist then this will be possible.
So then the issue would be, is this safe to attempt long term with the amount of writes being done to internal flash?

maxisma said:
There's another thing:
Recovery can't be used without /cache !
So we need to convert back each shutdown and convert to linux-swap each startup..
Click to expand...
Click to collapse
So I see recovery is using it but the only thing in it is the recovery logfile. Could we modify the recovery to read/write the logfile to system/sd/recovery or something, or is the cache used for other stuff as well?

Another idea:
Instead of using it as swap, what if we could find a way to use it as compcache space? That way we don't sacrifice any of the internal ram for compcache and still have compcache running

maxisma said:
There's another thing:
Recovery can't be used without /cache !
So we need to convert back each shutdown and convert to linux-swap each startup..
Click to expand...
Click to collapse
What part of recovery needs /cache ? Does the recovery img mount the /cache partition? I'm wondering if the recovery img could be modified to write to the ext2 partition such as /system/sd/cache .

persiansown said:
Another idea:
Instead of using it as swap, what if we could find a way to use it as compcache space? That way we don't sacrifice any of the internal ram for compcache and still have compcache running
Click to expand...
Click to collapse
This is actually a good idea and it'd be interesting to see which way would benefit us more.

shafty023 said:
What part of recovery needs /cache ? Does the recovery img mount the /cache partition? I'm wondering if the recovery img could be modified to write to the ext2 partition such as /system/sd/cache .
Click to expand...
Click to collapse
When you activate an option in recovery, for example install update.zip, it writes the command into /cache/recovery and does a "quick reboot", which reads out the command and does this.

ok first dont even try it would one
reduce life on nand
cause issues with flashing
slow down any thing accessing the nand.
to the person thinking the ram is using flash. you must have never been to high school cause even typing class explains what ram is. plus g1 has memory write speeds of about 120mb/sec thats sdram. swap will never replace ram 120mb/sec versus 6 is huge

if you want real improvements you must change the code to something more efficient.
less bloat
more native code
fewer clock cycles
improved app states(cache on sd that would save the state by dumping part of ram to load later reducing clock cycles and reducing ram)
and my favorite rewrite system libs and apps to include more native code and reducing ram usage.
btw native code is about 10-100x faster than android java. that means 10%-100% more speed(or less load)

you could make a swap file and put it on there and just mount the file...........

yeah what he said!
zenulator said:
you could make a swap file and put it on there and just mount the file...........
Click to expand...
Click to collapse
This sounds like a good idea, altho we would have to get compcache working with backing swap file.
do we know if the /cache speed is compaired to Class 6 microSD?

good luck hardheads

jokersax11 said:
ok first dont even try it would one
reduce life on nand
cause issues with flashing
slow down any thing accessing the nand.
to the person thinking the ram is using flash. you must have never been to high school cause even typing class explains what ram is. plus g1 has memory write speeds of about 120mb/sec thats sdram. swap will never replace ram 120mb/sec versus 6 is huge
Click to expand...
Click to collapse
Dude honestly get a life. I have a Bachelor's degree in Computer Science. And I could code in Intel Assembly accessing each and every register in memory while you're left googling what Intel Assembly code means. Maybe you should keep up with the latest technology. If you did you'd know there are all-in-one memory chips that support NOR, NAND flash, and RAM.
http://www.eetasia.com/ART_8800466798_499486_NT_ab9e537f.HTM

jokersax11 said:
if you want real improvements you must change the code to something more efficient.
less bloat
more native code
fewer clock cycles
improved app states(cache on sd that would save the state by dumping part of ram to load later reducing clock cycles and reducing ram)
and my favorite rewrite system libs and apps to include more native code and reducing ram usage.
btw native code is about 10-100x faster than android java. that means 10%-100% more speed(or less load)
Click to expand...
Click to collapse
Exactly what type of "native code" are you referring to? And do you have any idea what type of memory the G1 uses for RAM? Go back to your typing class.

Shafty did you see Zenulator's idea? That seems like it could work.. Does the phone ever format the cache partition?

[New Solution] - Faster Virtual Ram !!!!!

Hi,
I'm not a developer but I think my idea is great.
1. Using CACHE partition (50Mb) as virtual ram​
- I see that we are using swap ext partition to increase virtual ram, but access data speed of sdcard is not fast enough (for class2 or class4 sdcard),
- And I see that we have 50Mb free in cache partition which are only used when we use OTA update. But if we use a moded rom, why do we use OTA update ? access data speed of internal memory is really good, why don't we use cache partition as virtual ram ???
2. Runing faster with dalvik-cache on internal memory.​
- As you know, after install an application, we will have apk file in data/app or data/app-private and some odex file in dalvik-cache.
- If we only use app2sd with data/app and data/app-private, our application will run faster.

There was already an idea like that.
You mean using CACHE as swap partition.
At the moment this isn't possible because we would have to format this partitions and recovery wouldnt work then.

maxisma said:
There was already an idea like that.
You mean using CACHE as swap partition.
At the moment this isn't possible because we would have to format this partitions and recovery wouldnt work then.
Click to expand...
Click to collapse
What are the partitions formatted as now?

ivanmmj said:
What are the partitions formatted as now?
Click to expand...
Click to collapse
YAFFS2
Swapfiles aren't compatible with it.
Only 1 solution:
Create a loop-ext-file and loop mount it, put a swapfile on it and you're set.
But thats slow.

The partition wouldn't have to be formatted, you can use zeroed images for swap space, I've done it in Solaris before and I'm sure some of the memory techniques for Android have implemented this before, don't quote me!

maxisma said:
There was already an idea like that.
You mean using CACHE as swap partition.
At the moment this isn't possible because we would have to format this partitions and recovery wouldnt work then.
Click to expand...
Click to collapse
Is it not possible to use something like a swap file on the cache partition? I'm not familiar enough with the workings of Android, but I know that swap files are supported in Linux and I've seen it mentioned for Android, so is there any reason this wouldn't work? It would be an amazing improvement. I see you answered while I was typing this post, thanks.
I do wish people would stop putting their ideas as [new solution] and stuff and just put it forward as an [idea] or [concept] until there's a working implementation.

maxisma said:
YAFFS2
Swapfiles aren't compatible with it.
Only 1 solution:
Create a loop-ext-file and loop mount it, put a swapfile on it and you're set.
But thats slow.
Click to expand...
Click to collapse
Would the loop mounting be slow to create or slow for performance? I wouldn't mind a bit more boot time if it meant onboard swap, but obviously if it's a slowdown vs SD or none at all, it's probably not worth it.

How would u format a folder like that??? We could elimate the recovery and instead of useing it for 50 mb swap. Let's use it on the system folder. And then we could than drag eclair in our phones but then we would have to be stuck have just one rom. We could also ask rom devs to make us img instead of .zip roms so we could just flash the images

What about Nandroid etc?
This is a must-have.
Also I think the formation of the partition would be overwritten with every reboot..
But idk.
Won't try it out.

there's an issue for using /cache as swap or compcache like maxisma said.. recovery uses a small portion of /cache to write it's files to temporarily so this wouldn't be an easy work around. this idea was already started a long time ago by shafty and well, it never turned out.

maxisma said:
YAFFS2
Swapfiles aren't compatible with it.
Only 1 solution:
Create a loop-ext-file and loop mount it, put a swapfile on it and you're set.
But thats slow.
Click to expand...
Click to collapse
You don't need a swap partition, neither real nor looped
Code:
# dd if=/dev/zero of=/cache/swapfile bs=1M count=XX
# mkswap /cache/swapfile
# swapon /cache/swapfile
Ooops, forget about it: swapon (as a busybox internal) can't do that on my CM. It does work on my machines :/

But remember guys we also have the spl as backup. We could make the nandro-backup into a .nbh file to flash but it takes a longer time to do it. We could make an app or script that does it for as. I tried to customize the init.rc, build.prop, and fstab but it changes everytime - reboot. What should I do? These files were rewriten when I rebooted. If you guys want to talk this over with me and some of my friends on gtalk, I'll create a group for us to talk more in depth. My email address is [email protected].

jroid said:
there's an issue for using /cache as swap or compcache like maxisma said.. recovery uses a small portion of /cache to write it's files to temporarily so this wouldn't be an easy work around. this idea was already started a long time ago by shafty and well, it never turned out.
Click to expand...
Click to collapse
How much code do we have for recovery? I know there are modified recovery images, I have used Cyano and RA. It seems like it would be possible to create a small ramdisk or loop mount for recovery to use and recover this RAM for use by our applications. Or a mod to the init script to format that device properly for recovery when it boots and run mkswap on it when we boot to normal mode. I'm not real familiar with Android at this level, so perhaps I'm way off base here.

Do you know how to put a custom init.rc on to the g1? When I reboot it is gone and goes back to the orginal init.rc

mohsinkhan47 said:
Do you know how to put a custom init.rc on to the g1? When I reboot it is gone and goes back to the orginal init.rc
Click to expand...
Click to collapse
You have to change the one in the initial ram disk.
1) unpack boot.img
2) change initrc
3) repack boot.img
4) flash

farmatito said:
You have to change the one in the initial ram disk.
1) unpack boot.img
2) change initrc
3) repack boot.img
4) flash
Click to expand...
Click to collapse
I did that but got " broken image file ". What do I do now?

If the 50MB on the cache partition is never used, would it be possible to re-map it to /system? Probably something akin to what the "Death" SPL did.

That is what I said but then I guess you would have to reformat the phone to accept a different amount of info in this case megabytes. We would have to reformat the system folder to hold more info on it.

rgawenda said:
You don't need a swap partition, neither real nor looped
Code:
# dd if=/dev/zero of=/cache/swapfile bs=1M count=XX
# mkswap /cache/swapfile
# swapon /cache/swapfile
Ooops, forget about it: swapon (as a busybox internal) can't do that on my CM. It does work on my machines :/
Click to expand...
Click to collapse
YAFFS2 doesn't support swapfiles.
Thats why would need a loop mounted ext-image.

Placing a swap file/partition in internal memory is risky business. Flash memory has a limited number of write cycles, and swap does many writes per second. Whereas an SD card can be replaced if worn out, internal memory cannot, so the phone becomes a brick if the flash memory is damaged by swapping.

howto apps2sd as ext4

Ok. So lots of talk about ext for apps2sd and how it's being mounted as ext2 even if you formatted to ext4 etc.
Also talk about how journalling hurts performance and such. We'll tackle all those points here.
1. Howto fix the mounting of your ext partition in the meantime till your mod updates their software.
A. Pre-requisites
You must have already partitioned your sd card with an ext4 fs for apps2sd. You can do this in recovery or some other app.
You should also get the program file expert. It's awesome. But any filemanager that has root ability will work. You can do all this in terminal too so whatever floats your boat. I'll use file expert below.
B. Remount your system fs as read/write (not sure if it's actually needed since /system is mounted rw by most mods already... but it can't hurt). You can do this in file expert by hitting menu and selecting More, then the top option labeled "Mount", select read write. It should say success.
You can also do this in terminal but like i said above, i dont think this is absolutely needed but i always assumed it was so never tried this not doing it.
C. go back down directories using the back key till you hit /. Then goto /system/etc/init.d/
D. select S05mountsd (the S0# isn't important here so if yours is different dont worry). Once selected, hit menu and select file operations. Select permissions and change permissions to read/write all.
E. Open the file with a text editor of choice. Scroll down the file to the part that has the command with -t auto in it. It should be pretty obvious, and it's near the end.
Change -t auto to -t ext4. It doesn't matter if your fs is actually ext2 or ext3, mounting as ext4 is always better. The ext4 block allocator and general code is more efficient even if you dont use ext4 features.
You may also want to add an option to the options list data=writeback. This disables journal caching of actual data and only caches metadata. This is really integral to performance on such a slow medium like sd cards and we'll discuss that later.
Save your file and exit.
F. Reboot your phone. You should now have your sd-ext mounted as ext4.
2. data=writeback and why?
there are multiple modes of journalling available to ext4. The default is ordered. This means that there is a specific order that information is written to the fs to ensure that nothing is lost that is said to be written. This means metadata about the fs and file data is both cached in the journal. Safe but slow.
data=writeback still allows crash protection but means you may lose actual newly written data. Your fs wont be corrupt, but your file may not have gotten those last updates you were writing and had not synced yet. It's a pretty decent compromise.
There are other forms that offer better performance but basically remove journal integrity and we're not concerned with those.
There are also options that determine the interval of journal syncing and you can find that information on ext4 man pages.
Why do we care so much about performance here? because sd flash is slow and finite. A journal file is like swap in that it is a static file that's contents get written to anytime you write to your fs. On a normal hdd, it's not noticeable really but on a class 6 SD card, you are going to notice having to write everything twice to your sd card. So we choose writeback mode to remove that double writing and only cache fs metadata updates so that the integrity of the filesystem isn't hurt, and only that particular file that we crashed prior to syncing gets effected. 99% of the time this is fine and perfectly safe and way better than nothing.
3. So if journalling can hurt performance by using up write bandwidth to a medium we are writing other stuff on, wouldn't swap be bad too?
Yes. Swap should only be put on sd cards if you absolutely need it or your phone wont work. Aside from stressing the card, the bandwidth to read/write is extremely small for something you basically have to be interactive with and move to and from ram. Additionally android complicates matters by managing memory at a higher level than the kernel and this means that swap is ignored as a portion of memory. So while technically userspace apps deal with swap transparently, the low memory manager of android will operate as if you didn't have swap...which defeats the purpose.
Use swap as a last choice.
Also, the ability to make swap partitions and swap files is a common issue in linux-land. It's best to use a partition as this removes fs overhead but it becomes even more of a concern when you put it on an ext4 fs with journalling active. With default settings, a swap file will be considered just another file and the journal will cache the data and metadata updates (very bad for performance). if you use swap on an ext4 fs that's actually mounted as ext4 you _NEED_ to make the data option writeback. You will be very disappointed if you dont.
4. Other important notes:
If you use any ext4 specific options like extents, you will no longer be able to read the files written by recovery programs that have not been updated to mount fs's as ext4. Also, just enabling extents doesn't convert data on the fs to use them. You have to write the files while the option is enabled to see their benefit. This is more easily done by putting the card in an actual linux computer and mounting with the correct options and copying the data from the filesystem and then back again ....not for the novice. Though any new files written will automatically have extents even if you dont redo all current data. So best to bother with that after a fresh install prior to restoring apps.
You may also be thinking... Why do i even need a fat fs, i can make the whole thing ext4 and it will be awesome! You would be right, it would be awesome. But it's not so easy. A couple things have to be done in order to make everything written world rewritable and accessible on such a partition. you would have to setgid to a group like "sdcard" to a directory in this new fs and give it rwx group permissions and then have all users current and future become part of that group and mount bind the directory to /sdcard. Needless to say, recovery apps would have to also be updated. Then you could make your main sd partition any fs your kernel supports.

Nice post, cm7 and diet gb I build already have -t ext3 set, do you think ext4 would make much difference?

ext4 has a better block allocator and other features both backwards compatible and ext4 specific. There's really no reason to not mount ext2/3 and 4 as ext4

Thanks for the answer, This is also in the kernel I compile,
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_XATTR=y

Yes, i began my work on the kernel tonight and noticed that in the latest kernels at least that ext4 is being masqueraded as ext2 and as i thought early on as might be the case, that the reporting of the partition as being mounted as ext2 is a bug that has been squashed in the newest vanilla kernels.
This is still an issue, but not a serious one. FS repair tools and such may bug out when trying to found out what fs they're dealing with as the kernel reports the wrong ext version but the functionality of the fs is not affected ( you still get journalling and all that jazz).
So this is less of a performance issue (cept the adding of data=writeback) and more of an issue with other programs needing to see the correct fs.

Ur Diet 09.05 still did not see ext4 partition by Titanium.
carz12 said:
Thanks for the answer, This is also in the kernel I compile,
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_XATTR=y
Click to expand...
Click to collapse
Sent from my Gingerbread on Sapphire using XDA Premium App

If the kernel you're using is one of the newer kernels (probably anything after 2.6.32...but you should be using something much newer) then it likely doesn't even have ext2 or ext3 compiled in. It's all just ext4 like it should be. The issue you still may have however is that the kernel reports the lower revision extfs because ext4's driver accepts the ext4 fs but the kernel is checking for ext2....
Basically, we are asking if this is an ext2fs. The ext4fs driver answers cuz it's all there is and it's masquerading as ext2 and ext3 and ext4. So it looks and sees ext4 and says, hey, i can read this and returns YES. Now the answer yes was to the question of ext2, so that's what gets reported everywhere, but internally, ext4 knows it's got an ext4 fs. mount options that normally would error out with -t ext2 are accepted because the fs driver isn't really ext2 and it knows the fs isn't either so the mount options pass.
This is a kernel bug in an aesthetic and practical sense. Functionally it's all working as intended but other apps and users will get confused as to what exactly is the filesystem, ext2, ext3 or ext4. It can't know without probing the fs itself. Which is stupid for many applications.
Supposedly the latest kernels have a patch to fix this to correctly report the correct fs revision of ext. Though perhaps that hasn't been pushed to linus's branch yet.
In any case, you still should be modifying the mount options to use data=writeback so you're not clogging up the little SD bandwidth you have with data caching in the journal. You really only need fs metadata caching so you dont bork the entire partition on a crash. data=writeback does that, but default is data=ordered.

thank you for such a good post!

[q] ext3 and ext4 question

I am noticing that clockwork now has support for both ext3 and ext4. Can someone please explain to me what I would use ext3 or ext4 for. Are any of the new roms compatable with darktremmers app2sd? I used ext3 on my cliq for that reason but on the g2x with all the confusion with internal and external and internal-external sd, I am kind of lost and need a little clarification if someone wouldn't mind. My sd card i use now I have partitioned to ext3 but so far I have had no need for it and was about to remove that partition but if I may need it in the future when I flash a new rom I will just leave it or upgrade it to ext4. Thanks.

I'm not a dev, so forgive my simplistic explanation. Basically EXT4 is a much faster I/O file system. After flashing a kernel that has EXT4 support my I/O scores in Quadrant Advanced shot up and my total score is now over 4000. Before that I could get in the 3700's. So it makes a big difference there.
As for apps2sd I'm honestly not sure.
So basically it's a more efficient file system.

iamlilysdad said:
I'm not a dev, so forgive my simplistic explanation. Basically EXT4 is a much faster I/O file system. After flashing a kernel that has EXT4 support my I/O scores in Quadrant Advanced shot up and my total score is now over 4000. Before that I could get in the 3700's. So it makes a big difference there.
As for apps2sd I'm honestly not sure.
So basically it's a more efficient file system.
Click to expand...
Click to collapse
SSo I guess they are not talking about the sd card?? Just the file system. I think I am beginning to understand> Thank you

ICE DS v 4.2 ICS 4.0.4 Sense 3.6 with ext3

hello everybody!
I tried to find an answer on this on the ROMs thread but I couldnt so here it is;
Can I use ICE DS v 4.2 with ext3 only? If so, can I disable afterwards ext4 support?
Thanks

why wouldnt you use ext4? it's an advantage thou. but if you dont want to use ICE DS in future, just revert to ext3 by using 4EXT Recovery.

For the amount of internal memory that our devices have there is no matter whether you use ext3 or ext4. Also the install script of the ROM itself will format the partitions as ext4 upon flashing so there is not much choise actually If you like to go back to ext3 after flashing 4EXT Recovery can do this without losing any data
Sent from my HTC Desire S

Thanks for the quick replies..
When I used a stock ROM with a rooted Desire, I remember discussions about the disadvantages of ext4.
The idea as I understood, was that ext4 was more likely to create filesystem corruption in case of an abnormal shutdown of the phone, eg battery removal, expecially if it used to extend phone memory to the sd card via proper partitioning.

It is exactly the opposite. The ext4 file management will prevent corruption in case that a read/write process gets interrupted by a battery pull or similar. Search Google for ext4 there are all the benefits described on a Wikipedia article.
Sent from my HTC Desire S

amidabuddha said:
It is exactly the opposite. The ext4 file management will prevent corruption in case that a read/write process gets interrupted by a battery pull or similar. Search Google for ext4 there are all the benefits described on a Wikipedia article.
Sent from my HTC Desire S
Click to expand...
Click to collapse
This.
ext4 wouldn't exist, or developed if ext3 is better.