Are there any plans to implement a Button into the App-Manager, to move the Data-Part of an app to the /ext2 Partition ?
Especially on low memory devices (like the G1) this will give us the chance to install a lot of apps without runnung to "low space on device".
I have written a very small bash-script to manually do this job, but it would be more comfortable with a Button.
move.sh
Code:
#!/system/bin/sh
#
#data_to_move=com.alk.copilot
#data_to_move=com.camelgames.blowup
#data_to_move=com.drodin.tuxrider
#data_to_move=com.gameloft.android.AMEU.GloftAsphalt5.asphalt5
#data_to_move=com.navigon.navigator
#data_to_move=com.polarbit.ironsightlite
#data_to_move=com.polarbit.ragingthunder
#data_to_move=com.polarbit.rthunder2
#data_to_move=com.polarbit.waveblazerlite
#data_to_move=com.estrongs.android.pop
su
mount -o remount,rw /data
mkdir /sd-ext/data
cd /data/data
echo About to Move ${data_to_move}
cp -r -p ${data_to_move} /sd-ext/data
rm -r -f ${data_to_move}
ln -s /sd-ext/data/${data_to_move} ${data_to_move}
unmove.sh
Code:
#!/system/bin/sh
#
#data_to_move=com.alk.copilot
#data_to_move=com.camelgames.blowup
#data_to_move=com.drodin.tuxrider
#data_to_move=com.gameloft.android.AMEU.GloftAsphalt5.asphalt5
#data_to_move=com.navigon.navigator
#data_to_move=com.polarbit.ironsightlite
#data_to_move=com.polarbit.ragingthunder
#data_to_move=com.polarbit.rthunder2
#data_to_move=com.polarbit.waveblazerlite
#data_to_move=com.estrongs.android.pop
su
mount -o remount,rw /data
mkdir /sd-ext/data
cd /data/data
echo About to unMove ${data_to_move}
rm ${data_to_move}
mkdir ${data_to_move}
cp -r -p /sd-ext/data/${data_to_move} ${data_to_move}
Apps that need lots of space should use the fat32 partition gracefully, using this hack would considerably slow down your system(even class6). Why double the reads and writes to the slow mmc?
Because not all apps using the fat32-Partition.
Navigon for example uses 14 MB of internal storage.
If you install such apps on a G1, you can quickly run out of space.
Low on cache space will considerably slow down the overall system performance.
I would not suggest to move the data of any installed apps, but from the big ones.
Apps like Navigation and 3D-Games are writing not frequently their data and if you own a G1 you are already swapping when using an Eclaid based ROM. So this wouldn't make a big difference in the livetime of your sdcard.
Perhaps someone could implement a partial move (e.g. only libraries) to avoid massive writes to the card ... and/or setting a threshold value, so apps with small data could not be moved.
For me, moving the apps in the Script above has given me additional 35 MB internal space and everything is still runnung "fast".
/data/data
TheGenesis said:
Because not all apps using the fat32-Partition.
Navigon for example uses 14 MB of internal storage.
If you install such apps on a G1, you can quickly run out of space.
Low on cache space will considerably slow down the overall system performance.
I would not suggest to move the data of any installed apps, but from the big ones.
Apps like Navigation and 3D-Games are writing not frequently their data and if you own a G1 you are already swapping when using an Eclaid based ROM. So this wouldn't make a big difference in the livetime of your sdcard.
Perhaps someone could implement a partial move (e.g. only libraries) to avoid massive writes to the card ... and/or setting a threshold value, so apps with small data could not be moved.
For me, moving the apps in the Script above has given me additional 35 MB internal space and everything is still runnung "fast".
Click to expand...
Click to collapse
there is already a script in existance to move things like that:
# lucid -d -sd
would move app data to the sdcard and symlink .. this (however) does not move individual pieces .. i would be interested to know the speed difference on these apps that you moved .. also .. this will create extra difficulty when attempting any nandroid backup/restore .. i have seen people putting up comments because their phone crashed due to the excessive number of symlinks across the phone .. if you are not careful with them you could lose all your data
The Space allocated by the "big apps" is mainly used by their ./lib dir. Because of this, it would be enough to move and symlink only those dirs.
With Games the "rest" goes to settings and scores.
On a G1 there is absolutely no difference in speed when moving data to sd.
Perhaps its because the G1 is not the top performer at all
I have played those games in the list with data in internal and on sd-ext and there is no difference in speed ... loading time is also the same.
I'm satisfied with the results, but these scripts didn't remove the data when apps where uninstalled or re-installed and thats the reason of my request.
Take a look on your storage and see how much space (libraries) should be on the sdcard.
Code:
du -sk /data/data/* | sort -rn | head
btw ... did you ever enabled JIT on a G1 and played ExZeus or Armageddon Squadron ?
Its amazing what is possible on this "outdated" Hardware!
P.S. Nandroid Backup runs perfectly with this symlinks (no recursive/double Backups)
Update:
Nandroid Backup only saves app and app-private ... could you edit this to save everything excluding "crap-dirs" ?
If nandbackup uses standard tar calls, you can use the following command:
Code:
tar pcvf /sdcard/nandroid/sd-ext.tar . -C /sd-ext --exclude dalvik-cache --exclude lost+found
... it would save everything the user place on the partition including userinit.sh
Restoring such a tarball works perfectly with nandbackup.
Thx in advance
Thom
Int. mem for SWAP ?
well, after the moving some heavy apps data
I have 54mb free Int. mem (out of 90)
is that possible to use 24-32mb of this (fast?)memory
for SWAP ? instead of linux swap
and does it make any sense?
...just tried to enable /swapfile.swp via Swapper2
though it says
-creating swap - ok
-changing permission - ok
-formatting swap - ok
but
- enabling swap(file) - FAIL
sorry if it's just another stupid question
G1, stock cm5.0.8 test4, 32mb linux swap
TheGenesis said:
Update:
Nandroid Backup only saves app and app-private ... could you edit this to save everything excluding "crap-dirs" ?
If nandbackup uses standard tar calls, you can use the following command:
Code:
tar pcvf /sdcard/nandroid/sd-ext.tar . -C /sd-ext --exclude dalvik-cache --exclude lost+found
... it would save everything the user place on the partition including userinit.sh
Restoring such a tarball works perfectly with nandbackup.
Thx in advance
Thom
Click to expand...
Click to collapse
Have you tried BART? http://forum.xda-developers.com/showthread.php?t=562292
It's included in the recovery, and lets you backup/not_backup whatever you want.
zelipukin said:
well, after the moving some heavy apps data
I have 54mb free Int. mem (out of 90)
is that possible to use 24-32mb of this (fast?)memory
for SWAP ? instead of linux swap
and does it make any sense?
...just tried to enable /swapfile.swp via Swapper2
though it says
-creating swap - ok
-changing permission - ok
-formatting swap - ok
but
- enabling swap(file) - FAIL
sorry if it's just another stupid question
G1, stock cm5.0.8 test4, 32mb linux swap
Click to expand...
Click to collapse
It's not a stupid question ... I have had the same Idea yesterday ...
I have googled about life spawn of the internal flash memory, but I haven't found any satisfactory answer yet.
Anywhere here who know how fast the internal flash is ?
What about write cycles and wear levelling ?
If it has no integrated wear levelling, swapping will kill the phone in a few days.
I think your enable swap has failed due to wrong permissions ... try to enable with a defered call in your userinit.
Update: I have checked my filesystem ... /cache has actually 29 MB free ... is it correct, that /cache is only used by OTA updates ? Probably we can create a priorized swap there in addition to ext.
Keep up the good work dude...this sounds great
TheGenesis said:
Navigon for example uses 14 MB of internal storage.
If you install such apps on a G1, you can quickly run out of space.
Low on cache space will considerably slow down the overall system performance.
I would not suggest to move the data of any installed apps, but from the big ones.
Apps like Navigation and 3D-Games are writing not frequently their data and if you own a G1 you are already swapping when using an Eclaid based ROM. So this wouldn't make a big difference in the livetime of your sdcard.
Perhaps someone could implement a partial move (e.g. only libraries) to avoid massive writes to the card ... and/or setting a threshold value, so apps with small data could not be moved.
For me, moving the apps in the Script above has given me additional 35 MB internal space and everything is still runnung "fast".
Click to expand...
Click to collapse
If navigon is using that much internal storage it's a very poorly written application. Low cache doesn't slow the system down, just apps that require such huge amounts of it.
I'm already swapping when using Eclair? Wrong, using swap on the sdcard is horrible, I'd never recommend it to anyone and I personally don't use it. I have 114 apps installed and my cache is 9% used, it seems to me like your apps aren't clearing their cache correctly, or they're just poorly written. The argument that 'you are already swapping so this wouldn't make a big difference in the livetime of your sdcard' is untrue, you are effectively doubling the amount of read/writes to the mmc, if not more, so the lifetime could potentially be cut in half, of course depending on use.
I'd say nice try, but this really just working around crap apps.
I use a 16 GB class 6 SD-Card with static wear-levelling.
Assuming that a standard-flash-nand-cell lasts about 10.000 write cycles, and my swap-write-turnover is currently about 1,6 GB per day, my SD-Card will last about 273 years (minus regular writes).
So I don't care about livetime.
Besides Navigon, there are many apps, that store their huge libraries to /data .... Games in most cases ... If you aren't using "bad written apps" its fine for YOU ... everyone else has to do some tweaks when installing some of them to the limited internal storage.
I have 278 apps installed and the only limit for me, is currently the free space on my sdcard.
If you haven't enabled swap since you have flashed your first Eclair ROM, you have probably never felt what is "speed" or you never need more than 1 app running simultaniously .... or you are using a different phone instead of a G1
You say "swapping to sd is horrible" ... I think you have used the wrong parameters ... when I diable swapping my system is lagging ... even when I work with one app the same time.
Did you enabled compcache while swapping ? Did you use a swapfile on FAT32 ? Is your swappinness levor 50 or above 60 ? Are you using al class 4 or slower sd-card ? Are you running heavy memory consuming apps without killing them from time to time ?
All these can turn a fast swapping system into an unusable phone.
You cannot enable swap and use the system like before.
Update:
I have copied 18 MB from cache to data and it tooks round about 18 seconds.
Same file from sdcard (FAT32) to sdcard tooks 6 seconds ...
I will use the sdcard
I should have listened to internal voice telling me not to argue with a fool cause people might not know the difference..
Something strange
I did some stupid (owing to absence of linux knowledge)
experiments regarding to swap_2_/
I believe if it possible it should be done through userinit/config
or smth during boot to enable r/w. give necc permissions etc.
I just used Swapper and RootManager
If I create .swp (it creates but does not work) in any place but /cache
it (just existence of this file) does system unstable, slow and unresponsive
in /cache or any existing or newly created folders inside /cache/ it's OK
before reboot when those new folders/files disappear
=
After a wile something happened with my phone (not a first or last time)
many apps caused FC, settings were lost etc
tried "fix uid missmatches" - dots filled out numerous screens
and after ~20min I decided to reboot
tried nandroid - same endless ....................................................
after the reboot I found no FCs but still missing settings for some apps
(sim_linked_data apps like CoPilot were OK) so I Titaniumed non-working apps
data (5-6 apps) and evrthng seems fine
=
BUT when I look at internal memory available
I find 73Mb (out of ~90) FREE
there was 53Mb free before the accident
Is that normal? And whats the limit?
I have my laps and brain scratched to find some application of this
=
just my experience
-compcache always gives me horrible slow phone - not using
-linux swap - best results compare to no_swap - allways use
Some apps are storing many data to /data and sometimes to /cache.
I think your restores have cleared some of them.
Try CacheMate instead of such manouvers
I have checked the write throughput using dd:
Internal Storage: 3,5 MB/s
Class 6 SD Card: 7 MB/s
... annoying ... USB to SD-Card is 3,5 MB/s and SD-Card via Card-Reader (PC) is about 9 MB/s.
Hey man, i had the same problem and decided to go a head and write a small tool that does exactly this, this is a UI tool that shows all the folders in /data/data (excluding system folders) and let you move your apps to your sd, you should have an APP2SD ROM installed with root (of-course) and sd card partitioned to EXT and FAT32.
Contact me if you want to check it out, i never found the time to publish it ([email protected])
hi,
I have tried that and it works, but...it works until reboot...
After reboot I don't see directory /sd-ext/data....
I don't know why it always been deleted....
Information
JIT Compiler
Going Deeper With Android 2.2′s JIT Compiler
by Quentyn Kennemer on May 26th, 2010 at 3:09 am
Before the official Froyo announcement at I/O, we’d learned that an Adobe employee’s phone was running at ridiculously high speeds. Most outlets quickly chalked this up to the possibility that Google implemented a Just-In-Time compiler for their Dalvik Virtual Machine.
Sure enough, they confirmed our suspicions at the keynote in San Francisco, and we even got a taste of real-world performance (there’s no “theoretical” performance hikes here, folks). We know what JIT does, but I’m sure there are a lot of non-developers out there that can’t understand how it’s able to provide the huge bump in performance for Dalvik.
Technical lead for Android’s Dalvik team – Dan Bornstein – signed in with a blog post over at the Android developers site going into a bit more details about what’s really going on beneath the hood to significantly improve performance of Android (in certain cases) without needing to touch the hardware. Thankfully, he puts it in plain English so you can get – at the least – a pretty basic understanding of what’s going on. Head over there now if you want to learn more about what makes your Froyo so sweet.
Linux-Swap
What is SWAP?
Swap space is an auxiliary storage, such as a portion of a hard-disk, which can be used as memory by the operating system when system RAM is insufficient. This is especially useful on systems with very little system RAM, such as most DD-WRT compatible routers, as it helps prevent the system from running out of memory when multiple background processes are installed.
The difference between Froyo A2SD, A2SD and A2SD+
One of the glories of using Android is having an SD card for storage rather than having internal-only memory. Thanks to Android’s Linux blood, you can even harness the power of the SD card beyond its usual file storage capabilities. On Android, your internal memory is precious and you do not want to install games and applications – especially those which are up to 50MB in sizes – on your internal memory just to make your phone run slower. Instead, you can install the applications on SD card and let your phone take care of the system apps instead, saving more internal memory and speeding up your phone.
The Partitions
Android natively supports fat32 partition. However, thanks to the Android community, support for swap and ext partitions can be enabled too. Depending on the ROM, some can support up to ext4, while others support up to ext3. The explanation about the differences between these partitions is indeed very lengthy and not part of our chapter today. What is important to know, is that Android has support for fat32, swap, ext2, ext3, and ext4 partition support.
Your SD card is by default formatted to fat32. In order to use swap and ext partitions, you need to repartition your SD card. There are several ways to do this, but the most common way is by using a custom recovery installed on your phone (ClockworkMod or AmonRA recovery). Swap is virtual memory which uses extra space on your SD card for virtual memory. However, since Android already has DalvikVM, swap is not really needed. I myself don’t use swap space on my SD partitions. Ext partition is extended partition which was the first ever type of partition created specifically for Linux. It is based of the standard UNIX file system and was designed to overcome the limitations of Minix file systems. Ext 2 is second extended partition, ext3 is third extended partition and ext4 is the fourth extended partition respectively.
Dalvik Virtual Machine
One of the best functions of Android has to be the Dalvik cache. Dalvik cache is a wonder from the point your Android starts up, runs, hibernates and all the way till you device shuts down. Dalvik cache collects the information about the installed applications and frameworks, and organizes them into a writeable cache. Under this writeable cache, it stores the “optimized” bytecode of the applications which is used by the applications themselves later for a smoother operation. This dalvik cache can grow immensely huge as more applications are installed on your phone. It is safe to wipe dalvik-cache. It will be rebuilt again when the phone boots. This also explains why your phone takes ages to start up for the first time. As for my Nexus One, having about 145 applications installed, it takes about 13 minutes to build the cache.
If you ever extract an APK installer file, you will always find a file named classes.dex. This is the file Dalvik finds to build the cache. What makes the process slow? APK is an archive (which is why you can open it up with an unarchiver such as WinRAR or 7-Zip). Being an archive, it provides limited write access to the files contained within and the fact that archives are compressed. Not to forget, APKs are encrypted archives too. Therefore, DalvikVM has to extract the classes.dex files and build the Dalvik table accordingly which makes it easier to write data on it too. With this collective set of data, the Android OS no longer needs to index the applications and find their classes.dex when the phone is already running. Instead, it will just look into one place, and will know what to do next. Nifty huh?
To know what is going on inside the Dalvik VM, you can read about it here.
Froyo A2SD (F-A2SD)
When Froyo was released to Android community, one of its new features was the A2SD implementation. F-A2SD uses fat32 partition natively for application storage. This means, all you have to do is just slot in your SD card and its all ready to go. There is no need to partition the SD card whatsoever. This was a great effort from Google to include A2SD to the Android OS as it gives you an option to choose which applications you want to move to SD card, and which you want to leave on internal memory – BUT – with a condition! If the application developer decides to protect his application and not include A2SD support, you would not be able to move it to SD card. One major problem that F-A2SD has is that it only uses fat32 partition. This way, when the SD card is mounted to the computer, the applications become inaccessible. Not only that, if the applications have widget support, the widgets are removed too when the SD card is mounted. This can become rather a hassle especially if you have to mount your computer several times in a day.
A2SD
The A2SD method is much more interesting. It harnesses the glory of ext partitions. This way, the applications (protected or non-protected), will all be installed on the SD card ext partition. The good thing about ext partition is that when you mount your SD card, the ext partition is NOT mounted together. This said, when your SD card is mounted, the applications will still be accessible and separated from the files and folders on your fat32 partition. On A2SD, the dalvik cache resides on the phone memory.
A2SD+
A2SD+ takes the A2SD one step further. Its pretty useless to have a 512MB A2SD capacity if your dalvik cache is still on phone memory and you have lots of applications installed. This is because the dalvik cache can become pretty huge and just by using HALF of your A2SD’s ext partition, your internal memory can become FULL because of dalvik. Therefore, in A2SD+, the dalvik cache is also moved to SD card. This way, your internal memory is free as a highway. However, remember that Dalvik cache is accessed very frequently. If you have a slow SD card, the overall performance might be affected. I recommend using a class 6 or class 10 SD card for the purpose.
So make your pick guys. In Android’s world, you always have choices. Depending on your needs, use the partition that satisfies you. Hope this article clears any misunderstanding that anyone might be having. Cheers~
Filesystems (post is for galaxy s but all android is ~ same in filesystems)
Reality behind RFS Lag
Background
All data is stored on an 8gb or 16gb MoviNAND chip, of which 2GB is ‘system data’, and the rest is for user storage. The MoviNAND is one of the first mobile ‘smart SSD’ chips. That means that the MoviNAND handles all operations such as data wear leveling, physical data lookup, as well as having it’s own internal buffers. This cleverness is both good… and very bad.
RFS
RFS has a fairly badly written driver, that will call an fsync on file close.
Basically, RFS runs in ‘ultra secure’ mode by default. This security may not be really needed – I personally don’t want it if it means enormous slow downs. It also doesn’t help data security if the system/app is holding a file open, only if it closes the file. The MoviNAND is also fairly smart, and appears to write it’s cache to disk before turning off, and also appears to have capacitors to keep it alive for a little bit of time in the event of a power cut.
SQLite
Most Android apps use SQLite – a fairly simple database that is easy to embed. Sqlite has ‘transactions’ – not real transactions, but a transaction in sqlite is where the database is locked for the duration of a database write, and multiple databases writes can be included in one transaction. At the end of a transaction, sqlite will call FSYNC on the database file, causing a possibly long wait while the MoviNAND does it’s thing. Certain applications will not bunch up writes into a single transaction, and will do all of their writes in new transactions. This means that fsync will be called again and again. This isn’t really a problem on most devices, as fsync is a very fast operation. This is a problem on the SGS, because MoviNAND fsync is very slow.
The various fixes and why they work
Native EXT4 to replace RFS (Voodoo)
By replacing RFS with EXT4, the ‘sync on fileclose’ problem is removed. The EXT series of filesystems is also more efficient at allocating information into blocks than RFS/FAT32 is. This means less real writes to MoviNAND, which means that the MoviNAND buffer should be smaller, and when a sync is called, fewer commands have to be run. When a sync is called on EXT4, it will still be very slow, as the MoviNAND’s sync is still slow.
Basically, EXT4 improves filesystem grouping which leads to less commands, and does not have the broken ‘sync on file close’ that RFS does. It will not heavily improve sqlite database access in certain apps, as the full fsync on transaction end will still have to go through MoviNAND, and will be slow.
When pulling out the battery, there is a chance to lose data that has been written to a file but has not yet been told to sync to disk. This means that EXT4 is less secure than RFS. However, I believe the performance to be worth the risk.
Loopback EXT2 on top of RFS (OCLF)
By creating a loopback filesystem of EXT2, the ‘sync on fileclose’ problem is removed as well. Since the Loopback File is never closed until the EXT2 is unmounted, RFS will not call fsync when a file in the EXT2 loopback is closed. Since a single large file is created on RFS instead of multiple small files, RFS is unable to mis-allocate the file, or fragment it. The actual allocation of filesystem blocks is handled by EXT2. As a note, care should be taken in making the large file on RFS – it MUST align correctly with the MoviNAND boundries, or operations will be slowed down due to double-disk accesses for files, etc. It is unknown whether OCLF is aligning this correctly (how to determine this? 4KB block size gives double the performance of 2KB block size, so it might be aligning it correctly already).
Loopback also has the benefit of speeding up Sqlite databases (at the expense of a transaction being lost in power outage, as it could still be in ram). As always, this is a performance tradeoff between data security when the battery is pulled out, and performance. When pulling a battery out while using the loopback filesystem, there is a chance to lose the last few seconds of database writes. In practice, this isn’t a huge deal for a mobile phone – most lost data will be resynced when the phone reboots. In my opinion, the performance is worth it because of the very slow speed of a sync on MoviNAND.
Loopback EXT2 on top of EXT4
All of the above for normal loopback EXT2 applies. In addition, when the loopback flushes data, it will be flushed to EXT4 instead of RFS. This will probably be better than flushing to RFS, as the RFS driver is not as well written as the EXT4 driver. The difference should not be very large, though.
Journaling
Journaling on an SSD is not required. Your data will not be lost, your puppy will not die. Here is a post made by Theodore Tso -http://marc.info/?l=linux-ext4&m=125803982214652&w=2
But there will be some distinct tradeoffs with
omitting the journal, including possibility that sometimes on an
unclean shutdown you will need to do a manual e2fsck pass.
Not using a journal is not a big deal, as long as you take care to do a full e2fsck pass when an unclear shutdown has occurred. This is the main reason for a journal – to prevent the need to do a full disk check, and instead the journal can be easily read, and the full disk check avoided.
EXT2 vs EXT4
EXT2 appears to work better on the SGS than EXT4. This is because EXT4 has more CPU overhead than EXT2. Journaling is also very bad on MoviNAND. Why? It appears to be the command buffer in the MoviNAND controller. A call to update the journal will use a command slot in the MoviNANDs buffer, that could otherwise have been used for a real disk write. This means that journaling on MoviNAND is a VERY expensive operation compared to journaling on a ‘dumb’ disk.
Well, you could technically use EXT4 and simply disable the high cpu and other features until you are left with EXT2, since EXT4 and EXT2 are basically the same thing.
At any rate, the difference between EXT4 and EXT2 is not very large, and there’s no need for flamewars over it – it comes down to a choice of ‘running’ performance vs ‘startup’ performance, with EXT2 edging out EXT4 for everyday speed, while EXT4 not required a long disk check at boot.
Future Work
Rewrite the firmware for the MoviNAND’s flash to handle fsyncs properly and not bring the system to it’s knees. I joke, but this is really the true solution.
Other solutions include hacking EXT’s fsync method to return instantly, and ensuring that the real fsync is called when the system shuts down. Or doing nothing, fsync is there for a reason, I guess, and would be fine if MoviNAND’s fsync wasn’t so very slow.
There is probably a lot of small details missing from this writeup. They’ll be updated when we learn more. Thanks for all the useful discussions and arguments, everyone!
Many Thanks to Dennis for this awesome information (officially posted here)
Nice cp, but you could take out everything about movienand, which is not present in g3 we only have onenand...
Ah, and don't forget to give credits or link to op...
FadeFx said:
Nice cp, but you could take out everything about movienand, which is not present in g3 we only have onenand...
Ah, and don't forget to give credits or link to op...
Click to expand...
Click to collapse
Of course Done.
kyrillos13 said:
Of course Done.
Click to expand...
Click to collapse
hi
very useful info for the members very nice of u added to the roll up thread
I don t know why I never read this. Very nice and interesting info.
thank you very much for this!!
What about the link2SDapp? Is this a good choice? I'm using it but think it has probs with the second partition.
Godyn said:
What about the link2SDapp? Is this a good choice? I'm using it but think it has probs with the second partition.
Click to expand...
Click to collapse
Better use app2sd scripts
Thx,
I used it, but it filled my ROM.
What app do you suggest?
I used app 2 SD https://market.android.com/details?id=com.a0soft.gphone.app2sd&feature=search_result
But that didn't solve it.
Questions on external storage - SD cards - seem to go on and on and on...
There is a large amount of confusion about them and the information (and mis-information) that exists is all over the web.
Which one is the fastest? The best? How do I partition and format them? Or, the best for my device?
There is no simple answer since the performance of these cards depends on many factors, not of the least of these being the card itself.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Note that I do NOT have a DONATE button anywhere.
I am not looking for donations.
If you feel that you should donate something, by all means,
send it to your favorite XDA developer and/or XDA itself!
And don't be shy about the
button for the many posters who were of help to you!
If you don't like to read you are likely in the wrong thread.. Anyway, if you just want my bottom line - here it is..
Purchase and use a quality card - they will often perform better.
Generally, though not always, higher speed rating means faster.
Learn how to optimise your device (read buffers, caches, etc. all impact performance)
Unofficial recommendation: SanDisk has been a good performer and reliable in my experience.
Find a SanDisk card compatible with your device at the SanDisk Product Compatibility page (thanks Szczepanik!).
Counterfeit card info in post 2.
Formatting info in post 3.
COMMONLY USED CARD TYPES
The most commonly used card type for mobile phones is the SD (Secure Digital) card. It includes 4 families available in different form factors. The four families are the original Standard-Capacity (SDSC), the High-Capacity (SDHC), the eXtended-Capacity (SDXC), and the SDIO, which combines input/output functions with data storage. The three form factors are the original size, the "mini" size, and the "micro" size. There are many combinations of form factors and device families.
DEFINING BASIC CARD SPEED
SD memory card manufacturers use different types of flash memory to create cards, so actual transfer speeds can vary. Varying speeds make it difficult to determine which card will provide reliable recording of streaming content (which is what most manufacturers seem to be concerned with these days).
The two primary classifications are Speed Class and UHS Speed Class.
"Speed Class" has been used by many as a guide to their performance. The Class of a card is a general indication of the read/write speed. Unfortunately things are not that simple in real life. A card's speed depends on many factors, a few of which are:
The likelihood of soft errors that the card's controller must re-try. Writing data requires the controller to read and erase a larger region, then rewrite that entire region with the desired part changed.
Buffers used by the operating system which, in combination with the card's format, sector size and even the device's card bus speed can impact performance.
The possibility of fragmentation - that a body of information the host views as a unit is written to non-contiguous regions of memory. This does not cause rotational or head-movement delays as with magnetic media, but it does vary the amount of computation the card's controller must do.
Speed Classes 2, 4, and 6 assert that the card supports the respective number of MB/s as a minimum sustained write speed for a card in a fragmented state. Class 10 asserts that the card supports 10 MB/s as a minimum non-fragmented sequential write speed.
The following speed classes are defined:
Class 2 - 2 MB/sec
Class 4- 4 MB/sec
Class 6- 6 MB/sec
Class 10 - 10 MB/sec
UHS-I (UHS class 1) - a theoretical 50 MB/sec
USH-II - a theoretical 312 MB/sec
The Ultra-High Speed (UHS) designation is available on some SDHC and SDXC cards. UHS rated cards support a clock frequency of 100 MHz (a quadrupling of the original "Default Speed"), which in four-bit transfer mode could transfer 50 MB/s or a clock frequency of 208 MHz, which could transfer 104 MB/s. Double data rate operation at 50 MHz (DDR50) is mandatory for microSDHC and microSDXC cards labeled as UHS-I. In this mode, four bits are transferred when the clock signal rises and another four bits when it falls, transferring an entire byte on each full clock cycle. UHS-II cards further raise the data transfer rate to a theoretical maximum of 312 MB/s.
It should be noted that it is quite possible for a good Class 4 card to outperform a not so good Class 10 card in various devices (including the HTC HD2 which is what I am using at the time of this writing) as a result of various factors, including card quality, buffers, read/write errors, operating system and application software methodology used.
So, in theory Class 10 is faster than Class 4. As an example, recording video requires a constant minimum write speed while random access (such as our phones mostly use) does not rely on the same type of access. This is really great if you are often recording long videos but I suspect most of us use our phones a little differently.
Often you can get higher speeds from a card than it is rated for. For example, my Class 4 32 GB SanDisk card reads much faster than one might expect it and all too often it is the read speed that makes the device "snappy" (like loading an app).
RANDOM TESTS
It should be noted that these tests were not done by myself. There are many places where tests are described and many methodologies used for them. Benchmarking card performance is not a simple task.
Some of the cards tested include Sandisk 16G class 4, Memorystar 8 G class 10, Memorystar 16 G Class 10, Kingston 8 G class 10, Kingston 16 G class 10, Lexar HS Mobile 32 G class 10, Lexar HS Mobile 16 G class 10, Patriot LX 16 G Class 10, Adata 8 G class 6, Samsung plus 8 G class 6, Sandisk 32G class 4.
Random reads of 512 KB blocks show the SanDisk Mobile Ultra microSDHC to be the fastest card at 21.8 MB/s.
Reducing the block size for the random read test to 4 KB significantly impacts performance, which tops out at a mere 3.4 MB/s, achieved by the Samsung and Adata Class 6 cards. There is almost no difference between queue depths of one and 32 when it comes to memory cards.
Click to expand...
Click to collapse
Click to expand...
Click to collapse
You can review the entire test and the results here.
The random tests and results, which impact mobile phones more, are located here.
BENCHMARKING YOUR CARD
Benchmarks are a rather elusive science since there are too many variables that can impact the results, not the least of which is the card itself. Anyone who has worked with SD cards in the real world has probably noticed that the brand of card can make a dramatic difference in performance. Tests have shown as much as a 12x difference in performance between best and worst cards in real world tasks. As already noted, the "class" ratings used by manufacturers as a measure of performance are not reliable indicators of performance either. In fact, some manufacturer's class 2 and class 4 rated cards routinely outperform cards rated as class 6 or class 10!
Consistency of testing is important since different devices and bottlenecks, such as bus speeds, buffers, file systems and activity can all impact and alter the results. This is why a card that has performed extremely well on a test may turn out to be a dog in your device. Short of testing all cards in a lab environment under exacting circumstances may well yield inconsistent results.
There are a lot of apps available for benchmarking, for Android, as well as Windows and Linux. The following short list is not a recommendation of any software.
HD Tune is a hard disk utility with many functions. It can be used to measure the drive's performance, scan for errors, check the health status (S.M.A.R.T.), securely erase all data and much more. It was designed for hard disks and it has not been updated in recent times. While it could be used for SD Cards (and has been) it is not the ideal tool for it.
More info is available here.
AndroBench is a popular and up to date benchmark application that measures the storage performance of Android devices. It includes Micro benchmarks, SQLite benchmarks and Macro-benchmarks (see screenshot below).
More info is available here.
Crystal Disk Mark is another hard disk test utility which is also somewhat dated (last revision dates back to 2010 as far as I know). It runs under various versions of Windows and Internet Explorer.
More info is available here.
AnTuTu Benchmark is Android Benchmarking tool for Android devices. It can run a full test, through the "Memory Performance","CPU Integer Performance","CPU Floating point Performance","2D 3D Graphics Performance","SD card reading/writing speed","Database IO" performance", etc. The many tests it can perform and the frequent updates make it a viable tool for benchmarking.
More info is available here.
Vellamo is a reasonably easy-to-use suite of system-level benchmarks for devices based on Android 2.3 and later. Some of the functions it offers include CPU subsystem performance, scrolling and zooming, 3D graphics, video performance, memory read/write and peak bandwidth performance.
[ EDIT ] The app has recently been updated and is available on Google Play and is now compatible with KitKat.
More info is available here.
SD Tools is probably one of the most often used tools to check microSD cards (Name, Date, MID, OEMID). You can check if your card is fake. (Check serial number and MID and OEMID). You can also benchmark sd card writing and reading speeds. It is fast and easy to use. See screenshot below.
More info is available here.
Here's my 32GB Sandisk Class 4 card on the Android speed test:
And here is the same card as seen through Androbench:
FILE SYSTEMS FOR SD CARDS
The traditional view has been that storage is not a huge factor for performance on mobile devices. Flash storage (the type most commonly used today) draws little power, and its performance is thought to exceed that of the network subsystem. Yet, oddly enough, just by varying the flash storage, performance over WiFi can typically vary between 100% to 300% across applications; in one extreme scenario the variation jumped to over 2000%. The relationship between storage and application performance seems to be a combination of flash device performance, random I/O from application databases, and use of synchronous writes. Changes to the storage subsystem can significantly improve user experience.
The three primary (not sole) factors impacting device performance as it relates to storage seem to be the media (the card itself), the file system used and the quality of the applications used.
Speed Class is largely irrelevant as it is not necessarily indicative of application performance; although the class rating is meant for sequential performance, there are several cases in which higher-class SD cards performed worse than lower-class ones overall. If not addressed, lower performing storage not only makes the application run slower, it also increases the energy consumption of the device. This part is intended to deal with file systems used in mobile devices and will not address media or application quality (sorry, no tutorial on proper programming techniques) in great detail although they both have a significant impact on performance.
Briefly, Android (as it pertains to storage) consists of flash storage, operating system and Java middleware, and applications; the OS itself is based on Linux and contains low-level drivers (e.g., flash memory, network, and power management), Dalvik virtual machine for application isolation and memory management, several libraries (e.g., SQLite, libc), and an application framework for development of new applications using system services and hardware.
The Dalvik VM is a fast register-based VM providing a small memory footprint; each application runs as its own process, with its own instance of the Dalvik VM. Android also supports true multitasking and several applications usually run as background processes; processes continue running in the background when you leave an application (e.g., a browser downloading web pages).
Android uses SQLite database as the primary means for storage of structured data. SQLite is a transactional database engine that is lightweight, occupying a small amount of storage and memory; it is thus popular on embedded and mobile operating systems. Applications are provided a well defined interface to create, query, and manage their databases; one or more SQLite databases are stored per application on the data partition.
The YAFFS2 file system, managing raw NAND flash was traditionally the file system of choice for the various internal partitions including /system and /data; it is lightweight and optimized for flash storage. Recently, Android transitioned to Ext4 as the default file system for these partitions. Several other files systems have been implemented with varying success in Android devices.
More than 50 file systems have been documented for Linux alone and attempting to document all of them here is simply not possible for me. So many file systems, such little time! You can find plenty of information on all them by doing only a few internet searches. I will concentrate on relatively few of them here. Note that not all attributes of a given file system may be implemented in a given Android system. The omission of a file system from this post should not reflect negatively on it, but rather on my time, or lack thereof.
YAFFS (Yet Another Flash File System) was designed and written by Charles Manning. It is a log-structured file system that holds data integrity as a high priority. Yaffs1 works on NAND chips that have 512 byte pages + 16 byte spare areas. Newer NAND flash chips have larger pages (2048 bytes + 64 bytes spare areas) and stricter write requirements. Each page within an erase block (128 kilobytes) must be written to in sequential order, and each page must be written only once. YAFFS2 was designed to accommodate these newer chips.
YAFFS has been used on Linux, WinCE, pSOS, eCos, ThreadX, and various special-purpose OSes. YAFFS initialization simply erases flash memory. When a bad block is encountered, it follows the smart media scheme of marking the fifth byte of the block's spare area. Blocks marked as such remain unallocated from then on. To write file data, YAFFS initially writes a whole page (chunk in YAFFS terminology) that describes the file metadata, such as timestamps, name, path, etc. The new file is assigned a unique object ID number; every data chunk within the file will contain this unique object ID within the spare area. YAFFS maintains a tree structure in RAM memory of the physical location of these chunks. When a chunk is no longer valid (the file is deleted, or parts of the file are overwritten), YAFFS marks a particular byte in the spare area of the chunk as ‘dirty’. When an entire block (32 pages) is marked as dirty, YAFFS can erase the block and reclaim the space. When the filesystem's free space is low, YAFFS consolidates a group of good pages onto a new block. YAFFS then reclaims the space used by dirty pages within each of the original blocks.
When a YAFFS system mounts a NAND flash device, it must visit each block to check for valid data by scanning its spare area. With this information it then reconstitutes the memory-resident tree data structure.
The extended file system, or ext, was implemented in 1992 as the first file system created specifically for the Linux kernel. It has metadata structure inspired by the traditional Unix File System and was designed by Rémy Card. It was the first implementation that used the virtual file system and it could handle file systems up to 2 gigabytes in size.
The ext2, ext3 and ext4 file systems were all derived from this one. Most ext discussions center around ext3 and ext4 in the Android world.
ext3 is a journaled file system that is commonly used by the Linux kernel. Its main advantage over ext2 is journaling, which improves reliability and eliminates the need to check the file system after an unclean shutdown. Generally, ext3 is slower than competing Linux filesystems, such as ext4, JFS, ReiserFS and XFS, but it has a significant advantage in that it allows in-place upgrades from ext2 without having to back up and restore data. Benchmarks suggest that ext3 also uses less CPU power than ReiserFS and XFS. It is also considered safer than the other Linux file systems, due to its relative simplicity and wider testing base. ext3 does not do checksumming when writing to the journal and if the hardware is doing out-of-order write caching, you run the risk of severe filesystem corruption during a crash.
ext4 was created as a series of backward compatible extensions to ext3. In January 2010, Google announced that it would upgrade its storage infrastructure from ext2 to ext4. In December 2010, they also announced they would use ext4, instead of YAFFS, on Android. The ext4 advantages include large file system support, extents, persistent pre-allocation and journal checksumming.
NILFS (New Implementation of a Log-structured File System) is a log-structured file system for Linux. It is being developed by Nippon Telegraph and Telephone Corporation (NTT) CyberSpace Laboratories. It uses a copy-on-write technique known as "nothing in life is free", NILFS records all data in a continuous log-like format that is only appended to, never overwritten, a design intended to reduce seek times, as well as minimize the kind of data loss that occurs after a crash with conventional file systems. For example, data loss occurs on ext3 file systems when the system crashes during a write operation. When the system reboots, the journal notes that the write did not complete, and any partial data writes are lost. NILFS also includes fast write and recovery times, minimal damage to file data and system consistency on hardware failure, 32-bit checksums, etc.
Android kernels do not routinely include NILFS although mods to make it available can be found.
F2FS (Flash-Friendly File System) was created by Kim Jaegeuk at Samsung for the Linux operating system kernel. The motivation for it was to build a file system that from the start takes into account the characteristics of NAND flash memory-based storage devices, which have been widely used in computer systems ranging from mobile devices to servers. Samsung chose a log-structured file system approach, which it adapted to newer forms of storage. F2FS also remedies some known issues of the older log structured file systems, such as the snowball effect of wandering trees and high cleaning overhead. Because a NAND-based storage device shows different characteristics according to its internal geometry or flash memory management scheme (such as the Flash Translation Layer), Samsung also added various parameters not only for configuring on-disk layout, but also for selecting allocation and cleaning algorithms. Introduced in the second half of 2012 this new file system shows promise but is not yet generally available in any Kernels that I have seen. Samsung has submitted these patches for integration into the Linux kernel, which means it’s likely to appear on Android releases in the future.
JFFS2Journalling Flash File System version 2 or JFFS2 is a log-structured file system for use with flash memory devices. It is the successor to JFFS. JFFS2 has been included in the Linux kernel since the 2.4.10 (2001-09-23) release. JFFS2 is also available for a few bootloaders, like Das U-Boot, Open Firmware, the eCos RTOS and the RedBoot. Most prominently JFFS2 is used in OpenWrt. At least three file systems have been developed as JFFS2 replacements: LogFS, UBIFS, and YAFFS.
JFFS2 adds support for NAND flash devices which have a sequential I/O interface and cannot be memory-mapped for reading. It does not include hard links (this was not possible in JFFS because of limitations in the on-disk format) but it does have compression. Four algorithms are available: zlib, rubin, rtime, and lzo. It claims better performance - JFFS treats the disk as a purely circular log which generats a great deal of unnecessary I/O. The garbage collection algorithm in JFFS2 makes this mostly unnecessary.
Today's predominant file system, YAFFS2, will likely be replaced in the future by the likes of ext4, nilfs or f2fs. In order to do a fair comparison one must compare I/O throughput, user data access latency, application execution latency and data safety. Not a simple task.
If you ever want to get Linux techies arguing just talk about which file systems are the best.
Google, which knows a thing or two about fast systems, has decided, for their purposes anyway, that Ext4 is the best and close to the fastest file system of all. Google also hired Ted T'so, who also happens to be the leading Ext4 programmer. In a note to the Ext4 developer mailing list, Google's Michael Rubin, a senior staff engineer, wrote, "Google is currently in the middle of upgrading from ext2 to a more up to date file system. We ended up choosing ext4." So, if you are using an Android phone and you are not a kernel developer you may want to take Google's word for it, at least for now, and go with the ext4 file system on your SD Card.
In all fairness, the numerous tests that have been ran over the years will prove different winners. Some show ext2 to slightly outperform ext4 but we must also consider data safety and journaling. While not many of us will have vital, enterprise data on our mobile devices, reconfiguring and restoring a device can be tedious, at best. Some will argue for NILFS, others for ext4, and yet others.. well, you get the idea.
I cannot tell you which is the best or fastest or safest. What I can tell you is that, based on my experience, I am staying mostly with ext4 for the time being - reasonable speed and safety combination for my needs.
Comments? Additions? Suggestions? They are all welcome.
Flame wars (relating to SD Cards or otherwise) are not. :-]
HOW TO identify counterfeit Secure Digital Cards -and- backup your card to your PC!
IDENTIFYING COUNTERFEIT CARDS
A significant number of buyers have been deceived by inferior quality, cheap, slow SD cards rebadged as SanDisk or other brand names for quick profit. Many buyers get scammed by fraudulent sellers and products each day. There are a number of blogs showing images of fake cards.
This post is intended to help those who already have the card and would like to be certain.
With flash memory being able to be manipulated into displaying a set or upgraded fake capacity, there must be a way to efficiently test the flash memory. Not only is this testing critical for end users, it is essential for product manufacturers further up the supply chain to have a reliable way to detect fake Nand flash, otherwise they will be unwittingly producing products with incorrect capacities and creating marketplace chaos as well as soiling their reputation. The industry standard for testing memory is the burn in test, which essentially writes a set amount of data onto the memory, and then verifies said data. Errors signify that the memory is unstable and of lower quality (downgrade), or possibly that the memory has been upgraded to a fake capacity.
By far the most widely used and long-standing champion of burn-in testing for Nand flash is the h2testw.exe program, or affectionately known as the H2 burn-in test. Other benchmark and burn in testing programs have come and gone, all defeated by the upgraders in China. H2 was originally written by Harald Bogenholz for c't Magazin (Magazin für Computertechnik), a German computer magazine, and has been used extensively in the flash memory industry from China, Taiwan, to Korea, since 2008. The same version 1.4 has been in use since 2008 and has never been updated, which just goes to prove the reliability of Mr Bogenholz's awesome burn-in testing program.
Here's how:
Insert the flash memory card that will be tested into a reader and launch the H2 program.
Using the Select Target button, choose the drive letter corresponding with the flash memory device.
Once selected, leave "Data volume: all available space" selected and "endless verify" unselected.
Click the "Write + Verify" button to begin the testing.
H2 will "burn-in" the full capacity of data into the flash memory device, and then verify the burned-in data.
If there are errors, chances are the flash is faulty or has been upgraded to a fake capacity.
As long as the flash IC has been attached onto a PCB with a controller, it is possible the memory has already been faked. Each time flash memory exchanges hands post SMT production, from half finished PCBA, to finished product QC, or the end distributor's inspection; H2 is there every step of the way to verify the memory capacity. Hopefully the champ can continue to defeat all of the most sophistaced hackers and upgraders to ensure everyone gets the memory capacity that they paid for.
BACKING UP YOUR SD CARDS
For those of us who have only Windows systems backing up the ext4 partition with the rest of the card has been a bit of a chore. Not any longer! Thanks to MarkAtHome for finding W32ImageWriter we can now create a complete image backup of the card - all partitions - and restore it in tact! The following is a step-by-step of how I did it.
Win32DiskImager is a freeware project currently in beta release v0.6. It will create and restore an image copy of your entire card to and from your Windows PC without a Linux system. Note that image copy means a sector by sector copy of the card into an image file. The program does not care what is on the card!
You can read more about it here (ignore the Rasberry Pi headings!) - I am not going to duplicate their entire page here, sorry.
Once you download the binary (link below) you can unzip it into any folder of your choice. It does not require "installation". The unzipped content of the download should look like this:
Place your SD Card into a USB card reader - it should have a Windows drive letter assigned to it. Now simply launch W32DiskImager.exe on your PC. The card I was using in my phone was a SanDisk Class 4 32 GB card. Looking at it in Minitool showed the following information. You can see my FAT32 and ext 4 partitions:
Keep in mind that this is beta - it still has some minor issues. Upon launch I received an error which may have been caused by Windows Explorer still being open. It seems this app expects exclusive control of the card although clicking OK continued operation without any further issues.
Once at the main screen of the app there is a "read" and "write" button choice. Select a folder and a file name on your PC for the image you are about to create and click "read" to copy the content of your card to your PC. The program prefers a file extension of .img.
This process will take a while since the entire card is being copied sector by sector, not only the files on the card - be patient. When the process completes you will have a large image file of your entire SD card - roughly the size of the card (again, not the content). To test the process I used a second SanDisk Class 4 32 GB card. It was clean formatted with FAT32 without any additional partitions.
I launched W32ImageWriter again, located the image file I created above, made sure the correct drive letter was shown for my card and clicked "write" to copy the saved image from the PC to the card. This process took considerably longer than the creation process since write speeds to the cards are slower than read speeds. Leave it alone, go get some coffee or tea and relax. If you watch the progress bar you will slow the process. When the restoration completed I looked at the card in Minitool and compared it to the first card - I could not see a difference.
The final test was to put the second card into my phone and power up. All was fine, as before. Just to be certain I didn't mix things up and shut the phone down and inserted the other (the original, I think) card and powered up again. As far as the phone was concerned the cards were identical.
And, I now had a complete card image saved on my PC, including the FAT32 and ext4 partitions, which I could recreate any time on the same or another card! Very cool!
A few caveats! As noted above W32ImageWriter is still in beta. There is a known memory leak in this version that has been fixed and will be in the next release. Due to this I recommend running this app in a non-repetitive manner (i.e. use once then exit and restart if necessary).
If you'd like to give it a try you can find the free binary here.
I would appreciate hearing about your experience with it if you try it.
Formatting and Troubleshooting
PARTITIONING AND FORMATTING
The many file systems available for these cards is beyond the scope of this post but a brief mention is needed, nonetheless. Likewise, deciding on read buffers and cluster sizes depend a great deal on your operating system, how the card is used, the speed of the card bus and other factors. You will have to read and experiment more if you want to get optimum performance from your card.
Because the host views the SD card as a block storage device, the card does not require MBR partitions or any specific file system. The card can be reformatted to use any file system the operating system supports such as UFS, Ext2, Ext3, Ext4, btrfs, HFS Plus, HFS Plus, NTFS, FAT16, FAT32, exFAT, etc.
Most consumer products that take an SD card will expect it to be partitioned and formatted in some way. The universal support for FAT16 and FAT32 allow the usage of SDSC and SDHC cards on most host devices. On such SD cards, standard utility programs can be used to repair a corrupted filing system and sometimes recover deleted files. Defragmentation tools for FAT file systems may be used on such cards but are generally not recommended. The resulting consolidation of files may provide a marginal improvement in the time required to read or write the file, but not an improvement comparable to defragmentation of hard drives, where storing a file in multiple fragments may involve a time penalty to move between physical areas of the drive. Moreover, defragmentation performs writes to the SD card that count against the card's rated lifespan.
An SD card should have a life span of roughly 10 years (which doesn't mean it always will). In theory, the more often you write/erase it the faster it will wear. Of course, replacing an SD card is fairly inexpensive these days.
The memory of a card is divided into minimum memory units. The device writes data onto memory units where no data is already stored. As available memory becomes divided into smaller units through normal use, this leads to an increase in non-linear, or fragmented storage. The amount of fragmentation can reduce write speeds, so faster SD memory card speed standards help compensate for fragmentation.
Reformatting an SD card with a different file system, or even with the same one, may make the card slower, or shorten its lifespan. Some cards use wear leveling, in which frequently modified blocks are mapped to different portions of memory at different times, and some wear-leveling algorithms are designed for the access patterns typical of the file allocation table on a FAT16 or FAT32 device. In addition, the preformatted file system may use a cluster size that matches the erase region of the physical memory on the card; reformatting may change the cluster size and make writes less efficient.
The SD Association provides free formatting software to overcome many problems described above. This software formats all SD memory cards, SDHC memory cards and SDXC memory cards. It was created specifically for memory cards using the SD/SDHC/SDXC standards. It is generally recommended to use the SD Formatter first if the card is to be reformatted by another method. Using generic formatting utilities may result in less than optimal performance for your memory cards.
Nota bene: cards greater than 32GB will be automatically formatted as exFat by this software! If you require another file system, such as FAT32, you will have to do a second format with another utility! If you need to format a 64 GB card to FAT32 you can use various utilities to get it done, including this FAT32Format program.
The SD/SDHC/SDXC memory cards have a "Protected Area" on the card for the SD standard's security function. The SD Formatter does not format the "Protected Area". This is genrally reserved for the use of the device, such as cameras and mobile phones.
To create other partitions, such as an "ext4" Linux formatted partition, grab a copy of MiniTool Partition Wizard. The Home Edition is a free partition manager software designed by MiniTool Solution Ltd. It supports 32/64 bit Windows Operating Systems. Functions include: resizing partitions, copying partitions, create partition, extend partition, split partition, delete partition, format partition, convert partition, explore partition, etc.
Similar to the MiniTool Partition Wizard is the Paragon Partition Manager which also has a free version you can download. This one will allow you to increase an ext4 partition's cluster size, if you wish to.
This is a very good article on partition alignment if you are interested in knowing more about that subject - thanks MarkAtHome!
The free SD Association formatting software is available here.
The free Minitool Partition Manager software is available here.
The free Paragon Partition Manager software is available here.
There are many ways you can format a card, including various recovery tools, some bootloaders, PC card readers, etc. I am not suggesting that other methods do not work or that they are bad. This is simply my preferred way of getting it done.
Let's take a 32 GB SanDisk card and set it up for some of the devices. I am using a card reader attached to my PC for these steps. If this is not a brand new card the first step I will take is format it with SD Formatter to make sure it is in optimal form. Loading the program on your PC will give you a screen like this one:
This program will create the "proper" format for your card. Note that this means a 64 GB card which was originally sold with the eXFat format will be so formatted. If your device requires FAT32 format (which many phones require) you will have to do that with another program.
Note the "Option" button! Clicking this button will let you select the format type (quick, full or erase). I generally select full although it takes quite a bit longer. I am seldom concerned with securing old data from other people. I do, however, like to have format adjustment turned on.
When SD Formatter is finished you should see something like this:
If you need only a FAT32 partition you can stop here. You are done. If your setup requires a Linux type partition also for apps or data or whatnot you keep going.
My ROM uses the extended partition for most everything except the Android system. The advantage of this separation is that reads and writes will deal with two different devices, hopefully making things a little faster than waiting for just one.
I prefer to use the ext4 file format which includes journaling (see writeup on file system for more info). I will accomplish the creating of this partition with the Minitool Partition Manager. When you first load the program you should see something like this image although the number of drives and their order will depend on your system.
Note that the FAT32 partition is marked as "Primary"! This was done automatically by SD Formatter but if you are using another program you must remember to do this yourself or chances are the partition will not been properly seen on your device!
Creating the second, ext4 type partition also requires it to be set as "Primary"!
WARNING!! This program requires you click the apply button in order for your selections to be completed. If you do everything and forget to click apply nothing will actually be done!!
As you can see, the FAT32 partition spans the entire card. The simplest way to create room for another partition is to first decrease the size of the FAT32 partition. You can select the "Partition" menu choice on top or right click the FAT32 partition and you will see a "Move/Resize" choice. Selecting it will bring you to this window:
Reduce the size of the FAT32 partition and you will see available space increase as you are doing it. I am aiming for a 3 GB ext4 partition:
Once you apply the changes you should see the unallocated space available for the second partition:
Right click the newly created space and select "Create" partition. Don't forget to set the new partition as "Primary"! Since I am not planning to create another partition (such as Linux swap) I am using all the free space on the card.
Select Apply again and you should see your second partition being created:
Once the process is completed both of your partitions should be clearly shown and, if you did it right, both will be flagged as "Primary" partitions! Now, that wasn't so difficult, was it?
A FEW WORDS ABOUT PARTITION TYPES
There is apparently a lot of information floating around on the topic of "partition types". While they are fairly well defined and documented I could find nothing specific about requiring FAT32 type "b" partitions for Android. That does not mean I could not have missed the info but one would think it should be quite easy to find if this was a blanket requirement.
I read through Tytung's git repositories (okay, kinda speed reading since there is way too much source code there for limited time reading) and I could not find any restrictions on FAT32 type, though again, admittedly I could have missed it.
As importantly, I created a FAT32 partition as type "b" on one card and another as type "c" on another card - they both worked equally well for me. Minitool Partition Wizard creates type "c" (the FAT32 default for the program) unless you change it yourself. I have apparently never used anything other than type "c" on my HD2, on Typhoon GB ROMs, and Tytung's ICS and JB ROMs. So, you got me. If it matters I have not seen that yet.
Briefly, the partition type (or partition ID) in a partition's entry in the partition table inside a Master Boot Record (MBR) is a byte value intended to specify the file system the partition contains and/or to flag special access methods used to access these partitions, such as CHS (Cylinder/Head/Sector) mappings, LBA (Logical Block Addressing) access, logical mapped geometries, special driver access, hidden partitions, secured or encrypted filesystems, etc.
It is up to an operating system's boot loader and/or kernel how to interpret the value.
Type "b" was the original (or older) WIN95 OSR2 FAT32 design. It is limited to partition sizes of 2047GB or less and it depended on the BIOS INT 13 of those PCs. Type "c" also dates back to WIN95 OSR2 FAT32 but is LBA-mapped and is considered an extended-INT 13 equivalent of 0b. Of course, both of these specs were designed for actual (hard) disk drives. According to some notes I have seen the "c" revision should be faster but there is no concrete evidence of this as far as I have seen.
By changing the partition type ID, users can prevent the system from using or initializing partitions - this is up to the operating system which will be accessing it.
According to Microsoft the following limitations exist using the FAT32 file system with Windows operating systems:
Clusters cannot be 64 kilobytes (KB) or larger. If clusters were 64 KB or larger, some programs (such as Setup programs) might calculate disk space incorrectly.
A volume must contain at least 65,527 clusters to use the FAT32 file system. You cannot increase the cluster size on a volume using the FAT32 file system so that it ends up with less than 65,527 clusters.
The maximum possible number of clusters on a volume using the FAT32 file system is 268,435,445. With a maximum of 32 KB per cluster with space for the file allocation table (FAT), this equates to a maximum disk size of approximately 8 terabytes (TB).
The ScanDisk tool included with Microsoft Windows 95 and Microsoft Windows 98 is a 16-bit program. Such programs have a single memory block maximum allocation size of 16 MB less 64 KB. Therefore, The Windows 95 or Windows 98 ScanDisk tool cannot process volumes using the FAT32 file system that have a FAT larger than 16 MB less 64 KB in size. A FAT entry on a volume using the FAT32 file system uses 4 bytes, so ScanDisk cannot process the FAT on a volume using the FAT32 file system that defines more than 4,177,920 clusters (including the two reserved clusters). Including the FATs themselves, this works out, at the maximum of 32 KB per cluster, to a volume size of 127.53 gigabytes (GB).
You cannot decrease the cluster size on a volume using the FAT32 file system so that the FAT ends up larger than 16 MB less 64 KB in size.
You cannot format a volume larger than 32 GB in size using the FAT32 file system in Windows 2000. The Windows 2000 FastFAT driver can mount and support volumes larger than 32 GB that use the FAT32 file system (subject to the other limits), but you cannot create one using the Format tool. This behavior is by design. If you need to create a volume larger than 32 GB, use the NTFS file system instead.
Keep in mind the above is Windows (Microsoft). Linux and Android handle things a little differently. All of the Linux filesystem drivers support all three FAT types, namely FAT12, FAT16 and FAT32. As far as I know Android should handle them the same way. Other common features that they all support are various Linux mounting options (specified with the -o option to the mount command), such as "uid", "gid", "umask", etc.
There really is so much more to partitioning and formatting that I simply cannot cover it all here, however, all the information is already available someplace on the internet. Search is a wonderful tool, everyone!
SOME OTHER SD CARD RELATED STUFF TO PONDER
Finding and Fixing (?) Bad Blocks
SD cards include controller circuitry to perform bad block management and wear leveling. Although there is software to find bad blocks considering the price of these cards today if you're starting to see bad blocks the device is just plain going bad. Time to get a new one before you have serious problems.
You could run diagnostics to try and find/fix it if you really want to. This can be done on your PC and a card reader or in Android Terminal:
Code:
su
fsck_msdos /dev/block/yourdevice
For ext2/3/4 partitions you can use e2fsck but it needs to run on an unmounted partition. If this is not clear you probably should not attempt it until you read a lot more, sorry.
You cannot fix bad blocks. Once a block is bad - it remains bad. Kind of like trying to fix a battery that cannot hold a charge any longer.
There are many supposed fixes claiming to do magic. None actually work. You may be able to tell the system a block is bad and it should not be used, thus avoid problems, but there is no fix.
Eventually, when your device develops too many bad blocks (around 50% of NAND) it will go to join that big phone company in the sky - time to get a new phone.
The best way to check bad blocks is to format all partitions in ClockWorkMod (warning: this will wipe your phone completely!) and check the recovery log when finished - after the format go to Advanced in ClockWorkMod Recovery and choose report error, this will save your log in your /sdcard/clockworkmod You can view this file with most text editors or copy/email it to your computer for viewing.
Alternately, you can try to reboot your device. When the boot process is completed (before you do anything else) load the TERMINAL program and enter the following:
Code:
su
dmesg > /sdcard/dmesg.txt
This will create a text file called dmesg.txt in the root of your card which can be viewed with a text editor. Look for the kernel name early in the file. Bad blocks, if any, should be listed not too far after the name. Or try Lumberjack.
Thanks to Robbie P for the reminder on a HD2/MAGLDR option:
Robbie P said:
If using MAGLDR, go to services/DMESG to SD to get Dmesg from last boot.
Click to expand...
Click to collapse
Since we seem to be talking about bad blocks - it has been some time since this info was originally posted. For everyone who missed it (or forgot), here it is again.
The HD2 aka Leo is manufactured with 2 different NAND Flash ROM chipsets.
You can find the type by entering the tri-color boot mode.
If you see PB81120 SS-B3 on the first line that means your NAND Flash ROM is from Samsung (KBY00U00VM)
If you see PB1120 HX-B3 on the first line your NAND Flash ROM is from Hynix (H8BFS0WU0MCR)
After about one year of usage the Samsung chipset usually has a dozen or so bad clusters.
Hynix normally has none of them or just a couple.
A quick way to delete the Dalvik cache from your ext4 partition.
There are many, many ways to get this done should it be desired. One quick and easy way to do it in Android Terminal is:
Code:
su
cd /data/dalvik-cache
rm *
exit
Reboot your device and wait for the boot process to rebuild the cache.
Checking the system logs for problems
Problems with SD Cards as well as NAND and applications are often recorded in one of the Android system logs. Developers are often asking for logs so they can identify the source of a complaint. While not difficult to get, these logs are not in plain view. One the quick and easy ways to view and save the Android logs is a utility called Lumberjack.
The free Lumberjack software is available here.
If you prefer to use Android Terminal you can dump a lot of this info into text files which you can view later or email. The following will dump DMESG, KMSG and LOGCAT to your SD Card:
Code:
dmesg > /sdcard/dmesg.txt
cat /proc/kmsg > /sdcard/kmesg.txt
cat /dev/log/system > /sdcard/logcat.txt
Comments? Additions? Suggestions? They are all welcome.
Flame wars (relating to SD Cards or otherwise) are not. :-]
Thanks. Very interesting information.
Sent from my Nexus 7
Benchmark
ny_limited said:
Place holder .......
Click to expand...
Click to collapse
Great thanks it would be helpful to have guidance on the various Android apps that benchmark read and write speed, for example
AnTuTu
Androbench
Cheers
Tom
Szczepanik said:
Great thanks it would be helpful to have guidance on the various Android apps that benchmark read and write speed, for example
AnTuTu
Androbench
Cheers
Tom
Click to expand...
Click to collapse
Good idea, thanks! Not enough time at the moment but I'll see if I can come up with some..
i'm using Transcend 16GB Micro-SDHC (Class10)
and using swapper2, SD Card Boost
lucubrb said:
i'm using Transcend 16GB Micro-SDHC (Class10)
and using swapper2, SD Card Boost
Click to expand...
Click to collapse
About swappers....
Swapper2 is an app that lets you run a virtual memory on your sd card instead of using the internal phone memory. All swappers will decrease the life of your card (or NAND) simply by their nature, but SD cards are pretty cheap these days.
The intent of swapping is to create "make believe" RAM for working processes by moving applications in and out of working memory.
Unlike traditional swap, Android's Memory Manager kills inactive processes to free up memory. Android signals to the process, then the process will usually write out a small bit of specific information about its state (for example, Google Maps may write out the map view coordinates; the browser might write the URL of the page being viewed) and then the process exits. When you next access that application, it is restarted: the application is loaded from storage, and retrieves the state information that it saved when it last closed. In some applications, this makes it seem as if the application never closed at all. This is not much different from traditional swap, except that Android apps are specially programmed to write out very specific information, making Android's Memory Manager more efficient than swap.
Personally, I much prefer tweaking the Memory Manager settings than using swap files.
About SD Card Boost...
Not sure what this is. Is this something like the read buffer or something else?
Szczepanik said:
Great thanks it would be helpful to have guidance on the various Android apps that benchmark read and write speed, for example
AnTuTu
Androbench
Cheers
Tom
Click to expand...
Click to collapse
Updated post 1 with a few benchmark apps in no particular order or recommendation. Some are dated while others quite popular. I suspect that the methodology used for benchmarking - keeping all variables the same - is possibly more important than the test app itself.
Wow... this is what I have been looking for, hence it has made it to my signature :good:
Ultimate guide about SD cards.
apurohit said:
Wow... this is what I have been looking for, hence it has made it to my signature :good:
Click to expand...
Click to collapse
Thanks, but it is a work in progress..
I have a few more ideas to put up (and another thread or so) but time is not always cooperating.
So sad when when work gets in the way of having fun!
New ROM Toolbox
The very versatile ROM Toolbox was just updated and Jared has essentially re-written some SD Card related code. It may just be worth checking out. There isa a Lite and Full version - sorry I do not recall which features are paid only.
ROM Toolbox is the must have app for every root user. ROM Toolbox combines all the great root apps into one monster app with a beautiful and easy to use interface. ROM Toolbox has every tool you need to make your Android device fast and customized to your liking.
The October 23, 2012 update adds a re-written SD Booster (you can now set read buffers as high as you wish!) and new SD benchmarks & storage chart.
More info is available here.
ny_limited said:
About swappers....
Swapper2 is an app that lets you run a virtual memory on your sd card instead of using the internal phone memory. All swappers will decrease the life of your card (or NAND) simply by their nature, but SD cards are pretty cheap these days.
The intent of swapping is to create "make believe" RAM for working processes by moving applications in and out of working memory.
Unlike traditional swap, Android's Memory Manager kills inactive processes to free up memory. Android signals to the process, then the process will usually write out a small bit of specific information about its state (for example, Google Maps may write out the map view coordinates; the browser might write the URL of the page being viewed) and then the process exits. When you next access that application, it is restarted: the application is loaded from storage, and retrieves the state information that it saved when it last closed. In some applications, this makes it seem as if the application never closed at all. This is not much different from traditional swap, except that Android apps are specially programmed to write out very specific information, making Android's Memory Manager more efficient than swap.
Personally, I much prefer tweaking the Memory Manager settings than using swap files.
About SD Card Boost...
Not sure what this is. Is this something like the read buffer or something else?
Click to expand...
Click to collapse
what buffer size good for card?
coohdeh said:
what buffer size good for card?
Click to expand...
Click to collapse
Interesting question. Too often people tend to make very large or very small read buffers and performance suffers. Ideally, the buffer should parallel the cluster size of your card.
The best cluster size can usually be determined by using SD Formatter. 32k clusters are not always the best although they generally are for big cards (e.g. 32 GB and bigger).
Using 1024KB or 2048KB buffer with 32 KB clusters is, in my view, rather inefficient since it will save your device several read operations to read multiple clusters (which is the card's unit, so to speak).
In my experience setting a read buffer which is not less than 100% and not more than 500% of the cluster size is ideal. Experimenting is generally easy and a good idea as your 'mileage may vary'. Please remember to experiment - while a large buffer may do wonders (as it did for me) it is not always the best choice. See this post for a little contrast.
To set the read buffer the init.d scripts used to do the job - they still work, of course. The latest version of ROM Toolbox lets you set the buffer to any size you want and it will automatically reset it on each boot (a necessary evil). This is a very easy way to experiment without rebooting after each change.
Currently, my cluster size is 32KB and my read buffer is 16MB (16384 KB).
Using 1024KB and 2048KB read buffer my devise was extremely sluggish!
File systems for SD Cards - the debate goes on!
I have updated post 3 of this thread: File systems for SD Cards - the debate goes on!.
Have fun!
great
Wow. Now that was a lot of infos in the morning. I never thought that I ever gonna find so detailed and article about SDcards. Thanks
crancker said:
Wow. Now that was a lot of infos in the morning. I never thought that I ever gonna find so detailed and article about SDcards. Thanks
Click to expand...
Click to collapse
You mean you read it all?! I am impressed!
Was my typing / spelling OK?
I am using samsung 32GB class 10. It's really fast in my Samsung!
ny_limited said:
I have updated post 3 of this thread: File systems for SD Cards - the debate goes on!. Have fun!
Click to expand...
Click to collapse
ny_limited thanks again. It will be good to move any debate about file systems out of Xylograph's thread.
Meanwhile I recommend that readers of this thread view it in a browser and, in the top right corner, rate it five star and "newsworthy".
Cheers
Tom
Sent from my HTC HD2 using Tapatalk
Are there any reasons to use a class 10 memory card on a Android phone?
Essentially, most of the apps are installed on your internal memory of the phone itself. Isn't the memory card just used as a storage medium for music and videos?