I've seen a lot of people complaining about not being able to boot after using A2SD for a while and I think it's definitely necessary to make clear the danger in using A2SD and some ways to prevents them.
Do not move /data/data to SD.
You can safely move /data/app and /data/app-private to SD. Be cautions with /data/dalvik-cache. (See below)
Make sure the ext2 partition are mounted with noatime and nodiratime (rom maker's job)
Do regular file system checks (using Linux)
Back up your ext2 partition and redo them regularly
Here're the theories:
(NAND) Flash memory has two serious limitations when used to store frequently changing data: you cannot do random write unless you erase a whole page first and any bit can only be written a limited number of times (typically a couple millions). When one bit in a page is detected to be faulty, you lose the whole page. (More details: http://en.wikipedia.org/wiki/Flash_memory#Limitations )
Most mobile os vendors overcome this problem by using special file systems, namely yaffs and jffs, which arrange files according to page size and do write operations only when necessary. They also provide journals so hardware faults could be reliably detected and corrected.
EXT2, which is the file system used in A2SD, DO NOT have these features. It may stupidly allow several pages to be erased hundreds of times of just to write some small files into it. Things are made even worse because some fixed part of the file system (inode table and bitmap) has to be written EVERYTIME a file operation is done. The will accelerate the wearing of those pages and when they become inaccessible, you lose the whole file system.
A2SD would work fine if you only move /data/app to it, because the application files are never modified. It is a completely different story for /data/data because the sqlite databases in there are modified almost every other second!! (And I suspect the OS commit them to disk very often to ensure data integrity.)
In addition, android does not have system check tools for ext2, so it will not be able to detect any problem with the file system until it’s too late.
Edit:
I am not sure how frequent the system updates files in dalvik-cache, but I would say you only move it when you are running out of space in /data.
(Also changed title)
billc.cn said:
I've seen a lot of people complaining about not being able to boot after using A2SD for a while and I think it's definitely necessary to make clear the danger in using A2SD and some ways to prevents them.
Do not move /data/data to SD.
Do regular file system checks (using Linux)
Back up your ext2 parition and redo them regularly
Here're the theories:
(NAND) Flash memory has two serious limitations when used to store frequently changing data: you cannot do random write unless you erase a whole page first and any bit can only be written a limited number of times (typically a couple millions). When one bit in a page is detected to be faulty, you lose the whole page. (More details: http://en.wikipedia.org/wiki/Flash_memory#Limitations )
Most mobile os vendors overcome this problem by using special file systems, namely yaffs and jffs, which arrange files according to page size and do write operations only when necessary. They also provide journals so hardware faults could be reliably detected and corrected.
EXT2, which is the file system used in A2SD, DO NOT have these features. It may stupidly allow several pages to be erased hundreds of times of just to write some small files into it. Things are made even worse because some fixed part of the file system (inode table and bitmap) has to be written EVERYTIME a file operation is done. The will accelerate the wearing of those pages and when they become inaccessible, you lose the whole file system.
A2SD would work fine if you only move /data/app to it, because the application files are never modified. It is a completely different story for /data/data because the sqlite databases in there are modified almost every other second!! (And I suspect the OS commit them to disk very often to ensure data integrity.)
In addition, android does not have system check tools for ext2, so it will not be able to detect any problem with the file system until it’s too late.
Click to expand...
Click to collapse
these are very good points. Actually come to think of it we better mount the ext2 partition with noatime. Because right now every read will wear the flash down. flash storage is really not meant for ext2 filesystem.
knaries2000 said:
these are very good points. Actually come to think of it we better mount the ext2 partition with noatime. Because right now every read will wear the flash down. flash storage is really not meant for ext2 filesystem.
Click to expand...
Click to collapse
In JF1.5 build it is mounted with noatime, i believe:
#mount
/dev/mmcblk0p2 on /system/sd type ext2 (rw,noatime,nodiratime,errors=continue)
But I totally agree with the point of the thread /data/data should not be moved to sd. Not that it's only dangerous (as described), I even don't see any advantages of it.
Dimath said:
In JF1.5 build it is mounted with noatime, i believe:
#mount
/dev/mmcblk0p2 on /system/sd type ext2 (rw,noatime,nodiratime,errors=continue)
But I totally agree with the point of the thread /data/data should not be moved to sd. Not that it's only dangerous (as described), I even don't see any advantages of it.
Click to expand...
Click to collapse
really. that's good, but I am running haykuro's build right now and it is not mounted with noatime. I will have to change the init script.
I'm using JF's 1.5 A2SD build, and I'm pretty certain I moved over /data/data. I didn't really ask myself the question when doing it, but what exactly is stored in /data/data? Is there a command I can run to move it back off the SD card?
Are ext2 and fat the only supported file systems in the android kernel? If not maybe it would be best to move to a wiser file system.
Rekna said:
Are ext2 and fat the only supported file systems in the android kernel? If not maybe it would be best to move to a wiser file system.
Click to expand...
Click to collapse
Yeah, why can't we use yaffs etc?
Dimath said:
Yeah, why can't we use yaffs etc?
Click to expand...
Click to collapse
i don't think most partitioners support yaffs
i know the partition manager on ubuntu 8.10 doesn't
tubaking182 said:
i don't think most partitioners support yaffs
i know the partition manager on ubuntu 8.10 doesn't
Click to expand...
Click to collapse
Indeed, and I'm fairly sure that's why it hasn't been done. But keep in mind several million write cycles is a heck of a lot and apps only really write their data caches occasionally, so it'll likely be a while before anything bad happens(on the scale of years) so this is slight sensationalism. Actually the term "cache" is a slight misnomer here since it's really just storage the apps use for temporary data. If it were used as some kind of extended RAM or a real cache then I could see problems but with what it's used for it should be a non-issue.
billc.cn said:
Here're the theories:
Click to expand...
Click to collapse
SD cards are a form of removable flash that have their own write controllers. In most cases, the write controllers also perform wear levelling. This means even if a program writes to the same file on the sd repeatedly, each time it writes, it is not writing the same physical location on the flash. The reason for this is because due to the way flash works, entire blocks have to be erased before they can be rewritten. To make writing faster, the memory controller keeps a list of empty blocks that are ready to use. When a file is changed, the entire file with the new changes is written to a new memory block and then the old block with obsolete data is then reset to zeros (data deleted).
That said, I still think A2SD is a red herring that only contributes to newbies spamming these forums.
jashsu said:
SD cards are a form of removable flash that have their own write controllers. In most cases, the write controllers also perform wear levelling. This means even if a program writes to the same file on the sd repeatedly, each time it writes, it is not writing the same physical location on the flash. The reason for this is because due to the way flash works, entire blocks have to be erased before they can be rewritten. To make writing faster, the memory controller keeps a list of empty blocks that are ready to use. When a file is changed, the entire file with the new changes is written to a new memory block and then the old block with obsolete data is then reset to zeros (data deleted).
That said, I still think A2SD is a red herring that only contributes to newbies spamming these forums.
Click to expand...
Click to collapse
Which is exactly why I made my new method that's able to deal with a lot of user mistakes and can be incorporated into ROMs to make it take almost 0 user effort
Dimath said:
Yeah, why can't we use yaffs etc?
Click to expand...
Click to collapse
yaffs on SD cards can in some common cases invalidate the wear leveling in hardware that SD cards do, as they so the wear leveling in software.
You are simply understimating ROM makers, SD cards are different from the internal flash in that they do auto wear leveling, that's why you can put common filesystems like FAT which have statically placed allocation tables and writes to the same logical sector will always land on very different places in the card every time.
Yes, noatime will help a lot as it will _reduce_ writes to your SD card.
ext2 is not journalled so it will have less writes too than ext3 or any other journalled filesystem.
So ROM makers are already doing a good job, don't understimate them please.
Newbie NAPS
Newbie’s Noobguide to Android Partitions and Schematics
[A work in progress]
I’ve been there... You want to mess around with rooting your phone, or even getting that sweet CyanogenMod everyone’s been talking about. But let’s face it. You don’t know diddlysquat. And the little bit that you do know is just confusing, and those random half-answers on forums are just no help. Why isn’t there an “Android Hacking 101” out there? Well, here it is.
So should I read this guide?
This guide is actually not a guide. It’s meant to explain how the foundations of an Android device work in order to understand what you’re doing when you’re modding an Android phone. It will also help you figure what’s wrong when something goes bad, because you’ll know what you’re actually doing.
This guide is not a step-by-step guide. This guide is meant to educate on the basics of how the software within Android devices work. It covers the different partitions, the software/files within these partitions and their objectives.
Foreword about Android and LinuxIt is important to note that Android is based on Linux. Linux is open source, and so Google took the source code and used it as a basis for Android, which is open sourced as well. Linux is a Unix-like operating system, while Android can be considered Linux-like (and Unix-like). The Unix-like family of operating systems (sometimes known as *nix operating systems) is very large, and they all share much of this schema of how the foundations of the system work. Mac OSX is a BSD variant (also *nix) and iOS is derived from Mac OSX, and therefore is also very similar in many aspects to this schema.
Table of contents
1. The awesome flowchart
2. Basic terminology and definitions
3. The partitions
-------a. Download Mode (specific to Samsung)
-------b. Recovery
-------c. Kernel (sometimes known as Boot)
-------d. /system
--------------i. What is actually in this partition?
1. Drivers
2. User Interface framework
3. System apps (and bloatware)
4. Media
--------------ii. Manufacturer skins
-------e. /data
--------------i. What is actually in this partition?
1. Apps
2. App data
--------------ii. Factory reset
-------f. /cache
-------i. What is actually in this partition?
1. Thumbnails
2. Cached apps
--------------ii. Why wipe cache
-------g. /mnt (SD card and mass storage)
4. Apps, root permissions and SU
-------a. The root folder
-------b. System apps vs. data apps
-------c. Root permissions
-------d. Why get root
-------e. Fix permissions
5. Custom operating systems (custom ROMs) and Recovery
-------a. Custom ROMs
-------b. Recovery
-------c. Nandroid
-------d. Simplified installation process for (custom) operating systems
6. ROM manager
7. Bricked devices
8. Why not use task killers
-------a. How memory and storage works
-------b. Memory management
-------c. The task killer
-------d. True multitasking
9. Ending notes
10. License
Still missing:
● Bootloader
● Baseband (Radio)
● Troubleshooter
● Battery stats / “Battery memory”
----------------------------------------------------------------------------------------------------------------------------------------------
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Basic terminology and definitionsThis section is for reference and will probably only confuse you if you don’t read the terms in some sort of context. So go read the guide and if you run into a term you’re not familiar with, come back here and see if it’s explained here.
Partition - A section of a storage drive (such as a hard drive or sd card) that is divided so that it functions as an entirely separate storage drive. This is usually used in order to differentiate parts of a drive as if they are physically separate. For the sake of software, there is usually no difference between a hard drive and a partition.
Format/wipe - To completely erase all data on a partition (or drive) and reset the file system (which is the mechanism with which files are stored to and read from a partition).
Image - One file which usually contains an entire partition (or drive). It is called an image because it is as if one would simply take a picture of the partition as it is in its entirety and saved it as an image. Usually done for making backups.
Flashing - Copying an image onto a partition and thus setting the entire partition as defined by the image. Usually done for restoring backups or installing entire operating systems.
ROM (Read Only Memory) - This term is a little tricky, because it’s almost always used incorrectly. ROM is a form of storage that can’t be changed. Since ROM would historically store the operating system, many people use the word “ROM” like it means “operating system”. However, almost anything you’ll ever touch on your Android device is actually not ROM. Many simple (primitive) electronics used to store the operating system (which was usually very simple and didn’t ever need to change) on ROM - it was “printed” on the hardware in the factory, shipped, and it never needed to be changed. Time went along and the ROM was used only to store tools meant to install the operating system, so that even if the operating system was damaged or needed an update, the tools in the ROM could do that. Nowadays, more and more things are stored in volatile memory because technology moves so quickly and most things need to be able to be updated - even the tools used to install the operating system need to be updated (like the recovery in Android).
RAM (Random Access Memory) - The type of memory used for quick access to data which is frequently used. It has smaller capacity and is more expensive than other mass storage (such as hard drives) but it performs much faster.
Binaries - The executable files (usually of applications). These files contain actual instructions for what the application does and how it does it. It mostly does not include data the application is using or making. In Android, binary files end with ‘.apk’ (while in Windows binaries end with ‘.exe’).
Read-Write, Read-Only (R/W, R/O) - Software permissions for reading and/or writing to a partition. Read-Write (rw) permissions allow the software to see what’s in the partition and to modify it. Read-Only (ro) permissions only allow the software to see what’s in the partition without being able to modify it.
Stock/vanilla - A term used for the plain, unmodified form of software, usually meaning what the software is “supposed” to be like. In the case of Android, it would usually be the Android that Google distributes (that which the source can be found in the AOSP).
AOSP - Android Open Source Project. This is the actual project Google is working on called Android. After they complete a new version and release it, the source code is released. Hence, this source code is colloquially known as AOSP.
Source Code - The source code is the actual programming code that software engineers (programmers) write to make software. This code alone does nothing, it’s just the “instructions” for software, and for it to work, you need to take the code and “compile” or “build” it so that when you put it on a machine or in an operating system it will actually work.
Open Source - Open source is a way of saying that someone took their code and opened it up to the world, so that anyone can copy that code and use it for whatever they wish (depending on the license). Many times open source software allows anyone to contribute to it and add their own code. Open sourcing is also a philosophy that basically says that software can or should be “open” in such a way that people can contribute, reuse and build upon existing code.
Details needed
Baseband / Modem / Phone / Radio - The driver software for the actual cell radio - the chipset that actually connects to the cellular network and deals with calls, texts and cellular data.
The partitions
Download Mode (specific to Samsung)
Diagnostic and recovery mode for Samsung technicians. Flashes whole images - including baseband and system, even when partitions are not intact (can be used as last resort recovery). Done with a Samsung PC application straight to the phone’s memory via USB.
Recovery
Alternative boot for updating system. May also include diagnostic, flashing, and other miscellaneous tools. All custom ROMs and mods are based on using the (usually modified) recovery to “update” the operating system (in the system partition).
Kernel (sometimes known as Boot)
The kernel is, as implied, the kernel of the operating system, which includes the most basic instructions for the operating system on how to interact with hardware, such as how to read and write files or how much power to draw from the battery. Such critical instructions must be loaded first, before anything else is to take place.
/system
This is the actual operating system. This is what is called “Android” or “CyanogenMod” or “MIUI” (or on parallel systems, “iOS” or “Windows”). “Updating your phone” or “Updating Android” literally means flashing an image on top of the system (flashing = writing on to the entire partition).
Android on a phone is like Windows on a PC. Just like you can differentiate between programs, files, etc. and Windows itself - that is essentially the distinction with Android.
Because the system software is very critical, the entire system partition is read-only - meaning that it can’t be modified or changed without “root” permissions. This read-only attribute is a well established security measure present in all *nix OSs, and allows the user to rest assured that (theoretically) no matter what, the system itself is safe and cannot be compromised (but just so you know, as a rule of thumb: nothing is uncompromisable).
What is actually in this partition?
● Drivers
The operating system includes the drivers, each of which is the specific language with which the apps speak with the hardware of the phone (the screen, memory, camera, speaker, microphone, gps, wifi, etc). The drivers are different with every device because every device uses a specific make of hardware. That is to say, the processor on the HTC Nexus One is different than that of the Samsung Galaxy S II. The GPS of the HTC Sensation is different than that of the Motorola RAZR.
● User Interface framework
The operating system includes the framework for most of the UI elements. Basically, that means that an app can tell the OS to show a sliding bar, and the OS will show the sliding bar from the UI framework. That’s usually why almost all sliding bars, checkboxes, buttons, fonts, etc. almost all look the same - they’re all taken from the system’s UI framework. Each and every app may or may not use the built-in framework, and most fullscreen apps (such as games) have their own dedicated UI. In Android, the status/notification bar is part of the system’s UI framework, which is why it’s not as easy to customize.
● System apps (and bloatware)
There are a few permanent applications that are installed on the phone by default, such as the launcher (“homescreen app”), the phone app, the messaging app, etc. Bloatware are apps that are pre-installed on the phone as system apps that the user doesn’t feel s/he needs. They can’t be uninstalled without root permissions (because they are in a read-only partition). In Android 4.x system apps (including bloatware) can be disabled by the user (making them virtually non-existent on the phone and is usually just as good as uninstalling).
System apps can be found in /system/app/.
● Media
The system media is the folder that includes ringtones, notification ringtones, alarm rings, and UI interaction sound effects - all (usually) stored in .ogg audio format (an open source format). This folder includes the boot animation as well (the animation that appears as the phone is booting).
System media can be found in /system/media/
Manufacturer skins
Manufacturer skins (such as HTC’s Sense, Samsung’s TouchWiz, Motorola’s Blur, etc.) is an umbrella term for all the modifications a manufacturer has done to change the UI, Media, and sometimes also referring to the system apps installed in the system. Since Android is open source, manufacturers may do whatever they wish to it before selling it, and they believe they add functionality and value to Android by doing so.
Skins are very controversial, because they modify the read-only partition which cannot be undone by the user without root permissions. Many people dislike the modifications and find it makes the phone lose value, or simply find it a drag to have to modify the system in order to revert what the manufacturers have done
.
That being said, I’ve heard many people praise skins for actually making phones better than stock android. It is simply a matter of taste - and that is the reason there are so many custom ROMs out there - because different people like different modifications to the operating system.
/data
The system partition includes all functionality and it is read-only, which means it supposedly never changes after it comes out of the factory. However, the phone does get new things written to it (such as apps, saved data, etc.) and every user’s phone is different than another. There needs to be a place where the user can store everything done to the phone. That’s the data partition - it stores everything that has been added on top of the system after it left the factory and the user begins using it.
What is actually in this partition?
● Apps
Apps (such as those downloaded from the market, or sideloaded from the sd card) are the executable binaries (.apk’s) that the system recognizes as applications the user can run. These can usually be seen in the app drawer.
Apps can be found in /data/app/.
● App data
App data includes all data saved by apps. Apps need to save data; games need to save campaign saves and high scores, the contacts app needs to save contacts’ info and the call log, chat apps save logs of chats, email apps save emails, etc. Everything the app needs to remember - is saved as app data. The data partition is read-only, however each app gets its own folder and can only read and write to that folder (so that apps can’t snoop around what you’re doing in other apps).
Frequently asked, app data includes: SMSs and MMSs (Messaging app’s data), saved wifi networks and their passwords (not sure which app’s data), call logs and phone contacts (Contacts app’s data), and the phone’s settings (Settings app’s data). When you wipe data (factory reset), all these will be lost if you don’t back them up.
App data can be found in /data/data/.
When one does a “factory reset”, the data partition (along with the cache partition) is wiped. This ensures the operating system is intact, however, anything and everything the user installed or added or changed while using the installation will be erased, including all apps and their data, all settings - basically everything except for Android itself. It is important to note that a factory reset does not erase files and data that are saved on the SD cards (see /mnt below).
/cache
The cache partition is designated for storing files meant for easy, quick access. Files such as thumbnails that keep being called up are copied to cache. Cache has a similar role as RAM, however it is not stored in RAM - it is more akin to virtual memory (such as in Windows). Apps don’t actually write to cache, the OS determines which files are being asked for frequently, and decides which files are worth being copied to cache.
What is actually in this partition?
● Thumbnails
Thumbnails are the little preview pictures for images in the gallery. These are created so that you can see the images without opening them, however they are much smaller and are only stored for the sake of convenience to the user (to get a preview of the images) and for the sake of convenience to the system (so that it doesn’t have to make a preview every time you glance at the gallery).
● Cached apps
Apps that have been loaded into RAM and are used frequently are not cleared after the user closes them (see memory management below). If RAM is full and space needs to be made, loaded apps and data (such as a paused game) are copied over to the cache so that they can be loaded later on without having to restart the application from the data partition.
Sometimes issues occur when cache files are corrupt or misplaced - such that applications may be calling the wrong files or bad cache files. In such cases, wiping cache is a good thing - it clears up any files that may be cause issues. Cache files are not critical - they can be deleted without causing any issues, and therefore the cache can be wiped safely at any time (thumbnails will be erased, however the gallery will simply create them again).
Wiping the cache will most likely slow the system down temporarily (as it can no longer rely on files stored for quick access); but as it re-creates cache files, the system will return to its normal performance.
Cache files are obviously related to data, and therefore when wiping data, cache must be wiped as well (a “factory reset” does this automatically). For example, if an app (in data) is using thumbnails (in cache), wiping the app requires wiping the thumbnails as well.
/mnt (SD card and mass storage, also known as storage)
The SD cards are meant for storage. Usually, the internal storage is small and limited, and an internal or external SD card can be added to the phone to allow storage of very large files (such as photos and music). For the sake of proportion, normally the internal storage is no larger than several GBs (which are divided up between /system, /data, /cache, etc.), while storage is minimally several GBs, and are usually as large as several dozen GBs (most SD cards sold today are 8 to 64 GBs). Storing music and photos in internal storage is not realistic, hence the ability to mount SD cards. Newer versions of Android (Honeycomb and ICS) allow for mounting USB storage (such as hard drives), which expands the potential storage size to terabytes.
The difference between an internal and external SD card is simply its physical location. Many phones today are sold with internal SD cards that are actually SD cards built into the phone and cannot be physically removed, however, it is identical in function to an external SD card placed in an SD card slot.
When doing a “factory reset” (wiping /data and /cache), files on storage are not erased. When wanting to remove all personal information on a phone, one must do a factory reset and also remove (or wipe) all storage media.
Storage can be found in /mnt/ or /sdcard/ (e.g. /mnt/sdcard/, /mnt/emmc/, /mnt/USBDisk1/, etc.)
Apps, root permissions and SUThe root (sometimes also known as root folder) is, as implied, the root of the file system, the root of all folders. It is the “folder of all folders”. Sometimes this can be hard to grasp, so here’s an example: if I have one single file on my entire hard drive called File.ext, then it is in the root of the hard drive. If it were in a folder called Folder, that folder would have to be in the root of the hard drive. Root is usually symbolized simply as /. So the file would be in /File.ext, or in /Folder/File.ext. In Windows, the root of the local disk would usually be C:/. The file in the root of the local disk in Windows would be in C:/File.ext.
System apps are installed in the folder /system/app/. They are usually not given root permissions. Data apps are stored in /data/app/, and do not have root permissions. Both /data/app apps and /system/app apps have a folder which they can use to read and write files in /data/data/. Without root permissions, this is the only place on the system they may read and write files to. They may also read and write files to storage (for pictures, music, and other large files that wouldn’t fit into /data/data/.
This concept is the basis for *nix OSs theoretically being more secure: if apps are sandboxed (are given limited place to work with), they can do less damage. Without being able to access anything other than it’s own folder, an app cannot do any harm: it cannot tamper with critical system files, it cannot read personal data from other apps, and it cannot make the phone unusable. Theoretically, the worst an app can do is either damage itself (such as with a bug), or read unencrypted files from an SD card (that’s why personal information on an SD card should always be encrypted!).
Having root permissions means you have permission to read and write anywhere in the file system - as far deep as the root itself (that’s why it’s called root permissions). Theoretically with root permissions, one can modify everything in the system and (if the intentions are evil) take complete control of the operating system. Root permissions is therefore normally a sensitive concept, and it is carefully guarded and not given to just any app. Bad apps with root permissions can cause great damage and potentially misuse these permissions. That’s why in stock Android, root permissions are reserved and the user is never allowed to mess with it.
“Getting root” (colloquially known in iOS as “jailbreaking”) is usually considered the ability of a user to manually allow specific apps to get root permissions. This is usually done in Android by the user with an app called Superuser (or just SU for short). With Superuser no permissions change, however if an app request root permissions the user may grant them to the app. The Superuser app has root permissions, and all it does, with the user’s consent, is modify permissions of other apps (such as changing their permissions from read-only to read-write). Getting root is usually done by attacking and exploiting a bug in the operating system (different with every device) and installing SU, or by flashing an operating system with SU pre-installed (also different with every device).
The reasons for getting root is to install apps that can do things normally not allowed by the operating system. Such examples may be internet tethering on devices from some networks, using the camera’s flash, etc. It also allows one to manually modify files of the system (usually with a file manager), such configuration files, scripts, or adding and removing system ringtones.
Many “Force Closes” in Android (when an app crashes) occur due to inappropriate permissions. If an app expects to have permissions to write to a certain folder and the OS denies it because it doesn’t have those permissions, it may crash and Force Close. To fix this, a function called Fix Permissions is used to reassign all apps the permissions they are supposed to have (including giving apps root permissions if they’ve already been given by the user).
Custom operating systems (custom ROMs) and RecoveryCustom ROMs are modified operating systems that can be installed instead of the “stock” operating system installed on the phone when it leaves the manufacturing factory. Virtually all Android phones sold today are carrying already modified operating systems that are different from “vanilla Android” (the Android developed and released by Google). Since Android is open source, manufacturers (such as Samsung and HTC) can take the code, modify it to their taste and then install that on their phones before selling them. Google’s Nexus line is an example of vanilla Android - the “pure Google experience” - because there are no modifications on AOSP (Android Open Source Project).
Another type of modified operating system is CyanogenMod and MIUI, which are both based on AOSP and are developed and maintained by independent developers, usually with no interest in profit (though donations are always welcome). CyanogenMod is an example of a more vanilla approach (where the UI is relatively similar to AOSP) while MIUI takes a more modified UI experience (where the UI is known to draw inspiration from iOS). These modified operating systems usually offer extended functionality and are “pre-rooted” (meaning they have SU pre-installed in /system/app/).
These operating systems are usually distributed as zip files (one file containing everything needed to flash onto the intended partition). These zips are usually flashed via recovery, which itself is modified for being able to manually flash new images.
The recovery is usually meant for updating the operating system, such that when Samsung distributes a new update for their phones, the operating system images can be downloaded to the phone’s storage (SD card) and booted into recovery. When the phone is running the operating system, it can’t rewrite the whole /system partition because it is in use. Once in recovery, the phone can simply flash the whole image straight on to the /system partition. Therefore, the phone’s operating system is updated and the user’s data and apps and storage are left untouched.
Usually, the recovery has no interface or dialog, and the process is automated. This allows for an easy update process and customers do not have to get involved. A custom recovery will present an interface and allow many functions such as:
● backing up the partitions as images
● wiping partitions (including factory reset)
● flashing images to any of the different partitions
● other miscellaneous functions (such as wiping battery stats, fixing permissions, etc.)
NandroidSince the recovery can read and write entire partitions as images, this gives recovery the ability to backup entire partitions as images. A nandroid is a backup composed of a few images for each of the /system, /data, /cache and /boot partitions. Once these are backed up, the phone can be entirely restored (including the exact state of the operating system, apps, settings, data and everything except for storage) no matter what happens to the system installed. If a critical bug occurs because something terribly irreversible happens, you can restore to a nandroid backup you made before this happened.
Simplified installation process for (custom) operating systems
The usual process for installing custom OSs:
● root is obtained
● a custom recovery image is flashed on to the recovery partition (with root permissions)
Once in recovery:
● a nandroid backup is made for safe measure
● data and cache are wiped (if the operating system installed is different than the one previously installed)
● a custom operating system image is flashed to /system
ROM managerKoushik Dutta is a well known developer in the world of Android independent development. He is best known for developing ROM manager, the most comprehensive custom OS management app available, as well as the well known ClockworkMod (CWM) Recovery, one of the most widely used custom recoveries. He also happens to be the first person to root the first Android phone. Without praising enough, it is suffice to say that ROM manager and CWM recovery are some of the most important elements needed to know when getting started with custom OSs on Android.
Although gaining root is different with every device and every version of the operating system on the device (depending on the exploit found for that device), the rest of the process is similar across most devices. ROM manager requires root and can install CWM (if there is a version compatible with that particular device) and then do most of its functions straight from the app (such as telling it to wipe and then install a custom OS) as it reboots into recovery and does it automatically.
Bricked devicesA “bricked” device is a device that, by either software or hardware, has become as valuable as (or less valuable than) a brick. This is a very broad definition, however the idea is simple: if the device won’t boot into Android, and it can’t be fixed - it’s bricked. Sometimes, the device won’t boot into Android, however that does not mean it can’t be fixed. From my personal experience, everything can be fixed. This is thanks to the fact that even if the operating system is broken, recovery can be used to reflash the operating system. Even if recovery is broken, HBoot or Download Mode can be used to reflash the entire memory.
The exceptions for this are:
● the nand memory itself (the actual memory stick inside the device) is damaged
● the IPL or SPL are broken and therefore neither HBoot or Download Mode are unavailable (Samsung devices can even be restored from such situations with a hardware JIG)
● I’ve heard (not verified) that mismatching SPL and baseband can cause a brick, but haven’t ever had an issue with this.
I have personally reached situations very close to both of these cases on several devices, however I was always able to fix the device, and have never actually run into a brick. That being said, always read the full instructions given, and remember that you are taking the risk, not the person writing the instructions. If you’re in doubt, don’t do it. If you want to double-check, read forum threads and comments until you feel sure about going for it or not.
If that scared you, you should know that generally any person feeling confident with computer skills can manage to deal with this stuff, thanks to a great dev community revolving around Android working hard to make (usually free) solutions for everyone. If you know how to Google and someone has gone through it before (and someone has gone through it before), chances are you and your phone will be fine.
Why not use task killersAlthough this is not strictly related to Android modding, I felt it is important to stress how memory management works after explaining the different roles of each partition.
So first, a little bit about storage. Storage has two important properties (not including price): how much data can be stored on it (capacity/density), and how quick it is to write and read data to and from it (performance).
All data on a system that is meant to be stored, is usually stored on mass storage; high capacity, non-volatile storage. However, this type of storage is usually very slow and it wouldn’t make sense for the computer to run applications straight from mass storage, to keep reading and writing calculations it’s doing to the mass storage and to rely on such storage for second-to-second tasks. So when a user runs an application, operating system copies the binary and the data it is using to RAM, where it can quickly write and read what it’s doing. However, when something needs to be saved in the long term, it is copied back to mass storage. This is the difference between working with an open file, and actually saving the file after you changed it. When it is “open”, it is in RAM, where you can quickly read and write it. When it is “saved”, it is copied back to the hard disk (in the case of a desktop computer). Virtual memory is basically a part of the mass storage that is used as an extension of RAM. Although it is as slow as mass storage, it functionally extends the amount of RAM available (at a cost of performance).
However, when many applications are open, there may not be enough RAM to go around. Hence, you must manage your memory. Things you can do to manage your memory are:
● close applications that haven’t been used in a long time - and thus clearing RAM space for more frequently used applications (at the cost of losing potentially important data)
● copying the memory an application is using to virtual memory - and thus clearing the faster RAM space for more frequently used applications (at the cost of performance)
Every operating system has its own strategy and implementation of memory management. Linux is known for being particularly good at this, while Windows is known for being particularly not good at this. Linux manages to keep as many things in memory as possible for as long as possible, hence allowing applications to not close, open quickly and not lose data.
Applications in memory open faster because they don’t have to be loaded from the slow mass storage, they are already in memory. You can test this on your Android phone if you want: open an app and then press the back button until it closes, then open it again. Then open an app and press the home button until it is hidden, then open it again. You’ll see that after closing an app, opening it again takes much longer than loading it again without closing it. Makes sense, huh? Generally, used memory is good memory.
Used memory got a bad rap from Windows. For some reason, Windows works better the more free RAM there is. And it works terribly slow if there is no free RAM available. Why? Nobody knows. That is - I don’t know. Maybe it’s because Windows doesn’t know how to clear memory when more frequently used apps request it. Whatever.
Then came along task killers to clear your memory to speed your system. A task killer is a tool that pretty much closes applications in order to clear up free RAM. All you need to know is that on Linux used RAM is good RAM. Task killers on Linux will simply make loading applications take longer and won’t really help. A task killer is only good if you want to forcefully close an app for some reason (it’s misbehaving). This is possible within Android (Settings > Manage Apps > App > Force close, if you’re intertested). A task killer is mostly bad if there are apps that are supposed to run in the background in order to automatically update things, download stuff, etc. etc.
People believe that because of true, real multitasking in Android, apps are constantly open in the background hogging your data connection and battery life, doing crazy stuff and causing mayhem. But the engineers at Google are obviously smarter than that. Apps need to specifically request to run in the background, or run certain tasks in the background. That’s good! Because that lets you listen to music in the background while you browse the internet. It lets you download stuff in the background without having to stay on the download page like an idiot. It lets file managers finish copying large files without having to keep the file manager open. Assuming the application developer made sure to program the app properly. Sometimes you will end up with a developer who stupidly allowed an app to download stuff in the background non stop until you close the app. But let’s face it, the vast majority of your apps’ developers know how to program properly. Just leave the task killer alone.
Ending note
Although there can be drastic differences between devices from different manufacturers, the fact is that all Android distributions are *nix, and therefore share many traits, especially when dealing with such low-level processes. Understanding the underlying foundation under all Android devices is valuable knowledge for anyone interested in customizing the software on their device.
Composed: 2012-01-08 Google Docs link
Revised: 2012-04-24
Newbie NAPS is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License (a.k.a. Copyleft)
This is a community effort - no attribution necessary
Sorry my editing sucks, I'm kinda in a rush out the door. I'll clean it up later.
I've been having troubles with duplicates (and triplicates) of songs while playing through Music Play app (not Google Play Music).
I've heard it's common to other android 4.0 phones.
I'm going to clear my cache (albeit after my cribbage game) and see if it fixes the issue.
Thanks for writing this up. I've learned about registers and memory in an "beginner" class on assembly language, read about cache'd memory in my Upgrading and Reparing PCs book, but hadn't thought of clearing the cache as a solution. Your post gets technical while still staying terse.
If only XDA was filled with more material like this!
Google forces people to have at least 500mb of free space saying "some crashes may happen" (they actually don't and if they do I/people can manage that).
Freed up all space on phone, removed all files, removed almost all apps, but space used up (probably by internal partitions such as /data /system etc) increases so:
I mostly can't ****ing use phone any more;
can't install any app (even from USB, yeah I unchecked the damn "protect device" options),
can't enable ADB (because of some other WebView error, which too is related to how google/android works, can't access Dev Settings, they always crash , yes even with >500mb of free space),
can't view email (only in Gmail) (without a workaround,
can't download files (only in Google Drive) and so on...
Clearly this is a google problem, I wish I (and anybody else) never had to use an android/iphone etc.
When rooted etc, all this was never a problem, I also even could do all until there was actually 0mb available.
I need to do this to test some app I'd like to develop.
I can root, etc. each phone, but I'd rather keep putting it off (as I'm developing a hatred for all this, including anti user technology)
@StackExploit
It's NOT Google who notifies you when your Android device has less than 500MB free storage space, but Android OS.
The Android >=500MB free storage space rule is because of 'operating capital' - it needs space to store temporary files as you use apps, it needs space to download updates to and space to store the uncompressed update files prior to installing them ( it should delete the compressed and uncompressed update files automatically after install ), etc.pp.
If Android's storage memory is 32GB ( 32,768MB ) then 500MB correspond to ~2%, if it's 16GB ( 16,384MB ) then it corresponds to ~3%. IMO ridiculously low shares. Take note that Windows & Mac computers have the at minimum 10% free space rule.
In answer to your actual question: I don't believe there's a way around this restriction. Not unless you had rooted and used a custom ROM that specifically mentioned that as a feature. Even if you could bypass it, you'd probably run into other problems (e.g. performance) as a consequence.
Yes, of course, I didn't mention this need of space for temp files (which most often don't take up anywhere near as much as 500mb), because I explained instead for example that I can manage if things crash. These files should be in RAM instead when possible anyway(!).
Well, the min storage is 3% but so what when you can't upgrade any parts like storage and external storage is still lagging behind in so many cases. The higher prices for devices with more (of the same non removable) internal storage are ridiculous.