The question is in the title.
I searched the Android Hacking top-level-forum for fsck: 8 hits. 4 come from this particular 'general' sub-forum of 7826 threads. e2fsck isn't much better, 16 total, 8 in this sub-forum. And not all of these are relevant at all. None seem to describe how to properly utilize it.
Here's my take:
You run a filesystem with a journal but to keep things fast you have mount options of data=writeback and barrier=0. Or maybe no journal at all.
You reboot uncleanly. The journal may replay and everything's fine, and it works. Or the journal is corrupt, missing, or somehow cannot be replayed to make everything happy.
Why have people moved away from actually checking the filesystem on boot, like every sane desktop/server operating system does ? What seems to make people think that an Android smartphone isn't in the same class as the other devices?
Yes, your phone is a handheld computer. The phone you have in your pocket is probably near the speed of your second Windows XP / OSX machine by now, at least in perceived speed and ability to do stuff. Likely more-so in terms of web content access and likely a lot faster seeming for other tasks.
This is a lot of computing power in your hand and not fixing a corrupt filesystem on such a complex device is a crying shame...
Is there any good reason why it seems to me that so many roms exist without adding e2fsck to a boot script to properly check and clean the partitions before actually mounting them?
As you have pointed out that the filesystem check is usually not done at boot time.
Is there a way to catch it up at runtime?
Related
I understand that Android is not like Windows, but with all the app installing/uninstalling (and file transfers) we do on our devices - is there any need to perform a 'defrag' like function on the storage media?
I know on my Windows PCs in the past, a defrag sped up a lot of functions on the machine once completed.
is there such tool for the android OS?
EDIT: what the equivalent "defrag-like" operation i can perform on Android OS to improve internal drive's performance?
Android is Linux, it doesn't operate like windows and you won't ever need a defrag or anything like that.
FearedSkills say true .... but you can try Optimize ToolBox , for me - good app
Flash is your physical layer, you don't want to defrag it anyway as it burns write cycles.
Flash (SSD) 's life is reduced if you 'move' stuff around by increasing it's read and write cycles unnecessarily.
Totally agree with above especially if you use a ROM (or kernel) enabling ext4 filesystem, far more responsive and optimized than ext3.
To be precise, ext3 fs fragment, but fragmentation level is always low and is never a concern for efficiency of the system. Ext4 fragments less than ext3.
The reason that defragmenting in windows is good practice is primarily because the physical device (hard drive) will perform better when it is reading data in consecutive blocks.
SSDs and Flash chips do not organize their data the same way, and can actually have their service life reduced by moving data around into bigger chunks.
I was considering use the LUKS Manager or GuardianProject LUKS components to encrypt my /data partition and downloaded files in loopback AES volumes. Now, I noticed that Data2SD and other projects used to make use of loopback filesystems, but this was phased out over time. I am aware that there will inevitably be a performance hit, but it is not clear to me if my idea is as nuts as it sounds. So:
Are loopback filesystems on a FAT volume on a class 6 SDHC card on a Desire Z/Vision such a bad idea?
Is filesystem stability/corruption something I really need to worry about?
Are there any reasons this noob should know now before venturing into this approach or did Data2SD and others move to dedicated partitions purely for performance reasons?
Again, I realize performance will inevitably be a large factor, but I want to make sure that having loopback filesystems is not so detrimental I need to worry about other issues. I need Windows interoperability sort of when I am in a pinch.
Well sadly i don't have an answer for that question yet...
I'm trying to think of a way to put KSM to the test on my android device.
As far as i understand it is possible that the kernel actually causes high CPU usage trying to map and unmap memory pages over and over again.
This issue is known for linux and other virtual machines so it is possible that the Same effect will be on the android vm
Testings that i found are not relevant to android.
For example:
The result is a dramatic decrease in memory usage in virtualization environments. In a virtualization server, Red Hat found that thanks to KSM, KVM can run as many as 52 Windows XP VMs with 1 GB of RAM each on a server with just 16 GB of RAM. Because KSM works transparently to userspace apps, it can be adopted very easily, and provides huge memory savings for free to current production systems. It was originally developed for use with KVM, but it can be also used with any other virtualization system - or even in non virtualization workloads, for example applications that for some reason have several processes using lots of memory that could be shared.
Click to expand...
Click to collapse
http://kernelnewbies.org/Linux_2_6_32
What i would really want to know is what would happen if each of these VMs Would run a different application/game/audio/graphics software at the same time ? or what if the same vm will run many different apps ? and also to compare cpu usage with and without KSM
Guess i'll need a tool for that. something like 'iostat' but for memory diagnostic and another tool to see a per process CPU usage but 'top' is not good enough for that.
Any way, the best test should present clear results with precised data.
I'll keep looking for legit way to put it to the test.
If you can think of a way to test KSM with android, please let me know.
This is a technique that relates mostly to processes like virtualisation. For example, when you load 5 windows XP VMs, you'll have a good 10 - 20 services that are practically the same in memory in each VM. Instead of each service using 10mb (ie, 10mb x 5 = 50mb), you only need say 15 or 20mb using KSM. If you use different applications, it is very unlikely that anything would be saved FOR THAT APPLICATION. However, the main elements of a Windows XP System would still be there (drivers, explorer, firewall, logon, search and so on). Means little in one setup, but when you have several VMs it is shown to be a huge advantage. As we know a simple XP install can use 500mb of RAM actively, and this is fairly uniform across instals.
With android, i don't know if there are specific RAM savings to be had. Don't know enough about the inner workings and the sandbox android puts its apps in or how apps interact with system services. Sadly, i can't think of a good way to test it out either, but i'll be keeping an eye on this topic for someone (much) more knowledgeable to come along.
Harbb said:
Sadly, i can't think of a good way to test it out either, but i'll be keeping an eye on this topic for someone (much) more knowledgeable to come along.
Click to expand...
Click to collapse
Enter bedalus, stands there with a vacant expression on his face. Harbb looks disappointed.
kernels ; battery ; ROM ; gov/sched
That entire paragraph was dedicated to you bedalus, we both know that.
Lol
I hope someone can answer this though.
kernels ; battery ; ROM ; gov/sched
Wait for someone............
Sent from my Nexus S using xda premium
KSM does not improve performance on Android just like that - all enabling KSM does, is enable SUPPORT for the Feature but Applications would have to make use of the feature, which they don't.
You can easily verify this like that :
echo 1 > /sys/kernel/mm/ksm/run
<wait and/or run the Applications of your choice>
cat /sys/kernel/mm/ksm/pages_sharing
IF the above shows a value > 0 then you are making use of KSM else it's just available, without anyone using the feature.
Here's an interesting Article that gives a little more insight :
http://www.linux-kvm.com/content/using-ksm-kernel-samepage-merging-kvm
By the way, the same is true for ZCACHE. If you really want to make better use of your Memory (RAM) then using ZRAM as a Swapdevice does work (and may often make sense, too).
That all said : There appear to be efforts to make use of KSM http://forum.xda-developers.com/showthread.php?t=1464758 - so things may well change ...
any update on this...?
System.out.println("Hello peoples");
==>The purpose of this guide is to help people who don't know anything about programming,aren't modders,guys with knowledge about technology.
==>Initially I loved computers and their capabilities and have a little knowledge on the C and Java languages and just how computers (think).
You have to understand that computing tries to emulate human behaviors on how to solve problems.This is where programming kicks in.
So what is Android?
===================
1)Android in general is an operating system that was meant to run on mobile devices e.g. cell phones
and has expanded to tablets and notebooks.Android is divided into three language groups:
a)the system's framework and apps are written in java.
b)the Android's core [kernel] is pure C -language.
c)the Android's libraries are written in C++.
App libraries are called by apps that need more functionality that java can't provide.These are usually plugins like
decoders e.g. ffmpeg libraries that are used in video decoding,flash-players e.t.c. Here,java native methods are used and the android NDK platform
is used to enable the java apps call these libraries during execution.(Please read further on Android SDK,NDK and Jni).
So what happens when you have just flashed that new Rom.
====================================================================
First you have to understand that cell phones have their own embedded firmware not including the recovery and you will see why.
a)the recovery partition can be flashed to install aftermarket recovery roms.So even if you mess your recovery,you can still install again.(This varies with different phones).
b)their is that system chip which you cannot touch and there is a reason for it.Think of this partition like a PC BIOS.If you mess with your bios
your system is toast aka Bricking the system.Since phones are classed as embedded systems,manufactures don't want people messing with it as it would result into
cryptic errors and system vulnerability.
when the on button is pressed something simple yet complicated happens:
1==>The kernel which is compressed to save space usually in a (zImage) format is deflated or expanded.Since your NAND chips are partitioned,the kernel is given a very
special chunk in which is protected from user data.
2==>The kernel finds which base address it needs to start executing e.g.(0x00200000) and mostly when you put a wrong kernel base
address your phone enters into a boot-loop because arguments are being passed to invalid locations.It is important to know where your kernel base address
starts.You can try looking it up in your kernel sources(try searching for the mach msm folder and into the makefile) or just goggle it up or use Xda-Kitchen.
3==>with the correct kernel base address,execution starts.Usually the (init.rc) file in the (ramdisk folder) gives symlinks and creates structures i.e. folders that will house
modules and sets the correct paths to android files and framework.
4==>After arguments have successfully been passed,the handles are now passed to Android.Basically Android checks for (init.d) scripts that are available
this is true to GingerBread and Cyanogenmod 7.After that audio checks are done followed by camera services and then arguments are passed to (core.jar),a
critical framework file which is huge around 50Mb in size in CyanogenMod 7.
5==>Here the DalvikVM (Dalvik virtual machine) is called and the process of optimizing your system files starts.The framework get optimized first as this contains critical
code needed to run your device.Then your flingers and renderers and called.This are engines used by android e.g. (pixelflnger) which is used in touchscreen.
Your system's sensors are usually started around this point (your compass,light sensors e.t.c)your phone apps e.g. contacts,calendar get optimized around this point
and depending on the number of apps your manufacturer installed,it will take some time.
6==>your network get's activated around this point and is probably the time the capacitive buttons and lights on your gadget light up.This is usually a good indication
that your system has loaded.When your bootanimation ends the handles are passed to activate your home launcher.There is usually still a lot of activity going
on to fully ready your system and this is why if you try to use it immediately especially on low-end phones,your system lags or get (not responding) errors.
common misconceptions
=====================
1==>There is are reason why goggle gave minimum specifications needed to run Android because this is a full operating system(OS) unlike the past relic
phones that ran on 50MHz processors.
2==>please don't complain that some of your Ram is not the same as specified on the phones catalog(e.g. you have 256Mb of Ram but in the task manager it indicates you have 179Mb).
There is a reason to it.There are core processes that eat a chunk of your Ram and are hidden so that you don't try silly stuff like trying to kill them in thinking that your are trying to get more memory.
Think of it like this,it's like trying to kill (services.exe or svchost.exe) processes in Windows.You will just be trying to get system hangs,bluescreens
or just a system crash.
Will Add more info later.Please feel free to correct anything i might have not addressed properly or share your views.
Happy modding.
Kernel topic
============
In simple terms,the kernel is the core of any operating system e.g. windows, Mac,or any Linux distro like Ubuntu and Android.Android kernels come mostly
in the form of a compressed kernel (zImage).The kernel is written in pure C-language,which gives it direct access to memory and registers unlike java
which has to pass through the java VM(virtual machine).This makes code written in C-language to be very fast and robust but also dangerous.
==>Many Androids in the eclair regime ran on kernel 2.6.29. This was not a complete kernel and as by my experience there was alot of code missing from it.
2.6.29
======
==>a lot of androids did not have adb functionality due to the framework being embedded to allow USB mounting to PC.This was a very rigid method
of doing it(also a very old method).
==>In the case of other devices, when viewing the internal task manager,many processes were viewed as (0.00) byte files.In essence you could not determine
the amount of RAM your app was taking.This is true in the case of huawei u8120.
==>In the case of shutting down the phone,even in some cases under load,it did so very fast.It killed threads and handles mercilessly. Many people misunderstood
this concept and thought their phones ran faster as compared to 2.6.32 kernel.
2.6.32.9
========
Here there was a ton of improvements as developers and modders became more aware of support and tweaks.
==>Fixed issues like the internal task managers.It was now possible to accurately know how much RAM your apps were taking.
==>Resolved how the android system shutdown.Instead of merciless killing of handles and threads still running,it killed them appropriately.This is why
when shutting down your system takes a while.You can use adb to see these events.
==>Usb mounting to pc was also made somewhat generic and flexible across many devices.
==>It did change some methods on how the camera is being accessed mostly in eclair,donout and earlier versions of android phones.This issue made
cameras not function.
==>wifi methods were also changed and developers and modders had to re-write their code to allow compatibility.
==>This was also the year of many froyo phones and overclocking was a common thing.
extra notes
===========
During eclair era,many developers were in such a hurry to produce new android phones every few months that they never even thought of long-term support
to newer versions of android that would come much later.This is where i have to praise Iphones for standardisation of its OS across all of its devices.
==>If you try to copy paste code from 2.6.32.9 and paste it into 2.6.29 and expect it to work then expect tons of errors when compiling.The (g_android)
module was properly coded into 2.6.32.9 so if you try enabling it an older kernel,expect errors.
==>If you are a modder and looking to buy a new phone,please see if it has a fan base of coders or support.Avoid buying phones where you will not get
any support from the manufacturers or other devs e.g Huawei Technologies.This company sucks alot.After they produce a phone they forget about you the customer
so you will have to handle the upgrades all by yourself.
Do not use this command (make -i)when compiling kernels.It will skip errors and you may smile after it's finished but the end is just a tragedy.your kernel is bound
not to function properly or even function at all.
Happy modding.
this guide is under construction still and not quite finished - check back soo
Hi guys, I'd like to share some knowledge and as I'm going to start a blog doing so, I felt this would be as good as anywhere to start.
This technique will allow you to create disk images of your chosen size on as many storage devices as you want and that your device supports. There are a few advantages and reasons someone would do this. Storage flexibility, performance, data redundancy, easy backups. The techniques outlined here will likely be obvious to any linux pros, but perhaps not thought about in a mobile context.
While the current instructions and application of this technique is relatively complex, part of my goal in starting this is to gauge interest on whether it would be worth making a tool to do this. Perhaps there is one already, or another how-to. I didn't find any in my research in setting this up, albeit as a linux guru I didn't have to do much.
I will also include an APPENDIX including instructions on how to automatically compresss a filesystems contents (READ&WRITE) to save space with fusecompress. I may also add one for an encrypted filesystem if someone requests it.
DIFFICULTY: Hard for noobs, Easy for Pros.
REQUIREMENTS;
- Android device (rooted) preferably with multiple storage (sub-)systems (one will do if all you want is imaging or compression)
- Currently LINUX MUST BE INSTALLED ON YOUR DEVICE AND HAVE A BIND MECHANISM TO ACCESS THE MOUNT POINT IN ANDROID. As mentioned above, if the interest is there I will port the utils and make an app to manage it. I recommend Galoula Linux Installer w/ Debian stable. This tutorial is based off that setup but will likely work w/ Ubuntu and other Linux methods with few changes. Once all the tools are ported/compiled for Android, a Linux installation will not be required. You could very well compile it yourself..
- Working apt-get package manager and an internet connection.
Linux packages/tools needed & requirements/dependencies:
- lvm2, fuse, fuse-utils, libfuse, gvfs-fuse, libfuse2, libfuse-devel*, gcc/compiling tools*, fusecompress*+, busybox binaries or equivilent, fdisk, dd, mke2fs (or equivilent for your chosen filesystem), and iozone for benchmarking if you want.
- You will need a kernel & linux that supports fuse/loopback devices (most do!) or you can load the loopdev module. You will need at least 1 or 2 or more free loop devices depending on your configuration. There is a kernel parameter to change the maximum. **TODO: Add instructions for that **
- * and +: You will only need libfuse-devel and gcc for compiling fusecompress ONLY IF YOU NEED/WANT COMPRESSION. (If your chosen distro has a working fusecompress package you can use that, however at the time of writing DEBIAN-STABLE has not included an updated package for it. Squeeze had it, wheezy has some patches and updates for it but is not released as a package nor can I get it to work stabily after I spent hours convincing it to compile. I should note that this package is based off the CPP aka 1.9x branch (v2.6) and isn't completely stable compared to the older (slightly slower) but stable 0.9x branch. The APPENDIX instructions fot fusecompress ARE FOR THE STABLE/0.9 BRANCH)
My setup (as a reference):
Motorola XT875 aka Targa aka Bionic running Android 4.2.2, Avatar ROM (based on Cyanogenmod CM10.1) Latest Weekly Beta
16G internal device/SD storage, 16G patriot class 10 MicroSDHC. Dual-core ARMv7 armhf @ 1.2ghz, using DEBIAN-STABLE (Wheezy) under Galoula Linux Installer installed to it's own separate Ext3 partition on the external SD and Android Bind active. Also using my tablet, an a13 clone, with same Linux configuration.
I have a 2GB disk image on the internal, and a 2GB image on the microsd and they are combined as RAID0 (striping) w/ 4k stripes, ext3 fs w/1kb blocksize, 4kb stripe stride and compressed w/LZMA compression level 3-8 depending on my mood. It is setup like this for storing source code (cm-10.1 source actually, I am going to attempt to compile it entirely on mg phone, or at the very least my own modified portions.
Step 1.
--> If you're a pro, install all the stuff above, skim this section (mainly the part on finding an open loop) and skip to Step 3. Otherwise continue reading.
1a.
Boot up your linux distro on your Android. For me this means opening TerminalIDE, launching a term window, typing 'su' to gain root and then 'linuxchroot.sh' to launch the linux environment.
1b.
Then, ensure you have loop/fuse support.
Code:
ls /dev/block/loop*
.. should list all your systems loop devices, you can also type:
Code:
losetup /dev/block/loopX
(where X is a device number) to find out if the device is free or used by another app.
A response like:
Code:
losetup: /dev/block/loop0: 0
generally means it's available, while
Code:
losetup: /dev/block/loop1: 0 /mnt/asec/com.some.app/data.img
means another app is using it and you shouldn't.
Remember this as you will need to find anywhere from 1-4+ unused loop devices depending on your desired configuration.
1c. *
Next, it's best to ensure you're distro is up-to-date, this is optional* but recommended.
Code:
apt-get update
..then..
Code:
apt-get upgrade
..then, again (for good measure: update before you upgrade, upgrade, then update the upgrades..)
Code:
apt-get update
Step 2.
2a.
Once that completes, start installing the required packages:
Code:
apt-get install lvm2 fuse fuse-utils libfuse libfuse2 gvfs-fuse
Press Y and agree to any dependencies, wait for it to finisn.
2b.
You will need to choose which filesystem to use. I recommend ext3, but you will have to choose which is best for your needs. FAT is supported almost anywhere out-of-box, but is CRAP. NTFS is a bit better but is likely unsupported by your device, exFAT may be an option even. Whichever you choose you will need the appropriate tools (mkfs) to create/format it.
For both FAT and EXT support (or rather creation ability - mounting & read/write support may be limited by your kernel/modules loaded):
Code:
apt-get install dosfstools e2fsprogs e2fslibs
Step 3.
In this step you will create the disk image(s), initialize the partition table(s) and file system(s) and the steps will vary based on your goals, so read carefully. Since we are dealing with images, there shouldn't be much risk for damaging your device/data as long as you're careful. I should note, for the less experienced, that incorrect parameters with some of these commands can wipe your device or SD, if you read the directions carefully and only use the disk image(s) with these tools you will be safe, of course, after you put data within the image, it can also be overwritten with some commands.
3a.
You will now decide which, and how many devices to put images and how large they will be, and most importantly, your logical volume (RAID, non-RAID, span/JBOD, etc.) configuration. I will go into brief detail on the different configurations and use cases.
RAID0 aka STRIPING
Uses/Performance/Pros/Cons
Primarily aimed at performance, NO REDUNDANCY PROVIDED.
RAID0 will write one X byte block per Y devices looping in order. This effectively splits/distributes your data evenly among member devices.
Disk size will be size=devices*device size (or *disk img size)
All images (or devices) need to be the same size, extra bytes will be left unused.
Performance (P) (aka throughput) should be/is about (in theory) S*N=P where S is the average device speed (among all members, assuming similar/identical devices & busses, also note that P will never really exceed the slowest devices avg/max speed*number of drives, so slower drives are a bottleneck, try to use equal or identical devices) and N is the number of devices. On a physical hard disk drive, seek times can be lowest avg seek*members at best sometimes, though seek latency isn't an issue on our flash devices.
My benchmarks hardly show any impressive performance increase however my internal and external sd are obv. not matched, and I haven't done a ton of testing/benching/optimising.
Raid0 examples: 2x4GB disk images, each on a class 10 MicroSDHC card will appear as one 8GB device that can perform as fast as both devices can work together, about twice as fast as one, seek times won't decrease much. 4x200GB SATA drives striped will be one 800GB drive that is almost 4x as fast, however seek times may decrease to 1/4.
RAID1 aka MIRRORING
Uses:
Primarily aimed at data redundancy.
RAID1 will write one X byte block to EVERY device. This effectively MIRRORS your data among member devices.
Disk size will be equivilent to the device/img size regardless of the amount of devices it's mirrored to.
WRITE PERFORMANCE will be the same as the slowest single devicd, READ PERFORMANCE can increase up to x*member devices as it can act 'sort of like' RAID0 for reading, it doesn't need to read EVERY byte from EVERY device, it only needs to read 1/N (n=num of devices) from each device reading all simultaneously.
I have not tested mirroring in this setup yet.[/b]
JBOD aka JussaBunchO'Disks
Uses:
Just basically spans the devices for extra space. No performance techniques, striping or mirroring, rolls off the end of one disk (or img) onto the next. Devices/imgs can be any size and there's no performance hit for using devices/imgs on the same bus or device.
OK -- I started this a while ago, and here's my attempt at finishing it.
Not sure if you need fusefs unless you want to do file system compression.
You do need the lvm2 package, however.
Get down to it:
Now you have to create your image files, loop them, and raid them.
To create the images, use this command (this creates a ~100mb image, adjust the size as necesary):
Code:
dd if=/dev/zero of=/mnt/sdcard0/.myRaidImage-0.img bs=1024k count=100k
Code:
dd if=/dev/zero of=/mnt/sdcard1/.myRaidImage-1.img bs=1024k count=100k
Then, loop them to two free loop devices (see above to find out if they're free):
Code:
losetup /dev/block/loop1 /mnt/sdcard0/.myRaidImage-0.img
losetup /dev/block/loop2 /mnt/sdcard1/.myRaidImage-1.img
Before using any of the lvm2 tools below, you should check that your /etc/lvm/lvm.conf is correct, you may have to change the dirs from /dev to /dev/block for example.
Now, use pvcreate to create lvm physical volumes with those images:
Code:
pvcreate /dev/block/loop[1,2]
And vgcreate to create a lvm volume group for your lvm volumes:
Code:
vgcreate vg0 /dev/block/loop[1,2]
Now, we can create either a JBOD/spanned/contiguous disk, a mirrored disk, or a striped disk.
First, we will create a mirrored logical volume, with a log (in contrast to lvm2's raid1 mode which has no log):
Note: I had to force lvm2 to manage devices in /dev instead of relying on udev (which my device doesn't have), this is quite possibly the case for you as well, the option is in /etc/lvm/lvm.conf.
Code:
lvcreate -m 1 --mirrorlog mirrored -l 40%VG vg0
If that is successful, you now have a usable logical volume. To actually use it however, you will have to activate a filesystem on the device.
Code:
mkfs.ext3 /dev/vg0/lvol0
Then, you can create a mount point and use it:
Code:
mkdir /mnt/mirrored
mount /dev/vg0/lvol0 /mnt/mirrored
NOW YOU CAN USE IT.
The gentoo wiki has a great page on lvm2, and how to use it: https://wiki.gentoo.org/wiki/LVM
Most of it can be applied here with my instructions.
I will come back later and add specific instructions for striping and other modes, including stream compression and perhaps encryption - however - it make take me a while, which is why I posted the link - you should be able to figure it all out with this & that
reserved!
Hi,
do you have any update?
I'm wondering know how to do it.
Using madam or something else?
Thanks.
Sorry, I was sidetracked and forgot to finish. I will try to remember to finish this soon.
So, I got around to finishing it, at least to the point where it's usable and demonstrates mirroring. Not much different process for the other modes, but I will get around to exact instructions later if needed.
cheers.
So, I got around to finishing it, at least to the point where it's usable and demonstrates mirroring. Not much different process for the other modes, but I will get around to exact instructions later if needed.
cheers.