[TOOL] Lanchon REPIT: The Data-Sparing Repartitioning Tool For Android - Android Software/Hacking General [Developers Only]

[TOOL] Lanchon REPIT: The Data-Sparing Repartitioning Tool For Android - Android Software/Hacking General [Developers Only]

Lanchon REPIT: The In-Device Data-Sparing Repartitioning Tool For Android
REPIT is a simple, safe, device-only, data-sparing, and easily portable repartitioning tool for Android devices:
device-only: just flash a zip file in recovery to repartition the device.
simple: rename the zip file before flashing to configure your choice of partition sizes, file systems, wipes, etc.
safe:
a correctly ported REPIT can never hard-brick your device.
before starting, REPIT checks for the existence of all the tools that will be needed for the task at hand, verifies that the current partition layout passes several sanity checks, checks and fixes all the involved file systems, and verifies that the new partition layout will meet its sanity checks too. REPIT performs a dry-run of the complete repartitioning process to detect possible problems early on.
if REPIT fails, it will nonetheless try to restore your device to a working in-between state.
my estimate is that around 1000 users have already repartitioned with REPIT with no incidents of data loss, boot loops or soft bricks (2016-04).
easily portable: a simple configuration file is all that is needed to port REPIT to a new device.
Documentation (HELL YEAH, read this!) -> HERE
Downloads -> HERE
this is a user discussion thread. post here for general discussion and user support.
for official support create a Github issue, but only of you have read the docs.
new versions will not be announced here.
please 'watch' the Github project to receive notifications.
special thanks to the.gangster.
XDA:DevDB Information
Lanchon REPIT, Tool/Utility for all devices (see above for details)
Contributors
Lanchon
Source Code: https://github.com/Lanchon/REPIT
Version Information
Status: Stable
Created 2016-04-13
Last Updated 2019-06-07

Reserved

Reserved

Good to have a place again to discuss issues or simple questions.
Great tool, that meanwhile left its corner where it grew up (the Galaxy S2 - i9100), meanwhile supporting a growing number of devices.
RELEASES AND CHANGELOG

the.gangster said:
Good to have a place again to discuss issues or simple questions.
Great tool, that meanwhile left its corner where it grew up (the Galaxy S2 - i9100), meanwhile supporting a growing number of devices.
@Lanchon
you accidently linked the downloads to your flashize github instead of the Repit one.
Click to expand...
Click to collapse
lol thanks, mucho copy/pasta, fixed!

Thanks so much!
Perfect tool, used this evening on a old but still used I9100!

Tried this on a n7000 recently, tried to resize cache, which didn't work, saw partition structure with modem after cache, and understand your concerns in the comments. I imagine the we could so some experiments like you are doing for the Honor 4, however I'm not at the stage where I want to volunteer my device as potential sacrifice.
A question on "wipe", how sure are we that "wipe" actually wipes, ie is all the previous data non-recoverable down to a block level?
Does the wipe trigger some type of NAND erase cycle? This would be problematic on n7000 given the hard brick issues with NAND firmware there, and on the n7000 IIRC the ioctl() is filtered and purposely ignored, maybe this is also an issue with i9100? I don't recall.
If I have a used ext4 parition, and then format & wipe it then I think a fstrim happens. If I then change it to vfat, and dd the partition, am I able to recover blocks from the original ext4 partition, or does the trim cause the nand to regard the blocks as empty and return 0x0000000 or 0x11111111 or something deterministic?
I'm looking for a wipe that actually wipes, ext4 and vfat, rather then my current solution which is to dd if=/dev/zero of=/path, which is slow and is difficult to get a progress report, as kill -USR1 is a PITA.

samarium said:
Tried this on a n7000 recently, tried to resize cache, which didn't work, saw partition structure with modem after cache, and understand your concerns in the comments. I imagine the we could so some experiments like you are doing for the Honor 4, however I'm not at the stage where I want to volunteer my device as potential sacrifice.
A question on "wipe", how sure are we that "wipe" actually wipes, ie is all the previous data non-recoverable down to a block level?
Does the wipe trigger some type of NAND erase cycle? This would be problematic on n7000 given the hard brick issues with NAND firmware there, and on the n7000 IIRC the ioctl() is filtered and purposely ignored, maybe this is also an issue with i9100? I don't recall.
If I have a used ext4 parition, and then format & wipe it then I think a fstrim happens. If I then change it to vfat, and dd the partition, am I able to recover blocks from the original ext4 partition, or does the trim cause the nand to regard the blocks as empty and return 0x0000000 or 0x11111111 or something deterministic?
I'm looking for a wipe that actually wipes, ext4 and vfat, rather then my current solution which is to dd if=/dev/zero of=/path, which is slow and is difficult to get a progress report, as kill -USR1 is a PITA.
Click to expand...
Click to collapse
hi,
well, it's not that it didn't work lol! cache was intentionally left out of the resizable set. you've seen the comments and the 4X discussion so you know exactly why.
i wouldn't do the experiments anyway. in the Honor there is something useful to be gained: 1 extra GB to a 4GB /data, which is a lot. in the S2 cache is 100MB. cache is unused nowadays, so the only logical thing to do is shrink it. plus you want something left, like 32MB. so we will risk devices for 68MB? not on my watch
secure wipe is a very complex issue and there's an awful big can of worms there, believe me. it'll try to paint a picture.
ssds have a bunch of 'trim' commands. there's ERASE and DISCARD, and later TRIM was added, all with slightly different semantics. ans they all come in standard and secure flavors. most recently you can come across the standard non-queued commands or the new queued variants.
the non-secure flavored commands are performance-oriented: they are hints to help the ssd. the drive can ignore them if it wants. it can ignore requests for short ranges but honor requests for large extents, or whatever.
more recently devices can implement DRAT and RZAT (deterministic read after trim and read zero after trim), ending the previous non-deterministic behavior. take a look:
http://www.t13.org/documents/uploadeddocuments/docs2010/e09158r2-trim_clarifications.pdf
but there are a million bugs in ssd firmware so linux blacklists broken functionality. like RZAT in many drives (you can google it), even if the drive says it supports it.
in any case RZAT drives just alter the FTL (flash translation layer) structures to unmap the block, they don't erase flash, so your data is typically still there but normally inaccessible. and there is probably even a history of previous snapshots of the block, not just the latest.
that's where the secure commands enter the picture: supposedly the firmware must actually erase from flash all copies of the targeted blocks. in reality, firmwares are so buggy i wouldn't trust them for that, except maybe if the drive is intel brand.
back to the real world... the exynos:
>Does the wipe trigger some type of NAND erase cycle? This would be problematic on n7000 given the hard brick issues with NAND firmware there, and on the n7000 IIRC the ioctl() is filtered and purposely ignored, maybe this is also an issue with i9100? I don't recall.
you are living in the past. more than two years ago i started arguing that non-secure trim commands should in all probability work fine and could be used to speed up the S2 family. rationale is complex but you can google my threads about it. no-one listened so 2 years ago i learned to build kernels and did some tests with volunteers and began publishing trimming kernels (google i9100 trim for a bit of history). finally in 2016 trim was finally added to official CM 13, 12.1 and 11.
you could be trimming your device without knowing: if you are using recent official CM, you are already trimming. many other kernels switched to trim too.
but... in the S2, secure trim kills the eMMC, so only non-secure variants are issued by trim kernels. secure erase is not a possibility.
i don't know if the eMMC is RZAT, but i wouldn't trust the firmware anyway. if you want to wipe, dd is your best option. but don't use zeros!!! some ssd's compress data. use a pseudo random stream instead. after all, non-secure trim is not very different from a write, in the best case, so just write for safety and be done with it.
yes, dd won't wipe reserved blocks. no way to do that except:
-MAYBE... an eMMC full reset (including bootloader) by resizing the boot partitions (google) which is very dangerous and can easily hard brick the device.
-putting a power drill to you beloved S2
regarding what repit does:
-it trims only if the kernel has trim enabled.
-and it trims only ext4 partitions.
thanks!

Thanks for the detailed reply.
I'm currently using i9100 CM12.1 & n7000 CM13/Nightowl, and have been trimming vfat partitions.
Now you remind me, I do recall reading threads about trim safe a while ago, but I think I was more irritated at the time by the seemingly random kernel crashes & process terminations that eventually were tracked down to kernel interrupt fp register save/restore, and went down the Sony path for a while while the n7000 was unusable. Amusingly, the n7000 now has CM13, while the Sony Z Ultra is languishing and isn't looking good for CM13 at the moment.
While I agree that for general SSDs, zero blocks could be compressed, and blocks could be deduped, do you really think this is happening in n7000/i9100 emmc? I dont expect so, seems they seem too old. I agree that a /dev/urandom would be a better if= than /dev/zero for dd, however this is going to use much more power, and I'm usually wiping a phone with limited time and power.
I'm not trying to protect from something as serious as direct flash chip access, just want to be reasonably sure that whatever was on the phone has been overwritten and is not easily available to standard and even mildly esoteric extraction methods.

samarium said:
Thanks for the detailed reply.
I'm currently using i9100 CM12.1 & n7000 CM13/Nightowl, and have been trimming vfat partitions.
Now you remind me, I do recall reading threads about trim safe a while ago, but I think I was more irritated at the time by the seemingly random kernel crashes & process terminations that eventually were tracked down to kernel interrupt fp register save/restore, and went down the Sony path for a while while the n7000 was unusable. Amusingly, the n7000 now has CM13, while the Sony Z Ultra is languishing and isn't looking good for CM13 at the moment.
While I agree that for general SSDs, zero blocks could be compressed, and blocks could be deduped, do you really think this is happening in n7000/i9100 emmc? I dont expect so, seems they seem too old. I agree that a /dev/urandom would be a better if= than /dev/zero for dd, however this is going to use much more power, and I'm usually wiping a phone with limited time and power.
I'm not trying to protect from something as serious as direct flash chip access, just want to be reasonably sure that whatever was on the phone has been overwritten and is not easily available to standard and even mildly esoteric extraction methods.
Click to expand...
Click to collapse
ok. then i would format in ext4 and trim. you can use repit for doing both in a single shot, but you need a trimming kernel. take a dd snapshot of one or a couple of short pieces of the partition, both before and after the operation. compare them and post here, i'd surely like to know .
but... keep in mind that there are several different emmcs out there in the S2s, so you'd need to validate for each one. which reminds me, also post your device emmc IDs! use the eMMC brickbug check app for instance.
regarding the /dev/urandom vs /dev/zero thing, i'm not religious, i don't hold unwarranted beliefs about what i don't know. i investigate, i have no gut feelings on stuff
but if you want to know: i'm a bit obsessive, i certainly would have done at least something if i was coding the FTL, even if it is old. if your design is RZAT, the least you can do is trim the block instead of writing it when data is all zeros. you may think this would slow the emmc down, but this can be implemented in pseudo-zero time: writing usually entails generating some kind of checksum of the data. so note the magic checksum that corresponds to an all-zero block, and only if the checksum of the data being written matches the magic, actually check for all-zero data before writing. of course this can be 'attacked' by intentionally generating checksum collisions but who cares. the time saved by not garbage collecting, not using a free block, and not writing the block with all-zeros certainly pays for the checksum comparison against the magic value. also note that straight brute force comparison of all blocks is mostly ok too, as most written blocks will fail early anyway (ie, after one or a couple of bytes).
of course The One True Solution(R) is to encrypt the phone before using it...... assuming that you can actually wipe the key when you need to lol. ideally the key is in a separate hardware keystore adept at wiping keys. but on the S2 it just in the same eMMC . so this is a bit like explaining where the universe came from by saying that god created it, and then conveniently forget to explain where did god come from lol.
the power drill solution still mostly works on the S2 though

Pls help
##################################
Lanchon REPIT
A Data-Sparing Repartitioning Tool
Version: 2016-04-15
Device: i9100
Copyright 2016, Lanchon (GPLv3)
####################################
===== PRELIMINARY CHECKS =====
info: valid package names: <prefix>[-(system|data|sdcard|preload)=<conf>]...<suffix>
info: valid partition <conf> values: [<size-in-GiB>|same|min|max][+[keep|wipe][+[ext4|vfat|f2fs|swap|raw]]]
----- DEFAULTS -----
system = size:same + content:keep + fs:ext4
data = size:same + content:keep + fs:ext4
sdcard = size:same + content:keep + fs:vfat
preload = size:same + content:keep + fs:ext4
info: parsing package name
----- CONFIGURATION -----
system = size:1.0 + content:keep + fs:ext4
data = size:6 + content:keep + fs:ext4
sdcard = size:max + content:keep + fs:vfat
preload = size:min + content:keep + fs:ext4
info: disabling swap
BusyBox v1.22.1 bionic (2016-01-10 12:49 +0530) multi-call binary.
Usage: dirname FILENAME
Strip non-directory suffix from FILENAME
info: copying package to '/tmp'
cp: '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' and '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' are the same file
[ERROR 1]
Whats wrong

just change your current directory to /tmp and try to flash it again from there. (all within the recovery)

Lanchon said:
ok. then i would format in ext4 and trim. you can use repit for doing both in a single shot, but you need a trimming kernel. take a dd snapshot of one or a couple of short pieces of the partition, both before and after the operation. compare them and post here, i'd surely like to know .
but... keep in mind that there are several different emmcs out there in the S2s, so you'd need to validate for each one. which reminds me, also post your device emmc IDs! use the eMMC brickbug check app for instance.
regarding the /dev/urandom vs /dev/zero thing, i'm not religious, i don't hold unwarranted beliefs about what i don't know. i investigate, i have no gut feelings on stuff
Click to expand...
Click to collapse
bauner produced a new n7000 IsoRec with TWRP + OTG which I wanted, so while I was installing I tested a few things.
I generated a block binary file with each of the 1kB blocks annotated for the 200MB n7000 cache partition, and dd'd the file to the cache partition prior to each mkfs.
Code:
$ cat blocknull.pl
#!/usr/bin/perl
use strict;
my $count = $ARGV[0];
$count = 1 if $count == 0;
$count *= 1024 if $ARGV[0] =~ /M$/;
$count *= 1024 * 1024 if $ARGV[0] =~ /G$/;
for (my $i = 1; $i <= $count; $i++)
{
printf "\nBLOCK %10d kB %6d MB %2d GB BLOCK\n", $i, $i / 1024, $i / 1024 / 1024;
print chr(0) x (1024-43);
}
$
Code:
BLOCK 1 kB 0 MB 0 GB BLOCK
...
BLOCK 204800 kB 200 MB 0 GB BLOCK
Then operated on both vfat and ext4 file systems to determine which of the 204800 block signatures where remaining after various operations.
Code:
# tr -d '\0' < /dev/block/mmcblk0p7 | grep BLOCK.*BLOCK > cache.x.y.desc
...
# wc -l cache.?.?.*
204800 cache.1.1.dd
201632 cache.1.2.mkfs.fat32
197041 cache.1.3.copyfile
204800 cache.2.1.dd
0 cache.2.2.mkfs.ext4
204800 cache.3.1.dd
197380 cache.3.2.mkfs.ext4.twrp
60 cache.3.3.fstrim
60 cache.3.4.copyfile
#
So as expected vfat format doesn't change the underlying data returned, it just gets changed as it is used, as show by the signature counts on cache.1.*
ext4 is more interesting, as I tried a manual mke2fs -t ext4 for cache.2.* and TWRP wipe ext4 for cache.3.*. The TWRP wipe is not using secure discard as can be seen from recovery.log, but things are largely back on track after the first fstrim except for a hole from 131081 kB to 131125 kB.
Code:
recovery.log:
warning: wipe_block_device: Wipe via secure discard suppressed due to bug in EMMC firmware
Maybe more interesting is that even though it seems manual mke2fs -t ext4 does a wipe, when I do a fstrim is still says 180MB trimmed, which I don't expect, some inconsistency in fs metadata setup in mkfs maybe? Doesn't matter to me, 1 or 2 wipes, but seems I will be using mke2fs -t ext4 via terminal and then using TWRP wipe/format as required.
FYI, EMMC IDs PNG attached.

samarium said:
bauner produced a new n7000 IsoRec with TWRP + OTG which I wanted, so while I was installing I tested a few things.
I generated a block binary file with each of the 1kB blocks annotated for the 200MB n7000 cache partition, and dd'd the file to the cache partition prior to each mkfs.
Code:
$ cat blocknull.pl
#!/usr/bin/perl
use strict;
my $count = $ARGV[0];
$count = 1 if $count == 0;
$count *= 1024 if $ARGV[0] =~ /M$/;
$count *= 1024 * 1024 if $ARGV[0] =~ /G$/;
for (my $i = 1; $i <= $count; $i++)
{
printf "\nBLOCK %10d kB %6d MB %2d GB BLOCK\n", $i, $i / 1024, $i / 1024 / 1024;
print chr(0) x (1024-43);
}
$
Code:
BLOCK 1 kB 0 MB 0 GB BLOCK
...
BLOCK 204800 kB 200 MB 0 GB BLOCK
Then operated on both vfat and ext4 file systems to determine which of the 204800 block signatures where remaining after various operations.
Code:
# tr -d '\0' < /dev/block/mmcblk0p7 | grep BLOCK.*BLOCK > cache.x.y.desc
...
# wc -l cache.?.?.*
204800 cache.1.1.dd
201632 cache.1.2.mkfs.fat32
197041 cache.1.3.copyfile
204800 cache.2.1.dd
0 cache.2.2.mkfs.ext4
204800 cache.3.1.dd
197380 cache.3.2.mkfs.ext4.twrp
60 cache.3.3.fstrim
60 cache.3.4.copyfile
#
So as expected vfat format doesn't change the underlying data returned, it just gets changed as it is used, as show by the signature counts on cache.1.*
ext4 is more interesting, as I tried a manual mke2fs -t ext4 for cache.2.* and TWRP wipe ext4 for cache.3.*. The TWRP wipe is not using secure discard as can be seen from recovery.log, but things are largely back on track after the first fstrim except for a hole from 131081 kB to 131125 kB.
Code:
recovery.log:
warning: wipe_block_device: Wipe via secure discard suppressed due to bug in EMMC firmware
Maybe more interesting is that even though it seems manual mke2fs -t ext4 does a wipe, when I do a fstrim is still says 180MB trimmed, which I don't expect, some inconsistency in fs metadata setup in mkfs maybe? Doesn't matter to me, 1 or 2 wipes, but seems I will be using mke2fs -t ext4 via terminal and then using TWRP wipe/format as required.
FYI, EMMC IDs PNG attached.
Click to expand...
Click to collapse
>it seems manual mke2fs -t ext4 does a wipe
it shouldn't. mke2fs -i thought- doesn't trim unless invoked with a specific option. repit invokes with the flag:
https://github.com/Lanchon/REPIT/blob/master/repit-fs-ext4.sh#L51
as to the question of why your markers were cleared anyway, i've no answer.
>even though it seems manual mke2fs -t ext4 does a wipe, when I do a fstrim is still says 180MB trimmed
some kernels inform the newly trimmed space: if you run trim twice you'll get "0 bytes trimmed" the second time.
other kernels (like CM's i9100) always inform (and presumably trim) the full partition free space, so 180MB is ok. if you run trim twice you'll get the same number of bytes both times.
>things are largely back on track after the first fstrim except for a hole from 131081 kB to 131125 kB.
here are some candidate causes:
a) ext4 fs is based on 'blocks' with each block being a constant number of sectors. it makes sense that fstrim should work on whole blocks. also, when ext4 writes a 'short block', ie one not completely filled with data, some sectors and the end of the block might remain unused. it makes sense for ext4 not to write these sectors at all. but the complete block will be marked as used given that the block is the granularity for that, so no sector in the block will be trimmed by fstrim. so some pre-format data might survive in these short blocks after fstrim by this mechanism, which could explain what you observed.
b) another cause could be that the emmc might ignore trim requests for spans shorter than some size or for misaligned spans (remember that erases are performed in large chunks) giving rise to the observed data survival. when formatting no data survives because partitions are position- and size-aligned to 1MB or 4MB, which might be enough to guarantee trim. (i find this explanation unlikely, but who knows.)
about repit:
repit formats and trims ext4 and f2fs in a single operation.
to wipe /data a single repit should be enough:
-data=+wipe
to wipe /sdcard you can take advantage of repit's capability to convert file systems and do two repits:
-sdcard=+wipe+ext4
-sdcard=+wipe
an idea that comes out of this is that repit could temporarily format in ext4 before formatting in other modes that do not support format-time trimming, such as when wiping vfat and swap. this would make the above 2 steps unnecessary.

darkz16 said:
[...]
info: disabling swap
BusyBox v1.22.1 bionic (2016-01-10 12:49 +0530) multi-call binary.
Usage: dirname FILENAME
Strip non-directory suffix from FILENAME
info: copying package to '/tmp'
cp: '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' and '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' are the same file
[ERROR 1]
Click to expand...
Click to collapse
this is very interesting.
the relevant lines are these:
https://github.com/Lanchon/REPIT/blob/master/repit.sh#L318-L322
Code:
if [ -f "$packageName" ]; then
if [ "$(dirname $(readlink -f "$packageName"))" != "/tmp" ]; then
info "copying package to '/tmp'"
cp -f "$packageName" "/tmp/"
hint="this package copied itself to '/tmp'; please run it again from there"
the line:
if [ -f "$packageName" ]; then
assures that the file $packageName exists.
but these logfile lines:
Code:
BusyBox v1.22.1 bionic (2016-01-10 12:49 +0530) multi-call binary.
Usage: dirname FILENAME
Strip non-directory suffix from FILENAME
are proof that 'dirname' is being invoked without argument, which means that 'readlink -f "$packageName"' produced an error!
how can 'readlink -f <file>' (which means 'canonicalize') give an error when invoked on an existing file is beyond me. it looks like your implementation of 'readlink' is broken.
what exact recovery are you using? who made it or where is it being distributed?
EDIT:
btw there is a quoting error in the code here but it shouldn't affect this. the line should be:
if [ "$(dirname "$(readlink -f "$packageName")")" != "/tmp" ]; then

darkz16 said:
[...]
cp: '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' and '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' are the same file
[ERROR 1]
Whats wrong
Click to expand...
Click to collapse
so i committed what i think is a fix for this problem that i cant reproduce:
https://github.com/Lanchon/REPIT/commit/c7bc3655acc89ee554d1dafe09c7bb9948483c03
and attached is build for you to try.
IMPORTANT: EXTRACT THE ZIP FIRST, DONT JUST FLASH IT!

Lanchon said:
so i committed what i think is a fix for this problem that i cant reproduce:
https://github.com/Lanchon/REPIT/commit/c7bc3655acc89ee554d1dafe09c7bb9948483c03
and attached is build for you to try.
IMPORTANT: EXTRACT THE ZIP FIRST, DONT JUST FLASH IT!
Click to expand...
Click to collapse
Hi Lanchon.
I'm installed this zip but i have the same problem.
I'm copy, before zipping the achive, on the \tmp directory of emmc
What have i do wrong?.
Thanks.

chapito said:
Hi Lanchon.
I'm installed this zip but i have the same problem.
I'm copy, before zipping the achive, on the \tmp directory of emmc
What have i do wrong?.
Thanks.
Click to expand...
Click to collapse
no log, no kernel info, no recovery info. cant help you.

Lanchon said:
no log, no kernel info, no recovery info. cant help you.
Click to expand...
Click to collapse
Hi.
Forgive me for not giving more information to solve the problem.
Yesterday I could partition the memory.
I had to change the partition type of memory EMMC to vfat and the script to start.
I used the IsoRec TWRP 2.8.7.0.
I used the last script publicated "lanchon-repit-20160415-system=1.0-data=same-sdcard=max-preload=min+wipe-i9100.zip"
Thanks

chapito said:
Hi.
Forgive me for not giving more information to solve the problem.
Yesterday I could partition the memory.
I had to change the partition type of memory EMMC to vfat and the script to start.
I used the IsoRec TWRP 2.8.7.0.
I used the last script publicated "lanchon-repit-20160415-system=1.0-data=same-sdcard=max-preload=min+wipe-i9100.zip"
Thanks
Click to expand...
Click to collapse
you didn't need to change your sdcard to FAT !!!
you could have used: -sdcard=max++ext4
if only you had read the instructions! that's what they are there for lol
well, it's your data you had to format, not mine!
thanks!

Related

Need help creating blank DOS partitions images for use with sdsplit

Since there have been some problems trying to create dos partitions larger than 2GB directly on the G1 sdcards, I came up with an idea for a hack that might be a useful (but sad) way to get around the broken(?) mkdosfs provided by android. However, to do this, I will need help since I only have a 1GB card.
Here is my idea: Create a series of raw "normalized" gzipped dos partitions of various sizes and make them available for download. So, to do this we will need people with large cards to contribute images of various standard sizes, say 3G, 3.5G, 7G, 7,5G, 14G, 15G, 15.5G... that they create using a standard method. These images would then be made available for others to download, and I would potentially update the sdsplit to know how and where to find them.
What do you think, would it be worth it, would it be useful? It should be fairly easy to do, here are the steps that I envision:
1) Contributor uses fdisk on the phone to create a dos partition of the exact desired size. Using the G1 fdisk might be important to ensure that the partition is created with the exact same size than the person who is going to use it. Perhaps even using the sdsplit script like this (this will also create an unmkfsed ext2 partition on the remainder)
# sdsplit -p -fs 3500M
2) Contributor uses dd to zero out the new partition. I think that zeroing out the partition first would help create the smallest gzipped image for these partitions. (Do not try this unless you know what you are doing, it will wipe out the first partition of your sdcard!)
# dd < /dev/zero > /dev/block/mmcblk0p1
3) Contributor uses a windows partition tool or non G1 based mkfs tool to format the new dos partition with FAT 32. See the various other tutorials for this.
4) Contributor uses gzip -9 on linux, or gzip on G1 (no -9 available with busybox) to compress the raw "normalized" partition to a file named after its size.
# gzip -c < /dev/block/mmcblk0p1 > /data/fat32_4G_3500M.gz
Note the 4G for the sdcard size
5) Contributor uploads the raw normalized gzipped partition somewhere common.
I did some measurements, and on my phone I was able to create a ~467K file from my normalized 500M partition. That's not great, about a 1/1000 compression ratio, but I would not be surprised to see much better compression ratios with bigger partitions. Even at the same ratio, a 15.5G partition should only be ~15.5M, smaller than most software updates.
So, does anyone have any suggestions for places to put these files which would be easily accessible from a script? If anyone is willing to make a normalized image, perhaps posting it here would be a good start? We could later gather them somewhere better once we try to automated downloading them.

MartinFick said:
1) Contributor uses fdisk
2) Contributor uses dd
3) Contributor uses windows
4) Contributor uses gzip
5) Contributor uploads
Click to expand...
Click to collapse
sounds like a lot of complicated steps for something we're trying to make simple

Perhaps I didn't explain the important part. Those steps above would be done only once for each partition size by someone here on the forum (the contributor mentioned above) for the benefit of others in the future.
Once we have a few standard size images, these images can be used to "format" a partition to FAT by someone else (not the image contributor) in one easy step on their G1 like this:
# gzip -c -d < /data/fat32_4G_3500M.gz > /dev/block/mmcblk0p1
This single step above will replace the normal mkdosfs step that sdsplit currently does and has trouble with over 2G. This means that eventually, this step can be automated with sdsplit. Make sense?

MartinFick said:
Perhaps I didn't explain the important part. Those steps above would be done only once for each partition size by someone here on the forum (the contributor mentioned above) for the benefit of others in the future.
Once we have a few standard size images, these images can be used to "format" a partition to FAT by someone else (not the image contributor) in one easy step on their G1 like this:
# gzip -c -d < /data/fat32_4G_3500M.gz > /dev/block/mmcblk0p1
This single step above will replace the normal mkdosfs step that sdsplit currently does and has trouble with over 2G. This means that eventually, this step can be automated with sdsplit. Make sense?
Click to expand...
Click to collapse
ah .. peachy .. i'll see if i can follow your directions there

This idea is now obsolete. I got my hands on a mkdosfs which seems to work with large partitions and have updated the sdsplit script to use it.

Using dm-crypt to Encrypt the SD Card...?

Trying to setup a script to automatically encrypt/mount/dismount the SD card using dm-crypt (the same tool used in Froyo's app2sd).
Targets as below...
A wizard to setup disk/partition encryption.
The key can be randomized by the device itself, or manually configured by user, and stored on the phone's memory (the point is that the content on the SD card will NOT be readable once removed from the device.
Automatically mount and dismount during boot and shutdowns.
Compatibility with the native USB mount feature.
It's my understanding that the encryption/decryption will be done in the background and transparent to the user. I also understand that there will be a hit on the performance and battery life.
It seems that Android uses vold, and most dm-crypt guides out there involves editing /etc/fstab which is not used on Android at all.
Pardon my ignorance, and any pointer will be appreciated!

Anybody...?

Anybody knows how Froyo's A2SD system works? There's no cryptsetup whatsoever but somehow it can use dm-crypt to store the apps on SD card?
It seems that Android's vold has parameters hardcoded and compiled into the binary.
Furthermore, it's not necessary to to use vold to mount the sdcard, good ol' mount can be used (no auto-mount through), but there's no cryptsetup to setup the partition.
I would like to do some rough experiments but needs crypsetup, or setup a cross-compile platform for compiling a static cryptsetup first.
P.S. Someone had worked on encrypting /data but it wasn't finished before he's switched to N900... (my target is a bit difference through, to encrypt the sd card which is more easily stolen... if the phone itself is stolen, it wouldn't do much good anyway cause someone can use adb to access /data, assuming the key is stored on the device not entered manually during each boot)
http://androidvoid.wordpress.com/2009/09/30/android-encryption-using-cryptsetup-and-luks/
P.P.S. The DROID PRO is said to have both device and SD card encryption as well. I suspect dm-crypt is used. Look forward to see how it's being handled...

Any pointers...?

: ).......................

Sorry, that I am not able to help you - although I would in fact appreciate if you achieve the aim somehow!

I'm really surprised this hasn't gained more attention. I sincerely hope someone discovers a way to fully encrypt SD cards.

Damian works on a kernel (still alpha) which so far already has cryptsetup.
Section galaxy s
I think this could be more easy achieved with steam recovery than cwm.
So far full encryption of blackberries is THE killer argument why android still doesnt go professional and get more attention.
Lets change that!
Sent from my GT-I9000 using XDA App

Push for implementating it in Android, please *star* the issues.
Issue 11211 - android - Android too insecure - Encryption of the SDcard is crucial - Project Hosting on Google Code
Issue 3748 - android - Add support for partition/block device encryption - Project Hosting on Google Code

I was able to get encrypted file system to work within a chroot'd debian environment; the key was to get the right cipher to work with the android kernel (aes-cbc-essiv:md5). The only reason I couldn't get it to work directly in android was the lack of the cryptsetup command.
Here's a guide, where a 64M file "dummy" is created filled with random junk, turned into a device, LUKS formatted, and mkfs'd. The second section mounts it, the third dismounts and frees resources.
HTH
Paul
Code:
=== create ===
dd if=/dev/urandom of=cryptvol.raw bs=1024 count=65536
mknod /dev/loop21 b 7 21
losetup /dev/loop21 /home/cryptvol.raw
cryptsetup --cipher aes-cbc-essiv:md5 --key-size 256 luksFormat /dev/loop21
cryptsetup luksOpen /dev/loop21 cr_cryptvol
mkfs.ext3 /dev/mapper/cr_cryptvol
cryptsetup luksClose /dev/mapper/cr_cryptvol
losetup -d /dev/loop21
=== use ===
mknod /dev/loop21 b 7 21
losetup /dev/loop21 /home/cryptvol.raw
cryptsetup luksOpen /dev/loop21 cr_cryptvol
mount /dev/mapper/cr_cryptvol /data/local/cryptvol
mkdir /data/local/cryptvol
cd /data/local/cryptvol
echo "this is a file in encrypted file system" >> README
== close ==
umount /data/local/cryptvol
cryptsetup luksClose /dev/mapper/cr_cryptvol
losetup -d /dev/loop21

@speculatrix
Are you able to use the chroot Debian rmount in Android at all?

[GUIDE] Creating and using filesystem images on Android

Hey all, This is my first post on the forum, so I thought I'd make it a guide.
I've run into a few roadblocks with installing some development tools like python and stuff, so I figured I'd try and make a filesystem image to hold all of it, what with Android preventing executables on your SD card from actually executing. You should NOT try and use this for freeing up space by moving apps to it. That just breaks things.
Things to know
A mount point is where you access the contents of a filesystem through.
A filesystem is basically a virtual disk.
I'm not responsible for your phone melting into a pool of acid or anything else you want to blame me for. You executed the commands, not me.
If there's been guides for this already, I didn't find them.
Playing with your system files is dangerous. Be careful
Playing with filesystems is even MORE dangerous. Don't do it unless you know what's going on.
Reversing and then combining two words in a sentence, then calling it a bit more dangerous is incredibly dangerous and can kill you. Don't do it.
As I said earlier: This is not intended to free up internal storage on your phone. It is intended to create extra space to put command-line executables.
This was created for those of us who have older phones that don't have a whole lot of internal storage. (Like the Milestone) This should also work on anything that meets the requirements listed below. I use it on a Motorola Milestone A853 running CyanogenMod 7.2.4f UMTS/SHOLES, and it took some tweaking, but it worked. Commands you put into your shell are marked with `bolded text and surrounded by back-ticks. Leave the back-ticks out. (`)`
Requirements! (obtained using `which <command name>` or a specified command)
Free space on your mSD
losetup (configuring loop devices)
mknod (creating loop devices)
mount (mounting filesystems)
mkdir (making directories)
su (root access)
umount (unmounting)
mke2fs (formatting)
Linux filesystem support (`cat /proc/filesystems | grep ext`)
Here we go!
This assumes you are going to use an ext4-formatted filesystem image mounted on /data/development/, with the image located at /mnt/sdcard/development.img. You WILL need to determine your phone's supported ext<num> filesystems using `cat /proc/filesystems | grep ext`. Choose the highest-numbered one.
Create the mountpoint, the image, and then format the image to ext4.
`mkdir /data/development/``dd if=/dev/zero of=/mnt/sdcard/development.img count=409600 bs=1024``mke2fs -l /mnt/sdcard/development.img -b 1024 -t ext4`
Create the block device, set it to point at the image, then mount it.
`mknod /dev/block/loop250 b 7 250``losetup /dev/block/loop250 /mnt/sdcard/development.img``mount -t ext4 /dev/block/loop250 /data/development`
Test it to see if it works.
`touch /data/development/testfile;ls /data/development/`
Use it.
Unmounting (/!\ For connecting to a computer to avoid corruption /!\)
`umount /mnt/sdcard/development.img``losetup -d /dev/block/loop250`
Well, that's all I've got. Suggestions and corrections, as well as good feedback are appreciated.
Some PAQs (Possibly Asked Questions)
Q: I'm getting some errors. Here they are: pastebin.com/some-errors
A: Great, I'll see what I did wrong, and if it's something on your end, I'll try and help you out.
Q: It dun werk
A: Tell me the errors instead of complaining about them. Just saying that does not help me and wastes my time and bandwidth.
Q: "No space left on device"; "Operation not permitted"; "No such file or directory"
A: SD card full; not root (or SD card is full, or something broke); Mount point doesn't exist or (if 'mount' returns that and the point exists) something not cool happened.

Usage guide on Non-Android devices
Some interesting bits of information for use on computers:
There's probably not an easy way to read/write/mount EXT2 filesystems on a Windows machine, so if you have a solution that's stable, go for it.
Linux computers definitely should support this, and I'm not sure about OSX computers, but in theory would work if you had the right things installed. iPhones ... well I have no idea what you'd do with it there.
Linux:
On the device:
THIS PART IS EXTREMELY IMPORTANT, NOT DOING THIS WILL RESULT IN BROKEN FILESYSTEMS!
`umount /mnt/sdcard/development.img`
`losetup -d /dev/block/loop250`Plug your phone into your computer, making sure that the computer mounts it and reads it as removable media, and that the image is present.
On the computer: (Path of your user folder, and the mount point at which your device's SD is mounted can and very likely will vary from distribution to distribution. Check your paths.)
`mkdir /home/user/android-devtools/`
`mount -o loop /media/user/android-sd/development.img /home/user/android-devtools/`Now, you can change things within the image as you normally would a file or folder. Be warned, sometimes doing this can create conflicts in the file permissions, so running the right commands on the phone to change the permissions as root (use `chmod` for the permissions) can save you boatloads of trouble getting things to work, or editing files.
Mac OSX might or might not be the same, but seeing as I haven't picked up an Apple computer since maybe 5 years ago, let alone attempted filesystem management, I just don't know.

e2fsck command on cleanly formatted partition?

Last night I started to run e2fsck on my /data partition to do a bit of investigation to see if had any corruption (again)
Anyway I unmounted /data in TWRP and went to the terminal and issued the following command:
e2fsck -fvyD /dev/block/mmcblk0p8
12 hours later and it was still running mainly reporting "Clone multiply-claimed blocks? Yes" Yes is because I have y flag turned on to answer yes to all questions.
So I killed it as it has never run this long before (previously 3 hours max).
I then formatted data which I thought would make it unnecessary and ran the command again after unmounting data and it is doing the same thing.....
Why?
Code:
.
.
.
(
inode
#
869248
. mod time
Sat Aug 25 18:23:27 2012
)
Clone multiply-claimed blocks? Yes
It appears to be doing this for many (if not all) inodes
Over and over.
This makes it seem like my /data partition is very corrupted and also Aug 25 is when I bought the device in 2012 so it has gone right back to the start.
I suspect it is somehow related to turning data journaling off whilst testing hunds kernels.
What is the definitive command to ensure data journaling is on please?

sbdags said:
I then formatted data which I thought would make it unnecessary and ran the command again after unmounting data and it is doing the same thing.....
Click to expand...
Click to collapse
That's strange - after formatting a partition, e2fsck must run in seconds and without errors, and it should definitely not bring up anything related to the old data. How did you format it?

_that said:
That's strange - after formatting a partition, e2fsck must run in seconds and without errors, and it should definitely not bring up anything related to the old data. How did you format it?
Click to expand...
Click to collapse
I formatted it using the big format data button in TWRP but yes I agree it doesn't appear to have done what would call a format? Maybe the equivalent of a quick format? How do I properly format the mmcblk0p8 partition?
If I just issue e2fsck -n /dev/block/mmcblk0p8 it comes back with clean. It is only when I force a scan that it finds all these errors.
Just reformatted and run again and it is dong the same again. ??
Btw what is the command to turn data journaling on? Or once formatted does journaling reset itself?

sbdags said:
Just reformatted and run again and it is dong the same again. ??
Btw what is the command to turn data journaling on? Or once formatted does journaling reset itself?
Click to expand...
Click to collapse
Can you find the following line in your recovery log? This would be the command that TWRP used to format.
Code:
make_ext4fs command:
Normally, formatting ext4 with default options enables the journal.

_that said:
Can you find the following line in your recovery log? This would be the command that TWRP used to format.
Code:
make_ext4fs command:
Normally, formatting ext4 with default options enables the journal.
Click to expand...
Click to collapse
It reports the following:
Code:
make_ext4fs -l -16384 -a /data /dev/block/mmcblk0p8

sbdags said:
It reports the following:
Code:
make_ext4fs -l -16384 -a /data /dev/block/mmcblk0p8
Click to expand...
Click to collapse
Looks plausible. And look what I've found:
http://sourceforge.net/p/e2fsprogs/bugs/292/
You could try running the make_ext4fs command manually with the "-w" (wipe) option added - if I understand the code correctly, that should issue discard requests for all blocks, similar to fstrim.

sbdags said:
I formatted it using the big format data button in TWRP but yes I agree it doesn't appear to have done what would call a format? Maybe the equivalent of a quick format? How do I properly format the mmcblk0p8 partition?
If I just issue e2fsck -n /dev/block/mmcblk0p8 it comes back with clean. It is only when I force a scan that it finds all these errors.
Just reformatted and run again and it is dong the same again. ??
Btw what is the command to turn data journaling on? Or once formatted does journaling reset itself?
Click to expand...
Click to collapse
I mentioned this during the fiasco we had in the cromi thread awhile back, but I think it got lost in the rush of comments, let me know if you get it fixed, would love to be able to complete a check.

faustus1005 said:
I mentioned this during the fiasco we had in the cromi thread awhile back, but I think it got lost in the rush of comments, let me know if you get it fixed, would love to be able to complete a check.
Click to expand...
Click to collapse
Yeah something is not quite right with running fsck on the TF700. Oddly on my prime it runs without issue. I wonder if Asus has changed the way inodes are mapped. I know Samsung do this. I wonder if contributes to any lag as well?

_that said:
Looks plausible. And look what I've found:
http://sourceforge.net/p/e2fsprogs/bugs/292/
You could try running the make_ext4fs command manually with the "-w" (wipe) option added - if I understand the code correctly, that should issue discard requests for all blocks, similar to fstrim.
Click to expand...
Click to collapse
Let me do another backup and try the -w command as well. It's quite frustrating as I wonder if being able to run fsck correctly would help solve some of the lag issues we see.

_that said:
Looks plausible. And look what I've found:
http://sourceforge.net/p/e2fsprogs/bugs/292/
You could try running the make_ext4fs command manually with the "-w" (wipe) option added - if I understand the code correctly, that should issue discard requests for all blocks, similar to fstrim.
Click to expand...
Click to collapse
Adding the -w command fixed it (I think) e2fsk now completes in about a minute with no errors
Problem is now when I check if data journaling is on with a mount command there is no indication that it is - how to turn it off?
This is the command issued that turned it off I believe it's the "-O" that turns the journaling off:
Code:
run_program("/tmp/tune2fs.ext4", "-O", "^has_journal", "-c", "1", "-i", "1d", "-m", "0", "-o", "^journal_data_writeback", "/dev/block/mmcblk0p8");
is it as easy as issuing the same command without the "-O", "^has_journal" to get it back on?
OK I found some info looks like I need to issue this command in the installer?
Code:
run_program("/tmp/tune2fs.ext4", "-O", "+has_journal", "-c", "5", "-i", "5d", "-m", "0", "-o", "^journal_data_ordered", "/dev/block/mmcblk0p8");
I'll try that out now
Thanks for all the help @_that - I'm slowly getting there I think. :good:

sbdags said:
Last night I started to run e2fsck on my /data partition to do a bit of investigation to see if had any corruption (again)
Anyway I unmounted /data in TWRP and went to the terminal and issued the following command:
e2fsck -fvyD /dev/block/mmcblk0p8
12 hours later and it was still running mainly reporting "Clone multiply-claimed blocks? Yes" Yes is because I have y flag turned on to answer yes to all questions.
So I killed it as it has never run this long before (previously 3 hours max).
I then formatted data which I thought would make it unnecessary and ran the command again after unmounting data and it is doing the same thing.....
Why?
Code:
.
.
.
(
inode
#
869248
. mod time
Sat Aug 25 18:23:27 2012
)
Clone multiply-claimed blocks? Yes
It appears to be doing this for many (if not all) inodes
Over and over.
This makes it seem like my /data partition is very corrupted and also Aug 25 is when I bought the device in 2012 so it has gone right back to the start.
I suspect it is somehow related to turning data journaling off whilst testing hunds kernels.
What is the definitive command to ensure data journaling is on please?
Click to expand...
Click to collapse
You can try this.
tune2fs -l /dev/block/mmcblk0p8
You should see your system mounting has a "has_journal".
If it is not there, then your journal is off.
To turn it on.
tune2fs -O +has_journal /dev/block/mmcblk0p8
You should use the data_write for your journal to improve performance.
If you do a clean installation for your rom, the e2fsck should run less than a minute if you have a minimal apps installed. However, if you restore a nandroid backup for your system, it will take forever even though you don't have any app. I don't know why it does that with the nandroid backup..
I think that when you use e2fsck with a nandroid restore from your backup, the e2fsck is trying to convert your ext4 system to ext3 system.. It is just my guess. Maybe you and _that can figure why it does that.. Good luck...:fingers-crossed:

LetMeKnow said:
You can try this.
tune2fs -l /dev/block/mmcblk0p8
You should see your system mounting has a "has_journal".
If it is not there, then your journal is off.
To turn it on.
tune2fs -O +has_journal /dev/block/mmcblk0p8
You should use the data_write for your journal to improve performance.
If you do a clean installation for your rom, the e2fsck should run less than a minute if you have a minimal apps installed. However, if you restore a nandroid backup for your system, it will take forever even though you don't have any app. I don't know why it does that with the nandroid backup..
I think that when you use e2fsck with a nandroid restore from your backup, the e2fsck is trying to convert your ext4 system to ext3 system.. It is just my guess. Maybe you and _that can figure why it does that.. Good luck...:fingers-crossed:
Click to expand...
Click to collapse
Interesting, however, restored a nandroid and reran e2fsck and it finished in 30 secs with no errors Installed my ROM and ran it again and same again so it seems my problems are resolved for now
Thanks for the tips on data journaling.
BTW can you reup or send me the latest ET please - ready and confident to test it again now

sbdags said:
Interesting, however, restored a nandroid and reran e2fsck and it finished in 30 secs with no errors Installed my ROM and ran it again and same again so it seems my problems are resolved for now
Thanks for the tips on data journaling.
BTW can you reup or send me the latest ET please - ready and confident to test it again now
Click to expand...
Click to collapse
Haha, I found out that when I restore a good backup from nandroid and have more than 50% of problem with e2fsck.
I am still on a business trip. Let see if my thumb drive has the latest ET. Otherwise, I have it on my personal laptop at home. I will let you know in bit..

[KERNEL] [2 GB DATA] [ICS] [TWRP] [SD-EXT] Hefe Hook Kernel [2013/12/23]

First off, this is an ICS kernel that is intended for use with ICS ROMs. If you aren't using a ROM that is substantially similar to TeamAcid's CM9, then you may want to reconsider before you flash this.
This kernel is the same as the current TeamAcid CM9 kernel. Now that @FBis251 poked me about it, there is a version that also includes the "Hefe Kernel of Darkness" patch to turn off the touch-key backlight after it is touched.
If it's not all that different from the TeamAcid kernel, why would I want to flash this?
It's got TWRP
You can reliably mount /sd-ext or...
...move /data to your microSD, never worry about running out of app & data space, and speed your boot time
It allows you to get to the file system early in the boot process, to be able to change things that ordinarily you couldn't without creating a custom initramfs. This lets you, for example, create /sd-ext and mount something there, or get even wilder and do something completely untested like replacing /data entirely
TWRP can backup and restore /sd-ext
You hate the touch-key backlight
You want to be able to load the FUSE kernel module
How does it work?
When the SGS4G boots, it has a filesystem in the compressed boot/recovery image. It mounts that as the root file system, starts things running, and then all but that root filesystem sort of disappears as Android starts taking over. What happens and what is in that root file system is pretty much locked down unless you're going to create your own boot images. You can't edit a file on your running system and change that behavior. So....
The boot scripts in this build look for /system/etc/init-on-fs.sh right after /system gets mounted, but before much else happens. If it is executable, it will run the script. This means that you can make most of the changes to the environment or process you might reasonably want!
The "interesting" section of init.herring.rc is shown in the hidden text
Code:
on fs
exec /system/bin/logwrapper /system/xbin/echo "init.herring.rc: on fs enter"
# We chown/chmod /efs because mount is run as root + defaults
mkdir /efs 0775 radio radio
chown radio radio /efs
# Mount /system rw first to give the filesystem a chance to save a checkpoint
mkdir /radio 0775 radio radio
mount yaffs2 [email protected] /radio
chmod 0775 /radio
chmod 0775 /radio/modem.bin
chown radio radio /radio
chown radio radio /radio/modem.bin
mount yaffs2 [email protected] /efs
mount yaffs2 [email protected] /system
mount yaffs2 [email protected] /data
mount yaffs2 [email protected] /cache
exec /system/bin/logwrapper /system/etc/init-on-fs.sh
chmod 770 /efs/bluetooth
chmod 770 /efs/imei
mkdir /gps 771 gps system
chown gps system /gps
[...]
My init-on-fs.sh
Lists the mounts already in place (from the "stock" init commands)
Creates /sd-ext
Mounts the second partition of the microSD on /sd-ext
Lists the mounts, which should now include /sd-ext
and looks like this:
Code:
#!/system/bin/sh
/system/xbin/busybox mount
/system/xbin/busybox mkdir /sd-ext
/system/xbin/busybox mount -t ext4 /dev/block/mmcblk0p2 /sd-ext
/system/xbin/busybox mount
Since it's running under logwrapper the output goes to the logs and can be seen with logcat or the like.
Since this script runs before any "working" directories get set up and before any processes start running, if you're adventurous, you could even change the mounting of /data and just use microSD storage. I haven't gotten that adventurous yet, as TitaniumBackup seems to be very reliable in moving app data to the microSD partition.
Do I need to repartition my microSD and use /sd-ext?
Nope, you can just enjoy TWRP if you want.
How do I install it?
Download the current version from the next post. Push to your device and flash with your recovery.
init-on-fs.sh does not have an installer and is not distributed. Create it in /system/etc and make sure it's executable.
I have a question...
Check the FAQ right after the Downloads.
If you still have questions, especially if they are about how to use the Hefe Hook or how to configure your phone for 2 GB of /data, please use the Q&A thread.
.

Downloads
kernel-hefehook-kod-2013-12-23-111936.zip
Changes:
Properly incorporates "Hefe Kernel of Darkness" changes
Otherwise the same as kernel-hefehook-2013-12-18-084851.zip
2.6.35.7-Hefe-Hook-KoD-ICS-00184-g807d04b #1 PREEMPT Mon Dec 23 11:17:49 PST 2013
kernel-hefehook-kod-2013-12-19-171937.zip
Note: long-press of the touchpad doesn't work in this version. Please use a later version
Changes:
Incorporates "Hefe Kernel of Darkness" changes
Otherwise the same as kernel-hefehook-2013-12-18-084851.zip
As before, this is a hack to turn off the touchpad backlight as soon as it comes on.
2.6.35.7-Hefe-Hook-KoD-ICS-00176-gb430787 #1 PREEMPT Thu Dec 19 17:19:10 PST 2013
kernel-hefehook-2013-12-18-084851.zip
Changes:
TWRP fstab adjusted to be able to manage and backup mmc0p3 (potentially as /data)
fuse.ko available (~680 kB so not built into kernel)
2.6.35.7-Hefe-Hook-ICS-00171-g79fe663 #1 PREEMPT Wed Dec 18 08:47:03 PST 2013
Kernel code equivalent to 674765731f093fdb5f4a2416b03cf2c6c2c98e67 in TeamAcid's repository
with FUSE enabled as a module in the kernel build
Code:
diff --git a/arch/arm/configs/cyanogen_galaxys4gmtd_defconfig b/arch/arm/configs/cyanogen_galaxys4gmtd_defconfig
index 22303ee..5e44f0b 100644
--- a/arch/arm/configs/cyanogen_galaxys4gmtd_defconfig
+++ b/arch/arm/configs/cyanogen_galaxys4gmtd_defconfig
@@ -1,7 +1,7 @@
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.35.7
-# Sun Aug 12 20:38:04 2012
+# Wed Dec 18 07:54:02 2013
#
CONFIG_ARM=y
CONFIG_HAVE_PWM=y
@@ -38,7 +38,7 @@ CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
-CONFIG_LOCALVERSION="-cyanogenmod-ICS"
+CONFIG_LOCALVERSION="-Hefe-Hook-ICS"
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_LZMA=y
@@ -1900,7 +1900,8 @@ CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
-# CONFIG_FUSE_FS is not set
+CONFIG_FUSE_FS=m
+# CONFIG_CUSE is not set
CONFIG_GENERIC_ACL=y
#
kernel-hefehook-2013-12-12-231320.zip
Changes:
Initial public release
2.6.35.7-Hefe-Hook-ICS-00170-g6a2c60a [email protected]#1 Thu Dec 12 23:11:25 PST 2013
Kernel code equivalent to 674765731f093fdb5f4a2416b03cf2c6c2c98e67 in TeamAcid's repository
MD5 (kernel-hefehook-2013-12-12-231320.zip) = 0bb3950af5a09e1e0364a467c79fceb0
MD5 (kernel-hefehook-2013-12-18-084851.zip) = 38e76dc0d5dc5ae795b3d2ec57183d92
MD5 (kernel-hefehook-kod-2013-12-19-171937.zip) = 06f4ca9535d1baa88b8cc70984dbc661
MD5 (kernel-hefehook-kod-2013-12-23-111936.zip) = 9b638c335ecbcfdd998658b9d6346e63

FAQ
Why doesn't long-press-Home work? ("Darkness" version only)
Because I derped in the merge. I'm checking a version now and should have it up before Christmas.
Do I need to partition my microSD? Do I need to use /sd-ext?
Only if you want to use /sd-ext. You can enjoy TWRP without partitioning your microSD or using a boot-time script.
Will this work without init-on-fs.sh?
Yes
Will you give me instructions on how to replace /data completely with my microSD?
No, I haven't tried it yet myself
No, I need to make sure you can back it up and restore it under recovery
Yes, though first let me get rsync posted and some clear instructions worked out.
How do I format my microSD?
For my init-on-fs.sh script to work, you need your microSD formatted with the "old style" MBR partitioning scheme. My script looks for an ext4 partition as the second partition.
I do all my partitioning on Ubuntu or FreeBSD. You can boot Ubuntu on your Windows machine without overwriting anything ("Try Ubuntu"). I don't trust any Windows program to create a reliable Linux file system.
How do I ask about something else?
Please use the Q&A Thread.

Building
Build using the CM9 "ics" branch using the generally available instructions
Kernel source is on github. Hefe-Hook and Hefe-Hook-KoD are the two branches of interest. Both should be clearly tagged at each release.
galaxys4gmtd.xml:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remove-project name="CyanogenMod/android_vendor_cm" />
<project path="vendor/cm" name="teamacid/android_vendor_cm" revision="ics" />
<project path="packages/apps/SamsungServiceMode" name="CyanogenMod/android_packages_apps_SamsungServiceMode" revision="ics" />
<project path="packages/apps/CMFileManager" name="teamacid/android_packages_apps_CMFileManager" revision="ics" />
<project path="device/samsung/galaxys4gmtd" name="teamacid/android_device_samsung_galaxys4gmtd" revision="ics" />
<project path="vendor/samsung" name="teamacid/proprietary_vendor_samsung" revision="ics" />
<project path="kernel/samsung/galaxys4gmtd" name="jeffsf/android_kernel_galaxys4gmtd" revision="Hefe-Hook-KoD" />
</manifest>
<!--
teamacid's CM9 local_manifest.xml
=================================
vi: ts=4:sw=4:et:noai
-->
twrp.xml:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remove-project name="CyanogenMod/android_bootable_recovery" />
<project path="bootable/recovery" name="TeamWin/Team-Win-Recovery-Project" revision="twrp2.5" />
</manifest>
<!--
teamacid's TWRP local_manifest.xml
======================================
vi: ts=4:sw=4:et:noai
-->
TWRP is from the "twrp2.5" branch with the following commits reverted in git
35a35a67663127e42c8048ca0dc7465068a5606f
516e4e23721e4e5aa0a349ab8f27c22a81665880
Note that the "twrp2.5" branch is, at this time, pointing to the same commit as the "ics" branch.
The "twrp2.7" branch is heavily laden with selinux code for later Android releases and appears to have left ICS in the dust.
Changes to to the initramfs are self-documenting and visible in the distributed code.

Ooooo.... look! FIRST!

"My /data's bigger than your /data. Nah nah ne nah nah..."
Code:
localhost / # df -h /data
df -h /data
Filesystem Size Used Available Use% Mounted on
/dev/block/mmcblk0p3 5.6G 2.0G 3.4G 37% /data
localhost / # du -h /mnt/secure/asec
du -h /mnt/secure/asec
16.0K /mnt/secure/asec
It also boots in a minute, with all apps present and ready. No more three or four minutes until everything was read, mounted, and all off SD for all the "external storage" apps.
I'll probably remount the MTD partition as /data/dalvik-cache since my earlier check suggested that the internal flash was faster than the external ext4 for tiny, random reads.

Would you add FUSE (Filesystem in Userspace Linux kernel module) please to the kernel? I would like to try Paragon NTFS driver that said it even works in boot and recovery.
I tried the app, but it complained about the kernel not supporting fuse. I am using hefe kernel on AOKP-ICS.

Let me look at the config. It shouldn't be a problem. Nice to know that people are still interested in a solid ICS experience.
Posted from my SGS4G, thanks to Team Acid development

I really appreciate that you are continually working on development for this device, especially in this area
Sent from my SGH-T959V using Tapatalk 2

Rebel_X said:
Would you add FUSE ...
Click to expand...
Click to collapse
Today's build has the fuse.ko module built and available. I didn't build it into the kernel as it looked to be pretty large (~680 kB on disk) and I'm not sure something that many users would need. If it doesn't automatically load when you start your app, you may need to manually load it (or have it done in a start-up script).
Code:
localhost / # lsmod
lsmod
Module Size Used by Not tainted
bcm4329 213497 0
localhost / # insmod /system/lib/modules/fuse.ko
insmod /system/lib/modules/fuse.ko
localhost / # lsmod
lsmod
Module Size Used by Not tainted
fuse 58977 0
bcm4329 213497 0
Let me know if you have any issues or need help in getting it to load automatically for you.
I'm also curious about the use case. I just haven't run across a need or good use for FUSE on my phone. I have used it for ssh-based file systems in the past and it was pretty slick for that.

As this application of the Hefe Hook kernel is likely to generate more questions that "just an ICS kernel" would have, please use the Q&A Thread.
So, you want to have a nice, big, fast data partition so you don't have to worry about running out of space, or wait for all your apps to be mounted, one-by-one, off your SD. You want to be able to try a new ROM without worrying about your data at all as well, eh?
Oh, you're not on Team Acid CM9 or a similar ICS ROM? You can read along, but I don't know of any other SGS4G ROMs with a boot image / kernel that lets you do this the same way.
OK, all that said, breathe deeply and consider that you might lose all your data if things go awry. Breathe easier in that there are some great backup tools out there!
For this to work well, you'll need a good microSD. One that is at least "Class 10" and from a reputable manufacturer is highly recommended. I've had good luck with SanDisk and Transcend. I'm currently using the SanDisk Ultra 64 GB microSDXC Class 10 UHS-1 Memory Card 30MB/s with Adapter SDSDQU-064G-AFFP-A
Today's prices are not as bad as they were a year ago:
16 GB - $13
32 GB - $22
64 GB - $47
These 64 GB cards work just fine in my experience, being able to address all 64 GB in our phones.
These instructions assume that you have a blank card in your hands and aren't going to mess with the one you are presently using in your phone. You'll have to juggle a bit if you are going to "reuse" your current card.
Format your new card
First, we need to prepare that new card. I suggest four partitions, though you can get away with just the first three.
1) "sdcard" -- FAT
2) "sd-ext" -- ext4
3) "data" -- ext4
4) (spare) -- ext4
Here is how I think about partition sizes:
3) "data" -- 2 GB seems reasonable. I'm using about 1 GB and it would probably double if I installed the apps I've pulled off over the time I've had the phone since I didn't have enough space
4) (spare) -- 6 GB -- Why? I can do two full-image, uncompressed backups of 2 GB of data and 1 GB of internal storage. I can use rsync to very, very quickly do a backup as it only had to copy the added/changed files, not the whole thing. (More later on this.)
2) "sd-ext" -- 1 GB -- What do I do with this? All my apps and data are already on the microSD. I need something here so that scripts that grab the second partition don't grab our data partition. It could be smaller, but I figure I've got 1 GB in my pocket in case I fill everything else up.
1) "sdcard" -- 64 GB - 9 GB = 54 GB left. If you have a 16 GB card, it would be 7 GB, pretty tight, and I'd probably sneak some off of sd-ext and the fourth parition. Well, no, I'd spend a couple more dollars and at least buy a 32 GB card.
I do all of my partitioning on machine running Ubuntu. If you run Windows, you can boot into Ubuntu and run it without installing it.
Apologies here that there aren't a lot of details here yet on how to run the disk tool on Ubuntu. I wanted to get this up sooner than later.
Format the drive with a Master Boot Record (MBR) partition table
Add partitions (my 64 GB sizes shown):
1) FAT, 54 GB, "SD64-2-sdcard"
2) ext4, 1 GB, "SD64-2-sd-ext"
3) ext4, 2 GB, "SD64-2-data"
4) ext4, 6 GB (the remainder), "SD64-2-p4"
I name the partitions so that they are easier to manage on desktop machines. Unfortunately, I don't see the kernel picking up the names on the phone. My naming convention is the "SD64-2" identifies a chip -- a SanDisk 64 GB chip that I've scrawled the number "2" on with a marker. I've got a bunch of them and can't remember which is which without marking them.
Now, put that "new" chip somewhere safe, not in the phone!
You've still got your "old" microSD in the phone.
Power it off, remove your SIM card.
Boot it up and plug it in to your computer, if it isn't already. You don't want to run out of battery half-way through something important. Some things are easier through adb than on the phone. You do have adb up and running on your computer, right?
Get Backups
1) Turn off WiFi (yes, really -- you don't want anything changing your data after you've backed it up)
2) Backup with TitaniumBackup
2) Reboot to recovery
3) Run a full backup in recovery
4) Remove your day-to-day microSD
Get your information over to the "new" microSD
There are a lot of ways to do this, the easiest is probably to get both the old and the new chip mounted on a Linux machine and use rsync, along the lines of
Code:
rsync -avihP /media/jeff/SD64-1-sdcard/ /media/jeff/SD64-2-sdcard/
rsync -avihP /media/jeff/SD64-2-sd-ext/ /media/jeff/SD64-2-sd-ext/
Be careful with the trailing slashes! Especially if they are missing on the source, you'll get a very different result. The "-a" option is the important one, "-vihP" are all about getting some verbose output that supposedly a human can read as it is copying everything.
Grab a beer, cup of coffee, whatever. There is no magic if you are copying over a bunch of GB from one chip to another, especially over USB.
You now have a copy of your old card on your new card, with the new card having the right partitions.
You can run it again to convince yourself that the two have the same files. I do it myself. If it doesn't copy anything, it should be all good. It is a lot faster when you don't have anything to copy!
Put away your old card somewhere safe. Really. Take it out of your computer.
Get utilities on microSD
Before you take the new chip install a couple things. I use the 4th partition for this as "scratch" space. You want a copy of rsync for your phone, Hefe Hook kernel (2103-12-18 or later), and a version of init-on-fs.sh that mounts your microSD for /data (instead of the internal flash). Do not push init-on-fs.sh to the phone yet!
Get your old /data onto the new card
5) Put the new microSD in your phone. Leave the SIM out!
6) Boot to recovery -- I do this by having the phone connected and hitting "return" on
Code:
adb reboot recovery
as soon as I see the Team Acid logo ger replaced my the CyanogenMod boot animation. If it gets to be running, don't panic! No SIM, no WiFi remember? You won't get calls, SMS, push updates, new mail, nothin' Just reboot into recovery and continue.
7) Fire up adb shell, mount the partitions, and get rsync available
Code:
mount -a
cp /mmc0p4/rsync /system/xbin/
chmod 755 /system/xbin/rsync
8) Copy over your /data to the new partition!
Code:
rsync -avihP /data/ /mmc0p3/
Same warnings on trailing slashes, as well as making yourself confident that everything was copied over by running the rsync command twice.
9) Install init-on-fs.sh that will mount your microSD as /data
Code:
cp /mmc0p4/init-on-fs.sh /system/etc/
chmod 755 /system/etc/init-on-fs.sh
My /system/etc/init-on-fs.sh is chatty, but works for me:
Code:
#!/system/bin/sh
/system/xbin/busybox mount
/system/xbin/echo
/system/xbin/busybox mkdir /sd-ext
/system/xbin/busybox mount -t ext4 -o noatime /dev/block/mmcblk0p2 /sd-ext
/system/xbin/echo
/system/xbin/busybox mount
/system/xbin/echo
/system/xbin/busybox umount /data
/system/xbin/busybox mount -t ext4 -o noatime /dev/block/mmcblk0p3 /data
/system/xbin/echo
/system/xbin/busybox mount
10) Reboot to Android
Woo hoo!! Check your system space, you should have a bunch of free space on internal now!
Get everything "internal"
You want your stuff off "android secure" since the way that it works is that Android creates a file that is a filesystem for each and every app you've moved to SD. At boot time, it reads the file, mounts it, checks the filesystem, then "hooks up" the app inside. It can take a couple minutes for this to complete! Putting them on "internal" storage does away with that. They are "just there" like your internal apps used to be. That means you get all the widgets and everything too.
(No, I don't know how to turn off notifications when you move the XDA app to "internal" -- the settings don't seem to do anything.)
Don't rush this. You want to be able to get back to /data on the internal flash later if you need to. We haven't erased /data -- it is a handy "backup" there right now.
This assumes you have TitaniumBackup, perhaps Pro is required. If you don't own it already, I think it is a solid product. If you use something else to manage your apps and backups, take this as an outline.
This looks like a lot to do -- but if you ever have to restore to your internal storage, you'll know what you already had on SD so you can restore it there!
11) Mark your on-SD apps -- this is a pretty-well hidden feature, so stick with me
Launch TiB
One it gets going and "all application information has been loaded"
Go to the Backup/Restore tab
Click on "Click to edit filters"
Clear any existing filter and set
Filter by type: User
Filter by location: External
Click the green checkmark in the top right
It should return to the list of apps, all your on-SD apps should be shown
Click on "Click to edit filters" again
Click on the "Create label..." button
Enter a name, "Apps on SD, December 2013" or whatever makes sense to you
Click on "Add/Remove elements..." button
Click on "Select shown elm." button
You should now see only the "filtered" elements
Click the green checkmark in the top right
Click the "Save" button on label dialog
Click the green checkmark in the top right
You should see your on-SD apps tagged. Phew!!
12) Move all your on-SD apps and data to internal storage
Click the grey checkmark in the top right
You should see the Batch actions screen
Run "Move user apps to internal memory"
Run "Move app data to internal memory" (if you have it)
13) Kick back -- you've done it
I'd reboot to recovery at this point and backup your new data partition. The partions you will want to backup now are labeled
boot (mtd)
recovery (mtd)
system (mtd)
efs (mtd) -- once in a while is fine
cache (mtd) -- can skip, it's cache after all
Android Secure -- should be 0 MB, if so, can skip
sd-ext (p2) -- backup if you have important stuff here
data (p3) -- <== this is all your stuff now!
You can skip "userdata (mtd)" now as it is your "old" internal data partition.
You should also see that TWRP will let you put your backup on your choice of partitions on your microSD:
sdcard (p1)
sd-ext (p2)
mmc0p4

WhyTF is this guy not RC/RD already!?!?

Hey I flashed this on an sgs4g and your darkness patch didn't seem to work, this is on the Dec 18 build.

bhundven said:
WhyTF is this guy not RC/RD already!?!?
Click to expand...
Click to collapse
will put in a nomination when I get home if it's ok with Jeff for RC.
sent from within the white knight

FBis251 said:
Hey I flashed this on an sgs4g and your darkness patch didn't seem to work, this is on the Dec 18 build.
Click to expand...
Click to collapse
I'll take that as a polite request to dust off the cobwebs, patch the current kernel sources, and provide a "Darkness" version with the touch-key backlight turn-off hack
@FBis251 -- it's been checked and is up now -- I had forgotten how nice it is not to have the lights on all the time!
Amusingly, my look at the CM10 and CM11 sources look like they do what my kernel hack does, but in the Android layer.

jeffsf said:
I'll take that as a polite request to dust off the cobwebs, patch the current kernel sources, and provide a "Darkness" version with the touch-key backlight turn-off hack
@FBis251 -- it's been checked and is up now -- I had forgotten how nice it is not to have the lights on all the time!
Amusingly, my look at the CM10 and CM11 sources look like they do what my kernel hack does, but in the Android layer.
Click to expand...
Click to collapse
Ah I misread the OP, I thought I had read "This includes the kernel of darkness patch" but you said that this wasn't derived from the kernel of darkness .
Thanks for the update, flashing it now.
EDIT
Everything's working as expected, thanks again!

moved to Q&A thread (sorry, I need to get glasses).

I'm going to respond to you in the Q&A thread, if you don't mind.
I'll also make links to it a little more evident once I get back with you.

Bumping this thread to verify that this procedure works well with Team Acid AOKP (milestone 6 build 5). Feels smoother than CM9 too.
EDIT
Since it's a dev thread, I'm gonna post a tip that @bhundven gave me. He said to try using noop as the scheduler for the SD card since it seems to get better performance than deadline or cfq.
You can do this by adding an init.d script under /system/etc/ (or just edit one of the ones that are already there) and add the following:
Code:
echo "noop" > /sys/block/mmcblk0/queue/scheduler

FBis251 said:
Bumping this thread to verify that this procedure works well with Team Acid AOKP (milestone 6 build 5). Feels smoother than CM9 too.
EDIT
Since it's a dev thread, I'm gonna post a tip that @bhundven gave me. He said to try using noop as the scheduler for the SD card since it seems to get better performance than deadline or cfq.
You can do this by adding an init.d script under /system/etc/ (or just edit one of the ones that are already there) and add the following:
Code:
echo "noop" > /sys/block/mmcblk0/queue/scheduler
Click to expand...
Click to collapse
FB...you're the only one who could get away with a bump. haha So glad to see you are still around, even if its just lurking around back in the shadows. you're one of 4 who helped me get to where I am with Android from day 1. I love this phone! Thanks. (I know this is off topica sn a dev thread I apologize in advance)

Database Info

welcome