Return to Contents Page - doubleshot Developers Reference
**For some reason the board merged all of my posts...sorry guys, just gonna have to deal I suppose.
Since sharing my notes here about what i've learned/am learning for Android development seems to be helping people beyond myself, i'd like to share my notes regarding Edify scripting so far.
WARNING: You can really screw up your phone once you start making your own flashable zips. While this guide gathers scripting information from all over XDA and distills it into MT4GS implementation, it is not a difinitive resource.
There are blatant gaps of knowledge in this thread. I am trying to include as many of the helpful resources I can that were found here at XDA. Links are at the bottom of this post - if you found a good edify-script reference here at XDA, please reply with a link to this thread for people researching this topic. Thanks!
I highly recommend you have a good understanding of what you are doing before you go ahead and do it. Follow the links, drift through the XDA knowledge base and soak it all in.
Feel Free to ask any questions or add anything related to Edify scripting to this thread.
Here's how this thread is laid out:
Post 01 - Contents
Post 02 - Edify Scripting Explanations.
Post 03 - Before we begin...
Post 04 - Mounting your way out of RAM.
Post 05 - Delete, Demolish and Destroy.
Post 06 - Break more stuff: Format <-- Flashable zip 'format' utility here.
Post 07 - Reserved. ( Installing )
Post 08 - Reserved. ( Permissions )
XDA Links:
Tools:
nubecoder's amend to edify script converter
Utility i've been signing my zips with by benjamminzIS
Posts/Threads:
Single post by nubecoder, syntax
Tutorial by coolexe on Amend/Edify script syntax
Thread by DooMLoRD, pieces of the mounting puzzle
...and more when I get time to track them down and put them in.
One thing to note about this thread is that it's not just a quick synopsis of syntax. There are plenty of threads like that around XDA, and they are all helpful. The MT4GS has some specifics to keep in mind, and there are some things to work around in CWM, but we can start to tackle some of this with this guide.
This thread is more of an explanative review of the process, an attempt to point out some common, or not so common mistakes that are easily made...and a bit more of a look at why things are happening beyond just how to do them.
More of a theory behind the application of principles and so not a quick read.
I would like to say thank you to computerkid23 for putting an end to trying out every update-binary I could find looking for the right one. By allowing me to re-use the one from the "Tasteless Venom" ROM, I stopped treading water and started making forward progres again.
This is the update-binary file used in all of my scripts, and has the proper syntax for the commands to work with the MT4GS.
---------- Post added at 04:54 PM ---------- Previous post was at 04:51 PM ----------
Edify Scripting Explanations
ClockworkMod Recovery 3.0+ and update-script
Read the post from the above link to understand that Amend scripting is the old way and Edify scripting is the new way.
While the commands are essentially the same, the syntax and expression is much different in Edify then Amend.
If you are trying to flash a zip, and want to figure out if it's an Amend or Edify:
Both Edify and Amend flashable zips have the following directory:
/META-INF/com/google/android/
In the android folder:
Amend = update-script
----------------------
Edify = update-binary
updater-script
Click to expand...
Click to collapse
This is the easiest way for you to tell what kind of script you have - in simplest form and first point of recognition:
1 File = Amend
2 Files = Could be Edify
If two files exist in this directory, check to make sure they match the quote above for Edify. Make sure it's not a mix of an Amend update-script and Edify update-binary.
If everything matches up to Edify so far, then browse through the thread and look at the command syntax laid out for Edify. If what you're looking at doesn't match, then you'll have to seriously downgrade your CWM version to run it - meaning you will have to write your own recovery from scratch.
With that understanding, for the remainder of this thread we will only be focusing on Edify scripting.
update-binary
This is a not so trivial piece of your scripting puzzle. There are an unknown number of different update-binary files out there. They support different script syntax, make sure your updater-script and update-binary match.
---------- Post added at 04:54 PM ---------- Previous post was at 04:54 PM ----------
Before we begin...
...there are just two things we have to go over:
1 - Backups.
2 - The ui_print("") command.
First, read my Thread about backups. Specifically post 2 about making a Clockworkmod backup.
You should have multiple copies of each backup you keep, on different machines or media. If your only backup is on the sdcard of your phone, and you accidentally erase it, that's it - it's gone.
Make sure your CWM backup works before relying on it. The only thing worse then losing a backup is keeping a bad one you think is good.
Your can't just pull individual images out of it and restore it through CWM, you need the whole backup file that was generated by CWM. You can restore only specific partitions from the complete backup file if you want.
Basically, your backup is your security. Once you start writing scripts you run into the problem of being able to affect changes on things you weren't intending to and you just can't fix - you need that backup.
Second, the ui_print("") command.
Code:
Syntax:
ui_print("some message goes here");
ui_print("another one here");
ui_print("the next line is a blank line");
ui_print("");
This command simply prints whatever text is in between the quotes to the screen when the line is executed.
This includes white-space, so any spaces you make in the quotes are preserved, even if it's the first thing after the quoatation mark.
Okay - so here's one of those MT4GS tips that you find through trial and error:
- The MT4GS screen in recovery, after executing a file, fills up to a certain point with printed text.
1 - Verticle:
To get your title, introduction line, whatever, as the top line erasing all code that came before it, you must print 23 lines of text to the screen. The first line must be blank.
Example:
ui_print("");
ui_print("MT4GS - Blue6IX A01 Wipe - Stock Mod");
ui_print("");
Click to expand...
Click to collapse
That's how your first 3 ui_print("") commands should be entered into your script to make your title be the top line of code on the screen when the script is finished in CWM.
2 - Horizontal:
To keep your text from being split at an awkward place, remember that you get about 40 characters on a line before it starts printing to the next line on the screen. If this happens, that second line counts as one of your 23.
I attached some screenshots of some of my scripts at the bottom of this post. They show what clockworkmod looks like as soon as the script is finished, before the user changes anything.
With the parameters of 23 lines down (first line blank) and no more then 40 characters to any individual line, you can format the final output of your script to be much more useful to the user.
If a person sees all the commands they entered leading up to executing your script, but only a few of them and some are already pushed off screen, it can be confusing.
This is a part of the development approach that is appealing to me, the end user experience and what I can do to enhance it. By making the final output of your script consist of only what you want to be shown on the screen, you have an opportunity to communicate to the user.
Make sure to check all your spelling - honestly, when I see typos in the printed output of an installation script, on any software on any platform, I start to worry about what's in the rest of the code. By presenting yourself well here, you can establish a level of trust with the end user in your product. ( Obvious slang and intentional misspellings notwithstanding )
Something else to consider is that if you don't use the progress commands to show that the script is doing something, a long operation might confuse a user into thinking their phone froze. They might then do something really bad in the worried and irrational state of mind that could put them in...
"oh...no... I think it froze somewhere between writing the radio and imaging the boot partition...
Click to expand...
Click to collapse
I put that up there as an extreme example, writing radios is not part of this thread ( initially... ) but imagine being that user who doesn't really know what's happening to their device, doesn't have the technical background to know what to do in a situation like that, and they might irrationally pull the battery while it really is writing the radio in a panic.
When contemplating the effects of what actions you do and do not take, always consider the worst possible thing you can think of happening as being the most likely to occur.
Simply using a progress bar, or as I prefer, the ui_print(""); command to keep something happening on the screen keeps the user from guessing. If enough space is left over, you can even tell them what they have to do next.
Or you could just chalk it up to me being too particular about the details...
Anyways, check out these screenshots to see how you can use the space available to you to direct the user and possibly eliminate some confusion:
---------- Post added at 04:55 PM ---------- Previous post was at 04:54 PM ----------
Mounting your way out of RAM
-Mount Points
-"mount" command
-"unmount" command
The most feature-rich script for reprogramming your Android device just the way you want loses all value if all you are doing is writing to RAM.
If you do everything else right, but don't mount your partitions correctly, then your script will either error out or just zip through the install in no time flat.
First piece of information you need to know-
doubleshot mount points:
Code:
/dev/block/mmcblk0p22 /system ext4
/dev/block/mmcblk0p23 /data ext4
/dev/block/mmcblk0p24 /cache ext4
These are the mount points for system, data, and cache.
Thanks to nubecoder for the how to on finding them out Here:
Not a direct quote.
From ABD:
adb shell "mount > /sdcard/PHONENAME_mountinfo.txt"
From Terminal:
mount > /sdcard/PHONENAME_mountinfo.txt
Click to expand...
Click to collapse
You do not need root to generate this info from Terminal Emulator.
Part 1 - Mount point usage:
Code:
mount("ext4", "EMMC", "/dev/block/mmcblk0p22", "/system");
** I have had mixed results using update-binary files that called for 3 parameters to mount. I may not know enough to explain why, but my theory is that:
-"EMMC" parameter allows me to force the action to take place not in RAM, can someone verify this?
Ever since using this method of mounting I haven't suffered the problem of the script running itself out in RAM and not doing anything.
By executing the above line of code, you will mount the system partition of the internal phone memory. Partitions must not be mounted if the format command is executed. For all other actions, mounting is the first step.
Part 2 - unmount when finished.
Code:
unmount("/system");
unmounts the partition and makes it safe to format.
If you are done with it for the time being in your script, you can unmount and remount it later when you need it.
All mounted partitions should be unmounted before the script is done.
Part 3 - Example
Dummy code in green, mount & unmount in bold
Code:
[COLOR="Green"]Lines(of code);
in_your(script);[/COLOR]
[B]mount("ext4", "EMMC", "/dev/block/mmcblk0p22", "/system");[/B]
[COLOR="Green"]More(code);
the(stuff you do);
to_the(system partition);[/COLOR]
[B]unmount("/system");[/B]
[COLOR="Green"]possibly_even(more code);
to(do more things);
after_you(are done with);
the(system partition);[/COLOR]
So here we learned how to mount, unmount, and some reasons why we need to do or not do this.
I am a big fan of unmounting a partition when I am done using it, and you'll see examples of this throughout my scripts as they surface here at XDA. I know some people will disagree with me about this, but here's my reasoning:
-The only way to change something on a partition is to mount it first, so if a partition is not mounted nothing is happening to it.
*Format is the exception. A partition must not be mounted if you want to format it. See the post on format below to find out more.
By not mounting a partition if I don't need to, or unmounting a partition when you aren't using it for a time, you can prevent accidents and put another layer of protection on the execution of your code.
As much as you should mold to the way the machine works, it is here to serve you and that's gotta give both ways. At some point the code has to be decipherable to a human, and there's no harm in spreading things out a little bit to also make sure everything happens in it's proper way at the right time.
---------- Post added at 04:55 PM ---------- Previous post was at 04:55 PM ----------
Delete, Demolish and Destroy
-delete
-delete_recursive
Warning: Delete means exactly that - gone. There is no un-delete, there is only a backup. If you don't have a backup, you just lost whatever you deleted.
Please read my guide on backups before playing with this.
Delete Files:
Code:
Single Item:
delete("/path/to/file.extension");
Multiple Items:
delete("/path/to/file01.extension","/path/to/file02.extension","/path/to/file03.extension");
Pretty simple with the delete commands. In the Stock App Reference i've listed the direct edify delete commands for every stock app, so I won't get back into it like that here.
We will pull an example, though, the annoyingly broken stock calculator:
Calculator
Version: 2.1.2115332058.70840
201.25KB - /system/app/Calculator.apk
116.23KB - /system/app/Calculator.odex
/data/data/com.android.calculator2
Click to expand...
Click to collapse
Okay, so what we see here is this app has an .apk file at the location /system/app/Calculator.apk
In order to delete that, we need to put this line in our script:
Code:
delete("/system/app/Calculator.apk");
But wait! There's more. HTC includes a .odex file with most of the stock software on this device. The weirdly broken, poorly coded calculator is no exception, and we can delete that one too:
Code:
delete("/system/app/Calculator.odex");
Of course, that would turn into a huge script, if you had to type in the whole command every time you wanted to delete a file. To that end, we can group multiple targets to one action like this:
Code:
delete("/system/app/Calculator.apk","/system/app/Calculator.odex");
Which matches what I quoted from the guide.
Again, be sensible about this. You could group a large amount of apps into a single delete command, but then you're going to have to browse one line of code that goes on forever when you check it or have to change/fix it.
I'd recommend grouping a handful together, then just running another delete command with another group of apps if you have to do a whole lot of them. It makes it easier for you to figure out what's going on, and troubleshoot the script if necessary.
What's the point of fitting 100+ apps on one line of code, if it takes you 10 times as long to check it? Being too efficient to be productive, break it up a little bit.
delete is one of the most powerful commands at your disposal. It takes a lot of work to make something, but one stray delete command to wreck it all. Use with caution.
Delete Directories:
Code:
Single Item:
delete_recursive("/path/to/directory");
Multiple Items:
delete_recursive("/path/to/directory01","/path/to/directory02","/path/to/directory03");
If you thought delete was a powerful command, then delete_recursive just upped the bar.
Instead of being restricted to just single files, you can now nuke a whole folder. It also deletes everything in the folder, in case that didn't click yet.
Be even more careful in using this command, because doing somethine like delete_recursive("/system/customize/resource") to get rid of all the stock wallpapers and browser bookmark pictures you don't want sounds like a great idea...until you realize that folder also has your boot animations, default ringtones, and other things that you probably want to have.
( But that's okay, you made a backup first, right? )
To continue our calculator example from above, we might want to also delete the data file from the app. If you're going to get rid of something, you should try to get rid of all of it's pieces to prevent problems on the system.
Code:
delete_recursive("/data/data/com.android.calculator2");
That line of code deletes the data directory for the stock calculator, so now we can go ahead and replace it with something working right.
If we were going to be replacing just the calculator, or only one or two things here and there, then this method is acceptable.
If you can manage resetting all the permissions correctly, you can go ahead and delete a directory you have a copy of, and then just install it back there later in the script minus the stuff you didn't want. This is a hazardous road to take, but if you're deleting 37 out of 43 files in a folder, and you have to call them all by name...
( I have had mixed results with this, nothing that makes me eager to take this road if I can avoid it )
I'd like to get more into delete_recursive, but it's actually better addressed in the next post, so skip ahead to format.
---------- Post added at 04:56 PM ---------- Previous post was at 04:55 PM ----------
Format:
Well, here we are. The big fish of the "break stuff" commands. Often times, though, it's like using an axe when you need a hobby knife.
Unfortunately, the format commands seem to be broken for us at best, not usable at worst. If someone can put more time into it then i've been able to to figure out why or if there is a way to get them to work, then please don't hesitate to post here.
Meantime, while I was browsing XDA seeking answers, I stumbled across This Post and just started laughing and walked away from the computer for a little while.
And while that wasn't the answer I was looking for, I made the decision to drop it for now and just move on with the fix.
Code:
Here's the workaround:
delete_recursive("/cache")
Simply delete the directory.
Thanks Firon!
Now, here's where we get back to the part about mounting and unmounting the same partition multiple times in a single script. Follow along with this progression:
This block of code clears the cache:
Code:
mount("ext4", "EMMC", "/dev/block/mmcblk0p24", "/cache");
delete_recursive("/cache");
unmount("/cache");
...and this block of code clears all user data:
Code:
mount("ext4", "EMMC", "/dev/block/mmcblk0p23", "/data");
delete_recursive("/data");
unmount("/data");
And they are both self-contained. As long as neither is mounted before entering these 3 lines of code each, what you get here is Mount, Wipe, Unmount - done.
The reason why I picked /cache and /data to use as an example, is because when you put them both together like this:
Code:
mount("ext4", "EMMC", "/dev/block/mmcblk0p24", "/cache");
delete_recursive("/cache");
unmount("/cache");
mount("ext4", "EMMC", "/dev/block/mmcblk0p23", "/data");
delete_recursive("/data");
unmount("/data");
...you've just essentially called for a factory reset in your script. The next time the operating system boots, it will rebuild the cache and data partitions from whatever is installed on the device.
Remember, though, rebuilding the data partition will make it take noticeably longer to boot, and you should let it sit for a few minutes after it finishes booting.
The reason why people get choppy installs and errors from custom ROMs sometimes, is because as soon as the screen gets past the boot screens, they just start playing with it right away. What they don't realize is that the data partition is pretty complex, and a lot of that stuff is still being created, sorted and saved behind the scenes.
When you start making the device do stuff, before it's done what it's already doing, it tries to serve you first. It will clear parts of the data partition that are still being built from memory to do what you just told it to do.
Because of that, as you use it over time you run into the missing or corrupted files/directories/permissions/etc... because what was in memory was lost before it got saved on first boot.
Letting it sit for 5 minutes after first boot from a factory reset or any simulation thereof is vital to a smooth and proper install. Developers, the average user will not know this and getting the word out with your install will save you troubleshooting phantom problems users report to you that can't be duplicated by anyone else.
Now that we've gone through all that, here are two flashable zip files already signed and ready to use:
1 - Blue6IX_A01_Wipe.zip
- This will simply delete recursively /cache /data and /system.
Users: Flash this zip before installing a custom ROM or a CWM backup to eliminate 'bleed through' from the previous ROM/Install.
WARNING: If you do not have a /system partition waiting to install after flashing this zip, you will have a phone without Android on it. Check the ROM or CWM backup BEFORE flashing this zip to make sure you will have software on your phone when you are done. If this message scares you, then use the next listed flashable zip instead.
Developers: The three blocks of code that perform the wipes of these partitions should be the beginning of your installation script, if you are providing all the resources you need for your installation INCLUDING a pre-populated /system partition.
If you are not providing the /system partition in it's entirety, then see the next script:
2 - Blue6IX_B01_Wipe.zip
- This deletes recursively only /cache and /data.
Users: Flash this zip to simulate a factory reset. Your data partition will be rebuilt on the next boot by what's installed on the device. This does not delete any system programs, but any apps installed through any market or user data of any kind ( contacts, notes and etcetera... ) will be lost.
This flashable zip is just like doing a factory reset from the settings menu, except it's from recovery.
Developers: If you are just modifying the stock /system partition, then the code in this script that isn't ui_print("") should be at the beginning of your install script.
If part of your install instructions include "do a factory reset" or "format /data and /cache" then just insert this code as the first thing your installer does and you can take that hassle out of their hands.
Since we have people of all levels of experience and ability flashing ROMs, we should try to take as many extra steps out of their hands as possible - it's just easier for everyone around.
This will ensure that no random data gets left over from the old environment before you changed it. It ensures that your information gets written correctly - as long as you play nice with the pre-existing /system partition.
So notes on these scripts:
1 - A01 CAN LEAVE YOU WITH NO ANDROID SOFTWARE ON YOUR PHONE IF YOU HAVE NO /system PARTITION TO INSTALL BEHIND IT!!
* - On the user level, I run flashable.zip before I restore a clockworkmod backup. This cleans out those 3 partitions and lets the backup write to fresh memory space.
* - On an advanced user level, I usually just install the system.img file from the CWM backup and let it build the cache and data on the next boot. Seems cleaner to me that way.
- I wrote all the text in that prints to the screen to give it a cleaner, less confusing look. When the script is done, the interface is much less busy from random notices and not as intimidating to casual users.
+ - Neither of these flashable zips has anything to do with boot or recovery images or partitions. It's all cache, data and/or system.
( I don't know what would happen if you powered off without a /system partition. I'd think that it would be able to get to the recovery utility ( Clockworkmod ) where you could flash a /system partition, CWM backup or ROM. )
(( I'm not brave enough to find out on my phone, because while i'm 90% sure it would be fine, i'm not willing to try without more research first - Don't hold me responsible if you wipe /system and then power off and something bad happens...So make sure you have a new /system partition ready to install if you use Wipe A01 ))
And, last warning. If you don't know what you're doing with these flashable zips, then don't use them. Ask a question here first, i'll be happy to answer as best as I can.
Wow this is INCREDIBLY helpful and well-done. Thank you!
Dam this woulda came in handy two weeks ago when i was a newbie at these update scripts. LOL. I researched for hours trying to get the script right and i got it. oh well better late then never.
Sent from my Supercharged MT4G Slide Running Undead's Senseless ROM using xda premium
Undeadk9 said:
Dam this woulda came in handy two weeks ago when i was a newbie at these update scripts. LOL. I researched for hours trying to get the script right and i got it. oh well better late then never.
Sent from my Supercharged MT4G Slide Running Undead's Senseless ROM using xda premium
Click to expand...
Click to collapse
lol hahaha lets hope people take the time to learn this also makes Our jobs alot easier..again Great Job blue!
Hi! Well, i have a question.
Is that possible to Freeze an app with updater-script?
How could i do that?
Thanks, awesome guide, mate!
Related
I have spent a couple of days in hell and would now like to share my experience as a big thank you to the XDA community. However, I am fairly noobish when it comes to Android so if you feel that this does not help, please feel free to delete this entry.
It all began when I restarted my G Tablet and got lots of popus about various applications, including acore, failing. I thought that this was due to me screwing up the system in some way after more or less having followed r34p3rex's superb guide here: http://forum.xda-developers.com/showthread.php?t=827209.
Ignoring this for a moment I tried to install one of the NDK sample apps on the device. This failed with the message: "read only file system". Looking in logcat it was pretty apparent that this was also causing all other applications to fail, since they were logging errors with not being able to write under /data/.
Looking at dmesg output finally confirmed this, where it reported that the /data partition had journal errors and could only be mounted as read only (this was also confirmed by issuing the mount command from within an adb shell).
This led me on a journey from just trying to remount the partition as read only (which failed with no further message), through installing clockworkmod, different versions of tnt lite, formatting and repartitioning (or so I thought), to flashing the device through its APX interface (using nvflash). None of them worked.
Not until I read a post by raydog153 - in this archived thread: http://forum.xda-developers.com/archive/index.php/t-857875.html - did I succeed by re-partitioning the sdcard (under the advanced section).
You could argue that it was stupid to not realize that a re-partitioning was required in the first place - since the ext3 partition obviously had errors and Android does not provide a fsck command (or does it?) - but I guess I thought that this was what was actually being done in other clockwork commands or at least when doing the nvflash, but obviously that was not the case.
Anyway, sorry for the lengthy entry. I just wanted to give back to the community and help others who might get the same problems I had. Also, on the positive side I have learned a lot and, like someone else wrote, I have completely lost my noob fear of bricking my G Tablet.
Keep up the good work, guys! You Rock!
almost a week without problems with FCs, thanks for this usefull info.
I have an LG Optimus T. Not the most ground breaking phone ever, but a heck of a deal for the money. Nonetheless, it's my phone, and I'm not really happy with the internal memory barrier I'm facing. For what it's worth, I have it rooted. I heard a way to partition your SD card (with part of it being allocated to ext3 file system) and re-flashing your phone somehow so the phone utilizes your ext3 partition on the SD card for housing applications. I heard several users in the IRC chat confirm this works well.
However, now that it's show time and I'm trying to do it, I'm a little confused. So please, be gentle, and I apologize if I'm re-hashing simple steps. I'm just trying to make sure I get what I'm supposed to do.
I'm working from this link here:
http://forum.xda-developers.com/showpost.php?p=10692249&postcount=1299
So from what I understand, the factory ROM can do fine, so I don't need to locate a custom ROM. That said, I just need to reset the phone to factory settings (thereby getting the factory ROM), run A2SD (is A2SD any different from A2SD Killer that they speak of in this post? I never heard of it), then go through the motions of the initial setup as it suggests (language choice, time zone, etc).
Then, prior to setting up my gmail account, I'm supposed to reboot to recovery. What does it mean reboot to recovery? Am I supposed to hold down the keys to do a factory reset? Is that what they mean by recovery?
I hate to ask these questions in a form of hand holding, but I just want to make sure I'm doing it right. After re-hashing this post several times and talking to 2 buddies about it without any further progress, I figured I'd just ask.
That said, can anyone help guide me in the proper direction? I'd greatly appreciate it.
Thanks!
make sure that you are looking at the right forum next time that link was for htc desire (use it only if it says its compatible for your phone)
before you start worrying about a2sd, you need a custom recovery, found here. once you flash recovery and boot into it, everything will be clear (ext, partition, flashing, etc)
for apps2sd, id suggest using ungazes data2sd (apps2sd and data2sd are basically the same thing, at least thats my understanding)
follow the steps there. if anythings unclear, post any questions and ill check back later
I did notice it was an HTC forum, but I was speaking with users who had other devices and claimed they used the same method, so I didn't think much of it. At that point I figured it was more of an Android thing vs an actual manufacturer thing.
When you speak of the custom recovery, what *exactly* does that entail? I was told Android phones have the hard factory reset that cannot be altered as long as you don't mess with /system under root terminal. That said, this phone is pretty much brand new. I have nothing on it that isn't already stored in the cloud somewhere (google contacts, gmail, work's email, etc).
What I'm trying to get at is, is a custom recovery needed only to get my phone back the way it was before I screwed with it? Or if I'm accepting to doing a hard factory reset, is that acceptable as well?
Thank you very much for the link. This seems a little more straight forward than the last one was. I'm also trying to understand the differences between handsets and Android versions here. For example, you mentioned the previous post was for HTC. It's a difficult thing for me to grasp because I come from a very heavy Linux background. If I mess with Ubuntu on one machine, then mess with Ubuntu on a top end server, then tinker with it on a laptop, it's still Ubuntu no matter what machine it's on. So it's hard for me to grasp the concept of Android acting differently on different machines (phones) and having different methods to different devices.
That said, it sounds like the link you posted to me is more of a generic "everything should work" type of scenario. After all, it's just moving /data. It would make sense that it would work that way, right? If not, please correct me, as I'd rather be wrong 1,000 times over again and be corrected than make 1 mistake and mislead someone or brick my phone.
Thanks for your response. Hopefully the recovery thing isn't too much of a hassle and I hope this ext thing can do the trick! What kind of difficulty is involved here? Is this fix, say, grandma approved? Or is that asking for a lot?
EDIT - Also, I noticed you spoke of Apps2SD. I have that installed... is that not acceptable to what I'm running? After all, it just seems to move what parts of the apps it can to the SD card. I was aiming to have the entire application base MOVED to the SD card. Is that what it does and I'm just a little confused??
i understand what you mean (how ubuntu on one machine is the same as ubuntu on another)
its the same thing with android EXCEPT that each android device is made different by the hardware (i.e. my phone is limited by the fact that its cpu is not powerful enough to run some apps; different components make up each device in contrast to lets say iphone, where each iphone is the same; my understanding is that each rom utilizes and has commands for each of these components so if they're different then youll need a rom unique for each device)
heres a general scope over recovery:
custom recovery - allows you to flash a rom, make/restore a nand backup, well it allows you to do a lot of things
stock recovery - only a hard reset (done automatically)
a hard reset wont cut it if you screw up your phone. youll need a custom recovery (which allows you to back up your entire system as a nand backup)
when you say apps2sd, i think you are talking about moving the app to the sd card in settings. thats not true apps2sd. true apps2sd moves the whole app to the sd card, giving you unlimited space to store apps (well actually you are only limited by your sd card)
if i missed anything, tell me. im multitasking right now (sorry)
Hahah, no you're doing quite well for multitasking! I understand what you mean, and it makes sense to me. I mean, after all, isn't it still the same analogy to Ubuntu? (at first I didn't think so, but now I think it may be more relevant) After all, if I'm running a dual proc 6 core system with 24gb of RAM, it'll certainly move Google Earth (aka, an app) faster than a Pentium 3 with 128mb of RAM would, even though they're both packin Ubuntu, Fedora, Debian, or whatever your flavor may be. So in theory, I suppose you can apply the same analogy from Ubuntu as you do Android. The catch is, Ubuntu (or any Linux distro for that matter) runs on old hardware great, so the gap between old computer vs super computer is less relevant than today's Android based smart phones, where 300mhz could mean whether or not you can run a certain app. Compared to real life, Ubuntu operates just as quick to me on my dual proc 6 core (12 core total) server with 24gb of RAM as it does my P4 1gb RAM box on my test bench.
Anyway, going back on topic a bit more... yeah I was thinking of the Apps2SD from the market. So I suppose the "real" Apps2SD is a true "mv -R" type of application (linux recursive move of data).
Okay, so get the market based App2SD out of my head. Gone. Done. Okay... the thing I'm still not understanding is the recovery. I mean, I'm looking at the hard factory reset as, say, a recovery "Ghost based" partition that some Windows computers often come with.
Example - Windows gets fubar'd. Reboot. F8. Recover system. And it dumps the recovery image on the main partition. Ta da! Just like it was when you bought it at Best Buy!
I mean, if I could get it back to factory, then I could just start over, right? Trying to put it into words as best I can... I was thinking of it like this.
Hard factory reset = Windows recovery image from Dell factory (back to day 1).
Nandroid custom reset = Own "self made" Clonezilla/Ghost image (customized with your stuff as a backup you made yourself)
That's what I was thinking. So that being said, am I absolutely positively required to do a custom backup? Can I not just rely on the hard factory reset to take me back to day 1 from when I got it from T-Mobile and (if I so desire) start the process over if I tank it? You see, I was told that it's nearly impossible for me to permanently brick an Android phone, as the hard reset, while inconvenient since it would wipe my stuff, would at least get me back to a bootable, functional ROM.
If I'm re-hashing unnecessary things and just need to focus on the fact I have no choice, by all means, e-slap me. Thanks for your help so far. I'm just trying to learn what I can! Your time is appreciated.
the thing is, a time may come when you cannot even boot your phone (lets say you were messing around and modding and all of a sudden you get stuck in a bootloop) and you try doing a factory reset but it doesnt work. this might be because something is corrupted. for android, a factory reset only wipes the data but does not restore anything. this is where custom recovery comes in handy. not only can custom recovery do a factory reset, it can restore a backup if your phone ever gets screwed up (unless you screw it up so much that you cant even access your recovery, then things get much more complicated). think of this backup as a system restore on windows (creating a restore point on windows is like creating a nand backup in custom recovery)
and dont worry about all the questions a few months back i never knew this myself so i understand
Well my friend, your patience is certainly appreciated. I understand a little better now. So let me throw out another analogy just to further solidify what little understanding I feel I have.
Factory reset = Windows restore point
Nandroid reset = Self made recovery image via Clonezilla/Ghost
Your point is a Windows restore point is only good as long as the system is solid. If your hard drive dies, suddenly your backed up restore point is useless. But... dump on that Clonezilla/Ghost image you have saved elsewhere and you're good to go.
That said, how should I get started? How do I make a Nand backup? Any guides you recommend? Will this back up my current apps, settings, etc?
Once done, how would I go about the actual Apps2SD method for making the SD card act as app storage? I assume it's that ungazes link you provided earlier?
Again, thanks for your help!
Figure this is where I should come in to help. You can install Amon_ra's recovery or clockworkmod onto your device (you can choose if there is both) but you should be able to do all that from a backup and restore menu in the custom recovery. You can go get the free version of rom manager and it should be able to install clock work recovery all without that much input from you, then you boot into recovery from rom manager. It shows you this simple yet amazing GUI that you never thought possible with something so small such as your android device. You are then given choices, do you want to flash a zip to mod your system such as custom roms, new keyboards, etc. Backup and restore (nandroid backup) or even mount it in USB mode which is useful if you never backed up, and didn't keep a copy of a custom rom on your SD card. (I never made that mistake of atleast the latter option. lol) So basically, the recovery is the "safemode" of android, only you don't get to access all of your stuff until you leave "safe mode"
and a bit of advice: NEVER flash anything without 50% or more battery.
It probably will atleast softbrick your device if it dies mid-flash.
oh, I see. Yeah I thought recovery was the actual PROCESS of pulling an old backup over, not so much operating in a sterile system environment such as safe mode.
About flashing, I would suspect it would best be done plugged in with AC power. I've had a laptop or two nearly die on me in my flashing days (went to the bathroom, came back, flashing red light, uh oh!) so I think I'll be good there.
So which applications do I need to get started? The problem I face is when I google something, there's no less than 600 ways to do all sorts of various tasks. So I have a hard time separating which one is best practice, which one is the old method, which one is definitely a bad idea, etc. So you say I can choose which ROM to use. Are they pre-built into whatever recovery software I need to use to get this job done? Or do I have to download them somewhere and put them on my SD card and then boot into recovery?
EDIT - hey how about this lil guy here?
http://code.google.com/p/android-roms/wiki/SPL
Came across it when I was googling for Nandroid Backup.
EDIT 2 - Or this guy. Looks to be the same step as the Nandroid step within the first link. Looks like the first link, while a bit more dated, is also a bit more thorough.
http://theunlockr.com/2010/02/06/how-to-backup-and-restore-your-android-phone/
Since you're rooted, you can run thiss https://market.android.com/details?id=com.koushikdutta.rommanager&hl=en and it will be able to install clocwork mod recovery. Then you just boot into it and whatever rom you want to use (found in the android development forum for your device) you install that zip from SD card and voila! You're golden with a new rom on your phone. It isn't completely needed though. lol
if i were you, i wouldnt use rom manager (it is known to cause problems). for now, use the recovery i posted in the second post. if you want to flash a rom, heres a list of most (if not all) roms for our phone.
as you start getting familiar things, you should start referring here
heres a general scope of things when it comes to flashing a custom rom:
1. boot into recovery
2. make a nand backup
3. you MUST wipe data/factory reset, cache, and dalvik cache. wiping the sd card is not necessary
4. flash the rom
5. boot normally (first boot can take awhile)
and remember, never flash more than one file without rebooting normally in between or else you might run into some problems
When you say wipe the dalvik, cache, etc, is that done during a factory wipe or do I need to do that in a separate step?
Edit- which rom would you suggest? I hear a lot about cm7. Thats the only one I recognize.
they are separate processes but they are found under the same category
so itll be like this:
wipe
wipe data/factory reset
wipe cache
wipe dalvik cache
wipe rotation settings
wipe battery stats
something like that (at least this is how it works for amonra recovery. for clockwordmod recovery, i believe they are categorized differently but its still the same idea)
Ahh, I gotcha. I remember reading instructions on clearing the dalvik-cache, so that's why I thought maybe I'd have to re-hash those links I found and do that in a separate step. But it sounds pretty streamlined!
Well, I'll give this a shot tonight. I'm still going back and forth on which ROM to use. I hear a lot about CM7, but I also heard of 1 or 2 users saying on CM7 they lost their Google Navigation or Google Latitude functionality, both things I utilize. I also had no idea the Opt T was supported on CM7. But then again, if it's anything like Linux distros, using one of the more popular ROMs has its benefits as those are normally the highest supported.
People weren't lying when they said stop googling and just go to XDA forums. Thanks for the help yet again, bro.
im using cm7 and both google navigation and latitude work for me. and i know that the link i gave you said optimus p500 roms, but all p500 roms work for the p509 (since they both use the same hardware). just remember not to flash any roms from any other forums, flash only from the p500 forum
glad i could help. if you need any more help, just refer to that forum or pm me (since i have the same phone as you)
Sweetnsour, I was just suggesting Rom manager to be able to flash clockwork, then he could flash amon_ra from inside clockwork. I think that's possible? It was more of a hint for a newer user. I remember being on the other side and since I don't personally own your device, I was just offering general android advice. Glad to see you corrected me though, anyway, good luck OP. Hope you enjoy your device modifications.
I come here with my tail between my legs ducking from any backlash I may get from any users here. I spent the better part of the evening Googling around, asking in IRC, and trying to get this part working.
I ran into several issues. Some of them are quite comical to me.
I Googled around, as well as utilized the links that were provided to me in this thread. I learned that Clockwork and Rom Manager are from the same dev team, but they are two separate products. Likewise, I was hearing other users confirm that Clockwork is the bees knees but that Rom Manager is kind of "ehh" with predictability and reliability.
After I ran into some road blocks I ended up asking in the IRC Android chat room. Some users in there directed me to some guides. I read through them. Of course, each one was different. So to a user like me, I have no idea which one is the better one to go with. The one guide kept talking about how I need to run a .sh script. So, of course, I download the files supplied and extract them. No .sh script. I ask in the IRC chat and I caught hell because I should read the directions more clearly. I read through them several times and got no where. Ultimately, some users in the IRC chat decided to link me to the guide they recommended. Later, I too ran into a road block there.
Moving along, I went into the XDA-Devs chat once I found myself at the next road block. There some users spoke to me about what I was trying to do. I linked to them the guides the Android users told me to use and they were wondering why I was using such old guides to do this simple task.
This is where my frustration is brewing. There are a million and one ways to do everything. There's no structure. No consistency at all. I've done a ton of reading and I still feel like I know next to nothing about doing a "simple" recovery. I know phones are different. I know ROMs are different. But holy hell. Doing "xyz" task on one Android phone vs the same task on another Android phone is as different from driving a lawnmower vs a dump truck.
At this point I installed Android SDK on my Ubuntu machine. However, "adb shell" was yielding a command not found error. It's nearly 2 am. I'm packing it up for tonight.
That said, let's back up a bit here. My goal is this. I have an LG Optimus T. I have it rooted. I would like to utilize Clockwork in order to do a recovery as Clockwork came heavily recommended by many sources I read about. Then, I would like to utilize the XDA site for a list of available ROMs for my phone and choose one of my liking.
Is this guide, right here what I should be using to get this job done?
Along with that, how exactly do I need to install SDK on Ubuntu so it works without headache?
Thank you to everybody who has been patient with helping me learn this stuff so far.
that guide works. after you flash the recovery from that guide, don't think that you are always stuck with it. you can always flash another recovery in recovery. as for sdk, do you need it to flash recovery? those commands to flash recovery can be done on the phone using a terminal emulator. (about installing sdk on ubuntu, I will check on that tmrw because I'm doing this on my phone, and over where I'm at I should be sleeping)
EDIT: and here's a list of roms for our phone http://forum.xda-developers.com/showthread.php?p=16535249
EDIT: for sdk, have you tried this?
http://forum.xda-developers.com/showthread.php?p=11823740#post11823740
Sent from my LG-P500 using XDA Premium App
Okay. I wasn't sure if it was safe to run those commands from the phone. I was tempted to but, ahh, figured I'd wait.
Now these instructions are exclusive to the Clockwork mod, correct? So if I get any other recovery (Amon Ra, for example) there will be specific instructions for that exact recovery too as well, right?
I'll give this a shot later and see where I end up. Thanks for your help!
Edit... It's a shame those commands aren't in an sh script already. Just a fun idea... What I'd add them to a blank text file and on the phone just chmod +x it and run it. Would that work??
the commands are fairly similar:
cwm (from your guide):
#mount -o remount,rw -t yaffs2 /dev/block/mtdblock5 /system
#cat /sdcard/flash_image > /system/bin/flash_image
#chmod 755 /system/bin/flash_image
#mount -o remount,ro -t yaffs2 /dev/block/mtdblock5 /system
#flash_image recovery /sdcard/recovery.img
#reboot recovery
amonra (from here):
# mount -o remount,rw -t yaffs2 /dev/block/mtdblock1 /system
# cat /sdcard/flash_image > /system/bin/flash_image
# chmod 755 /system/bin/flash_image
# mv /system/etc/install-recovery.sh /system/etc/install-recovery.sh.bak
# mount -o remount,ro -t yaffs2 /dev/block/mtdblock1 /system
# flash_image recovery /sdcard/recovery-RA-GNM-thunderg-1.1.0.img
# reboot recovery
after you flash either cwm or amonra, you wont have to do this again when you flash another recovery. to flash another recovery, you would just have to flash the recovery zip in recovery. these zips can be found here
If you are going to flash and ICS ROM after this tool please make sure to first install a GB then ICS otherwise your SD card will not be accessible. I am trying to find a fix but it has been illusive. This statement will be removed once a fix has been found and OP updated.
Now with CWM Touch!
This method uses NVFLASH to recreate all partitions and formats them, thereby getting rid of the file allocation pointers and hence renders data inaccessible without effort.
Note: I have never seen a file come back to life or its data affect other files without effort on the users part or a temporary glitch once the partitions and pointers have been overwritten. Please post a reply with evidence if you have seen it.
Benefits:
The contents of the zip file can be used to super clean G2X before a new ROM flash. This tool recreates all the partitions in the phone so wipes the pointers to all files (everything). The method it employes has existed from before. The files here will stop short of pushing a ROM, so that you can push your favorite ROM.
Prerequisites:
G2X (Will not work on O2X due to different partition table)
Desire to have a clean install of your favorite ROM
Backup your data
*** WARNING ***
This tool will wipe both phone internal memory and internal SD card. External SD card content is safe.
Don't say you haven't been warned!
What to do:
Download the attached Zip file.
Take out the battery (assuming you have backed up your data on external SD card, computer or elsewhere), connect the device via USB while holding VOL+ and VOL- together. Don't let go of the volume buttons until the device is recognized.
(I typically leave Device Manager open to see when it shows up in the list under "USB Controllers").
Launch flash.bat, sit back and wait until the process finishes (generally a good idea to open a command prompt, CD to the directory and then run the file).
Unplug, put battery in, reboot into CWM recovery (pushed by this tool) by holding VOL- first, then pressing and holding POWER).
Flash your favorite ROM.
Enjoy!
I made this package because I could not get the bluetooth to work with G2x GB 2.3.3 (and below). The issue was a disconnect problem where both the phone and headset would still think they were connected but in audio would not be routed for a call or media. After this procedure my phone has not had an issue for almost 3 weeks now. I have tried it with two ROMs and 3 different headsets without any issues.
I guess I will not toss this phone after all .
Putting it out there in the hope that it may help others solve some of their issues as well. I suspect bad data stays in places not touched by other methods.
Your comments are most welcome.
[Update]
Klathmon has made an newer version with CWM 5.0.2.8. RP by Klathmon can be found here:
http://forum.xda-developers.com/showpost.php?p=24768294&postcount=66
I have tested the windows version but linux one is untested. Use at your own risk.
[Update]
Now with CWM Touch!
Thanks to everyone who has contributed on this thread with a drive to positive change! You work/comments/suggestions are very much appreciated.
References:
http://forum.xda-developers.com/showthread.php?t=1248644. This link was the inspiration for the current work.
Link to CWM Touch development thread. I took the image posted there: http://rootzwiki.com/topic/22984-recovery-touch-enabled-cwm-5813/
[EDIT] The zip file should be fixed now. Darn work machine does something to the upload.
was just about to flash the new hellfire rom and saw this tool! will try it out and see how things go
edit: yeah corrupt zip on my end too.
The zip file seems to be corrupt. Could you re-upload plz
worked great on my end, no problems.
khanggle said:
worked great on my end, no problems.
Click to expand...
Click to collapse
The real test is how stable your ROM would stay with this method to wipe than any of the other methods.
I guess only time will tell .
wow this method really blaze hellfire much thanks to your( you) & (rom dev)
Ohh I like this. Thank you for sharing .
This thing is faster than nullifier, and I especially like how you don't need to be in CWM to use it. It just seems like doing it in "update mode" would allow better manipulation of the phone's internals.
so this tool DOES format and nullify in addition to recreating the partitions?
deleting and recreating partitions does not remove data. only by writing over existing data can you get rid of it (like nullifier does)
dodgefan67 said:
so this tool DOES format and nullify in addition to recreating the partitions?
deleting and recreating partitions does not remove data. only by writing over existing data can you get rid of it (like nullifier does)
Click to expand...
Click to collapse
He states in OP that it doesn't write anything, therefor doesn't "nullify". It formats and partitions.
dodgefan67 said:
so this tool DOES format and nullify in addition to recreating the partitions?
deleting and recreating partitions does not remove data. only by writing over existing data can you get rid of it (like nullifier does)
Click to expand...
Click to collapse
Actually you are right. Deleting, recreating and formatting partitions does not get did of the 1s and 0s, so in that sense the data is still there. However the pointers are gone, and hence even though the data is still there it can't be accessed without effort.
What do you think the nand has on it when the device is shipped... I can assure you its not ask zeroes!
Sent from my LG-P999 using XDA
mansa_noob said:
Actually you are right. Deleting, recreating and formatting partitions does not get did of the 1s and 0s, so in that sense the data is still there. However the pointers are gone, and hence even though the data is still there it can't be accessed without effort.
What do you think the nand has on it when the device is shipped... I can assure you its not ask zeroes!
Sent from my LG-P999 using XDA
Click to expand...
Click to collapse
Well no, it has a next to worthless version of android Also, the point he was trying to get across is that it would be better for you to present the script for what it is and not try to compare it to other scripts that do something entirely different. You do kind of imply that you accomplish the same thing as the Nullifier script and more, when in fact you simply do something different.
Some may and probably will prefer your script, at least for specific purposes, while others will prefer the Nullifier script.
MWBehr said:
Well no, it has a next to worthless version of android Also, the point he was trying to get across is that it would be better for you to present the script for what it is and not try to compare it to other scripts that do something entirely different. You do kind of imply that you accomplish the same thing as the Nullifier script and more, when in fact you simply do something different.
Some may and probably will prefer your script, at least for specific purposes, while others will prefer the Nullifier script.
Click to expand...
Click to collapse
Done. OP updated to reflect. Thanks for clarifying that. It was not my intention to state that this tool does anything any other does, or does not.
I think we are all trying to do the same thing... get rid of the issues when swapping ROMs like _____. I can't seem to think of a good analogy here .
thanks for clarifying the OP, that makes much more sense
and no i wasnt trying to say you were reinenting the wheel, just trying to understand what your tool is actually doing
and also i do understand that it would take effort to get data off any type of storage device once the partition has been erased. i deal with wiping hard drives in a data center all the time. we use a DOD wipe tool that writes 1s and 0s over a drive a minimum of three full times to eradicate the data
but with this tool doing it outside of CWM, it sounds like it may do a better job of it
but i must ask, why do you need to push CWM? what if i have a newer version of it already on my phone?
dodgefan67 said:
but i must ask, why do you need to push CWM? what if i have a newer version of it already on my phone?
Click to expand...
Click to collapse
There is no need to push CWM 5.02. As I said in the OP it is a mod of the script found in the reference. In that script it was pushing default recovery. Since the recovery partition is being overwritten as well you will loose existing one and you need a recovery to be able to push a new ROM. Hence the need to push recovery (instead of using super one click).
As long as the recovery is not bigger than the partition that holds it we should be able to push a newer version. However I can't leave it empty or people will look for me with a shotgun (myself included ).
dodgefan67 said:
but i must ask, why do you need to push CWM? what if i have a newer version of it already on my phone?
Click to expand...
Click to collapse
I was thinking the same thing but seems easy enough .. add the cwm img to the dir and edit the .cfg file. Hopefully that should get the new CWM flashed
So would it be benefitial or pointless to nullify to clear the data then run this script for a... super clean?
Sent from my LG-P999 using XDA
mansa_noob said:
There is no need to push CWM 5.02. As I said in the OP it is a mod of the script found in the reference. In that script it was pushing default recovery. Since the recovery partition is being overwritten as well you will loose existing one and you need a recovery to be able to push a new ROM. Hence the need to push recovery (instead of using super one click).
As long as the recovery is not bigger than the partition that holds it we should be able to push a newer version. However I can't leave it empty or people will look for me with a shotgun (myself included ).
Click to expand...
Click to collapse
ahhhhhh now i understand. my eyes have been opened LOL
---------- Post added at 07:36 PM ---------- Previous post was at 07:35 PM ----------
PL0X_Kleiner said:
So would it be benefitial or pointless to nullify to clear the data then run this script for a... super clean?
Sent from my LG-P999 using XDA
Click to expand...
Click to collapse
i think it would be pointless unless you are super paranoid
dodgefan67 said:
ahhhhhh now i understand. my eyes have been opened LOL
---------- Post added at 07:36 PM ---------- Previous post was at 07:35 PM ----------
i think it would be pointless unless you are super paranoid
Click to expand...
Click to collapse
Nobody is twisting any arms here... Use it if you want to, don't if you don't want to.
I saw benefit so posted so that others may benefit as well.
Sent from my LG-P999 using XDA
Ok so question, why are you killing all the data on your internal SD card? Why not just the individual partitions that the phone uses? Your destroying data that don't need to be destroyed before flashing a rom...
Sent from my LG-P999 using xda premium
It does not have to be destroyed. I just haven't had a chance to test without recreating emmc yet.
Next step. Likely tomorrow. Would like to test before release.
Sent from my LG-P999 using XDA
[edit] BTW, I do want to restate that there is some data that needs to be destroyed which is not being taken care of by other methods. I don't believe it is in the emmc, but rather it resides in one of the other partitions. Most methods out there are only working in the /data, /system and /cache domain.
My original problem (bluetooth disconnect) which happens to be a problem for some in the community even now was only completely fixed by this method.
The background: thought I had everything backed up but didn't realize there was something I missed. A quick wipe of all partitions on the phone has removed the data partition's file index. Then I realized I was missing data from my backup.
The problem: file recovery tools I've come across thus far don't seem to want to look for this particular type of file... simple XML. I'm also trying to limit use of the phone itself because installing of new apps to try and recover the data may overwrite sectors that are needed for the recovery. I'm already feeling sort of lucky that this is a 32GB internal memory device and my estimate of the file size is going to be <200MB. In that respect odds of data already having been overwritten in the minimal use I've had since should be slim, and if it has occurred it is probably something I can fake my way around.
Steps taken so far: from TWRP recovery's terminal I performed a dd of /dev/block/mmcblk0p38 (data partition on my Moto XT926M) to a file on an external card. This has enabled me to more easily work with the data from my Ubuntu desktop and also to not have to worry about further changes of the data from phone usage. I've already located via grep and parsed out via dd two XML files that I lost but those were easy as they were each under 100KB. I've located the start of the last XML file I want to recover but after about 1.5MB the trail goes cold. Either the data was contiguous and has already been overwritten at that point or the original file was being written, encountered a next block that was already occupied and continued writing elsewhere. I'm hoping for the latter.
SO...
With this image of the data partition and the knowledge of "this is where the file starts" does anyone have thoughts on how I can continue to work to find the missing pieces of this file? Assuming that from the staring point I've found that the original write just had to skip and continue writing at a different block, in f2fs is there a way to see that from the vicinity of the data I've located? I seem to think when looking at low-level data for some disk format method that the last handful of bytes (or maybe in some sort of header bytes?) of one file segment would indicate what block/sector/offset the next file segment would start but I don't know if that is the case here. Better yet any Linux based utilities that can take an f2fs partition dump and do advanced forensic recovery? I'm able to instruct it that the file starts at byte X of the image. I could try an Android app based solution but 1) that partition will continue to evolve and risk further destruction of desired data and 2) apps I've examined so far are great for finding pictures and videos, not so much at anything else.
My alternative is going to be to continue grepping through the image searching for known XML tags and manually trying to piece things together. At that point the 32GB MMC size changes from a blessing to a curse. Needle, meet haystack.
Oops, in hindsight I probably should have had this under "Android General" instead of this sub forum. Mods, I leave it to your discretion regarding movement but you can't argue that this is highly technical.
Any Luck
Pow_2k said:
Oops, in hindsight I probably should have had this under "Android General" instead of this sub forum. Mods, I leave it to your discretion regarding movement but you can't argue that this is highly technical.
Click to expand...
Click to collapse
Hi mate .. have you been able to recover files .. i am in a same situation ( forgot to back up my pictures and formatted internal sd to F2FS ..
please let me know
Thanks
I would recommend the link here.
At the end you will find all the interesting tools as link.
https://articles.forensicfocus.com/...bile-devices-running-android-operating-system
Best regards!
Sent from my HTC One_M8 using XDA-Developers mobile app
just some more useless link
https://www.magnetforensics.com/resources/recovering-evidence-from-f2fs-file-systems-with-ief
has anyone ever undeleted single file?
My thread and project are inactive now. Please accept my apologies for this inconvenience.
ast.ls
Blackout
ast.np
Blackout
ast.pm
Blackout
ast.ops
Blackout
ast.settings
Blackout
ast.storage
Blackout
2nd Feedback
Thanks for the dedicated thread. If I may, some more nitpicking about rudimentary things.
ast.install has hard coded file names of ast scripts, that's not very portable:
l_Scripts=( "ast.ls" "ast.np" "ast.pm" "ast.ops" "ast.settings" "ast.storage" "astcore" )
I have packages ( "ast.ls" "ast.net" "ast.pm" "ast.ops" "ast.set" "ast.sto" "astcore" )
so more generic pattern for handling like ast* may be considered. In case of renaming, ast.install could have a file renaming template (ast.rename) with say key value rows for generic renaming procedure before starting the actual transfer/installation. Another thing it could try to perform chmod 755 after installation.
EDIT: Looks like you have it but it doesn't work as system is read-only for say /system/xbin installation directory. So installation can be performed either by hand from a file manager with root access or TWRP recovery (install script for easy TWRP install may be a solution).
Do you have any comparative performance observations regarding background network access?
2a. The interesting are the influence of size of the blacklist on system/apps behaviour (fluency, speed, memory usage or possibly noticed negative consequences re some specific apps). By default whitelist contains less than a dozen of items, blacklist is empty. What packages would you consider safe to remove from the whitelist, like LG weather widget? Will 300 blacklisted items be completely practical to have?
2b. Battery consumption changes and practical benefits measured re having hundreds of packages blacklisted vs normally empty blacklist.
2c. I remember you mentioned a FW app you installing on the end of device restoration process, what functionality separate FW app provides for your use case, or is that just an app like AdAway?
hotcell said:
Thanks for the dedicated thread. If I may, some more nitpicking about rudimentary things.
ast.install has hard coded file names of ast scripts, that's not very portable:
l_Scripts=( "ast.ls" "ast.np" "ast.pm" "ast.ops" "ast.settings" "ast.storage" "astcore" )
I have packages ( "ast.ls" "ast.net" "ast.pm" "ast.ops" "ast.set" "ast.sto" "astcore" )
so more generic pattern for handling like ast* may be considered. In case of renaming, ast.install could have a file renaming template (ast.rename) with say key value rows for generic renaming procedure before starting the actual transfer/installation. Another thing it could try to perform chmod 755 after installation.
EDIT: Looks like you have it but it doesn't work as system is read-only for say /system/xbin installation directory. So installation can be performed either by hand from a file manager with root access or TWRP recovery (install script for easy TWRP install may be a solution).
Do you have any comparative performance observations regarding background network access?
2a. The interesting are the influence of size of the blacklist on system/apps behaviour (fluency, speed, memory usage or possibly noticed negative consequences re some specific apps). By default whitelist contains less than a dozen of items, blacklist is empty. What packages would you consider safe to remove from the whitelist, like LG weather widget? Will 300 blacklisted items be completely practical to have?
2b. Battery consumption changes and practical benefits measured re having hundreds of packages blacklisted vs normally empty blacklist.
2c. I remember you mentioned a FW app you installing on the end of device restoration process, what functionality separate FW app provides for your use case, or is that just an app like AdAway?
Click to expand...
Click to collapse
I "liked" your post because your nitpicking is only indication of your care and liking AST scripts, I respect that.
Now, onto addressing your 2nd phase of nitpicking, mildly jocking. As I mentioned previously, perhaps in Info Bank thread, once I get a question, in order not to repeat myself I would cover it, thoroughly, in its relevant post/tool. Please read post #3 for everything I could share on Netpolicy.
Naming issues: Please rename/copy ast.install to ast.setup or anything you wish then update the array list to your liking. This is a quick solution for anyone with the same issue. The beauty of AST is you can modify its scripts in any way you wish in a text editor, on any platforms. Please kindky, don't ask me and provide me with the solution when is not practical. I'm referring to your copying idea instead of hardcoded list of scripts. I know what I'm doing in this instance.
ast.install automatically mounts /system to 'rw' if it was 'ro'. If it was 'rw' already it never set it back to 'ro' to avoid causing problems to the process which set it to 'rw' before ast.install was executed. There are times, /system cannot be mounted as 'rw'. I can guess why, but don't wish go into details for now. A reboot often solves the problem. ast.install will show an error message if it fails but you didn't mention that in your post. So I have no idea what you have complained about at the end.
You have mentioned TWRP before as well. In order to avoid confusion to other users, AST has nothing to offer in TWRP and I don't know why you think it does. I have installed AST to /system/xbin on two LG V20 phones with the same script with no issues. If it is easier to do this manually please do by all means. When you mention installing AST from TWRP, you would give the wrong impression to users, as if AST is an overly complicated thing.
I don't have the time nor the patience to measure battery consumption. Battery consumption is very subjective, as you know. However, I would appreciate it if you like to share your future findings with the community in here.
I'm sorry, I don't recall what was exchanged in reference to 2.c.
Since I know you like AST, how about sharing something with the community about how it made things easier for you? That is a feedback too. No one would take my word for it because they think I'm on my way to become a millionaire.
3rd Feedback
xdav20 said:
I "liked" your post because your nitpicking is only indication of your care and liking AST scripts, I respect that..
Click to expand...
Click to collapse
I never do posts asking or hoping for likes. That would be rather strange assumption. Regarding your obviously negative connotation here, it would be more appropriate if you simply remove the "like" you made.
Now, onto addressing your 2nd phase of nitpicking, mildly jocking. As I mentioned previously, perhaps in Info Bank thread, once I get a question, in order not to repeat myself I would cover it, thoroughly, in its relevant post/tool. Please read post #3 for everything I could share on Netpolicy.
Click to expand...
Click to collapse
Thank you for more detailed explanation. I have to note that you added the explanations in your post #3 after I asked the relevant net policy questions. Either way, IMHO your statement "I think the performance loss or gain is debatable, totally dependant on how Android internally implemented it, and, frankly, this is something I don't wish to get heavily involved with. Please kindly, keep me out of future debates." is not very encouraging. To me it seems that you aren't interested in the discussion of benefits in performance or possible battery savings here, thus logically negating the major possible purpose of using your tool from users perspective. That's strange indeed, but I do respect your wish, so that's it.
Naming issues: Please rename/copy ast.install to ast.setup or anything you wish then update the array list to your liking. This is a quick solution for anyone with the same issue. The beauty of AST is you can modify its scripts in any way you wish in a text editor, on any platforms. Please kindky, don't ask me and provide me with the solution when is not practical. I'm referring to your copying idea instead of hardcoded list of scripts. I know what I'm doing in this instance.
Click to expand...
Click to collapse
Here you are contradicting yourself. As you previously mentioned, you made a deliberate effort to ensure any script can be renamed and still maintain its integrity thus boosting flexibility of usage even more. But here you are basically insisting such move would not be practical. In previous post I suggested you to maintain this flexibility by introducing renaming templates file, so updating the tool will be a simple seemless step/command, without manual intervention required, regardless of a chosen naming scheme. Well, again, it's your call. Perhaps you'd reconsider, just like you did with the inner AST folder in the distribution archive (Thanks BTW for that!).
ast.install automatically mounts /system to 'rw' if it was 'ro'. If it was 'rw' already it never set it back to 'ro' to avoid causing problems to the process which set it to 'rw' before ast.install was executed. There are times, /system cannot be mounted as 'rw'. I can guess why, but don't wish go into details for now. A reboot often solves the problem. ast.install will show an error message if it fails but you didn't mention that in your post. So I have no idea what you have complained about at the end.
Click to expand...
Click to collapse
Indeed I mentioned this due to failure to obtain the expected result on execution of ast.install. Note this script can't be run from ext SD card due to the obvious permission / media format issue. But, normally the /system is in RO state, so even if jackterm has required root permission, the ast.install still would fail when run from internal storage:
elsa:/storage/emulated/0/Download/ast #sh ast.install /system/xbin
cp: /system/xbin/ast.ls: Read-only file system
chmod: chmod '/system/xbin/ast.ls' to 100755: Read-only file system
cp: /system/xbin/ast.net: Read-only file system
chmod: chmod '/system/xbin/ast.net' to 100755: Read-only file system
cp: /system/xbin/ast.pm: Read-only file system
chmod: chmod '/system/xbin/ast.pm' to 100755: Read-only file system
cp: /system/xbin/ast.ops: Read-only file system
chmod: chmod '/system/xbin/ast.ops' to 100755: Read-only file system
cp: /system/xbin/ast.set: Read-only file system
chmod: chmod '/system/xbin/ast.set' to 100755: Read-only file system
cp: /system/xbin/ast.sto: Read-only file system
chmod: chmod '/system/xbin/ast.sto' to 100755: Read-only file system
cp: /system/xbin/astcore: Read-only file system
chmod: chmod '/system/xbin/astcore' to 100755: Read-only file system
I) ast.install: Installation completed.
elsa:/storage/emulated/0/Download/ast #
Why and how reboot supposed to resolve that?
EDIT: Looks like it works fine when performed right after reboot! It would be nice if install script would try to perform a mount, cp, chmod and report about the failure. Instead of showing the (fake) message "Installation completed." the hint to reboot and retry would be much more useful.
You have mentioned TWRP before as well. In order to avoid confusion to other users, AST has nothing to offer in TWRP and I don't know why you think it does. I have installed AST to /system/xbin on two LG V20 phones with the same script with no issues. If it is easier to do this manually please do by all means. When you mention installing AST from TWRP, you would give the wrong impression to users, as if AST is an overly complicated thing.
Click to expand...
Click to collapse
It has nothing to do with wrong impressions. All I tried to suggest was an attempt to provide a path to easy and working solution to install/update AST and have a workaround for system RO state issue as described above. One still will have to ensure the system partition is mounted in TWRP before doing AST install/update. Sure if ast.install or its exec environment can be fixed, there would be no need to reboot to recovery or perform manual steps with a file manager.
I don't have the time nor the patience to measure battery consumption. Battery consumption is very subjective, as you know. However, I would appreciate it if you like to share your future findings with the community in here.
Click to expand...
Click to collapse
If and when I'll proceed with net policy customisation and start to have some comarative data on battery consumption, I'll try to post here without involving you to the subject as per your wish to keep you out of future debates.
@hotcell
I have attached a modified version of ast.install that would double check if 'mount' command remounted /system rw or not. If the script still fails to copy your scripts, without displaying an error message, then, and previously, the problem has nothing to do with my script. I haven't developed 'mount' binary nor know why on your particular system things doesn't work normally. There is no 'fake' message, because of what happens the script assumes everything went well. If the scripts were previously copied to the target location, there is no easy way to know if the copy was successful or not. For such non-critical job, doing more is waste of time.
"ast.storage -lb online" displays the mounted partitions. The very last data is the current read/write status of the partition. After a reboot, could you run the above command to see what it prints about /system partition before and after running ast.install please.
I'm a tool maker, I'm not going to tell anyone where you can use my tool. I can provide guidance how the tool should be used, for productivity purposes only. What thing you have forgotten was, ast.np is a helper tool for Android Netpolicy tool. Without ast.np it is, literally, impossible to use it due to the input format it takes. Knowing how over analytical you can be, I made it clear I wish to stay away from any future debates myself. I have not stated, no one else can debate about it on my thread. I'm sorry, if you took my statement personally and I encourage you to use my thread to discuss anything related to AST. I would response, if I had to.
I thought for a while, how can I prove to @hotcell I wasn't being sarcastic when I "liked" his post? My proof is at the very end of that post when stated: "Since I know you like AST, how about sharing something with the community about how it made things easier for you?" If I didn't mean it, I wouldn't have stated a factual thing, I would had said; "If you like AST then why don't you...". Since I meant what I said without a menacing connotation, I stand by my previous decision. However, I do share your sentiment about the "like" system in place.
I have a serious issue with you or anyone else who think I have to accommodate to their specific needs in the same release that is meant for the public and no matter how accommodating I have been to you, you still have an approach that I don't feel comfortable with it. I will response equally to anyone who would want to dedicate terms to me. Once you renamed the scripts, I do not have to change the installer with your suggestion method, period. Have you thought, in the same directory users might have scripts or files starting with the "ast" prefix? That was the reason I hard-coded the script names to avoid potential disasters.
Edit: Noticed your edited comment about everything started working after a reboot. I do not like to discuss about something, specially in a public forum, when I do not have factual information. I'm normally good at observing patterns. I'm purely guessing, the reason behind the failure of mount command is to do with how certain file managers that support root handle mounting /system partition. Of course, there is always more than one reasons to such conditions.
Here is what you can do to see if that is the caase or not. Do everything as you would do after a reboot. Use the file managers you use and access the /system partition and copy something over to force the file manager to make the /system partition rw. Exit the file manager and then try the installer script again. Repeat the same thing, this time don't use the same file managers and after the system has been up and running for awhile, try the installer script again. What did you observe?
4th Feedback
Yes, you were right. After writing to the /system with MiXplorer file manager, the ast.install script fails. See screenshots: first made befre using MiX on /system, after that script worked; the second pic shows the ast.install.sh failure after using MiX, please note the (fake because of actual failure to mount and errors reported before) success report on the last line; the trird pic was made after the failed install. Also, with Root explorer, it asks to remount the system partition as R-W for operation, after that the ast.install(.sh) fails exactly as shown on the middle pic.
xdav20 said:
There is no 'fake' message, because of what happens the script assumes everything went well. If the scripts were previously copied to the target location, there is no easy way to know if the copy was successful or not.
Click to expand...
Click to collapse
The easiest way to know is probably to try to write an empty file into installation dir and check for its existance. Another way would be checking file timestamp.
I have a serious issue with you or anyone else who think I have to accommodate to their specific needs in the same release that is meant for the public and no matter how accommodating I have been to you, you still have an approach that I don't feel comfortable with it. I will response equally to anyone who would want to dedicate terms to me. Once you renamed the scripts, I do not have to change the installer with your suggestion method, period. Have you thought, in the same directory users might have scripts or files starting with the "ast" prefix? That was the reason I hard-coded the script names to avoid potential disasters.
Click to expand...
Click to collapse
Of course you don't have to accomodate or appease anyone. If maintaining renaming flexibility and extending that to install script is not you goal AST tool may benefit in terms of flexibility, just forget about this. And yes, I thought about ast prefix oversimplicity. That is the exact reason why I suggested ast.rename file template where one can explicitly define source-destination name pairs. So having a single fixed named file name for such template is all needed to avoid any possible clash.
hotcell said:
Yes, you were right. After writing to the /system with MiXplorer file manager, the ast.install script fails. See screenshots: first made befre using MiX on /system, after that script worked; the second pic shows the ast.install.sh failure after using MiX, please note the (fake because of actual failure to mount and errors reported before) success report on the last line; the trird pic was made after the failed install. Also, with Root explorer, it asks to remount the system partition as R-W for operation, after that the ast.install(.sh) fails exactly as shown on the middle pic.
The easiest way to know is probably to try to write an empty file into installation dir and check for its existance. Another way would be checking file timestamp.
Of course you don't have to accomodate or appease anyone. If maintaining renaming flexibility and extending that to install script is not you goal AST tool may benefit in terms of flexibility, just forget about this. And yes, I thought about ast prefix oversimplicity. That is the exact reason why I suggested ast.rename file template where one can explicitly define source-destination name pairs. So having a single fixed named file name for such template is all needed to avoid any possible clash.
Click to expand...
Click to collapse
Thank you for the quick response. Of course, if my script fails, so as everything else from command line. Now that we established my guess is an actual fact, I have another immediate concern.
Thanks to your screenshots, I have noticed storage.ast -lb command is not displaying the partition fs type and partition read&write status of /system and /userdata and there were errors. I installed Mixplorer and navigated to xbin and created a file. Went back to terminal and used ast.storage -lb online again and it reported everything correctly. You either have a corrupt fs table or there is a condition that I haven't seen before. To make sure, I would like to ask you to send me the following files by running these commands please:
Code:
cat /proc/mounts > mounts
cat /proc/self/mountinfo > mountinfo
cat /proc/self/mountstats > mountstats
Please make sure you have used Mixplorer or the usual things you do that may cause the problem first.
Edit: After a reboot, please run the ast.storage -lb online again and please report back if there were errors. I'm guessing if you do, there is something wrong on your end.
Disclaimer: At the time of publication of this article, I have not found a way to remedy the problem with the application mentioned in this article. If you did, please notify me to make the appropriate changes to this article.
This article came to existence after exchanging messages with @hotcell to which my further investigation revealed a problem with MIXplorer app. I understand MIXplorer has a thread on XDA so there is a chance the developer to be contacted to rectify the problem.
The problem is after /system partition is remounted 'rw' (Read & Write) MIXplorer does not set the partition back to 'ro' (Read-only) even after the application is closed explicitly. This can leave you extremely vulnerable in case a malicious app awaits for such opportunity. Fortunately, after a reboot all partitions will be set back to their default status. However, users might not reboot their phones for days.
Root Explorer from SpeedSoftware has a mount toggle (rw,ro) button for each partition you use which it gives you the control to change the status at will. Then I tested ES File Explorer, as soon as I closed the app the /system was remounted back to 'ro'.
Thanks to ast.storage -lb command I could easily tell what was going on. My advice to you is, to either contact the developer of MIXplorer and request the partitions to be remounted as 'ro' upon exiting the app or similarly to provide a toggle button. In the mean while, I wouldn't use MIXplorer on a device that is connected to the internet.
5th
@xdav20
Although irrelevant in the thread common context, please see the attachment below (as it looks like you disabled PM functionality).
I see your AST gives formatting errors, but practically I can't see any problems with my system. Also, looks like if /system left R-W mounted by a file manager or other app, AST has no way to proceed. As you noticed about re-mounting on-the-fly from UI, reboot can be avoided by remounting the /system as RO in Root Explorer. Also, another decent file manager X-plorer doesn't need an (clean) exit as it will re-mount the system as RO after FS operation finished.
hotcell said:
@xdav20
I see your AST gives formatting errors, but practically I can't see any problems with my system. Also, looks like if /system left R-W mounted by a file manager or other app, AST has no way to proceed..
Click to expand...
Click to collapse
Thank you for the files. I will start looking into them.
I have already stated, two posts up, ast.storage's -lb command works fine on my phone even with MIXplorer installed and /system remounted rw. While I'm trying to establish if your phone has a problem or not, I have to be here correcting your incorrect statement. Also, I'm wondering why you never reported the error messages before when you did about other things?
Edit: I have attached a screenshot to prove everything works. I deliberately set the /system to rw.
6th
xdav20 said:
Thank you for the files. I will start looking into them.
I have already stated, two posts up, ast.storage's -lb command works fine on my phone even with MIXplorer installed and /system remounted rw. While I'm trying to establish if your phone has a problem or not, I have to be here correcting your incorrect statement. Also, I'm wondering why you never reported the error messages before when you did about other things?
Click to expand...
Click to collapse
Perhaps my statement is incorrect, I've no idea. But how would you explain that after Root Explorer toggling to RO state ast.install works perfectly, what Root Explorer can do that ast.install can't?
For the last question there is an easy answer: When I reported about "fake installation success" clause, it was obvious the copy failed, screenshots were attached. Timing to report wasn't/isn't critical for me. Also, looks like you are using SuperSU while I've Magisk, maybe there are some diffs/settings related?
EDIT: After playing with renaming ast.net in /system/xbin via X-plore, Root Explorer and MiX more:
1. X-plore always know how to rename the file
2. Root Explorer knows and warns when in RO state, but fails to rename after subsequent renamings in other FMs. Toggling RW->RO>RW restores renaming capability.
3. MiX fails to rename first.
hotcell said:
Perhaps my statement is incorrect, I've no idea. But how would you explain that after Root Explorer toggling to RO state ast.install works perfectly, what Root Explorer can do that ast.install can't?
For the last question there is an easy answer: When I reported about "fake installation success" clause, it was obvious the copy failed, screenshots were attached. Timing to report wasn't/isn't critical for me. Also, looks like you are using SuperSU while I've Magisk, maybe there are some diffs/settings related?
EDIT: After playing with renaming ast.net in /system/xbin via X-plore, Root Explorer and MiX more:
1. X-plore always know how to rename the file
2. Root Explorer knows and warns when in RO state, but fails to rename after subsequent renamings in other FMs. Toggling RW->RO>RW restores renaming capability.
3. MiX fails to rename first.
Click to expand...
Click to collapse
Found the cause, I think I have the solution, and I have to leave to my dentist appointment soon. I will release a fix as soon as I can. There is nothing wrong with your device but with what you use that commonly cause problems to everyone else's (not literally) code. Thanks again for the files. The future fix will not fix the mount issue, mount issue is a common issue in Android because so many processes use /system partition and bound to be conflicts.
7th
xdav20 said:
Found the cause, I think I have the solution, and I have to leave to my dentist appointment soon. I will release a fix as soon as I can. There is nothing wrong with your device but with what you use that commonly cause problems to everyone else's (not literally) code. Thanks again for the files. The future fix will not fix the mount issue, mount issue is a common issue in Android because so many processes uses /system partition and bound to be conflicts.
Click to expand...
Click to collapse
Thanks, although I have no idea what you are going to fix (as it seems the issue is in mounting)
Anyway, I thought you can try to implement an algo like this:
1. check if /system looks mounted as RW, remember the state for later restoration, if not RW, try to re-mount
2. create am empty file named as current time in installation dir
3. try to check if it exists. if yes, delete it and set flag ready=true and proceed to #5.
4. if flag tried is not set (false), try to make double re-mount RW>RO>RW, set flag tried=true, return to #2.
5. if flag ready=false, return error message asking to either reboot or mount /system as RW and exit
6. if ast.rename is absent from install dir, perform scripts copy, otherwise the same with renaming according to the template
7. set chmod on script files as required. Done.
@hotcell
I have attached a pre-release version of AST to this post. This release contains support to handle multiple mountpints on online blocks. Android by default has one single mountpoint per block. However, components/su such as Magisk creates additional mountpoints, one on /dev/block/sda14 and two on /dev/block/sda18. AST can now handle multiple mountpoints on any online blocks, referring to -lb command. Thank you again for the files and your assistance in resolving this issue.
As you have guessed by now, Magisk was the cause of errors and I'm guessing the source of your /system mount issues. Now, that '-lb' command can show correct information on your device, you can keep your eyes on read and write status of both mountpoints on /dev/block/sda14. Please kindly provide a screenshot of '-lb online' so I can make sure everything was okay before releasing it on OP. Since you have installed AST in /system/xbin, please do not run anything locally without specifying its full parh in command line. I'm updating OP about this point shortly.
One minor matter, I have to respectfully ask you to refrain from referencing to AST scripts as your renamed versions. I got confused when you referenced ast.np to ast.net. For anyone following our conversations it can workout confusing too. At this point of time, I don't wish to make a rule and publish it on OP. I hope you can understand the point I raised. Thank you for your understanding.
Edit:01
Added two screenshots. Magisk-Partitions image shows the online partitions with Magisk being installed (based on your device data files) and Default-Partitions image shows a typical device without Magisk? Have you spotted the additional entries?