Database Info

[DEV][SCRIPT] First-Boot App Install

First-Boot Install System​
I have searched Far and wide for something like this since i first put out SleeperROM in November and always come up empty.
So with the latest release, i decided it was finally time to do it myself.
All you have to do is, using the following package as a template either on its own or in your ROM, make sure your batch folder with the .apk's to install are in /data/.firstboot/
Why
Some apps like QuickPic, ConnectBot, TinyFlashlight, Flash, Google Goggles and others that rely on linked libs don't like to simply be copied to their install dir because many won't install their own libs unless the PackageManager does it and/or they won't add themselves to the packages list (like QuickPic). The old solution is to include the lib either in the /data/data/appdir/lib with the rom install OR in /system/lib but this is quite wasteful especially in the case of big apps like Flash where including the libs separately from the app effectively doubles the space taken up on the rom by that single app since the apk still contains the lib files within.
So the solution is to install on first boot by including the apps in a batch folder for the script to process.
How it works
What it does is run from the init scripts, as one of the last scripts to run, it waits until the Android core system is up (checks to be sure by waiting for the SystemUI process is running then waits for an additional 10 seconds)
Then runs /data/.firstboot.sh, which is where you should put your first boot routines. Included in this script is the batch app installer which looks for the apps in /data/.firstboot/ and stores temporary app backups in /cache/tmp -- so be mindful that on MTD roms, the cache partition is smaller so may have to modify the $tmp variable to wherever you want to store temporaries
After installing, it sleeps for a proportional number of seconds (5 seconds per app installed) to ensure there is no race condition between installs or the final permissions_fix, zipalign and tmp cleanup. The /data/.firstboot.sh script removes itself when everything is complete.
The remaining components are kept in there for additional use by the /etc/init.d/Y02firstboot script in the future, so then all that needs to be put on the phone is a new /data/.firstboot/ dir and replacement /data/.firstboot.sh to run a batch of updates.
The Apps MUST be named by their package name.
I.e. Titanium Backup MUST be named com.keramidas.TitaniumBackup.apk
the file name must correspond with the name of the data dir in /data/data/ the script is lazy in that way, i didn't feel like keeping a file manifest of installs, instead just like to drop in apps to install.
The installer
consists of the following components:
- /system/etc/init.d/Y02firstboot
- /system/xbin/rsync
- /system/xbin/fix_permissions
- /system/xbin/batch_zipalign
- /system/xbin/sleeperlog (echos, logcats, and writes a log of activity to /sdcard/sleeperlog.txt)
- /data/.firstboot.sh
- /data/.firstboot/ (batch app directory)
- NOT INCLUDED, there must be a busybox bin somewhere in $PATH - this is not a problem for most roms
See the package link for a ready to use first-boot installer [CWM] flashable. Just drop your apks into /fs/data/.firstboot/ for a batch app updater
Download the template/usable package
DOWNLOAD MOD_CWM-FirstBoot-20120112.zip
File size: 341.07 KB / MD5 23d88c349b8d2fa3cd2f9958ae99a1f6
​
THE MAIN COMPONENTS:
/system/etc/init.d/Y02firstboot
Code:
#!/system/bin/sh
# execute post-install script on First boot
# 01022012 SENSEISIMPLE
SLEEP=3
FBSCR="/data/.firstboot.sh"
BB="busybox"
pg () {
$BB ps | $BB grep "[email protected]" | $BB grep -v "$( echo $BB grep [email protected] )"
}
if [ -f "$FBSCR" ]; then
#install apps on first boot after system services have started
sleeperlog "Found $FBLOC"
sleeperlog "Waiting for system"
$BB chmod 0755 $FBSCR
while : ; do
if pg systemui; then
$BB sleep 10
sleeperlog "system loaded."
log -p i -t boot "Executing $FBSCR script"
sleeperlog "Running FirstBoot init"
$FBSCR
break
fi
sleeperlog "WAITING FOR SYSTEM SERVICE: sleeping for $SLEEP s..."
$BB sleep $SLEEP
done
fi
/data/.firstboot.sh
Code:
#!/system/bin/sh
#
# 20120107 - SENSEISIMPLE
#
log -p i -t init:firstboot "INIT.firstboot BEGIN: USER SCRIPT $FBLOC.sh"
sleeperlog "FirstBoot Script OK TO RUN"
FBLOC="/data/.firstboot"
tmp="/cache/tmp/firstboot"
bak="$tmp/bak/data/data/"
BB="busybox"
RM="$BB rm"
CP="$BB cp"
MV="$BB mv"
MKDIR="$BB mkdir"
LS="$BB ls"
CHMOD="$BB chmod"
SLEEP="$BB sleep"
GREP="$BB grep"
pg () {
$BB ps | $BB grep "[email protected]" | $BB grep -v "$( echo $BB grep [email protected] )"
}
$CHMOD 0777 /data
#install apps on first boot
if [ -d $FBLOC ]; then
sleeperlog "Found $FBLOC"
if [ "$($LS $FBLOC)" ]; then
cd $FBLOC
$MKDIR -p $bak
for pkg in *.apk; do
log -p i -t init:firstboot "INIT.firstboot INSTALLING: $pkg"
sleeperlog "PREPARING: $pkg"
pkgname=${pkg%.*}
sleeperlog "BACKING UP APP DATA - $pkgname"
#back up data, delete the original data dir to prevent install errors (pm isn't very good at what it does)
if [ -d /data/data/$pkgname ]; then
$CP -a /data/data/$pkgname $bak/$pkgname
$RM -rf /data/data/$pkgname
fi
#WAIT, then install, then WAIT SOME MORE
$SLEEP 2
sleeperlog "INSTALLING $pkgname"
pm install -r $FBLOC/$pkg &
$SLEEP 10
#REIntegrate application data
if [ -d "$bak/$pkgname" ]; then
sleeperlog "\nSYNCING APP DATA \n\n$(/system/xbin/rsync -auv --exclude=lib $bak/$pkgname/ /data/data/$pkgname/)\n"
fi
i=$((i+1))
#Move the install .apk to tmp
if $LS /data/app | $GREP "$pkgname"; then
sleeperlog "Package appears to have installed."
$MV "$pkg" $tmp/
fi
#If the firstboot batch dir is empty, delete it now
! [ "$($LS $FBLOC)" ] && $RM -rf $FBLOC
done
#WAIT for [#ofapps x 5 seconds each] to avoid a race condition
waitsecs=$(( i * 5 ))
sleeperlog "Waiting for ${waitsecs}s before Fixing Permissions"
$SLEEP $waitsecs
sleeperlog "Fixing Permissions $(/system/xbin/fix_permissions)"
sleeperlog "Running batch zipalign \n\n $(/system/xbin/zipalign)"
sleeperlog "Clearing tmp $tmp"
fi
fi
sleeperlog "FIRSTBOOT SCRIPT COMPLETE"
log -p i -t init:firstboot "INIT.firstboot SELF DESTRUCT FIRSTBOOTSCRIPT"
if ! [ "$($LS $FBLOC)" ]; then
$MV $0 $tmp/.firstboot
# COMMENT THIS OUT FOR DEBUGGING, TO NOT REMOVE THE TMP DIR
$RM -r $tmp
echo ""
fi
echo -e "#\n#COMPLETED ON $(date)\n#" >> $0
sleeperlog "FirstBoot Apoptosis"
log -p i -t init:firstboot "INIT.firstboot END: USER SCRIPT $FBLOC.sh"
THANKS
CyanogenMod team for the Fix_permissions script
DarkyROM for the Batch Zipalign script

Saving this seat . . .

does this work with a bml rom?

^^^^In theory, it would work with an NTFS rom, if one existed - different filesystems don't [usually] create any changes on the surface... if you try to copy a file from a partition of any file system to a partition of any other file system, you don't have to explicitly tell the system the fs types involved. I haven't looked through the entire script, but if there are any commands that must specify the partition type, it would be a simple matter of changing any occurances of EXT4 to YAFFS2, etc.
This could be a great way to make a clean reinstall with all of your apps intact (minus app data, do a separate backup with your backup app of choice) - first, backup your installed apps by copying them to the appropriate directory, then do a full wipe and install, with the script automatically reinstalling your apps on first boot...
EDIT: I just looked through the script more closely. First, it appears that data is backed up. Second, I can confirm that the standard, non-file system-specific linux commands are used for file operations (cp to copy, etc), so this script shouldn't need any adjustment to accomodate different file systems
Sent from my SPH-D700 using XDA App

Is it possible for someone to develop an application to backup your personal apps and then running a script in clockwork or on first boot that does the same thing as this, alllowing for personal apps to be in the rom directly after flashing. I understand that sometimes apps may not be compatible, but can randomroms rdu script be modified to choose to do this?
Sent From My Cyan4g

Good looking script SenseiSimple!!

Okay, I'm trying to use this script, but you said to put the apps in fs/data/firstboot/ but where is "fs"? I found the "firstboot.sh" file, but I'm sure I can't put the apk's in there...
In loving memory of my son "Jeffrey Ryan Giles" 11/17/1992 to 11/25/2011 :'(

sniperkill said:
Okay, I'm trying to use this script, but you said to put the apps in fs/data/firstboot/ but where is "fs"? I found the "firstboot.sh" file, but I'm sure I can't put the apk's in there...
In loving memory of my son "Jeffrey Ryan Giles" 11/17/1992 to 11/25/2011 :'(
Click to expand...
Click to collapse
If you look at the folder structure inside the SleeperROM zip, you'll see he uses an fs directory to hold his system and data folders for installation. If you're developing a ROM, you can adapt the script to point to wherever your data folder is (most other ROMs just have system and data in the root directory of the zip, so the path would just be data/firstboot/).
EDIT: no need to look in SleeperROM zip, the firstboot zip in the OP has the same folder structure. It just doesn't seem to have the firstboot directory inside data. You'll need to add that.
Sent from my SPH-D700 using XDA App

bbelos said:
If you look at the folder structure inside the SleeperROM zip, you'll see he uses an fs directory to hold his system and data folders for installation. If you're developing a ROM, you can adapt the script to point to wherever your data folder is (most other ROMs just have system and data in the root directory of the zip, so the path would just be data/firstboot/).
EDIT: no need to look in SleeperROM zip, the firstboot zip in the OP has the same folder structure. It just doesn't seem to have the firstboot directory inside data. You'll need to add that.
Sent from my SPH-D700 using XDA App
Click to expand...
Click to collapse
Thanks for the reply buddy, so what I THINK your saying is to create a folder called "first boot" in my data folder?
In loving memory of my son" Jeffrey Ryan Giles" 11/17/1992 to 11/25/2011 - RIP :'(

sniperkill said:
Thanks for the reply buddy, so what I THINK your saying is to create a folder called "first boot" in my data folder?
In loving memory of my son" Jeffrey Ryan Giles" 11/17/1992 to 11/25/2011 - RIP :'(
Click to expand...
Click to collapse
Are you trying to do this in a ROM or on the phone? Sorry if that's a dumb question, but I just want to be sure we're on the same page.
Sent from my SPH-D700 using XDA App

sniperkill said:
Thanks for the reply buddy, so what I THINK your saying is to create a folder called "first boot" in my data folder?
In loving memory of my son" Jeffrey Ryan Giles" 11/17/1992 to 11/25/2011 - RIP :'(
Click to expand...
Click to collapse
bbelos said:
Are you trying to do this in a ROM or on the phone? Sorry if that's a dumb question, but I just want to be sure we're on the same page.
Sent from my SPH-D700 using XDA App
Click to expand...
Click to collapse
i hadn't noticed my zip routine apparently skips empty .folders
in the zip, there SHOULD have been /fs/data/.firstboot into which you place whatever apk files named according to their actual package name i.e. com.keramidas.TitaniumBackup.apk for Titanium Backup, com.alensw.PicFolder.apk for QuickPic and so on...
you can find out this info in Titanium Backup by finding the app in the list and tapping its name to show the data store location.
it HAS to correspond to the actual package name in order to properly back up the data files because the package manager binary as run from a script won't overwrite the data dir so the old data has to be synced back into the new install (rsync)
I am currently refactoring the script to reduce the number of dependencies... the system will then consist of the init.d script, zipalign and rsync bins, and the /data/.firstboot.sh, /data/.firstboot dir all in one place rather than scattered throughout the system.

irule9000 said:
Is it possible for someone to develop an application to backup your personal apps and then running a script in clockwork or on first boot that does the same thing as this, alllowing for personal apps to be in the rom directly after flashing. I understand that sometimes apps may not be compatible, but can randomroms rdu script be modified to choose to do this?
Sent From My Cyan4g
Click to expand...
Click to collapse
it would be possible to create an app to do this, and i considered it because of the need to make sure the android system is fully loaded for package manager to work, but given the requirement for full root access to start running before the system is up without having to worry about android permissions or getting SuperUser approval it's easier/more effective to do it as part of the system init in a script.
as far as randomking's rdu script, they would be compatible, but they exist separately as in one shouldn't interfere with the other, because the firstboot script runs long after install once the system is up and running.
you could modify the rdu setup to include/exclude the /data/.firstboot dir and .firstboot.sh script based on the selected config, since the init script does nothing if it doesn't see the /data/.firstboot.sh file, and the .firstboot.sh script does nothing if it doesn't see the .firstboot dir

SenseiSimple said:
it would be possible to create an app to do this, and i considered it because of the need to make sure the android system is fully loaded for package manager to work, but given the requirement for full root access to start running before the system is up without having to worry about android permissions or getting SuperUser approval it's easier/more effective to do it as part of the system init in a script.
as far as randomking's rdu script, they would be compatible, but they exist separately as in one shouldn't interfere with the other, because the firstboot script runs long after install once the system is up and running.
you could modify the rdu setup to include/exclude the /data/.firstboot dir and .firstboot.sh script based on the selected config, since the init script does nothing if it doesn't see the /data/.firstboot.sh file, and the .firstboot.sh script does nothing if it doesn't see the .firstboot dir
Click to expand...
Click to collapse
Expanding on the rdu script idea, how about a prop variable in the config to specify a user apps directory on the sdcard (best if standardized)? The user can store the apks to be installed after a clean flash in this directory. The rdu script would be modified to copy those apks to the firstboot directory. Although how to handle data? Maybe another script to be run before flashing that checks the apks in the sdcard folder, and copies the current data folder to the sdcard. This data is then restored somehow with the firstboot script. This all can be in addition to the firstboot apps included with the rom.
Sent from my SPH-D700 using XDA App

app
I honestly have no developing experience. To clarify, I was I simply asking if the app could back up the users personal apps/data to the specified folders, to make this script fool proof. Then when a rom is flashed the randomromkings rdu script which currently decides on lite roms or full roms could be modified to either install apps or not install apps depending on the rom compatibility. Finally data is wiped, rom flashes, and your script runs on boot allowing the apps to all be there. Expanding on that if the ROM allows apps to be re installed via your script, through the modified rdu, your script could be moved from one directory to another allowing the process to happen and at the end moves itself back to the original directory. This means that the end user who has no clue how to do anything but flash roms (like myself, but I want to learn) has done nothing accept backing up their apps. I know this is all hypothetical, but would this be possible, and also I have another idea that is probably impossible. If a script could be written so that the apps are backed up when the rom flashes, then the rom automatically does a factory restore and wipes the various caches, meaning that less people are going to have force close issues or other avoidable bugs because they didnt wipe. thanks for all your hard work

Clear Me Here !!
You said that the script (.firstboot.sh) removes itself, but does it remove Y02firstboot script from init.d?? I don't think so, also, there are no disadvantages of this after first boot, as it won't find .firsboot.sh and skip the operation, but why shud it execute unneccesarily everytime??
@OP--Plz. tell me if i am right or wrong.......If right, plz. add a function to remove that script from init.d. However files in /xbin are OK though, so no need to remove them............

Everybody is writing and making this appear as complicated as possible! So let me clarify what I believe the idea is here. First of all, the 'RDU script' I assume you're referring to is simply a flash-able zip to set you phone for a lite or full installation. It sets a config file. I use it in my rom, and I also currently use that config file to set several other features upon installation, and changeable at anytime one of my themes is flashed.
RDU was simply meant to jump start the idea of developers making installations more uniform. Which has happened to various extents, for example, I've used this first-boot install to solve my quickpic(I love the app) and flash pre-install problems. It also fixes usb tether, vlingo, terminal, etc.
SO, what are we saying here? We'd like to be able to backup our apps to the sd card, as many apps can do for us. Then, we'd like to either:
A. Build a script into a rom to restore these apps prior to or upon installation.
or
B. Build a separate script which would be run AFTER a rom installation to restore these apps from CWM.
Yes?
P.S. I see this is very late to the game, didn't realize this thread was being revived...
Still. Neat idea, sorry I hadn't noticed it yet.

+1 Thanks
After integrating this into my custom ROM, my phone (Galaxy S 4G) would only install the first .apk file in the .firstboot directory. After I removed the code which tells it to backup/restore the /data/data directories, it worked fine. I won't need that code since the ROM does a full wipe of /data every time anyway, but I'm not sure why it doesnt work when it's there. I'm not well versed enough with Java's syntax yet to comprehend why.
Also, your first post says that it should record logs to \sdcard\sleeperlog.txt but the script tries to record it to \sdcard\sleeperromlog.txt and neither one of those files actually appear on the sd card.
Edit: I had to remove the check to make sure the .firstboot directory is empty before deleting it, but it works fine on my Galaxy S 4G ROM as long as there are no /data/data directories for the programs I am installing.

does anyone still have a template of this? The link is down.
Edit: nevermind I don't need it anymore.

[Scripting][ARM][Installer][Port] Python (static) 2.7.9 and 3.4.2

(Check out https://www.python.org for information on what python is.)
Yes you read that correctly, static python for Android! It took a while to figure out how to get this compiled, but I finally did it. I present you with an installer and a little bit of testing. The second post contains information on compilation and the sources. Btw, any cool python scripts are welcomed!
INSTALLATION
Recovery flash installers are attached that will install python 2.7.9 and/or 3.4.2 to /system/pythonX.X.X. You can install both if you want. Installation size is about 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
After installation, the python director(ies) in /system will contain the static python binary, some scripts, and a bunch of modules and documents. Separate scripted executables will be installed to /system/bin/python or /system/bin/python3 depending on which one is installed. These basically just set the PYTHONHOME environment variable and execute the python binary.
TESTING
When you boot up to Android after installation, you should be able to just open up a terminal and fire off some python commands.
Test using python 2.7.9:
Code:
python -c 'print "Hello World!"'
Test using python 3.4.2:
Code:
python3 -c 'print("Hello World!")'
You can also write scripts shelled with python:
Code:
#!/system/bin/python
print "Hello World!"
Make sure to set them as executable with "chmod +x".
Adjust your screen brightness via /sys (root required, tested on Galaxy Nexus):
Code:
python -c 'f=open("/sys/devices/omapdss/display0/backlight/s6e8aa0/brightness","w"); f.write("40"); f.close()'
A cool script I made to tweak file system I/O like rq_affinity, rotational, etc for I/O blocks:
Code:
#!/system/bin/python
import os,re,sys
list=[]
# find all directories containing rq_affinity
for roots, dirs, files in os.walk('/sys'):
for file in files:
match=re.search(r'\S+/rq_affinity',os.path.join(roots,file))
if match:
list.append(match.group().replace('rq_affinity',''))
# write specific values to files in each directory found before
for dir in list:
for name in 'rq_affinity', 'rotational', 'read_ahead_kb', 'nr_requests', 'iostats', 'nomerges', 'add_random':
try:
f=open(dir+name,'w')
if name is 'rq_affinity': f.write('1')
elif name is 'read_ahead_kb': f.write('512')
elif name is 'nr_requests': f.write('512')
else: f.write('0')
f.close()
except IOError:
sys.stderr.write('Problem writing to ' + dir+name + '\n')
Or execute "python" or "python3" without any parameters to open up the interpreter and go from there. When you start for example "python" (python 2.7.8), you should see something like:
Code:
Python 2.7.8 (default, Dec 2 2014, 05:15:18)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
***I haven't fully tested this yet, so let me know how it goes. I know basic commands work, but there are still some complexities in the compilation that may need to be figured out.***
(update) -- Install Extra Packages/Modules via "easy_install" -- (root required)
1) In a shell, remount /system read-writable with:
Code:
mount -o remount,rw /system
2) Change directory to /system/pythonX.X.X, i.e.,
Code:
cd /system/pythonX.X.X
3) Execute easy_install followed by a package name, i.e.,
Code:
./easy_install [i]package_name[/i]
4) Let it download and install! Then test it out.
5) Remount /system read-only with:
Code:
mount -o remount,ro /system
Also, to remove a package, execute:
Code:
./easy_install -m [i]package_name[/i]
rm -r ../lib/pythonX.X/site-packages/[i]package_name[/i]*.egg
*** Flashing the resolv.conf patch may be required to make an internet connection to download modules.
Also, a few things need to be done to get easy_install working:
1) For python 2.7.8 or 2.7.9 you need to link python2.7 to python like this:
Code:
ln -s /system/python2.7.9/bin/python2.7 /system/python2.7.9/bin/python
I'll make sure this is automatically done in future installs.
2) SSL certificates need to be added to "/etc/pki/tls/certs/" to use SSL with easy_install.
Code:
mkdir -p /etc/pki/tls/certs
curl http://curl.haxx.se/ca/cacert.pem -o /etc/pki/tls/certs/ca-bundle.crt
ISSUES
The python installations have the following modules compiled in with external dependencies such as libraries not included at this point:
Code:
--------------PYTHON 2.7.8-2.7.9----------------------
_hashlib _multiprocessing _ssl
_testcapi bz2 crypt
dbm
Code:
--------------PYTHON 3.4.2-----------------------------
_crypt _dbm _decimal
_hashlib _multiprocessing _ssl
_testcapi
I will look into this more.
As @bubbleguuum points out, name resolution with python 3.4.2's urllib wasn't working. I found a workaround by adding the line
Code:
options single-request-reopen
to /system/etc/resolv.conf. This still needs some testing but seemed to work for me. I've included a recovery flash zip to patch /system/etc/resolv.conf if the line is not there since this is a root operation.
UPDATES
* 2014-12-02 * Compiled in more modules for each installation. This makes the installation a bit bigger, but it's worth it. You get a more complete python! To get stuff like help functions and math and readline modules, you need the larger installations linked below under "DOWNLOADS".
* 2014-12-03 * Thanks @cybojenix for pointing out some incapatibility issues with the original edify installer I was using. I have now updated the installers to use SuperSU's great non-edify sh-script installer.
* 2014-12-05 * Some updates to the installers to include more modules, plus size reduction for python 3.4.2. Modules added to python 2.7.8 : _bsddb _ctypes, _ctypes_test, _hotshot, _json, _lsprof, _sqlite3, future_builtins, and ossaudiodev, easy_install, pip(broken for now). Modules added to python 3.4.2: _bz2, _ctypes, _ctypes_test, _gdbm, _json, _lsprof, _opcode, _sqlite3, _testbuffer, _testimportmultiple, ossaudiodev, xxlimited
* 2014-12-05 * Some big updates to the installer to include almost the rest of the modules that wouldn't compile. Also easy_install is working on my end with these new experimental builds. With a little effort, should be able to get pip working as well. I had to use some hacky flags to get it to compile, ignore unresolved-symbols and such, so it definitely needs some testing. Another thing I did was tweak the install script to write over previous installations from these installations. I recommend downloading the experimental builds if you read this. The worst thing that might happen is one of the new modules I've included won't work completely.
* 2014-12-14 * Added python 2.7.9. Trimmed the installation sizes down to the much more reasonable 41MB, 43MB, and 49MB respectively for python 2.7.8, python 2.7.9, and python 3.4.2.
* 2014-12-24 * Attached a recovery flash.zip to patch resolv.conf to try and fix a urllib name resolution error.
* 2014-12-26 * A couple updates: Tweaked python installations so that modules looks for "/system/bin/sh" instead of "/bin/sh" when needed. Also a few "#!" corrections for scripts in the python bin directory. This shouldn't break anything from what I've tested, only make Android python more capable, but I'll keep the old installations attached for now just in case since the modifications involved using "sed". The second update is nice. I've added a little section on installing extra modules with easy_install(included and working with all installations).

Compiling
The environment I compiled in was a debian "wheezy" image mounted as a loop device on Android. The reason I compile this way is so I have full access to the arm environment tools, gcc compiler, etc. without having to go through the trouble of using a cross-compiler (which doesn't work in this case from my testing with uClibc, or eglibc. Code sourcery's might work but I didn't feel like booting an x86 linux installation to try). You can recreate the same environment pretty easily by using either Complete Linux Installer or Linux Deploy apps. I use both.
To get a static python compiled, I first downloaded the following packages with APT:
Code:
[i]apt-get build-dep python python3[/i]
build-essential gcc make # building tools
binutils-gold # awesome gold linker
zlib1g-dev # zlib
libreadline-dev, libncurses5-dev # readline, curses
libbz2-dev # bz2
libsqlite3-dev # sqlite3
python-bsddb3, python3-bsddb3
libgdbm-dev # gdb
libssl-dev # ssl
python-tk, python3-tk # tkinter
libdb-dev # db
python-gdbm python-bsddb3
libffi-dev # _ctypes
tcl8.6-dev # tkinter
libx11-dev # tkinter
libmpdec-dev # decimal
Then I manually installed binutils-gold by unpacking the rpm for armv7h (this isn't available with APT yet). You can get it here though. (Update: this should be available in the debian repositories now.)
For static compilation, you need to make some changes to Modules/Setup from the python source directory. Specifically, you need to add
Code:
*static*
to the top of the file. Then you need to uncomment any commented modules you would like compiled in. When you compile and see a list of failed modules, search for each of these in Modules/Setup and uncomment them. For example, change
Code:
#math mathmodule.c _math.c
to
Code:
math mathmodule.c _math.c
. Then recompile. There may be some shared dependencies and packages as well you need to figure out to get them to compile. Take a look at the attached "Setup" files for more detail.
The actually compile steps look like this:
(1)
Code:
./configure --build=arm --prefix="$PWD"/out LDFLAGS="-static -static-libgcc -Wl,--unresolved-symbols=ignore-all -Wl,--export-dynamic" CPPFLAGS=-static CXXFLAGS=-static CFLAGS="-Os -static" LDFLAGS=-static LD=ld.gold
(2) Modify the Modules/Setup file.
(3)
Code:
make clean; make install
After compiling, there are some things you can do to make your installation in "./out" smaller. Strip the large binaries in "out/bin", i.e.
Code:
strip -s ./out/bin/python2.7
Also remove the python archive library, *.o's, etc. with:
Code:
make clean
This might seem kind of weird, but it works and cleans out unnecessary files from your installation.
**Make sure to backup your Modules/Setup file if you run "make distclean". I lost my 3.4.2 Modules/Setup file after doing this. :silly: Now I gotta recreate it, dammit.
**I should also mention that compile time is very fast given the amount of data generated. It takes about 10 minutes to get python and all it's modules compiled on my Galaxy Nexus.**
:EDITS:
: Updated some configure parameters. Removed unnecessary, extra "./configure". Also added some updates to the included "Setup" files.
: Hacky update to "./conifgure" to ignore unresolved symbols, export dynamic. This allows certain modules to be compiled which have shared dependencies.
: Update to trimming down the installation size after "make". Use another "make clean".
Downloads - Extract in the source directory
Setup (python 2.7.8)
Setup (python 3.4.2)
Setup (Experimental) (python 2.7.8)
Setup (Experimental) (python 3.4.2)

Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference

cybojenix said:
Something that would be interesting to look in to would be to have pip/easy_install running, so you can install extra non c packages onto the sdcard/data/wherever.
Would you also consider opening up the sources please? There is a project I'm interested in doing involving python on Android, however a lack of time meant I couldn't finish building it.
Give me a shout if you need anything. I hope to see even more come out of this thread
Edit:
I've just looked at the updater-script. The mount command won't work on the majority of devices. Can you turn the update-binary in to a shell script please? See the SuperSU updater for reference
Click to expand...
Click to collapse
Thanks for your reply! I've changed the installer to use SuperSU's no-edify sh-scripted updater. I tested on my device, but let me know if there are any issues though. I kept it pretty basic.
I added some stuff on the source modifications and compilation to the 2nd post. The only file I've actually modified so far in the source is the generated Modules/Setup file, and I've included the one I used for python 2.7.8. Unfortunately, I over-cleaned the python 3.4.2 directory, which deleted my modified Modules/Setup there, so I'll have to add it later when I re-edit it.
That's a great idea on pip/easy_install. I did download the pip_installer and tried out installing on the static python I made. Got some errors though pointing to a few modules I haven't gotten compiled into the static python installation (listed in the OP near the bottom, specifically _ctypes). I'll see what I can do about that. The line for compiling the module might just need to be added to Modules/Setup. Hopefully that's the case.

@7175
Thank you very much for these binaries.
There is however a problem: name resolution (DNS) doesn't seem to work at all (with both python 2 and 3 downloads).
All attempts to use urllib.request.urlopen('http://somehost.com') fail with "<urlopen error [Errno -2] Name or service not known>" (running python as root but it doesn't matter, and a rooted Nexus 4 running 4.4.4 stock ROM):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 455, in open
response = self._open(req, data)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 473, in _open
'_open', req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1202, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/system/python3.4.2/lib/python3.4/urllib/request.py", line 1176, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
Click to expand...
Click to collapse
it works if replacing somehost.com by its ip address.
To reproduce above stack trace with python 3:
import urllib.request
urllib.request.urlopen('http://www.google.com')
while this work:
urllib.request.urlopen('http://173.194.45.229')
nslookup and general name resolution work anywhere else:
[email protected]:/ # nslookup www.google.com
nslookup wwwgoogle.com
Server: 8.8.4.4
Address 1: 8.8.4.4 google-public-dns-b.google.com
Name: wwwgoogle.com
Address 1: 2a00:1450:4007:80c::1014 par03s15-in-x14.1e100.net
Address 2: 173.194.45.242 par03s15-in-f18.1e100.net
Address 3: 173.194.45.240 par03s15-in-f16.1e100.net
Address 4: 173.194.45.244 par03s15-in-f20.1e100.net
Address 5: 173.194.45.241 par03s15-in-f17.1e100.net
Address 6: 173.194.45.243 par03s15-in-f19.1e100.net
Click to expand...
Click to collapse
This issue has probably something to do with how python was compiled to do name resolution. I remember vaguely a similar issue
on Ubuntu 12.04, with a ffmpeg compile that would always fail to resolve hostnames in URLs. No sure what the exact fix was but it had something to do with the libc and its name resolution mechanism. Or maybe something ipv6 related ?

@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
EDIT: Hmm, looks like re-compiling with "--disable-ipv6" didn't fix the problem. Will have to look more into this issue and hold off on uploading ipv4-only python3 since it didn't fix it. In the meantime, I've attached a recovery flash.zip to the OP to add the above mentioned line to /system/etc/resolv.conf if needed, which seems to fix the issue for me.

7175 said:
@bubbleguuum : Thanks for checking out that important issue.
I did a little research and it seems there is some issue with DNS name resolution related to ipv6. I recompiled with "--disable-ipv6" and that seemed to fix the issue. I also found that adding the line
Code:
options single-request-reopen
to /etc/resolv.conf did the trick as well. Apparently this tells the resolver to use a new socket for ipv6 resolution instead of same one as ipv4. It thereby reduces wait-time as well. Maybe check on your end and see if that works. Otherwise I'll upload ipv4-only installations since this issue at least needs to be side-stepped.
Click to expand...
Click to collapse
Thank you for looking into this so fast.
My intended usage is for running the python binary from a regular non-root app (which works great!), so editing resolv.conf is not an option.

I've been researching that DNS resolving issue and could not find a clear explanation on why it fails on Android and not on other systems, and why exactly the added line in resolv.conf fixes (or rather workaround) it.
All seems to point to socket.getaddrinfo() failing for an unknown reason.
Is Python compiled with HAVE_GETADDRINFO defined ? If that's the case it uses the libc definition of getaddrinfo, otherwise
it uses an emulation function found in getaddrinfo.c.
My guess is that it is the latter, and what is causing this issue on Android.
It's probable this issue has already been solved on Android since other Python binaries exists, but Googling around do not give
much answer.

I get this error when trying to install anything with easy_install (using /system/python3.4.2/bin/easy_install-3.4 because /system/python3.4.2/easy_install doesn't seem to exist like in the OP):
Code:
# ./easy_install-3.4 feedparser
Searching for feedparser
Reading https://pypi.python.org/simple/feedparser/
Download error on https://pypi.python.org/simple/feedparser/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'feedparser' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or download links found for feedparser
error: Could not find suitable distribution for Requirement.parse('feedparser')
i got curl and the certs installed in /etc/pki/tls/certs/ as per instrucions in the OP, any help?

Thoughts on PIL?

Hmm... new problems here. Android Lollipop have no /system/etc/resolv.conf by-default and urllib patch is no more works.
How it can be fixed?

http://qpython.net/index.php

Where I can find source code and related documents for project

Pratik Raj said:
Where I can find source code and related documents for project
Click to expand...
Click to collapse
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?

hackintosh5 said:
I would like to know the same, as well as what license it is under... Also, I don't care about the network, but does it work on Oreo? Also, do you think I could customise the installer to install to a non-system location (/tmp) so that my flashable zip can use python?
Click to expand...
Click to collapse
Idk about OP, but I can surely post my source for python 2.7 on arm/arm64. I compiled it like a year ago, but it wasn't too bad, just required a bunch of patches and code for dlopen

I just want to say thank you so much.

[TOOL][LOG] Easy-to-share log and report for developers and users

Hello XDA comunity!
I want to share you a very useful tool that I have created, its name is: "kielyd".
What's keilyd?
Keilyd is a BASH daemon that constantly dumps the system's logs, warnings and other useful info in a easy-to-share "ZIP" with the main purpose to create a very informative and useful bug report.
keilyd is expansible and configurable. It's very easy to add new features because BASH interacts directly with the Linux/BSD commands in the Android System.
How does it works?
Keilyd is launched at device boot, with the "init.d support" and as root (see requeriments below). After launched, keilyd relaunches itself in a "daemon mode", that runs in the background, then, it dumps several information in the /sdcard and compress it in a single zip, so, the user can send that zip (in fact, are 2 zip's or more if you use the "snapshot" feature) to the developer of their ROM, to a forum or send it to a 3rd party developer and report a bug with all the system logs included.
Keilyd stores 2 zips: the first one is the current log (the "very last file") and the second one is the last log before keilyd were launched again.
If you won't enable de daemon, you can create a single zip when you use the "snapshot" feature, if you take two or more snapshots in the same minute, the ZIP will be the same, otherwise, one script per different minute will be created. The Snapshot feature can create as much zips as needed.
What info is collected?
That's a good question!
The info that is collected is (this applies ONLY for a vanilla copy of keilyd):
Kernel log (aka "dmesg") and other kernel info
Logcat (aka "logcat")
List of process running and memory statics
A copy of all tweaks in init.d (including keilyd itself)
A copy of build.prop and sysctl.conf
A copy of /proc/config.gz (if exists)
Some cpuinfo
Mounted filesystems and usage
How does this "daemon" can help me?
Developer:
Have you had the need to get some system log from a user?
Probably the answer is "yes", but, the user may have no idea how to do that, so, you have to give them a "step-by-step" guide of "how to get a system log in the terminal".
This simplifies the process, so, the user can report a bug with logs with no terminal (if the daemon is enabled) or with a single command that looks like
Code:
keilyd --snap
User:
Well, the complement of the developer... You can share "golden" information just by sending 2 zips files to the dev. Minimal Android knowledge is the only prerequisite if the daemon is working as spected.
This daemon is not a demon, but it can be very bad if not implemented as spected. Please, ensure that ALL THE COMMANDS ARE KNOWN and that THERE IS NO PORPRIETARY interference.
Why BASH?
A good idea is to create a daemon in C/C++ and share it as a "binary", BUT, as we are recolecting information from the device, the user must have to be concient of what are the script recollecting, if the user can't read bash, it can ask to a 3rd developer. You can feel safer because it's under the GPLv3 licence!
In addition, BASH run in (almost) any UNIX based OS, including but not limited to Apple OS X, GNU/Linux, Android, iPhone OS, etc...
This prevent a "blind trust" from the user, and give them facility of enable-disable the script, add features and so on.
Requeriments
- A rooted device
- Busybox (or run-parts)
- init.d support
- BASH 4.3.x ONLY SUPPORTED by the moment.
- Terminal emulator/HIDE]
How can I include it in my ROM?
Take a look of my repo in GitHub, it contains the last version of keilyd and the implementation during compilation in the Makefile.
If you are a ROM modder, download the latest version from above, include init.d and bash support in you ROM.
Or you can create a CWM/TWRP/Philz... flashable zip (it would be great!)
Users: how to use
When something goes weird, open the terminal emulator and write:
Code:
keilyd --snap
and send to the developer all zips in
Code:
/sdcard/keilyd/out
type
Code:
keilyd --help
and a helpful message will be printed.
If something goes wrong and you need to reboot the device (or the device reboots itself), just send both zips to the developer or the forum where you are asking help (This feature only works if the daemon is enabled).
Developers: how to use
You can add/remove things in order to fit your needs, the only thing that we ask you is that you write the whole script in BASH, without "blobs" that may private the user from know what is exacly logged and stored.
If some user sends you the zip, you will notice that the files are plain-text, and you can view it in any text editor.
TODO:
List all apps installed (system and user)
Secure "user-modificable" settings (Allmost DONE)
A graphical interface (an app) to manage the daemon
A flashable ZIP
Aports and sugestions are WELLCOME, please, coment in the box below
Install redistributable package:
If the avobe requeriments are meet...
1. Open this GItHub URL
Click in the "Download ZIP" icon in the right.
2. Put the zip in your phone
3. Unzip it.
4. If you want to use the script... A; otherwise, B.
A.
a. Open the terminal emulator.
b. cd to the folder that contains
Code:
install.sh
c. As root, type "
Code:
sh install.sh
" in the terminal.
B.
a. Open ES File Explorer
b. Enable root explorer and mount /system as rw
c. Copy
Code:
daemon/18keily
to
Code:
/etc/init.d
d. Tap and hold the
Code:
18keily
file and open the poperties
e. Change te permisions to
Code:
rwx r-x r-x
f. Copy the
Code:
redistrib/bash
file to
Code:
/system/xbin/bash
g. Change the permisions as above.
h. Symlink /etc/init.d/18keily to /system/xbin/keilyd
i. Reboot
NOTE:
Keilyd can be used in 2 ways:
1. As a daemon that does it job without human interaction, that constatantely and automaticaly creates 2 zips in the output directory.
2. As a Script that only needs to be called with the '--snap' parameter, but it requieres human interaction and does not constantely repeat the process.
Use the first one if you are a beta tester, or if you know that you ROM can hang sudently. Recomended for Unstable ROM'S.
Use the Last One if your ROM is stable or official, but you want to do a bug report.
You can allways use the second option if you use the first one.
By default, the daemon is disabled. You can enable if you change the "ENABLED" variable to "Y". I personally recommend values for "MINS" between 5 and 15.
BUGS REPORTS in the GITHUB REPOSITORY

Reserved

Hi there,
This looks like what I need, did you close the repo? Can you share a link to the script? I have a few devices, and I want to have some forensic information for troubleshooting (I.E. log all the warnings, errors and critical logs, + a snapshot every n minutes about the system usage). Right now I have to go where the device is, connect it with ADB and query the logs (most of the time, the logs were overwrited)... Thanks in advance.

[TOOLS][ZIPS][SCRIPTS]Android Backup and Restore Tools[Multiple Devices/Platforms]

android-backup-project
The android backup project provides a set of scripts and tools to backup and/or restore applications installed on Android devices.
This is not adb backup which didn't work for my requirements as
it doesn't backup applications if those are requesting not to be backed up
it does not support restoring to different devices quite well
Google warns that it might be discontinued in future versions of Android
Motivation
I never was really happy about the possibilities to backup / restore Android devices. Especially when you are developing with (different) Android devices, you wish you were able to transfer "configurations" from one device to a different one. Or you would like to revert to an older version of an app or ...
The same is valid when you change your "main" device. The Google mechanism to set up a new device from another one or from a backup works quite well, but has almost the same deficits as adb backup: it doesn't restore all of your apps and data.
Also TWRP or other custom recovery implementation do not really help to get out of this situation. It's fine for creating a backup and restoring it later on the same device but there is no support for switching devices. In addition Android often changes architecture so that TWRP support for a specific device is not guaranteed and at the time of writing there is not even TWRP support for Android 10 yet.
There is another very interesting project related to backup and migration on XDA (https://forum.xda-developers.com/android/apps-games/app-migrate-custom-rom-migration-tool-t3862763). Unfortunately it relays on TWRP, too, and has still some issues to be solved.
Last but not least there is Titanium Backup which is available for a very long time now. It works quite well but the free version is very limited in functionality and provides only the possibility to store the backup on the device itself which is a kind of contradiction for a backup.
Some time ago I started already a similar project (https://github.com/AndDiSa/ART) to manage backup and restore of Android devices remotely. Unfortunately it never finished and due to architectural changes of Android it would need a complete workover so that I decided to start over from scratch
Precondition
adb installed and in execution path
Ubuntu Linux (other should work, too)
a rooted Android device (either by adbd unsecure or by e.g. Magisk / SuperSU)
Usage
Theses scripts kind of do that TiBackup is doing, but controlling the backup- and restore process remotely.
backup_apps.sh
./backup_apps.sh [--system-apps]
This script creates a backup of all user (and system) applications and their data. The backup files will be stored in a directory newly created named by the device and the current date.​
restore_apps.sh
./restore_apps.sh [<directory_name>]
Ths script restores the apps and their data of a previous backup created by backup_apps.sh Either the directory is identified automatically by the device connected and the current date or you can pass a directory name as parameter. In that case all the apps and data found in the given directory will be restored.
If you want to restore only a part of them, please copy them into a different directory and give that directory as a parameter to the script.
Important
Keep in mind that restoring all apps may cause issues as some have unique IDs that end up causing problems if you use the same unique ID on different devices.
You also may want to consider only restoring apps that are missing from google play or apps that sadly decide to prevent backups of their (i.e. your) data.​
full-backup.sh
./full-backup.sh [--data-backup][--no-data-backup][--media-backup][--no-media-backup][--image-backup][--no-image-backup]
This script does a full backup of (different parts of) the /data partition. There are several options to control the behaviour.
--data-backup will backup /data without /data/media and /data/mediadrm
--media-backup will backup /data/media and data/mediadrm
--image-backup will backup the whole partition which is mounted as /data as a 1:1 copy
As it cannot be guaranteed that during the backup process there will be no single modification on the /data partition it's quite common to get a checksum validation failure after the backup is finished.​
full-restore.sh
./full-restore.sh [--data-backup][--no-data-backup][--media-backup][--no-media-backup][--image-backup][--no-image-backup] <directory_name>
This script is restoring data previously created by full-backup.sh It accepts the same parameters as full-backup.sh and takes in addition the name of the directory from where the backup shall be restored.
Important
Be very careful in unsing this method, especially when you are restoring /data It will overwrite everything on the /data partition and this may cause major issues up to the point that your device becomes no longer usable. In that case you probably need to do a full wipe to get it working again.​
restore-single-partition.sh
./restore-single-partition.sh <path-to-image-to-be-restored> <name-of-the-partition-on-device>
This script is restoring a single partition image previously created by backup-partitions.sh back to the device. The image can be either in compressed or plain format. As parameters you need to give the path to the partition image and the name of the partition as is is configured on the device (see: /dev/block/by-name/<name>).
Important
This command is in experimental state and may be very dangerous as it can fully brick your device. Be very careful in using this method, you have been warned(!). ​
Downloads:
android-backup-project-v2.1.zip (md5: b0c71d602963696f7c372d3c2222ae75)
android-backup-project-v2.0.zip (md5: 14be02aa3e2a458ba7d7456dc152dad8)
android-backup-project-v1.6.zip (md5: 00f8dc8630b63fe1398ea61a436c2a0f)
android-backup-project-v1.5.zip (md5: 9cb1183d385f854c1f4b7858855cde58)
android-backup-project-v1.4.zip (md5: 9ad78f33a86e117a72058a336b34ab28)
android-backup-project-v1.3.1.zip (md5: 737a7d3483d98616f650d51eea033619)
android-backup-project-v1.3.zip (md5: 3bb11d4da02754a82f8614a53aacd4df)
android-backup-project-v1.2.zip (md5: 27f365663a97b3c7451938d24ffb7779)
android-backup-project-v1.1.zip (md5: c9806edb32b68109c2fc8e087927ac7a)
android-backup-project-v1.zip (md5: b754cf72fe10c89cdb8e4cb8e94f9e0a)
Special thanks
Credit goes to Raphael Moll who initiated a similar project some time ago and Marc Merlin who improved it to work with Android O. I took some ideas and inspiration from these projects and from my first trial I started years ago. The current implementation does not have much in common with neither of those versions.
Especially I would like to thank @topjohnwu for his great Magisk project and @osm0sis for his great collection of scripts and tools.
XDA:DevDB Information
Android Backup Project, Tool/Utility for all devices (see above for details)
Contributors
AndDiSa
Source Code: https://github.com/AndDiSa/android_backup_project/
Version Information
Status: Stable
Current Stable Version: V1.0
Stable Release Date: 2019-12-08
Created 2019-12-08
Last Updated 2020-01-19

Missing Permissions
Magisk Canary in combination with newer Android versions are restricting permissions even on rooted devices more and more. Due to that not all actions required to do a backup / restore are allowed any longer. To circumvent this restrictions there is a new Magisk Module available which enhances the permissions so that the backup / restore scripts are working.
If you are observing permission issues using the backup / restore scripts please install the AndroidBackupRestore Module in Magisk.
Edit: Latest Magisk Canary (>=20425) is working again without the need for additional changes

Reserved

I there and thanks for the work.
I tried to use your script and I am stuck at "Checking root access"
I am using Magisk. Shell is autorized in #Superuser.
In command line "su" is working see #
Here is my terminal screen
[email protected]~
$ bash full-backup.sh --data-backup --media-backup --image-backup
~/busybox-ndk ~
Déjà à jour.
~
Waiting for device...
Devices detected:
List of devices attached
da279704 device
Checking for root access...
[email protected]~
$ adb shell
gemini:/ $ su
gemini:/ #
Click to expand...
Click to collapse

@cocolol What Android version / device are you using? I did my tests with e.g. a Magisk rooted Pixel 3a on Android 10 without any issues.
Can you try from your command line the following:
adb shell su root -c whoami
and if this is blocked the following:
adb shell su root whoami
probably for your device / configuration
adb shell su whoami
is working

updated to version 1.1 as there was a quote mismatch in one of the scripts

Can you backup and restore just user apps?

str8str said:
Can you backup and restore just user apps?
Click to expand...
Click to collapse
If you use backup_apps.sh without passing the --system-apps option ... but on Android 10 most likely you will still backup (some) system apps. If there is need for that one could add a filter ...

@andisa thank you for your prompt reply and sorry for my late answer
I updated script to 1.1 and still be blocked.
Thus "adb shell su root -c whoami" is working answer is "root"
But other commands didn't work (see log below)
My aim is to jump from PE 9 to PE 10 on xiaomi MI5 and MI8 (dipper) with all my apps.
I came here from Migrate app forum which is not working for me since it is TWRP dependend.
[email protected] ~
$ bash full-backup.sh --data-backup --media-backup --image-backup
~/busybox-ndk ~
Déjà à jour.
~
Waiting for device...
Devices detected:
List of devices attached
da279704 device
Checking for root access...
[email protected] ~
$ adb shell su root -c whoami
root
[email protected] ~
$ adb shell su root whoami
[email protected] ~
$ adb shell su whoami
Click to expand...
Click to collapse

@cocolol ok, that's just as it should be, i.e. if you are rooted with Magisk
adb shell su root -c
prefix should work. Question is: why it is blocked within the script.
Edit: I expect that the script is blocked while checking a sane /data partition. Can you try to remove that part from the script? It doesn't harm and we can prove that my expectations are correct. It's sufficient the remove / comment out the checkForCleanData line.

I got an error: "Unrecognised architecture armv7l".
After adding that to line 82 in functions.sh (and switching flaky usb port) it went through.
Edit:
Device gohan (BQ Aquaris X5 plus)
OS LineageOS 14.1-20191115 (unofficial)
Host Manjaro, bash v5.0.11 (ah well, I had to install PV command)
Edit: fixed device name

Keule-T said:
I got an error: "Unrecognised architecture armv7l".
After adding that to line 82 in functions.sh (and switching flaky usb port) it went through.
Edit:
Device gohan (BQ Aquaris X5 Pro)
OS LineageOS 14.1-20191115 (unofficial)
Host Manjaro, bash v5.0.11 (ah well, I had to install PV command)
Click to expand...
Click to collapse
@Keule-T Thank you very much for the info. I'll add that this change into the next version.
Would be great if others could report issues / success, too, so we can improve the scripts to be able to run on (almost) every device.

AndDiSa said:
Would be great if others could report issues / success, too, so we can improve the scripts to be able to run on (almost) every device.
Click to expand...
Click to collapse
Well, that was a disturbing experience. I downloaded and un-zipped the v1.1 on Arch Linux, opened a terminal from within that extracted folder, and ran the command ./backup_apps.sh with no additional arguments.
The terminal reported this:
Checking for root access...
Determining architecture...
Pushing busybox to device...
busybox-ndk/busybox-arm64: 1 file pushed. 26.0 MB/s (1478208 bytes in 0.054s)
### Creating dir qcom_2019-12-21_QQ1C.191205.016.A1
~/Downloads/android-backup-project-v1.1/qcom_2019-12-21_QQ1C.191205.016.A1 ~/Downloads/android-backup-project-v1.1
Then the terminal listed about 70 lines of .apk files, and then disaster happened. These are the last few lines of what the terminal reported:
## Stop Runtime
## Pull apps
package:/data/app/com.touchtype.swiftkey-80xpz-vz5Aii7U6vdFVHnQ==/base.apk=com.touchtype.swiftkey
/data/app/com.touchtype.swiftkey-80xpz-vz5Aii7U6vdFVHnQ==/base.apk
/data/app/com.touchtype.swiftkey-80xpz-vz5Aii7U6vdFVHnQ==
com.touchtype.swiftkey
./backup_apps.sh: line 55: pv: command not found
Then after that, complete silence. I mean COMPLETE. My phone did a Force Close and powered off. Wouldn't reboot back into system, just looped. I had to do a a Nandroid restore to get back into my phone again. Not good....
OnePlus 3T rooted with Magisk 20.1 and running MSM Xtended XQ v4.0 Android Q rom.
I like the concept, but I think I'll wait until you get it a bit more developed before I do a beta test again... Maybe you can make use of the above information to further your project.
---------- Post added at 12:12 PM ---------- Previous post was at 12:11 PM ----------
Forgot to add that the only thing that made it into the directory your script created on my PC was a file named
app_com.touchtype.swiftkey.tar.gz with a filesize of 0 . That's all that was there.

@Oblias thank you very much for your report. I am very sorry for the mess but it's exactly those reports which help really a lot.
pv is a program which helps logging the communication between the device and the pc and I did not have in mind that it probably is not installed on a Unix system. I'll add a check for the presence of the important programs so that there is a stable basis before starting the scripts.
The normal behaviour of the script is as follows:
- shutdown runtime (to prevent modifications during the backup process)
- backup apps and data
- restart runtime
So a "warm" restart of the device is normal after the backup script finishes. I am only wondering why you had to restore a nandroid ...

AndDiSa said:
@Oblias thank you very much for your report. I am very sorry for the mess but it's exactly those reports which help really a lot.
pv is a program which helps logging the communication between the device and the pc and I did not have in mind that it probably is not installed on a Unix system. I'll add a check for the presence of the important programs so that there is a stable basis before starting the scripts.
The normal behaviour of the script is as follows:
- shutdown runtime (to prevent modifications during the backup process)
- backup apps and data
- restart runtime
So a "warm" restart of the device is normal after the backup script finishes. I am only wondering why you had to restore a nandroid ...
Click to expand...
Click to collapse
Well, you said in the OP that one of the things needed for these scripts to run was to have "Ubuntu Linux (other should work, too)" installed. Arch Linux and Ubuntu Linux both use the Linux kernel, so I thought that would be alright. Which leaves me a bit confused as to why you said that PV is not installed on a Unix system??? But anyways.......
As far as the normal warm restart is concerned, perhaps something got messed up in the command when the script halted?? I'm not all that good with scripts, so I don't know. Also, perhaps whatever made the phone do a poweroff instead of a reboot, also corrupted some system file? Just guessing, of course... But it's not biggie really. Anyone who tries anything like this had better have a backup on hand, just in case any sort of unpredictable event happens.
So, let me know about the PV command issue, and I'll give it another try, if you'd like.
---------- Post added at 01:27 PM ---------- Previous post was at 01:24 PM ----------
Also, as an added thought, when you get a chance, perhaps a small FAQ in the OP as to what the normal behavior is when running these scripts. For example, letting folks know that their phone will do a reboot on the script completion, and things of that nature.... Just a thought....
---------- Post added at 01:31 PM ---------- Previous post was at 01:27 PM ----------
UPDATE: I also just checked my repos, and pv is available to be installed, it just wasn't installed during a 'normal' install of Arch Linux (if there is such a thing).
So, if I install pv, are there any other commands I should check for to see if I have installed before trying the script again?

@Oblias beside those "normal" programs like git, tar, gzip, ... which are normally always installed there is no additional program needed. But nevertheless I agree that probably some additional compatibility checks should be done especially as there are different versions of the programs existing which take probably also different parameters ...

AndDiSa said:
@Oblias beside those "normal" programs like git, tar, gzip, ... which are normally always installed there is no additional program needed. But nevertheless I agree that probably some additional compatibility checks should be done especially as there are different versions of the programs existing which take probably also different parameters ...
Click to expand...
Click to collapse
Makes sense. Alright then, I'll wait until you check things out, and once you have posted back, I'll give it another try. I don't mind some beta testing..

AndDiSa said:
@cocolol
Edit: I expect that the script is blocked while checking a sane /data partition. Can you try to remove that part from the script? It doesn't harm and we can prove that my expectations are correct. It's sufficient the remove / comment out the checkForCleanData line.
Click to expand...
Click to collapse
After some investigation script was blocked on line 29 of functions.sh file on this condition "if [[ $($AMAGISK whoami) == "root" ]]"
I dont know why since when I replace this line with "$AMAGISK whoami" I get "root" as answer
Thus I replace "checkRootType" directly with "AS=$AMAGISK"
But I get the same with next function
Determining architecture...
Unrecognized architecture aarch64
Click to expand...
Click to collapse
Seems to be does'nt get the answers same as problem above
I am using cygwin maybe it come from this.
I replaced == with =~ and it works but for the case condition I found nothing

@cocolol I suppose cygwin has some issues with evaluating the expressions as aarch64 is supported. Probably also here you need to change the comparison ... If you know the type, you can also execute the code for the matching architecture.

Uploaded version 1.2 of the tool with the following improvements:
- support for armv7l architecture
- checking prerequisites needed by the tool, e.g. adb, pv, ...

Overview of installed package names, app names, version name and code on a specific device

I want to have an overview of all apps listed on my system with their package name, app name, version name and code and whether it is a system or third party app.
If it is a separate list with third party and another one with system apps that's ok to.
I keep al list of installed apps (third party and system) where I have the version number and whether it is an updated system app or not.
This is to have an overview if I ever have to reinstall my phone.
I know I can get names with adb commands but I do not know to get all of this data on one line with semi-colons as separators.
Up to now I haven't found any app giving me this list.
Both solutions are good but maybe a script or command(s) is easier then I can figure out how it works if I ever need more information.
Up to now I have found
adb shell commands but after struggling to get a good version of aapt and to move and execute it on the device.
In the end I keep having problems with the following lines
adb shell /data/local/tmp/aapt-arm-pie d badging $pkg | awk -F: '
$1 == "application-label" {print $2}'
Before I had a problem that aapt could not be executed but now I get
bash: syntax error near unexpected token `|'
So If somebody can tell me how to fix this it is already ok but other solutions are very welcome
I could get a similar script (the oneliner in the text) running where I should get application name and version but the ouput was garbled the first part of all the different lines seems to be written only with a line feed, anyway the first part of the output was all overwritten on the same line.
For information , my phone is on Android 11, I saw somewhere that you do not have execution rights on /data/local/tmp but for me that did not seem to be the problem.

AFAIK neither awk nor print are applets that come with Android's Toybox.
Also aapt2 isn't needed to gather the infos you want to get, they all can be got by means of pure ADB commands, IMO.
BTW:
A directory as /data/local/tmp never can have execution rights, because it's merely a container where files can get stored.
As with Android only binaries ( as aapt2 ) and scripts can get executed and this only if they are allowed to get executed.