Database Info

Patching system.img from the Android SDK build code

Hi,
Does anyone have any idea where in the android source tree system.img is created when running "make sdk"?
I'm working with the emulator, and want 'make sdk' to add in a custom binary to /system/bin in system.img.
So currently the sdk build places the system file system here: out/target/product/generic/system
This gets packaged up into system.img (yaffs2 file system) somewhere, and gets deleted/reset everytime the sdk is built so I cant just drop my binary in there. Strangly though it does work if I overwrite an existing binary, I tried replacing system/bin/ping with my binary and it got packed up into system.img and worked perfectly. It's almost as if there's a whitelist of system/bin files?
Anyone know how I can do this?
Thanks a lot.

CM13 compiling error?

I'm trying to compile CM13 for the OPO, and I keep coming with an error, I've tried to compile 3 times, and each time getting the same error. I wiped ccache and tried make clean and still have had the same results, any ideas?
http://pastebin.com/CvJT8RKZ

These could be the culprit :
Code:
expr: syntax error
stat: illegal option -- c
usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
[...]
/Volumes/CM/android/system/kernel/oneplus/msm8974/scripts/xz_wrap.sh: line 23: exec: xz: not found
The second error could be fixed by properly setting up xz binary (either by installing it or exporting it to the current terminal window). Can you also mention what operating system you're using to compile the ROM ?

F4uzan said:
These could be the culprit :
Code:
expr: syntax error
stat: illegal option -- c
usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
[...]
/Volumes/CM/android/system/kernel/oneplus/msm8974/scripts/xz_wrap.sh: line 23: exec: xz: not found
The second error could be fixed by properly setting up xz binary (either by installing it or exporting it to the current terminal window). Can you also mention what operating system you're using to compile the ROM ?
Click to expand...
Click to collapse
I'm using OS X 10.11.4, and this is my first-ish time creating a ROM (well from source anyway) so I have no idea what that means, any chance you could walk me through it (I'm reasonably competent, just need some general direction).

Vekhez said:
I'm using OS X 10.11.4, and this is my first-ish time creating a ROM (well from source anyway) so I have no idea what that means, any chance you could walk me through it (I'm reasonably competent, just need some general direction).
Click to expand...
Click to collapse
Are you trying to build on OS X, or are you running a Linux VM? Either way, I think you might be missing some packages like the above poster mentioned. Here's a guide I found for setting up your OS X environment: http://forum.xda-developers.com/showthread.php?t=2510898, make sure you grab the required packages for compiling android. Good luck!

ryanmat said:
Are you trying to build on OS X, or are you running a Linux VM? Either way, I think you might be missing some packages like the above poster mentioned. Here's a guide I found for setting up your OS X environment: http://forum.xda-developers.com/showthread.php?t=2510898, make sure you grab the required packages for compiling android. Good luck!
Click to expand...
Click to collapse
Following all those instructions everything has already been done, all the brew things have been installed, everything is up to date etc. So I'm assuming the error will be the same?

Vekhez said:
Following all those instructions everything has already been done, all the brew things have been installed, everything is up to date etc. So I'm assuming the error will be the same?
Click to expand...
Click to collapse
You can always sync the repo again and retry the build. If it doesn't work you can always ask in the thread I linked. The other place to check out is here: https://source.android.com/source/initializing.html
Here's a link to the xz package for Mac you seem to be missing: http://macpkg.sourceforge.net/

[Q] help about SMS/MMS, yaffs extractormmssms.db to xml is not enough

Hello all,
I got the following issue:
I dirtyflashed a custom Marshmallow ROM and now I am stuck with UI issue (theme conflict). Unable to access to home or parameters.
I don't care about all my stuff in it, except SMS/MMS.
I made a nandroid backup of all my FS and reinstall a clean rom (but it is another story).
So, I search how to extract my sms from this nandroid backup.
I found this thread fyi (and many other ways..)
http://forum.xda-developers.com/showthread.php?t=1585957
It is about to convert DB which host SMS/MMS (mmssms.db) to xml.
BUT, mmssms.db is depracted on Android 5+
Anyway, I found all my sms/mms into bugle_db file. Database from Google Messenger app, not OS DB (/data/data/com.google.apps.messaging/databases)
And I tried many restore options (using Titanium and nandroid extract, etc..)
Messenger reload DB from OS. (maybe security conflict too, well..)
Conclusion, I got many way to resolv this issue:
- Find the new DB used (into /system I guess), to restore it
If you know that, I will be glad to know I found nothing about.
- Find or dev a tool to convert sms app store file to xml
i.e. bugle_db file for Google Messenger app to xml for SMS&backup (same as yaffs extractor)
Maybe someone can easiley adapt yaffs extractor to bugle_db structure.
- Go forget about all my sms
hope someone can help me on it

I have a similar issue and this is what I found, I hope it will help you if you still need it or at least someone else that is searching for an answer.
Tim O'Brien has created a tool in python call smstools and here is the git. (github.com/t413/SMS-Tools)
p1ne has added the support for bugle_db to this tool and can be downloaded here (github.com/p1ne/SMS-Tools/archive/master.zip)
I have never used python so I needed to figure things out.
I have Windows 10
First it did not work with python 3.6.1
I did get it working with python 2.7.13,
download and install python 2.7.13 "Windows x86-64 MSI installer" (seemed like the best option)
I set up the PATH for less typing but its not needed, here is a link for how to (howtogeek.com/118594/how-to-edit-your-system-path-for-easy-command-line-access/)
downloaded the above zip from p1ne github and extracted to c:\tsms (doesn't matter the location i just wanted it short)
copied my saved bugle_db file to the same directory c:\tsms
opened cmd or powershell and navigated directory C:\tsms\SMS-Tools-master\
installed smstools by typing
C:\Python27\python.exe setup.py install
I then typed
python C:\tsms\SMS-Tools-master\bin\smstools C:\tsms\bugle_db C:\tsms\out.xml
Since this defaults to an android xml backup I did not add any flags to the end but you could convert the output to several outputs with --type flag. You will need to read the git to see the supported outputs.
If you didn't set your PATH you would type
C:\Python27\python.exe C:\tsms\SMS-Tools-master\bin\smstools C:\tsms\bugle_db C:\tsms\out.xml
Hopefully all went well and you get a message like this.
6708 messages read in 0 seconds from bugle_db using parser smstools.bugle.Bugle
latest text: 2017-04-23 07:36:50 PM US Time Zone to +**************:
Parts arrived
sorting all 6708 texts by date
You should have a new file called out.xml in the C:\tsms directory
Copy this to your android phone and restore with SMS Backup & Restore
Hope this all works out for anyone in need.

o1hitman1o said:
I have a similar issue and this is what I found, I hope it will help you if you still need it or at least someone else that is searching for an answer.
Tim O'Brien has created a tool in python call smstools and here is the git. (github.com/t413/SMS-Tools)
p1ne has added the support for bugle_db to this tool and can be downloaded here (github.com/p1ne/SMS-Tools/archive/master.zip)
I have never used python so I needed to figure things out.
I have Windows 10
First it did not work with python 3.6.1
I did get it working with python 2.7.13,
download and install python 2.7.13 "Windows x86-64 MSI installer" (seemed like the best option)
I set up the PATH for less typing but its not needed, here is a link for how to (howtogeek.com/118594/how-to-edit-your-system-path-for-easy-command-line-access/)
downloaded the above zip from p1ne github and extracted to c:\tsms (doesn't matter the location i just wanted it short)
copied my saved bugle_db file to the same directory c:\tsms
opened cmd or powershell and navigated directory C:\tsms\SMS-Tools-master\
installed smstools by typing
C:\Python27\python.exe setup.py install
I then typed
python C:\tsms\SMS-Tools-master\bin\smstools C:\tsms\bugle_db C:\tsms\out.xml
Since this defaults to an android xml backup I did not add any flags to the end but you could convert the output to several outputs with --type flag. You will need to read the git to see the supported outputs.
If you didn't set your PATH you would type
C:\Python27\python.exe C:\tsms\SMS-Tools-master\bin\smstools C:\tsms\bugle_db C:\tsms\out.xml
Hopefully all went well and you get a message like this.
6708 messages read in 0 seconds from bugle_db using parser smstools.bugle.Bugle
latest text: 2017-04-23 07:36:50 PM US Time Zone to +**************:
Parts arrived
sorting all 6708 texts by date
You should have a new file called out.xml in the C:\tsms directory
Copy this to your android phone and restore with SMS Backup & Restore
Hope this all works out for anyone in need.
Click to expand...
Click to collapse
Thanks! I am a python newer too. It works fine for me.
Btw, I found it needs another module 'unicodecsv', I taken this project 'python-unicodecsv' (github.com/jdunck/python-unicodecsv),and can be downloaded here(https://codeload.github.com/jdunck/python-unicodecsv/zip/master)
Hope it will help someone else.

Thanks
jiangchuanso said:
Thanks! I am a python newer too. It works fine for me.
Btw, I found it needs another module 'unicodecsv', I taken this project 'python-unicodecsv' (github.com/jdunck/python-unicodecsv),and can be downloaded here(https://codeload.github.com/jdunck/python-unicodecsv/zip/master)
Hope it will help someone else.
Click to expand...
Click to collapse
Thanks you just saved my day

I'm reviving this post to say it worked but not at 100%. But when I restore my messages, only recieved messages are saved and restored with bugle_db :/

It worked 100% for me but im using google messenger.

beware that write-ahead logging is enabled, so you need three files. make sure you work on copy because the files disappear/change on access.
Code:
bugle_db
bugle_db-wal
bugle_db-shm
here is how to use it on linux. I have tested on clean Xubuntu 20.04 LTS (from pendrive) so all dependencies should be fulfilled.
Code:
sudo apt update
sudo apt install wget git python2
wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
git clone https://github.com/jdunck/python-unicodecsv
git clone https://github.com/Batambatam/SMS-Tools
sudo python2 get-pip.py
sudo python2 -m pip install ./python-unicodecsv blessings
cd SMS-Tools
sudo python2 setup.py install
cp -v com.google.android.apps.messaging/databases/bugle_db* .
smstools bugle_db messaging.xml

[GUIDE] Extracting proprietary files from OTA updates easily

Hi all
I thought I'd try and write an easy and simple guide on how I extract blobs from vendor updates, cause I couldn't find a guide online myself.
A few things first:
- I will be writing this guide on Windows. You will need to find the Linux/MacOS tools yourself cause this is what I use.
- I use Windows Subsystem for Linux in this guide. It's not needed, but will make your life a lot easier. Follow this guide if you haven't got it installed already.
- This will assume you know how to build a ROM, know what device/vendor trees are, have some slight knowledge of the command line and know how to use basic Git.
First, we need to grab an OTA. I own a MI 8 Lite, so I usually use xiaomi.eu to update my blobs as it's easy to capture the zips. Depends on your device.
Now you've captured your OTA, you need to get the blobs extracted out. I use UR-Tool on Windows 10 but it's completely your choice. I will be proceeding this tutorial with UR-Tool.
Extract your zip into the UR-Tool build environment. Run the tool and choose your zip, extract it, like here: https://i.imgur.com/Y50Y3IZ.png
Okay, so now we have two directories, system and vendor (I'm fairly sure if you're non-treble you'll have one folder which means you can skip this next step.)
Go into system, and remove the symlink named vendor. You can then copy in the vendor folder into system.
In my device tree, I also had /sbin/charge needed in my proprietary-files.txt. I needed to extract my boot.img and put this folder into the system folder. Just check your device tree's needed files.
Copy the entire folder to somewhere easy to find and preferably a path which doesn't contain any spaces. We will be using it later.
Okay, so now we've done all this, we are going to move to the vendor and Git side of things. I will be using WSL from here!
So, now we have our fully ready to go blobs... but what now?
Well we need to prepare our vendor of course!
Like I said above I'll be using WSL from here as I needed "patchelf" for mlipay blobs. You will need something that can execute .sh files and using WSL is probably the easier option. If you're on Linux you don't need to do anything really.
We need 3 things to clone in. I've taken the liberty of stripping down Lineage's vendor to just the critical stuff for extracting blobs. You can find my repository here.
Lets make a new directory. We will clone in our trees like this:
Code:
git clone (device tree link) device/(vendor)/(codename)
git clone (vendor tree link) vendor/(vendor)/(codename)
git clone https://github.com/nysadev/extract_blobs_vendor vendor/lineage
This will create a directory that looks like a ROM, but severely stripped down. The reason we do the folders like this is because the scripts in device tree will point to a structure like this. This will help us avoid script location errors later. Make sure in your device tree that extract-files.sh points to vendor/lineage (should be in the first few lines).
Your root directory now should look like this: https://i.imgur.com/YiqMaDJ.png
Now go to your device tree. This is where all the good stuff happens. You need to know the path here of your extracted blobs from earlier.
Run these two commands:
Code:
chmod +x setup-makefiles.sh
./extract-files.sh (path/to/your/extracted/system)
Hopefully if all goes well you should see this with no "file missing in source errors": https://i.imgur.com/jErpyCv.png
Just to be sure, at the end of when it extracts, run "./extract-files.sh" again, just to prepare your vendor.
Now you're done! You'll hopefully see a few good modified libs, changes to Android.mk and your device-vendor.mk.
You're probably going to want to use git here to upload your changes. Just run:
Code:
git add .
git commit -m "(codename): update to (ota-version)"
git push -f
or something of the sort.
Well done! You're now done! Feel free to reply with any errors you get.
My Telegram is @nysadev; you're maybe better off catching me there.
I hope my guide helped you. Drop a like or something, idek.

It will be very useful

You rock! Love ya

how do u get proprietary-files.txt if u r building from scratch?

wheres UR-tool?

[Guide] Building post 2017 android kernel from source

About this guide
I have been building kernels for many machine types and have been fiddling around with custom android kernels for my own phone but to my great frustration the rules changed quite a lot for my new phone. It seems every kernel after 2017 is A LOT harder to build and needs quite some fiddling around te get it to boot. The documentation is terrible the least on the steps and all howtos are all outdated by now. This guide shares with you the secrets to build yourself a modern post 2017 kernel from source. I only own a op7 at this moment but I am pretty sure the rules apply to all devices.
So why do we need to build our own kernel? Well that's simple. I have scrolled through many custom kernels missing a simple module that I need. In my case binfmt. Perhaps you need a specific device driver compiled into your kernel or you want to hack in/patch in some special functions that are only needed by you. Or you love a certain kernel but need that one little adjustment in the configs that this kernel doesn’t have and so on.
Lets get through it step by step but I do assume you know something about Linux, you know how to use the terminal and that you have some knowledge on building binaries from source. Ready? Ok lets get started.
These steps are required to be followed to be successful:
1. Set up a building environment
2. Obtain the kernel source
3. Download the proper tool-chain
4. Adjust configuration files to match your needs
5. Build the actual kernel image
6. Backup your current boot.img!!!
7. Trow the image into a anykernel zip and flash it.
8. Close your eyes prey and run the kernel.
Prerequisites
You will need:
A fairly fast computer with lets say 4 gigs of RAM and a decent CPU.
A workable linux distribution that you can boot into (I prefer ubuntu bionic at this moment).
The kernel sources for your device.
ROOT!!
Lots of spare time.
A attitude that makes you unstoppable.
1. Setting up the building environment
Ok lets first open a terminal. You will need the terminal a lot. As a matter of fact it will be your friend through this guide so better get used to it. Assuming you are on a ubuntu like distribution lets fetch the important packages first
Code:
$sudo apt install build-essential bizon flex git
This command will install some of the prerequisites you need to do anything. More packages will need to be installed on the way but at least you will be able to type $make
Now lets create a working directory. Go into your home folder and create a working directory which will later on contain your kernel sources and tool chain.
Code:
$cd
$mkdir [COLOR="MediumTurquoise"]name-of-your-working-folder[/COLOR]
2. Obtain the kernel source
This one is tricky as my experience have learned me the kernel sources supplied by my own vendor are usually badly documented and contain a few bugs here and there resulting in either build errors or a unbootable kernel. So usually I grab a custom kernel that has minimal changes to the stock kernel and boots by only using a single kernel file without extra kernel modules attached. The custom kernels usually have the nasty building bugs flawed out and thus your chances for success are higher. I used the exkernel in my case and out of respect for the hard work of flang2 I bought the app that came along.
To get the sources google around and find the sources on for example git and click the download or clone button there and copy the link of the depository.
Then simply clone the depository to your working directory so after changing to your working directory type
Code:
$git clone link-you-just-copied
Git will now automatically clone the entire kernel source into the source sub-directory. You can change the name of this directory to make you life easier.
3. Download the proper tool-chain
This is where I lost many days and weeks of my life. Where do we find the proper tool-chain for our kernel?
You basically need 3 compilers
clang
GCC
device-tree-compiler
There are many versions of clang and GCC and you will need to find the one that matches your kernel and builds for your cpu architecture. I will assume in this guide that your CPU is aarch64 or arm64 (both names are used at the same time).
To obtain clang head over to the following address:
"https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/"
now in the tags and branches you can find the version of android you are looking for. If your kernel is for android 9 then use the android 9 tag. Click on it and then click tgz which will provide you with a huge tarball.
Create a folder in your working directory named toolchain en extract the contents of the tarball into the sub-folder clang.
You can try any subversion of clang there is but I recommend to simply peek into the build.config files located in the root of your kernel source directory which usually specify the exact subversion of clang used.
Time to get GCC which is easier since there is only one version right now.
Head over to
"https://android.googlesource.com/platform/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/"
and use the correct version here as well. In my case I picked the latest build. Then do the same as for clang but name the sub-folder GCC or whatever you please to do.
And then finally get yourself the device-tree-compiler
Head over to
"https://github.com/dgibson/dtc/tree/v1.4.4"
and clone this depository into ~/your-working-directory/toolchain/
and navigate to the subdir you cloned it into
then build by typing
Code:
$make
any missing package errors will need you to install them using
Code:
$sudo apt install missing-package-name
and do make again until the build completes.
If all went well your tool-chain and working directory will look like this:
Code:
/[COLOR="mediumturquoise"]current-working-dir[/COLOR]/[COLOR="mediumturquoise"]kernel-source-dir[/COLOR]/
|
/toolchain/
|
/clang/
/gcc/
/dtc/
4. Adjust configuration files to match your needs
Lets first grab the correct defconfig file and setup the build directory. This defconfig file contains the default kernel configuration parameters which you propably want to adjust.
Head over to ~/working-directory/kernel-source-dir/arch/arm64/configs/
see what you can find there. I found the elementalX_defconfig file which is the file I needed to build the Exkernel from scratch. If you are trying a stock kernel dive a little bit deeper into the vendor folder and see if you can find a good config file there.
Alternatively sometimes the config is hidden on your device itself. Have a look at
/proc/config.gz
the file inside the archive contains a valid defconfig for your device but beware your may need to open it with menuconfig first and save it again to make it workable for the compiler script.
Place the defconfig file you like to use into the configs folder.
(Skip the following step if your are going to build your kernel for the first time because we first want to see it work.)
Now head over to your kernel source dir and type the following command
Code:
$PATH="~/[COLOR="mediumturquoise"]working-dir[/COLOR]/toolchain/clang/[COLOR="mediumturquoise"]clang-r353983c[/COLOR]/bin:~/[COLOR="mediumturquoise"]working-dir[/COLOR]/toolchain/gcc/a[COLOR="mediumturquoise"]arch64-linux-android-4.9[/COLOR]/bin:${PATH}" \
make menuconfig O=out \
ARCH=arm64 \
CC=clang \
CLANG_TRIPLE=aarch64-linux-gnu- \
CROSS_COMPILE=aarch64-linux-android-
now remember to use the correct paths to your tool-chain. In this command I use clang-r353983c but you may use a different version. The same applies for linux-android-4.9.
I would like to thank Nathan Chance for supplying documentation on the clang tool-chain on his github.
If all well you will be presented with a menu in which you can adjust things. First open the defconfig file you wish to use and then save the file after you are done. Now REMEMBER! With this command the menuconfig will save your new file into the /kernel-source-dir/out folder.
(end of the skipped step)
Now it is time to prepare our kernel build using the following command (do this from your kernel source dir) and remember that this command will look in the /kernel-source-dir/arch/arm64/configs/ directory
$make O=out ARCH=arm64 name-of-your-defconfig
If all goes well you will see something like configuration written to .config
Everything is up and ready to go. Your hart will start pounding from this moment on
5. Build the actual kernel image
This is where things get complicated. I hope you have many years to live because this part consumes time, a lot of it.
type
Code:
$PATH="~/[COLOR="mediumturquoise"]working-dir[/COLOR]/toolchain/clang/[COLOR="mediumturquoise"]clang-r353983c[/COLOR]/bin:~/[COLOR="mediumturquoise"]working-dir[/COLOR]/toolchain/gcc/[COLOR="mediumturquoise"]aarch64-linux-android-4.9[/COLOR]/bin:${PATH}" \
make -j$(nproc --all) O=out \
ARCH=arm64 \
CC=clang \
DTC=~/[COLOR="mediumturquoise"]working-directory[/COLOR]/toolchain/dtc/dtc \
CLANG_TRIPLE=aarch64-linux-gnu- \
CROSS_COMPILE=aarch64-linux-android-
ready set GO!
This will depending on your machine take between forever and eternally. You will witness many warnings on the DTC build seems normal these days and a few warnings on the CC part. Most important is that no errors are thrown at you.
If all goes well you will see a normal exit status and you will have a “working” kernel image.
“Error?: Well that happens. Try a different build of clang check your command line. And if all that fails try to find out what is wrong in the source and that means digging through thousands of forum pages until you find out whats wrong. But using the google tools usually goes well.”
Almost there go and check
kernel-source-dir/out/arch/arm64/boot
and there should be a image-dtb or image.gz-dtb file depending on you settings.
That is your kernel image right there. The difference in size between image and image-dtb should not be huge. 10Megs in difference usually means your dtb is not good but trying is the only way to find out if it works.
6. Backup your current boot.img!!!
You know what to do here right? Do not skip this step unless you like bricked devices or want to reflash and lose your data and all that kind of stuff. Not sure what you are doing stop here or backup your entire device including system vendor etc.
7. Trow the image into a anykernel zip and flash it.
Ok something changed in the last few years. Unpacking repacking and booting using fastboot somehow gives me problems. Dm-verity errors and all kind of red screens. Signing the boot image results into new errors. Well this is how I did it.
Get yourself a anykernel zip file. I used the Exkernel.zip because it only contains a kernel image which I like. Open the zip in a good zip tool (I used ark) and replace the image-dtb file with the one you created. Place this new zip you created on a memory stick and then….
Flash it using twrp or any tool of choice.
8. Close your eyes prey and run the kernel.
Two things can happen.
1. Blank screen nothing happens. Only god can help you, repeat all the steps.
2. Your android starts booting. Start crying of joy
Check in your android if this is indeed the kernel you build. If so time to make some adjustments to your configs or happily enjoy your boosted phone.
Now please remember. If you plan to distribute your kernel that you do the correct steps of accrediting the original programmer and trow the source online. If you use a already custom kernel please respect the hard work of the maker and communicate your plans with him/her.