Database Info

Toolchain / Wakeup Lag

Various developers have noticed differences in the amount of lag between different toolchains. Dkcldark reported having a toolchain that nearly eliminates this problem, and was able to share with me how he built it using ct-ng. In essence, the versions and options selected in the arm-iphone sample config for ct-ng produce a toolchain that seems to give much better results. A binary package for linux-x86 is available here. I find that my device is typically ready to use after sleep in about one second, running a kernel built with this toolchain and my fast-stepup and OC modifications, and this is similar to the worst-case behavior for stock kernels. Hopefully this will be helpful to others who want to build kernels - I'm going to try other gcc versions with ct-ng without changing any other options and see what results I have with them.
Results (linked to a toolchain download when a kernel has been built successfully):
binutils 2.19.1, gcc 4.3.4, newlib: about 1s lag on wakeup
binutils 2.19.1, gcc 4.4.3, newlib: kernel panic at UI startup

Good work Unhelpful! Glad you decided to share this with everyone rather than keeping it to yourself and ignoring people.

Excellent work! Thank you SO much for sharing . I know JAC is already making a kernel using this toolchain for the Vibrant. Your work is MUCH appreciated

Thanks..
Honestly, friend... Thank you. I wanted to start looking at kernel sources and trying to build them or at least follow the changes.. I really appreciate your openness with what you've found.

Good find.

Optimize Options using the GNU Compiler Collection

I haven't found a good source of discussion on this topic but has anyone experimented with gcc optimization flags when compiling the kernel or a ROM? I dug around and found sparse information on the subject and rarely is Android specifically mentioned. Just curious if anyone has 2 cents on the subject.
I've been trying to figure out how to implement using these flags in kernel compilation but I'm confused if you're supposed to put these in your makefile (both main one and the arch\arm\... makefile?)
is ffast-math preferred over some of the Linaro blog post flags ("We’ve switched from a base of -O2 -fno-strict-aliasing to -O3 -fmodulo-sched -fmodulo-sched-allow-regmoves.")
I've tried sifting through Linaro git repos to see anyone actually using any flags they mention but have yet to find anyone using them yet they say they do somewhere.
GCC Optimization Flags Reference http://gcc.gnu.org/onlinedocs/gcc-4.0.4/gcc/Optimize-Options.html
Some background information on the Linaro blogs about -o3 make flag
Compiler flags used to speed up Linaro Android 2011.10, and future optimizations
People trying out the latest Linaro Android builds may notice they’re faster than the older versions. One of the reasons for this is that we’re using a new set of compiler flags for this build.
We’ve switched from a base of -O2 -fno-strict-aliasing to -O3 -fmodulo-sched -fmodulo-sched-allow-regmoves.
To optimize application startup, we’ve also switched the linker hash style to GNU (ld --hash-style=gnu), and patched the Android dynamic linker to deal with those hashes.
Getting rid of -fno-strict-aliasing enables rather efficient additional optimizations – but requires that the code being compiled follows some rules traditionally not enforced by compilers.
Given the traditional lack of enforcement, there’s lots of code out there (including, unfortunately, in the AOSP codebase) that violates those rules.
Fortunately, gcc can help us detect those violations: With -fstrict-aliasing -Werror=strict-aliasing, most aliasing violations can be turned into built time errors instead of random crashes at runtime. This allowed us to detect many aliasing violations in the code, and fix them (where doable with reasonable effort), or work around them by enabling -fno-strict-aliasing just for a particular subdirectory containing code not ready for dropping it.
-O3 enables further optimizations not yet enabled at -O2, including -finline-functions, -funswitch-loops, -fpredictive-commoning, -fgcse-after-reload, -ftree-vectorize and -fipa-cp-clone.
While some of them are probably not ideal (e.g. -finline-functions tends to increase code size, thereby also increasing memory usage and, in a bad case, reducing cache efficiency), overall -O3 has shown to increase performance.
-fmodulo-sched and -fmodulo-sched-allow-regmoves are fairly new optimizations not currently automatically enabled at any -O level. These options improve loop scheduling – more information can be found here.
We optimized further by adding link-time optimizations in some relevant libraries, and by using -ffast-math in selected parts of the code. -ffast-math is a bit dangerous because it can cause math functions to return incorrect values by ignoring parts of the ISO and IEEE rules for math functions (parts that make optimizations harder, or that simply require additional checks that slow down things considerably), so it should usually not be used for an entire build – but enabling it for parts of the code verified to not rely on exact ISO/IEEE rules can produce quite a speedup.
We also added the option to specify extra optimizations in a board specific config – so now our Panda builds are optimized specifically for Cortex-A9 CPUs while the iMX53 builds optimize for Cortex-A8 instead of relying on the least common denominator.
We also experimented with Graphite related optimizations, such as -fgraphite-identity, -floop-block, -floop-interchage, -floop-strip-mine, -ftree-loop-distribution and -ftree-loop-linear – those optimizations can rearrange loops to allow further optimizations. We’ve turned them back off for the release because they introduced some stability problems, and the benefit seemed minimal.
However, chances are they will come back in a future build. They can help make more efficient use of multi-core CPUs (with the addition of -ftree-parallelize-loops) once the compiler knows how to to threading (currently, Android is built using a generic arm-eabi targeted compiler that has no knowledge of the underlying OS).
Other possible future improvements – though probably not as efficient as the ones already made, or the switch to -ftree-parallelize-loops for multi-core boards – include seeing what effect a switch between ARM and Thumb2 instructions has on performance (enabling the right mode in the right modules), identifying places where the increased code size of -O3 actually hurts, and add the likes of -fno-inline-functions there, identifying further performance critical parts that can handle -ffast-math, fixing the aliasing violations we’ve worked around this time, and – of course – switching to gcc 4.7 when it becomes usable.
​

make -o3 -o -fmodulo-sched -o -fmodulo-sched-allow-regmoves -Wl,--hash-style=gnu -Werror=strict-aliasing ARCH=arm CROSS_COMPILE=~/(toolchain path)/bin/arm-linux-gnueabihf- zImage modules -j 4
is this how you'd do a make according to the instructions on the linaro blog? This is the only way I could do it for it to actually run something but it doesn't tell me that it's passing these flags

[COMMIT] [AOSP] JustArchi's ArchiDroid Optimizations V4.1 - Unleash the power!

Hello dear developers.
I'd like to share with you effect of nearly 300 hours spent on trying to optimize Android and push it to the limits.
In general. You should be already experienced in setting up your buildbox, using git, building AOSP/CyanogenMod/OmniROM from source and cherry-picking things from review/gerrit. Solving git conflicts would also be nice. If you don't know how to build your own ROM from source, this is not a something you can apply to your ROM. Also, as you probably noticed, this is not a something you can apply to already prebuilt ROM (stocks), as these optimizations are applied during compilation, so only AOSP roms, self-compiled from source may use this masterpiece.
So, what is it about? As we know, Android contains a bunch of low-level C/C++ code, which is compiled and acts as a backend for our java's frontend and android apps. Unfortunately, Google didn't put their best at focusing on optimization, so as a result we're using the same old flags set back in 2006 for Android Donut or anything which existed back then. As you guess, in 2006 we didn't have as powerful devices as now, we had to sacrifice performance for smaller code size, to fit to our little devices and run well on very low amount of memory. However, this is no longer a case, and by using newest compilers and properly setting flags, we can achieve something great.
You probably may heard of some developers claiming using of "O3 Flags" in their ROMs. Well, while this may be true, they've applied only to low-level ARM code, mostly used during kernel compilation. Additionally it overwrites O2 flag, which is already fast, so as you may guess, this is more likely a placebo effect and disappears right after you change the kernel. Take a look at the most cherry-picked "O3 Flags commit". You see big "-Os" in "TARGET_thumb_CFLAGS"? This is what I'm talking about.
However, the commit I'm about to present you is not a placebo effect, as it applies flags to everything what is compiled, and mostly important - target THUMB, about 90% of an Android.
Now I'll tell you some facts. We have three interesting optimization levels. Os, O2, O3. O2 enables all optimizations that do not involve a space-speed tradeoff. Os is similar to O2, but it disables all flags that increase code size. It also performs further optimizations to reduce code size to the minimum. O3 enables all O2 optimizations and some extra optimizations that like to increase code size and may, or may not, increase performance. If you want to ask if there's something more like O4, there is - Ofast, however it breaks IEEE standard and doesn't work with Android, as i.e. sqlite3 is not compatible with Ofast's -ffast-math flag. So no go for us.
Now here comes the fun part. Android by default is compiled with O2 flag for target ARM (about 10% of Android, mostly low-level parts) and Os flag for target THUMB (about 90% of Android, nearly everything apart from low-level parfts). Some guys think that Os is better than O2 or O3 because smaller code size is faster due to better fitting in cpu cache. Unfortunately, I proven that it is a myth. Os was good back in 2006, as only with this flag Google was able to compile Dalvik and it's virtual machine while keeping good amount of free memory and space on eMMC cards. As or now, we have plenty of space, plenty of ram, plenty of CPU power and still good old Os flag for 90% of Android.
I've made countless tests to find out what is the most efficient in terms of GCC optimization, two selected tests I am about to present you right now.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
As you may noticed, I compiled whetstone.c benchmark using three different optimization flags - Os, O2 and O3. I set CPU to performance, maximum frequency, and I repeated each test additional two times, just to make sure that Android doesn't lie to me. Source code of this test is available here and you may download it, compile for our beloved Android and try yourself. As you can see O3 > O2 >> Os, Os performs about 2.5x times worse than O2, and about 3.0x times worse than O3.
But, of course. Android is not a freaking benchmark, it's operating system. We can't tell if things are getting better or worse according to a simple benchmark. I kept that in mind and provided community with JustArchi's Mysterious Builds for test. I gave both mysterious builds and didn't tell my users what is the mysterious change. Both builds have been compiled with the same toolchain, same version, same commits. The one and only mysterious change was the fact that every component compiled as target thumb (major portion of an android) has been optimized for speed (O3) in build #1 (experimental), and optimized for size (Os) in build #2 (normal behaviour). Check poll yourself, 9 votes on build 1 in terms of performance, and 1 vote on build 2. I decided that this and benchmark is enough to tell that O2/O3 for target thumb is something that we want.
Now it doesn't matter that match if you wish to use O2 or O3, but here is some comparison:
1. Kernel compiled with O2 has 4902 KB, with O3 4944 KB, so O3 is 42 KB bigger.
2. ROM compiled with O3 is 3 MB larger than O2 after zip compression. Fast overview: 97 binaries in /system/bin and 2 binaries in /system/xbin + 283 libraries in /system/lib and other files, about 400 files in total. 3 MB / 400 = 7,5 KB per file size increase.
3. It's unlikely that code working properly with O2 level might break on O3 level, most issues are on the Os <-> O2 part.
4. If it doesn't cause any issues, and speeds up a binary by a little bit, why not use it?
5. The only real reason to not use O3 is potential higher memory usage due to oversized binaries.
In general, I doubt that this extra chunk of code may cause any significant memory usage or slower performance. I suggest to use O3 if it doesn't cause any issues to you compared to O2, but older devices may use O2 purely for saving on code size, similar way Google did it back in 2006 using Os flag.
[SIZE="+1"]Now let's get down to business[/SIZE].
Here is a list of important improvements:
- Optimized for speed yet more all instructions - ARM and THUMB (-O3)
- Optimized for speed also parts which are compiled with Clang (-O3)
- Turned off all debugging code (lack of -g)
- Eliminated redundant loads that come after stores to the same memory location, both partial and full redundancies (-fgcse-las)
- Ran a store motion pass after global common subexpression elimination. This pass attempts to move stores out of loops (-fgcse-sm)
- Enabled the identity transformation for graphite. For every SCoP we generate the polyhedral representation and transform it back to gimple. We can then check the costs or benefits of the GIMPLE -> GRAPHITE -> GIMPLE transformation. Some minimal optimizations are also performed by the code generator ISL, like index splitting and dead code elimination in loops (-fgraphite -fgraphite-identity)
- Performed interprocedural pointer analysis and interprocedural modification and reference analysis (-fipa-pta)
- Performed induction variable optimizations (strength reduction, induction variable merging and induction variable elimination) on trees (-fivopts)
- Didn't keep the frame pointer in a register for functions that don't need one. This avoids the instructions to save, set up and restore frame pointers; it also makes an extra register available in many functions (-fomit-frame-pointer)
- Attempted to avoid false dependencies in scheduled code by making use of registers left over after register allocation. This optimization most benefits processors with lots of registers (-frename-registers)
- Tried to reduce the number of symbolic address calculations by using shared “anchor” symbols to address nearby objects. This transformation can help to reduce the number of GOT entries and GOT accesses on some targets (-fsection-anchors)
- Performed tail duplication to enlarge superblock size. This transformation simplifies the control flow of the function allowing other optimizations to do a better job (-ftracer)
- Performed loop invariant motion on trees. It also moved operands of conditions that are invariant out of the loop, so that we can use just trivial invariantness analysis in loop unswitching. The pass also includes store motion (-ftree-loop-im)
- Created a canonical counter for number of iterations in loops for which determining number of iterations requires complicated analysis. Later optimizations then may determine the number easily (-ftree-loop-ivcanon)
- Assumed that loop indices do not overflow, and that loops with nontrivial exit condition are not infinite. This enables a wider range of loop optimizations even if the loop optimizer itself cannot prove that these assumptions are valid (-funsafe-loop-optimizations)
- Moved branches with loop invariant conditions out of the loop (-funswitch-loops)
- Constructed webs as commonly used for register allocation purposes and assigned each web individual pseudo register. This allows the register allocation pass to operate on pseudos directly, but also strengthens several other optimization passes, such as CSE, loop optimizer and trivial dead code remover (-fweb)
- Sorted the common symbols by alignment in descending order. This is to prevent gaps between symbols due to alignment constraints (-Wl,--sort-common)
Click to expand...
Click to collapse
Sound cool, doesn't it? Head over to my ArchiDroid project and see yourself how people react after switching to my ROM. Take a look at just one small example, or another one . No bullsh*t guys, this is not a placebo.
However, please read my commit carefully before you decide to cherry-pick it. You must understand that Google's flags weren't touched since 7 years and nobody can assure you that they will work properly for your ROM and your device. You may experiment with them a bit to find out if they're not causing conflicts or other issues.
I can assure you that my ArchiDroid based on CM compiles fine with suggested steps written in the commit itself. Just don't forget to clean ccache (rm -rf /home/youruser/.ccache or rm -rf /root/.ccache) and make clean/clobber.
You can use, modify and share my commit anyway you want, just please keep proper credits in changelogs and in the repo itself. If you feel generous, you may also buy me a coke for massive amount of hours put into those experiments.
Now go ahead and show your users how things should be done .
Cherry-picking time!
Android "Lollipop" (5.1.1 & 5.0.2 tested)
JustArchi's ArchiDroid Optimizations V4.1 for CyanogenMod (latest)
A set of commits you may want to pick to fix O3-related issues:
external_bluetooth_bluedroid | hardware_qcom_display | libcore | frameworks_av #1 | frameworks_av #2
Older entries are provided for reference only. I suggest using only latest commit above.
Android "Lollipop" (5.1.1 & 5.0.2 tested)
JustArchi's ArchiDroid Optimizations V4 for CyanogenMod
Android "Kitkat" 4.4.4:
JustArchi's ArchiDroid Optimizations V3 for CyanogenMod
JustArchi's ArchiDroid Optimizations V3 for OmniROM
JustArchi's ArchiDroid Optimizations V2
JustArchi's ArchiDroid Optimizations V1
AFTER applying above commit and AFTER EVERY CHANGE regarding flags, ALWAYS make clean/clobber AND empty ccache (rm -rf ~/.ccache)
Q: How to properly change toolchains used in local manifest?
Open from your source rootdir .repo/local_manifests/roomservice.xml (or create one). Here is a sample manifest that replaces default 4.8 toolchain (both eabi and androideabi) with 4.8 SaberMod and 4.9 ArchiToolchain:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remove-project name="platform/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8" />
<project name="ArchiDroid/Toolchain" path="prebuilts/gcc/linux-x86/arm/arm-eabi-4.8" remote="github" revision="architoolchain-5.2-arm-linux-gnueabihf" />
<remove-project name="platform/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8" />
<project name="ArchiDroid/Toolchain" path="prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8" remote="github" revision="uber-4.9-arm-linux-androideabi" />
</manifest>
This is only an example, you should use the toolchains that suit you best. My ArchiDroid/Toolchain github repo is a good start to test various different toolchains and decide which one you like the most, or which one causes the least problems for you. I do not suggest any other magic tricks to include custom toolchains, putting your selected one in proper path is enough, avoid magic android_build modifications.
[size=+1]Troubleshooting[/size]
Q: Compiler errror:
Code:
(...)/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/../libexec/gcc/arm-linux-androideabi/4.8.x-sabermod/cc1: error while loading shared libraries: libcloog-isl.so.4: cannot open shared object file: No such file or directory
This error can be fixed by installing missing library. libcloog-isl.so.4 is provided by libcloog-isl4 package, so on debian-like OSes, you should be able to fix it with:
Code:
apt-get install libcloog-isl4
Q: Compiler errror:
Code:
(...)/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/../libexec/gcc/arm-linux-androideabi/4.8.x-sabermod/cc1: error while loading shared libraries: libisl.so.13: cannot open shared object file: No such file or directory
This error is very similar to above, but considers other shared library. libisl.so.13 is provided by libisl13 package. Now the problem is that this package is in testing/sid, so we'll need to install it from there.
Add to your /etc/apt/sources.list following entries:
Code:
deb http://ftp.debian.org/debian testing main contrib non-free
deb-src http://ftp.debian.org/debian testing main contrib non-free
Then apt-get update && apt-get install libisl13.
Issues below are for older commits and should be used for reference only
Kitkat THUMB O2+ errors?
These are the most common issues.
* Change -O3 flag from TARGET_thumb_CFLAGS back to -Os, make clean/clobber, empty ccache and try again. This fixes most of the issues.
* RIL problems for for the Exynos 4210 family? Add -fno-tree-vectorize to TARGET_thumb_CFLAGS.
* Broken exFAT -> https://github.com/JustArchi/android_external_fuse/commit/78ebbc4404de260862dca5f0454bffccee650e0d
Errors caused by toolchain?
1. Try Google's GCC 4.8 if you used Linaro 4.8 or SaberMod 4.8
2. Fallback to Google's GCC 4.7 if above didn't help (change TARGET_GCC_VERSION back to 4.7)
Errors caused by GCC 4.8+?
* ART Fix (bootloop) -> https://github.com/JustArchi/android_art/commit/71a0ca3057cc3865bd8e41dcb94443998d028407
* Not booting kernel -> https://github.com/JustArchi/androi...mmit/9262707f4ea471acf40baa43ffe4bfb3cff64de9 and https://github.com/JustArchi/androi...mmit/41a70abcdad746d9415f3ee40f90528feb0c9bdd
Errors caused by GCC 4.9+?
* Graphical glitches in PlayStore -> https://github.com/JustArchi/android_external_webp/commit/36c6201fbb108d6b757f860e2cd57f3982191662
Errors caused by Linaro?
* error: unknown CPU architecture -> https://github.com/JustArchi/androi...mmit/5e5e158a7c147725beae1eeb6785174baacecb03 (Keep in mind that this is a sample fix for smdk4412 kernel, you may need to use similar solution in your own case. Also, this error happens only with Linaro toolchain, doesn't happen with Google's GCC)
Other errors?
* error: undefined reference to 'memmove' -> https://github.com/XperiaSTE/androi...mmit/679a4e571ef77f47892a785e852d8219c1e6807a
[size=+1]Credits[/size]
@IAmTheOneTheyCallNeo - For inspiration and first steps
@metalspring - For some nice commits
@sparksco - For SaberMod, some nice commits and support for the optimization idea

Bravo! I see flags in here that I am yet to try. Thanks for all your work and dedication to this effort.
Going to do a comparison this week against my "neofy" initiative
Thanks!
-Neo
Forum Moderator

This is amazing man!
Enviado desde mi Moto G mediante Tapatalk

Damn... Wish I had an international gs3. I'd be on SlimKat, find your thread and be like:
I'm gonna be 20 years old in about an hour... Should I ask?

Great work :highfive:
I saw you had already a custom 4.10 linaro. If I use it, do I need to change anything on your commit?

Thanks
Sent from my SAMSUNG-SGH-I747 using Tapatalk

_MarcoMarinho_ said:
Great work :highfive:
I saw you had already a custom 4.10 linaro. If I use it, do I need to change anything on your commit?
Click to expand...
Click to collapse
It won't boot, stl port has segfaults when using GCC 4.9+. If you want to use it, sure, change TARGET_GCC_VERSION and include proper toolchain.

JustArchi said:
It won't boot, stl port has segfaults when using GCC 4.9+. If you want to use it, sure, change TARGET_GCC_VERSION and include proper toolchain.
Click to expand...
Click to collapse
Is this the best toolchain [stability/performance] to compile a ROM?
https://github.com/JustArchi/Linaro/tree/4.8-androideabi

_MarcoMarinho_ said:
Is this the best toolchain [stability/performance] to compile a ROM?
https://github.com/JustArchi/Linaro/tree/4.8-androideabi
Click to expand...
Click to collapse
Yes.

JustArchi said:
Yes.
Click to expand...
Click to collapse
Just one last question. Do I need to configure envsetup.sh to the custom toolchain directory?

_MarcoMarinho_ said:
Just one last question. Do I need to configure envsetup.sh to the custom toolchain directory?
Click to expand...
Click to collapse
No, but you need to add Linaro entries to your local_manifest.
https://github.com/JustArchi/ArchiDroid/blob/2.X-EXPERIMENTAL/__dont_include/roomservice.xml#L3

Thank you @JustArchi
i make a build for Slimkat (nexus 5)
Everything is fine but just a noobish question,
I have to enable "TARGET_USE_O3=true" in BoardConfig.mk ?

micr0g said:
Thank you @JustArchi
i make a build for Slimkat (nexus 5)
Everything is fine but just a noobish question,
I have to enable "TARGET_USE_O3=true" in BoardConfig.mk ?
Click to expand...
Click to collapse
No, you shouldn't have any commits apart from mine. It already specifies O3 for everything.

Can I use a 4.7 toolchain with the ArchiDroid optimizations? I've tried compiling my kernel with different 4.8 toolchains before, and it's always resulted in boot loops.

Codename13 said:
Can I use a 4.7 toolchain with the ArchiDroid optimizations? I've tried compiling my kernel with different 4.8 toolchains before, and it's always resulted in boot loops.
Click to expand...
Click to collapse
Actually you can. Just change TARGET_GCC_VERSION back to 4.7.

Well done mate, I'll try contributing from my side to this project

great work archi !! also after porting a fully working 4.4.2 touchwiz to i9300 is it possible to make aosp for i9300 more stable now?

@JustArchi which flags could be used for kernel compiling? And where should I put it in in the Makefile? I don't compile the complete ROM because I have to low machine for that, but I am developing a custom kernel for my Xperia Z. BTW Fajnie trafić na innych Polaków

dragonnn said:
[B @JustArchi[/B] which flags could be used for kernel compiling? And where should I put it in in the Makefile? I don't compile the complete ROM because I have to low machine for that, but I am developing a custom kernel for my Xperia Z. BTW Fajnie trafić na innych Polaków
Click to expand...
Click to collapse
Ktoś tu mówił o Polakach?
Świetna robota @JustArchi :highfive:

dragonnn said:
@JustArchi which flags could be used for kernel compiling? And where should I put it in in the Makefile? I don't compile the complete ROM because I have to low machine for that, but I am developing a custom kernel for my Xperia Z. BTW Fajnie trafić na innych Polaków
Click to expand...
Click to collapse
If you have whole ROM tree then you cherry-pick this commit, lunch your target and make bootimage. This is enough.
If you have standalone kernel, take a look at main Makefile .

[APP][4.0+][Snapdragon][CAF]Code Aurora SWE Browser (Unofficial)(Stock)(Adblock)

This is the fully stock and unchanged Code Aurora Snapdragon Optimized Browser version m60. Everything is open source and I will also answer any questions how I built this APK. I am just a lone wolf with a Ubuntu 14.04 machine and Linux has been my hobby. I am NOT a full fledged developer but I know enough to build source and I am happy to share this here at XDA and help out.
I will gladly have this thread locked if anyone can point me to a vanilla and unchanged CAF Snapdragon Optimized Browser. Why Code Aurora does not build and make available this awesome browser is beyond me. I searched https://codeaurora.org and they do not offer their SWE Browser APK! I've searched here at XDA and elsewhere and most of those offering CAF browsers are not sharing their source code and/or are making changes or adding patches to the original source.
Before I begin, I'd first like to thank Code Aurora for their source code and this guide by ZeroX7C and gwinlord for his guide. I did NONE of the hard work at all. Those who I cited deserve all the thanks for their work. All I did was piece it together, strip out any changes that are non-CAF, build it, and offer it here. You may not donate any money to me as I feel I have done nothing to deserve it.
Just as ZeroX7C says, the documentation on how to build this browser is incomplete. It took me several hours and many curse words to finally get this browser to build. ZeroX7C's guide is also incomplete and gwinlords guide makes changes and adds patches to the original source so here I will help to complete the guide for an original unmodified CAF browser that ZeroX7C started.
Below is a guide to build your own Code Aurora Snapdragon Optimized Browser version m60:
Requirements:
Ubuntu 64 bit, preferabely 12.04 or 14.04
8GB RAM, 8GB Swap, 100GB Free Space
1) First, run
Code:
sudo apt-get update
2) Set up your build machine by following the Code Aurora Build Machine Requirements Guide. There will be an error about depot_tools using a deprecated URL. Allow the setup to install newer versions of depot_tools from Google. Do not install the Oracle JDK. Code Aurora uses OpenJDK 8 in build m60 and not Oracles JDK. Run the following code if your machine does not have OpenJDK 8 installed.
Code:
sudo apt-get -y install openjdk-8-jre openjdk-8-jdk
3) Create the .gclient file in the same directory where you git cloned depot_tools and build dependencies. Add the following code to .gclient
Code:
solutions = [
{ "name" : "src",
"url" : "git://codeaurora.org/quic/chrome4sdp/chromium/[email protected]/remotes/origin/m60",
"deps_file" : "DEPS",
"managed" : False,
"safesync_url": "",
},
]
target_os = ["android"]
4) Change directory to src, run
Code:
cd src
git clean -f -d
git reset --hard HEAD
GYP_CHROMIUM_NO_ACTION=1 gclient sync -v -r refs/remotes/origin/m58
./build/install-build-deps-android.sh
. build/android/envsetup.sh
GYP_CHROMIUM_NO_ACTION=1 gclient runhooks -v
GYP_CHROMIUM_NO_ACTION=1 gn gen out/Default --args='target_os="android" symbol_level=0 is_debug=false optimize_for_size=false remove_webcore_debug_symbols=true proprietary_codecs=true'
5) Add adblocking. Credit goes to AdAway, Adblockplus, ublock origin, and adguard for their recommended adblocking host files and @ArslanTheGreat for all the help.
Code:
nano chrome/java/res/raw/web_refiner_conf
Modify the code in web_refiner_conf to,
Code:
{
"version": "1.0",
"update": {},
"subscriptions": [
{
"name" : "Adaway",
"category" : "Hosts",
"format" : "WebRefiner 1.0",
"url" : "https://adaway.org/hosts.txt",
"expiration" : "5 days"
},
{
"name" : "hpHosts’ Ad and tracking servers",
"category" : "Hosts",
"format" : "WebRefiner 1.0",
"url" : "https://hosts-file.net/ad_servers.txt",
"expiration" : "5 days"
},
{
"name" : "Peter Lowe’s hosts file",
"category" : "Hosts",
"format" : "WebRefiner 1.0",
"url" : "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext",
"expiration" : "5 days"
}
]
}
6) To apply all the modifications all at once, download the attached patch and apply using
Code:
git apply swebld58ver0004.patch
7) (a.) Optional: If you want to create logs of the build for error checking, then
Code:
mkdir logs
nano build/makeswebrowser.sh
and add the following code
Code:
#1/bin/bash
set -e
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>logs/swebuild.log 2>&1
time ninja -C out/Default swe_browser_apk
then run,
Code:
chmod +x build/makeswebrowser.sh
build/makeswebrowser.sh
Logs will be in src/logs/swebuild.log
(b.) OR, if you just want to skip making logs, simply run
Code:
ninja -C out/Default swe_browser_apk

*Changelog*
Why build another CAF browser?
I am building this because I wanted a nicely developed and secure open source browser for my own personal use and for my family. I appreciate the offerings on f-droid and the open source chromium build, but I wanted the snapdragon optimized version and there is none to be found that are entirely open source and not making changes to the original Code Aurora source code! I am also somewhat concerned that some of the browsers on F-Droid may possibly have security vulnerabilities despite their respective developers best intentions.
Is this browser fast?
Yes! After optimizations posted in the build guide, the SWE Browser is faster than Chrome in every which way.
Will you be keeping this browser up to date?
Well, I do plan on keeping this up to date as long as there is no official Code Aurora browser being published. Would someone please complain to Code Aurora and ask why they are not building it and publishing it? My Ubuntu 14.04 machine runs all day and while I'm gone at work, why not have it build this browser and other things I'd like to have?
Will you add new features to this browser?
No, I will not. I plan on keeping these builds completely stock as it would come from Code Aurora themselves. There are however hidden features that may be enabled. I am looking into it.
I have a bug, can you fix it?
No I will not fix it. This is a beta build, m55. It probably does have some bugs! If there is a problem with the build, it's up to Code Aurora to fix and not me.
Hey, want to try a new compiler to optimize this build?
Sure! Or if you happen to know of other ways we can compile SWE_Browser and have suggestions for other compilers or libraries we can use to tune this build, please post them here! Let's gather a master list of ways we can all compile this APK ourselves!
*edit 12/23/2016*
After careful consideration, I decided not to try new compilers. I decided I am going to follow Code Aurora's build environment exactly as recommended on their own website. After all, I think they probably know what's best for their own software they created.
Can I post benchmarks in this thread?
Sure, please do! I ran several benchmarks and they are all over the place. The SWE Browser is faster than Chrome but runs about the same speed as other CAF based browsers.
If you have any suggestions at all about the browser or the build environment, please post!
--Helpful References--
https://github.com/scweeny/CAF/
https://forum.xda-developers.com/general/general/guide-building-chromium-snapdragon-t3255475
https://github.com/gwindlord/caf_chromium_build
https://source.codeaurora.org/quic/chrome4sdp/
Import Bookmarks from Chrome
Fix AMP pages

Thank you for the guide and i'm looking forward to the complete version, i been missing with this for pass few day and i still cant figure it out, also could you include how to customize it to add google sync and patches and stuff.

hello,
does it support cm 11?

TheUser123 said:
Thank you for the guide and i'm looking forward to the complete version, i been missing with this for pass few day and i still cant figure it out, also could you include how to customize it to add google sync and patches and stuff.
Click to expand...
Click to collapse
I just now updated the guide, see above. I am sorry, I don't plan on adding any patches at the moment. You can check gwinlords guide linked above. In it, there are several patches that can be added.
arjuntsgowda said:
hello,
does it support cm 11?
Click to expand...
Click to collapse
I'm sorry, it does not. The browser will only run on LP or MM based OS's at this time. I updated the title of this thread to reflect that.

Everyone who uses SWE Browser should always get the latest version. I notice people are still downloading the first build but I encountered several errors during that build. I've gotten much better at setting up the environment since then. I've noticed benchmark improvement on the latest build too, just sayin!

Why is Web Refiner not part of this build? Otherwise the browser is running fine and is pretty fast even compared to other CAF browsers, good job OP.

10goto10 said:
Why is Web Refiner not part of this build? Otherwise the browser is running fine and is pretty fast even compared to other CAF browsers, good job OP.
Click to expand...
Click to collapse
Web refiner must not be part of Code Aurora's m55 build. Is it part of other Code Aurora's builds? The SWE Browser is built exactly as recommended per their own sync and build instructions. I can't even use the Qualcomm LLVM even if I wanted to because m55 uses a newer toolchain.
I agree, the browser is pretty fast especially the last build. This is the only browser I use now. I compiled SWE browser last Saturday and the version number did not increment so I did not upload it in case anyone was wondering.

mattnin said:
Web refiner must not be part of Code Aurora's m55 build. Is it part of other Code Aurora's builds? The SWE Browser is built exactly as recommended per their own sync and build instructions. I can't even use the Qualcomm LLVM even if I wanted to because m55 uses a newer toolchain.
I agree, the browser is pretty fast especially the last build. This is the only browser I use now. I compiled SWE browser last Saturday and the version number did not increment so I did not upload it in case anyone was wondering.
Click to expand...
Click to collapse
Yes, I think. It's included in some custom builds: YuBrowser (m54, customized by the Mokee team) and TugaPower Browser (m54, m55, and m56). I like to browse the web without ads
I'm not a dev myself, but maybe these links can help you:
https://wiki.codeaurora.org/xwiki/bin/Chromium+for+Snapdragon/Customization+for+M42
https://source.codeaurora.org/quic/...5&id=950ba75f9296bfe4318ac39ff966cc2e48512ff5

10goto10 said:
Yes, I think. It's included in some custom builds: YuBrowser (m54, customized by the Mokee team) and TugaPower Browser (m54, m55, and m56). I like to browse the web without ads
I'm not a dev myself, but maybe these links can help you:
https://wiki.codeaurora.org/xwiki/bin/Chromium+for+Snapdragon/Customization+for+M42
https://source.codeaurora.org/quic/...5&id=950ba75f9296bfe4318ac39ff966cc2e48512ff5
Click to expand...
Click to collapse
Interesting. Looks like I should be able to enable this feature by including a proper configuration. Thanks for heads up! Once I have that completed, I'll update the download and the guide. :good:

Thanks again @10goto10 for the heads up on web refiner. It's working perfectly with the default host files recommended by AdAway.
The browser version number is the same as the previous build but I did sync the repository as of 07 Jan. Updated the guide with new build instructions and androidfilehost with the latest build.
*edit*
Dammit, I had a copy paste error in the adblocking conf file. I didn't have a lot of time tonight to work on the SWE Browser and I must have rushed a bit and didn't notice I used the same URL twice. So the last build's web refiner is only using two host files. Even though it is only using two files and not three, it's still doing a good job according to my tests.
Also, in case anyone is wondering, bookmarks are saved after updating manually.
*edit again*
Ok, fixed the guide and updated the pgl.yoyo.org url. Synced the local repo, didn't see any new changes from Code Aurora. Set it to build. I'll update androidfilehost with the build as soon as it's done!

The new build is up, and version number incremented to 55.0.2883.2641472, so get the newest version with web refiner in the top post.

mattnin said:
The new build is up, and version number incremented to 55.0.2883.2641472, so get the newest version with web refiner in the top post.
Click to expand...
Click to collapse
Just wanted to say thanks for doing this. I've had the same thoughts as you for a while now... wondering why CAF doesn't put out an official apk, and wanting to figure out how to build it for myself as well. Looks like you beat me to it haha.
The guide is awesome too. :good:

Can confirm now webrefiner is working as intended! @mattnin you rock!

FanDroid09 said:
Just wanted to say thanks for doing this. I've had the same thoughts as you for a while now... wondering why CAF doesn't put out an official apk, and wanting to figure out how to build it for myself as well. Looks like you beat me to it haha.
The guide is awesome too. :good:
Click to expand...
Click to collapse
You're welcome. Glad I could give back to this great community! I'm still working on the guide, it's not finished! I'll make it easier to follow and understand.
10goto10 said:
Can confirm now webrefiner is working as intended! @mattnin you rock!
Click to expand...
Click to collapse
Holy cow, it blocked 23 advertisements! That's gotta be a record!
Good news everyone. I know why the other CAF builds are larger than mine, and I also know why they are just a little faster than mine. Not anymore...
After taking the tip by @10goto10 and looking into webrefiner, I began to dig into the source code and look into other 'hidden' features of Code Aurora's SWE Browser. I came across the same optimizations that the other CAF browsers are running. I found that the optimizations sacrifice build size for speed. See the code below.
Code:
GYP_CHROMIUM_NO_ACTION=1 gn gen out/Default --args='target_os="android" symbol_level=0 is_debug=false optimize_for_size=false remove_webcore_debug_symbols=true'
Not only that, but there may be some other code aurora features I can enable. I will keep plugging away and see what I can accomplish, but here we are again with a new build. It's the same version as the last but optimized for SPEED! :laugh: run some benchmarks and see for yourself.

Here are benchmark scores using Octane,
Chrome 4792
SWE Browser 5443
SWE Browser smokes Chrome! :good:

Done some JS & DOM benchmarks myself against the latest Samsung Browser and the previous version of this browser.
~14% speed improvement compared to the previous version, awesome! :good:
...and it's even ~5% faster than TugaBrowser M55. YuBrowser M54 is ~8% faster though, but it eats RAM like nobody's business

10goto10 said:
Done some JS & DOM benchmarks myself against the latest Samsung Browser and the previous version of this browser.
~14% speed improvement compared to the previous version, awesome! :good:
...and it's even ~5% faster than TugaBrowser M55. YuBrowser M54 is ~8% faster though, but it eats RAM like nobody's business
Click to expand...
Click to collapse
I'm working on the YuBrowser speed advantage... It could be because YuBrowser is based on stable CAF, not sure at this point but I will get to the bottom of it.

10goto10 said:
Done some JS & DOM benchmarks myself against the latest Samsung Browser and the previous version of this browser.
~14% speed improvement compared to the previous version, awesome! :good:
...and it's even ~5% faster than TugaBrowser M55. YuBrowser M54 is ~8% faster though, but it eats RAM like nobody's business
Click to expand...
Click to collapse
Try the yubrowser comparison now. Maybe try clearing app cache after new install. Now getting 5759 on Octane. New build up, version incremented.

mattnin said:
Try the yubrowser comparison now. Maybe try clearing app cache after new install. Now getting 5759 on Octane. New build up, version incremented.
Click to expand...
Click to collapse
Man, you're a frikkin machine, cranking out these builds like it's nothing
On the left is your latest build, and on the right is the latest version of YuBrowser. (I cleared data and cache, etc.)
Still no luck on the JS performance forefront; YuBrowser M54 is still ~8% faster.
But DOM / Page load+manipulation is faster by ~6% on your (latest) build, and that's the reason why I prefer your browser over YuBrowser. :good:
Also, I can't confirm this right now since I'm on TouchWiz, but on CM13 peak memory usage under the same workload is higher on YuBrowser than.... any other browser I could think of, really. That sucks because other background apps get closed when I do some intensive browsing with it..
Your hypothesis might be correct: JS on M54 is probably more optimized because it's CAF's stable release, and there's probably something about M55 using a newer toolchain.
p.s: I attached a BT keyboard now while running the Speedometer benchmarks, because it likes to open the virtual keyboard from time to time, and that might affect the results. So that's why the benchmark score for that is higher than my previous runs.

[H850,H830,RS988] mk2000 Kernel [STOCK]

THANKS:​
shminer: For his repository
eoghan2t7: For his repository
Mentalmuso: For his repository
savoca: For the invaluable KCAL
USA-RedDragon: For his repository
nathanchance: For his pre-merged sources
TheCrazyLex: And developers of AOSPA, grabbed lots from there
emdroidle: For his work with rooting, and kernel patches
eng.stk: For his blu_active governor, and inspiration
frap129: For his Maple scheduler, and repository
alucard_24: For his governor, and G5 repository
franciscofranco: For his many kernel patches
Lord Boeffla: For his Wakelock Blocker etc
sultanxda: For his many kernel patches
warBeard_actual: For his VM tweaks
neobuddy89: State notifier etc
arter97: Adreno idler etc
flar2: Adreno boost etc
And other's I probably forgot
Previous Versions:​
Features:
1.4 Nougat
Linux 3.18.85
KCAL - Screen panel control.
ElementalX and Zzmoove CPU governors.
Modules signed and secure, as in stock rom.
Disable software CRC checking of SD Card - More performance.
CRC32 and CHACHA20 HW ACC, CRNG based on CHACHA20.
Many patches that improves or fixes things.
Maple I/O scheduler.
LZ4 compression.
RCTD removed.
1.5 Nougat
Linux 3.18.92
KCAL - Screen panel control
Alucard, and blue_active (by eng.stk) Governors
Integrated Virtual Memory tweaks (war_mod) and Blu_active settings into ramdisk
Many work-queues converted to *system_power_efficient_wq*
Patches from CAF mostly, to various subsystems
Support for UAC2 external DAC
Boeffla Wakelock Blocker
Maple I/O scheduler
Adrenoboost
On/Off Fsync
Adreno Idler
1.5.5 Nougat
Linux 3.18.97
KCAL - Screen panel control
blu_active (by eng.stk) cpu governor
Integrated memory tweaks (war_mod), and blu_active settings into ramdisk
Many threads converted to *system_power_efficient_wq*
Toggle USB2 900mA Fast-Charge
Support for UAC2 external DAC
Toggle HiFi mode on/off *
Maple I/O scheduler
Toggle Fsync on/off
Adreno Boost
Adreno Idler
WireGuard
* /sys/kernel/hifi_mode/enable_hifi_mode "1" = enable
- WCD9335 HiFi mode not enabled by default in LG source. This toggle will enable it on the fly.
Download: AndroidFileHost
Anyone is free to include it in their ROM as they like
Compiled with GCC 9.1.1
[2.2] Features:
Linux 3.18.140
SultanXDA's CPU Boost
KCAL - Screen panel control
Backported kernel improvements
Toggle USB2 900mA Fast-Charge
Boeffla Dynamic Sync Control
Boeffla Wake-Lock Blocker
Various patches and fixes
RCTD/Triton disabled
Maple I/O scheduler
DriveDroid support
Modules disabled
WireGuard
K-Lapse
Download: Oreo G5 Folder
The (AnyKernel2) flashable zip will flash kernel as well as /system/lib/modules. If you want to go back you should backup modules as well
Sources:
https://github.com/stendro/msm8996_lge_kernel
LGH830SV_LGH830TN_Android_Oreo_v30c.zip - From http://opensource.lge.com

Questions
Q: Will this work on newer bootloaders?
A: It should. The kernel is based on the most recent of LG sources.
Q: What control app do you suggest using?
A: I suggest using SmartPack, or EX Kernel Manager.
Q: Where can I donate? I want to donate!!
A: I don't accept donations because of two reasons. One: I don't really need it. Two: I like the idea of this being totally free, it adds to the enjoyability of the kernel.
Q: I'm having some problems with kernel not working or booting?
A: If you have downgraded your bootloader (i.e by flashing Asgard kernel first) then that is likely why. My kernel works on new firmware, not exactly sure how far back the compatibility goes.
Q: I flashed this zip in recovery, but it didn't seem to work, I'm still on my old kernel. What can I do?
A: With AnyKernel you shouldn't need to re-flash ROOT after flashing kernel (if already installed, if not, do it before flashing kernel). SuperSu has been known to revert the kernel install if you do so.
Q: OK so I just rooted my phone and installed this, and OMGOSH! so excited, now what do I do - do I tweake-dy freak out huh huh? How do I do it??
A: Put simply, don't do anything. I don't know how things where in the past, but now you don't need to do anything. Not with this kernel anyhow. Such thing are better discussed in a separate thread anyway.

Thanks for your work. Tested your kernel with H850 and Fulmics 4.5. Works well but bluetooth isn't working, it's not possible to activate.
Returned with backup to stock.
Best regards
mkpcxxl

mkpcxxl said:
Works well but bluetooth isn't working, it's not possible to activate.
Click to expand...
Click to collapse
Thank you for your report. How silly of me, I forgot to test bluetooth since I never use that.
I'll see about uploading a fixed version soon.

Version 0.8 is up.

askermk2000 said:
Version 0.8 is up.
Click to expand...
Click to collapse
Flashed and now bluetooth is working.
Thanks again for your work.

I am sorry, but where can I find the source code for this kernel? If it is one of github links, which of them is the one?

dadittoz said:
I am sorry, but where can I find the source code for this kernel? If it is one of github links, which of them is the one?
Click to expand...
Click to collapse
I'm sorry too, but when I started this project, I did it in the only way I know (and what seemed simplest to me) and that is to do stuff manually. I don't have experience with github or any elaborate development tools, heck I hardly know how to do this stuff, but I'm learning more and more.
I can upload the latest source in zip if that is satisfactory.
Sorry if this breaks xda rules, but I'm sure I've seen other projects not having their full, direct sources posted.

askermk2000 said:
I'm sorry too, but when I started this project, I did it in the only way I know (and what seemed simplest to me) and that is to do stuff manually. I don't have experience with github or any elaborate development tools, heck I hardly know how to do this stuff, but I'm learning more and more.
I can upload the latest source in zip if that is satisfactory.
Sorry if this breaks xda rules, but I'm sure I've seen other projects not having their full, direct sources posted.
Click to expand...
Click to collapse
Well, as far as I know GPL requires sources to be published. But it's not that I brag about complying to licenses. I am genuinely interested in changes made. I enjoy reading @eoghan2t7 commits on his kernel tree every few days. As for me, I am not really interested in a single zip Version control systems are awesome and it would be great if your kernel tree could use it.

dadittoz said:
Well, as far as I know GPL requires sources to be published. But it's not that I brag about complying to licenses. I am genuinely interested in changes made. I enjoy reading @eoghan2t7 commits on his kernel tree every few days. As for me, I am not really interested in a single zip Version control systems are awesome and it would be great if your kernel tree could use it.
Click to expand...
Click to collapse
Yes, as I got further into this project I figured it would be better to open a qithub account next time.
So I'll probably do that with the next major source drop from LG.

Flashed in fulmics 4.5, running fine so far. I will report battery life, thanks for your job!
Enviado desde mi LG-H850 mediante Tapatalk

.
battery life is not quite as good

Nick cutter said:
battery life is not quite as good
Click to expand...
Click to collapse
I think you might be right.
May be a problem or conflict with cpu hotplug. I will look into it.

that sounds good but otherwise a great kernel keep it up :good:

Tha.ks for this great kernel. Only two flaws so far: Battery and overheating.

Can you upload your fully kernel source ?
GPL

Nick cutter said:
battery life is not quite as good
Click to expand...
Click to collapse
Hadi99 said:
Tha.ks for this great kernel. Only two flaws so far: Battery and overheating.
Click to expand...
Click to collapse
Hm, don't have this problems, Battery is like stock and temps are normal.
I remember I've had overheating with Asgard kernel.
Regards
mkpcxxl

@walter79
Link added to first post. Bottom.

mkpcxxl said:
Hm, don't have this problems, Battery is like stock and temps are normal.
I remember I've had overheating with Asgard kernel.
Regards
mkpcxxl
Click to expand...
Click to collapse
Temps are alot more stable right now, but battery life is still horrible for me.
Asgard did indeed have overheating in versions 5.x, but since then it had been fixed

Version 1.0
Super excited about this one
Compiled with gnu11, graphite, O3 and other optimizations. Most of them was borrowed from @jahlex D.O.T.S Kernel (Thanks!) - but gnu11 was my idea. I really don't know if is good to do or not, but hey...
It's been tested "adequately" I'd say. But that probably means inadequate in most tongues, however, I like to take few risks and gamble every now and then
Performance seems very much increased, battery as well.
Added Maple i/o
Alucard and Zzmoove (bLE) governors
Fixed small omissions here and there
Maple is default IO (was initially going for FIOPS), since that hooks into "display_state" making it aware of screen on/off.
Zzmoove is default cpu governor. Alucard could not be set as default at build time - kernel would not boot
I've added a zip to fix lg bootup script in /system/etc - it, amongs other things, sets *interactive* governor as default
Hotplugging characteristics can be customized.
May it serve you all well!