Database Info

[Think Tank] Cache DNS Requests on Android to speedup browsing

I was wondering if there is a method to cache DNS requests? I use CM latest and was browsing around in the system folder, when i came by /system/etc/resolv.conf. Here you find the nameservers used by android to resolve DNS.
This got me thinking, because I use a DNS cache on my Ubuntu Box. Might there be a way to do this on Android, to speed up browsing regularly visited websites? Does anyone have any idea how to use a local DNS cache to speed up browsing?
Here's the deal:
- In /system/etc/resolv.conf you find 4 nameservers (4.2.2.5 - 4.2.2.2)
- Doing a ping test (count = 1000) to those, i find big differences in latency. Average was 800 on my test on 3G (I live in the Netherlands). I found this number rather large.
- On IRC, when asking this question, they mentioned porting dnsmasq to android and change nameserver to localhost
A ported version of dnsmasq is found here: http://code.google.com/p/android-wifi-tether/source/browse/trunk/res/raw/dnsmasq?spec=svn120&r=120
Copied the file to /system/bin, chmod +x the file, and this one works.
Now here's the thing:
- dnsmasq --help mentions a configurationfile: /data/local/dnsmasq.conf. This one have to be made for android
- how to configure dnsmasq so it uses the DNS servers and create a cache, thus speeding up browsing (in my humble belief)
- how to start dnsmasq on booting android (and making sure it stays alive)
And finally, does it make sense to use this kind of method? Like to apply for a think tank to make this work

That does sound like a good idea. For me it isn't just 800ms to find out the DNS but on edge rather something like about 3 seconds! this would seriously speed the whole thing up! However, how are you gonna find the IP adresses for the first time?

Well dnsmasq creates a cache, using nameservers found in the config file, if i am correct. I am looking into the dnsmasq.conf examples to see what workaround i need.. Should be pretty straightforward
resolv.conf should have nameserver 127.0.0.1, and dnsmasq should use configured nameservers like 4.2.2.5 and create a cache. Therefore apps use the dnsmasq as DNS server.
Furthermore, i am trying to find out what nameservers are the fastest. 4.2.2.2-5 are Verizon, while i am in the Netherlands. OpenDNS has a datacenter in Amsterdam, so i figured changing nameservers to OpenDNS would speed things up some more..

Even better would be unbound, though the stated overhead of ~11 MB might be too much for G1/mT3G v1.
Also, do you have to do any tricks to prevent resolv.conf from being overwritten when you switch networks? I know that usually dhclient will rewrite the file, and there's a couple of workarounds, but I have no idea which ones would work on Android.

CM builds already have a dnsmasq binary in /system/bin. It's used for tethering. You could launch it with a different config file if needed (there's a dnsmasq.conf in /system/etc already for tethering). Also, the values in resolv.conf aren't really used by much anymore unless you have stuff linked with uclibc. The resolver in Bionic uses the values of the net.dns* system properties.

cyanogen said:
CM builds already have a dnsmasq binary in /system/bin. It's used for tethering. You could launch it with a different config file if needed (there's a dnsmasq.conf in /system/etc already for tethering). Also, the values in resolv.conf aren't really used by much anymore unless you have stuff linked with uclibc. The resolver in Bionic uses the values of the net.dns* system properties.
Click to expand...
Click to collapse
Thanks for the reaction... You are right indeed, i found the dnsmasq.conf:
Code:
no-resolv
no-poll
server=4.2.2.2
server=4.2.2.3
interface=usb0
dhcp-authoritative
dhcp-range=192.168.77.100,192.168.77.105,12h
dhcp-leasefile=/data/misc/dnsmasq.leases
pid-file=/data/misc/dnsmasq.pid
user=dhcp
no-negcache
What you're saying is that i should create another one, but i am wondering what values i should use..
interface=?
Also, regarding your remark on DNS, how to see the values of net.dns* and how to change them? sysctl -n doesn't show these values, I must be looking in the wrong direction...

Hey,
You don't have to set an interface at all.
The interface delcaretion limit dnsmasq to listen only on a specific interface (for both DHCP and DNS requests).
As far as I know dnsmasq is caching dns by default you can limit the cache size and set not to cache negative queries but it will cache by default so no special settings is needed.
In addition, Dnsmasq is also function as dhcp server and if you don't want it to try removing all lines that declaring on dhcp settings.

True on that one, i want to use dnsmasq mainly on 3G
On my remark about the dns properties, already found out to use getprop and setprop
getprop shows different values on DNS compared to resolv.conf:
[net.dns1.195]: [84.241.226.140]
[net.dns2.195]: [84.241.226.9]
[net.dns1]: [84.241.226.140]
[net.dns2]: [84.241.226.9]

You need to be aware of one thing with regards to the resolv.conf file.
It is NOT USED BY ANDROID.
If you use the command "getprop", you will see several dns entries in there -- none of which matches the static dns servers set in resolv.conf. The resolv.conf is used by TERMINAL BINARIES.

zrubi said:
Hey,
You don't have to set an interface at all.
The interface delcaretion limit dnsmasq to listen only on a specific interface (for both DHCP and DNS requests).
As far as I know dnsmasq is caching dns by default you can limit the cache size and set not to cache negative queries but it will cache by default so no special settings is needed.
In addition, Dnsmasq is also function as dhcp server and if you don't want it to try removing all lines that declaring on dhcp settings.
Click to expand...
Click to collapse
might as well restrict it to the local interface
Code:
interface=lo
you can then turn off dhcp with
Code:
no-dhcp-interface=lo

I use OpenDNS on my home network which is claimed to be better than ISP dns servers. When you signup/setup you are required to input their values for dns servers, perhaps you set these values from your phone?

Not sure if Android supports this... but the easiest and least resource intensive way to do this would be to simply add your commonly used domains to the hosts file. I believe some ad blocking software used to use this to block ads.
Aside from that... Bind can be run as a caching nameserver and would probably work in conjunction with setprop to overwrite the nameserver values android tries to use...
Another interesting idea might be to run a squid cache with data stored on the SD card... this should theoretically be faster than pulling the pages over 3G, but could be memory constrained.
FWIW I did some testing with timing page loads over 3G/2G/Wifi... and I found that the bottleneck for page loads on the G1 was not in fact the speed of the network connection(3G and WiFi were virtually identical - 12Mbps cable, 2G slightly slower), but CPU time spent in rendering(well, possibly memory related as well, but the G1 is always memory constrained so its hard to tell). The biggest speed improvement I experienced was in turning off Javascript.
I also tested Stock vs Dolphin vs Steel and found Stock & Steel to have very similar numbers, with Steel having a slight edge sometimes. Dolphin was always orders of magnitude slower.
IMHO The only real way to speed up browsing on the G1 would be to utilize an external compression proxy to reduce the amount of data being sent across the network to the G1, or to rewrite the browser/Dalvik VM/whatever to be more efficient. I tried the external proxy method but couldn't seem to get it to work.

good thought...i'll tinker w/ this some
also dot folder

@equid0x Good thoughts... I used Opera on Android, which uses a compression proxy. Runs fast. Might be an idea to look into this for the native browser and abandon the idea to cache DNS.

cyanogen said:
CM builds already have a dnsmasq binary in /system/bin. It's used for tethering. You could launch it with a different config file if needed (there's a dnsmasq.conf in /system/etc already for tethering). Also, the values in resolv.conf aren't really used by much anymore unless you have stuff linked with uclibc. The resolver in Bionic uses the values of the net.dns* system properties.
Click to expand...
Click to collapse
I think I have dnsmasq playing nice w/ this config:
Code:
no-resolv
no-poll
server=4.2.2.2
server=4.2.2.3
listen-address=127.0.0.7
interface=lo
pid-file=/data/misc/dnsmasqcache.pid
user=dhcp
no-negcache
it's listening on UDP port 53 -- seems to be doing its thing..
you're right..resolv.conf doesn't affect anything I do.
but I don't see any entries like net.dns* when i
#sysctl net
What am I missing about how Bionic does things?

alapapa said:
but I don't see any entries like net.dns* when i
#sysctl net
Click to expand...
Click to collapse
Use:
# getprop

Brut.all said:
Use:
# getprop
Click to expand...
Click to collapse
Thanks.
i can:
# setprop net.dns1 127.0.0.1
# setprop net.dns2 127.0.0.1
and it uses dnsmasq for a while, then they change back to t-mobile's (seems like it happens when i change towers or it goes from 3g->edge or vice-versa
does net.dnschange have any ability to control this?
know where I can find any documentation on the dhcp process that populates these values?
edit: just tested again and the settings persisted all night. performance wasn't noticeably different than normal. I'll try to do some more scientific tests and report back..

was this project dropped? anything goin on here?

dnsmasq cpu problem
Hi Guys,
I've implemented the dnsmasq with my adhoc connection. My config file is:
no-resolv
no-poll
server=10.50.30.254
listen-address=127.0.0.1
interface=lo
pid-file=/data/misc/dnsmasqcache.pid
user=dhcp
no-negcache
addn-hosts=/data/misc/dnsmasq-host
cache-size=65536
local-ttl=86400
where the dnsmasq-host file is a copy of http://www.mvps.org/winhelp2002/hosts.txt to avoid advertising site.
After few minutes I run dnsmasq I get the cpu at 100% and the process sayd:
dnsmasq: Unknown cmd ''
dnsmasq: Unknown cmd ''
dnsmasq: Unknown cmd ''
I've tried only with few config options but the result is the same.
Do you know what is that?

[Q] Android VPN configuration, through ADB?

Hello
I would like to configure multiple (let's say 100) devices with VPN settings.
Configuring the devices one by one manually is something that I want to avoid.
I do not have any MDM software.
Devices are Android 4 and possible 2.3-3.
Questions:
- Where are the VPN settings located?
- Can I edit the settings with ADB/sqlite3
- Are root rights needed?
- Is it possible to change settings from application (if I would create VPN config app for this purpose)?
Any links to the documentation or examples are appreciated, couldn't find these with google.

from the terminal in a rooted g1.
> adb shell
# sqlite3 /data/data/com.google.android.providers.settings/databases/settings.db
sqlite> INSERT INTO system VALUES(99,’http_proxy', 'proxyort');
sqlite>.exit

wilsonj said:
from the terminal in a rooted g1.
> adb shell
# sqlite3 /data/data/com.google.android.providers.settings/databases/settings.db
sqlite> INSERT INTO system VALUES(99,’http_proxy', 'proxyort');
sqlite>.exit
Click to expand...
Click to collapse
Thanks, but system table does not contain any VPN settings.
I have manually configured couple on VPN connections from settings, but cannot find those from settings.db.
Do you know which database and table contains VPN settings?
Regards,
Kimmo

I am not that much technical but what i have understood from your question is that you are willing to use your android devices instantly on 100 devices i guess! well the simple way of doing this is going through router because it will absolutely safe your time from configuring it on each device. however, you did not mention that are you willing to use from your home/office or plan to use it while travelling as well.
The best thing what i can recommend you is to go with a business vpn which will give you access to 100+ vpn accounts.
The best option will be to use router which will safe you from the hazel of setting again and again...

Thank you for you answer. You did not understand my question, which basically is:
- how to configure VPN settings using sqlite3

working

[POC][WIP] Easy Reverse Bluetooth Tethering (ZeroBT)

ZeroBT - Reverse Bluetooth Tethering
This was a modification I released on Android Central for the LG Optimus S. I know there's quite a few tutorials on Reverse Tethering however I made this so it would be easy to connect/disconnect for tablets and phones without data service, so without further to do...
Some notes before I give the download link and Instructions:
This is a proof-of-concept, the concept? That using the SQLite3 binary you can create Android services that watch for specific changes to the Android settings database. This Reverse Bluetooth Tethering mod watches /data/data/com.android.provider.settings/databases/settings.db for changes to WiFi, Bluetooth and Mobile Data states.
This currently will not work with phones that have an active data service, this is because it watches the Mobile Data setting and dials if it's turned on at the same time as Bluetooth, However I cannot block Android's mobile data at the same time.
Warning:
If your kernel does not have PPP support in it's kernel, You'll have to either compile the kernel modules yourself or find a kernel that does have support.
Originally, this package contained the PPP kernel modules but they were compiled for the iScream Kernel found on Android Central. Likewise, the binaries included might not work or might overwrite the ones included in your ROM if they already exist, if this is the case, please copy the files from your ROM's ZIP back into the xbin folder (rfcomm, chat)
Download:
ZeroBT 0.0.1.0 Beta 1 (Signed): Download the attached version, I can't post links yet.
Unless the carrier you plan to be tethering from is Verizon, you might need the Peers and Chat script files from a Berry4All package (Or download the Berry4All-PPPd-stuff.zip file attached).
Installation:
Flash the ZIP file and then mount your USB Storage without exiting recovery,
On your SD Card you'll see the ZeroPPP folder, copy your carrier's Options and Chat script here. Open Configuration.ini to begin configuration.
Configuration:
Let's say your a Sprint customer and want to reverse tether your Sprint phone to your Android device.
Your configuration.ini would look like this:
Code:
# Bluetooth Settings
BluetoothAddress=XX:XX:XX:XX:XX:XX
# PPPd Settings
PPPOpts=/sdcard/ZeroPPP/sprint
CSisEXE=no <-- This is only yes if the chatscript file is a bash script that calls chat.
ChatScriptFile=/sdcard/ZeroPPP/sprint-chat
EnableLogging=no
PPPLogFile=/sdcard/ZeroPPP/pppd.log
Use:
Now this is the reason that I stated not to install this mod if you have an active data service/plan..
To trigger connection while the Daemon is running:
Toggle Wifi Off, Bluetooth On, and Mobile Data on.
within aprox. 10 seconds (Some debug statements I forgot to take out makes it wait 5 so you can check logcat and to keep the log clean) your bluetooth phone (in my case a MotoRAZR) should connect to your Android device and start dialing.
Enjoy.
I posted this here on XDA-Developers because it could be of some use to someone later on. This is by no means a completed reverse tethering modification, however maybe someone could give me ideas for a new trigger. (I was planning to create a Settings UI using SL4A and Python.)
Please give some feedback, this is my second real Android mod and I would love for it to be available for public scrutiny. I know this might be better on Tablets so feel free to take the scripts included and tinker with them as you wish

Thanks! I don't think this thread got the attention it deserved! Im looking to do similar, but using computer to phone. The information has been helpful.

[MOD] DNS Server for Android - Notable speed increase

Hi,
This is a potential game changer, DNS server for Android, the open source, DNSMasq and some specific config files for it.
It runs on your Android device and you point your DNS queries to it instead of your remote ISP DNS server.
Using this setup will speed up your Android online experience many fold because DNS (specially over 3G) is one of the slowest blocking components of the entire browsing activity.
Features/Points to note -
- Caching, multithreaded. The replies from the cache are instantenous, hence browsing speed is enhanced significantly
- Query multiple DNS servers simultaneously. Fastest reply is used. Hence redundancy is also achieved along with speed. Currently set to Google DNS servers, Open DNS servers and your home Wifi router. Best not to use more than 5.
- Works with all versions of Android from 1.5 -4.x and all Android phones and tablets
- You can add your ISP's (both 3g and home) DNS servers to /data/local/dnsmasq.conf.
- Don't use your ISP's servers (and remove your Wifi router) if anonymity is a concern.
- Can help circumvent DNS restrictions.
- If your home router has DNS and an IP other than 192.168.1.1, you can change the entry in /data/local/dnsmasq.conf
- It may not start automatically if your ROM doesn't have init.d support. In that case and in either case, you can download SManager and set /etc/init.d/97dns to 'su' (run as root), 'boot' (run on boot) and 'net' ( run on network change)
- To make it stick all the more, you can download SetDNS and enable it and set the DNS server to 'Custom' -> 127.0.0.1 primary and 8.8.4.4 secondary ( backup Google DNS in case DNSMasq is not running for some reason). Make sure that you have Wifi and 3G options checked.
- Will significantly speed up your tethering experience if you set your DNS server to the Android phone and to use this DNS server. Currently the dhcpd option is disabled. You can either enable it or tell your tethering DHCP software to send DNS server=192.168.x.x ( or your Android phone IP) to your PC.
- Block Ads with the MVPS HOSTS file located at /data/local/dnsmasq-host
- VPN connections with and without split tunneling are supported. Check out commented sections in config files. Can use different DNS servers for different domains.
- You need to use this version of DNSMasq. Others ( eg those distributed with most ROMs) have a runaway CPU problem or don't work without the -d (debug) option. Any help resolving those issues welcome.
- You'll need root and su. busybox optional. Simply run Install.bat to install it and configure it automatically. Then make it stick with SManager and SetDNS as mentioned above.
- It's not mine, I just found the correct version and configured it.
- Enjoy! Please click the Thanks button if this helps you.
25th May - Fixed bug where Wifi DHCP wasn't working

Any idea why this breaks my native tether on an og evo 4g on team dirt cm9 port?

Is it necessary to use the install.bat from a pc to make this work or can I just add the dns script to the init.d folder?

Firstly, thank you.
I am trying to manually setup this so I can use some dns redirection (using the --server option) but when I run the install.bat file, it breaks tethering (which uses dnsmasq as well). Would you please help me out understanding what steps the script did to get it running?

Hi,
I think that this version of DNSMASQ may need some different startup options to support dhcp. I have specifically disabled dhcp in the config file.
Or it (dhcp) simply may be compiled out of the binary and it currently may not be usable for both DNS and DHCP.
YMMV. BTW the install script backs up the dnsmasq binary so you can replace it with the backup in case of any issues.
Cheers.

I will try this one if it'll bring any good to my connection. Thanks.

[GUIDE] How to avoid Google on Android (privacy)

In any ROM (even without gapps) there will always be automatic connections with Google(captive portal mode, ntp) and Qualcomm (ntp, gps) The theory says that we have to deactivate automatic date and time and use only the integrated GPS (or otherwise contact Google / Qualcomm) but in practice these hosts will connect to the internet when they can ... The file in a LineageOs ROM is usually in /vendor/etc/gps.conf. On the other hand, there are geolocation alternatives, such as theUnifiedNIp project. Later, we can use other servers such as Mozilla. For this purpose we need to have MicroG installed, although I leave the information in case you are interested in.
The exception would supposedly be Replicant, but it supports few devices.
To avoid the espionage of Google we need the following:
-Before installing / formatting / flashing, we export our contacts in .vcf format and then recover them by importing the file from the Contacts application. If you want synchronization you can use, for instance,DAVdroid in Nextcloud / Ownclowd. This file and our photos / videos are taken to the PC and viceversa using a wire. We also download F-Droid, Afwall firewall and Adaway apks. We do not connect to the internet till I say even we have to avoid mobile data or wifi in the setupwizard.
-We flash a LineageOs ROM, without gapps and without MicroG(because it generates too many connections with Google). In your options we will uncheck Automatic Date / Time. We become root with Magisk or su.
-The next step is to deactivate the captive portal mode. All Android phones send a ping to Google to verify that the internet works. In Development Options we will enable the Local Terminal / Shell. Afterwards, we look for the new app and we open it.
In order to have root access we write:su
Then we will put:settings put global captive_portal_mode 0
And finally:reboot (also in the terminal, because if we restart manually it will be activated again)
-We disable "Intent Filter Verification" system app. It connects to Google and Amazon to verify the net. It is not necessary to be root. We force stopping and afterwards we disable it.
-If we use Android Pie we change Private DNS from automatic (default) to No.
-Install the firewall Afwall + We will give access only to the apps that interest us. However, there is a "bug" in Android that produces another inevitable data leak for any Firewall. This occurs at the boot of the system in which the program is incapable because it is loaded later, and the OS takes advantage to skip the locks. In its experimental options, there is one that controls this behavior. "Fix the data leak at boot" To let us mark the option, which by default will be gray, we must indicate in the immediately superior option "Path of the home directory for script", the first one that appears/sbin/.core /img/.core/service.d. Besides, we mark IPv6 compatibility. Thus, Afwall could "see" some IP's and block them. Nowadays I do not know what they are but there are several (an failed) unknown connections.
-AdAway. Now It is the moment to connect to the internet. We add the lists I put below, update the app and reboot. We are going to block Google servers (time.google.com) and Qualcomm servers (Izat, izatcloud.net) because despite blocking them in the firewall, disabling automatic date / time and using only the integrated GPS for time, they will try to connect to the time servers as soon as the phone connects.
To simplify, we added the host that I created for that purpose:
https://gitlab.com/Jorgu81/hosts/raw/master/HostsGoogle
Most recommended is Steven Black, although it is not essential:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
- After that, we install the F-Droid store. We are going to avoid Yalp Store/Aurora because they generate too many connections with Google, but you can install / uninstall if you need any app.
-IceCat web browser. Startpage, Qwant or SearX search engines. The reason, here:
https://spyware.neocities.org/articles/browsers.html
First of all we disable its addons because they break the webs. Afterwards, we put the ublock addon Origin.. If you want to avoid Google webcrawling we must block its domains with the aforementioned add-on. These are some of its trackers:
adservice.google.com
admob.com
adwords.com
adservice.google.es
adservice.google.com
doubleclick.net
googleapis.com beware! Some forums will not load correctly if we block ajax.googleapis.com
google-analytics.com
googletagservices.com
googleusercontent.com
googletraveladservices.com
googlesyndication.com
googleadservices.com
gstatic.com (it tends to break many websites, so use it manually where possible)
-SD Maid. This program is very complete and with it we can disable system applications or avoid autobooting them. But what interests us is to remove tracking permissions of the apps, specifically those referred to Google. These are boot (auto-start),analytics, tracking, firebase and in general those that refer to google. If the apps contain any of them, we will remove them. We look for "Application Control" (previously we can mark it to indicate also those of the System in Settings) and after selecting them, we select the app and choose "Permission Manager". Do not forget to give the 3 points above (Other) to see all of them. Example with Whatsapp. Uncheck "com.whatsapp.Bootreceiver","google.android.gms.measurement.AppMeasurement Receiver","com.google.firebase.iid.FirebaseInstanceIdReceive r","com.google.android.gms.measurement.AppMeasurementInstallReferrerReceiver"
https://sdmaid.darken.eu/download/production/
If we want to avoid its automatic connections we could disable the options in Settings, General Settings, Bug reporting.
Do not forget Trust in LineageOs in order to change, deny or allow permissions of our installed apps.
With this we have finished the initial configuration to avoid, as far as possible, the espionage of our system.
----------------------------------------------------------------------------------------
Applications / recommended settings (from F-Droid, as usual):
-Beta Updater for WhatsApp. To update the well-known application.
-XprivacyLua. We need Xposed:
Xposed (not Android 9)
https://forum.xda-developers.com/xposed/xposed-installer-versions-changelog-t2714053
EdXposed (Android 9, alpha)
https://forum.xda-developers.com/xposed/android-9-0-xposed-solutions-t3889513
It will block all the information from our terminal that apps can takefor their indeterminate purposes. Few of them will make an incorrect use of it because they come from F-Droid. If we use WhatsApp we have to leave at least permission to clipboard and Contacts. In IceCat only for clipboard.

Removed old info.

Updated info:
-Disabling Intent Filter Verification due to its connections with Google, Amazon, and so on...

Another update and I hope the last...
We mark IPv6 compatibility in the firewall. Thus, Afwall could "see" some IP's and block them. Nowadays I do not know what they are but there are several (an failed) unknown connections.
Nevertheless, if we do not apply this, we could see data leak (AdAway, wireshark, tcpdump)
Now, there is no automatic connections at all.
On the other hand, I will install Pie soon so I will hope not to find too many problems...

Last update with Pie.
Only strange connections between mobile and router with Private DNS that is in automatic mode by default. We change it and select No.
-Added more hosts
-Changed hosts from GitHub to GitLab.
Now, we have 0 automatic data again when we turn on/off wifi or mobile data.