Database Info

[Think Tank] Cache DNS Requests on Android to speedup browsing

I was wondering if there is a method to cache DNS requests? I use CM latest and was browsing around in the system folder, when i came by /system/etc/resolv.conf. Here you find the nameservers used by android to resolve DNS.
This got me thinking, because I use a DNS cache on my Ubuntu Box. Might there be a way to do this on Android, to speed up browsing regularly visited websites? Does anyone have any idea how to use a local DNS cache to speed up browsing?
Here's the deal:
- In /system/etc/resolv.conf you find 4 nameservers (4.2.2.5 - 4.2.2.2)
- Doing a ping test (count = 1000) to those, i find big differences in latency. Average was 800 on my test on 3G (I live in the Netherlands). I found this number rather large.
- On IRC, when asking this question, they mentioned porting dnsmasq to android and change nameserver to localhost
A ported version of dnsmasq is found here: http://code.google.com/p/android-wifi-tether/source/browse/trunk/res/raw/dnsmasq?spec=svn120&r=120
Copied the file to /system/bin, chmod +x the file, and this one works.
Now here's the thing:
- dnsmasq --help mentions a configurationfile: /data/local/dnsmasq.conf. This one have to be made for android
- how to configure dnsmasq so it uses the DNS servers and create a cache, thus speeding up browsing (in my humble belief)
- how to start dnsmasq on booting android (and making sure it stays alive)
And finally, does it make sense to use this kind of method? Like to apply for a think tank to make this work

That does sound like a good idea. For me it isn't just 800ms to find out the DNS but on edge rather something like about 3 seconds! this would seriously speed the whole thing up! However, how are you gonna find the IP adresses for the first time?

Well dnsmasq creates a cache, using nameservers found in the config file, if i am correct. I am looking into the dnsmasq.conf examples to see what workaround i need.. Should be pretty straightforward
resolv.conf should have nameserver 127.0.0.1, and dnsmasq should use configured nameservers like 4.2.2.5 and create a cache. Therefore apps use the dnsmasq as DNS server.
Furthermore, i am trying to find out what nameservers are the fastest. 4.2.2.2-5 are Verizon, while i am in the Netherlands. OpenDNS has a datacenter in Amsterdam, so i figured changing nameservers to OpenDNS would speed things up some more..

Even better would be unbound, though the stated overhead of ~11 MB might be too much for G1/mT3G v1.
Also, do you have to do any tricks to prevent resolv.conf from being overwritten when you switch networks? I know that usually dhclient will rewrite the file, and there's a couple of workarounds, but I have no idea which ones would work on Android.

CM builds already have a dnsmasq binary in /system/bin. It's used for tethering. You could launch it with a different config file if needed (there's a dnsmasq.conf in /system/etc already for tethering). Also, the values in resolv.conf aren't really used by much anymore unless you have stuff linked with uclibc. The resolver in Bionic uses the values of the net.dns* system properties.

cyanogen said:
CM builds already have a dnsmasq binary in /system/bin. It's used for tethering. You could launch it with a different config file if needed (there's a dnsmasq.conf in /system/etc already for tethering). Also, the values in resolv.conf aren't really used by much anymore unless you have stuff linked with uclibc. The resolver in Bionic uses the values of the net.dns* system properties.
Click to expand...
Click to collapse
Thanks for the reaction... You are right indeed, i found the dnsmasq.conf:
Code:
no-resolv
no-poll
server=4.2.2.2
server=4.2.2.3
interface=usb0
dhcp-authoritative
dhcp-range=192.168.77.100,192.168.77.105,12h
dhcp-leasefile=/data/misc/dnsmasq.leases
pid-file=/data/misc/dnsmasq.pid
user=dhcp
no-negcache
What you're saying is that i should create another one, but i am wondering what values i should use..
interface=?
Also, regarding your remark on DNS, how to see the values of net.dns* and how to change them? sysctl -n doesn't show these values, I must be looking in the wrong direction...

Hey,
You don't have to set an interface at all.
The interface delcaretion limit dnsmasq to listen only on a specific interface (for both DHCP and DNS requests).
As far as I know dnsmasq is caching dns by default you can limit the cache size and set not to cache negative queries but it will cache by default so no special settings is needed.
In addition, Dnsmasq is also function as dhcp server and if you don't want it to try removing all lines that declaring on dhcp settings.

True on that one, i want to use dnsmasq mainly on 3G
On my remark about the dns properties, already found out to use getprop and setprop
getprop shows different values on DNS compared to resolv.conf:
[net.dns1.195]: [84.241.226.140]
[net.dns2.195]: [84.241.226.9]
[net.dns1]: [84.241.226.140]
[net.dns2]: [84.241.226.9]

You need to be aware of one thing with regards to the resolv.conf file.
It is NOT USED BY ANDROID.
If you use the command "getprop", you will see several dns entries in there -- none of which matches the static dns servers set in resolv.conf. The resolv.conf is used by TERMINAL BINARIES.

zrubi said:
Hey,
You don't have to set an interface at all.
The interface delcaretion limit dnsmasq to listen only on a specific interface (for both DHCP and DNS requests).
As far as I know dnsmasq is caching dns by default you can limit the cache size and set not to cache negative queries but it will cache by default so no special settings is needed.
In addition, Dnsmasq is also function as dhcp server and if you don't want it to try removing all lines that declaring on dhcp settings.
Click to expand...
Click to collapse
might as well restrict it to the local interface
Code:
interface=lo
you can then turn off dhcp with
Code:
no-dhcp-interface=lo

I use OpenDNS on my home network which is claimed to be better than ISP dns servers. When you signup/setup you are required to input their values for dns servers, perhaps you set these values from your phone?

Not sure if Android supports this... but the easiest and least resource intensive way to do this would be to simply add your commonly used domains to the hosts file. I believe some ad blocking software used to use this to block ads.
Aside from that... Bind can be run as a caching nameserver and would probably work in conjunction with setprop to overwrite the nameserver values android tries to use...
Another interesting idea might be to run a squid cache with data stored on the SD card... this should theoretically be faster than pulling the pages over 3G, but could be memory constrained.
FWIW I did some testing with timing page loads over 3G/2G/Wifi... and I found that the bottleneck for page loads on the G1 was not in fact the speed of the network connection(3G and WiFi were virtually identical - 12Mbps cable, 2G slightly slower), but CPU time spent in rendering(well, possibly memory related as well, but the G1 is always memory constrained so its hard to tell). The biggest speed improvement I experienced was in turning off Javascript.
I also tested Stock vs Dolphin vs Steel and found Stock & Steel to have very similar numbers, with Steel having a slight edge sometimes. Dolphin was always orders of magnitude slower.
IMHO The only real way to speed up browsing on the G1 would be to utilize an external compression proxy to reduce the amount of data being sent across the network to the G1, or to rewrite the browser/Dalvik VM/whatever to be more efficient. I tried the external proxy method but couldn't seem to get it to work.

good thought...i'll tinker w/ this some
also dot folder

@equid0x Good thoughts... I used Opera on Android, which uses a compression proxy. Runs fast. Might be an idea to look into this for the native browser and abandon the idea to cache DNS.

cyanogen said:
CM builds already have a dnsmasq binary in /system/bin. It's used for tethering. You could launch it with a different config file if needed (there's a dnsmasq.conf in /system/etc already for tethering). Also, the values in resolv.conf aren't really used by much anymore unless you have stuff linked with uclibc. The resolver in Bionic uses the values of the net.dns* system properties.
Click to expand...
Click to collapse
I think I have dnsmasq playing nice w/ this config:
Code:
no-resolv
no-poll
server=4.2.2.2
server=4.2.2.3
listen-address=127.0.0.7
interface=lo
pid-file=/data/misc/dnsmasqcache.pid
user=dhcp
no-negcache
it's listening on UDP port 53 -- seems to be doing its thing..
you're right..resolv.conf doesn't affect anything I do.
but I don't see any entries like net.dns* when i
#sysctl net
What am I missing about how Bionic does things?

alapapa said:
but I don't see any entries like net.dns* when i
#sysctl net
Click to expand...
Click to collapse
Use:
# getprop

Brut.all said:
Use:
# getprop
Click to expand...
Click to collapse
Thanks.
i can:
# setprop net.dns1 127.0.0.1
# setprop net.dns2 127.0.0.1
and it uses dnsmasq for a while, then they change back to t-mobile's (seems like it happens when i change towers or it goes from 3g->edge or vice-versa
does net.dnschange have any ability to control this?
know where I can find any documentation on the dhcp process that populates these values?
edit: just tested again and the settings persisted all night. performance wasn't noticeably different than normal. I'll try to do some more scientific tests and report back..

was this project dropped? anything goin on here?

dnsmasq cpu problem
Hi Guys,
I've implemented the dnsmasq with my adhoc connection. My config file is:
no-resolv
no-poll
server=10.50.30.254
listen-address=127.0.0.1
interface=lo
pid-file=/data/misc/dnsmasqcache.pid
user=dhcp
no-negcache
addn-hosts=/data/misc/dnsmasq-host
cache-size=65536
local-ttl=86400
where the dnsmasq-host file is a copy of http://www.mvps.org/winhelp2002/hosts.txt to avoid advertising site.
After few minutes I run dnsmasq I get the cpu at 100% and the process sayd:
dnsmasq: Unknown cmd ''
dnsmasq: Unknown cmd ''
dnsmasq: Unknown cmd ''
I've tried only with few config options but the result is the same.
Do you know what is that?

[Q] Android VPN configuration, through ADB?

Hello
I would like to configure multiple (let's say 100) devices with VPN settings.
Configuring the devices one by one manually is something that I want to avoid.
I do not have any MDM software.
Devices are Android 4 and possible 2.3-3.
Questions:
- Where are the VPN settings located?
- Can I edit the settings with ADB/sqlite3
- Are root rights needed?
- Is it possible to change settings from application (if I would create VPN config app for this purpose)?
Any links to the documentation or examples are appreciated, couldn't find these with google.

from the terminal in a rooted g1.
> adb shell
# sqlite3 /data/data/com.google.android.providers.settings/databases/settings.db
sqlite> INSERT INTO system VALUES(99,’http_proxy', 'proxyort');
sqlite>.exit

wilsonj said:
from the terminal in a rooted g1.
> adb shell
# sqlite3 /data/data/com.google.android.providers.settings/databases/settings.db
sqlite> INSERT INTO system VALUES(99,’http_proxy', 'proxyort');
sqlite>.exit
Click to expand...
Click to collapse
Thanks, but system table does not contain any VPN settings.
I have manually configured couple on VPN connections from settings, but cannot find those from settings.db.
Do you know which database and table contains VPN settings?
Regards,
Kimmo

I am not that much technical but what i have understood from your question is that you are willing to use your android devices instantly on 100 devices i guess! well the simple way of doing this is going through router because it will absolutely safe your time from configuring it on each device. however, you did not mention that are you willing to use from your home/office or plan to use it while travelling as well.
The best thing what i can recommend you is to go with a business vpn which will give you access to 100+ vpn accounts.
The best option will be to use router which will safe you from the hazel of setting again and again...

Thank you for you answer. You did not understand my question, which basically is:
- how to configure VPN settings using sqlite3

working

[5.x,6.x,7.x] How to use VPN in CM builds on mt65xx devices

Install Shadowsock VPN : Use NAT mode to tunnel whole device
Download
Other OPTIONS
A workaround for VPN tunneling whole device
Follow these steps :
1. Go to psiphon and choose Browser-Only mode don't tunnel whole device.
2. Go to its logs tab and check the port of http proxy.
3. Now open WiFi setting and long press to your WiFi network (from which u are connected) go to modify tab and from check the advanced options.
4. Chosen manual proxy option and put host name as 127.0.0.1 and port as the no. which you noted in step 2 (eg. 8080) and click save .
Enjoy you have set up a VPN connection successfully.
You can open your browser and check your IP
Note: You have to undo this after you don't need VPN by changing manual proxy to none .
For those who want to use Orbot vpn
This is very similar to previous vpn workaround for psiphon
Just follow these step for Orbot
1. Open Orbot and long press to start orbot.
2. When all bootstrapped done go to check browser it will ask for normal browser or install Orweb .
3. Open Orweb and volla you r connected to tor but actually its not to tunnel your traffic to vpn go to Orweb settings.
4. Now note the proxy host and proxy port generally it is localhost(same as 127.0.0.1) and 8118 respectively but it may change so better to keep updated with port numbers.
5. Now go to wifi setting and long press the wifi (from which u r connected) and check advanced options.
6. Choose manual proxy option and put the host name 127.0.0.1 and port no. (e.g. 8118 ) which u noted in step 4
Now u can check https://check.torproject.org from any browser and volla Congo this browser is configured to use tor.
Sorry for long guide but it is detail enough so that anyone can easily setup tor in cm builds.
Note: You have to undo manual proxy option to none after you don't use vpn.
All credit of this workaround goes to - Suraj Rawat (@nervehammer) : thanks man
Don't forget to press thank ​

[GUIDE] How to avoid Google on Android (privacy)

In any ROM (even without gapps) there will always be automatic connections with Google(captive portal mode, ntp) and Qualcomm (ntp, gps) The theory says that we have to deactivate automatic date and time and use only the integrated GPS (or otherwise contact Google / Qualcomm) but in practice these hosts will connect to the internet when they can ... The file in a LineageOs ROM is usually in /vendor/etc/gps.conf. On the other hand, there are geolocation alternatives, such as theUnifiedNIp project. Later, we can use other servers such as Mozilla. For this purpose we need to have MicroG installed, although I leave the information in case you are interested in.
The exception would supposedly be Replicant, but it supports few devices.
To avoid the espionage of Google we need the following:
-Before installing / formatting / flashing, we export our contacts in .vcf format and then recover them by importing the file from the Contacts application. If you want synchronization you can use, for instance,DAVdroid in Nextcloud / Ownclowd. This file and our photos / videos are taken to the PC and viceversa using a wire. We also download F-Droid, Afwall firewall and Adaway apks. We do not connect to the internet till I say even we have to avoid mobile data or wifi in the setupwizard.
-We flash a LineageOs ROM, without gapps and without MicroG(because it generates too many connections with Google). In your options we will uncheck Automatic Date / Time. We become root with Magisk or su.
-The next step is to deactivate the captive portal mode. All Android phones send a ping to Google to verify that the internet works. In Development Options we will enable the Local Terminal / Shell. Afterwards, we look for the new app and we open it.
In order to have root access we write:su
Then we will put:settings put global captive_portal_mode 0
And finally:reboot (also in the terminal, because if we restart manually it will be activated again)
-We disable "Intent Filter Verification" system app. It connects to Google and Amazon to verify the net. It is not necessary to be root. We force stopping and afterwards we disable it.
-If we use Android Pie we change Private DNS from automatic (default) to No.
-Install the firewall Afwall + We will give access only to the apps that interest us. However, there is a "bug" in Android that produces another inevitable data leak for any Firewall. This occurs at the boot of the system in which the program is incapable because it is loaded later, and the OS takes advantage to skip the locks. In its experimental options, there is one that controls this behavior. "Fix the data leak at boot" To let us mark the option, which by default will be gray, we must indicate in the immediately superior option "Path of the home directory for script", the first one that appears/sbin/.core /img/.core/service.d. Besides, we mark IPv6 compatibility. Thus, Afwall could "see" some IP's and block them. Nowadays I do not know what they are but there are several (an failed) unknown connections.
-AdAway. Now It is the moment to connect to the internet. We add the lists I put below, update the app and reboot. We are going to block Google servers (time.google.com) and Qualcomm servers (Izat, izatcloud.net) because despite blocking them in the firewall, disabling automatic date / time and using only the integrated GPS for time, they will try to connect to the time servers as soon as the phone connects.
To simplify, we added the host that I created for that purpose:
https://gitlab.com/Jorgu81/hosts/raw/master/HostsGoogle
Most recommended is Steven Black, although it is not essential:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
- After that, we install the F-Droid store. We are going to avoid Yalp Store/Aurora because they generate too many connections with Google, but you can install / uninstall if you need any app.
-IceCat web browser. Startpage, Qwant or SearX search engines. The reason, here:
https://spyware.neocities.org/articles/browsers.html
First of all we disable its addons because they break the webs. Afterwards, we put the ublock addon Origin.. If you want to avoid Google webcrawling we must block its domains with the aforementioned add-on. These are some of its trackers:
adservice.google.com
admob.com
adwords.com
adservice.google.es
adservice.google.com
doubleclick.net
googleapis.com beware! Some forums will not load correctly if we block ajax.googleapis.com
google-analytics.com
googletagservices.com
googleusercontent.com
googletraveladservices.com
googlesyndication.com
googleadservices.com
gstatic.com (it tends to break many websites, so use it manually where possible)
-SD Maid. This program is very complete and with it we can disable system applications or avoid autobooting them. But what interests us is to remove tracking permissions of the apps, specifically those referred to Google. These are boot (auto-start),analytics, tracking, firebase and in general those that refer to google. If the apps contain any of them, we will remove them. We look for "Application Control" (previously we can mark it to indicate also those of the System in Settings) and after selecting them, we select the app and choose "Permission Manager". Do not forget to give the 3 points above (Other) to see all of them. Example with Whatsapp. Uncheck "com.whatsapp.Bootreceiver","google.android.gms.measurement.AppMeasurement Receiver","com.google.firebase.iid.FirebaseInstanceIdReceive r","com.google.android.gms.measurement.AppMeasurementInstallReferrerReceiver"
https://sdmaid.darken.eu/download/production/
If we want to avoid its automatic connections we could disable the options in Settings, General Settings, Bug reporting.
Do not forget Trust in LineageOs in order to change, deny or allow permissions of our installed apps.
With this we have finished the initial configuration to avoid, as far as possible, the espionage of our system.
----------------------------------------------------------------------------------------
Applications / recommended settings (from F-Droid, as usual):
-Beta Updater for WhatsApp. To update the well-known application.
-XprivacyLua. We need Xposed:
Xposed (not Android 9)
https://forum.xda-developers.com/xposed/xposed-installer-versions-changelog-t2714053
EdXposed (Android 9, alpha)
https://forum.xda-developers.com/xposed/android-9-0-xposed-solutions-t3889513
It will block all the information from our terminal that apps can takefor their indeterminate purposes. Few of them will make an incorrect use of it because they come from F-Droid. If we use WhatsApp we have to leave at least permission to clipboard and Contacts. In IceCat only for clipboard.

Removed old info.

Updated info:
-Disabling Intent Filter Verification due to its connections with Google, Amazon, and so on...

Another update and I hope the last...
We mark IPv6 compatibility in the firewall. Thus, Afwall could "see" some IP's and block them. Nowadays I do not know what they are but there are several (an failed) unknown connections.
Nevertheless, if we do not apply this, we could see data leak (AdAway, wireshark, tcpdump)
Now, there is no automatic connections at all.
On the other hand, I will install Pie soon so I will hope not to find too many problems...

Last update with Pie.
Only strange connections between mobile and router with Private DNS that is in automatic mode by default. We change it and select No.
-Added more hosts
-Changed hosts from GitHub to GitLab.
Now, we have 0 automatic data again when we turn on/off wifi or mobile data.

Where to find GPS certificates?

I want to create my own mod (for personal use) based on a downloadable Magisk GPS mod found here - https://github.com/Magisk-Modules-Repo/optmizedgpsconf . That mod doesn't work on my Pixel phone (with Lineage OS 17.1). It forces GPS to not lock on any satellites. With that mod, only assisted, network-based location works. However, that mod that have some entries in GPS.conf files that are an improvement over stock/Lineage GPS.conf settings security-wise.
Here's my own GPS.conf change:
NTP_SERVER = time.cloudflare.com
I've gone Cloudflare and DNS-over-HTTPS all the way - on every device in my network, router, modem, OS settings, YogaDNS (for Windows), AdGuard settings, Magisk Cloudflare DNS modules, Bromite, and set Cloudflare NTP server globally - "adb shell settings put global ntp_server time.cloudflare.com" . If there are other places where I can replace stock NTP Server address with time.cloudflare.com with relative ease, then please let me know.
Here are the GPS.conf entries I wanted to take from the Optimized GPS Magisk module to add to my own GPS.conf:
XTRA_SERVER_1 = https://xtrapath1.izatcloud.net/xtra3grc.bin
XTRA_SERVER_2 = https://xtrapath2.izatcloud.net/xtra3grc.bin
XTRA_SERVER_3 = https://xtrapath3.izatcloud.net/xtra3grc.bin
XTRA_VERSION_CHECK = 3
XTRA_CA_PATH = /system/etc/security/cacerts
I don't think LineageOS 17.1 comes with certificates for version 3 (the most secure) HTTPS version of XTRA servers and their BIN files... The default GPS.conf uses "XTRA_VERSION_CHECK = 1" (poorly secured HTTP XTRA servers). Can I get those Version 3 XTRA certificates somewhere? I don't think my mod is going to work without them. It could be that the existing GPS Magisk module doesn't work because those certificates are missing. Whenever I use Assisted GPS, AdGuard (in VPN mode) does not display connections to anything other than Google to determine my location via network...
Second major question - is a simple repack tool for those who want to create simple file replacement Magisk modules without scripts? I know nothing of programming, but I do have PC game modding experience via modding tools. Some basic syntax is easy to understand with common sense, such as "IF bla bla, THEN bla bla" stuff. I plan to use the existing GPS Magisk module to create my own GPS module simply via GPS.conf file substitution and zipping with WinRAR. That should work...