What is the best way to watch HTTPS traffic from apps now? I will collect what I have found so far, but hoping someone more knowledgeable will add some points. Feel free to correct or point out other ways of accomplishing this. It feels like regardless of the options, the root of the problems are how to get around certificate pinning.
Emulator vs Phone
This is the first question and probably the most dependent on what you want to achieve. Working on a real device gives more space between your device and the proxy which makes things easier. The extra space is costly in other ways. For example, I would prefer to have a single instance running on the computer to collect information, but using a phone is easier but has the physical requirement of a device connected to the network.
Phone
Physical separation allows for clearer testing. Fully functional device means your input and output work as expected.
Emulator - Waydroid
Emulator running on the same computer causes more complicated networking to ensure you don't block your own traffic. Troubleshooting is trickier as it's more difficult to easily access parts of the emulator that a phone is easy to access. For example, I spent much more time than I would have expected to move a VPN configuration file from my computer to the virtual machine emulator than I would have ever expected. Adding the same configuration to the phone was a simple QR code scan.
Emulator running in a virtual machine allows for a future use case of running the whole thing in the cloud without a physical device.
Proxies
As far as I know, the only way to capture the HTTPS traffic is to use a proxy. This is in the form of an application running on a separate (virtual or physical as mentioned above) device. The hardest part here is the Certificate Authority which signs the HTTPS traffic when it leaves the app. More sophisticated apps, to prevent fraud, do a variety of actions to prevent the user or 3rd parties from capturing the data in each HTTPS request.
mitmproxy
open source, link
I tried this first as it comes with Python library which would make capturing data for later analysis much easier. Mitmproxy has a few different modes, and ultimately I found that `mitmproxy --mode wireguard` which runs via VPN captured a good amount of traffic, but still had target SDK traffic unable to be opened. Mitmproxy has a built in tool to help installing the certificate in Android as a user certificate. This will capture some HTTPs traffic, but for some apps and many SDKs this does not capture their traffic. Traffic can be captured in several ways: CLI tool for analysis of live traffic in memory, CLI dump to file and in memory live in browser of choice.
Charles Proxy
free for 30 days, shareware, link
I first used Charles nearly 10 years ago, and it doesn't feel like it's changed much, but is actively maintained. When I first started using Charles it was a breeze to use, CA was less of a problem. But as Android changed it also now has the problems of CA needing to be installed, and helps the user by providing it's own signed certificate which can be installed as a user certificate. Charles is a standalone program that you run and as such it does have a fair amount of issues on my linux environment related to it's display sizes. .
Burp Suite - Community Edition
paid/free, link
Community edition that is free to use. Runs in browser and comes with it's own CA tool.
Android Certificate Authority
These are the certificates used to sign HTTPS traffic to keep it secure. In Android there are three levels: User, System (root) and App Pinned Certificates. In Android settings you can add a CA which will be considered "user". Apps can choose whether to ignore this certificate. System CAs can only be set by a root user. While a user can install user CA's, apps do not have to use these. CAs can be set by users as root certificates. I believe this must be set regardless of device or VM. The majority of the certificates provided by the proxies don't seem to open a lot of HTTPS traffic. This is likely because Android N (API level 24) certificate pinning was introduced in 2016 and at this point most SDKs and Apps use this for transferring traffic.
JustTrustMe
open source, link
This is installed on a device or emulator. An Xposed addon that can be installed to force apps to use root authorities and prevent them from pinning their own CA.
apk-mitm
open source, link
This can be installed in a separate linux environment and is used to modify an app's apk before being installed into a VM emultator or phone. It attempts to get around the app's certificate pinning by patching the APK to disable certificate pinning.
This is just my notes on what I'm looking into. I figured I'd post here to see if anyone has some advice or pointers. Please feel free to correct / add to this! Meanwhile I'll also keep my notes here if it helps anyone.
To anyone later who is interested in this topic, I was able to finally get a working solution using Magisk + LSPosed and two certificate modules which unpinned certificates and set my user certificate to system. I wrote my detailed steps here if anyone needs the help.
Related
Here is an idea I have thought of:
Making a GUI (root needed of course) applaction for turning on/off and managing WIFI routing. Basically, it would package the tethering script, and give the ability to see who is connect/block/etc. Possibly even mac filtering down the line.
My one main stumping block is that I would really love to get a little DHCP server running on it in the background, so that it would be much easier to manage. For instance, say I am on customer site with my coworker, and there is no net access. I would like to just allow him to autoconnect with out having to go through the process of setting up a new static ip and having him need to do that on the client side.
I went through the current linux distro on the phone, and, not surprisingly did not find dhcpd One solution that I am thinking of is to install debian, and get one that way, but that would not work too well as it would also require users to install debian and apt-get dhcpd.
Another thought that I had was to extract the files from the .deb, and include them in app, and have it just copy the files to the proper places (include shell script) then call out to dhcpd start/stop when needed.
The last option would be implementing a simple DHCP server in Java, kind of my last resort, as I am not even sure the API access to java.util.net.*
Just found this:
http://www.dhcp.org/javadhcp/
Seems to be J2SE compliant, so I am thinking (if the android JVM is compliant) it could be leveraged as a simple DHCP server. I still like the idea of using a premade linux one though
See other thread..
very cool.... dnsmasq is the way to go me things
Hi,
I have some doubts and do not know how to do, maybe you can help me.
so, I need to communicate between two different android apps on different devices.
Case study: i have tablet1 with app1 and tablet2 with app2 on a local network. I need app1 on tablet1 send and receive data to/from app2 on tablet2. So its a client/server communication (or better, i need many clients to one server).
There is a mechanism AIDL but only works different apps on same device, but i need on different devices. I have done some examples of services and messengers handler but without success. Applications must be version 4.0 and android app ( I say this because there is a NSD mechanism that discover services of other devices / apps android in local networks, but the minimum version is 4.1 and I already have several customers with tablet v4.0. I know it is possible but also do not want to install a web server on a tablet ( like a KWS or PAW server and others) ).
The documentation and examples that i use are google android site, like trainning, guide, references and others.
Thanks in advance for any tips.
I am new to today's device apps. However have taken a big project which I am not sure is deliverable!! I want to develop two cross-platform application (desktop-windows/mac/android, mobile-windows,iOS/iPAD/IPOD etc), lets call them site-access and remote-access.
LAN(Option1, site-access) Front end: HTML/CSS/JavaScript Database:H2 Database access language:GO programming language webserver/web application server: Go programming language server running on a pc in company (company server). I am hoping that I could use JavaScript to trigger some functions/libraries in GO to query H2 database? Will it really work like that?
LAN(Option2) Front end:HTML/CSS/JavaScript Database:H2 webserver/application server:Apoche Tomcat database access language: Java servlet/Node.js
In this case, I am hoping that I would use javascript to communicate with node.js running in back end that will then communicate with Apoche Tomcat over servlet. Will it work?
remote-access (hosted on google app engine) Front-end:HTML5/CSS3/JavaScript Employee seamlessly easily use remote-access icon on devices to connect to company server- backend running under architecture 1/2 above- and access files off company server? I am hoping that I could use some additional database access conditions for remote-access app using GO programming language to design simple login features? I am sorry about my naivety in web-development. But your input will surely put me in the right direction. Thank you
Dear Members,
Imagine: An Android Black Box without any screen but only USB Power+Data and Wi-Fi ports to be connected with a Lap- /Desk- top computer and the combo used as a Superphone
I have been planning for a long time to use internet — like the thread I had posted in Unix StackExchange in
unix.stackexchange.com/questions/86380/reading-sim-data-via-file-managers-using-usb-datamodem, around September 2013.
There I had posted the links of an idea: If Mobile-> Internet access Modem, why not datacard->mobile, posted in both Knoppix and Debian forums, around March 2013.
A killer of an idea came to me while I began using web.whatsapp.com
I have been doing research on the alternatives of the Android OS available on the web. These two links are sufficient for what I am going to present:
beebom.com/android-alternative/
itsfoss.com/open-source-alternatives-android/
Won't it be easier if, rather than to build free and Open-source alternatives to Android, Android itself is enhanced for its use with a computer, keyboard and mouse, using an app like the Whatsapp Digital Optical Code scanner, to have the display and button- and touchscreen- controls transferred to our lap- / desk- top computers, like we can in Whatsapp via web.whatsapp.com?
In Linux there already are ways to remotely control a desktop via appropriate permissions with a GUI interface.
This way, Google remains happy, while we too remain free from restrictive policies.
There are many emulators already available on the Open Source Linux systems, like QEMU, VirtualBox, and so on, not to mention the proprietary VMWare.
The app needs to have two parts:
(1) A rudimentary functionality of a Transceiver/Emulator, to slip right between the Hardware and the Android OS, creating a "What You Ask Is What You Get" one to one virtual communicator, and side by side, relaying the signals to the main app.
(2) A virtualisation of the user input signals and transceiving the same with the Android OS.
The main application having all the remaining functionalities to connect the Android OS with the Lap- / Desk- top via Wi-Fi, internet and its in-built optical scanner.
Of course, the App needs to have a cloud application to store all the data of the users on the cloud securely via SSL security like Whatsapp.
The App could earn its profits from the revenue structure Google has erected to have the app paid via advertisements. Interested users like us would also be more than willing to pay for the app, I believe.
In the end, again, a device could as well be developed to combine an Android SmartPhone Black-Box (without screen) Hot-plugged with a standard lap- / desk- top and forming a seamless combination of the two into one super-unit via Free and Open Source Software.
To conclude, I seek this opportunity to inform that I am a very empowering closet-entrepreneur, but I have my own limitations because of my inability to accept certain existing structures. So rather than forming an entrepreneurship venture, I like freely to share information. FOSI instead of FOSS, I for Ideas.
Hello guys,
Finally, I decided to post my question here because I couldn't find any useful information online. What is the problem?
We are looking for a management solution for our Android devices, which can support deploying AD-based user e-mail certificate. We are obligated to deploy a solution for signing and encrypting e-mails. We have AD CA in our windows domain which works ok. The user has to logon, open Outlook, Open the settings and the certificate is there, ready to use. Which for most of the users is ok. The problem is with the mobile devices (Android). We've tested TrendMicro Mobile Security (it is more antivirus as management tool), Sophos Mobile (looks pretty ok, containers etc.) but still can't deploy automatically the user e-mail certificate, We've checked as well XenMobile but there is as well an option only for device certificate. In most cases (solutions), the user should open the AD CA page, generate certificate, download it, deploy it, and then use, which is very difficult for most of the non-technical users and it is as well a security issue. Is there a solution to do this automatically?
I see that there are a lot of management tools for Android but it will be enormous work to test all of them.
So, does someone already did such thing and which tool was used?
Thanks in advance