Related
NOTE:
I'm not a developer or Android expert. All information provided here is copied from different internet sources and is to the best of my knowledge. I'll not be responsible for any harm to you or your device resulting from this.
1. PC BOOT PROCESS
Before diving into Android boot process, let's have a look at Linux PC first.
Power Button Pressed
Power On Self Test (POST); identify the devices present and to report any problems
BIOS / UEFI
Necessary hardware initialization (keyboard, disk etc.)
Disk (MBR)
DOS Compatibility Region code (optional)
Bootloader
Active/boot partition (Boot sector)
Kernel
Initrd / initramfs (init)
Services/daemons/processes
BIOS / UEFI is the first software code that is hard-coded on board and runs after we press power button. BIOS runs in real (16 bit) mode of processor, thus it can not address more than 2^20 bytes of RAM i.e. routines can't access more than 1 MiB of RAM, which is a strict limitation and a major inconvenience.
When creating partitions, MBR is saved in LBA0, GPT header in LBA1 and primary GPT in LBA2-33, LBA34 (35th) is the first usable sector. Backup or secondary GPT is saved in last 33 LBAs, last usable sector by OS is ( Total LBAs - 33 ). Partitioning software aligns GPT partitions at larger boundaries, e.g. at LBAs that are multiple of 2,048 to align to 1,048,576 bytes (512 bytes * 2048 = 1 MiB) boundaries. So first sector of first partition is LBA 2048 and so on.
When a system boots, driver of a filesystem is to be loaded in RAM in order to use that filesystem, but driver is itself a file, inside some filesystem. It's like a chicken and egg scenario. So the solution is to always load (as a BIOS/UEFI standard) the first sector on the bootable storage (0/0/1 C/H/S in older schemes and LBA0 in newer), which is (legacy or protective) MBR. This communication between BIOS/UEFI and storage media is through commands which are specific to host controller e.g. ATA commands for devices with SATA/AHCI interface on PC.
Master Boot Record (MBR)
1st 512 bytes (1 sector) at the start of 1st valid disk
Bootstrap code (446 bytes) + Partition Table (64 bytes)
Executable code: Bootloader 1st stage scans partition table and finds 1st sector of active partition (or may point towards intermediate stage)
Partition table provides information about active/bootable partition (and all others as well)
Small size of 64 bytes limits the number of maximum (primary) partitions to 4
Since bootloader unable to understand filesystem (inodes etc.) yet, so MBR is itself executable
Last 2 bytes are boot signatures i.e. to find immediately if disk/drive is bootable or not and hence switch to the next
DOS Compatibility Region
This stage is specific to legacy GRUB, GRUB 2 (default bootloader on most of modern Linux ditros) splits this stage to stage 2 and 3
31.5 KiB / 63 sectors next to MBR, contains filesystem utilities
Still loaded by BIOS routines (or bootloader may use it's own drivers)
Required by certain hardware, or if "/boot" partition (sector containing stage 2) is above 1024 cylinder heads of disk, or if using LBA mode
Volume Boot Record (VBR) / Partition Boot Record (PBR)
Sector no. 63 (64th sector) and above may contain Volume Boot Record or Partition BR, very similar to MBR
Also called Volume Boor Sector, it may be the first boot sector on any partition
NTFS saves VBR as metadata file name $Boot at first clusters, which also contains cluster number of file $MFT. $MFT describes all files on the volume; file names, timestamps, stream names, lists of cluster numbers where data streams reside, indexes, security identifiers (SID's), and file attributes like "read only", "compressed", "encrypted", etc.
If disk isn't partitioned, it's the first boot sector of disk
Boot Partition (if exists)
In MBR scheme, a partition can be marked bootable / active using a flag, usually the first partition of disk
Windows stage 1 bootloader reads and loads only the "Active Partition" from MBR Partition Table
Bootsector or VBR/PBR is read by stage 1 or 1.5 (2 or 3 on GRUB2) bootloader which loads stage 2 (4 on GRUB2) or actual bootloader
MBR / VBR Contains:
Jump instruction (first 3 bytes) i.e. "goto boot code" command
Filesystem header
Executable boot code, usually contains jump instruction for next adjacent sector(s) containing stage 2 bootloader
End of sector (similar to boot signature)
Stage 1 or 1.5 (or 3 on GRUB2) bootloader reads the filesystem table (like MFT / FAT) on partition and loads actual bootloader as a regular file
Bootloader (Actual)
Loaded by previous bootloader from the filesystem of same partition
Loads all necessary filesystem drivers (if any further required)
Configuration is read from database e.g. /boot/grub/ on Linux (GRUB) and <"System Reserved" Partition>/Boot/BCD on Windows (BOOTMGR)
Windows:
BCD is binary file, can be read and modified by commandline tool bcdedit.exe or GUI tool EasyBCD
NTLDR on XP simply used C:\ as active partition reading C:\Boot.ini
Linux:
GRUB makes use of modules to offer extra functionality for complex boot processes
It can show a boot menu to user if needed or configured e.g. for multi-booting or in safe/recovery mode or boot from USB/Network etc.
Locates and loads the kernel of desired OS and ramdisk in RAM
If GRUB is unable to handle the kernel of an OS like Windows, it can be configured for CHAINLOADING i.e. read and execute bootsector of the partition containing Windows bootloader
'os-prober' helps 'grub-install' and 'grub-update' finding Windows boot partition (System Reserved) by reading bootloader configuration in that partition
Kernel
1st MB of kernel from same partition (/boot) loaded in RAM by bootlader in read mode, then switch to protected mode (32-bit) and move 1MB ahead clearing 1st MB
Then swith back to real mode and do same with initrd (if it's separate from kernel)
Kernel contain ramfs drivers to read rootfs from initrd and mount it
Initramfs
Contains minimal filesystem and modules (required drivers which aren't carried by kernel) to access real rootfs (hard driver, NFS etc.)
udev or specific scripts load required modules
<ramdisk>/init is usually a script which loads necessary drivers and mounts real rootfs
finally init switch_root's to real rootfs and executes <real rootfs>/sbin/init; sysV (traditional), upstart (Ubuntu's initiative) or systemD (the latest widely accepted)
init > getty (on virtual terminals) > login (program) > motd > login shell > bashrc / bash_profileRead more about LINUX CONSOLE & VIRTUAL TERMINALS
UEFI
UEFI can understand filesystem contrary to BIOS, hence no limitation of MBR code (446 bytes)
Needs an EFI System Partition (ESP), preferrably of minimum 550MB
ESP partition is formatted as FAT32 but can understand other filesystems such as FAT12 (floppy), FAT16, ISO9660 (CD/DVD), UDF etc.
EFI firmware reads directly <ESP_Partition>/EFI/<vendor>/<boot_programs> as configured in boot manager (which disk, which partition, which program)
Boot programs make use of EFI firmware or EFI shell or GUI Boot Manager to load kernel
If boot program is just the disk, (no partition and no program configured), then fallback program <disk>/<ESP partition>/BOOT/BOOTX64.EFI is executed
Secure boot feature verifies signature of boot program before loading
Multi-booting is easy, just read different entry from ESP partition unlike relying on single bootloader to chain load all available OS's
EFISTUB feature of Linux kernel allows booting kernel directly as a boot_program
UEFI works better with GPT than MBR
Must read:
ANDROID PARTITIONS & FILESYSTEMS
2. ANDROID BOOT SEQUENCE
There might be a single or multiple bootloaders (to give directions how to boot). For a typical android device (most common Qualcomm SoC / ARM processor), boot sequence is as follows:
BootROM (like BIOS on PC). It's integrated with SoC.
Processors, bootloaders
POST
SBL
Parallel loading related stuff from different partitions.
Application BootLoader (aboot)
Primary Boot Mode (if no Kernel detected or if bootloader/download mode key combination applied)
Bootloader/Download Mode
Secondary boot
Kernel (hardware detection and populating /sys, /dev/ and /proc directories as the processes start) and initramfs (creating rootfs and other pseudo filesystems on rootfs)
Init (first process with PID "1". It initiates further loading of processes and daemons)
System / OS (ROM)
Recovery (if recovery mode key combination applied. It's a kernel with UI to perform basic troubleshooting operations)
3. BOOTLOADERS
Bootloader(s) facilitate the the initial starting up of device by taking control from SoC, performing necessary checks, loading required components and then hand over the charge of booting to kernel. RAM is detected at first stage to start loading configuration of other hardware (like keypad, display etc.) in it.
There exist(ed) multiple bootloaders which are executed by different processors, on different devices with different (partition) names like RPM (PBL), DBL (Device Boot Loader; CFG_DATA or sbl1), SBL2, SBL3 (QCSBL) and OSBL (Operating System Boot Loader) etc.
In a nutshell, on modern ARM devices (Qualcomm SoC):
BootROM / iROM and PBL
iROM run by CPU0 on power button press, loaded in iRAM (before RAM is initialized)
It may set up RAM and execute PBL in RAM or leave this for SBL. iROM/PBL is hard-coded on SoC, written during CPU production process and it's closed source.
On devices (such as open boards or some tablets) which support booting from multiple sources like eMMC/sdcard/USB/UART/Network like a PC BIOS, there is an extra stage between iROM and PBL:
IBL (Initial BL)
It's also loaded in iRAM. Depending on CPU pin settings (hidden and soldered or exposed for manual switching) informed by iROM, IBL passes boot mode selection to PBL and optionally checks PBL integrity if itself e-signed by iROM.
SBL or XBL (Preloader)
IBL calls SBL from eMMC/SDCard which supports LCD output. SBL initializes the DDR RAM, loads the trusted firmware (TZ) and the RPM firmware if not loaded by BootROM. SBL calls the final bootloader after self testing the device.
Uboot is open-source secondary bootloader for embedded devices. However sources of SBL can also be obtained from Qualcomm.
ABOOT (APPSBL; predecessor of Little Kernel)
ABOOT loads Partition Table, kernel, splash screen (logo) and modem. It's also responsible for charging mode and fastboot mode. Memory addresses in RAM for boot/recovery partitions are hard-coded in aboot.
Other examples of final (i.e. just before kernel) bootloaders are uboot (traditional Linux bootloader for embedded devices) or manufacturers' developed BL's like hboot (used by HTC) and redboot etc.
Manufacturers put their limitations (say of network carrier i.e. SIM lock and others) at this stage. USB protocol isn't enough and communication with bootloader to hack such restrictions require special devices (called Flashing Box or Service Box in common language), even sometimes a protocol like JTAG i.e. talk directly to microprocessor.
As a norm, all of these stage-1,2,3... bootloaders are simply called BOOTLOADER. While on some devices there is no bootloader partition at all and bootloader(s) resides on SoC.
Coming back to the booting process, after initializing boot process, bootloader (if it's locked) checks the integrity of boot.img (normal boot) or recovery.img (recovery boot), loads them in RAM and transfers control to kernel offering it with "phys_initrd_start" address of compressed (cpio, gzipped) initramfs.
4. KERNEL & INITRAMFS
Once the kernel is loaded and extracted in RAM by bootloader along with parameters, kernel starts executing. Kernel is in fact a self-contained (static) executable binary, made up of many object files (.o) linked together at compile time. Once the architecture and CPU are identified, architecture-dependent code is executed as per parameters passed from bootloader. Then arch-independent stage is executed which includes setting up drivers (display, touch etc.), filesystems like rootfs, tmpfs, proc, ext4 etc. and initializing console as well (if configured). Here the kernel-space ends and user-space begins (what they call it).
Kernel extracts compressed initramfs in rootfs (which itself is ramfs or tmpfs) and executes /init binary which subsequently reads its configuration files /init.rc and other /*.rc files written in Android specific init language. With the help of kernel, init mounts pseudo filesystems /sys and /proc and populates /dev directory containing device node files. Then it mounts /system and all other partitions including /data (also decrypts it if encrypted) and sets (SELinux security) policies, system properties and environment variables (PATH, EXTERNAL_STORAGE etc.). Additionally init also look after any hardware changes (ueventd) and started services changes (watchdog) occurring dynamically.
Finally init starts the runtime located on the system partition. One of the major last processes started by init is Zygote (Java virtual machine) which compiles apps to run for specific architecture (mostly arm / arm64).
DEVICE TREE BLOB
Device Tree Blob (DTB) - created by DT Compiler (DTC) from DT Source (DTS) text - is a mapping of hardware components on a board/SoC and usually a part of kernel source.
PC hardware usually support hardware enumeration through ACPI i.e. kernel may enquire (probe) the buses - PCI (internal devices), USB (external devices), SCSI (storage devices), HDMI/DVI/VGA (display devices) etc. - which device is connected to it.
Buses on embedded devices (including Android devices) mostly don't support enumeration (hardware discovery) because there are usually fixed set of devices and no option for a different OS to be loaded on device. Therefore OS needs to be informed of all connected devices and this is done by providing a standard DTB to kernel. DTB is provided by SoC / motherboard vendor and is usually a part of kernel source. During boot process, DTB is loaded by bootloader at boot time and passed to kernel so that it can discover hardware and create node points accordingly.
We can view device tree on Adroid device by:
Code:
~# ls /sys/firmware/devicetree/base
~# ls /proc/device-tree
DTB may live on a separate dtb/odm partition as specified by AOSP (and was the proposed solution for ARM based embedded Linux devices before Android's birth) but that isn't widely practiced. Usually DTB is appended to kernel zImage/Image.gz or placed at second stage inside boot.img.
VERIFIED / SECURE BOOT
Ensuring a chain of trust from Power ON up to loading of kernel is with the domain of SoC vendor (Qualcomm, Intel etc.) and OEM's. Injecting some malicious or harmful code at any point during booting is made harder to the extent of impossibility.
To ensure a secure booting chain, PBL verifies authenticity of SBL which subsequently verifies integrity of bootloaders (TZ, RPM, DSP, HYP and aboot) so that to avoid loading of unsigned images (boot, recovery, system and others). TZ, after being loaded by SBL also verifies ABOOT using a hardware-based root certificate.
A bootloader with Verified/Secure Boot implementation verifies boot.img or recovery.img (kernel, initramfs and DTB appended to kernel or on second stage of boot.img) by matching their signature with key(s) stored in "OEM keystore" (some partition like CMNLIB, KEYMASTER or with some other name) which itself is signed by OEM. Some vendors allow replacing/appending this keystore with custom one so that custom signed images can be flashed followed by re-locking of bootloader. A simple detail is given here.
At this stage, the chain of trust is handed over to "dm-verity" key stored in boot image initramfs, responsible for "Verified Boot" process of Google/AOSP. Dm-verity (a part of Verified Boot implementing Linux Device Mapper by Google) is a kernel feature i.e. it comes into action after boot image (kernel and ramdisk) is loaded in RAM. It verifies subsequently loading block devices; /system, (/vendor if it exists) and optionally others.
For details see this, this and this.
Google suggests integrating libavb (native code to verify integrity of boot.img) in bootloaders starting from Verified Boot 2.
Unlocking Bootloader
Read here to know about the risks of BL unlocking.
Unsigned kernel or recovery cannot be loaded unless bootloader is unlocked. To make any modification to OS, a critical piece of process is disabling a security system built into the Android's bootloader (aboot) that protects the read-only partitions from accidental (or intentional) modification for privacy, security and DRM. This is what's referred to as "unlocking NAND" or "unlocking bootloader." You have to firstly unlock bootloader to modify partitions "boot" or "recovery" and to gain root access on /system. If bootloader is locked, you only have write access to /cache and /data partitions. Everything else is read-only on device and bootloader will prevent unsigned images from being flashed to the phone. Unlocked bootloader ignores signature verification check which was initiated by BootROM and then transferred to "SBL" and then to "ABOOT" while loading kernel or recovery.
Some newer devices don't allow unlocking of bootloader directly (FRP) without permission from manufacturer to ensure more security i.e. contents of partition "devinfo" are signed by the OEM and can't be modified without their approval. After having permission, an official method is provided to unlock BL using PC. Still some functions related to Proprietary Content might be lost due to bootloader unlocking.
DRM is used to protect content from being copied.
Certain pre-loaded content on your device may also be inaccessible due to the removal of DRM security keys.
Click to expand...
Click to collapse
Android Rooting
Must Read: Root User and Linux Capabilities: Linux vs. Android
Note: Unlocking Bootloader and Rooting breaks "Verified Boot". It can be dangerous.
In order to perform some privileged task on Android, we need to "root" the device first. Since it's impossible to start a process with elevated privelages from within running Android OS, rooting usually involves running a root process (su-daemon) from boot with all capabilities. Superuser requests are made by any non-privelaged programs by executing "su" binary and permissions are managed by an app.
In early days, rooting usually involved booting into a custom recovery which in turn mounted and modified /system files. Usually some daemon's executable binary was replaced with a custom script. In order to address the OTA and other issues caused by improving security features (SELinux, Verfied Boot, SafetyNet etc.), systemless root method was introduced which is used by latest apps like Magisk. It involves modifying /boot image and putting some files on /data as well. So a new init service is injected fulfilling all necessary requirements of new security mechanisms.
In both cases, a locked bootloader won't boot custom recovery or modifed kernel (boot.img). See Verified Boot. Therefore bootloader needs to be unlocked for rooting.
However it is possible to gain root sometimes without unlocked bootloader but not always.
Other methods of rooting a phone from within a running ROM using some sort of One-Click rooting solution (KingRoot, Z4Root, KingoRoot etc.) depend on some vulnerability or exploit in Android OS. Making such security breaches is getting harder and harder with every new release of Android and with improved defense mechanisms, though it varies for different vendors too. The most prominent was with the release of Lollipop and Marshmallow when systemless method had to be introduced beacuse the previous methods failed to work. When phone is rooted using one of such improper root methods, there is a high probability to face "incomplete root" like messages at some point. If such a rooting method works for your device, it's alarming. This exploit is also a way for malware to enter your device. For examples, see Magisk Installation - Exploits, this and this. A very popular exploit dirty cow was patched later.
In addition to that, there are some hacks for certain devices to flash custom recovery without unlocking bootloader using some kind of Firmware Flasher tool (SPFlasher, MiFlasher etc.) in Download Mode because Download Mode provides access to device even before bootloader/fastboot is loaded. Or if you are expert in coding, you can mimic the custom recovery image look like the factory signed firmware and flash it through stock recovery. But this exploit isn't a universal solution either.
So the proper way to rooting which doesn't need any vulnerability, goes through unlocked bootloader. While buying a new phone this must be considered. Keeping you away from root access and unlocked bootloader goes in favor of vendors. By forcing you to use their ROMs (with bundle of useless bloatware apps), they earn a lot from you - money as well as forced loyalty - by collecting data, showing ads and using a lot of other tactics. Go for a brand that provides kernel source and ability to unlock bootloader (on customer's responsibility and with voided warranty obviously).
FIRMWARE UPDATE PROTOCOLS (BOOTLOADER MODE)
Likewise BL, on every device there might be a single or multiple BL modes with different names like bootloader mode, download mode, emergency mode (EDL), ODIN (Samsung), nvFlash tool etc. When we boot in BL mode, device is stuck on boot logo. Some factory flashers work in these modes such as MiFlasher (Xiaomi) and SP Flash Tool (for MTK devices). Bootloader or Download Mode is accessible even if device is soft bricked i.e. if Recovery and/or ROM isn't accessible.
Download Mode
Download Mode (certain button combination while powering on device; usually Vol. Up + Vol. Down or Vol. Down for longer duration + Power) is an official method used by many vendors to flash factory firmware / updates using Flasher (software). Emergency Download Mode (EDL), as it's called on Xiaomi Devices, can also be accessed through fastboot/adb commands or by using some jigs/jumpers. However, to ensure more security, EDL is disabled on some newer devices.
Download Mode is primary to bootloader mode (at PBL or SBL stage) and can be used without unlocking bootloader.
Odin (Samsung), QPST/QFIL work in Download mode (Qualcomm HS-USB QDloader 9008).
When we boot in Download mode, device is stuck on blank screen.
Fastboot Mode
Fastboot - provided by ABOOT - is a software development tool and a standard communication protocol for Android bootloader. It's an alternate of recovery flashing that works in BootLoader mode (aboot) and comes bundled on most of the recent ARM Qualcomm devices. It's a minimal UI through commandline to interact with device in case of failure or to modify / flash partitions. Some OEM's provide fastboot with limited functionality e.g. 'fastboot oem' commands not working and some devices haven't at all. It's up to the discretion of mobile phone vendor.
Fastboot mode is used to perform operations through commands when device is connected to PC through USB. It works even when phone is not switched on in Recovery or ROM or even if android isn't installed on phone. You can read here what operations we can perform through fastboot mode.
Only NAND (eMMC) and USB modules (drivers) are activated at this stage.
INIT PROCESSES & SERVICES: ANDROID vs. LINUX
FILESYSTEM TREE MOUNTED BY INIT: ANDROID vs. LINUX
RESOURCES:
From the bootloader to the kernel
RESERVED
RESERVED
RESERVED
RESERVED
You have to firstly unlock bootloader to modify partitions "boot" or "recovery" and to gain root access on /system. If bootloader is locked, you only have write access to /cache and /data partitions. Everything else is read-only on device and bootloader will prevent unsigned images from being flashed to the phone.
Click to expand...
Click to collapse
I'm under the impression that unlocking the bootloader is not mandatory for rooting the device.
You can root the device with a locked bootloader and gain full access to /system partition.
NikosD said:
I'm under the impression that unlocking the bootloader is not mandatory for rooting the device.
You can root the device with a locked bootloader and gain full access to /system partition.
Click to expand...
Click to collapse
Yeah I think my brief statement is a bit misleading because rooting is out of the scope of this thread. I have added some details to first post.
Thank you very much for all this useful info.
Some more comments.
A locked bootloader won't boot custom recovery or modified kernel (boot.img)
Click to expand...
Click to collapse
It happens to have a budget Chinese tablet with Oreo 8.0 and MediaTek SoC, which I can root using a modified/patched boot.img with Magisk v17.1 inside of course - I mean full root without problems - keeping the bootloader locked before and after rooting.
In addition to that, there are some hacks for certain devices to flash custom recovery without unlocking bootloader using some kind of Firmware Flasher tool (SPFlasher, MiFlasher etc.) in Download Mode because Download Mode provides access to device even before bootloader/fastboot is loaded
Click to expand...
Click to collapse
The tablet mentioned above, belongs to this category too.
Using SPFT (Smart Phone Flash Tool), I can flash custom recovery TWRP for my device without unlocking the bootloader.
So, I have two questions:
1) Is it rare to have such a device or is it common nowadays to be able to root and flash custom recovery TWRP with locked bootloader ?
2) How is technically possible to patch boot.img for rooting and flash TWRP using SPFlashTool (even in download mode before bootloader) without complains afterwards from bootloader, verified boot, dm-verity and all these safety checks that validate digital signature of Vendor ?
I mean you can do whatever you want before bootloader starts, but how can you escape from security traps after the initialization of bootloader verifications ?
Thank you.
NikosD said:
1) Is it rare to have such a device or is it common nowadays to be able to root and flash custom recovery TWRP with locked bootloader ?
Click to expand...
Click to collapse
I'm not sure how common it is but I must say these are exploits. Developers are making use of these vulnerabilities for positive and negative purposes. But these are not a "long-term" solution for rooting.
2) How is technically possible to patch boot.img for rooting and flash TWRP using SPFlashTool (even in download mode before bootloader) without complains afterwards from bootloader, verified boot, dm-verity and all these safety checks that validate digital signature of Vendor ?
I mean you can do whatever you want before bootloader starts, but how can you escape from security traps after the initialization of bootloader verifications ?
Click to expand...
Click to collapse
That's what my point is. Fastboot code verifies signatures/hashes only when flashing the image and doesn't verify or fails to verify integrity if image is already flashed. This is not the desired behavior so it's an exploit and it should be closed. Letting unsigned images be flashed in Download Mode is another exploit which is common with Chinese vendors as far as I have come across some instances. They don't address "loopholes" seriously. Failure to stop security breaches at or after bootloader level is definitely on SoC Vendor or OEM's part. I have added a paragraph in first post with some useful details and links.
This link explains:
The Qualcomm SoC is analyzed in the previous chapter dload / edl mode, the mode in the firmware image download process does not do any verification, can be directly written into the brush.
Click to expand...
Click to collapse
It's badly translated from Chinese but is informative.
Exploiting Qualcomm EDL Programmers is a complete series on this subject summarized here.
mirfatif said:
Only NAND (eMMC) and USB modules (drivers) are activated at this stage.
Click to expand...
Click to collapse
Hey pal, I'd like to know if you could help me with an issue I'm facing. I have a Moto G5 that isn't booting to any ROM (it either bootloops in bootlogo or in boot animation), and also on TWRP and during the boot animations the device is slow as hell (like 0.5 FPS on TWRP and even less on boot animation; on TWRP the device also takes a few seconds to complete even the simplest tasks - like the press of a button or the swipe of a slider - here's a video that shows differences between how stuff works on fastboot and how slow things are on TWRP, it takes like 2 hours to completely flash a custom ROM, i.e.).
I know much of the issue will be device-specific, but my point (and the reason I quoted that specific part of your OP) is that, on fastboot mode, the device is snappy and responsive. When I press a button it completes the corresponding task immediately, frames don't stutter (not that there are any animations to be rendered in fastboot, but when I switch from one option to another using the volume keys, it does so on screen as it should, with no lag), and so on. Stock recovery also seems to be ok with speed, but it's even harder to measure than fastboot because, in almost 10 years meddling with android devices, I have always found stock recoveries (and CWM in the pre-TWRP times) to be somewhat slow. Stock recovery definitely looks snappier than TWRP, though. Tried several ROMs, both custom and stock, and the issues remain on all of them.
I got to this post by researching if fastboot mode was stored on the same NAND chip as recovery, OS and so on (found out that yes, it's all on the same chip). If it wasn't, I could just assume it was a hardware fault on the NAND chip, and that would be the reason that fastboot was running fine but recovery and OS weren't, but since they're all on the same cell, I can only think that some part of the system (I mean as in every single code that runs on the device, not only the OS) that loads on TWRP and on normal boot, but not on fastboot (and possibly not on stock recovery) are faulty, thus being a software issue (either solvable with just a normal USB cable or needing a flash box).
So, my question is: which are the differences in the parts of system loaded by fastboot and by TWRP? Are there any parts that are loaded by TWRP that aren't loaded by the stock recoveries on most devices?
I know it's a rather complicated question and some stuff might be device-specific, but if there is anything you could tell me that are more generic to every Android device, it would help me a lot. Thanks in advance.
[android-security-discuss] Apply for key attestation for hardware-backed keystore authentication
android-security-discuss.narkive.com
It says in this link that the TEE attestation keys aren't generated in the TEE and are batch keys issued by a keymaster? So, if the manufacturer has access to your phone's individual key, should they be able to restore it to your device if your bootloader has stayed locked with all official software installed?
Perhaps we might have the ability to regenerate our own keys? They appear to be generated by the secure bootloader and the attestation key seems to change on updates. So, if we give the generator what it needs from the secure bootloader (Which shouldn't be lost if it stays locked), the Android version, and patch level could we generate a new key, like what the system seems to do on the very first boot?
Keymaster Functions | Android Open Source Project
source.android.com
AFAIK the hash keys stored in the vbmeta files are generated when manufacturers compile Android for their phones. Also Android's boot.img gets signed by OEM.
May be you by means of AVB2TOOL can regenerate the vbmeta files on base of installed Android at your own. But IDK it.
FYI: On device's very 1st boot the ANDROID_ID key gets generated, not the vbmeta files.
I'm having trouble understanding the architecture of mobile (and Android) devices. I compare it a lot to the design of PCs, laptops, etc, which I know quite well.
Here's my understanding on how PCs work when booting:
The hardware has firmware stored in ROM (Read Only Memory). Actually, Flash memory is used nowadays, on which the stored content can of course be changed, unlike real ROM memories in the old days. Because the firmware is hardware-specific and its operation is very critical, its content is rarely updated or otherwise changed. Installing new firmware is called flashing. Firmware in a PC is most commonly BIOS or UEFI, the task of which is (briefly) to first run the POST tests, provide some interfaces and finally start the software in the mass storage. By mass storage, I mean memory separate from the firmware's Flash memory, which can also be Flash memory, such as an SSD disk, or a more traditional hard disk.The BIOS (i.e. firmware) in the specified order (which first is the internal NVMe SSD or the external USB hard disk?) tries to load the software into the RAM memory for execution from mass storage MBR (Master Boot Record) part . Master boot record is a physical defined area in mass storage. Bootloader software is stored on this MBR part.When the bootloader (located on the MBR part) is loaded into RAM and run, it knows the contents of the end of the disk and starts the kernel from there.The kernel starts (in Linux) the init process, nowadays often Systemd, which starts the rest of the software.--------------------
What kind of memories and storages are most commonly found in Android devices? One main memory (i.e. RAM)? One Flash memory for firmware (i.e ROM)? Another separate flash drive that acts as mass storage? Possibly SD card and USB stick as external mass storage?
What is firmware on Android devices?
What is the bootloader in (located in MBR part) on Android?
Linux is the kernel used by Android, which is started by the bootloader? After that, Android continues to boot, how?
A pile of terms, which I have ambiguities:
Bootloader; What's it like on Android? It is often characterized as hardware specific. So is it the case that the bootloader in Android is firmware? So in Android, the firmware runs the tasks of the PC world BIOS and bootloader (located in the MBR part), and then starts the Android located on the mass storage?
Recovery; What is this technically?
Android ROM; I can't understand this. As far as I know, Android is an operating system located mass storage, not Read-Only-Memory firmware.
Rooting; On a PC, we are used to the fact that the owner of the device has root rights. Is it just that the manufacturers have decided to set the default root password to some generated random string, and by default, the user only has access to the basic user account?
After the above has been answered, I would like someone to explain to me (separately) technically, starting from the hardware level (where and how), how do Android devices booting and work? Links to additional information are also welcome. hank you very much! If anyone can answer my questions, thank you very much!
Your questions should put you to shame.
Start reading yourself, building up your knowledge as you read.
Anyway, welcome to the forum. After a year of reading, you will laugh at your post.
ze7zez said:
Your questions should put you to shame.
Start reading yourself, building up your knowledge as you read.
Anyway, welcome to the forum. After a year of reading, you will laugh at your post.
Click to expand...
Click to collapse
I know my questions are stupid, but I'm impasse. It seems that there is much less information about designing for mobile devices than PCs. Could you link some articles on this? As the last article I read this, but it didn't help much, because I compare too much what I learned on PCs.
There are no stupid questions, there are only stupid answers.
Start with the basics based on information from google:
Architecture overview | Android Open Source Project
source.android.com
ze7zez said:
There are no stupid questions, there are only stupid answers.
Start with the basics based on information from google:
Architecture overview | Android Open Source Project
source.android.com
Click to expand...
Click to collapse
That is useful, but there is a reason why I asked about mobile/Android device design/architecture. Android itself is as far as I know (if I'm not mistaken) just an operating system, like the desktop operating systems Windows and Ubuntu, but mobile/Android devices are very different from PCs in terms of hardware and firmware. For example: https://www.quora.com/Is-there-anything-like-BIOS-in-mobiles-How-do-they-boot
How long is a huge ball of string?
No simple answer...
This is for those who are new to Android development and basically have NO understanding about the partition structure. I will give a high-level introductory explanation. PC GNU/Linux users: please note this is completely different from x86 (PC Linux) partition table. You will not come across partitions denoted as sda1, sda2, sdb1, sdb2, and so on. Instead, it will be structured as follows:
/boot
This is the partition that has all the data that is necessary for the phone to boot. It includes the kernel and the RAMDISK (these are the only components of the operating system that are stored in this partition. The remaining are stored in /System). Without this partition, the device will simply not be able to boot. Wiping this partition from recovery should only be done if absolutely required and once done, the device must NOT be rebooted before installing a new one, which can be done by installing a ROM that includes a /boot partition.
/system
This partition basically contains the entire operating system, except the kernel and the RAMDISK (as mentioned in /boot explanation). This includes the Android User Interface as well as all the system applications that come pre-installed on the device. Wiping this partition will remove Android from the device without rendering it unbootable, but you will still be able to boot into the /recovery partition to install a new ROM.
/recovery
The recovery partition can be considered as an alternative boot partition that lets you boot the device into a recovery console for performing advanced recovery and maintenance operations on it. Think of this like a proprietary recovery partition that PC companies put on prebuilt PCs. When you flash a custom recovery such as TWRP or CWM, you are overwriting this partition.
/data
Also called userdata, the data partition contains the user’s data – this is where your contacts, messages, settings and apps that you have installed go. Wiping this partition essentially performs a factory reset on your device, restoring it to the way it was when you first booted it, or the way it was after the last official or custom ROM installation. When you perform a wipe data/factory reset from recovery, it is this partition that you are wiping.
/cache
This is the partition where Android stores frequently accessed data and app components. Wiping the cache doesn’t effect your personal data but simply gets rid of the existing data there, which gets automatically rebuilt as you continue using the device.
/misc
This partition contains miscellaneous system settings in form of on/off switches. These settings may include CID (Carrier or Region ID), USB configuration and certain hardware settings etc. This is an important partition and if it is corrupt or missing, several of the device’s features will will not function normally.
/sdcard
This is not a partition on the internal memory of the device but rather the SD card. In terms of usage, this is your storage space to store your media, documents, downloads, pictures, videos, ROMs etc. on it. It is like the equivalent of the ' Users/[Username] ' folder in Windows and ' /home/~ ' folder in x86 Linux. Wiping it is perfectly safe as long as you backup all the data you require from it, to your computer first. Though several user-installed apps save their data and settings on the SD card and wiping this partition will make you lose all that data.
On devices with both an internal and an external SD card – devices like the Samsung Galaxy S and several tablets – the /sdcard partition is always used to refer to the internal SD card. For the external SD card – if present – an alternative partition is used, which differs from device to device. In case of Samsung Galaxy S series devices, it is /sdcard/sd while in many other devices, it is /sdcard2. Unlike /sdcard, no system or app data whatsoever is stored automatically on this external SD card and everything present on it has been added there by the user. You can safely wipe it after backing up any data from it that you need to save.
/sd-ext
This is not a standard Android partition, but has become popular in the custom ROM scene. It is basically an additional partition on your SD card that acts as the /data partition when used with certain ROMs that have special features called APP2SD+ or data2ext enabled. It is especially useful on devices with little internal memory allotted to the /data partition. Thus, users who want to install more programs than the internal memory allows can make this partition and use it with a custom ROM that supports this feature, to get additional storage for installing their apps. Wiping this partition is essentially the same as wiping the /data partition – you lose your contacts, SMS, market apps and settings.
/Boot (Is NOT viewable in Android)
/Recovery (Is NOT viewable in Android)
/Data (Userdata) (Is viewable in Android)
/Cache (Is viewable in Android)
/System (Is viewable in Android)
/Misc (Is NOT viewable in Android)
Ram
https://developer.android.com/topic/performance/memory-management
Understanding Firmware naming:
N986USQU1ATGM
N=Note
986U or F etc, the model of device
SQ, FX etc = CPU and model specific
U,S,E = Update, Security, Engineering, respectively
1,2,3,4,5 etc = bootloader revision (This is important! You cannot go to a previous revision)
A,B,C,D = Android version
T, U = Year (T=2020, U=2021 etc)
A,B,C etc = month (January A - December L)
1 - 9 and then A - Z =build compilation. This basically means how many builds there are in a month. They start at 1 and go to Z
So N986USQU1ATGM would be
N968-U-SQ-U-1-A-T-G-M
N968U (Note 20 Ultra Carrier version), SQ (Snapdragon), U (Update), 1 (Bootloader version), A (Build 10), T (2020), G (July), M (22nd build)
How to enter Download Mode:
Turn off the device.
Connect USB cable to your PC (Leave it disconnected from the phone)
Press and hold down the Volume Up and Volume Down buttons. While they are still pressed, plug in the USB cable into your phone.
The phone will go into download mode press volume up. In Odin you will see that phone is added.
Dirty Flash:
I would only do this if you are having to manually update to the newer firmware and would not do it if you are coming/going to U/U1 or from beta firmware or if you are on an old firmware. I'd also highly recommend doing a back up prior to the doing this
Load these into Odin
BL
AP
CP
HOME_CSC
Do NOT flash CSC or USERDATA, either of these WILL wipe your device
This is a "dirty flash" and these can sometimes cause issues. Keep in mind if things start going sideways and stuff starts not working right, your first step to a solution will be to wipe the device.
Tips on flashing U1 Firmware:
You will have to wipe, can NOT dirty Flash going between U and U1 firmware
Use the patched ODIN linked in post #2 or #3, Odin3_v3.13.3b (They are exactly the same)
Have an active US Carrier SIM installed to get carrier features
If you get your CSC Stuck on XAA/XAA/(Insert your carrier here), and can not get Carrier options back.
PIT files
https://ihax.io/samsung-pit-files-explained
plus_rlus said:
I know my questions are stupid, but I'm impasse. It seems that there is much less information about designing for mobile devices than PCs. Could you link some articles on this? As the last article I read this, but it didn't help much, because I compare too much what I learned on PCs.
Click to expand...
Click to collapse
The are no stupid questions.
Questions are asked when we do not understand something and want to learn.
There is nothing wrong or negative about asking questions.
Questions are a part of how we learn.
Cheers.
plus_rlus said:
<SNIP>
What kind of memories and storages are most commonly found in Android devices? One main memory (i.e. RAM)? One Flash memory for firmware (i.e ROM)? Another separate flash drive that acts as mass storage? Possibly SD card and USB stick as external mass storage?
What is firmware on Android devices?
What is the bootloader in (located in MBR part) on Android?
Linux is the kernel used by Android, which is started by the bootloader? After that, Android continues to boot, how?
A pile of terms, which I have ambiguities:
Bootloader; What's it like on Android? It is often characterized as hardware specific. So is it the case that the bootloader in Android is firmware? So in Android, the firmware runs the tasks of the PC world BIOS and bootloader (located in the MBR part), and then starts the Android located on the mass storage?
Recovery; What is this technically?
Android ROM; I can't understand this. As far as I know, Android is an operating system located mass storage, not Read-Only-Memory firmware.
Rooting; On a PC, we are used to the fact that the owner of the device has root rights. Is it just that the manufacturers have decided to set the default root password to some generated random string, and by default, the user only has access to the basic user account?
After the above has been answered, I would like someone to explain to me (separately) technically, starting from the hardware level (where and how), how do Android devices booting and work? Links to additional information are also welcome. hank you very much! If anyone can answer my questions, thank you very much!
Click to expand...
Click to collapse
Firmware is the hardware specific drivers, library files and other resources that are supplied by the manufacture(s) and are chipset specific.
The firmware is proprietary and normally closed source. Basically the parts that make the hardware work.
The bootloader is what actually boots the device.
This is supplied by the device manufacture(s) and is device specific.
It is separate from the system.
Recovery is a mini Android environment.
- Factory (Stock) recoveries are restricted to the user but have unrestricted (root) access to the device.
- Custom recoveries (TWRP, OrangeFox, ..) allow the user unrestricted (root) access to the device.
Android ROM (rom) is the actual system (OS) and normally you would include the version that you are running.
Stock roms - Google 12L, AOSP xx, OOS 12, MIUI xx, ColorOS xx, ...
Custom roms - Lineage 19.1, crDroid 12.1, AospExtended 12.1, ...
In computer terms it would be..
Windows 7, Linux (Fedora 34), MacOS Monterey.I am not sure what the current versions of MIUI and ColorOS are, hence the xx.
Once the bootloader boots the device, a few things can happen.
- The system boot image (system kernel) takes over and boots the device into system (rom).
- The recovery boot image (recovery kernel) takes over and boots the device into recovery (mini Android environment).
- If system fails to boot, device reboots into recovery (Recovery Party) if recovery can boot.
- If no boot image takes over, you will stay in the bootloader, reboot into some special mode or just a good old fashion boot-loop.
There have been a lot of changes to Android though the years..
Each device, manufacture, Android version.. can be different from another.
The most common bootloader is (or supports) fastboot but, this is manufacture and device specific.
Not to be confused with fastboot_d (new story that started with Android 10/11?).This has also changed though the years, some manufacture use their own variation of bootloader.
HTC had H-BOOT, Samsung does their own thing along with some other manufactures.
Rooting....
By default the substitute (switch) user su command is removed from Android.
This is what most refer to as superuser since it defaults to root user if you do not specify a substitute user.
This has been a long and changing story in the Android world also.
Old but, well worth the read.
How-To SU - [chainfire.eu] - Link
The current most popular used root solution is Magisk.
It is a little more than just su. Magisk - [GitHub] - Link
---
It might be easier if you see an actual partition table.
Nexus 7 16 Gig WiFi - [PastBin] - Link
Might as well make it an ... interesting one.
In this example, userdata only has 1.2 Gigs since the rest is used by other partitions.
userdata is mounted as /sdcard.
Save for boot, cache, system, misc, recovery and userdata.
The other partitions would be considered firmware.
When the device boots, the partitions get mounted to /dev/block.
Hope it helps more than confuse.
Cheers.
Hello,
the proper section for my question was a bit tricky for me, so I will put it in General Q&A:
I use a device that comes with File based Encryption - namely Samsung Knox.
(But its not a Samsung Knox specific question)
For that I use some AOSP based ROM - namely LineageOS
(Im not too sure if thats some LOS specific question ... )
Now: File based Encryption comes as default.
But I want some Full Disk Encryption for personal preference.
The GUI of my ROM does not allow me to change the encryption method.
Neither can I deactivate FBE in my ROM, nor can I manually put a second layer with FDE ontop.
So: How to enforce FDE?
I guess with building my own ROM it could be achieved - but obviously I look for some easier way, like shell commands to get rid of FBE first and later use regular ROM capabilities to just use the AOSP internal FDE feature.
Anyone with any experience in this field?
Thanks
IMO you had better asked for help here:
Android Q&A, Help & Troubleshooting
This forum is for all of your questions about Android Development and Hacking. If you need help troubleshooting a problem, please be as specific as possible by describing your software configuration, including the ROM, kernel, and any modifications you've done.
forum.xda-developers.com
It's simply not a choice, you have to live with whatever the ROM provides. Of course you can disable encryption and format userdata at wish unencrypted.
May I ask what's the benefit of FDE compared to FBE?
(btw Samsung encryption is unrelated to Samsung Knox)
aIecxs said:
May I ask what's the benefit of FDE compared to FBE?
Click to expand...
Click to collapse
File Based Encryption (FBE)
Note: That's became default with the release of Android 10 in 2019.
Cold Device – contains a stock background image, user data is locked, and needs bruteforce to access.
Hot Device – background image is visible, the camera is accessible, so data collection can be performed on the phone with the proper tools without knowing the passcode.
Full Disc Encryption with Secure Start-Up (FDE)
Note: That's default encryption since Android 6.
Cold Device (Samsung) – must enter the user’s password before the device will even start
Cold Device (LG) – the operating system is not fully booted without the password
Understanding how to differentiate between cold and hot devices while collecting data will help ensure you use the proper tools.
BTW:
If on device USB-Debugging is enabled - most users do so to have an emergency entrance - a hacker can use ADB to easily intrude in device's Android and deduct data even if they are FBE encrypted, whereas on a FDE encrypted device a hacker stands in front of a closed door, IMO, although ADB may be fully functional.
If you use a screen lock on your Android smartphone, full-disk encryption is enabled by default.
jwoegerbauer said:
If on device USB-Debugging is enabled - most users do so to have an emergency entrance - a hacker can use ADB to easily intrude in device's Android and deduct data even if they are FBE encrypted, whereas on a FDE encrypted device a hacker stands in front of a closed door, IMO.
Click to expand...
Click to collapse
Thx but I asked OP.
adbd is running only after userdata is decrypted, as adb_keys is located /data/misc/adb. therefore that would make no difference (btw hacker would need adbkey.pub of victims PC) despites most user don't have enabled usb debugging at all.
There is no difference between FBE and FDE on "Hot Device" after first unlock (decrypt), except that FBE is more secure before first unlock, as the moment android lock screen appears, FDE whole disk is already decrypted, while FBE is splitted in (ce) credential encrypted + (de) device encrypted storage.
the so called secure start-up is optional. most stock FDE encrypted devices are simply encrypted with default_password and are easier to break (I have done many times). In such case it's even possible to bypass screen lock by simply deleting locksettings.db, while the same would for sure destroy FBE and make files unrecoverable.
But even on secure start-up it's easier to bruteforce passphrase online, as the (encrypted) DEK is saved in userspace (crypto-footer) only, which (in theory) allows attacker to backup & restore status quo to cheat gatekeeper timeout and it's even possible to reset failed decrypts counter, while on FBE encryption key is stored in TEE only which makes it impossible to backup (and therefore harder to cheat gatekeeper).
Furthermore, once decrypted, on FDE it's (hard but) possible to recover deleted files, while on FBE it's impossible to recover deleted files per design.
I see no reason to believe FBE is less secure than FDE and don't see the benefit.
btw Samsung devices provide (FBE) "Strong protection" which is successor to (FDE) Secure startup:
-> Settings -> Biometrics and security -> Other security settings -> Strong protection
jwoegerbauer said:
If you use a screen lock on your Android smartphone, full-disk encryption is enabled by default.
Click to expand...
Click to collapse
Completely unrelated to each other. Two cases:
1) I personally prefer unlocked bootloader and use no encryption at all (I have disabled forcefully), but still I am using pin as screen lock. (there is no private data on phone, that's my decision)
2) stock devices are encrypted on first boot, even before user reaches initial setup. user can decide to just swipe with no pin at all, still device itself is encrypted, despites it has no screen lock.
This applies to both FDE and FBE.
aIecxs said:
Thx but I asked OP.
adbd is running only after userdata is decrypted, as adb_keys located /data/misc/adb. therefore that would make no difference (btw hacker would need adbkey.pub of victims PC) despites most user don't have enabled usb debugging at all.
Click to expand...
Click to collapse
Only allowed me to do you often do.
If you could boot a device into recovery mode - what method used ever - then ADB is accessible and fully functional, AFAIK.
jwoegerbauer said:
If you could boot a device into recovery mode - what method used ever - then ADB is accessible and fully functional, AFAIK.
Click to expand...
Click to collapse
Right, if you can boot into custom recovery, which avb/dm-verity prevents on locked bootloader, at least it should
Still it makes no difference to bruteforce FDE or FBE then, assuming TWRP magically bypasses gatekeeper timeout.
btw Cellebrite, isn't that the company that got hacked by signal founder after false claim they could crack signal?
https://arstechnica.com/information...turns-the-tables-on-forensics-firm-cellebrite
So I have encrypted a phone, Lineage OS 14.1 UNOFFICIAL. DIDNT WORK. i couldnt boot to os
I went into twrp made backup of everything except EFS partition,, transfered to my PC. Formatted data then even tried new OS install it does work. So i just went and tried restoring the data partition only and it restored it succesfully, when I try booting, It takes some time and encryption starts, automatically.
Is there a way to not start encryption process? Are there some files in data partition that are causing encryption process to start on boot which I can remove? Help
NOTE: TWRP did ask for my password when I was making my backup, so I entered correctly and It worked. I made a backup of that partition. My guess is that something is initiating the encryption process on boot
Encrypting is default behavior: since Android 4 it was FDE ( full-disk encryption ) , since Android 9 it's FBE ( file-based encryption ). Since Android 10 FBE is mandatory.
Encryption takes place when device gets booted if in Android's FSTAB file this flag is set.
jwoegerbauer said:
Encrypting is default behavior: since Android 4 it was FDE ( full-disk encryption ) , since Android 9 it's FBE ( file-based encryption ). Since Android 10 FBE is mandatory.
Encryption takes place when device gets booted if in Android's FSTAB file this flag is set.
Click to expand...
Click to collapse
First of all, thank you for your reply. Can I open and change that flag inside the fstab file?
android?boy said:
First of all, thank you for your reply. Can I open and change that flag inside the fstab file?
Click to expand...
Click to collapse
jwoegerbauer said:
Encrypting is default behavior: since Android 4 it was FDE ( full-disk encryption ) , since Android 9 it's FBE ( file-based encryption ). Since Android 10 FBE is mandatory.
Encryption takes place when device gets booted if in Android's FSTAB file this flag is set.
Click to expand...
Click to collapse
Also, Where is the fstab file located. I saw it in /etc/fstab, but I didnt change that partition when I restored only Data, so it must be there. Am I wrong?
To answer your question "Can I open and change that flag inside the fstab file?":
Even if fstab file is present - it's present in several locations, you cannot simply edit it and be done. You'd have to rebuild the boot image instead – a task that goes beyond the scope of this thread ( which is end-user orientated, and creating/rebuilding a boot image rather is in the domain of developers ).
first, determine encryption type.
Code:
adb shell getprop ro.crypto.type
adb shell getprop ro.crypto.state
next find the fstab. for older devices it is in boot ramdisk (needs un-/repacking boot.img with AIK)
https://forum.xda-developers.com/t/...-kernel-ramdisk-win-android-linux-mac.2073775
for (SAR) system-as-root devices it is in /vendor/etc.
assuming dm-verity is disabled and in TWRP system/vendor is mountable rw, for (FDE) full disk encryption you can replace the forceencrypt= flag with encryptable=
https://forum.xda-developers.com/t/4061571/post-82849947
for (FBE) file based encryption it is not possible to disable the first boot behaviour. either keep encryption as is (recommended) or destroy encryption completely by removing the fileencryption= flag and formatting /data.
there exist flashable zips for this
https://forum.xda-developers.com/t/...encrypt-disk-quota-disabler-11-2-2020.3817389