SD card mirror / symlink (like ts-bind, but w/o Magisk) - General Questions and Answers

Hi.
This general question has been asked a several times and has a several different and (mostly) acceptable answers, but I have a separate requirement I'd like to address. As far as I can tell, the closest tool to what I need is the ts-bind Magisk module, but since it requires the SD card to be present at boot it will not work for me.
Here's my scenario:
I've added an SD card to my device, and I'd like to use it as "Adoptable storage". However, I can't do that directly since I'd like to be able to manipulate the driver (in this case, specifically to manually suspend the device when it's not being used). ts-bind allows you to unbind the storage via a terminal, which is what I would need, but as my device is only loaded after boot that specific module will not work.
What I'm after is a way to emulate Adoptable storage via a script / symlink, but I need to be able to control it over the command line. The catch is that the mirrored drive needs to stick around, as the "source" folder is effectively unmounted when it's suspended until it's accessed again.
Ideally there would be a way to create a "replica" of the storage device (again, like symlink) and the use that with the built-in adoptable storage function.
Is there some kind of combination of symlinks / swapping apps or anything you can think of that might accomplish the above?
Cheers,
Rafael

Related

[GUIDE] MTP and PTP (Media Transfer Protocol) vs UMC (Mass Storage Class)

Here I will post the advantages and disadvantages of the two.
Advantages of UMC
UMC works at block level. It means that you access the partition from the operating system like if were local, and you have complete control of it.
For the previous reason, UMC keeps timestamps for files and folders. This is specially important for pictures, photos or videos that don't have EXIF, where you rely on file creation or modification time to arrange them and to know when were taken. MTP sets for all transferred files current system time, thus overwritting the real creation or modification time and ruining your gallery.
UMC allows the use of recovery software (Recuva) or partition software (Easeus Partition Master) and even format with a different filesystem. Without UMC you have to use android side software that usually is less powerful, needs a rooted phone, etc.
As you have total access to the partition, you can access all files there, even hidden, system, etc. With MTP you only have access to the files that the controlling operating system (android) wants. For example, with MTP, files starting with dot (.file) are not accessible. Some file formats are not allowed.
UMC is compatible with any operating system that allows connecting a FAT32 / exFAT pendrive, while MTP requires specific support. Linux and OS X may have built-in support or not.
With UMC you directly access the files, so modification is instant, and viewing. On the other side, with MTP you download it, modify it and re-upload the edited version, but you never do it directly. Even to play files you need to completely download it first. Imagine downloading a 5 GB MKV.
UMC is always the same, while for MTP there are several implementations: MTP, MTPZ (Microsoft Zune), Sony SonicStage, Apple DMAP.
UMC is faster than MTP because requires less time to initialize transfer, but isn't that big difference for an average user. The higher the number of files to be transferred and the lower the size of each file, the higher the ratio MTP/UMC.
Advantages of MTP or PTP
In MTP mode, the android device controls the input/output to the filesystem, so there is no risk of data loss because of cold disconnecting the device from USB. You work on a layer over the filesystem. With UMC you must be careful.
MTP allows you to use the sdcard from both the android phone and the computer at the same time, even to any desired number of them. On the other side, with UMC you only can do it from a device at a time, meaning that you have to unmount the sdcard from the android to view it on the computer, stopping apps, etc. Of course there are software that can force viewing UMC from both sides, but is a highway to corruption.
MTP does not show the native filesystem to the computer (it uses a hierarchery simulated by the MTP driver), so it will always be compatible. For example, on devices with the same partition for data and sdcard (like Galaxy Nexus, Galaxy S3) you are writing from the computer to an ext4 partition and you don't need windows to support it. The same if the device partition were in any imaginable filesystem, the MTP will show you it in a standard hierarchy. On the other side, with UMC your local operating system (windows, linux, mac) must support natively the filesystem of the partition or download some software that allows you to do it.
With MTP you use all capabilities and disadvantages of the device filesystem. If the filesystem is in ext4 you can copy files over 4GB to the device, that you couldn't in UMC mode because usually it will come formatted in FAT32, that is the most compatible fs for all operating systems.
MTP enables Windows Media DRM, UMC doesn't.
MTP allows the use of password for accessing the files (on compatible devices). On the other side, with UMC, of course you could use powerful tools like TrueCrypt, but you need the corresponding software on android that reads it.
How to preserve timestamps
- Use File Timestamp app. Root is required, and works recursively too.
- Use Mass Storage Mode when possible.
- zip or tar the files when sending them to the phone or receiving from.
- If the device has external sd, you can use it as a man in the middle.
- samba (smb protocol) allows preservation of timestamps, however applications for android don't support it.
- FTP allows preservation of timestamps, however applications for android don't support it.
- NFS allows preservation of timestamps, however couldn't get any application for android working properly (Servers Ultimate Pro).
- Rsync allows preservation of timestamps, however couldn't get any application for android working properly (Servers Ultimate Pro).
- adb push and pull does not preserve timestamps. Furthermore doesn't work recursively with folders.
- MTP does not preserve timestamps.
- Cloud services like Google Drive, Dropbox, etc. usually never preserve timestamps.
Questions and answers
Is possible to implement MTP on "put here your device"?​Possibly yes, if has USB. MTP works on software side, so updating your rom or installing a new one will do the job
Is possible to implement UMC on "put here your device"?​That depends on hardware mainly. If the internal sdcard and the data folders belong to the same partition, you can't. The reason is that you can't enable access to a part of a partition at block level, the whole or nothing. This is the case for Galaxy Nexus and Galaxy S3, in order to take advantage of all space, and discard the case where you have filled a partition and the other plenty of space.
Dan Morill said:
It isn't physically possible to support UMS on devices that don't have a dedicated partition for storage (like a removable SD card, or a separate partition like Nexus S.) This is because UMS is a block-level protocol that gives the host PC direct access to the physical blocks on the storage, so that Android cannot have it mounted at the same time.
With the unified storage model we introduced in Honeycomb, we share your full 32GB (or 16GB or whatever) between app data and media data. That is, no more staring sadly at your 5GB free on Nexus S when your internal app data partition has filled up -- it's all one big happy volume.
However the cost is that Android can no longer ever yield up the storage for the host PC to molest directly over USB. Instead we use MTP. On Windows (which the majority of users use), it has built-in MTP support in Explorer that makes it look exactly like a disk. On Linux and Mac it's sadly not as easy, but I have confidence that we'll see some work to make this better.
Click to expand...
Click to collapse
Sources:
Easy UMS, USB Mass Storage and Media Transfer Protocol – XDA Developer TV
DifferenceBetween: Difference Between MTP and MSC
Directions on Microsoft: What is MTP?
CrackBerry: On startup - Media Transfer Protocol
Ice Cream Sandwich supports USB mass storage after all, Galaxy Nexus does not
Issues and questions:
[Q] Hidden folders through Android MTP
Nexus 4 not showing files via MTP
[Q][MTP] Certain files/directories hidden to windows file manager?
Connecting to PC files are different!
MTP and hidden files
[Q] Hidden files and MTP/Windows
[Q] Do the S3 still using 2gb partition for data and 12 as virtual sd?
Just got meself a Nexus 7... But!
Upload to dropbox: file timestamps should be preserved
Android File Transfer - For Mac users only
Thanks for this guide !
If I understand this correctly, then devices like the Asus Transformer Eee Pad (TF300T) which has a 16GB or 32GB internal storage, PLUS a microSD card (and standard SD card slot when docked) could use UMC, if the manufacturer made it so (which they haven't), right?
invertedskull said:
If I understand this correctly, then devices like the Asus Transformer Eee Pad (TF300T) which has a 16GB or 32GB internal storage, PLUS a microSD card (and standard SD card slot when docked) could use UMC, if the manufacturer made it so (which they haven't), right?
Click to expand...
Click to collapse
The UMC should appear then for the microsd, which you can physically remove from the phone, plus the dock if existent.
For the 16/32 GB internal storage it depends if there is a dedicated partition for internal sdcard. Check it for the Eee Pad.
Well, then I assume(?) Asus were lazy or just didn't think to give us the choice, cos I only have MTP or PTP mode. I miss UMC.
invertedskull said:
Well, then I assume(?) Asus were lazy or just didn't think to give us the choice, cos I only have MTP or PTP mode. I miss UMC.
Click to expand...
Click to collapse
Well, assuming that you have your device rooted and with busybox, can you post the output of these three commands
Code:
ls -lR /dev/block
Code:
mount
Code:
su
parted /dev/block/mmcblk0
print
scandiun said:
Well, assuming that you have your device rooted and with busybox, can you post the output of these three commands...
Click to expand...
Click to collapse
Sorry, I don't have it rooted. I have been asking around if rooting / unlocking the TF300T would give me the option for UMC, but pretty much everyone told me chances are zip.
invertedskull said:
Sorry, I don't have it rooted. I have been asking around if rooting / unlocking the TF300T would give me the option for UMC, but pretty much everyone told me chances are zip.
Click to expand...
Click to collapse
Rooting can't change that because it's hardware implemented. You can't do anything about it. Probably the Eee pad transformer doesn't have a dedicated partition for sdcard.
A quick way to check it is see if the free space for the sdcard and the userdata is exactly the same, and is always like that no matter on which of the two partitions you write.
About the issue anyway, modern phones like Galaxy S3, Galaxy Nexus, Galaxy Note and probably newer like Nexus 2 and Note 2 won't have it anymore, it's just the way to go because takes all the advantage of the free space on the device. It will be the standard from now on.
EDITED:
If you want UMS on devices that don't have it, you can use DriveDroid from market (there's both free and paid). With it, create a blank file that will be used as "partition". Then you can connect your phone to your computer and put there the songs you want. Then do the same in the car. For example, in my Nexus 7 I've created a 2 GB file which serve for that purpose (takes a while when is big).
http://forum.xda-developers.com/showpost.php?p=39203658&postcount=21
I have 20.71GB free internal, and 1.89GB free on the mSD card. :/
invertedskull said:
I have 20.71GB free internal, and 1.89GB free on the mSD card. :/
Click to expand...
Click to collapse
Because the mSD is another sdcard, different than the internal, in case it has it.
Would be great if you could get the partition scheme as I requested before, or ask for it, to definitely rule it out.
scandiun said:
Because the mSD is another sdcard, different than the internal, in case it has it.
Would be great if you could get the partition scheme as I requested before, or ask for it, to definitely rule it out.
Click to expand...
Click to collapse
Is this what you need?
when i connect my device with computer(MTP conection), i saw 1drive appear, then i go inside, i saw 2drive(iternal&external with how many free space left), go inside again always empty... how to use it?
I get that also. Just go into either one (internal or external) that you want to copy your files to and do your stuff.
invertedskull said:
Is this what you need?
Click to expand...
Click to collapse
More or less. The internal sdcard and the external (microsd) are different, one has 27.15 GB and the other 29.80 GB.
Click in the More... and see if you have UMC for the external microsd
scandiun said:
More or less. The internal sdcard and the external (microsd) are different, one has 27.15 GB and the other 29.80 GB.
Click in the More... and see if you have UMC for the external microsd
Click to expand...
Click to collapse
Nope, I only have MTP or PTP. This makes me sad.
Of UMC, MPT, and PTP, which is the most benigh?
I don't know if this is the best place to ask my question, but at least you all seem to understand this all pretty well.
I plug my phone into my PC to just use ADB. Under Gingerbread, I would select "Charge only" as my USB connection type.
With ICS, there is no longer the "Charge Only" option for an USB connection to a PC. I have to pick UMC, MTP, or PTP.
Given I really don't want to use any of those, just ADB, which of the three is the most benign one?
WaltA said:
I don't know if this is the best place to ask my question, but at least you all seem to understand this all pretty well.
I plug my phone into my PC to just use ADB. Under Gingerbread, I would select "Charge only" as my USB connection type.
With ICS, there is no longer the "Charge Only" option for an USB connection to a PC. I have to pick UMC, MTP, or PTP.
Given I really don't want to use any of those, just ADB, which of the three is the most benign one?
Click to expand...
Click to collapse
Either the MTP or PTP are the safer ones. There is no possibility to select "none" like was possible in previous version as you say. The UMC has the disadvantage that if you activate the Mass Storage usually the /sdcard won't be available from ADB commands.
Anyway, the ADB is separate from those options. The ADB option is usually found under Developer options. You can use adb with any of the options you say.
Thanks a lot very very helpfull!
oops, delete post. wrong window lol
scandiun said:
Rooting can't change that because it's hardware implemented. You can't do anything about it.
...
About the issue anyway, modern phones like Galaxy S3, Galaxy Nexus, Galaxy Note and probably newer like Nexus 2 and Note 2 won't have it anymore, it's just the way to go because takes all the advantage of the free space on the device. It will be the standard from now on.
Click to expand...
Click to collapse
I thought I would let you (and anyone else reading this thread) know that your post here is wrong. It's not "hardware implemented" in the sense that these two protocols are software implementations of data transfer. In fact, in many cases, even where there is NOT a microSD card that is mounted as a separate partition from system/data, the UMS/MSC (USB Mass Storage Transfer/Mass Storage Class) can still be implemented by this procedure:
1. Root (and optionally flash a custom ROM, that can be easily modded)
2. boot into alt OS, such as Recovery or OS from alt. source (USB OTG?)
3. Partition such that you have the required "separate partition"
4. add kernel modules/libraries you want and will need for UMS/MSC
5. Boot up and enjoy UMS/MSC as well as MTP (if you don't mind crappy speed)
Done.
Also, regarding MTP/PTP (Media Transfer Protocol, which is actually different from Picture Transfer Protocol), being a future replacement for all "modern phones", I am not so sure. It is designed for that indeed, but the Android community is different from most others. I will personally be going out of my way in the future, to ensure all phones I buy have removable uSDcard slots, as long as I can still find them, because it makes recovery in a bind, much easier (like, if there's a problem establishing communications with a computer, and numerous other possibilities). There's nothing like popping in a 32 or 64 Gig card of movies, training videos, eBooks, etc., within 10 to 20 seconds, vice the hours it will typically take to transfer even 5 Gigs of videos over the MTP protocol. Again, I don't like programs handling all my media for me. I prefer to see the files where they live (a higher fidelity simulation, is afterall, a hallmark property of how sharper minds represent reality, and abstractions from it are always mere shortcuts we use in a hurry). You got an easier method to rapidly check file hashes on your phone, from your laptop? Think of it this way, saying that MTP will "replace" UMS, is like saying Apple will replace Linux.
Regards,
Paul
:good:

[Q] Why does no Mod Rom or Custom ROM allow official ext4 external?

Dear All,
I have a simple question to open a discussion here. We all know that modern android phones are having the ability to understand ext4 (and if my research is right - the internal storage in most Android 4.x phones is formatted in ext4). Why is dev or mod build like Cyanogenmod or even a OEM like Xiaomi which develops MIUI not removing emulated sdcard functionality (optionally) and instead allow users choose to reformat their actual sd to ext4 when inserted (and mark that as internal SD - to allow standard App2sd)? Thereby enabling secure app + media storage a reality? And in the same way as Internal Storage, I think we can use MTP to read the external storage (ext4) when connected to PCs.
Furthermore, we can have a setting to warn the users about this compatibility change (that once they take up/choose this method)- they will not be able to read cards in normal way when connected directly via card reader to Windows PCs because of no direct ext4 support in Windows.
Additionally, can't we at least have this as an advanced or developer only setting, if we want novice users to not mess with the functionality?
Do you all agree? Or am I having a over-simplified understanding of making such a change?
Lastly, given an option, I would definitely choose such a change at the cost of compatibility rather than paying tons of extra money for higher internal flash storage
Nobody wants to answer this? We have so many experts here - I am really surprised to see no replies

VZW GS5 Adoptable storage, how to access.

I have adoptable storage working per the S7 command line method with ADB. Now, as for how I can access it.. Since its encrypted the PC can't read it directly. Is it possible to use linux commands to create a linked folder on the phone's actual internal storage to the root of the adopted sd card? Wouldn't this cause the phone to handle the encryption work and let us have access to it? The other question is would this created link survive a reboot? I remember when I did a linked folder I had to have a script autorun on boot of the Android 3.0 tablet to recreate the link.
Nova5 said:
I have adoptable storage working per the S7 command line method with ADB. Now, as for how I can access it.. Since its encrypted the PC can't read it directly. Is it possible to use linux commands to create a linked folder on the phone's actual internal storage to the root of the adopted sd card? Wouldn't this cause the phone to handle the encryption work and let us have access to it? The other question is would this created link survive a reboot? I remember when I did a linked folder I had to have a script autorun on boot of the Android 3.0 tablet to recreate the link.
Click to expand...
Click to collapse
Since adopted storage becomes internal, it's best to think of it as you would a hard drive in your PC. You don't physically connect PCs or remove hard drives to transfer files; you networking or apps. There are a myriad ways to do this: FTP, SMB, BitTorrent Sync (my preferred way), SendAnywhere, SuperBeam, etc.
WiFi is far slower than USB3. USB3 would be my preferred method to load up music on it vs wireless methods.
Nova5 said:
WiFi is far slower than USB3. USB3 would be my preferred method to load up music on it vs wireless methods.
Click to expand...
Click to collapse
The interface is only theoretically faster. The S5 uses MTP to connect to your PC over USB, which means writes are made at the file level, not the block level. Translation: if a file has changed on the S5, your PC rewrites the entire file instead of just the changes. This makes routine USB file transfer extremely inefficient.
OTOH, using BitTorrent Sync with an 802.11ac router gives me block level write speeds of up to 16 MB/s, which is pretty good for the S5's notoriously slow SD card R/W performance. And if you have a slower router you can just let the network transfer run overnight anyway.
All fine and dandy, but not important to the answer sought. What im looking at doing is creating a symlinked "folder" that I can drop large amounts of data on, and have it sent directly to the card. Permissions are currently the issue on looking into the "private" directory on the phone for its storage. I just have to look up the command structure for the permissions changes when I get time to dig into it. Might not work depending on what protections google put in place to prevent those changes as they could compromise security.

SSH server on Android N (no root) with write access to the SD card

I want to mount all of my phone's storage read/write with ssfs, but I haven't yet found an SSH server able to do that properly. SimpleSSHD came closest, but it doesn't get write access to /storage/148C-40DE, which is my removable SD card. Also, it doesn't support setting the file attributes and date/time, which is annoying (all files I copy to the phone will have the current date/time).
It's not the protocol, because I've also tried the open source Primitive FTPD app, and it had the same problem. So back to SSH, I've tried a bunch of other free and paid apps from Play, and all had the same problem with the external SD Card, except for WiFi FTP Server, which was able to write on folders in the external SD card, but the connection kept breaking to the point of being unusable, but it did show that it's possible for a server app to offer write access to the SD card without root.
So is there a current solution to run an SSH server on Android Nougat with write access to the external SD card? The guides I've found were very old (2011).
Nothing?
I'm looking for the same thing, since I really want wireless file syncing with my microSD card, but NFS/SMB is out due to not being rooted.
This is the best one I've seen so far -- https://forum.xda-developers.com/android/apps-games/app-ssh-sftp-server-terminal-interface-t3740091 -- https://play.google.com/store/apps/details?id=net.xnano.android.sshserver
It can write to the SD card. But it does have some odd issues -- file modification times aren't kept, and mounting from windows causes an error if you have more than one root per user (workaround: I just created a user for the internal memory root and one for the SD card). But the modification time is a big problem. I haven't found any SSH server on Android that keeps modification times, where openSSH out the box does it on Linux.
In the reviews for this app someone mentioned a better app that's not on the playstore, but I couldn't find one like that.
I'm surprised no one has really gotten wireless sharing on Android working well yet (and frankly, I'm surprised it just isn't support by Android directly on the correct port numbers.

Encrypted portable microSD card in Settings; apps can't read it. Help?

Not sure if many of you are aware, but the U11 allows you to encrypt portable microSD cards in the settings, which is nice for privacy purposes. The problem is since I did that (I even rebooted after) many of my apps can't properly read or write to the microSD card. I've noticed many of said apps don't implement Google's expandable storage permissions properly either (i.e. they don't ask you to select the root directory of the microSD card in Android File Explorer so they have proper read-write access to everything.)
My questions, therefore, are:
What am I doing wrong? Is there some permission I'm missing?
Are 3rd party apps supposed to be able to work with encrypted portable storage?
Is encrypted portable storage even an AOSP feature? Or something specific to 3rd party OEMs?
Why do so many apps not implement microSD storage read/write permissions correctly?
Any ideas?

Categories

Resources