Related
So you’ve got a nice, shiny, new G1 and you’ve been hearing about all the amazing things you can do with it but you “MUST HAVE ROOT”. As far as you know, you’re not a plant (although you may feel as smart as one at this point) and beyond that, you have no clue what any of the terms or concepts mean in context.
Well, I’m bored so I’m going to try and clear some things up.
There are a lot of threads that cover each of these things but I’m going to try and put as many basics into one post as possible. Hopefully it can be a perfect start for n00bs and good reference in lieu of search for others. Please feel free to correct any semantic (or blatant) mistakes I make.
I’ll keep the glossary here and update terms as I add to this post:
Android OS - Like Windows Mobile but based on Linux, using a Java based front end.
Linux - Open Source operating system used instead of Windows XP/Vista, Mac OSX etc... it's free (as in beer).
Open Source (From Wiki) – Free and open source software, also F/OSS, FOSS, or FLOSS (free/libre/open source software) is software which is liberally licensed to grant the right of users to study, change, and improve its design through the availability of its source code.
Root (as in access)- root is like the administrator account on a windows machine (also referred to as su, or superuser). It allows you to have complete access to the underlying OS of a linux or *nix based machine. For the G1, it allows for the use of themes, native backup functionality, manually selecting which apps can utilize root access, auto-rotate screen, multi-touch in browser, moving applications/caches to the sd card etc...
Root (as in location)- the 'root' of a folder or drive is the top most area of that location. In windows, C:\ is the 'root' of your hard drive. The 'root' of your SD card just means you haven't moved into any subfolders.
Shell – (also heard as terminal, bash, command line) This is a loose definition, but it’s basically a command line to run specific actions against the OS.
Bootloader – the SPL and IPL of a flash based device. See jashu’s description here.
SPL (Secondary Program Loader) - You get to the SPL by holding the camera button while powering on your phone. This is where you flash NBH images. See bootloader above.
Recovery Mode - Holding the 'Home' key while while powering on the G1 will take you into Recovery Mode. From here you can perform a NANDroid backup, wipe your phone, access a command line and of course, flash your phone with an update.zip file.
RC## (or release candidate) – In context to the G1, it is an official release of Android from T-Mobile meant specifically for the G1 (not ADP).
ADP (Android Developer Phone) – A Google specific (or carrier non-specific) version of the G1/Dream that has root access by default and is meant for developers writing apps for the G1, or Android in general.
ADP vs. RC## - Neither RC’s or ADP versions are tied to their respective hardware. With the right bootloader, you can flash an ADP image to a G1 or an RC image to an ADP.
JFV1.## - Is a specific Version of a JesusFreke ROM. JesusFreke is a developer on this website that has graciously spent his time to modify the G1 OS to allow us to have root access to our phones. This gives us the ability to explore and modify our phones via a command line.
Cupcake – a development branch of the Android OS that contains many improvements that was merged into the master build of Android and is currently being released to new phones as Android 1.5.
Nandroid – a utility, accessible through Recovery Mode, that allows you to backup your phone and restore to the exact condition at backup.
Apps2SD – Applications moved to your SD card instead of internal memory. Some people like the extra room, some people don’t want to hassle with the partitioning.
Partition – just like the partitions that separate cubicles in an office, a partition separates parts of a drive.
File system – there are many. It’s basically a specific way of organizing data on a partition. FAT(32) is generally windows, ext2 is generally linux. This is not a hard and fast rule, just most common in context with what you’ll see here.
Scripts – scripts are text files that contain a list of commands to perform. Instead of typing each command out multiple times, a script can be run that will initiate all steps listed in the script.
Android SDK (System Developer’s Kit) – This includes all tools (sans fastboot) that a developer needs to create applications for the G1. It also has tools for interacting with the phone via a command line (ADB).
ADB - is a part of the SDK that allows you to run commands against the G1 in lieu of using the terminal on the phone itself.
Fastboot - is a tool used to flash system images (.img files) to the G1 from a command line on your pc. IMG files are created when you do NANDroid backups and official images can be downloaded from HTC as well. To get to fastboot mode on your phone, hold the back button while powering on.
When T-Mobile first released the G1, they left a bug in the Android OS that allowed anything typed on the keyboard to be passed on to a root shell running in the background. This really was a major flaw and needed to be patched. Unfortunately, when they patched it, they really patched it. RC29 was the last version that still had root. With all versions RC30 on, it was removed. It completely denied us any hope at modding our “open-source” phone.
Somehow, the base image for RC29 (dreaimg.nbh) was leaked and some enterprising developers were able get access to the bootloader and return an updated G1 (RC30+) to RC29 and use this to regain root.
Somewhere along this road, Google released the ADP (Android Developer Phone), which has root enabled and uses a specific SPL (EngineeringSPL) that was the base for the modified HardSPL that most of us use now. Nandroid was included to allow us to back up our phones and shortly after, JesusFreke modified RC30 to keep root and still provide the fixes and improvements that came with it.
I’m not sure where it all started, but eventually, LucidREM released a modified version of JesusFreke’s ROM. This made moving applications to SD painless and freed up system storage and now we can have 32 flashlights and 62 tip calculators installed all at once.
Apps2sd has been the bane of many peoples existence. It requires you to partition your SD card in to separate file systems (FAT32 to remain compatible with windows computers as a mass storage device and ext2 to maintain compatibility with the underlying linux OS of the G1). It also requires you to move your apps to the SD card and then create symbolic links (similar to a windows shortcut) from the internal location pointing to the SD card. Lots can go wrong in this process and that’s why LucidREM, MartinFick, MarcusMaximus04 and others have created tools to help simplify the process.
Now of course, to achieve any of the things you want to do with the G1, you have to interact with it. There are at least 2 main ways to do this. Some prefer to do everything from the phone itself using a terminal, while some prefer to use their PC with the phone connected via USB. Others avoid both of these, as best they can, and use other peoples apps or scripts anywhere they can get away with it. This is why you may find many different explanations of the same goal.
In order to interact with your phone from a pc, you need the Android SDK, which includes ADB. ADB is basically a linux shell that communicates with the G1. It is easier to copy and paste from threads and insert commands without worrying about making typos. It also requires it’s own bit of hoops to jump through (unless you use a mac or linux ;-)) and sometimes scares people away. It is highly recommended if you plan on hacking at your phone with any regularity.
Of course, there are some sadists (I once was one) that like to type line after line of code on a tiny keyboard and use the terminal directly from the phone. This is fun and it makes you feel 1337, but it also leaves a lot of room for error. Remember, you are a root user now, and any mistake you make can be potentially huge.
Well, that’s all for now folks. Please feel free to add, subtract, reorganize, correct anything I’ve said, in the comments. Also, I’ve tried to add links to any relevant threads and sources that I used in making this… this, whatever you wanna call it.
Thanks to everyone in this community for doing what you do. We are all geeks and enjoy doing this stuff. It's good to have so many talented people taking an interest in Android and the G1 in general. It is open source communites that keep technology interesting and exciting.
Thanks to Haykuro, TheDudeofLife, all the theme devs, and all the big players that I didn't reference in this post. Oh, and SolemWishing for the Timeline! It helped, thanks!
Reserved for future posting
Awesome post for nuubs. This should be permanently stickied!
Very cool. a couple terms you should add:
Fastboot
SPL
Recovery mode
(including what key strokes you need to hit to get into spl and recov. modes)
Thanks for the feedback!
I added SPL and Recovery... let me get my facts straight on fastboot and I'll add that tomorrow.
I don't know if it is the right place for it, but there seem to have been lots of question about "radio" or more specifically "radio update".
I feel smarter already.
I particularly found the file system explanation useful, i put it together that fat32 and the other were the two partitions but didnt realize which was for cpu. Not ready to attempt but definitely closer (although Im not even sure if I want to partition i have no need for all that space at this point) It doesnt affect performance does it?
Agreed. Good stuff. Definitely noob required reading material.
Yes, indeed a very nice guide for the beginners. Hell we ALL started that way...i remember when I first got this phone ~6 months ago (no root), and there was almost nothing about it, no support, no add-ons, no hacks, nothing. It was boring, and for me I was coming from a motorola (motomodders?), so going to something that was far superior but didnt have community support made me almost cry.
Though look now, 3 months later the market was filling up and being abundent of new stuff to play with (I didnt even try rooting for a while, until it became a lot more well-known [fixes and the likes], and themes became a necessity because they started to get really good), and now 6 months later people are hacking away figuring out soo much stuff about it. Amazing work everyone, seriously.
Something good: It all starts at the roots .
Well, I added info on fastboot and exceeded my 10000 character limit. Now I understand why so many people reserve the second post. lol...
I'll make some changes so I can add info about the radio, however the link to fastboot explains the radio fairly well.
Thanks for the feedback everybody, I hope this helps some people out.
skri11a said:
So you’ve got a nice, shiny, new G1 and you’ve been hearing about all the amazing things you can do with it but you “MUST HAVE ROOT”. As far as you know, you’re not a plant (although you may feel as smart as one at this point) and beyond that, you have no clue what any of the terms or concepts mean in context.
Click to expand...
Click to collapse
When i read those lines i knew that this post would be worth reading
It was a really nice n00b guide, though ive done some WiMo flashing so some of the terms sounded familliar i certanly learned a thing or two
I would say it should be stickied and put on the wiki - oh and perhaps list it in alphabetic order, it would make it more usefull as a "I dont understand this term so ill just look it up"-thread...
Perhaps you can get a mod to give you post #2 & #3
//M
DMaverick50 said:
I feel smarter already.
I particularly found the file system explanation useful, i put it together that fat32 and the other were the two partitions but didnt realize which was for cpu. Not ready to attempt but definitely closer (although Im not even sure if I want to partition i have no need for all that space at this point) It doesnt affect performance does it?
Click to expand...
Click to collapse
I'm glad it helped. As far as performance issues, I've had none. In fact, when I was skating along with 12-19MB of free space, my phone would crall and cause me all sorts of grief. Since I've moved the apps and dalvik-cache, it's been very responsive and reliable. That being said, get a GOOD sd card. I see a lot of people having problems using cheap or < class4 sd cards.
m.klinge said:
When i read those lines i knew that this post would be worth reading
It was a really nice n00b guide, though ive done some WiMo flashing so some of the terms sounded familliar i certanly learned a thing or two
I would say it should be stickied and put on the wiki - oh and perhaps list it in alphabetic order, it would make it more usefull as a "I dont understand this term so ill just look it up"-thread...
Perhaps you can get a mod to give you post #2 & #3
//M
Click to expand...
Click to collapse
lol... I'm glad you liked it.
And thanks for the tips. I'm gone for the weekend, but I'll alphabetize it when I get back. Not sure what I can do about getting it stickied but I'll look into the wiki on monday too.
can you add busybox
in the nandroid instructions it requires busybox but I didnt see an explanation for what busybox is. Thanks and this thread has already been very helpful for me
speaking of stickying this....
who is in charge of stickies? A lot of stickable topics aren't stickied and a lot of topics that should be unstickied are still stuck...
DMaverick50 said:
in the nandroid instructions it requires busybox but I didnt see an explanation for what busybox is. Thanks and this thread has already been very helpful for me
Click to expand...
Click to collapse
Hey sorry for the late reply... Been pretty busy since Friday.
I'll try and add this to the first post shortly. I may have to remove some of the history lesson if I can't get a mod to give me the second/third post.
BusyBox - This is a single executable utility that contains many common Linux commands, instead of having an individual executable for each command. As far as I know this is built into all of the JF releases, as well as Dude's. It is also usable in the JF recovery console by hitting alt-x. To use busybox, just type "busybox" in front of the command you want to use (i.e. #busybox ls -L --to get a list of your directory).
AbsoluteDesignz said:
speaking of stickying this....
who is in charge of stickies? A lot of stickable topics aren't stickied and a lot of topics that should be unstickied are still stuck...
Click to expand...
Click to collapse
If you figure this out, let me know. I haven't tried to PM a mod or anything but I'd still be curious to know what the magic requirements are.
This will definitely help a lot of newcomers. Thanks for taking the time to make it.
Way to get stickied! Now I can stop copying pasting updates worrying I might not be able to find the post...
skri11a:
BusyBox - This is a single executable utility that contains many common Linux commands, instead of having an individual executable for each command. As far as I know this is built into all of the JF releases, as well as Dude's. It is also usable in the JF recovery console by hitting alt-x. To use busybox, just type "busybox" in front of the command you want to use (i.e. #busybox ls -L --to get a list of your directory).
Click to expand...
Click to collapse
Wouldn't "Unix utilities" be more correct than "Linux commands"?
And saying Android is "like windows mobile" is blasphemy! You can't compare Unix to Micro$oft Windoze (Yes, that's a very sophisticated and mature way to express my hatred of the OS )! >:|
Anyway, it's good that you wrote this, theres probably a lot around here not having a clue about how things work in the world of unix
So, I've been at this for about 48 hours now (not continuously, but closer than you might think) and I figured I should take a break from modifying project files and puzzling over alignment issues to discuss the project, share some of the problems I've been having and ask if anybody can help, and so on.
The general idea is "Chromium build for Windows (on x86/x64) and build on ARM (for Linux), so there must be a way to build it for Windows on ARM". For the most part, that even looks like it's true. Probably at least 80% of 654 Visual Studio projects (no, that's not a joke) either build just fine with only minor amounts of work, or are things that we don't actually need (I'll try building the test suites... once everything else builds!!)
Areas that have given me problems (caution: some chance of brief rants ahead):
v8. Less than you might think, though. Setting the flags for Arm seems to have been enough.
Sandbox. There's a fair bit of thunking coded in assembly going on in the sandbox for x86. Not sure what's up with it (I don't know exactly how the Chromium sandbox works) but it'll have to come out or be replaced. The Linux (including ARM) sandbox seems to be SELinux-based, which doesn't help at all.
Native Client (NaCl). I think all the assembly is in test code, though, so I may just boldly #ifdef if all away.
libjpg-turbo (libjpg). Piles of carefully optimized assembly... for x86 and x64. There is a set of ARM assembly (for Linux) that Visual Studio won't compile, but something else might... or I may tweak until it works. Of course, I could also just accept the speed hit and use the version of libjpg implemented in nice, portable C.
Anything where the developers tried to use some SSE to speed things up. I may be able to replace it with NEON code, or I may just remove it and hope **** doesn't break. We'll see.
Inline assembly in general. Even when it's ARM assembly, Visual Studio / CL.exe don't want anything to do with it (__asm is apparently now an invalid keyword). I suspect I'll have to just pull the assembly out into stand-alone functions in their own files, then compile them to object files and link them back in later. If I can figure out the best way to do this (for example, I'll want to inline the asm functions) then it shouldn't impact performance. Seriously though, I kind of hate inline assembly. I can read assembly just fine, but I'm usually staring at it in a debugger or disassembly tool, not in the middle of source code I'm trying to build...
Everywhere that the current state of the CPU is cared about (exception and crash handlers, in particular) because the CONTEXT structure is, of course, CPU-specific. They're pretty easy to get past, though.
Low-level functions, like MemoryBarrier. Fortunately, it's implemented in ntdll.h... but as a macro, which breaks at least half the places it's referenced. Solution: where it breaks things, undefine the macro and just have it be an inline function that does what the macro did.
Running out of memory. Not even joking... well, OK, a little bit. I've got 32GB; I won't actually run out. Both Visual Studio and cl.exe do at times, though!. Task Manager says VS is currently using 1,928 MB, and before I restarted it, it broke 2.5GB private working set. Pretty good for a program that for some reason is still 32-bit...
Goddamn compiler flags. Seriously, every single project (I mentioned there are over 600, right?) has its LIBPATHs hardcoded to point at x86. Several projects have /D:_X86_ or similar (that's supposed to be set by the build tools, not the user, you idiots...) which plays merry hell with the #ifdef guards. Everything has /SAFESEH specified, not in the actual property table where the IDE could have removed it (unneeded and invlaid on ARM) but in the "extra stuff we'll pass on the build command line" field, which means every single .EXE/.DLL project must be modified or the linker will fail.
My current biggest goal is the JPG library; nobody wants to use a browser without it. After that, I'll tackle the sandbox, leaving NaCl for last... well, last before whatever else crops up.
Anyhow, thoughts/comments/advice are welcome... in the mean time, I'm going to go eat something (for the first time in ~22 hours) and then get some sleep.
Kudos for having the patience to look though this monster.
It's my understanding that NaCl is still a pretty niche thing at the moment. Is it possible to easily either disable it or completely hack it out, or do other more critical parts of Chromium now depend on it?
I don't think anything truly depends on it. I'll look in the VS dependency hierarchy and see how many things list it, and how awful it would be to remove them.. after I get the other stuff working. I may pass on the sandbox as well, if possible; it makes the security guy in me cringe something awful, but as they say, shipping is a feature..
great
Please make that happen !
Working on it! I've gotten over half of the projects to build and link, but some other stuff is adamantly refusing to work. I'm beginning to suspect I'll need to work from the other direction - rather than starting at the bottom and building all the dependencies, then combining them into browser components, and then eventually combining all the components into a complete piece of software, I may have to work from the top, removing components until the whole thing builds (at which point it will likely be useless, or all-but) and then seeing what I can add back in. I thought it would be faster to just assume everything can be made to work and only exclude something if it proved intractable, but at this point I've got a ton of very small components and almost no ability to combine them.
It would also help if VS was better at managing such truly immense tasks. For example, I have no simple graph of what all is and is not building, so I'm being forced to manually map that onto the VS dependency tree and see what is blocking a given component from building successfully, and how much is dependent upon it, one erroring project at a time (and there are a *lot* of erroring projects - my last attempt to build any substantial part of the system saw 50 of 400 projects fail).
GoodDayToDie said:
Working on it! I've gotten over half of the projects to build and link, but some other stuff is adamantly refusing to work. I'm beginning to suspect I'll need to work from the other direction - rather than starting at the bottom and building all the dependencies, then combining them into browser components, and then eventually combining all the components into a complete piece of software, I may have to work from the top, removing components until the whole thing builds (at which point it will likely be useless, or all-but) and then seeing what I can add back in. I thought it would be faster to just assume everything can be made to work and only exclude something if it proved intractable, but at this point I've got a ton of very small components and almost no ability to combine them.
It would also help if VS was better at managing such truly immense tasks. For example, I have no simple graph of what all is and is not building, so I'm being forced to manually map that onto the VS dependency tree and see what is blocking a given component from building successfully, and how much is dependent upon it, one erroring project at a time (and there are a *lot* of erroring projects - my last attempt to build any substantial part of the system saw 50 of 400 projects fail).
Click to expand...
Click to collapse
I thinkt tht is a mutch better taktic and mutch less frustrading.
I would love to see just a minimal version of it. After that all the small componens can follow.
50 of 400 is pretty good i think. Better then i expected
Bear in mind that the entire thing is 650 projects. If 50 fail at that level, many of the higher-level ones (dependent upon the lower-level) will fail too. I'll see what I can do. I may or may not be able to get v8 actually working (without it, the JS speed will be very bad, think IE8 at best) and I may have to fall back to the legacy libjpeg (which will cut JPEG render speeds by at least a factor of 2). Skia (2D drawing library used by Chrome) has a bunch of assembly optimizations that I need to get it to use the Arm version of instead. There's a couple of total hacks with the library files I've had to pull, which may or may not result in a working final build. We'll see.
GoodDayToDie said:
Bear in mind that the entire thing is 650 projects. If 50 fail at that level, many of the higher-level ones (dependent upon the lower-level) will fail too. I'll see what I can do. I may or may not be able to get v8 actually working (without it, the JS speed will be very bad, think IE8 at best) and I may have to fall back to the legacy libjpeg (which will cut JPEG render speeds by at least a factor of 2). Skia (2D drawing library used by Chrome) has a bunch of assembly optimizations that I need to get it to use the Arm version of instead. There's a couple of total hacks with the library files I've had to pull, which may or may not result in a working final build. We'll see.
Click to expand...
Click to collapse
the v8 engine ( used in nodejs ) has been ported to ARM :
I still can't link : htt p://ww w.it-wars.com/article305/compiler-node-js-pour-arm-v5
perhaps it will help you
Edit : oups, I just see that another great user of this forum made the port of nodejs to RT
Yep... but they did it without v8. That's not an encouraging result, but I feel like I'm so close...
Is there a GitHub repo so we can help or track the progress of the project ?
Sorry, not at present. There probably should be. The sheer size of the codebase is incredible (about 2.4GB) and having some way to share it practically would be good.
Also, I suspect this would go a lot faster if I don't have to repeat the work of others. I know that there's a working Webkit DLL out there, for example (though with several features, including the V8 JS engine, missing) and if I could get my hands on that it would drastically reduce the number of additional components I need to build. Currently I'm working on the sandbux, but expect that I will need to rip the whole thing out and basically have the browser run as though it was always passed the --no-sandbox parameter, at least for the first build. Too damn much assembly.
http://www.engadget.com/2013/01/22/google-chrome-native-client-arm-support/
This wouldn't have any impact on this project, would it?
Sent from my SCH-I535 using xda-developers app, complete with annoying signatures.
It probably means that NaCl on Windows RT will be possible in the future. At present, I'm cutting it out of the build - too much x86-specific stuff there to port it over myself, and it owuldn't be able to run x86-compiled NaCl code anyhow.
You might have bit off more than you could chew. It'd better if you put your current progress under version control on some public site so that other people may be able to help you.
It's a big and complex project. You are taking a lot of time, and understandably so. But just open up to other people and you could get this done faster.
Yeah, this is probably true. My life also got unexpectedly *busy* in the last week; a couple weeks ago I had many times as much free time as I do now, and so porting has slowed down.
My upload speed would take ages (literally probably at least a day of solid activity; it's embarassingly slow) to push the full source anywhere, but I may make the effort anyhow. I'll have to post it somewhere for GPL compliance in any case...
You may upload only the diff files, they'll probably be smaller then the whole distribution.
Not to pour cold water on you however, IE10 is already faster than the latest Chrome build in Windows Phone, Windows 8.
I don't see the point of this.
I have personally jumped from IE8 > FF > Chrome and finally back to IE10 over the years depends on its usability, smoothness, speed, etc
Speed isn't the only reason to use a browser. I actually prefer IE myself, but there are some things that other browsers do better than it (in the case of Chrome, parts of HTML5, the syncing across Google services, etc.) Also, Chrome gets updated far more often than IE; IE9 was equal with Chrome on speed at its release, and was far behind by the time IE10 came out.
The reason for this project, though, is a mixture of interest in what it takes, and a desire to benefit the community. Microsoft has deeped that only software which they have blessed may run on the Windows RT desktop. I disagree, and have chosen (among several other things) to port a web browser because I feel that it's important for users to have choice.
LastBattle said:
Not to pour cold water on you however, IE10 is already faster than the latest Chrome build in Windows Phone, Windows 8.
I don't see the point of this.
Click to expand...
Click to collapse
Some websites do not get along with the trident rendering engine. Some webdevs are so "Oh f*** IE I don't care" and block access to features just because it is IE. I have experienced this first hand on IE10 on my surface where it tells me to come back when I have a decent browser, only to not have the choice to do that.
This really isn't the webdevs fault either, for years IE was the scum of the internet, only recently has IE caught up to the rest of the browsers (and in my opinion exceeded some) but the years of IE being bad have left a lot of disjointed webdevs who won't even consider giving the latest IE a chance.
I implemented a browser based encryption solution which runs on Windows RT (and many other Windows computers). All I wrote was the HTML page, I am leveraging Crypto.JS javascript library for encryption algorithm. I am using the HTML 5 File API implementation which Microsoft provides for reading and writing files.
I make no claim on this but seems to work good for me. Feel free to feedback if you have any suggestions. The crypto.js library supports many different algorithms and configuration so feel free to modify it to your own purposes.
You can download the zip file to your surface, extract it and load the TridentEncode.htm file into Internet Explorer.
If you want to save to custom directory you probably need to load it from the Desktop IE instead of metro IE (to get the file save dialog). I usually drag and drop the file onto desktop IE and from there I can make favorite. This should work in all IE 11 and probably IE 10 browsers... if you use other browsers you may need to copy paste into the fields since the File API implementation seems rather browser specific. Running the html page from the local filesystem means that there is no man-in-the-middle which helps eliminate some of the vulnerabilities of using a javascript crypto implementation. You could also copy the attached zip file to your skydrive to decrypt your files from other computers.
Skydrive files in theory are secure (unless they are shared to public) so this might be useful for adding another layer of protection to certain info.
Again, use at your own risk, but feel free to play around and test it, and offer any suggestions or critiques of its soundness, or just use it as a template for your own apps.
Ok... this is really cool! Nice idea, and a good first implementation.
With that said, I have a few comments (from a security perspective). As an aside, minimized JS is the devil and should be annihilated with extreme prejudice (where not actually being used in a bandwidth-sensitive context). Reviewing this thing took way too long...
1) Your random number generation is extremely weak. Math.random() in JS (or any other language I'm aware of, for that matter) is not suitable for use in cryptographic operations. I recommend reading http://stackoverflow.com/questions/4083204/secure-random-numbers-in-javascript for suggestions. The answer by user ZeroG (bottom one, with three votes, as of this writing) gets my recommendation. Unfortunately, the only really good options require IE11 (or a recent, non-IE browser) so RT8.0 users are SOL.
NOTE: For the particular case in question here (where the only place I can see that random numbers are needed is the salt for the key derivation), a weak PRNG is not a critical failing so long as the attacker does not know, before the attack, what time the function is called at. If they do know, they can pre-compute the likely keys and possibly succeed in a dictionary attack faster than if they were able to generate every key only after accessing the encrypted file.
2) Similarly, I really recommend not using a third-party crypto lib, if possible; window.crypto (or window.msCrypto, for IE11) will provide operations that are both faster and *much* better reviewed. In theory, using a JS library means anybody who wants to can review the code; in practice, the vast majority of people are unqualified to either write or review crypto implementations, and it's very easy for weaknesses to creep in through subtle errors.
3) The default key derivation function (as used for CryptoJS.AES.encrypt({string}, {string})) is a single iteration of MD5 with a 64-bit salt. This is very fast, but that is actually a downside here; an attacker can extremely quickly derive different keys to attempt a dictionary attack (a type of brute-force attack where commonly used passwords are attempted; in practice, people choose fairly predictable passwords so such attacks often succeed quickly). Dictionary attacks can be made vastly more difficult if the key derivation process is made more computationally expensive. While this may not matter so much for large files (where the time to perform the decryption will dominate the total time required for the attack), it could matter very much for small ones. The typical approach here is to use a function such as PBKDF2 (Password-Based Key Derivation Function) with a large number of iterations (in native code, values of 20000-50000 are not uncommon; tune this value to avoid an undesirably long delay) although other "slow" KDFs exist.
4) There's no mechanism in place to determine whether or not the file was tampered with. It is often possible to modify encrypted data, without knowing the exact contents, in such a way that the data decrypts "successfully" but to the wrong output. In some cases, an attacker can even control enough of the output to achieve some goal, such as compromising a program that parses the file. While the use of PKCS7 padding usually makes naïve tampering detectable (because the padding bytes will be incorrect), it is not a safe guarantee. For example, a message of 7 bytes (or 15 or 23 or 31 or any other multiple of 8 + 7) will have only 1 byte of padding; thus there is about a 0.4% (1 / 256) chance that even a random change to the ciphertext will produce a valid padding. To combat this, use an HMAC (Hash-based Message Authentication Code) and verify it before attempting decryption. Without knowing the key, the attacker will be unable to correct the HMAC after modifying the ciphertext. See http://en.wikipedia.org/wiki/HMAC
5) The same problem as 4, but from a different angle: there's no way to be sure that the correct key was entered. In the case of an incorrect key, the plaintext will almost certainly be wrong... but it is possible that the padding byte(s) will be correct anyhow. With a binary file, it may not be possible to distinguish a correct decryption from an incorrect one. The solution (an HMAC) is the same, as the odds of an HMAC collision (especially if a good hash function is used) are infinitesimal.
6) Passwords are relatively weak and often easily guessed. Keyfiles (binary keys generated from cryptographically strong random number generators and stored in a file - possibly on a flashdrive - rather than in your head) are more secure, assuming you can generate them. It is even possible to encrypt the keyfile itself with a password, which is a form of two-factor authentication: to decrypt the data that an attacker wants to get at, they need the keyfile (a thing you have) and its password (a thing you know). Adding support for loading and using keyfiles, and possibly generating them too, would be a good feature.
The solutions to 3-5 will break backward compatibility, and will also break compatibility with the default parameters for openssl's "enc" operation. This is not a bad thing; backward compatibility can be maintained by either keeping the old version around or adding a decrypt-version selector, and openssl's defaults for many things are bad (it is possible, and wise, to override the defaults with more secure options). For forward compatibility, some version metadata could be prepended to the ciphertext (or appended to the file name, perhaps as an additional extension) to allow you to make changes in the future, and allow the encryption software to select the correct algorithms and parameters for a given file automatically.
Wow thanks GDTD that's great feedback
Not sure about his minified sources, the unminified aes.js in components is smaller than the minified version (which I am using) in rollups. I'll have to look into what his process for 'rollup' is to see if I can derive a functional set of non-minified script includes. If I can do that it would be easier to replace (what I would guess is) his reliance on Math.random.
His source here mirrors the unminified files in components folder : https://code.google.com/p/crypto-js/source/browse/tags/3.1.2/src
msCrypto that would be great, I had no idea that was in there. I found a few (Microsoft) samples so I will have to test them out and see if I can completely substitute that for crypto.js. Would be more keeping in line with the name I came up with.
Currently this version only works for text files, I am using the FileAPI method reader.readAsText(). I have been trying to devise a solution for binary files utilizing reader.readAsArrayBuffer but as yet I haven't been able to convert or pass this to crypto.js. I will need to experiment more with base64 or other interim buffer formats (which Crypto.js or msCrypto can work with) until I can get a better understanding of it.
Metadata is a great idea, maybe i can accommodate that with a hex encoded interim format.
You seem extremely knowledgeable in the area of encryption, hopefully i can refine the approach to address some of the issues you raised by setting up proper key, salt, and IV configuration... I'm sure I will understand more of your post as i progress (and after reading it about 20 times more as a reference).
Too bad we don't a web server for RT, that would at least open up localStorage for json serialization (mostly for other apps I had in mind). I guess they might not allow that in app store though. Could probably run one of a developers license though (renewed every 1-2 months)?
nazoraios said:
Too bad we don't a web server for RT, that would at least open up localStorage for json serialization (mostly for other apps I had in mind). I guess they might not allow that in app store though. Could probably run one of a developers license though (renewed every 1-2 months)?
Click to expand...
Click to collapse
I cant comment too much on the encryption, GoodDayToDie has covered anything I could contribute and more. But there is a functioning web server on RT. Apache 2.0 was ported: http://forum.xda-developers.com/showthread.php?t=2408106 I dont know if everything is working on it, I dont own an RT device and last time I tried I couldnt get apache to run on 64 bit windows 8 anyway (needed it at uni, spent hours going through troubleshooting guides and it never worked on my laptop, gave up and ran it under linux in virtualbox where it took 2 minutes to have functioning the way I needed it to).
Curious about the performance. Speaking of encryption, 7-Zip has it built-in, and from the discuss in StackExchange, it seems pretty good.
One of the neat things about this thing (local web app? Pseudo-HTA (HTml Application)? Not sure if there's a proper name for such things) is that it runs just fine even on non-jailbroken devices. That's a significant advantage, at least for now.
Running a web server should be easy enough. I wrote one for WP8 (which has a subset of the allowed APIs for WinRT) and while the app *I* use it in won't be allowed in the store, other developers have taken the HTTP server component (I open-sourced it) and packaged it in other apps which have been allowed just fine. With that said, there are of course already file crypto utilities in the store anyhow... but they're "Modern" apps so you might want to develop such a server anyhow so you can use it from a desktop web browser instead.
Web cryptography (window.crypto / window.msCrypto) is brand new; it's not even close to standardization yet. I'm actually kind of shocked MS implemented it already, even if they put it in a different name. It's pretty great, though; for a long time, things like secure random numbers have required plugins (Flash/Java/Silverlight/whatever). Still, bear in mind that (as it's still far from standardized), the API might change over time.
Yep, I think of them as Trident apps since trident is what Microsoft calls their IE rendering engine, but I guess they are sort of offline web apps (which come from null domain). Being from null domain you are not allowed to use localstorage which is domain specific. You also are not allowed to make ajax requests. You just have file api and json object serialization to make do with I/O.
Another app I am working on is a kind of Fiddler app similar to http://jsfiddle.net/ where you can sandbox some simple script programs.
Kind of turning an RT device into a modern/retro version of a commodore 64 or other on-device development environments. Instead of basic interpreter you've got your html markup and script.
I have an attached demo version which makes available jquery, jquery-ui, alertify javascript libraries in a sandbox environment that you can save as .prg files.
I put a few sample programs in the samples subfolder. Some of the animation samples (like solar system) set up timers which may persist even after cleared so you might need to reload the page to clear those.
It takes a while to extract (lots of little files for all the libraries) but once it extracts you can run the html page and I included a sample program 'Demo Fiddle.prg' you can load and run to get an idea.
I added syntax highlighting editors (EditArea) which seems to work ok and let's you zoom each editor full screen.
The idea would be to take the best third party javascript libraries and make them available and even make shortcuts or minimal API for making it easier to use them. Common global variable, global helper methods, ide manipulation. I'd like to include jqplot for charting graphs, maybe for mathematical programs and provide api for user to do their own I/O within the environment.
These are just rough initial demos, and obviously open source so if anyone wants to take the ideas and run with them i'd be interested in seeing what others do. Otherwise I will slowly evolve the demos and release when there are significant changes.
Hello,
I want to figure out the Samsung Accesory Protocol in order to create a "open source" Gear Manager app replacement. This thread is to ask if anyone has been trying to do the same thing as well as try to gather as much information about this protocol as possible. Generic discussion is also accepted, in case anyone has better ideas.
Right now all I know is that this protocol is based on RFCOMM, albeit it can be transported over TCP too. It has a level 1 "framing" which consists basically on
Code:
packed struct Frame {
uint16_be length_of_data;
char data[length_of_data];
}
packed struct FrameWithCRC {
uint16_be length_of_data;
uint16_be crc_of_length;
char data[length_of_data];
uint16_be crc_of_data;
}
I also know that there are various types of packets. "Hello" packets are exchanged early during the connection and contain the product name, etc. Authentication packets are exchanged right after the initial "hello" and contain some varying hashes (crypto warning!). Then the normal data packets are "multiplexed", as in usbmuxd: they have 'session' IDs which described towards which watch program they are talking with. All Hello and authentication packets are sent without CRC, but normal data packets are. The CRC implementation used is crc16, same poly as in the linux kernel.
I suspect that whatever we uncover about this protocol might be useful to e.g. pair Gear with an iPhone, with a PC, things like that.
Note: most of this comes from viewing Bluetooth logs. However it's clear that reverse engineering will be required for the cryptographic parts. In this case I believe it's legally OK to do so in the EU because it's purely for interoperability reasons. I don't want to create a competitor to the Gear2, I just want to talk to it.
Motivation: I bought a Gear2 in order to replace a LiveView that was dying (buttons wearing out, broken wriststrap clips, etc.) . I used it both for notifications as well as map/navigation.
Since I have a Jolla, no programs are available to pair with most smartwatches, but I've been developing my own so far (MetaWatch, LiveView). Thus I decided on a replacement based purely on hardware characteristics and price. Also Tizen seems more open than Android, thus I figured out it would be easier for me to adapt to the watch.
However it seems that I understimated the complexity of the protocol that connects the Gear with the GearManager. So my options in order to make use of this watch are:
Sell Gear2 back and buy something that's easier to hack (e.g. another LiveView ),
Figure out the SAP protocol and write a replacement Gear Manager app (what this thread is about),
Write replacement Tizen applications that don't use SAP. This involves writing new programs for Calls, Messages, Notifications, Alarms, Camera, watchOn, Pulse monitor, etc. i.e. a _lot_ of work if I want to exploit all features of the watch.
But at least one can reuse the existing Tizen settings app, launcher, drivers, etc. (I started porting Qt to the Gear2 with this idea)
Use a different Linux distro on the Gear 2. Such as Sailfish, Mer, etc. This involves all the work of option 3 + possibly driver work.
As of now I've not decided which option is easier for me so I'll keep trying to push them all.
javispedro said:
Hello,
I want to figure out the Samsung Accesory Protocol in order to create a "open source" Gear Manager app replacement. This thread is to ask if anyone has been trying to do the same thing as well as try to gather as much information about this protocol as possible. Generic discussion is also accepted, in case anyone has better ideas.
Right now all I know is that this protocol is based on RFCOMM, albeit it can be transported over TCP too. It has a level 1 "framing" which consists basically on
Code:
packed struct Frame {
uint16_be length_of_data;
char data[length_of_data];
}
packed struct FrameWithCRC {
uint16_be length_of_data;
uint16_be crc_of_length;
char data[length_of_data];
uint16_be crc_of_data;
}
I also know that there are various types of packets. "Hello" packets are exchanged early during the connection and contain the product name, etc. Authentication packets are exchanged right after the initial "hello" and contain some varying hashes (crypto warning!). Then the normal data packets are "multiplexed", as in usbmuxd: they have 'session' IDs which described towards which watch program they are talking with. All Hello and authentication packets are sent without CRC, but normal data packets are. The CRC implementation used is crc16, same poly as in the linux kernel.
I suspect that whatever we uncover about this protocol might be useful to e.g. pair Gear with an iPhone, with a PC, things like that.
Note: most of this comes from viewing Bluetooth logs. However it's clear that reverse engineering will be required for the cryptographic parts. In this case I believe it's legally OK to do so in the EU because it's purely for interoperability reasons. I don't want to create a competitor to the Gear2, I just want to talk to it.
Motivation: I bought a Gear2 in order to replace a LiveView that was dying (buttons wearing out, broken wriststrap clips, etc.) . I used it both for notifications as well as map/navigation.
Since I have a Jolla, no programs are available to pair with most smartwatches, but I've been developing my own so far (MetaWatch, LiveView). Thus I decided on a replacement based purely on hardware characteristics and price. Also Tizen seems more open than Android, thus I figured out it would be easier for me to adapt to the watch.
However it seems that I understimated the complexity of the protocol that connects the Gear with the GearManager. So my options in order to make use of this watch are:
Sell Gear2 back and buy something that's easier to hack (e.g. another LiveView ),
Figure out the SAP protocol and write a replacement Gear Manager app (what this thread is about),
Write replacement Tizen applications that don't use SAP. This involves writing new programs for Calls, Messages, Notifications, Alarms, Camera, watchOn, Pulse monitor, etc. i.e. a _lot_ of work if I want to exploit all features of the watch.
But at least one can reuse the existing Tizen settings app, launcher, drivers, etc. (I started porting Qt to the Gear2 with this idea)
Use a different Linux distro on the Gear 2. Such as Sailfish, Mer, etc. This involves all the work of option 3 + possibly driver work.
As of now I've not decided which option is easier for me so I'll keep trying to push them all.
Click to expand...
Click to collapse
I think your thread should probably go in the Dev section for Tizen. Have you made any development? If your want it moved, report your own post with the button in top right labeled report. You can then suggest your thread be moved to the new Tizen Development section. Ok, I wish you all the luck, you seem to be very talented programmer/dev. Thanks for your contributions.
Chris
noellenchris said:
I think your thread should probably go in the Dev section for Tizen.
Click to expand...
Click to collapse
Well, some mod already moved this thread from Development, where I originally posted it, into Q&A. This is not exactly "Tizen" development (SAP is used in may Samsung devices seemingly).
noellenchris said:
Have you made any development?
Click to expand...
Click to collapse
Yes, lots of progress. I have been able to write a program that connects to the Gear2 from my PC, succesfully "completes" the setup program and synchronizes the date&time. Things like changing the background color etc. are now trivial. I will soon port it to my Jolla.
I am now looking into how to send notifications to the watch. I've not been able to get Gear Manager to actually send any notifications (to use as "reference"), because goproviders crashes when I try to simulate notifications on my android_x86 VM
If anyone can send me an HCI / Bluetooth packet capture of their Android device while it is sending notifications to the Gear2 I would really appreciate it.
Unfortunately, the main problem here is that Samsung uses some cryptographic authentication as a form of "DRM". I am not exactly sure why.
There was no way for me to discover how the crypto worked so I took the unclean approach and dissasembled their crypto code (libwms.so). That means there's no way I would be able to distribute the code now without risking a lawsuit from Samsung.
Sadly this means that while I can distribute the protocol specifications I obtained, legally distributing "Gear Manager replacements" is probably impossible.
javispedro said:
Well, some mod already moved this thread from Development, where I originally posted it, into Q&A. This is not exactly "Tizen" development (SAP is used in may Samsung devices seemingly).
Click to expand...
Click to collapse
Ya, I was kinda in a Gear 1 mind set, and they have separate threads for Android and Tizen....
Chris
javispedro said:
Unfortunately, the main problem here is that Samsung uses some cryptographic authentication as a form of "DRM". I am not exactly sure why.
There was no way for me to discover how the crypto worked so I took the unclean approach and dissasembled their crypto code (libwms.so). That means there's no way I would be able to distribute the code now without risking a lawsuit from Samsung.
Sadly this means that while I can distribute the protocol specifications I obtained, legally distributing "Gear Manager replacements" is probably impossible.
Click to expand...
Click to collapse
I would gladly write a MIT-licensed C library implementing your protocol specifications. That would be correctly following the chinese-wall approach to reverse-engineering, right?
Anyway, AFAIK, being in Europe decompiling for interoperability purposes is allowed -- I know that wikipedia is not to be taken at face value, but: en.wikipedia.org/wiki/Reverse_engineering#European_Union
Antartica said:
I would gladly write a MIT-licensed C library implementing your protocol specifications. That would be correctly following the chinese-wall approach to reverse-engineering, right?
Anyway, AFAIK, being in Europe decompiling for interoperability purposes is allowed -- I know that wikipedia is not to be taken at face value, but: en.wikipedia.org/wiki/Reverse_engineering#European_Union
Click to expand...
Click to collapse
Well, the problem is not the protocol specifications per se, which I'm actually quite confident I'd be able to redistribute (I'm in EU). The problem is the cryptography part, which is basically ripped off from the Samsung lib "libwsm.so" . Unless we can find out what cryptographic method that lib uses, distributing alternate implementations Is a no-go.
javispedro said:
Well, the problem is not the protocol specifications per se, which I'm actually quite confident I'd be able to redistribute (I'm in EU). The problem is the cryptography part, which is basically ripped off from the Samsung lib "libwsm.so" . Unless we can find out what cryptographic method that lib uses, distributing alternate implementations Is a no-go.
Click to expand...
Click to collapse
If you have the time, I don't mind researching the possible crypto used (although I've only studied DES/3DES, AES and Serpent, hope that whatever scheme used is not very different from them).
Some ideas to start from somewhere:
1. As you have used its functions, it is a block cipher? I will assume that it is.
2. What is the key size and the block size?
3. Are there signs that it is using a stack of ciphers? (that is, applying one cipher, then another to the first result and so on)
Antartica said:
If you have the time, I don't mind researching the possible crypto used (although I've only studied DES/3DES, AES and Serpent, hope that whatever scheme used is not very different from them).
Some ideas to start from somewhere:
1. As you have used its functions, it is a block cipher? I will assume that it is.
2. What is the key size and the block size?
3. Are there signs that it is using a stack of ciphers? (that is, applying one cipher, then another to the first result and so on)
Click to expand...
Click to collapse
Hello, I've not forgotten about this, just somewhat busy and been using the MetaWatch lately
1. Yes it is clearly a block cipher, and the block size Is 16bytes.
2. I don't know about the key size, it is obfuscated.
3. Doesn't seem like a stack of ciphers. It looks like some overcomplicated AES. But to be honest AES is the only encryption I know of
By the way I think I will upload my current test "manager" source code to somewhere after removing the crypto specific files . Since the protocol itself has been obtained cleanly. Note I've used Qt (not the GUI parts) so it's useless for creating a library; the code will probably need to be rewritten to do so, but it may be useful as "protocol specs".
javispedro said:
Hello, I've not forgotten about this, just somewhat busy and been using the MetaWatch lately
Click to expand...
Click to collapse
No problem. Curiously, I've transitioned from the metawatch to the Gear1 fully (null rom, not pairing with bluetooth to the phone but gear used as a standalone device).
[off-topic]I'm not using my metawatch anymore. I was modifying Nils' oswald firmware to make it prettier and to have some features I wanted (calendar, stopwatch), but it was very inaccurate, supposedly because of missing timer interrupts (the existing LCD drawing routines were too slow). I rewrote the graphics subsystem just to stumble into a known mspgcc bug, and trying to use the new redhat's mspgcc resulted in more problems (memory model, interrupt conventions). In the end I couldn't commit enough time to fix that and my metawatch is now in a drawer[/off-topic]
Returning to the topic:
javispedro said:
1. Yes it is clearly a block cipher, and the block size Is 16bytes.
Click to expand...
Click to collapse
Good. We can at least say it isn't DES/3DES nor blowfish (64 bits block size). Regrettably there are a lot of ciphers using 128-bits block size; that I know: AES, Twofish and serpent.
Perusing the wikipedia there are some more of that size in use: Camellia, sometimes RC5 and SEED.
javispedro said:
2. I don't know about the key size, it is obfuscated.
3. Doesn't seem like a stack of ciphers. It looks like some overcomplicated AES. But to be honest AES is the only encryption I know of
Click to expand...
Click to collapse
I understand that to mean that you cannot use that library passing your own key, right?
What a pity! One way to test for these ciphers would have been to just cipher a known string (i.e. all zeroes) with a known key (i.e. also all zeroes) and compare the result with each of the normal ciphers :-/.
javispedro said:
By the way I think I will upload my current test "manager" source code to somewhere after removing the crypto specific files . Since the protocol itself has been obtained cleanly. Note I've used Qt (not the GUI parts) so it's useless for creating a library; the code will probably need to be rewritten to do so, but it may be useful as "protocol specs".
Click to expand...
Click to collapse
Perfect. I don't need anything more .
Ok, so I've uploaded my SAP protocol implementation: https://git.javispedro.com/cgit/sapd.git/ . It's "phone" side only, ie it can be used to initiate a connection to the watch but not to simulate one. In addition, it's missing two important files: wmscrypt.cc and wmspeer.cc which implement the closed crypto required to "pair" the watch. The most important file is sapprotocol.cc which implements the packing/unpacking of the most important packet types. The license of those files is GPLv3 albeit I'm very happy if you use the information contained on them to build your "Gear Manager" program under whichever license you'd prefer.
For anyone who hasn't been following the above discussion: I've figured out a large part (useful for at least establish contact with the watch and syncing time/date) of the SAP protocol used between the Gear watch and the Gear manager program on the phone. This has been done mostly by studying traces and afterwards talking to the watch using my test implementation above to figure out the remaining and some error codes. The debug messages left by the watch's SAP daemon were also immensely helpful. As long as I understand this is perfectly safe to do, publish and use as I'm in the EU and is basically the same method Samba uses.
Unfortunately, the protocol contains some crypto parts required for the initial sync (subsequent connections require authentication). However, the communication itself is not encrypted in any way, which helped a lot with the process. Because it's impossible for me to figure out whatever authentication method is used, I had to disassemble the library implementing this stuff (libwms.so). This is still OK according to EU law, but I'm no longer to release that information to the public. I'm looking for alternatives or ideas on how to handle this fact.
In the meanwhile, let's talk about the protocol. It's basically a reimplementation of the TCP(/IP) ideas on top of a Bluetooth RFCOMM socket. This means that it's connection oriented and that it can multiplex several active connections (called "sessions") over a single RFCOMM link. Either side of the connection can request opening a connection based on the identifier of the listening endpoint (called a "service"). Strings are used to identify services instead of numeric ports as in TCP. For example, "/system/hostmanager" is a service that listens on the watch side. Once you open a session towards this service (i.e. once you connect to it) you can send the time/date sync commands. In addition to be the above the protocol also seems to implement QoS and reliability (automatic retransmission, ordering, etc.). It's not clear to me why they reimplemented all of this since RFCOMM is a STREAM protocol, and thus reliability is already guaranteed!! So I've not focused much on these (seemingly useless) QoS+reliability parts of the protocol.
Let's start with the link level. There are two important RFCOMM services exposed by the watch: {a49eb41e-cb06-495c-9f4f-aa80a90cdf4a} and {a49eb41e-cb06-495c-9f4f-bb80a90cdf00}. I am going to respectively call those two services "data" and "nudge" from now on. These names, as many of the following ones, are mostly made up by me .
The communication starts with Gear manager trying to open a RFCOMM socket towards the "nudge" service in the watch. This causes the watch to immediately reply back by trying to open a connection to the "data" service _on the phone_ side. So obviously this means that your phone needs to expose the "data" RFCOMM service at least. In addition, the watch will try to open a HFP-AG connection (aka it will try to simulate being a headset) to your phone. Most phones have no problem doing this so no work is required. Of course, if your phone is a PC (as in my case ) then you'll need to fake the HFP profile. I give some examples in my code above (see scripts/test-hfp-ag and hfpag.cc).
Once the RFCOMM socket from the watch to the phone "data" service is opened, the watch will immediately send what I call a "peer description" frame. This includes stuff such as the model of the watch as well as some QoS parameters which I still don't understand. The phone is supposed to reply back to this message with a peer description of its own. See sapprotocol.cc for the packet format.
After the description exchange is done, the watch will send a "authentication request" packet. This is a 65 byte bigint plus a 2 byte "challenge". The response from the phone should contain a similar 65 byte bigint, the 2 byte response, and an additional 32 byte bigint. If correct, the watch will reply with some packet I don't care about. Otherwise the connection will be dropped. It obviously looks like some key exchange. But this is the crypto part that's implemented in libwms.so....
After these two exchanges link is now set up. The first connection that needs to be opened is towards a service that is always guaranteed to be present, called "/System/Reserved/ServiceCapabilityDiscovery". It is used by both sides of the connection to know the list of available services present on the other side. Despite this, you cannot query for all services; instead, you must always know the name of the remote service you're looking for. There's some 16-byte checksum there which I don't know how to calculate, but fortunately the watch seems to ignore it!! I suspect that you're expected to actually persist the database of available services in order to shave a roundtrip when connection is being established. But this is not necessary for normal function. This service is implemented in capabilityagent.cc, capabilitypeer.cc . This part was actually one of the most complex ones because of the many concepts. I suggest reading the SDK documentation to understand all the terms ("service", "profile", "role", etc.).
If everything's gone well, now the watch will try to open a connection to a service in your phone called "/system/hostmanager". Once you get to this message things start to get fun, because the protocol used for this service is JSON! It's implementation resides in hostmanageragent.cc, hostmanagerconn.cc . For example, Gear Manager sends the following JSON message once you accept the EULA: {"btMac":"XX:XX:XX:XX:XX:XX", "msgId":"mgr_setupwizard_eula_finished_req", "isOld":1}. At this point, the watch hides the setup screen and goes straight to the menu.
Well, this concludes my high-level overview of the SAP protocol. Hope it is useful for at least someone!
Things to do:
Personally I'm looking for some traces of the notification service. Ie the one that forwards Android notifications towards the watch. For some reason it doesn't work on my phone, so I can't get traces. I suspect it's going to be a simple protocol so a few traces will be OK. It's the only stuff I'm missing in order to be able to actually use the Gear as a proper smartwatch with my Jolla.
We still need to tackle the problem of the cryptographic parts. Several options: either "wrap" the stock libwms.so file, try to RE it the "proper way", .... I'm not sure of the feasibility of any of these.
Many other services.
javispedro said:
After the description exchange is done, the watch will send a "authentication request" packet. This is a 65 byte bigint plus a 2 byte "challenge". The response from the phone should contain a similar 65 byte bigint, the 2 byte response, and an additional 32 byte bigint. If correct, the watch will reply with some packet I don't care about. Otherwise the connection will be dropped. It obviously looks like some key exchange. But this is the crypto part that's implemented in libwms.so....
Click to expand...
Click to collapse
About that 65-byte bigint... that is a 520-bit key. The usual length of ECDSA keys is exactly 520-bits, so we may have something there: it is possible that they are using ECDSA signing (just like in bitcoin, so there are a lot of implementations of that code).
Not forgotten about this!
Just an status update:
I'm still in the process of defining the API of the C library using javispedro's sources as template.
It's tougher than I originally supposed because the C++ code has a lot of forward-declarations of classes, which is very difficult to map into C. To counter that I have to move elements between structures and I'm not so comfortable with the codebase yet.
And then there is still the hard work of translating the Qt signals/slots to plain' old callbacks... and implementing the bluetooth part using bluez API... and... well, I hope that is all.
Anyway, patience .
I've now had access to a Samsung S2 and thus I have been able to obtain more traces. The latest Git now contains code to connect to the notification manager service, thus allowing to send notifications from the phone to the watch.
That was the last missing part to be able to use the Gear 2 as a 'daily' smartwatch with my Jolla, so I've now also ported the code to run under Sailfish. In fact I'm using this setup at the moment. My first comment is "wow the vibrator IS weak".
You can find a log of sapd's (ie my code) startup qDebug() messages; they may be useful (if you can't yet get your code to run)
I suspect that there may still be some important battery issues because the watch keeps printing error messages about SAP services it can't find on the phone (and instead of sleeping, it starts busy polling for them.... :/ ). It does not seem to happen while the watch is out of the charging cradle, so it may not be important, but not sure yet.
As for the encryption, I'm not sure how to proceed. I could describe the code to you, but that would be risky, because I don't understand what it does. Thus the only way (for me) to describe it would be to pass on the mathematical formulas/pseudocode ... Apart from that, we also have the problem of the keys...
Antartica said:
The usual length of ECDSA keys is exactly 520-bits, so we may have something there: it is possible that they are using ECDSA signing
Click to expand...
Click to collapse
They do use ECDH indeed, and they link with OpenSSL and import the ECDH functions. However it's not clear if they use ECDSA; while the crypto algorithm DOES resemble DSA, I cannot fully identify it.
Congratulations for managing to make it work with the Jolla .
I have finally found a suitable "flattened" class hierarchy as to be able to map your code into C; see the attachs. Basically, I have to move the functionality of SAPConnectionRequest, SAPSocket, CapabilityPeer and SAPConnection into SAPPeer, and then it is suitable for my needs.
javispedro said:
As for the encryption, I'm not sure how to proceed. I could describe the code to you, but that would be risky, because I don't understand what it does. Thus the only way (for me) to describe it would be to pass on the mathematical formulas/pseudocode ... Apart from that, we also have the problem of the keys...
They do use ECDH indeed, and they link with OpenSSL and import the ECDH functions. However it's not clear if they use ECDSA; while the crypto algorithm DOES resemble DSA, I cannot fully identify it.
Click to expand...
Click to collapse
If you manage to describe it using mathematical formulas as in
http://en.wikipedia.org/wiki/Ellipt...ture_Algorithm#Signature_generation_algorithm
it would be perfect, but I reckon that to be able write that you need intimate knowledge of the code and don't know if you have time for that :angel:
And identifying the hash function used would be a problem in itself...
One idea: how about a ltrace so we have the calls to the openssl library? That may uncover new hints.
Anyway, I have a lot of work before me until I need that, so don't fret over it.
Hi there! Any chance that the Gear can (really) work with an iPhone?
gidi said:
Hi there! Any chance that the Gear can (really) work with an iPhone?
Click to expand...
Click to collapse
agreed. Needs iPhone support please.
Antartica said:
Congratulations for managing to make it work with the Jolla .
I have finally found a suitable "flattened" class hierarchy as to be able to map your code into C; see the attachs. Basically, I have to move the functionality of SAPConnectionRequest, SAPSocket, CapabilityPeer and SAPConnection into SAPPeer, and then it is suitable for my needs.
Click to expand...
Click to collapse
You may want to look at the official Samsung SDK docs to match their class hierarchy. I tried to match my hierarchy to theirs, but this happened very late in the development process, so there is some weirdness.
Antartica said:
One idea: how about a ltrace so we have the calls to the openssl library? That may uncover new hints.
Click to expand...
Click to collapse
I more or less know what it is doing with OpenSSL, but that's because I looked at the dissassembly. They use OpenSSL for key derivation (ECDH), but the actual cryptographic algorithm is their own. This 'block cipher' is the part they have tried to obfuscate. Not much, but still enough to require more time than what I have available It is basically a set of arithmetical operations with some tables hardcoded in the libwsm.so binary, so no external calls to any library. The hardcoded tables are probably derivated from their private key, which is most definitely not on the binary. In fact I suspect this is basically AES with some changes to make it hard to extract the actual key used, so that's where I've centered my efforts.
Technically it should not even be copyrightable, so maybe I could just redistribute my C reimplementation of the algorithm, but as with any other DRM who knows these days... and that still leaves the problem of the tables/"private key".
Digiguest said:
agreed. Needs iPhone support please.
Click to expand...
Click to collapse
Well you are welcome to implement one such iPhone program yourself. Will be happy to resolve all the protocol questions you have.
(But please stop with the nagging).
Wasn't nagging at all. Just agreeing with him. I am no programmer so I have to rely on others for answers. Sorry if you thought otherwise.
Looking for to see more work on it though. Keep it up.
Hi there! Nice work on getting Gear2 to work with Jolla.
I'd love to get Gear1 to work with WP8.1. Do you have the code for Jolla
on github/bitbucket so I could give it a peek? Thanks in advance.
Duobix said:
Hi there! Nice work on getting Gear2 to work with Jolla.
I'd love to get Gear1 to work with WP8.1. Do you have the code for Jolla
on github/bitbucket so I could give it a peek? Thanks in advance.
Click to expand...
Click to collapse
javispedro had the sources in gitorius, but they are not there anymore (surely related to gitlab buying gitorius).
I attach a tarball with javispedro sources as of 19 October 2014.
Note that it lacks the files implementing the crypto, so just porting it is not enough to be able to communicate to the gear. OTOH, I know that there are some differences in the protocol between the Android Gear1 and the Tizen Gear2 (if the gear1 has been updated to Tizen, it uses the same protocol as gear2). Specifically, to be able to communicate with both watches, the gear manager package has both gear manager 1.7.x and gear manager 2.x. javispedro's code implements the gear 2 protocol.
Personally, I have my port on hold (I have problems with bluetooth in my phone, so there is no point in porting sapd right now as I would not be able to use it).
ABANDONED
Hi! Does anyone here use Linux desktop distributions in chroot environment on Android device?
I am developing wayland protocol server for Android devices. If anyone is interested in checking my project, latest version of apk is always available here:
ftp://ftp.drivehq.com/mogryph/sparkle/
Currently I am only focused on running Xwayland as client. Also apk supports audio output.
Simplest instruction:
1. Android 6 or newer required, busybox required, root required
2. Prepare linux distribution in directory, image or on partition. Make sure you have Xwayland installed in it. Make sure you specify which DE to run (or at least xterm) in ~/.xinitrc
3. Install and start sparkle.apk
4. Press "edit user.sh", uncomment (remove #) line starting with start_generic_container. Change rest of this line to match your device:
first arg - image or partition where distribution is installed. If distribution is installed in directory and mouting is not needed, leave this arg unchanged.
second arg - mount point or directory with distribution. If you use mounting (first arg), this arg can be left unchanged.
third arg - name of the user which will be used to start Xwayland and DE. Its better to specify non-root. Also this is the user who must have .xinitrc in his home dir (see step 2).
5. Save user.sh and click "Start".
6. Any problems and crashes will be reflected in the log.
If you want audio output:
1. Compile and install driver from pcm_sparkle.tar.gz in your distribtion
2. cp 1.asoundrc ~/.asoundrc
If you have blinking problem, change upload_mode from 1 to 2 in settings. If you have bad performance, setting no_damage to true may help, but in most cases no_damage=false is better. Fastest upload mode is 0 (if it works).
If you don't trust me and don't want to give sparkle root permissions (I perfectly understand this) you don't have to. Also you can do without busybox.
But in this case, you need to understand and do a lot of things. Check sparkle's user.sh to get idea about what needs to be done. Basically:
1. You need to make /data/data/com.sion.sparkle/files accessible from inside chroot container. You can use bind bound.
2. Make sure you have tmpfs mounted over /tmp in container.
3. You may need to change selinux context on /tmp to match sparkle's context or disable SELinux.
4. You need to create new directory in /tmp, symlink sparkle's wayland socket from /data/data/com.sion.sparkle/files/wayland-0 to this dir. And export XDG_RUNTIME_DIR to point to this dir. Dir must be (ch)owned by user who will be running Xwayland and DE.
5. After all this, you can try to start Xwayland and your DE.
new version
New version
rgho.st/8Fbz64Rxj
Added x86 and x86_64 support. Actually it is rewritten almost from scratch but x86 support is the only thing others can notice...
Hello! This project is interesting. I tried you app and it works on my Xiaomi Redmi Note 4X(chromium and glmark from chrooted environment works very well)! Can you publish source code on Github, because it really interesting project?
Also I'm interested, please post it on github!
Did you put this up on github or move this thread? Looks very interesting.
1
Argh, sorry, I decided to abandon this project. You are free to delete thread. Also no copyleft-licensed components were used so I don't have to bother releasing sources.
Hentacler said:
Argh, sorry, I decided to abandon this project. You are free to delete thread. Also no copyleft-licensed components were used so I don't have to bother releasing sources.
Click to expand...
Click to collapse
Check your PM please!
1
Hello again.
For last two weeks I was rewriting it from scratch (yes. again... yes, third time).
Probably need another week to make it stable.
Currently I am not sure it runs on any device except my own 5-year old phone (LineageOS 14).
I will maintain last version here:
ftp://ftp.drivehq.com/mogryph/sparkle/
There is no English documentation, but you can see script "user.sh" to get idea about how to start xwayland. In most cases it should be enough to edit few lines in that script to make it work on another device. If you execute this script on your device with "install" argument, it is supposed to place itself into sparkle's directory and sparkle is supposed to run it ("start" function) automatically. Sparkle doesn't request root unless script does.
Here is video of sparkle working:
https://www.youtube.com/watch?v=tOSFYxCF7Q8
But it seems that KDE + video recording was too much for my old phone
Still, if you going to see video, don't close it until 2:00 where I turned of composition which caused lags.
Also on device everything looks much smoother than on video, even after 2:00.
When I watch fullscreen (1280x720) video on my device, sparkle + xwayland together add just 5% of CPU load (20% load of single core).
Thats it I guess... I tried to to discuss sparkle on 4pda.ru (russian forums), but got very bad reception. "xsdl is perfect, dont reinvent the wheel" they say. So I started to hate humanity and I decided to make sparkle personal project. Also this is last time I am solving reCAPTCHA to leave post on XDA.
Still alive
We are still alive. I've changed first post to reflect actual state. Now sparkle supports audio, auto-mouting containers and is lot more stable.
Yet there are still many things I want to improve in sparkle's core before adding new functions.
Also there are few demo videos on ftp.
Amazing!
Working great on my redmi 6 pro. Stock miui 9.9.3 rom. With linuxdeploy and sparkle from your ftp. No lag on visual and sound. My Linux distribution is alpinelinux arm64 arch.
Since first time I see your posting on 4pda. I'm interested in it. And finally it's on xda.
Thanks dev.
---------- Post added at 02:52 AM ---------- Previous post was at 02:44 AM ----------
For anyone interested in the topic. Please follow the instructions in documentation from ftp. And Translate it to eng from rus.
This sounds amazing! Just curious, is it related to https://github.com/twaik/sparkle ?
I now have it working very well on my Samsung Tab S3 using Xwayland and a tiling window manager. Firefox runs amazingly well!
Is it meant to be used only with Xwayland or will it also work with native Wayland applications?
BTW, I think if you open sourced this project and promoted it a bit, it could become quite popular. It's basically the first way to run X11 GUI applications on Android devices at full speed. If you set up a donation link, you could also get compensated for your time and effort. I'll personally contribute $20 if it's open sourced, and I'm sure others will chip in as well.
robsmith11 said:
This sounds amazing! Just curious, is it related to https://github.com/twaik/sparkle ?
Click to expand...
Click to collapse
Thanks for feedback. Nice to hear that someone managed to start this thing
Twaik's repository is clone of my very very old version of sparkle. I made that version years ago when I was just starting to learn linux and C++. Sparkle was rewritten from scratch two or three times since that version. And (I believe) current version is much better.
Regarding making it open source... Few months ago I had to find real job. Can't spend much time on personal projects any more. But I have my own strange programming style and my own vision of what sparkle should be. Not sure I want others to paint on my picture. It's probably all because of Twaik! I hate how he used old open source version of sparkle. He did terrible things to it, outraging all my beliefs Sorry!
P.S.: Yesterday I've uploaded another apk to my ftp. The file is called "sparkle-testing.apk". This version is much newer and has many fixes. But I've also changed to many things since tested version including some fundamental changes. No guarantee it will run at all on other devices. Interest is mega low and I get no test reports at all.
Hi Hentacler, I've just found your project - it looks really promising. Unfortunately, the only link currently working on this thread is to github. Is this project still live?
I have a samsung galaxy note 10+, and am using it as a laptop replacement. In addition to the android apps using Samsung Dex (Samsung's desktop solution), I have several linux distributions installed inside a chroot using userLand - so far, its working great. I'd be keen to give you project a try if it's still live, and am happy to help out with testing from my device.
Re open source - while I like your project, I'm not super interested in investing time into something that's not open sourced - I appreciate your concerns about wanting to maintain the direction, but having transparent development is pretty important to me. Is Twaik's fork of your project a better place to go?
Cheers.
tillum said:
Hi Hentacler, I've just found your project - it looks really promising. Unfortunately, the only link currently working on this thread is to github. Is this project still live?
I have a samsung galaxy note 10+, and am using it as a laptop replacement. In addition to the android apps using Samsung Dex (Samsung's desktop solution), I have several linux distributions installed inside a chroot using userLand - so far, its working great. I'd be keen to give you project a try if it's still live, and am happy to help out with testing from my device.
Re open source - while I like your project, I'm not super interested in investing time into something that's not open sourced - I appreciate your concerns about wanting to maintain the direction, but having transparent development is pretty important to me. Is Twaik's fork of your project a better place to go?
Cheers.
Click to expand...
Click to collapse
ftp://ftp.drivehq.com/mogryph/sparkle/
Link to FTP should work and there you can get two versions:
sparkle.apk - old version, but confirmed to work by 3-4 people.
sparkle-testing.apk - latest version, but only briefly tested by me.
I don't ask anyone to invest anything... Sparkle doesn't request root access or any other dangerous permissions (unless you enable automatic container mounting and starting) so it's safe to try for anyone who wants.
Btw, somewhere between these two versions I've replaced BASH container initialization script with LUA version. That was probably a bad idea. LUA script is harder to start directly as root and hacks I used may not work (currently may even cause application freeze if root access is denied). Going to revert to BASH probably. But this only touches people who want sparkle to mount container and launch everything automatically on single button press.
p.s.: Why I need to solve captcha every time I post something?
Thanks for the new release! I've updated and everything seems to be working without any changes on my Samsung Tab S3 with chroot and Arch Arm Linux.
Your changes also solved the flickering for me! The old version would flicker the screen whenever my keyboard's trackpoint activated, but it's not flickering at all any more. Performance seems to be about the same.
I think this could be quite popular, but not many people know about it. Perhaps a post on Hacker News or Reddit would raise awareness.
I understand your position on open source and maintaining control. One idea if you haven't already considered it is releasing the code with a restrictive license that forbids any forks. But either way, I'm enjoying being to properly use X11 on my tablet.
BTW, have you tried any native Wayland compositors? I don't really understand the Wayland ecosystem that well. I gave Sway a brief try, but it didn't seem to work. I've only been using XWayland.
@Hentacler Thanks for your reply! Very keen to get this working, but having a few issues. I'm unsure how to configure the user.lua file - I'm using your latest apk.
I have a non-rooted device, and am running archlinux under termux. Works fine with xsdl. I have installed xorg-server-wayland for X11. I'd appreciate any advice you have.
@robsmith11 Are you able to share how you got this working on Arch? Thanks!!!!
tillum said:
@Hentacler Thanks for your reply! Very keen to get this working, but having a few issues. I'm unsure how to configure the user.lua file - I'm using your latest apk.
I have a non-rooted device, and am running archlinux under termux. Works fine with xsdl. I have installed xorg-server-wayland for X11. I'd appreciate any advice you have.
@robsmith11 Are you able to share how you got this working on Arch? Thanks!!!!
Click to expand...
Click to collapse
I am not sure it is possible to use sparkle without root...
Sparkle makes it's directory accessible for everyone (chmod 777). Before Android 8 or 9 this was enough and xwayland from termux was able to connect to sparkle. Here is how people used to start it:
export XDG_RUNTIME_DIR=/data/data/com.sion.sparkle/files
Xwayland
But newer versions of Android brought more restrictions and termux can no longer connect to sparkle. These new restrictions are implemented using SELinux if you know what it is. Applications now have different security contexts.
But that is not all. Newest versions of android brought even more terrible meaningless restrictions effectively "killing" applications like termux and many others.
In short, from now one applications are not allowed to execute code (binary) that comes from "untrusted" sources. Termux used to download a lot of such code from it's own repositories. And now it can't. We can't even unpack binaries from assets.
So I can only help with rooted devices.
P.S. Please forgive me, but I am leaving this website. Making people solve recaptcha every time they want to post something is unacceptable level of contempt.
My mail: [email protected]
Thanks for that, will have a play. I could always just root my device. Weird about recaptcha, not having this issue. Currently through termux I have access to the whole sdcard, and am able to download packages (and distros) in it - will have a play and see what else is possible.
@tillum
I basically just followed the instructions on the first post for using Sparkle without busybox. I didn't need to modify the Lua scripts.
I'm guessing SELinux may be a problem without root. I'll try setting it up without root when I have a chance later.