There is a simple solution to that all of this problem that anyone can use. This will effectively circumvent any legal issues and keep rom updates relatively simple.
1st Problem- A developer cannot distribute Google closed source aps.. This however does not prohibit legitimate users from holding a backup copy. (Fair Use protects the end user of this). If this were not so you could not have over the air backups on your phone, (update.zip)
2nd Continual updates of closed source google apps--- The updates provided by google will be pushed to us legitimately (hence always an approved up to date source for the closed source apps for our liscenced personal devices)
Solution--
-A rom cooker simply needs to build the rom with the applications in place, test as if a complete distribution and right before packaging pull the APKs that are closed source (the Rom does not have to be functional without them.
-The user simply downloads the rom as a "kitchen" places their apk files in a folder in the kitchen.. A provided script or simple program can be put in the kitchen to add these programs back into the package in their proper locations and resign. The output saved into a folder in the kitchen and boom functional rom
---This keeps the letter of the law and spirit, and since the developer is not distributing their code they are safe because they are only distributing android open code, but since YOU hold a legitimate ROM backup provided BY Google for YOUR liscenced device you can simply place such files in the provided folder. Run script and you are done. This program would be simple to create or scripted. File version would have to be simply pre agreed upon in the post much as some developers suggest a radio version with their roms.
--If you dont have a copy or dont hold the knowledge to extract them, then I am sure some kind souls would have accidentally placed them on rapid share for your backup enjoyment... (Liability is not on developers)
I took a screen shot of what I see as a possibility, I would make the application but I am not a ROM developer and these guys clearly have some headway on me....
Look at the picture attached....
By the way this would be legal and not infringe on the law. Since the developer is only providing a "blue print for a possible rom"
Interesting...I'm not a dev...but it seems it could possibly work.
Good idea, I was literally thinking the same thing. It could definitely work.
It would be simple enough.... very simple... Unfortunatly by the time I lay done the first line of code somebody here will have created 10 versions of this.. LOL I am rusty in programming.. Which for something so simple would probably not be needed.
it would definitely make sure that people had to step it up and learn more. I'm totally down for anything that makes people smarter!
I prefer a method such as this over the creation of replacement apps for the google suite (for the short term anyways..).l
http://twitter.com/cyanogen/status/4384352484
If a person can root their phone, or even copy their rom file to the phone then they already have more than enough skill to do a simple drag and drop and a double click... but then again I have been surprised (scared) many times by the average intellect of most people.
Mismatching is definately a problem but not one without a solution. Again some of this will have to be a little sneaky at first. But this is for a few apks. True they are intertwined in the os. Thats why a standard has to be followed for this...
It will make rom development a little more regid but still very much doable. The apks/odexs would have to be controlled a certain way but this is not out of the real of possibility... The roms would come at a slower pace for sure..
afbcamaro said:
Mismatching is definately a problem but not one without a solution. Again some of this will have to be a little sneaky at first. But this is for a few apks. True they are intertwined in the os. Thats why a standard has to be followed for this...
It will make rom development a little more regid but still very much doable. The apks/odexs would have to be controlled a certain way but this is not out of the real of possibility... The roms would come at a slower pace for sure..
Click to expand...
Click to collapse
The new, improved fix_permissions script available in this forum will fix all the mismatching and do the odexing. Perhaps a few lines of code added to the beginning of that (OS) script could transfer the Google apps from their storage spot into the newly installed CMUpdate. Just store backups of the necessary apps on the phone, install the new update, run the script that reinstalls the Google apps, fix permissions and Odex, and re-boot into your new ROM. It looks pretty straight-forward to me.
I am sure that in the beginning it will be complicated with multi steps, but soon there will be an auto-update ap that will do the lot!
Are there going to be compatability issues, even we will end up wiping everytime we get a new rom + closed apps in?
This is so stupid. Can anyone "outside US" take over the roms so we can move on unaffected, as it is happening with everything else in the net that they try to close/block/control?
zaqwsxzaqwsx said:
I am sure that in the beginning it will be complicated with multi steps, but soon there will be an auto-update ap that will do the lot!
Are there going to be compatability issues, even we will end up wiping everytime we get a new rom + closed apps in?
This is so stupid. Can anyone "outside US" take over the roms so we can move on unaffected, as it is happening with everything else in the net that they try to close/block/control?
Click to expand...
Click to collapse
that sounds good?!
Ok, so if they like the work Cyanogen has done, but they have problems with his distribution of certain elements, maybe he just needs to TALK to them and see what can be done. You cant tell me that they don't see the following, and publicity that his roms draw. That is advertising, and companies pay BIG Bucks for good advertising. They need to C.Y.A on their end, but I bet they woulds be happy to tell him what he can do to comply. Anything "gray area" that is done with the roms to come will certainly bring back the lawyer talk, so why not see what they have for ideas.
There's also the problem that software backup is NOT covered under Fair Use, which you can read about Here http://www.copyright.gov/title17/92chap1.html#107
There's also
http://www.copyright.gov/help/faq/faq-digital.html
That's an interesting read, specifically
It is also important to check the terms of sale or license agreement of the original copy of software in case any special conditions have been put in place by the copyright owner that might affect your ability or right under section 117 to make a backup copy. There is no other provision in the Copyright Act that specifically authorizes the making of backup copies of works other than computer programs even if those works are distributed as digital copies.
Click to expand...
Click to collapse
And out of the Google Market TOS(First one I found)
Section 3.5
Unless you have been specifically permitted to do so in a separate agreement with Google, you agree that you will not reproduce, duplicate, copy, sell, trade, or resell the Market for any purpose.
Click to expand...
Click to collapse
Our best bet is to convince Google to give Cyanogen a licence providing he takes certain actions to make sure end user has a licence, such as having Cyanogen updater only run on MyTouch, Hero and G1. Or get Google to give us these apps in push fashion after initial setup.
ohwut said:
Our best bet is to convince Google to give Cyanogen a licence providing he takes certain actions to make sure end user has a licence, such as having Cyanogen updater only run on MyTouch, Hero and G1. Or get Google to give us these apps in push fashion after initial setup.
Click to expand...
Click to collapse
Pretty much what I was leaning towards- there IS a way, it is a matter of figuring out what will make them happy. If it benefits them, and helps us it's a win-win
btw, greetings from Gresham Small world
What I would do is simple just get the dev to not include the google apps and add a standalone app which will then once on ur pc push your backup google apps back into the rom zzip and sign and voila. Simples
I really doubt google will attack anyone for holding copies on ur pc. How can they find out without invasion of privacy
Look any script that anyone compiles will be viewed as warez.
What you need to do is use existing apps. But installing and backing up should be done at the User level. Writing instructions on how to backup and reinstall applications will in no way violate Google agreement at the Dev level. It would however violate it at the User level. And even at that level it is not real clear as you are just using what was entitled to you at the purchase of the phone. You can technically go after Google for violating their end of the deal and not allowing you to use the content on the device.
They dont know how these apps ended up back on your phone they have no case.
And lets face it technically installing a custom rom is violation of the T-Mobile agreement in its self. Using the Tether app is violation of it also. So no matter what people are breaking the law as it is.
A script can not violate the law in this case. The user using it for illegal purposes can be.. but let's be honest most of the ground these coorporations use for infringement are in murky waters and can be defended with a pathetic lawyer, especially a user... most of us break the law with or without knowing it one or twice a year maybe more. Microsoft doesn't go after a user because it will do them more harm than good. A crime right that profiteers from this they will go after. Google has to prove monetary loss due to you action and with a user it will fall under to minor to show up on the radar. A lot of people are more afraid of these companies than they should be.... microsoft has taken action in the past with windows mobile.. xda could not host the roms... so they went to private host like rapidshare... whay they would have to do in order to go after a user cost more than is worth especially something that they know is shaky ground.... I would dare develop and host them offsite. We are trully making it much bigger than it is by being so fearful to come up with solutions. We are convincing outselves that every action to circumvent this is illegal as well
If google just made all of these apps available on the market it would solve the problem.
Then we could still have custom ROM's minus the propitiatory apps, then we could just install them from the market.
As long as the source already included all the dependency's I don't see how this would not work. There apps are free anyway and we would be getting them from the market, an "official" distributor of said apps.
Win win for everyone.
The market is not one of the propitiatory apps is it? Because that would kinda **** up my master plan here.
HOLY CRAP, i just had a realization... if Cyanogen could obtain a lisence from google, then we could probably put roms in SAM or even the market and just update or have an updater app that could save your homescreen and your google sign-in and other user data.... even have a rooting app on there
this could be a new step towards a more open android putting root and custom roms in the hands of regular users and then if that happens i think android would become truly great then we would surpass the iphone on so many levels not just the developer/flasher level, but on the reggy consumer level
I came across this article while surfing the internet. I wanted to share this with you guys, and see what your feelings were on this.
"Mobile Device Security and Android File Disclosure
Back in November, Thomas Cannon brought to light an issue within the Android operating system. Specifically, he found that it was possible to obtain the contents of files on an Android device by simply persuading its owner to visit a web site under attacker control. The issue only garners a 3.5 CVSS score, but yet it’s still fairly serious.
Thomas reported this issue responsibly to Google and they took it seriously. However, since then they have come back with a ridiculous remediation plan. Granted, its probably not entirely Google’s fault, but the overall situation looks very bleak for Android.
The problem is that Google stated that a fix will be available as part of an update to the upcoming Android 2.3. While that, in itself, may not be totally ridiculous, the reality of the situation is that Google is only one party involved in Android. There are two other groups, namely OEMs and Carriers, that must also do their part in getting the fix to users. Although Android devices are becoming increasingly functional, the security posture remains abysmal.
The security posture for desktop applications has improved vastly with all of the sand-boxing, automatic updates, and various other exploit mitigation technologies. Meanwhile, Android includes almost none of existing security protections. In fact, mobile users are being left out in the cold, unable to get a patch for a trivially exploitable cross-zone issue. For that matter, they can’t even control whether their device’s browser automatically downloads files or not.
This situation is not news, rather it is a sad fact. It is totally unfair for end users to be left out to fend for themselves. After all, they are paying a small fortune for these devices and the service to be able to use them. Hopefully the vendors involved will wake up before a network worm outbreak occurs.
Originally, Thomas disclosed the details of his bug on his blog. Later, he removed some details to help protect users. I believe that responsible disclosure is a two-way street that requires responsibility on both sides. Since Google, OEMs, and carriers all continue to act irresponsibly, it is necessary bring more attention to this issue and the situation as a whole.
I spent a little time and managed to recreate the issue with nothing more than HTML and JavaScript. As of today, I have released a Metasploit module to take advantage of the flaw. It is available in the latest copy of our Framework product, or you can view the source via the link to our Redmine project tracker above.
Before I go deeper into the consequence of this bug, I want to point out that Thomas outlined several workarounds for this vulnerability in his blog.
Now, take a deep breath give some thanks to the fact that, under Android, most every process runs under a separate, confined, unix-style user account. This design feature partially mitigates this issue, lowering confidentiality impact to “Partial” and bringing the CVSS score from 5 to 3.5. That said, an attacker can still gain access to some pretty interesting stuff.
For starters, an attacker can steal any world-readable file. In my tests it was possible to get potentially sensitive information from the within the “proc” file system. This type of information could include kernel versions, addresses, or configuration that can be used enhance further attacks.
Also, you can snarf any files that are used by the browser itself. This includes bookmarks, history, and likely more. This kind of information could potentially be embarrassing or possibly even give an attacker access to any saved passwords or session cookies you might have stored.
Perhaps the easiest win though, is that you can grab anything off of the SD card. You might ask, “Anything?! What about the user separation?” Well, because the SD card has been formatted with the “vfat” (aka “fat32”) file system, there is no concept of ownership. All files are owned by the same user id since the file system itself cannot encapsulate who created which file. As Thomas said, files in the SD card that have predictable names are ripe for the picking. This includes pictures and movies. These may in fact be some of the most private data on your device.
In conclusion, I hope that the Android security debacle will get resolved as soon as possible. If Google, OEMs, and carriers can’t work it out, perhaps another party will step in to maintain the operating system. I believe this could be very similar to the way various Linux distributions operate today. If the situation is not resolved, I fear the Android device pool could become a seething cesspool of malicious code..."
Here is the address
http://blog.metasploit.com/2011/01/mobile-device-security-and-android-file.html
Sent from my PC36100 using XDA App
Shocking. Thanks for the info.
Nice find. You are right that oems and manufactures need to stay on top to mantain security. Hopefully meaningful post like this will make users aware of the possible dangers of the internet, data, and phone usage
Sent from my ADR6300 using Tapatalk
Ouch. Wish Android updates were like iOS..
Android is open, one of the main assumptions is that there is no single company, which controls it. I could create my own phone with Android, sell it to people and give them no support at all - Google can't do anything about it.
There is only one solution to this problem: people have to choose their phones wisely. People look at phone specs, at CPU, RAM, camera, but they ignore future support and openess. Recently Motorola has stated they will lock bootloaders in their future phones. People will go for these phones anyway and then they will complain they can't do anything with some horrible bugs, they will complain about Android and Google, but they should complain about Motorola and themselves. While Nexus S owners will have same bugs fixed by both Google and community.
Choose your phones wisely.
SD with vfat...good catch. Horrible bug while many users trying to move their apps to SD. And maybe 80-90% of the apps in the market require modify SD card perm? Horrible. Verizon SGS is screwed since that phone have little internal and lots of external SD.
I'm so glad you guys came across this thread, and it didn't get lost in all the other threads. I hope some of the devs see it. Can a fix be implemented at the Rom or kernal level?
Sent from my PC36100 using XDA App
Cydia Substrate is a code modification platform. It can modify the code of any major process, whether that code is written in Java or C/C++. It has been designed to support an ecosystem where many developers are interested in hooking the same processes. It is designed to be powerful and efficient.
== How do I get Substrate? ==
You can either download it from the Play Store or directly from its website.
== How do I develop for Substrate? ==
You download its SDK using the Android SDK manager or from its website. There is extensive documentation on the website.
== What is Substrate's website? ==
http://www.cydiasubstrate.com/
== How is this different from Xposed? ==
Many compare it with Xposed, but Xposed only supports a single use case: hooking Java functions inside of app_process. Substrate can hook native code, such as is required to modify the way styles are loaded inside of the Android asset manager. There are many other differences, however, as Substrate's API is based on five years of experience managing a community of runtime code modification for iOS. I normally avoid doing direct comparisons, but after attending Big Android BBQ and presenting on Substrate, I have been encouraged to make the differences and advantages of Substrate's approach more explicit here on XDA.
Xposed requires an inverted form of logic based on "before" or "after" hooks while Substrate lets the developer use more straightforward "replace and call previous" implementations. This also enables more complex interactions with the previous implementation that have been shown to be valuable among the thousands of developers using Substrate on other platforms. Xposed attempts to offer something similar with "replace" hooks, but those do not provide access to the previous implementation, and while Xposed provides a way to call the "original" implementation, that skips any other hooks that might be stacked.
Xposed requires the developer to find a safe moment to interact with the class being hooked. To make this possible, there are numerous lifecycle events such as "VM loaded", "package loaded", and "command line application started". However, this does not solve the problem that touching classes can change the order in which they statically initialize. This also means that it will not be possible to provide declarative syntax wrappers (such as Logos, which developers use on iOS) on top of Xposed, as this context will have to be made implicit in imperative logic. Substrate solves this class initialization problem by allowing developers to hook the classloader itself, getting a callback when a class is "linked" so that the developer can find a class loaded in any classloader (even as a plugin to an application an hour after that application starts, where the code is downloaded as a .dex from the Internet).
Xposed has a method hook implementation that makes it lose track of which method was hooked, requiring it to do a lookup every time such a method is called. This implementation is currently linear in the number of hooks, making it slow down the more hooks you install. Worse, there is a high constant multiplier on this algorithm, as the comparison between entries is very expensive (and was made more expensive when recently fixing a longstanding bug caused by this lookup being slightly incorrect). Substrate, in comparison, uses runtime code generation to avoid the need to every look anything up at runtime: you can use Substrate to hook small functions in tight loops without experiencing the same kind of performance issues you would see with Xposed.
Substrate is also designed with a different user focus: while it currently has a setup interface, it would prefer to not have any UI at all (and this will be strived for in subsequent versions, assuming anyone cares to use it). Upgrades to Substrate can be automatically installed by the Play Store and do not require the user to interact with Substrate for the changes to "stick". Substrate itself is distributed via Play. Rather than confine these kinds of modifications to advanced users who use forums such as XDA, the idea is that everyone should have access to using this kind of technology. If you have a ROM or another store in which you'd like to see Substrate distributed, I would be more than happy to talk to you about this to make that happen, and these installations will be fully supported.
For some more information on the differences between Xposed and Substrate (or if you are wondering why you should bother paying any attention to things that I say, as maybe you don't remember me from my earlier Android projects), I encourage you to read the comments I left a couple posts down from here on this thread that describe the history of Substrate, how I fit into the Android ecosystem, and more about how Substrate differs from Xposed. I will also likely be posting the talk I gave at Big Android BBQ (with either notes to go along with each slide or in the form of a video I will record re-giving the talk and advancing the slides), which might make some of these things more clear.
Current Changelog
[this is the changelog from Play, which has been compressed slightly. I will bring the more full changelog back, as I have it saved somewhere, and put it here or link it here]
v0.9.4011:
* fix decoder bug inside ARM emulator
* support Genymotion Intel emulator
* add symbol names for Moto X
v0.9.4010: critical Android 4.3 fix, avoid old Superuser bug
^^ must install before Android 4.3 OTA!
v0.9.4009: work around Xposed bug, 4.2 fix, better errors
v0.9.4008: HTC linker path patch, limit symbol exports
v0.9.4007: RAZR i 4.1.2, detect HTC override, avoid ps
v0.9.4005: incompatibility detector, avoid mount/ln/mkdir
v0.9.4004: Holo, Script Failure, detect physical /vendor
Comments from Developer
So, yeah: I'm the developer of Cydia Substrate, the framework everyone uses on iOS to do runtime code modification. Back in 2011, I gave a talk at Android Open along with a demo of Substrate running on Android 3.0. However, after some in-depth discussions with people there who were interested, I realized that what I had at the time "wasn't sufficient": it was just the core of an implementation, not an end-to-end offering. By the time it had everything I felt it needed to launch--including a comprehensive website filled with documentation, a configuration application to install with it, fully tested support for both ARM and x86, a forward-compatible pure Java API vetted by a bunch of the top people in the iOS modding community (as I feel like breaking APIs after launch is one of the more evil things a framework can do), and an extension that would make sense to end users that they could try (so that trade press wouldn't be horribly confused, as I knew they would report on the release)--it was already 2013.
http://www.youtube.com/watch?v=tA9cnemnQ0A <- Android Open 2011 keynote teaser
As many people then know, I released it in June. A lot of people have tried it (165k installs just from Play, and another 20k downloads of the APK off the site), and many of those people even like it enough to keep it installed and leave positive reviews, despite there being almost nothing available to use with it except WinterBoard (which I really only did as a demonstration). However, I also get comments from people who seem to believe I'm some kind of "interloper" in the world of Android. Additionally, there are the people who leave reviews saying stuff like "this is stupid, we already have Xposed" (sometimes then explicitly adding in the "go home to iOS" kind of spiel). The #1 complaint, however, is "nothing I can do with it", because developers never seem to talk about it or use it much, and the people installing it are all end users. Clearly this isn't the kind of reaction that I thought would happen, especially after having discussed Substrate at length with pulser_g2 before launch (who said that the community here tends to be very good about judging things on their usefulness and technical merits as opposed to having emotional attachments).
http://www.cydiasubstrate.com/ <- Cydia Substrate
Given this, and after an encouraging back/forth I had with some people on reddit's Android subreddit a few days ago in some threads about the analysis I did of that recent Android iMessage client (people who didn't know much about the ways in which Substrate is very different than Xposed in capability and focus), I figured I'd finally make a post on XDA. I kind of had been waiting to make this post as well, honestly (as again: I like things to be more perfect before I release them than maybe people are used to around here ;P), but it seems like I'm now waiting for something that is itself causing the delay (I had really expected to do this in July, before the whole thing got more actively depressing). This is clearly that post ;P. I've responded to a bunch of other threads here talking about Substrate (and the many other Android projects I've released) in the past, but this is the first time I've actually started a thread.
(In specific, Substrate currently doesn't support some Samsung devices due to a change they make to the linker paths, and I wanted to have 100% device coverage before making the inaugural XDA post. However, I'm finding it very demotivating to spend the time to think through all the options I've been considering for workarounds given the overall lackluster reaction to my work, so I'm not even making fast progress anymore: I tend to work on the things that people react positively to, and while I got a lot of positive reactions on the balance from users, I got much less than I expected from developers given how many people use Substrate on iOS and how powerful the framework is. I think, from some conversations I've had, this is largely due to confusion over how Substrate on Android relates to Xposed, which many people seem to think of as the "home-town competitor" "that does the same thing". I thereby figure that I may as well attempt to directly address that core motivation problem, to see if I should even bother continuing spending time helping out in this community, hence this ludicrously long and highly personal post about what is essentially a technical framework ;P.)
[Readers who find the next section boring should skip below to "=== Substrate ===".]
I imagine I (sadly) thereby need to start by defending my history in the Android community, as many people seem to not be aware of much of it; it actually goes back very far, as I had promised the overall mobile community that if Android were ever rooted, I'd immediately start looking at it in earnest (before there was a device, I had already been messing around with the emulator, but the device concepts Google had at the time were more like slightly souped-up feature phones, not competitors to something like an iPhone). So, in 2008, when that first "root console attached to keyboard" mistake was found on the G1 that let you get a root telnetd running by just typing it into any search field, I dropped everything and drove two hours to Los Angeles to pick up a G1 (they were not selling them in Santa Barbara yet, due to T-Mobile not really having a presence here at the time). As promised, I immediately set to work attempting to help out.
As I ran a number of mailing lists already for iOS, I set one up for Android called g1-hackers, which attracted a good number of people, and even a few Google employees who worked on bionic and the kernel. On this list is where the G1's bootloader was first dumped: if you've ever heard the stories about Eddie Dost figuring out how to do it, this is that. In fact, it was from my G1, with a kernel I compiled (following Eddie's direction: I did not know much about flash drivers), that that first Android NAND was obtained (as Eddie had already updated his device and thereby didn't actually have root). Here is a link to the mailing list thread, directly to the post where we finally succeeded and I provided the kernel image I used so that others could perform the same dump on their own devices.
http://www.telesphoreo.org/pipermail/g1-hackers/2008-December/000096.html <- [g1-hackers] G1 boot code
Around that same time, I was also contributing to AOSP, providing a bunch of patches to things like mount and init, as I wanted to be able to get Android devices to a state where they could run something much closer to Debian than Android (I had my eyes set on kind of a hybrid). In the process of doing this, I wrote a guide that for a couple years subsequent were the canonical instructions for getting a bootstrapped build of Debian installed as a chroot under Android. At the time the patch turnaround on AOSP was sometimes over half a year (and almost never shorter than a couple months), which made contributing to the project sufficiently painful that I eventually stopped. If you search through Android's codebase, though, you still find some of my work.
http://www.saurik.com/id/10 <- Debian & Android Together on G1
At the time, I honestly do not remember XDA having yet become "the place" where people spent much time talking about Android: instead, a lot of conversation happened on IRC (which is where the iOS community had already been, and where it remains). There was a channel that I was a part of which included a bunch of people whose names would hopefully be familiar to people around here, including JesusFreke (and, much later, Cyanogen). I got to see the birth of a lot of great websites and tools (such as JesusFreke's smali/baksmali) while participating on that channel. Apparently, I was talking about "Substrate for Dalvik" on that channel in November of 2008 (which is also when I first joined XDA): that's how long I've been staring at this ;P.
During the next couple years, I ended up developing and maintaining a website called Cyrket, which had the mission to allow developers and users to search the contents of the Android Market using their desktop web browser. It also solved a few key problems that developers had with comments, in that you could only see comments for apps your device had access to that were then written in your language. Developers without devices, or with devices that could not see their product (which often included those that paid extra for the ADP1, which could not see copy-protected apps) could not see comments at all. Cyrket presented all of the comments for your application in all regions in all languages (and even used Google Translate to translate them all into your own language).
The way Cyrket had worked is that I scraped the contents of the Market using the same protocol Google's client used, indexed it (supporting find-as-you-type search), and exported it all to the site (well, originally, it was actually just a live client, but then it got really popular ;P). It got me into some mild trouble occasionally with the Android Market team, but overall no one seemed to mind it that much. Cyrket was actually the primary site people used for this purpose for a long time, and I even got the impression that people at Google were begrudgingly using it as it was more convenient than the alternatives. There were a few times where it had to be taken offline (due to changes and rate limits from Google), one time for months, but I'd usually figure out some new way to get it running. Honestly, though: I was really glad when Google finally launched a website for the Market and I was able to stop working on Cyrket ;P (and also glad that Google added most of Cyrket's features for developers to their publishing console, features that Apple actually still doesn't have AFAIK).
http://www.androidtapp.com/cyrket-android-market-browser-back-from-awol/ <- Cyrket Android Market Browser Back from AWOL!
Since those times, I mostly felt the need to get Substrate "awesome" (which started to really come together during 2011, after Cyrket was no longer needed), and so didn't do many larger projects on Android until recently. That said, I have been involved in things related to exploits and security. One of the higher impact things that I did was to release mempodroid, an implementation of the mempodipper exploit described by Jason A. Donenfeld for Linux 2.6.39+, which became the primary method to root devices running Android versions 4.0.0 through 4.0.2. Much more recently, users have been using Impactor, my implementation of the various "Master Key" exploits (based both on bugs described by Jeff Forristal as well as techniques I pioneered against a random AOSP bug).
https://github.com/saurik/mempodroid <- mempodroid README
http://www.saurik.com/id/17 <- Exploit (& Fix) Android "Master Key"
Given all of this, I hope people can get a feeling for just how strange and depressing it feels to me when people seem to suddenly believe I'm some kind of foreign invader . (FWIW, I also feel rather awkward having to describe all of this in this fashion, but frankly I'm at a point where I'm realizing that if I don't explain it in this much detail myself, no one else will. While I'm certain I'll get some people responding really negatively with comments like "he's such a blowhard, going on and on about silly little things he did", so far when I've given similar spiels to people in person at conferences, they often go "oh wow, I remember that tool/happening, but didn't remember that that was also you", and so figure that this might go a long way to fixing this weird problem: I'm not just "that iOS jailbreak guy".)
=== Substrate ===
Alright, now with that aside: in time for Google I/O (which was arguably bad timing, as I was then immediately unavailable for days ;P), I finally released Substrate. Substrate (in my clearly biased opinion ;P) is actually really cool: as far as I know, it is currently the only tool available for Android that allows developers to easily modify native code without patching/replacing. I know, for example, that people often ask how to modify features like the holo themes that are implemented in C, and the answer is Substrate: if you can find the code (which is often exposed via a symbol as there are tons of C++ symbols available on most Android builds) you can use Substrate to hook it at runtime in a way that avoids having to patch the files on disk, allows developers to deploy their changes across multiple ROMs, and supports the idea that users should be in charge of the specific features that they have on their devices (as opposed to ROM distributions).
As another concrete example that maybe makes this more obvious: sometimes you download a program from the Play Store (which, incidentally, I have a very hard time not constantly still calling the Android Market ;P) that is pretty much just a massive JNI binary--maybe an OpenGL game or a media player of some sort--that refuses to run on a device that has been rooted. A really common way that developers implement such checks is to do things like verify the existence of files on disk. The simple/common checks are very easy to detect and defeat using Substrate as you can hook the native "open" call from the C standard library, check if the filename is something like /system/xbin/su, and return "nope, not there".
http://www.cydiasubstrate.com/api/c/MSHookFunction/ <- MSHookFunction()
Substrate lets you do this kind of hooking in any system daemon (not just those spawed via app_process). Yes: if there's a program running in the background of your phone, some native service written by the OEM that manufactured the device, you can use Substrate to modify it. A lot of very interesting extensions on iOS involve these kinds of hack; for an extreme example, the software unlocks that we used to have for earlier iPhones involved modifying CommCenter, a native program that initializes the radio hardware: by hooking some of the code in that daemon, it was possible to, at just the right moment, inject a different command sequence over the serial connection to the baseband, exploiting it for the unlock.
http://www.cydiasubstrate.com/inject/android/ <- Android Native Injection
Of course, Substrate also supports hooking Java code (yes, a little like Xposed, which at some level uses the same underlying trick I walked people through in my talk at Android Open 2011). Somehow, though, a lot of developers don't seem to catch all that other stuff that Substrate lets you do, and get hung up on this one part that Xposed also manages, leading to all those aforementioned irritating comments about how "there's no point to Substrate because we already have Xposed": Xposed can't do most of the things Substrate can do (and the developer has even told me that he actively tries to avoid Substrate-like techniques as they are "pretty complicated", so it isn't even moving in that direction). FWIW, on iOS it took a lot of time for Substrate to get these features (it did not have them in 2008 when I first released it): they aren't trivial ;P.
http://www.cydiasubstrate.com/api/java/MS.hookMethod/ <- MS.hookMethod()
Even within the restricted context of modifications to Java, however, I think Substrate has a lot to offer. Again: I actively refused to release Substrate until I felt I had truly nailed a few things, including in particular the Java API (at Android Open 2011, I only supported JNI, which developers there told me would not lead to traction). I was a major proponent of aspect-oriented programming when I was younger, I got into byte-code engineering in college, and I co-published a paper on a Java code modification framework called jMonitor in 2004: this is something I've been thinking about for a long time, and I think the approach I take has some merit in and of itself. I know a lot more can be done (I feel it would be really interesting to have AspectJ-style pointcuts, for example, or the kind of bytecode-level instruction matching that I implemented as part of jMonitor <- features not described in the paper, I think ;P), but I felt a good first step was be to directly leverage the iOS community's six years of experience.
http://www.cydiasubstrate.com/id/6dfa187d-6e04-4f97-b63a-ae75b5338e01/ <- jMonitor [RV '04]
To this end, Substrate provides an API for Java that is very analogous to the API that it provides for modifying C/C++ and Objective-C. The focus is on "I know about some code and I want to modify it", allowing you to not have to think much about the timing or execution details of the program that may be loading that code (so you never have to think about "packages" or "processes" or "applications": you just concentrate on "classes", and thereby don't need a million "helper APIs" to handle each narrow timing case). To enable this, I use the aforementioned ability of Substrate to modify native code to hack features into the VM itself, giving me the ability to instrument events like "a class has been loaded". If you want to hook a method of a class from Apache Commons, and you want to hook that class no matter whether it was loaded as part of an application or dynamically as part of a classloader for a plugin downloaded by an application, this is trivial to express with Substrate. AFAIK, that use case isn't even describable using Xposed.
http://www.cydiasubstrate.com/api/java/MS.hookClassLoad/ <- MS.hookClassLoad()
This kind of VM-level modification and runtime code generation support (that is heavily flexed on iOS Substrate, and thereby has had years of in-the-field testing; so far Android has exposed just one bug in its ARM reassembler after release, and that was only in the qemu emulator for some reason) also means that Substrate's implementation of hooks is highly efficient: to compare again to Xposed, every time a method that has been hooked is called via Xposed, there is a linear-time search through a linked list doing a rather heavyweight comparison to determine which method it was after the fact; with Substrate, every call is direct, there are no lookups, and there are no comparisons, so you can hook an arbitrary number of methods with no slow down, so even very small methods that are called very often can be hooked without issue.
Additionally, with Substrate I wanted to address a specific pain point that many people would bring up when I'd give talks: "how is this secure, and how do I control what apps can use these features". This became even more important, as I wanted Substrate extensions on Android to be easily deployable via conventional means, such as the Play Store (yes: Cydia Substrate itself is in the Play Store, as I believe it is important for these kinds of features to not just be in the hands of developers on forums, but to be used by end users everywhere). To this end, I integrate into the Android security model, providing a special permission that applications must have to install a Substrate extension. This helps enable the idea that Substrate mostly "gets out of the way", becoming more of a technical detail behind your extension rather than something users will need to interact with constantly to activate or update your product.
I also wanted to provide at least something that would help solve the "reflection hell" that developers seem to always find themselves in while attempting to do runtime code modification in Java (even back on desktop Java using AspectJ). I thereby provide the means to "bless" a class loader, allowing it to access private fields and classes without the overhead of reflection: the access checks, for just that one class loader, no longer apply. Substrate extensions are loaded into such a "blessed" classloader. (I do not, even though I could, ever just whack an access check VM-wide; Xposed does this, and I feel like it is going to have security implications on Java security contexts applied to class loaders for plugins.) In the case of WinterBoard, for example, I don't ever have to deal with invoking Methods or getting Fields: setAccessible is just a dim memory.
Being able to use this functionality, however, can be awkward, and in some cases is almost impossible: while testing this feature, I realized that developers would end up needing "public stubs" for all the classes they were working with, but the calling convention for a public method and a private one is different, so the calls fail at runtime. I thereby ship as part of the Substrate SDK (yes, there's an easily-updated SDK package that you can download using the Android SDK Manager ;P) an extension to javac itself (as you might imagine at this point, written using AspectJ) that turns off access control checks: you can thereby access private fields or call private methods with no extra work both during development and at runtime. This all works sufficiently well that I generally run all of ant under the modification, such that anything ant compiles becomes "blessed".
http://www.cydiasubstrate.com/id/c17c554f-b603-4e3b-8f99-ebb3528e3ef8/ <- Java Access Controls
(And yes: this is one of the things that caused Substrate to get delayed even longer than it already had been. There was also a rather serious delay caused by my attempts to really nail the boundary between "code that is shipped with Substrate" and "code that is shipped with the extension", something that burned me a lot throughout 2013 as it was the kind of problem that spending time actively thinking about didn't directly help, requiring an epiphany I had soon before Google I/O. Arguably had I been willing to ship without documentation at all, and had I generally cared less, I would probably have had everything out in very early 2012, but during January-May I started working on the initial draft of cydiasubstrate.com, as I had apparently incorrectly thought that such efforts would be critical to developer adoption.)
Again, I write this in the hope that it clears up misconceptions, either about myself or about Substrate. As far as I can tell, Substrate has a lot of very unique value propositions: things that currently are only made possible by Substrate; and, even within the restricted scope of hooking Java code inside of a service being managed by Zygote (the only area of overlap with Xposed), I think that it offers a bunch of advantages in security, performance, deployment, and ease of development that cannot be so casually dismissed with a flippant "we already have Xposed (go home)". A lot of these features (and I haven't even gone into all of them: I could write paragraphs about the advantages of how Substrate's API handles chained hooks, the ways I enable extensions that need to cross classloader boundaries, or the way Substrate makes it easy for end users to temporarily disable extensions without complex tooling) come from having spent over a decade now thinking about this problem and the last five years actively managing a developer ecosystem with tens of millions of users on iOS.
I am thereby happy to answer any questions about how to use Substrate, issues with Substrate on any device (I never blame the device: I might not have a fix immediately for a specific problem, but I always consider it Substrate's job to work around issues the device throws at it to get its functionality in place so the task will at least end up on my todo list), or even about me (as a lot of why I find writing this both so important and so painful are due to the occasional-yet-present more-personal attacks/misconceptions I often seem to receive about somehow being an "outsider"). (That said, please do have some patience: sometimes my ravenous need to do nearly 24/7 testing on a specific device has to give way so I can go to a conference I'm giving a talk at, or so I can focus on a different problem that might be more pressing or simply have a higher probability of near-term success: spending an infinite amount of time on one problem is unfair to all of the other problems that exist ;P.) [And, in fact, I have a meeting I have to be at tonight, but which hopefully won't take insanely long.]
Reserved Post
["reserved", as apparently you always should have at least one of these ;P]
Links to Extension Threads
[and finally, I can see ending up with a page that might link to other threads on XDA, although arguably I should put this on cydiasubstrate.com. right now, most projects that use Substrate are in Play. I am not certain if I'm now just misunderstanding how to use XDA, though: again, this is my first thread I've started myself]
Wow. The timing couldn't be any more perfect for you to post this.
I do not have an Android device yet and have been theorizing exactly how I could easily make modifications to applications.
Because I am just getting started in the Android development community, I don't have any biases towards one framework or the other.
Sooo.... this is on my watch list.
gugbot said:
Wow. The timing couldn't be any more perfect for you to post this.
Click to expand...
Click to collapse
The opinion of many (reasonable) people differ ;P.
gugbot said:
Sooo.... this is on my watch list.
Click to expand...
Click to collapse
Yay! If you have a moment, I'm curious: how/why did you find this thread? It seems like very few people actually go to this "Frameworks" sub-forum; there are almost no threads posted to it except the one about Xposed, which I'm presuming people must be finding by links from other places (whether random websites or other threads on XDA).
saurik said:
The opinion of many (reasonable) people differ ;P.
Yay! If you have a moment, I'm curious: how/why did you find this thread? It seems like very few people actually go to this "Frameworks" sub-forum; there are almost no threads posted to it except the one about Xposed, which I'm presuming people must be finding by links from other places (whether random websites or other threads on XDA).
Click to expand...
Click to collapse
I was browsing in development tools and was surprised to see that a Saurik posted about Cydia Substrate!
I was brought to this forum by one about theme development?... Maybe you should post this in a forum with more traffic. There seems to be an endless amount of categories for everything.
i have try your cydia substrate on cm10.1.3 stable..device samsung i9300..
install winterboard..apply icon pack but icon pack not applied..
then when want to open other apps the apps fc..except winterboard..
slipar said:
i have try your cydia substrate on cm10.1.3 stable..device samsung i9300..
install winterboard..apply icon pack but icon pack not applied..
then when want to open other apps the apps fc..except winterboard..
Click to expand...
Click to collapse
Yeah, as I mention in this thread WinterBoard was more of a demo that has been difficult to justify improvements to . This isn't an issue with Substrate, at least.
Would you mind sending me the crash report from the adb log? At least, would you mind telling me the name of the theme you applied? Also, thinking about it, CyanogenMod already has a theme engine... it never occurred to me how WinterBoard would interact with the existing theme engine in CyanogenMod (although I guess thinking even longer about it, I see no reason why it would fail horribly... it should just layer on top).
saurik said:
Yeah, as I mention in this thread WinterBoard was more of a demo that has been difficult to justify improvements to . This isn't an issue with Substrate, at least.
Would you mind sending me the crash report from the adb log? At least, would you mind telling me the name of the theme you applied? Also, thinking about it, CyanogenMod already has a theme engine... it never occurred to me how WinterBoard would interact with the existing theme engine in CyanogenMod (although I guess thinking even longer about it, I see no reason why it would fail horribly... it should just layer on top).
Click to expand...
Click to collapse
hope i send u the correct logcat..
im using ios7 concept theme..g play link here
slipar said:
hope i send u the correct logcat..
im using ios7 concept theme..g play link here
Click to expand...
Click to collapse
Thank you so much for the information. Here is a new version of WinterBoard that seems to work with this theme.
http://cache.saurik.com/apks/com.saurik.winterboard_0.9.3922.apk
thanx saurik..tested but this time winterboard just fc when try to change theme..
logcat attach..
slipar said:
thanx saurik..tested but this time winterboard just fc when try to change theme..
logcat attach..
Click to expand...
Click to collapse
I'm sorry about that issue... this is actually quite interesting to me as it might indicate that I need to do some more work on the blessed compiler as it relates to miranda methods. I had verified that the theme functioned, but had not gone back to attempt to re-verify the setup activity itself, which I guess hadn't been recompiled in a long time. I've added a temporary workaround to the issue while I investigate further. ("Humorously", if you have Xposed installed, I am pretty certain that the WinterBoard settings activity would have worked, as Xposed just destroys the access control checks for the entire VM.)
http://test.saurik.com/xda/com.saurik.winterboard_0.9.3922+1.gf733f01.apk
Hey there, I just happened upon this thread while deeply perusing the boards after just getting home from a 17hr drive and being unable to go to sleep yet. I am VERY interested in the substrates capabilities, it sounds like a very interesting concept. I am a new developer and am wanting to learn more and play more....I use xposed on my phone now and was considering starting to develop modules for it, buuuttt I think I just changed my mind I'm on an att sgs4 running a 4.3ge Rom. Going to install the substrate the night via Play Store and mess around with it starting tomorrow. Thanks for this
Sent from my GT-I9505G using Tapatalk
Sc4ryB3ar said:
I'm on an att sgs4 running a 4.3ge Rom. Going to install the substrate the night via Play Store and mess around with it starting tomorrow. Thanks for this
Click to expand...
Click to collapse
Yay! (Now, watch your GT-I9505G be one of those few Samsung devices Substrate detects as incompatible ;P. Samsung has so many model numbers that all map to the same high-level marketing names that it's difficult to keep track of what's what. If that happens, and you are interested in helping out, I can implement one of my alternative injectors quickly for you to work with.)
saurik said:
Yay! (Now, watch your GT-I9505G be one of those few Samsung devices Substrate detects as incompatible ;P. Samsung has so many model numbers that all map to the same high-level marketing names that it's difficult to keep track of what's what. If that happens, and you are interested in helping out, I can implement one of my alternative injectors quickly for you to work with.)
Click to expand...
Click to collapse
It installed just fine, quickly and with no apparent issues
winterboard, however rendered neither theme I chose correctly, wondering if its the themes though.... Didn't get a logcat and then I hosed my system last night messing around too much, so I started fresh and haven't gotten back to substrate and wb yet....I'll be back to it withing a couple of hours
Sent from my GT-I9505G using Tapatalk
substrate source code
Saurik,
I've been dabbling some with Cydia Substrate and it seems to offer a lot of unique possibilities for Android apps.
Do you have any plans to release the source code for this like you did on iOS? I'd be very curious to learn more about how it works. Also, is there a link to your talk from the Android Open conference?
Thanks,
Fred
(Ugh. I have no clue how people keep up with a forum, especially with the website as slow to load every page as it is ;P.)
fjones8856 said:
Do you have any plans to release the source code for this like you did on iOS? I'd be very curious to learn more about how it works.
Click to expand...
Click to collapse
I currently do have an intention to release the source code, but I'm not certain under what license (all of the licenses I normally use don't solve the specific issues related to Substrate). That said, no one seems to care much about Substrate on Android: on iOS people tend to (almost to a level of it being a problem) jump on new solutions to evaluate constantly, whereas on Android people seem to just snark "we already have X" even when there are compelling advantages to a replacement. Given this situation, I am highly unmotivated to spend the time to figure out the right solution, given that in a way Substrate is "my magnum opus": it is the culmination of the research and experience of so many years of my life, that passing up the ability to license it to the companies that sometimes talk to me about that (for either enterprise wrapping or security) to satisfy a group of people who are mostly asking for the source code specifically to replicate the technique *and then avoid using Substrate*, makes very little sense.
On the project side of it, Substrate on iOS only ever received a single code contribution from someone I wasn't already so close with that I was sharing code already. It isn't even the kind of project that one would expect getting many contributions: it is more of a backend technology, and the extent to which it has a GUI is actually a bug (I intend for it to be 100% seamless as part apps that use it: Substrate on iOS does not have a GUI and never will have a GUI, and that's how I think it really should work on Android as well, but of course right now I need the silly Install button). If anything, on iOS, we often end up with random companies that want to "own the scene", which ends up with them forking Substrate in ways that cause platform incompatibilities for other developers: Substrate on iOS has thereby actually been closed source now for almost two years, and it has actually improved the stability of the platform. I thereby am somewhat loath to "repeat the same mistakes from before" and end up with forks.
fjones8856 said:
Also, is there a link to your talk from the Android Open conference?
Click to expand...
Click to collapse
There was no recording of the actual talk, just of the keynote introduction that I already link to from my website. In the talk I walked people through a demonstration of using an early version of the JNI-level Substrate API, and showed how it worked (which was very simple at the time). In essence, I demonstrated, with my exact code on the projection, the technique that Xposed started using half a year later (which is just "oh, I'll change the contents of this Method object, as apparently the runtime doesn't care if the Method is allocated as part of a Class; if I do it right I can simulate registerNatives") and the most obvious way of implementing MSJavaHookClassLoad (which--for the really really low-level API I had at the time, on pre-4.0 VMs that didn't have complex JNI stacks--is clearly "MSHookFunction the class load and provide a callback"). Everything is going to be new for ART, though: the techniques are going to have to be much more sophisticated (which I'm excited by, as this is a game changer).
Pm sent
Sent from my GT-I9505G using Tapatalk
Hi again,
Bit of another weird question but i'm looking up applications that have issues such as memory leaks, Power drain Issues and a like. TBH, any application there is out there from sat nav to gaming, From simple notepads to full office suites. Everything and anything basically. Wanting to make a comprehensive list so that when we get our 'reports' sent to us it will flag up the particular application the customer is using that may be a issue. Even ones that have issues with certain versions of android.
Again, Thanks for any help
Ok then, Let me rephrase the question,
What applicatiuons do people know about that cause issues. From malware like GluMobi to Memory leaks of mGlow or Resource Hogs like hotmail to network hogs like netflix. Security issues like the one in apache cordova 3.5 and below to simple storage eaters like The SIms Freeplay.
ANY issue, not matter how big or small basically that can cause ANY potential problem. Technically, Its going be a HUGE list
Bugs, Battery Drain, Issues with certain versions of Android, battery drain, LITERALLY anything, No matter how big or small.
Thanks again
It's flat-out impossible to maintain an accurate list of what you're asking for. Most issues reported in most cases would be fixed within a few days as the apps get updated. Simply asking people to report these things is also a dangerous precedence and an ineffective way of doing it as there will be prejudice left and right, users reporting subjective information that isn't technically true and/or applicable to their specific phones and/or ROMs only. And how would you make comparison? How slow, leaky, disruptive etc does an app need to be to make it on the list? What if an app gets added that had real issues, gets fixed the day after, and then remains on your list for several more months because no new reports are coming in? It would be rather unfair to the developer(s).
Any truly disruptive apps are eventually removed and banned from ALL app repositories as the app host gets complaints about it (like Google bans apps from Play Store), so there's no reason to make a list of them here.
If i misunderstand your intentions with this list, i'm sorry. But you have more explaining to do before this idea makes any sense.
RobbyRobbb said:
It's flat-out impossible to maintain an accurate list of what you're asking for. Most issues reported in most cases would be fixed within a few days as the apps get updated. Simply asking people to report these things is also a dangerous precedence and an ineffective way of doing it as there will be prejudice left and right, users reporting subjective information that isn't technically true and/or applicable to their specific phones and/or ROMs only. And how would you make comparison? How slow, leaky, disruptive etc does an app need to be to make it on the list? What if an app gets added that had real issues, gets fixed the day after, and then remains on your list for several more months because no new reports are coming in? It would be rather unfair to the developer(s).
Any truly disruptive apps are eventually removed and banned from ALL app repositories as the app host gets complaints about it (like Google bans apps from Play Store), so there's no reason to make a list of them here.
If i misunderstand your intentions with this list, i'm sorry. But you have more explaining to do before this idea makes any sense.
Click to expand...
Click to collapse
Im in total agreement with what you say and this is just an extra feature that we are adding to what we already have. I work for a company in the UK and our intentions with this is we already have a system that checks clients hardware/software for what it has as we do a lot of work for many other big companies in the uk (all of them basically) as we have some very good engineers here. THe idea behind this database is just to flag certain things that may be causing issues and its more for internally than anything external although that as come up in meetings about adding this feature to the program we plan to release in the near future where 'certain' clients will be able to access our databases with our app we provide them. This is all preliminary at the moment and as i say, For our internal use only. This is why im looking for such a vast catalogue of problems, whether rumour or not
is not an issue at present. I'm just building the bare bone of this to test out how it works against our other databases and how easy/effective it will be working with what we already have.
Thanks for your answer and that's the conclusion we were at also. The fact that pre bundled software as total access to all information is kind of worring which we have dug up. This allows other programs that can get access to any of that information if it can pull a request from said bundled software. Example would be a program that requests use from the bundled program to read a PDF file (with the bundled software been a PDF reader). This is given access and then allows all the privileges of the bundled software. This is very very dangerous and a concern as most phone companies chuck plenty of bundled software (often not wanted by the consumer) on to there phones.
I was working on the 2G,3G,4G radios on all major phones the other week so im accustomed to A LOT of data entry
Thanks for your help my friend, Its good to know what we thought would be true but we have dug up a few other issues by doing this, So its not all a loss
EDIT: A piece of software still available and apparently malware/spyware is droiddream (bicchali.harish.droiddream) from what i can find on it. Also, Livelocker (net.livelocker) looks as if its got malware/spyware. As you say, What defines spyware is different in different peoples opinions but me personally am dubious about everything, As i think everyone should be but they are not. People just don't care as long as they have their facebook and crap lol. Point of interest about facebook, Funded to the tune of 12.8 Million by In-Q-Tel to get going, WHich was formed by the CIA. Just a little nugget there
I'm surprised no one as ANYTHING to say on the matter, Even if it's just on a whim that they hate app for x, y,& z. I have plenty personally lol