Recently, I put together a benchmark for general-purpose lossless data compression algorithms (think zilb and LZMA, not HEVC and MP3). While I was doing so it occurred to me that the standard corpora aren't very representative of the type of data people actually use data compression for these days.
To address this I'm putting together a new corpus and I would like to make sure it includes data relevant for mobile developers. Since I have virtually no experience with mobile development I was hoping some of the developers around here could tell me what kind of data they are compressing (or would like to in the future) so I can include something like it in the corpus, especially the differences from desktop usage.
Note that, in addition to developers using benchmarks run against this data to help decide what codec(s) to consider for their project, the data from this corpus will also be used by people writing compression codecs to help tune their algorithms and implementations—in other words, if the corpus includes data which is representative of what mobile apps use it will likely result in better compression (higher ratio, faster, lower memory…) for mobile apps.
I can't post links here yet (this is my first post), but for more details see the project page at <https://github.com/nemequ/squash-corpus>. There is an item in the issue tracker for "Data from a mobile app", but it's pretty vague—that's what I'm here trying to pin down, especially content that doesn't really fit into one of the other issues.
Related
Pronounced "say candy", the goal of SecAndy is to come up with as secure and private of an OS as possible. So as not to reinvent the wheel, we'll base this initiative on our open source code of choice (Android or maybe other developers' choice).
I am not a developer myself but I can without a doubt, because of former professional experiences, organize a project and gather the right people together as a community in order to make sure that project sees the light of day after it has acquired a life of its own if needed, which I think we will agree is something that this kind of project requires because of the scrutiny it will quickly attract.
I am officially calling upon this post all interested developers that could help us fork Android or other open source OS.
Let's get a kickstarter funded and let the party begin. I will update you later today on the advancement of such.
This thread welcomes constructive ideas and developer participation, but here are beginning requirements we'll need to fulfill eventually to privatize and secure android :
- default browser allowing custom search engines such as https://ixquick.com or duckduckgo
- default system search pointing to those custom engines for online component
- control of gps at firmware level to allow full disability
- peer to peer file exchange (think BitTorrent sync) with 1024 to 2048 bit encryption
- implementation of secure sms and mms exchange (think textsecure)
- implementation of encrypted voice channels (think redphone or SIP with end-to-end encryption)
- root vpn for all online access
- systemwide warning of insecure solutions (example : wanting to use gmail or regular email)
- PGP transparent email solution
- Tor option for root vpn (subject to mitm attacks but more on that later)
- peerguardian type auto-updated database to identify suspicious IP address ranges
- systematic in-out firewall control auto updated with peerguardian database and community based rules database
- hardened malware protection and app permissions with automatic permission audit based on application type
- full device encryption and lockup (in case of unauthorized user)
- full remote wipe out and bricking with auto IMEI reporting (in case of theft, might have to be amended because of attack vector)
- full remote location capability with real time tracking (that one might have to be scratched, high security risk because of attack vector)
This obviously doesn't cover all the bases but would be a good start... I know a lot of these options can be implemented with a mismatch of apps and custom Roms but having it all at an OS level AOKP style would greatly help in building an android by the people for the people community that could eventually loosen the stranglehold of less than transparent corporations.
60 views in 24 hours and not one comment. Obviously I'm approaching this the wrong way. More news at 11.
e-motion said:
60 views in 24 hours and not one comment. Obviously I'm approaching this the wrong way. More news at 11.
Click to expand...
Click to collapse
I don't want to be insulting, but no programming work has been done on your part, and you're just asking for people to dive in this project to get managed by someone they never heard of. It's not really surprising no one has commented yet.
I understand what you're saying but any comment, even if only just to show interest in such a project, will be key to drive developers to it.
I might not have started any development but I have clear understanding of how to design secure solutions. I can't go into details of why that is, however you can clearly see with my 2nd post that some research has been done. If I wanted a solution for me alone, I could just go on with my own little pudding of custom ROM and security apps.
However, because of the recent news events that SHOULD have awaken this population, I thought now might finally be the right time to try to get such a project off the ground. But without anyone even showing any interest, why would any developer be drawn to it ? If people would rather focus more on content consumerism than on what might happen under an umbrella of spooks that they're paying for with their taxes, then they have learned nothing from history and deserve what's coming to them, simple as that.
This is NOT a development thread in case you haven't noticed, so telling me I haven't developed anything yet is not even relevant.
In case anyone cares, this will be moved shortly in the t-mobile Note 2 Android development thread as a Touchwiz proof of concept ROM. Little steps, little steps...
Sent from my SGH-T889 using Tapatalk 2
mobile sec
While I am not a developer I would be interested in this project. I've been thinking about this a bit lately given recent events. I think a useful privacy preserving security related app and phone combo might have these features:
-some way to separate the baseband processor (radio) from the OS. It seems most phones share memory with the radio and this fact can and has been exploited. Own the bb processor and you own the phone. Perhaps a 3g dongle plugged into an android phone in host mode would work. Some of these usb "data only" radios can be unlocked for voice too. I believe a rooted phone with IP tables/firewall running would be much more secure than a conventional mobile phone.
-an anonymising network for connecting to servers/peers. I think the i2p network is well suited for this purpose. Rather than connect to services that are not designed with your anonymity/privacy in mind, connect to hidden/darknet servers that make it extremely difficult to ascertain your real IP and location. Perhaps an i2p router running on your home computer relaying i2p traffic while also maintaining a long lived encrypted connection to your mobile in order to "push" data to it. In this way the user benefits from the anonymising network, contributes to the network, but doesn't have the battery drain of relaying packets from the phone (if this is even possible).
-end-to-end encryption. Perhaps OTR messaging for texting and perhaps openPGP for transferring binary files as I don't believe file transfer in OTR is available at this time.
-an app that uses the above network that is capable of sending/receiving encrypted text, audio, video, gps location etc and does not leak any personal information that you don't want leaked. XMPP might be a good choice (with perhaps out-of-band binary transfers for efficiency). Giving your unique identifier to another person that is using the same app would allow you to communicate with them while not revealing your phone number, imei, imsi, etc. There would be some latency in the communication especially with binary transfers but I would gladly accept that for the added security.
anyway, just wanted to add this to the conversation and hope to see this project take shape as we definitely need more security enabled os's and apps.
I know this may seem like a very noob quetion but i wanted to get it cleared once and for all..
does cloud back end mean the following???
I have an app that collects data, processes it, and gives the output and the ui displays it.. all this done on the phone..
does cloud backend mean, my functions that process the data are in the cloud so i'm saving processing activity by letting it be done in that cloud..
all i have to do is send input and collect output. from my phone.???
Is this wat these services do???
Also if yes, what is the difference between the following..
Google app engine and compute engine??
cloud storage and cloud SQL??
which should i use to run functions and store temp data ( lasts not more than a week)..
and the data is processed once every minute but the size of data is not more than a few KB...
am basically doing a bit of number crunching.. collect a few numbers , run a function, return a number..
What would be ideal for me???
Cloud back end
Hi
Cloud computing is just a buzz-word, meaning that client application will use some dedicated network resource for some purpose. In some scenarios makes sense to use the back-end for data storage only. In others - for computations only. Others can combine between storage, analysis, computations, etc. Computations also can be performed partly on the client side, partly on the back-end side.
For deciding what is the boundary, splitting between the client and the back-end, usually the following considerations are taken:
Should the data be shared between client apps (may be of the same user or even across users)? If it should - most likely at least partial data storage needed to be cloud based
Is all the data, required for computation available at the client side? If not - may be makes sense to store missing data at the back-end and to cache it on the client side. If the data is dynamic and consistency across the clients is required - may be makes sense to perform at least part of the computation, requiring the back-end provided data to perform at the back-end, and then to continue computing on the client.
How often do you need to run the computation? If too often (e.g. more than once per second) back-end side computation will involve the network latency and data traffic overhead. So may be makes sense to perform the computation at the client side.
On the other hand - does the computation require computing power? Most of the client devices have rather weak CPU. Running heavy computations will take long time and will drain the battery. In this case at least the heavy parts of the computation makes sense to perform on the back-end
Etc, etc.
Regarding back-end storage as files or SQL: first of all, besides SQL databases there are a lot of other (NOSQL) kinds of DB. Such as: document DB, graph DB, etc. You need to decided which kind of data storage fits your needs best, based on the following considerations:
Which kind of data will be stored? black-box binary or composed from primitives (String/boolean/int/float/etc.)?
What are the relations between entries? For example for modeling social network, where entries are "Persons" and relations are from Person to many Persons - in this case graph usually DB fits best.
How data do you expect to retrieve the data? Is it just entry by some ID or you want to run queries by the fields of the entries?
In general, much deeper understanding of your problem is needed in order to give you more specific suggestions.
Good luck
Thanks
nvyaniv said:
I know this may seem like a very noob quetion but i wanted to get it cleared once and for all..
does cloud back end mean the following???
I have an app that collects data, processes it, and gives the output and the ui displays it.. all this done on the phone..
does cloud backend mean, my functions that process the data are in the cloud so i'm saving processing activity by letting it be done in that cloud..
all i have to do is send input and collect output. from my phone.???
Is this wat these services do???
Also if yes, what is the difference between the following..
Google app engine and compute engine??
cloud storage and cloud SQL??
which should i use to run functions and store temp data ( lasts not more than a week)..
and the data is processed once every minute but the size of data is not more than a few KB...
am basically doing a bit of number crunching.. collect a few numbers , run a function, return a number..
What would be ideal for me???
Click to expand...
Click to collapse
shwonder said:
Hi
Cloud computing is just a buzz-word, meaning that client application will use some dedicated network resource for some purpose. In some scenarios makes sense to use the back-end for data storage only. In others - for computations only. Others can combine between storage, analysis, computations, etc. Computations also can be performed partly on the client side, partly on the back-end side.
For deciding what is the boundary, splitting between the client and the back-end, usually the following considerations are taken:
Should the data be shared between client apps (may be of the same user or even across users)? If it should - most likely at least partial data storage needed to be cloud based
Is all the data, required for computation available at the client side? If not - may be makes sense to store missing data at the back-end and to cache it on the client side. If the data is dynamic and consistency across the clients is required - may be makes sense to perform at least part of the computation, requiring the back-end provided data to perform at the back-end, and then to continue computing on the client.
How often do you need to run the computation? If too often (e.g. more than once per second) back-end side computation will involve the network latency and data traffic overhead. So may be makes sense to perform the computation at the client side.
On the other hand - does the computation require computing power? Most of the client devices have rather weak CPU. Running heavy computations will take long time and will drain the battery. In this case at least the heavy parts of the computation makes sense to perform on the back-end
Etc, etc.
Regarding back-end storage as files or SQL: first of all, besides SQL databases there are a lot of other (NOSQL) kinds of DB. Such as: document DB, graph DB, etc. You need to decided which kind of data storage fits your needs best, based on the following considerations:
Which kind of data will be stored? black-box binary or composed from primitives (String/boolean/int/float/etc.)?
What are the relations between entries? For example for modeling social network, where entries are "Persons" and relations are from Person to many Persons - in this case graph usually DB fits best.
How data do you expect to retrieve the data? Is it just entry by some ID or you want to run queries by the fields of the entries?
In general, much deeper understanding of your problem is needed in order to give you more specific suggestions.
Good luck
Thanks
Click to expand...
Click to collapse
we get data which is hardly a few KB but it involves a lot of processing.. The issue is that we used to use google drive app script to do this where within a minute we process 30 or odd request(we are still testing on a small scale.. this is more of a info gathering or research phase for us right now) .. these requests have to be processed within the minute.. cause the very next minute we recieve our next data package... Apps script said we used too much CPU for the time... So we are looking into app engine...
Any advice??
Do you mean 30 requests per minute from each client? How many clients to you plan to have at production phase? I think it's too frequent for the option of performing all the calculation at the cloud. Can you split the calculation between the client and the cloud in such way that you can reduce the frequency of cloud based calculations? For example, may be not all the inputs of the calculation are indeed changing with this frequency. May be part of them are more static and you can calculate the part which depends on more static data at the cloud, and then to complete the calculation on the client side, using additional (more dynamic) inputs.
Also take into account that you can hardly assume that online connection is always available in mobile device. Sometimes your customers will be out of the coverage, sometimes the network will be switching from mobile data to WiFi and vice versa. It's a question of required reliability. From "...these requests have to be processed within a minute..." I understand that you need high reliability (otherwise it's not a problem to skip some requests if you are not able to process all of them).
nvyaniv said:
we get data which is hardly a few KB but it involves a lot of processing.. The issue is that we used to use google drive app script to do this where within a minute we process 30 or odd request(we are still testing on a small scale.. this is more of a info gathering or research phase for us right now) .. these requests have to be processed within the minute.. cause the very next minute we recieve our next data package... Apps script said we used too much CPU for the time... So we are looking into app engine...
Any advice??
Click to expand...
Click to collapse
New Relic is currently incentivizing new customers with three free months of Code School classes. But really, the bribe shouldn’t be necessary. We know, this sounds like the type of hyped-up pitch you might hear on a bad infomercial, but here at XDA, New Relic APM has completely revolutionized our ability to debug application processes, allowing us to speed up the site while reducing server costs. When they wanted to sponsor the site this month, we were thrilled to promote a product that we both know and use (we are very happy paying customers and have been for years). We’ve explored all options for monitoring site performance, but nothing comes close to the insight we gain from New Relic. Every web developer and mobile app developer should at least consider using New Relic. The company offers free trials and lite versions that are free-for-life on both its web app and mobile products, so there’s no reason not to.
XDA has a complex, layered infrastructure with html front-end caching, data store caching via Memcache and APC, and of course, a large database. When the site occasionally slowed down, it was impossible to determine why. On our local and staging development environments, we were able to trace application processes to determine hangups, but when working at scale, serving live pages to millions of users, we couldn’t get the same insight. After installing New Relic’s PHP plugin (they have plugins for just about every application server type), we instantly gained the ability to dig deeper into the application server processes to see where bottlenecks were occurring. The New Relic plugin software allows us to trace all the way back through the application to explore specific functions and database queries and see exactly how long they are taking to execute. We have even been able to use New Relic to assist third-party application developers to optimize their apps to work better on our servers.
Over the past few years as customers, we’ve watched New Relic launch an entire suite of products related to code-level monitoring of application performance. Specifically for mobile app developers, New Relic Mobile provides similar code-level diagnostics as the web app product. You can identify poor code paths; track CPU, memory, and database usage metrics; monitor interactions with external services and APIs; and get insight as to whether performance is being impacted disproportionately in specific geographies or on specific carriers. For a monitoring product that is used by many of the largest apps in the world, the “pro” level pricing is very reasonable ($29 per app per month), and even the free “lite” version still provides significant value.
Developers often come to us needing help debugging slow processes, and we almost always recommend installing New Relic to trace hard-to-find bottlenecks in both web and mobile applications. And today, we’re publicly making that same recommendation to you. Try it out for free. Deploy and get three free months of Code School.
I'm wondering, based on experience witch of these people would recommend for mobile app development. (Please mention which of these you actually have experience using.)
I'm an "old school" developer and am proficient in PHP/PERL, HTML, CSS and Javascript. I don't have time to learn a new language like C# and I refuse to use Microsoft's tolls (like .NET, Visual Studio, etc.) but I would spend the time maybe to pickup up RUBY (or maybe Java) if enough benefit was there in the associated mobile app development tool.
I want as close to native as possible with 90%+ cross-platform solution, meaning, 5-10% of the source code may differ due to differences in platform. I will only be developing for Android and IOS (sorry Windows) but may develop for Windows Down the road. Some apps I may develop may be enterprise class that need to get remote data from SQL Server, etc. I have done research and narrowed down my selection and am looking for further insight from those that have actually used these tools. Things that are also important to me include:
- Low cost (I can't afford to pay $100+ a month in fees)
- Good/Large Developer Community
- Good Support from Maker (good roadmap with improvements, bug fixes frequent, etc.)
PhoneGap - This seems like the most popular but it sounds like the "quickest" route for most web developers that are familiar with HTML and Javascript/CSS but to me it sounds like I might want something that is closer to native being that my primary core knowledge has always been as a coder first, and a web developer/designer second.
Appcellerator Titanium - This seems like the other most popular tool that claims to compile so the app uses the devices native controls (via Titanium API) instead of essentially an HTML page masquarading as an app (like PhoneGap) My gut says this one suits my situation better than Phonegap. Why do people use PhoneGap instead of this, because it's easier for non-programmers?
RhoMobile Rhodes - I am wondering how this compares to the two above assuming I learn RUBY on Rails.
Telerik Icenium (now called AppBuilder) - This one I have found the least discussion about.
I just ran across Codename One too which sounded intriguing as would require me to learn Java.
It seems most everyone (that isn't doing native) is using PhoneGap or Titanium and the second two are lesser known so I'm having a hard time finding people that has at least some brief experience using all four or five of these.
Any insight is greatly appreciated.
jazee said:
I'm wondering, based on experience witch of these people would recommend for mobile app development. (Please mention which of these you actually have experience using.)
I'm an "old school" developer and am proficient in PHP/PERL, HTML, CSS and Javascript. I don't have time to learn a new language like C# and I refuse to use Microsoft's tolls (like .NET, Visual Studio, etc.) but I would spend the time maybe to pickup up RUBY (or maybe Java) if enough benefit was there in the associated mobile app development tool.
I want as close to native as possible with 90%+ cross-platform solution, meaning, 5-10% of the source code may differ due to differences in platform. I will only be developing for Android and IOS (sorry Windows) but may develop for Windows Down the road. Some apps I may develop may be enterprise class that need to get remote data from SQL Server, etc. I have done research and narrowed down my selection and am looking for further insight from those that have actually used these tools. Things that are also important to me include:
- Low cost (I can't afford to pay $100+ a month in fees)
- Good/Large Developer Community
- Good Support from Maker (good roadmap with improvements, bug fixes frequent, etc.)
PhoneGap - This seems like the most popular but it sounds like the "quickest" route for most web developers that are familiar with HTML and Javascript/CSS but to me it sounds like I might want something that is closer to native being that my primary core knowledge has always been as a coder first, and a web developer/designer second.
Appcellerator Titanium - This seems like the other most popular tool that claims to compile so the app uses the devices native controls (via Titanium API) instead of essentially an HTML page masquarading as an app (like PhoneGap) My gut says this one suits my situation better than Phonegap. Why do people use PhoneGap instead of this, because it's easier for non-programmers?
RhoMobile Rhodes - I am wondering how this compares to the two above assuming I learn RUBY on Rails.
Telerik Icenium (now called AppBuilder) - This one I have found the least discussion about.
I just ran across Codename One too which sounded intriguing as would require me to learn Java.
It seems most everyone (that isn't doing native) is using Cordova/PhoneGap or Titanium and the second two are lesser known so I'm having a hard time finding people that has at least some brief experience using all four or five of these.
Any insight is greatly appreciated.
Click to expand...
Click to collapse
Both Phonegap and Appcelerator compile into a native package containing a mix of native and javascript and with both you develop your code using Javascript. As you've pointed out, one of the key differences is that Appcelerator will use the native device OS widgets rather than web widgets. In simple terms the layout is abstracted into an XML format which is then rendered into native equivalents during compilation for each of the platforms.
Depending on your application, native widgets may be important, or not. For example if you're developing a game then it could actually be advantages not to use the native widgets and have an essentially identical look and feel across all device platforms and screen aspect ratios.
The other difference is that Phonegap is free, (though their optional build service is not). Appcelerator is priced on a monthly subscription, but you also get some cloud features included in that.
Other popular cross platform tools to check out: Xamarin (native widgets, C# development, non-Free), ReactJS (native widgets, Javascript development, free-open source)
You can also try using Ionic and Cordova frameworks, They support cross-platform development and in my opinion have good documentation support
Hi everyone
I have recently been working on an Android development tool whilst completing my CS Msc [1]. This tool allows developers to submit their apps and a usage case (by defining a Monkey Runner script) and in turn will provide feedback on the energy usage of the applications code (at the method level). I am looking for feedback from the community as I am now planning to release this, as a free and open source tool.
Currently the tool will give you an approximation of the total usage of the code, a breakdown of what this means in terms of other usage (eg how long you could have watched an HD video using the device for the same amount of energy), and a breakdown of each methods total and average energy usage. There is also some support for estimation of hardware energy usage. This is all done without requiring a physical device (it runs on emulators). By using monkey runner, I am hoping that developers can define different use cases to also allow stress testing of their application.
This will not give you the exact energy usage of the app. It measures areas of code that are known to have high energy costs, assigning values based on findings from previous research that catalogued these. This research claims that these areas relate to 80% of the total energy usage of the code. The feedback is designed to highlight which routines are using the most energy and thus allow you to focus on them.
Is this something that the community is interested in? Are there any features you specifically would want added?
[1]To confirm, I have finished my Msc. This piece of work has been handed in and graded. I am not asking for help with the report or ideas for this. I just want feedback on what I feel is a worthwhile tool
This is obviously interesting. It opens us up to a whole new area of optimisation. Due to the device being mobile, power optimisation is very important.
To be able to determine what kind of features are required would need actual use of the tool.
Congrats and good work. Are you planning to open source this tool?