eXtended Sort/Merge for Android
Hello all !
I suppose a very few of you have ever needed to sort a file on your Android phone ...
Eventually, you can use the standard Unix 'busybox sort' program in a Terminal Emulator instance :
sort -k 1.1,1.10 -o myOutFile.txt myInFile.txt
This will sort the input file, ordering lines from the first ten characters of each line, ascending.
Applied to a 50 MB file (500,000 lines of 100 bytes each), it takes about 50 sec. on my Galaxy Note.
So, I did a port of a mainframe-like sort/merge, along with a Gui, that does the same job in 5 to 7 sec.
For those interested in testing this application (see attachments), remarks, ideas [and bugs ...] are welcome in this thread.
The attachment contains :
hxsm : the binary program
xsmgui-v.vv.apk : the Gui application
50mt.txt : a 50 MB text file, given as an example above
Installation (adb ...) :
download and unzip the attached file,
push or copy 50mt.txt to /sdcard/tmp (or any other directory you can access)
push or copy hxsm to /data/local/bin (or any other directory in the standard PATH)
make it executable : shell chmod 777 /data/local/bin/hxsm
install the Gui : xsmgui-v.vv.apk
Check you have to read/write/execute on all directories involved (chmod 777 ...)
SnapShots and further explanations on next post.
And run Xsm carefully, first using the default parameters showed by the Gui:
in all cases, I will not be responsible of any failure/dommage on your phone.
Please post if possible :
Brand and model of your Android box
Android version (Gingerbread, ICS ...)
Cpu speed
Size of files(s) sorted
Duration of the job(s)
Furthermore, if you have room on your sdcard(s), you can get a larger sample text file, by copying the basic sample (50mt.txt) several times on itself, and sort it :
on my G.N., a 1 GB file (ten millions lines of 100 bytes each) is sorted in 180 sec.
Still, enjoy !
eXtended Sort/Merge for Android
Advanced Use
Those wanting more may directly invoke the hxsm command in a terminal emulator
For help, just enter 'hxsm' at the command prompt :
usage :
hxsm -c, --check -q|-v, --quiet|--verbose -h, --help
-m, --merge |--copy[|--sort]
-k, --key=all | start,len[,order(A|D),[format(B|C|I|P|Z|Y)] ]
-r, --recfm=F|V|M
-z|-l, --lrecl=nnnn
-i, --infile=in_id1[,recfm=x[,lrecl=nnn] --infile=in_id2....
-o, --outfile=out_id[,recfm=x[,lrecl=nnn]]
--outrec=(in_pos1,len1,out_pos1,type1[in_pos2,...])
-f, --field-separator=TAB|BAR|COMMA|COLUMN|SEMICOLUMN|c|0Xhh
-uk|-ur, --unique-key|--unique-record
--include=start,len,op,val[AND|OR,start,len,op,val...]
--exclude=start,len,op,val[AND|OR,start,len,op,val...]
-t --sortwork=dir1,dir2,.. --sortwork=dir3,dir4,..
-y, --storage=nnnK|M|G
--keep-order
-f --field-separator=TAB|COMMA|COLUMN|SEMICOLUMN|c|0Xhh
--record-separator=c|0xhh
--collating-sequence=ascii|ebcdic
--skip-head=nnn
--throw-empty-records
-E --ignore-ioerror [ = ignore short lines ]
--norun
Example for the job given as an example in post #1 :
hxsm --verbose --recfm=V --lrecl=250 --key=1,10 --input=/sdcard/tmp/50mt.txt --output=/sdcard/tmp/6=50mt-sorted.txt/FONT]
or
hxsm -v -rV -l250 -k 1,10 -i/sdcard/tmp/50mt.txt -o/sdcard/tmp/6=50mt-sorted.txt/FONT]
Explanation for some usefull options
-k : sort/merge keys
You may specify several keys
For each key, specify the start column (byte number starting at 1), the key length in bytes, and eventualy the letter 'D' for descending order.
Furthermore, for each key, you may also have 'specific' fields format, most inherited from IBM main frames, like 'Packed' or 'Zoned' decimal fields, or just 'Numeric'.
example :
hxsm ... -k 14,3,D,P --key=1,10,A,Z -k35,40 ... --recfm=F --lrecl=100
This will sort the file on a global key including:
a packed, signed decimal field (5 digits + sign), descending order
a zoned, signed, decimal field (10 digits, including sign)
an alphanumeric field (40 bytes).
Note that in that case, the file cannot be a text file, because possible x'0D' inside the packed field : it should be specified as 'Binary Fixed' (--recfm=F)
-i : input file
you can have several input files, provided that the sort key are at the same place in all files
example :
hxsm ... -i file1 --infile=file2 -ifile3 ...
--skip-head=nnn : throws the nnn fist lines/records
-m : merge several input files already sorted on the same key
example :
hxsm -m -o resulting_file -i file1 --infile=file2 -ifile3 ...
-t : sort-work directories
It may improve perpormances, provided you have more than 1 sdcard (a fast one, class 10), or a fast usb stick connected
If this is the case, just specify a workibg directory on the corresponding mount point :
example :
hxsm ... -t /mnt/sdcard/external_sd/tmp (and check you've done a chmod 777 on it ...)
Input/Output
if the -i option is omitted, then the program will read from stdin
if the -o option is omitted, then the program will write onto stdout
Filtering
You may filter the lines or records you want to be written onto the output :
-uk : unique key
if several lines/records have the same key, then only one will be written
-ur : unique record
if several lines/records are strictly identical, then only one will be written
--include=start,len,op,val ...
--exclude=start,len,op,val ...
takes in, or omits, lines/records whose field(s) responds to some criterias
Examples:
hxsm ... --include=12,3,EQ,C'ABC' ...
this will keep only records where there is 'ABC' in cols. 12-14
hxsm ... --include=12,3,EQ,C'ABC',OR,12,13,EQ,c'ABD' ...
this will keep only records where there is 'ABC' or 'ABD' in cols. 12-14
hxsm ... --exclude=15,1,EQ,C'Z' --include=12,3,EQ,XC'ABC' ...
this will throw all records having a 'Z' in col. 15,
then, in the records left, keep only those having 'ABC' in cols 12-14
Sorting 'CSV-like' files
If the input file(s) has no fixed keys location, but rather varying length fields,
then the fields are "SOMETHING SEPARATED" (i.e. column, semi-column, tabulation ...)
In this case, just specify the 'SOMETHING' with the -f (--field-separator=...) option.
The program will considered that all lines are made of fields separated by 'SOMETHING',
and that each length given in the '--key=' option is a MAXIMUM length.
For 'SOMETHING', you may specify :
--field-separator=TAB (0x09) or BAR (0x7C) or COLUMN (0x3A) or SEMI[-]COLUMN (0x3B)
or any hexadecimal value 0x00 .. 0XFF
Example :
hxsm ... -f TAB -k 3,5,D --key=1,10,A -k 5,40 -k2,9,A,N ...
This will sort the file on a global key including:
an alphanumeric key in field #3, max length 5 chars, descending order
an alphanumeric key in field #1, max length 10 chars
an alphanumeric key in field #5, max length 40 chars
a numeric key in field #2, max length 9 digits
All fields being separated by a TAB char (0x09).
eXtended Sort/Merge for Android
Snapshots
All snapshots are in the following attachment
(if anybody can tell me how tu put the snapshots 'online' ...)
Thanks, not for the mass but useful tool though.
hhenault said:
All snapshots are in the following attachment
(if anybody can tell me how tu put the snapshots 'online' ...)
Click to expand...
Click to collapse
Edit post -> Go Advanced -> Manage Attachments -> in the popup window add your pictures instead of the zip file, you can upload picts too.
Attempting to modify your bootloader may brick your device!!
Updated to v1.2: August 23rd, 2014
-Embedded a solid jpeg extractor in rlimager1.2.c (java is no longer needed)
-Added a zero byte function to fix the encoding of rgb0, bgr0, 0bgr, & 0rgb pixel formatsA while back I wanted to make a boot animation for my Nexus 7 2013, that utilized the Google splash screen in the first part of the bootanimation. I started searching Google and XDA, and found that it was embedded in the bootloader somewhere, but nobody had a clue how to retrieve it. I could see the battery charging icons with a hex editor and use FFmpeg to convert them from their encoded rgb24 pixel format into usable images. That was easy, but I couldn't find what I was looking for. I then figured they were in some odd pixel format, maybe even proprietary. I made loops going through every conceivable pixel format at different widths, generating 1000's of images, still nothing. The last thing I tried was making an image where every bit in the bootloader was translated into a pixel, either on or off, still nothing. I pretty much gave up, until I came across this thread. I had heard of run length encoding, but didn't know it was used in Android. I soon found that Google and some manufacturers were using it for the graphics in their bootloaders.
Hit the button below to see the splash screen for the Nexus 7 2013, this is a pixel for pixel exact image, with the Google logo and Unlock Icon in their un-encoded form as seen on the devices screen. (unfortunately the splash screen wasn't encoded as a complete image, the text was encoded one place and the lock icon in another, so I did have to use photoshop to make a 1200x1920 black background and place the Google logo and the lock icon in the right position.)
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
The complete Google splash screen in full 1200x1920 can be downloaded as an attachment at the bottom of this post. It is a PSD with a black background layer and the 518 pixel "Google" and the 72 pixel "unlock" all on separate layers.
The "Tool":
This is a Windows batch script I wrote to find images in any type of file. Included in the zip at the bottom of this post is: a modified run length encoder/decoder, rlimager1.2.c, that does the encoding/decoding and the extraction of jpegs from any type of file.
Rlimager1.2.c is based off of the original to565.c put out by the AOSP several years ago. To565 was made to convert an rgb24 image into a rgb565 run length encoded file which could then be named initlogo.rle, and packed into the boot image for a kernel splash screen. Of course having the ability to encode images in this format left users the want to decode them as well (to make sure it worked before they flashed it to their device). XDA user @zebarbu in this thread, made a program to decode rgb565 rle as well. I must credit both of these authors, as rlimager1.2.c is based off of both these programs.:good: I have added additional decoding/encoding of some newer formats; as mentioned below; as well as a jpeg extractor and a function to write 0's where ffmpeg will put 1's in the case of certain pixel formats.
The executable rlimager1.2.exe was compiled with mingw32-gcc 4.8.1 on a 64-bit Intel running Windows 7, and has been tested on Windows 7 Home Premium and Windows 8. The batch script, RUN_length_imager_v1.2.bat, will not work with obsolete versions of Windows. The source file for rlimager1.2.exe and RUN_length_imager_v1.2 batch script are available in the second post, as well as in the `src\` folder of the download. FFmpeg is also in the download, and the source can be found here, while the included pre-compiled version was downloaded at Zeranoe.
Googles ADB is also included in the download, which is located at the bottom of this post.
"Run_length_imager_v1.2.zip", should be unzipped into a new directory.
Click the button below for instructions on how to use rlimager1.2 by itself. Rlimager1.2 is a standalone command line program.
Usage:
rlimager1.2.exe ([-e] [2-4] | [-d] [2-4] [-m] [max run] [-o] [offset]) < input_file > output_file
Mandatory, one or the other
-d (2-4)Run Length Decode input_file from 2, 3, or 4 byte color pattern
-e (2-4) Run Length Encode input_file to 2, 3, or 4 byte color pattern
-j (output root name) Extract Jpegs from file. Output name can include a full path.
-z (skip) -o (offset) Zero every (skip) bytes, starting with (offset)
Optional for [-d] decoding only:
-m (max run) Maximum pixel run to decode. Default is 0, which defaults to the maximum allowable for each color pattern
-o (offset) Offset (in bytes) to start decoding. Default is 0
Examples:
rlimager1.2 -d 4 -m 8064 < "C:\example_file.rle" > "C:\output\rle_decoded\example_file.rgb0"
rlimager1.2 -j root_name < "C:\users\downloads\system.img"
Notice with the jpeg extractor example above you have to use: < "inputfile"
rlimager1.2 -z 8 -o 5 -i "C:\output\example_file.rle" -e 4 < "C:\example_file.rgb0" > "C:\output\example_file.rle"
The above example run length encodes the input file, then takes the output file and zeros every eighth byte starting at 5
You can also zero byte any file without doing it all on one line, like:
rlimager1.2 -z 5 -o 8 -i "C:\file_to_be_zeroed_every_5th_byte_strarting_with_the_8th.raw"
To load a file into RUN_length_imager_v1.2.bat, you have two options:
-Drag and drop the file you wish to use onto the "RUN_length_imager_v1.2.bat" file.
-Run the program and choose option 4 at the menu. This will pull whichever partition you choose, from your device to your computer.
Note:
The second method requires you to have your ADB drivers for your device installed on your computer. You must also have busybox installed on your device. If for some reason ADB will not start and you end up with a blank partition list, simply close the program and open it back up and it should work. If it still doesn't work your ADB is probably not functioning correctly.
First and foremost before you get too much into searching for run length encoded images, you should check for jpegs first, especially if you have a Samsung device. It is really fast, and all you have to do is drag the file onto the RUN_length_imager_v1.2.bat file and select option 8 at the main menu. In five seconds or less, if there are jpegs in the file, you will have all of them in their entirety.
What is run length encoding?
RLE is a method of encoding an image with the intentions of having a smaller filesize. If you have 1000 pixels of the exact same color, you can save memory, by instead of encoding (R,G,B) 1000 times; you can add a count before the pixel description, 1000 (R,G,B). This works real efficiently for images that have lots of pixel runs, but an image where every pixel is different actually will take more memory than your conventional method of storage.
The Run Length Patterns:
I have found three different run length patterns so far. The first is the original one mentioned up above, rgb565, or as I call it 2 byte. This format can be used to encode any pixel format that uses two bytes per pixel. It works like this, the first 2 bytes is the count and the next 2 bytes is the color. The M8 uses this encoding for its bootloader graphics.
Note:
When I say "can be used to encode any pixel format that uses two bytes per pixel", I really mean pixel formats with sequential pixels. Some pixel formats use the YUV colorspace. Not all formats stored in the YUV colorspace store the pixels sequentially. For instance some YUV pixel formats will take 6 bytes of data for 4 pixels, but the first 3 bytes are not just for the first 2 pixels.
In the RGB colorspace there are BAYER pixel formats which don't encode sequential pixels either, the colors are split onto different lines in this method. Neither of the previous examples are candidates for this type of run length encoding.
The Nexus 5 and 4 use this next one, what I call 3 byte, because it can be used to encode any pixel format that uses...3 bytes per pixel. This encoding uses a 1 byte count and 3 byte color. This is easily the slowest method, for obvious reasons.
The last one, up to this point, is what I will reference as 4 byte. It works with four byte per pixel pixel formats, and uses 4 bytes for the count and 4 bytes for color. Both Nexus 7's use this format.
There may be more patterns, but it is hard to find partition dumps to download off the internet. I have found several though, and found run length encoded images in almost all of them. If anyone wants to upload a bootloader, I will gratefully add it to my collection and see what is in it.
This program will take any file, and decode it using whatever pixel format you choose, and if you have run length decoding turned on it will automatically associate the proper run length method to make a usable image, if there is one. Another important thing when dealing with raw image files, besides the pixel format, is the line length (width). This width is encoded in regular image files like bmp, jpg, and pngs. But raw images are nothing but pixel data. To get the correct image, if there is one, you have to specify when to go to the next line. I have built loops into the windows script that will let you specify multiple pixel formats to run, with an embedded loop that goes through all of the different widths you choose.
You may want to load a whole bootloader file to see if there are any images in it. To do so just drag and drop the file onto "RUN_length_imager.bat" Since the file contains different types of code and possibly images, you will want to set the maximum pixel count down to around 1000. This will limit the maximum number of a pixel run. This is necessary because not all of the data going through the decoder is meant to be run length decoded obviously, so FF FF FF FF 01 01 0A F0 in a 4 byte rle scheme will decode to over 16 gigabytes if you don't have your maximum pixel run set down to halt this.
Note:
You can stop the program at any time. I designed the program to save your settings before running and after changing anything. So if you accidentally don't set your max run and it seems like it is taking forever, and you are watching the output file size growing and growing, you can simply x out of it, or close it. Restart it and adjust whatever you have to.Another variable is the byte offset. Since the very first count byte has to go into the decoder as the very first count byte, you may have to offset the whole file up to a certain number of bytes as specified below.
For 4 byte rle patterns offsets to be tried are 0,1,2,3,4,5,6,7.
For 3 byte rle patterns offsets to be tried are 0,1,2,3
For 2 byte rle patterns offsets to be tried are 0,1,2,3
The last variable is the width. Hitting 2 in the program will allow you to set a range, or specific widths. Example: My Nexus 7 2013 bootloader has the Google logo as a 518 pixel wide image, the android as 712, the buttons as 1090, and some other things at 600 and 960. I would enter:
Code:
518 712 1090 600 960
You simply separate the different widths with spaces. You can't mess up, you can put whatever width (in any order) you want to in and it will make an image.
The second method is an exploratory method, where you enter the starting width, skips, ending width. All separated by commas.
Example:
Code:
712, 1, 1200
This will run through every width, incrementing by 1, from 712 pixels to 1200 pixels.
Note:
The height is automatically calculated based on the width, filesize, bytes per pixel. This happens at the core of the loops.Click the button below to see what the entire Nexus 5 bootloader screen looks like when the bootloader "file" is loaded with these settings used: pixel format rgb24, max run 1080, offset 0, line length 1080. The image is really big, because the whole file is run length decoded.
After you click the button, scroll down and you will see the bootloader screen towards the bottom of the image.
This is the Nexus 4, whole bootloader, pixel format rgb24, max run 1080, offset 0, line length 720
After you click the button, scroll down and you will see the bootloader screen towards the bottom of the image.
You've Identified there are images, but how do you extract just the images?
You need a good hex editor, don't worry though because you don't have to look at any hex. I prefer 010 Editor by SweetScape Software. You get a free 30 day trial. Download it. Load your bootloader file into it by right clicking on the file and selecting "010 editor". If you don't have your bootloader file you can use option 4 in RUN_length_imager.bat and pull the partition from your device. If you don't know what partition you are looking for or how to do it manually, here is an excellent guide. If you use my script, the partition will be saved in the "__partitions__" folder.
Go up to View:
Set Left Area to Char
Set Right Area to Binary
Change Line Width to Custom Width
Choose a value where the characters on your left pane almost fill up your screen so that when you start scrolling over, your right pane (binary) is right there. The right binary pane is a lot wider, that's why I prefer it on the right. Also choose a width that is evenly divisible by 8, I set mine at 224, using a small 1366x768 laptop.
A real handy shortcut in 010 editor is Ctrl + and Ctrl - to enlarge or shrink the font.The 4 byte run length encoding is the easiest to identify because there is 4 bytes for the count, and the biggest number you can store into a 4 byte address is over 4 billion. The 3rd and 4th bytes are usually nothing but zeros. The only 4 byte pixel format I've seen used so far is, bgr0, which also has the fourth byte as nothing but 8 zeros. Here is what a typical looking 4 byte rle image looks like.
Notice how the count value, mostly, show up as just four periods ....
Sometimes the first byte (period) will be a different character representing a higher value for a run of same colored pixels. Also notice that there is a lot of black and white (gray) shades in the color values. All colors on a grey scale, represented in rgb, are equal values. A value of 255,255,255 is white 200,200,200 is a light shade of grey, and 0,0,0 is black. The rgb values will always be identical to be an exact shade of grey. This image is part of the Google logo which is mostly black and white.
Any area that you highlight in the "char" pane will also be highlighted in the "binary" pane when you scroll over, you'll find this extremely handy. These 8 bytes is one complete encoded run, which represents 15 pixels pure white. You can see exactly how the four count bytes sit beside the bgr0 color bytes. Byte 1 is 00001111, binarys equivalent to decimal 15. Bytes 2, 3, and 4 are 0; then the color bytes 5, 6, 7, 8. Blue is represented with the 5th byte, green the 6th, red the 7th, and the 8th byte is not used in this pixel format. Don't get confused about the pixel format, because this method of encoding will cover all 4 byte per pixel formats, some could use the 4th color byte (or alpha), and the order they're in is not that important right now. Unless you know exactly what you are looking for, like the infamous green android laying on its back, you may be trying to pay more attention to the green column when trying to find where the image starts or stops.
(This is the binary copy of the android image you see if you have a Nexus 7 2013)
(from Nexus 7 2013 bootloader)
This part can be kind of tricky, I actually surprised myself at how good I was at "getting lucky" and finding the correct beginning and ending of an image in the bootloader files. So that probably means that it really isn't that hard, it is just not something you encounter every day. Being aware of how your image should be encoded helps a lot. If you know there are 100's of black pixels at the start and/or end of an image, keep that in your mind when looking at the characters.
An image will never start or end with the count byte(s) as 00000000!!! But, often times they DO start or end after some zeros, or before some zeros at the end. Take for example: This is the beginning of one of the lock images. (notice how it is right after the text ANDROID BOOT, hmm)
(Keep in mind that this is a 4 byte pattern run length encoding, which actually will consume 8 bytes total per run. Four bytes for the count and four bytes for the color. The other two patterns look as you would think: 3 byte .### 2 byte ..##)
Notice how the count bytes before what I have highlighted in blue is all zeros. So you won't start there, move right and see what that is. And that is your image start, it is a run of 28 solid black pixels (0,0,0,0) Once you have identified the start, and hopefully selected it as I have in the screenshot, write down the number after "Start:" Which is 3097596.
This is the end of the lock. Notice the characters all of the sudden changing, this is an indication of a color change. I visually scanned line by line and found this spot that separated all the ÿÿÿ and ÀÀÀ's up above to the newer assortment of characters like ## and 33Ì
Now you need to isolate just that image. With those last 8 bytes selected, go up to Edit - Select Range, and down at the bottom of the screen, enter in the start byte you wrote down earlier. Hit enter and the encoded image will be selected. Copy this by hitting Ctrl-C or going up to Edit - Copy.
Go to File - New - New Hex File
Hit Ctrl-V to paste, or go to Edit - Paste
Now save that file somewhere, and drag and drop it onto "RUN_length_imager.bat"
I already knew the width of the image before writing this, but here is the output with the Line length set at 60,1,80 and pixel format bgr0
Notes:
You can also encode jpg, bmp, and png files into whatever 2, 3 or 4 byte pixel format you choose. The run length encoding will be automatically adjusted according to which pixel format you choose. When encoding images into the pixel format rgb0, bgr0, 0rgb ...
Any of those 4 byte formats. FFmpeg will put 11111111 in place of the zero in the zero byte position. I'm not sure why it doesn't put 00000000 like it should, but I know how to fix it, but it involves another option, and I haven't gotten to it yet. If you were to try writing this to your device, I have no idea how Android would treat this 11111111 in the null spot, because as far as I know rgb0 (and cousins) is a made up definition by the writers of FFmpeg to describe rgb24 as rgb32 with 8 bits per color, and 8 padding bits.
Nexus 5 & 4 use the pixel format rgb24. The whole bootloader screen is encoded in one pass, as opposed to drawing several images at different spots on the screen.
The Nexus 7 1 & 2 use the pixel format bgr0. It's kind of ironic that at first when I was looking for the Google logo, I thought it was entirely made up of black and white pixels and stored in an 1 = white 0 = black, binary format (that would account for no one being able to find it, unless you do like I did and use the monow or monob pixel format in FFmpeg), but in actuality the format that it is stored in can potentially contain billions of colors. (4 byte color)
The Nexus 7 2013 Unlock Icon has an extra white pixel at the bottom right that you can clearly see on the device, now that I told you it's there. Does it mean something, or is it just poor editing by the people at Google?
The Nexus 7 2013 has the Google logo encoded in 3 different places in the Bootloader. They are binary copies of each other, so that leads me to believe that the reason for this is to discourage people from trying to edit the images. Maybe you'd brick if you just changed one and not all three? I don't know, but can't think of any other reason why they would do this. There is also two unlock icons, not counting the big one on the bootloader screen that asks you if you're sure you want to unlock it.
The Nexus 7 2013 has the only bootloader, that I've encountered, that not only stores images in a 4 byte run length manner, but there are also non-encoded 3 byte rgb24 images for the battery icon, and charging icons. Even the older Nexus 7 has all of the images run length encoded.
Samsung bootloaders use jpegs, although I did find the Linux penguin, non-encoded in the Note 3. I also found the "font mask" that uses bits to make some different fonts stored as bit masks. You can see it, using the monow or monob pixel format and setting your width at 16 32.
The HTC M8 uses the pixel format rgb565. The only device I own is the second generation Nexus 7, and it is the only one where I have been through the bootloader with a purpose. Since I don't own a M8 I really kind of just glanced at it in the hex editor and found the exclamation triangle, the htc silver logo, and the android laying on its back; all run length encoded. It also has the bit mask font embedded in it for the text, as I mentioned about the Note 3.I take no responsibility to any damage that is done to your device or computer with this program.
The source file and batch script can be found in the following post AND in the download!
Download Run Length Imager v1.2
Changelog:
Current version 1.2
-Embedded a solid jpeg extractor (java is no longer needed)
-Added a zero byte function to fix the encoding of rgb0, bgr0, 0bgr, & 0rgb pixel formats
-Added and Output folder to contain made images
-Several other code modifications
v1.1
-Added a java jpeg extractor
-Click the button below to view the additional credits related to version 1.1
The java jpegextractor.java I found here! It was written back in 2002 by Marco Schmidt. I have slightly modified it by putting in a polarity counter that takes care of embedded jpegs, that otherwise will destroy the jpeg extraction process.
v1.0
-initial upload
More images:
Nexus 5
HTC M8
2013 Nexus 7 where you can see all three logos
RUN_length_imager_v1.2.bat
Code:
@echo off
setlocal enabledelayedexpansion
color 0b
set "vers=1.2"
title Run Length Imager v%vers%
set "device_dir=sdcard"
set "partitionfind=mmc"
set autozero=sure
set outfiletype=png
set autoopen=sure
set "outfilefolder=output"
set "loglevel=-loglevel debug"
set "hidebanner=-hide_banner"
set "adblog=>>"%~dp0sys\adb_log.txt" 2>&1"
set "ffmlog=>>"%~dp0sys\ffmpeg_log.txt" 2>&1"
IF "%~1"=="-d" (
set ffmlog=
set adblog=
set "loglevel=-loglevel debug"
set hidebanner=
mode con:cols=1000 lines=4000
color 07
shift
)
cd /d "%~dp0"
if not exist "%~dp0%outfilefolder%\" mkdir "%~dp0%outfilefolder%"
if exist "%~dp0sys\adb_log.txt" del /q "%~dp0sys\adb_log.txt"
if exist "%~dp0sys\ffmpeg_log.txt" del /q "%~dp0sys\ffmpeg_log.txt"
if exist "%~dp0sys\rli_log.txt" del /q "%~dp0sys\rli_log.txt"
if exist "%~dp0sys\needfiles" del /q "%~dp0sys\needfiles"
if not exist "%~dp0bin\ffmpeg.exe" echo.FFMPEG Program "%~dp0bin\ffmpeg.exe">>"%~dp0sys\needfiles"
if not exist "%~dp0bin\rlimager1.2.exe" echo.rlimager1.2.exe "%~dp0bin\rlimager1.2.exe">>"%~dp0sys\needfiles"
if exist "%~dp0sys\needfiles" goto :help
if not exist "%~dp0sys\definitions" call :make_definitions_file
if "[%~1]" neq "[]" if exist "%~1" echo.%1>"%~dp0sys\last_file"
if exist "%~dp0sys\settings" (call :load_settings) else (
set "maxcount=0"
set "offset=0"
set /a "bpp=4"
set "pixel_format=bgr0"
set "line_length=50,1,180"
set "rldecode_on=1"
call :save_settings
)
if exist "%~dp0sys\last_file" call :load_last_file
goto :menu
:help
echo.&echo.&echo.&echo.&echo.&echo.&echo.&echo.
echo.The following necessary file^(s^) are not found in their proper location:
echo.&echo.
type "%~dp0sys\needfiles"
echo.&echo.&echo.&echo.&echo.&echo.&echo.&echo.
pause>nul
goto :egress
:make_definitions_file
cls&echo.&echo.&echo.&echo.&echo.&echo.&echo.&echo.
if not exist "%~dp0sys\" mkdir "%~dp0sys"
echo.Building FFmpeg Pixel Format List..
echo.This should only take a minute...&echo.
if exist "%~dp0sys\definitions" del /q "%~dp0sys\definitions"
echo.rgb565 2 >>"%~dp0sys\definitions"
echo.bgr565 2 >>"%~dp0sys\definitions"
for /f "skip=1 tokens=2" %%p in ('bin\ffmpeg -pix_fmts 2^>^&1^|findstr /rxic:"^I.*"') do (
%ffmlog%"%~dp0bin\ffmpeg.exe" -f rawvideo -s 1x1 -pix_fmt rgba -i "%~dp0sys\getres" -f rawvideo -s 1x1 -pix_fmt %%p -y "%~dp0sys\%%p"
for %%? in ("%~dp0sys\%%p") do if not "%%~z?"=="0" set /a bval=%%~z?/3&echo.%%~n? !bval!>>"%~dp0sys\definitions"
del /q "%~dp0sys\%%p"
)
echo.Finished!&call :load_settings
goto :eof
:make_images
call :drawhead
call :drawsettings
if defined list (set "for_type=for /l") else (set "for_type=for")
for %%p in (%pixel_format%) do (
call :setbpp %%p
if "!rldecode_on!"=="1" (
set "out_file=decoded_raw_!bpp!_byte_%in_filename%"
set "out_path=%~dp0%outfilefolder%\%in_filename%_max_%maxcount%_offset_%offset%"
if not exist "!out_path!\" mkdir "!out_path!"
if not exist "!out_path!\!out_file!" (echo.Decoding !bpp! byte rle)&("%~dp0bin\rlimager1.2.exe" -d !bpp! -m %maxcount% -o %offset% < %file% > "!out_path!\!out_file!")||(call :rli_help&goto :menu)
for %%? in ("!out_path!\!out_file!") do set /a "out_filesize=%%~z?"
if not exist "!out_path!\%%p\" mkdir "!out_path!\%%p"
if defined autoopen start "" "!out_path!\%%p\"
%for_type% %%R in (%line_length%) do (
set /a height=!out_filesize!/%%R/!bpp!
echo.Using ffmpeg to generate %%Rx!height! image from %%p pixel format.
if !height! NEQ 0 %ffmlog%"%~dp0bin\ffmpeg.exe" %hidebanner% %loglevel% -f rawvideo -vcodec rawvideo -pix_fmt %%p -s %%Rx!height! -i "!out_path!\!out_file!" -vframes 1 -y "!out_path!\%%p\%%Rx!height!.%outfiletype%"
)
) else (
set "out_path=%~dp0%outfilefolder%\%in_filename%_no_rldecode\%%p"
if not exist "!out_path!\" mkdir "!out_path!"
if defined autoopen start "" "!out_path!\"
%for_type% %%R in (%line_length%) do (
set /a height=!in_filesize!/%%R/!bpp!
if /i "%%p"=="monow" set /a height*=8
if /i "%%p"=="monob" set /a height*=8
echo.Using ffmpeg to generate %%Rx!height! image from %%p pixel format.
if !height! NEQ 0 %ffmlog%"%~dp0bin\ffmpeg.exe" %hidebanner% %loglevel% -f rawvideo -vcodec rawvideo -pix_fmt %%p -s %%Rx!height! -i %file% -vframes 1 -y "!out_path!\%%Rx!height!.%outfiletype%"
)
)
if "!auto_turn_back_on!"=="1" (set "rldecode_on=1"&set auto_turn_back_on=)
)
goto :eof
:rle_image
set "out_path=%outfilefolder%\Run Length Encoded_%in_filename%"
for %%p in (%pixel_format%) do (
call :setbpp %%p
if "%%p"=="rgb0" (set "zoffset=8"&set "zero=8")
if "%%p"=="0rgb" (set "zoffset=5"&set "zero=8")
if "%%p"=="bgr0" (set "zoffset=8"&set "zero=8")
if "%%p"=="0bgr" (set "zoffset=5"&set "zero=8")
set "out_file=encoded_raw_!bpp!_byte_%in_filename%"
if not exist "%~dp0!out_path!\" mkdir "%~dp0!out_path!\"
%ffmlog%"%~dp0bin\ffmpeg.exe" %hidebanner% %loglevel% -i %file% -f rawvideo -vcodec rawvideo -pix_fmt %%p -y "%~dp0!out_path!\raw_%%p"
if defined autozero (echo.Encoding !bpp! byte rle&"%~dp0bin\rlimager1.2.exe" -z 8 -o !zoffset! -i "%~dp0!out_path!\%in_filename%_rl_encoded_with_!bpp!_byte_pattern_and_%%p_pixel_format" -e !bpp! < "%~dp0!out_path!\raw_%%p" > "%~dp0!out_path!\%in_filename%_rl_encoded_with_!bpp!_byte_pattern_and_%%p_pixel_format"||(call :rli_help&goto :menu)
) else (echo.Encoding !bpp! byte rle&"%~dp0bin\rlimager1.2.exe" -e !bpp! < "%~dp0!out_path!\raw_%%p" > "%~dp0!out_path!\%in_filename%_rl_encoded_with_!bpp!_byte_pattern_and_%%p_pixel_format"||(call :rli_help&goto :menu))
set zoffset=&set zero=
)
if defined autoopen start "" "%~dp0!out_path!\"
goto :eof
:rli_help
echo.There was a problem encoding^/decoding the file:&echo.%file%
echo.Bpp is: %bpp%&echo.Pixel format is: %pixel_format%&echo.Offset is: %offset%
echo.Max run is: %maxcount%&echo.Width is: %line_length%&echo.Working file name is: %in_filename%&echo.&pause >nul
goto :eof
:load_last_file
set "encode="
set /p file=<"%~dp0sys\last_file"
for %%? in (%file%) do set /a "in_filesize=%%~z?" &set "in_filename=%%~n?" &set "ext=%%~x?"
if [%ext%]==[] goto :eof
if "%ext%"==".jpg" set "encode=true"
if "%ext%"==".png" set "encode=true"
if "%ext%"==".bmp" set "encode=true"
if "%ext%"==".jpeg" set "encode=true"
goto :eof
:load_settings
<"%~dp0sys\settings" (
set /p pixel_format=
set /p bpp=
set /p offset=
set /p maxcount=
set /p line_length=
set /p rldecode_on=
)
set /a bpp
echo.%line_length%|findstr ".*,.*,"&&set "list=1"||set list=
goto :eof
:save_settings
>"%~dp0sys\settings" (
echo.%pixel_format%
echo.%bpp%
echo.%offset%
echo.%maxcount%
echo.%line_length%
echo.%rldecode_on%
)
goto :eof
:menu
set auto_turn_back_on=&call :load_last_file&call :drawhead&call :drawsettings
echo.&echo.&echo.1 - Make Image(s)
echo.&echo.2 - Change Output Line Length ^(width^)
echo.&echo.3 - Change Pixel Format
echo.&echo.4 - Use ADB To List/Pull A Partition
echo.&if defined rldecode_on (echo.5 - Turn Off Run Length Decoding) else (echo.5 - Turn On Run Length Decoding)
echo.&if defined rldecode_on (echo.6 - Change Offset) else (echo. - - Change Offset ^(run length decoding is off^))
echo.&if defined rldecode_on (echo.7 - Change Max Run Length) else (echo. - - Change Max Run Length ^(run length decoding is off^))
echo.&if defined encode (echo.8 - Encode image) else (echo.8 - Extract Jpgs From File)
echo.&echo.9 - Exit&echo.
choice /n /m "Select A Menu Number:" /C:123456789
if errorlevel 1 set k=1
if errorlevel 2 set k=2
if errorlevel 3 set k=3
if errorlevel 4 set k=4
if errorlevel 5 set k=5
if errorlevel 6 set k=6
if errorlevel 7 set k=7
if errorlevel 8 set k=8
if errorlevel 9 set k=9
if %k%==1 call :make_images
if %k%==8 if "%encode%"=="true" (call :rle_image) else (call :jextract %file%)
if %k%==4 call :getpartitions
if %k%==3 call :change_pf
if %k%==2 call :change_line_length
if %k%==5 call :rle_toggle
if %k%==6 if defined rldecode_on call :change_offset
if %k%==7 if defined rldecode_on call :change_max_run
if %k%==9 goto :egress
goto :menu
:jextract
call :drawhead&call :drawsettings
set "out_path=%~dp0%outfilefolder%\%in_filename%_extracted_jpgs"
if not exist "%out_path%\" mkdir "%out_path%"
echo.&echo.Working...
"%~dp0bin\rlimager1.2.exe" -o %offset% -j "%out_path%\%in_filename%" <%file%
if defined autoopen start "" "%out_path%\"
echo.&echo.&echo Press any key to continue to the main menu..
pause>nul
goto :eof
:change_max_run
call :drawhead&call :drawsettings
echo.over 4 billion times. With a 3 Bpp rle pattern the highest value 0xFF will repeat the
echo.pixel 255 times maximum. And with a 2 Bpp rle pattern the value 0xFFFF will repeat
echo.a pixel 65535 times.&echo.
echo.When searching this way you need to set the max pixel run count to around ~1000-2000 to avoid
echo.unusable image files that can take forever to generate.&echo.
echo.Just press enter for a default value of 0 which will let the decoding process run its
echo.natural course.&echo.
set /p maxcount=Max pixel run:||set "maxcount=0"
call :save_settings
goto :eof
:change_offset
call :drawhead&call :drawsettings
echo.Enter the byte offset for the file.&echo.
echo.This should always be 0 when dealing with pure rle image files.
echo.The reason it is in here is because if you load an entire file with data/images all
echo.mixed together; the start of the Pixel count read for an image might just not be in
echo.the right spot, and the count will be read as color data instead of count data, resulting
echo.in really long^/short runs and no image. If using a whole partition^/file and a standard
echo.0 as the offset at several resolutions isn't generating the image you are looking for,
echo.taking the offset up, 1 by 1 until you get to 7 for 4 Bpp raw files, or an offset of
echo.3 for 2 ^& 3 Bpp raw files. It is advised to set your max pixel run count to ^~1000-2000
echo.when searching through complete files/partitions like this.&echo.
echo.This is the same thing as deleting "x" bytes from the beginning of the file.&echo.
echo.If you are using an image that starts with the count byte^(s^) at byte 1 in the file, then
echo.you don't have to worry about it.&echo.
echo.Just press enter for the default offset of 0.&echo.
set /p offset=Offset:||set "offset=0"
call :save_settings
goto :eof
:egress
"%~dp0bin\adb.exe" kill-server>nul 2>&1
endlocal&exit
goto :eof
:rle_toggle
if defined rldecode_on (set rldecode_on=) else (set "rldecode_on=1")
call :save_settings
goto :eof
:change_pf
:keepitinthecall
call :drawhead
call :drawsettings
echo.Enter the pixel format to use. The run length decoder will automatically adjust format accordingly.
echo.You may enter multiple pixel formats, but of course if will take longer and produce double the
echo.images if you enter two, or triple if you choose three. If you enter multiple pixel formats
echo.seperate them with a space or a comma. The bytes per pixel number seen at the top of the screen
echo.will only reflect the correct value when one pixel format is chosen, this however will have
echo.no bearing when you go to make images as the bpp is determined before each set of images is made.&echo.
echo.Note that any pixel format over 4 bytes per pixel will probably never be used in Android, for the
echo.purpose of displaying a static image at least.
echo.Threre is no run length decoding of those pixel formats over 4 bytes per pixel as of right now.&echo.
echo.The ones listed below, grouped by their bytes per pixel, are some of the more common pixel formats.
echo.Hit enter to see all input formats available with your FFmpeg build.
echo.&echo.Enter pixel formats in LOWER CASE ONLY.&echo.
echo.Enter "build" if you have updated your ffmpeg, and would like to update the pixel formats also.&echo.
echo.Enter "show" to show available pixel formats with your ffmpeg build.
echo.Common 2 Bpp android pixel formats ^[rgb565le, bgr565le^]
echo.Common 3 Bpp android pixel formats ^[rgb24, bgr24^]
echo.Common 4 Bpp android pixel formats ^[rgb0, 0rgb, 0bgr, bgr0, rgba, argb, abgr, bgra^]&echo.
set "old=%pixel_format%"
set /p pixel_format=Pixel format^(s^):||goto :eof
if /i "%pixel_format%"=="build" (set "pixel_format=%old%"&call :make_definitions_file&goto :keepitinthecall)
if /i "%pixel_format%"=="show" (set "pixel_format=%old%"&call :show_pixel_formats&goto :keepitinthecall)
call :setbpp %pixel_format%
call :save_settings
goto :eof
:setbpp
for /f "tokens=1,2" %%a in ('type "%~dp0sys\definitions"') do if /i "%~1"=="%%a" set /a "bpp=%%b"
if %bpp% GEQ 5 if defined rldecode_on (set "auto_turn_back_on=1"&set rldecode_on=)
if %bpp% EQU 1 if defined rldecode_on (set "auto_turn_back_on=1"&set rldecode_on=)
call :save_settings
goto :eof
:show_pixel_formats
cls&echo.Input pixel formats supported with your FFmpeg build:
for /f "skip=1 tokens=2,4" %%A in ('bin\ffmpeg -pix_fmts 2^>^&1^|FINDSTR /rxic:"^I.*"') DO echo.%%A--- %%B ^<^<^<BITS per pixel
echo.&echo.Press enter to continue...
pause>nul
goto :eof
:drawsettings
echo.Working Name: %in_filename%&echo.File Name: %file%&echo.Size: %in_filesize% bytes
echo.__________________________________&echo.______________________________________&echo.
echo.Pixel Format: %pixel_format%&echo.Line Length: %line_length%
if defined rldecode_on (
echo.Rle Format: %bpp% byte pp&echo.Max Run Length: %maxcount% pixels
echo.Offset: %offset%) else (echo.&echo.Run Length Decoding Is Turned Off.&echo.)
echo.______________________________________&echo.
goto :eof
:change_line_length
call :drawhead&call :drawsettings
echo.Enter the line length ^(width^) that you want to use. You can enter several, SEPERATED BY SPACES;
echo.or you can use a range SEPERATED BY COMMAS, in this format: start resolution, skips, ending resolution.&echo.
echo.Example: If you want to use the widths of 150 25 3000 400 98 16. You would enter in any order:
echo.150 25 3000 400 16 98&echo.
echo.Example: If you want a line length starting at 100 pixels going to 1200 pixels, while skipping to every
echo.10th pixel. You would enter:
echo.100,10,1200&echo.
set /p "line_length=Line length:"
echo.%line_length%|findstr ".*,.*,.*"&&set "list=1"||set list=
call :save_settings
goto :eof
:drawhead
cls&echo.&echo.___________________________________-_-
echo.__________________________________&echo.&echo.Run Length Imager: v%vers% by makers_mark
echo.__________________________________&echo.______________________________________&echo.
goto :eof
:getpartitions
if not exist "%~dp0partitions\" mkdir "%~dp0partitions"
call :drawhead&set /a index=1
"%~dp0bin\adb.exe" -d start-server
if %ERRORLEVEL% GTR 0 call :adb_error&goto :eof
for /f "skip=1 tokens=3,4" %%s in ('bin\adb.exe -d shell cat /proc/partitions^|findstr /rxic:".*%partitionfind%.*"') do (
if not "%%s"=="" (call set /a "_size[!index!]=%%s"&&call set "_partition[!index!]=%%t") else (set /a index=1)
call set /a index+=1
)
if %ERRORLEVEL% GTR 0 call :adb_error&goto :eof
set /a index-=1
echo. Partition Size&echo.
for /l %%c in (1,1,%index%) do (
if %%c LSS 10 (echo. %%c. !_partition[%%c]! !_size[%%c]!) else (
echo.%%c. !_partition[%%c]! !_size[%%c]!)
)
echo.&echo.If you want to pull one of these partitions from your device, enter the number
echo.to the left of it and press enter. Just press enter to go back to the Main Menu&echo.
set /p _pick=:||goto :eof
2>nul set /a _pick=%_pick%/1 || GOTO :eof
IF %_pick% LSS 1 GOTO :eof
if %_pick% GTR %index% goto :eof
cls&echo.&echo.&echo.WARNING: WARNING: WARNING: WARNING: WARNING:&echo.WARNING: WARNING: WARNING: WARNING: WARNING:
echo.WARNING: WARNING: WARNING: WARNING: WARNING:&echo.WARNING: WARNING: WARNING: WARNING: WARNING:&echo.&echo.&echo.&echo.&echo.
echo.Issuing this command:&echo.
call echo.bin^\adb.exe -d shell dd if^=^/dev^/block^/%%_partition[!_pick!]%% of^=^/%device_dir%^/%%_partition[!_pick!]%%
echo.&echo.PLEASE READ THIS TWICE....&echo.
echo.DD is a VERY powerful tool, and if the output directory^/file; listed after ^"of^=^" is not the
echo.name of the partition you chose to pull, DO NOT CONTINUE. If you are not sure what you are doing;
echo.what this command is, or what you are even looking at. DO NOT CONTINUE.&echo.
set "guess=%random%"
echo.Enter this number to continue: %guess%
echo.&set /p answer=:||goto :eof
if not "%guess%"=="%answer%" goto :eof
echo.&echo.Copying data into a file on your device...
call echo."%~dp0bin\adb.exe" -d shell "dd if=/dev/block/%%_partition[!_pick!]%% of=/%device_dir%/%%_partition[!_pick!]%%"|cmd /v:on%adblog%&&echo.&&echo.Copying file from your device to your computer...&&call echo."%~dp0bin\adb.exe" -d pull "/%device_dir%/%%_partition[!_pick!]%%" "%~dp0partitions\%%_partition[!_pick!]%%"|cmd /v:on%adblog%&&echo.&&echo.Deleting the file from your device...&&call echo."%~dp0bin\adb.exe" -d shell rm "/%device_dir%/%%_partition[!_pick!]%%"|cmd /v:on%adblog%
call echo."%~dp0partitions\%%_partition[!_pick!]%%">"%~dp0sys\last_file"
echo.&echo.&echo.Finished. The partition has been loaded and is saved as:&echo.&call echo."%~dp0partitions\%%_partition[!_pick!]%%"&echo.&echo.Press any key to continue.
call :save_settings
pause>nul
goto :eof
:adb_error
echo.&echo.&echo.&echo.&echo.&echo.&echo.&echo.
echo. ADB is not connected, working properly, or you have
echo. more than one device connected.
echo. Try "Safely Removing" your device from your computer.
echo. Then unplug your usb cable, and reinsert it.
echo.&echo.&echo.&echo.&echo.&echo.&echo.&echo.
pause>nul
goto :eof
rlimager1.2.c
C:
/*
* Copyright (C) 2008 The Android Open Source Project
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
& NOTICE:
& May 5th, 2014
&
& makers_mark @ xda-developers.com
& http://forum.xda-developers.com/showthread.php?t=2764354
&
& Original source:
& https://android.googlesource.com/platform/build/+/b6c1cf6de79035f58b512f4400db458c8401379a/tools/rgb2565/to565.c
& Based off of the original to565.c program to convert raw rgb888 r=8 g=8 b=8 images
& to, 2 byte count, 2 byte color run length encoded rgb565 r=5 g=6 b=5 files.
& Mainly, if not always to my knowledge, used for creating initlogo.rle files for kernel splash
& screens.
&
& Added decoding of 2, 3, and 4 byte rgb(x) patterns
& Added encoding of 3 and 4 byte rgb(x) patterns
& Added byte offsets and maximum pixel runs for decoding files not totally
& encoded in a run length manner
& Version 1.2 added:
& Jpeg extractor, to pull jpegs from any file or drive/device image
& A zero byte writer, to undo what ffmpeg does to rgb0, 0rgb, bgr0, and 0bgr pixel formats
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
unsigned int _CRT_fmode = _O_BINARY;
static char fileName[1024];
void zeroBytes(long long, long, char *);
void jpegExtract(char *, unsigned long long);
void headerToFooter(int);
int getFilename(char *);
void decode_rgb16_rle(unsigned int, unsigned long long int);
void decode_rgb24_rle(unsigned int, unsigned long long int);
void decode_rgbx32_rle(unsigned long int, unsigned long long int);
void encode_rgb16_rle(void);
void encode_rgb24_rle(void);
void encode_rgbx32_rle(void);
void zeroBytes(long long offset, long z, char *inputFile)
{
int readByte;
FILE *zeroStream;
FILE *inStream;
long long int cursorPosition;
char outputFile[1024];
sprintf(outputFile, "%s.zero", inputFile);
if ((inStream = fopen(inputFile, "rb")) == NULL){
fclose(inStream);
return;
}
if ((zeroStream = fopen(outputFile, "wb")) == NULL){
fclose(inStream);
fclose(zeroStream);
return;
}
fprintf(stderr,"\nZeroing every %ld bytes starting at %lld", z, offset);
fprintf(stderr,"\nInput file: %s", inputFile);
while ((readByte = fgetc(inStream)) != EOF){
cursorPosition = ftell(inStream);
if (((cursorPosition - offset) % z) == 0 && cursorPosition >= offset){
fputc(0,zeroStream);
}else{
fputc(readByte,zeroStream);
}
}
fclose(inStream);
fclose(zeroStream);
}
void jpegExtract(char outname[1024], unsigned long long o)
{
int readByte;
FILE *fileStream;
unsigned int header;
if (o != 0){
fseek(stdin, o, SEEK_SET);
}
while ((readByte = fgetc(stdin)) != EOF){
if (readByte == 0xff){
readByte = fgetc(stdin);
if (readByte == 0xd8){
readByte = fgetc(stdin);
if (readByte == 0xff){
readByte = fgetc(stdin);
if (readByte == 0xe0 || readByte == 0xe1){
header = 0x00ffd8ff | (readByte << 24);
if (!getFilename(outname)){
if ((fileStream = fopen(fileName, "wb")) == NULL){
fclose(fileStream);
return;
}
int streamNumber = fileno(fileStream);
fprintf(stderr, "\n%s", fileName);
write(streamNumber, &header, 4);
headerToFooter(streamNumber);
close(streamNumber);
continue;
}else{
break;
}
}
}
}
}
}
fclose(stdin);
}
void headerToFooter(int streamNumber)
{
short notFinished = 1;
int polarity = 1;
int readByte;
while((readByte = fgetc(stdin)) != EOF){
if (readByte == 0xff){
readByte = fgetc(stdin);
if (readByte == EOF){
readByte = 0xff;
write(streamNumber, &readByte, 1);
break;
}else if (readByte == 0xd8){
polarity += 1;
readByte = 0xd8ff;
write(streamNumber, &readByte, 2);
}else if (readByte == 0xd9){
polarity -= 1;
readByte = 0xd9ff;
write(streamNumber, &readByte, 2);
if (polarity == 0){
break;
}
}else{
readByte = 0x00ff | (readByte << 8);
write(streamNumber, &readByte, 2);
}
}else{
write(streamNumber, &readByte, 1);
}
}
}
int getFilename(char outputbase[1024])
{
unsigned int counter;
FILE *stream;
for (counter = 1; counter <= 99999; counter++){
sprintf(fileName, "%s_%05d.jpg", outputbase, counter);
if ((stream = fopen(fileName, "r+")) != NULL){
fclose(stream);
continue;
} else {
fclose(stream);
return(0);
}
}
fclose(stream);
return(1);
}
void decode_rgb16_rle(unsigned int m, unsigned long long int o)
{
unsigned short data[2], repeats;
if (o != 0){
fseek(stdin, o, SEEK_SET);
}
while(read(0,data,4) == 4){
if (data[0] > m){
continue;
}
for (repeats = 0; repeats < data[0]; repeats++){
write(1, &data[1], 2);
}
}
}
void decode_rgb24_rle(unsigned int m, unsigned long long int o)
{
unsigned char repeats, data[4];
unsigned long color;
if (o != 0){
fseek(stdin, o, SEEK_SET);
}
while(read(0, data, 4) == 4){
if (data[0] > m){
continue;
}
color = ((data[1]) | (data[2] << 8) | (data[3]) << 16);
for (repeats = 0;repeats < data[0]; repeats++){
write(1, &color, 3);
}
}
}
void decode_rgbx32_rle(unsigned long int m, unsigned long long int o)
{
unsigned long repeats;
unsigned long data[2];
if (o != 0){
fseek(stdin, o, SEEK_SET);
}
while(read(0, data, 8) == 8){
if (data[0] > m){
continue;
}
for (repeats = 0; repeats < data[0]; repeats++){
write(1, &data[1], 4);
}
}
}
void encode_rgbx32_rle(void)
{
unsigned long color, last, count;
count = 0;
while(read(0, &color, 4) == 4){
if (count){
if ((color == last) && (count != 0xFFFFFFFF)){
count++;
continue;
} else {
write(1, &count, 4);
write(1, &last, 4);
}
}
last = color;
count = 1;
}
if (count){
write(1, &count, 4);
write(1, &last, 4);
}
}
void encode_rgb16_rle(void)
{
unsigned short int last, color, count;
count = 0;
while(read(0, &color, 2) == 2){
if (count){
if ((color == last) && (count != 0xFFFF)){
count++;
continue;
} else {
write(1, &count, 2);
write(1, &last, 2);
}
}
last = color;
count = 1;
}
if (count){
write(1, &count, 2);
write(1, &last, 2);
}
}
void encode_rgb24_rle(void)
{
unsigned char count;
unsigned long int last, color;
count = 0;
while(read(0, &color, 3) == 3){
if (count){
if ((color == last) && (count != 0xFF)){
count++;
continue;
} else {
write(1, &count, 1);
write(1, &last, 3);
}
}
last = color;
count = 1;
}
if (count){
write(1, &count, 1);
write(1, &last, 3);
}
}
int usage(void){
fprintf(stderr, "\n\n\nUsage:\n\nrlimager.exe ([-e] [2-4] | [-d] [2-4] [-m] [max run] [-o] [offset]) < input_file > output_file\n\n");
fprintf(stderr, "Mandatory, one or the other\n\n");
fprintf(stderr, "-d (2-4) Run Length Decode input_file from 2, 3, or 4 byte color pattern\n");
fprintf(stderr, "-e (2-4) Run Length Encode input_file to 2, 3, or 4 byte color pattern\n");
fprintf(stderr, "-j (output root name) Extract Jpegs from file. Output name can include a full path.\n");
fprintf(stderr, "-z (skip) -o (offset) Zero every (skip) bytes, starting with (offset)\n\n");
fprintf(stderr, "Optional for [-d] decoding only:\n\n");
fprintf(stderr, "-m (max run) Maximum pixel run to decode. Default is 0, which defaults to the maximum allowable for each color pattern\n");
fprintf(stderr, "-o (offset) Offset (in bytes) to start decoding. Default is 0\n");
fprintf(stderr, "\n\nExamples:\n\nrlimager1.2 -d 4 -m 8064 < \"C:\\example_file.rle\" > \"C:\\output\\rle_decoded\\example_file.rgb0\"\n");
fprintf(stderr, "rlimager1.2 -j root_name < \"C:\\users\\downloads\\system.img\"\n\n");
fprintf(stderr, "Notice with the jpeg extractor example above you have to use: < \"inputfile\"\n\n");
fprintf(stderr, "rlimager1.2 -z 8 -o 5 -i \"C:\\output\\example_file.rle\" -e 4 < \"C:\\example_file.rgb0\" > \"C:\\output\\example_file.rle\"\n\n");
fprintf(stderr, "The above example rle encodes the input file, then takes the output file and zeros every eighth byte starting at 5\n");
fprintf(stderr, "You can also zero byte any file without doing it all on one line, like:\n\n");
fprintf(stderr, "rlimager1.2 -z 5 -o 8 -i \"C:\\file_to_be_zeroed_every_5th_byte_strarting_with_the_8th.raw\"\n");
return(1);
}
int main(int argc, char **argv)
{
unsigned int decode_opt = 0, encode_opt = 0;
unsigned long long int maxrun = 0;
long long int offset = 0;
long zeroByte = 0;
short jflag = 0;
char *d_string, *e_string, *m_string, *o_string, *j_string, *z_string, *inputFile;
int c;
while ((c = getopt (argc, argv, "i:z:j:m:o:e:d:")) != -1)
switch(c)
{
case 'i':
inputFile = optarg;
break;
case 'z':
z_string = optarg;
zeroByte = atol(z_string);
break;
case 'j':
jflag = 1;
j_string = optarg;
break;
case 'm':
m_string = optarg;
maxrun = atoll(m_string);
break;
case 'o':
o_string = optarg;
offset = atoll(o_string);
break;
case 'e':
e_string = optarg;
encode_opt = atoi(e_string);
break;
case 'd':
d_string = optarg;
decode_opt = atoi(d_string);
break;
}
if ((encode_opt > 1) && (encode_opt < 5)){
if (maxrun != 0 || decode_opt != 0){
usage();
return(1);
}
if (encode_opt == 2){
encode_rgb16_rle();
return(0);
}
if (encode_opt == 3){
encode_rgb24_rle();
}
if (encode_opt == 4){
encode_rgbx32_rle();
}
if (zeroByte){
zeroBytes(offset, zeroByte, inputFile);
}
return(0);
} else if (zeroByte){
zeroBytes(offset, zeroByte, inputFile);
return(0);
} else if ((decode_opt > 1) && (decode_opt < 5)){
if (encode_opt != 0){
usage();
return(1);
}
if (decode_opt == 2){
if (maxrun == 0){
maxrun = 0xFFFF;
}
fprintf(stderr, "Maximum pixel run set at %d\n",maxrun);
decode_rgb16_rle(maxrun, offset);
return(0);
}
if (decode_opt == 3){
if (maxrun == 0){
maxrun = 0xFF;
}
fprintf(stderr, "Maximum pixel run set at %d\n",maxrun);
decode_rgb24_rle(maxrun, offset);
return(0);
}
if (decode_opt == 4){
if (maxrun == 0){
maxrun = 0xFFFFFFFF;
}
fprintf(stderr, "Maximun pixel run set at %lld\n",maxrun);
decode_rgbx32_rle(maxrun, offset);
return(0);
}
return(0);
} else if (jflag){
jpegExtract(j_string, offset);
} else usage();
return(1);
}
@makers_mark
Very nice work! This is actually more interesting than it seem. It's also a way to inspect binary code in unknown blobs. Another thing to notice is that you can probably automate the correct image size, for images that you know contain vertical lines. If you look at the padlock image sequence, you see that the images contain parts that are "sloping" either upward or downward, this can be used to look for the correct direction to increment/decrement the size counter.
So if you remember your basic math for the slope of a line:
m = Δy/Δx = (y2 - y1)/(x2 - x1)
you can plug some pixel values in and get the slope, and subsequently the direction for the counter.
Obviously the most difficult part will be to detect what is a "line" when there are other pixels there.
E:V:A said:
@makers_mark
Very nice work! This is actually more interesting than it seem. It's also a way to inspect binary code in unknown blobs. Another thing to notice is that you can probably automate the correct image size, for images that you know contain vertical lines. If you look at the padlock image sequence, you see that the images contain parts that are "sloping" either upward or downward, this can be used to look for the correct direction to increment/decrement the size counter.
So if you remember your basic math for the slope of a line:
m = Δy/Δx = (y2 - y1)/(x2 - x1)
you can plug some pixel values in and get the slope, and subsequently the direction for the counter.
Obviously the most difficult part will be to detect what is a "line" when there are other pixels there.
Click to expand...
Click to collapse
Thank you, you can imagine my excitement the first time I had the logo I was looking for come across my screen
I was trying to think of a way to automatically choose "the right image", but cognitive dissonance kicked in and I figured that people would rather see lots of images. Kind of kidding, but I wanted it to work for almost all images, and my expectations kept leading me back to nothing.
Here is something, i forgot to mention, I found this is in the n72013 4.02 bootloader that I don't ever remember seeing on the device. It is the ASUS logo, with their slogan underneath it. But it is not all there, I've checked and checked and this is how it is in the hex editor too. It is located at the very bottom, right about the last Google logo, width is 600.
^^ And as you see above, the sloping lines is probably the remainder of the "missing" pieces. For some reason the image has been split and pieces moved around, just like for any other file. This is probably because you have extracted this as a blob from memory or EMMC in which case these pieces may have been shuffled around by internal MMC wear leveling. But the way these pieces are linked together should be via inodes and you'll need some better tools to read these. (It should be some kind of address immediately following an incomplete image.) Another possibility could be that the screen driver FW like to read images in pieces...
E:V:A said:
^^ And as you see above, the sloping lines is probably the remainder of the "missing" pieces. For some reason the image has been split and pieces moved around, just like for any other file. This is probably because you have extracted this as a blob from memory or EMMC in which case these pieces may have been shuffled around by internal MMC wear leveling. But the way these pieces are linked together should be via inodes and you'll need some better tools to read these. (It should be some kind of address immediately following an incomplete image.) Another possibility could be that the screen driver FW like to read images in pieces...
Click to expand...
Click to collapse
I was thinking that this data was from the factory download, but I just checked and you are indeed right. It is not in the bootloader.img from google, but only on my mmcblk0p12. As for the white lines surrounding it, the bottom lines are the Google logo (at 518 pixels) and the upper white areas are the up/down arrows and the second unlock icon. Your point was still well received! And thanks for your insight:good:
Here is the same image (I cropped out the upper irrelevant stuff, but the same data) at 518 width.
If you haven't read it yet, try a search for ARM_ELF.pdf
If the partition you're reading (like a boot.img) is an ELF file it will give you the offsets for the start of files (that's how I found my systems initial splash image) and the byte size for that file and any other files in that partition ie. Boot code, splash screen, text data.
Throw a boot.img/recovery.img into a hex editor (I use HxD) and alter the displayed byte count from standard 8/16/32 etc to 320 (or your screens pixel width) and scroll through and you'll see the data layout roughly of your splash screen (really basic way to find it)
Bashing away at my HTC Desire C
Antagonist42 said:
If you haven't read it yet, try a search for ARM_ELF.pdf
If the partition you're reading (like a boot.img) is an ELF file it will give you the offsets for the start of files (that's how I found my systems initial splash image) and the byte size for that file and any other files in that partition ie. Boot code, splash screen, text data.
Throw a boot.img/recovery.img into a hex editor (I use HxD) and alter the displayed byte count from standard 8/16/32 etc to 320 (or your screens pixel width) and scroll through and you'll see the data layout roughly of your splash screen (really basic way to find it)
Bashing away at my HTC Desire C
Click to expand...
Click to collapse
If only it would of been that easy...
But, if you are dealing with a non-rle images, you absolutely can see it's "likeness" in a hex editor with your width set correctly. An image that is run length encoded bears no resemblance to anything, except some of them do look like random waves, but that's another story.
Those images were the first things I noticed in my N7 bootloader, not what I wanted though. But since the non-rle images were in a rgb24 pixel format, with 3 bytes per pixel, my battery charging icon that is 340 pixels wide could only be seen correctly (in a hex editor) at 340*3=1020 for the width. That's one of the things I like about 010 editor, I think Hxd will only go up to 512 for the width, 010 editor goes up to 1024.
Ah yes, I'd forgotten about .rle :banghead: can't see it but can find it.
You will find the address of them in the ELF data, especially the splash and battery charge on power up, it's in (I think, off top of my head) hboot, if you know the partitions offsets from 0x00000000 (or you have a full nand dump) there's pointers to them
Bashing away at my HTC Desire C
Antagonist42 said:
Ah yes, I'd forgotten about .rle :banghead: can't see it but can find it.
You will find the address of them in the ELF data, especially the splash and battery charge on power up, it's in (I think, off top of my head) hboot, if you know the partitions offsets from 0x00000000 (or you have a full nand dump) there's pointers to them
Bashing away at my HTC Desire C
Click to expand...
Click to collapse
You must be referring to the kernel, I am talking about the bootloader in the OP. There is no elf header/data in the bootloader, they are typically proprietary and written in machine code &| assembly language.
You've spiked my curiosity now, especially as I've not been able to work at my PC in a while
Is now off to see where it's pointed at from....
Bashing away at my HTC Desire C
@makers_mark So I was sitting in the bar the other day...and suddenly realized where you got that name from.
Antagonist42 said:
You've spiked my curiosity now, especially as I've not been able to work at my PC in a while
Is now off to see where it's pointed at from....
Bashing away at my HTC Desire C
Click to expand...
Click to collapse
Here is the mmcblk0p12 from the Nexus 7 2013 and the whole bootloader.img from the factory download. I don't know if it would help you, I imagine you have a device to check, but I have the logo locations already if it would help you out any.
There are 4 locations of the image (the splashscreen isn't drawn all at once, it places the logo and the lock from different places in the bootloader) in the partition mmcblk0p12 they are all 43768 bytes long:
307760 o----c 351528
2390104 o----c 2433872
3075424 o----c 3119192
3166732 o----c 3210500
In the factory image there are three locations, they are:
1183172 o----c 1226940
3265516 o----c 3309284
3950836 o----c 3994604
@E:V:A
If you were at a bar and you figured out where I got my name from, you'd have to be right!
Came across this, simply because it's for LCD screens but explains the different encoding methods
http://www.demmel.com/download/ilcd/2d_rle_appnote.pdf
Bashing away at my HTC Desire C
Antagonist42 said:
Came across this, simply because it's for LCD screens but explains the different encoding methods
http://www.demmel.com/download/ilcd/2d_rle_appnote.pdf
Bashing away at my HTC Desire C
Click to expand...
Click to collapse
The possibilities are endless when it comes to run length encoding. The three different methods that I've found probably aren't the only ones being used. Most people probably confuse rle with the old microsoft file format, but actually rle these days is a method of encoding. So, if you want to have 3 bytes for the count and 3 bytes for the color, you can do it. But if you notice, all three of the methods are 4 or 8 byte reads. No 3, 6, 10.... I wrote an arbitrary decoder for rlimager, but took it out because I don't see them ever using rle that doesn't add up to 4 or 8. Like 2 bytes count 3 bytes color, would be a more efficient way to encode 3 byte color, as opposed to 1 byte count 3 byte color (like they use). But it would be horribly slow. Here is the arbitrary decoder:
Code:
void arbitrary(int bytesofcount, int bytesofcolor)
{
unsigned int i, j, k, totalperpixel = bytesofcount+bytesofcolor;
unsigned long times, count, counttotal;
unsigned char color, instream[totalperpixel];
unsigned char colorbyte[bytesofcolor];
while(read(0,instream,totalperpixel)){
for (i = 0; i<bytesofcount; i++){
count = instream[i];
if (i > 0) {
counttotal = (count << 8) | counttotal;
} else {
counttotal = count;
}
}
for (j=0; j<bytesofcolor;j++){
color = instream[j+i];
colorbyte[j] = color;
}
for (times=0;times< counttotal;times++) {
for (k=0;k<bytesofcolor;k++){
write(1, &colorbyte[k], 1);
}
}
count=0;
}
}
But in the end, there are countless ways to run length encode an image. I've seen vertical encodings, diagonal, and horizontal. I've just simply found 3 used to encode images in some bootloaders...
Just flicked through a power point doc on image encoding including rle for tv transmission, it can be done with the bitmap fields as well!!
With any luck they'll keep it simple
Thought the LCD doc may come in handy as just for the screen being lcd, I don't think I've come across anything on screen images and encoding in Qualcomm docs so it's probably down to manufacturer how they do it (samsung and LG may use same as used on their TV's they make )
Bashing away at my HTC Desire C
Antagonist42 said:
Just flicked through a power point doc on image encoding including rle for tv transmission, it can be done with the bitmap fields as well!!
With any luck they'll keep it simple
Thought the LCD doc may come in handy as just for the screen being lcd, I don't think I've come across anything on screen images and encoding in Qualcomm docs so it's probably down to manufacturer how they do it (samsung and LG may use same as used on their TV's they make )
Bashing away at my HTC Desire C
Click to expand...
Click to collapse
I think they'll keep it simple, as they have for this reason: The images that are run length encoded in android, usually so far, are meant to be decoded and displayed instantaneously. If you go to a more complex algorithm to encode images in a run length fashion it is only practical for storage! I went back and forth about whether to write this guide for weeks before ever releasing any of this information, because I feared they would change it up to something that would be almost impossible to figure out, barring a leak. Then I came to the realization that they are pressed for speed on boot, and I don't think they want to make it any more complex, and if they do, oh well the game is on again.
I think that is part of the reason that the nexus 5 and 4 which use the 1 byte count 3 byte color, encode the whole bootloader screen in one pass. Because that method of decoding is easily the slowest, so instead of decoding the android at specified location, then the text, then the buttons..... They just draw the whole screen. I really think they're trying to save as much time as possible when it comes to these decoding algorithms.
Not just that but more complexity would mean a decoder, screen drivers (possibly) memory space to decompress & read it from....probably more trouble than it's worth to go hi-tech just for (basically) a splash screen in general but definitely would be worth (in my eyes) having a GUI you could run it through and alter the bit fields (sad oldie stuck on windows and no time to really get into linux :what: ).
Bashing away at my HTC Desire C
Antagonist42 said:
Not just that but more complexity would mean a decoder, screen drivers (possibly) memory space to decompress & read it from....probably more trouble than it's worth to go hi-tech just for (basically) a splash screen in general but definitely would be worth (in my eyes) having a GUI you could run it through and alter the bit fields (sad oldie stuck on windows and no time to really get into linux :what: ).
Bashing away at my HTC Desire C
Click to expand...
Click to collapse
True! :good:
The following images were all extracted using option 8 - Extract Jpgs From File
AT&T Note 3
Crespo i9020
So I wanted to implement something like TeslaMate for my UIS7862. The idea being to be able to visualize trips, and various vehicle stats from Grafana (and maybe a live location tracker).
My original plan was to use TorquePro to log vehicle stats + GPS location, and then to send those logs to a listening webserver for storage in Prometheus and display via Grafana. I found an Automate script to hook this into HomeAssistant here: here. However, I wanted a few additional items:
I don't have a SIM card in my radio and do not normally have it connected to my phone as a hotspot, so internet connection is intermittent and I didn't want to lose data
I wanted to be able to upload to different IP addresses depending on whether I'm connected to the home network (i.e. at home) or otspot (i.e I'm driving)
I wanted to be able to store stats from the media canbus (like fuel level) that don't seem to be available on OBD-II (at least I haven't found them for my GTI)
I wanted to learn Kotlin and write a 'real' Android app
I was successful in writing an app that would send all unsent Torque data to my home server once it connects via wifi (basically reproducing Rob's Automate script), but getting the canbus data out of the radio required more work.
I decompiled 190000000_com.syu.canbus.apk and set about learning how it worked, and trying to connect my own app.
What I found so far:
Unlike the MKC/D units which appear to communicate with the canbus module via a serial port, the FYT radios seem to use I2C
The com.syu.ms.apk is responsible for the hardware communication
the com.syu.canbus.apk connects to the com.syu.ms.toolkit Intent to access hardware data.
This Service provides a getRemoteModule() procedure which seems to provide 4 different interfaces:
0: the 'Main' interface
4: the Sound interface
7: the Canbus interface
14: the 'CanUp' interface (no idea what this is)
each interface (IRemoteModule) provides 4 commands: cmd, get, register, unregister
The 'register' command registers a callback to a specific ID. That callback will be called when the value at the ID changes.
For instance, on my GTI, ID '6' of the canbus module is the fuel-level. I can register a callback at ID=6, and that callback will be called whenever the value changes
I haven't spent time to look at the other modules, nor what the 'get' or 'cmd' functions do
With the above, I now have a rudimentay application that will fetch the Fuel level from the radio (via the Canbox). My plan is to incorporate this with the OBDII capture to create a composte data-set to upload to my prometheus database. Interestingly, the com.syu.ms.toolbox only responds back to me if I use the 'com.syu.ipc.[IModuleCallback|IRemoteModule|IRemoteToolkit' descriptor.
I will make everything above available on GitHub once I've cleaned it up a bit. It should be possible to extract any canbus data the radio has (along with other internal state depending on what the other modules expose). However, what I've learned is that every CanBox has a different interface and presents diferent data, so the effort to make a generic interface would be very high and beyond the scope of what I plan to do. There are about 2500 unique CanBoxes listed in FinalCanbus. I see about 600 unique classes implementing these modules, each of which implements a different set of registerable IDs.
I plan to add an interface to register any ID if you know what you are looking for to my app. I think @surfer63 could do the same to FytHwOneKey if they were so inclined, but without a table of which features are available it would only likely benefit programmers.
I'll update this post with a GitHub link once available, but I thought there might be some interest in the canbus analysis stuff.
Here is the GitHub repository for the Canbus access library: https://github.com/AxesOfEvil/FYTCanbusMonitor
The CanBox ID is specified by ID=1000.
The low 16 bits appear to specify the canbox type, and the upper 16bits seem to represent the car make/model. This mapping happens in syu.ms.module.canbus.HandlerCanbus with the name mapping in module.canbus.FinalCanbus
Here is an example of the IDs for (some) Reise RZS CanBox to give an idea of what type of data is available:
Code:
U_CUR_OIL_EXPEND 0
U_MISC_BEGIN 0
U_LOW_OIL_WARN 1
U_LOW_BATTERY_WARN 2
U_LIFE_BELT_WARN 3
U_CLEAN_FLUIT_WARN 4
U_HANDLE_BRAKE_WARN 5
U_RESIDUAL_OIL 6
U_BATTERY_VOLTAGE 7
U_DRIVE_MILE 8
U_PARK 9
U_RADAR_MUTE 10
U_CUR_SPEED 11
U_ENGINE_SPEED 12
U_OUT_TEMP 13
U_AIR_BEGIN 14
U_AIR_POWER 14
U_MISC_END 14
U_AIR_BIG_WIND_LIGHT 15
U_AIR_LITTLE_WIND_LIGHT 16
U_AIR_AC 17
U_AIR_MAX 18
U_AIR_CYCLE 19
U_AIR_DUAL 20
U_AIR_REAR 21
U_AIR_BLOW_UP 22
U_AIR_BLOW_BODY 23
U_AIR_SHOW 24
U_AIR_BLOW_FOOT 25
U_AIR_WIND_LEVEL 26
U_AIR_TEMP_LEFT 27
U_AIR_TEMP_RIGHT 28
U_AIR_AQS 29
U_AIR_SEAT_HEAT_LEFT 30
U_AIR_REAR_LOCK 31
U_AIR_AC_MAX 32
U_AIR_SEAT_HEAT_RIGHT 33
U_AIR_TEMP_OUT 34
U_AIR_AUTO 35
U_AIR_END 36
U_DOOR_BEGIN 37
U_DOOR_ENGINE 37
U_DOOR_FL 38
U_DOOR_FR 39
U_DOOR_RL 40
U_DOOR_RR 41
U_DOOR_BACK 42
U_DOOR_END 43
U_AIR_FRONT 44
U_AIR_BLOW_MODE 45
U_CNT_MAX 46
AxesofEvil said:
I plan to add an interface to register any ID if you know what you are looking for to my app. I think @surfer63 could do the same to FytHwOneKey if they were so inclined, but without a table of which features are available it would only likely benefit programmers.
Click to expand...
Click to collapse
Nice work you are doing here.
But I do not know what you mean with above statement.
For further reading: lbdroid did some reverse engineering in 2006.
You might take a look at some of his repos: https://github.com/lbdroid/MCUd
In that github/readme are 5 other repos. They are outdated, but might still give you some clues.
surfer63 said:
Nice work you are doing here.
But I do not know what you mean with above statement.
For further reading: lbdroid did some reverse engineering in 2006.
You might take a look at some of his repos: https://github.com/lbdroid/MCUd
In that github/readme are 5 other repos. They are outdated, but might still give you some clues.
Click to expand...
Click to collapse
Sorry, maybe that was inappropriate. I guess I was thinking about ways to give users access to the canbox data since Tasker doesn't seem able to hook into services this way. One use case would be direct access to steering wheel buttons from the canbox (my understanding is that in some cases FwHwOneKey can't handle canbus related buttons...maybe I'm wrong). Or, perhaps there isn't really any use at all for this info to trigger user applications.
I know there was a request to access Canbox data for widgets (for instance to be able to display the outside temperature on a custom screen). This method should be able to support something like that, but I have no idea if there is an existing app that could make use of it. Maybe I could write a proxy that would turn service updates into system broadcast events? Just spitballing here.
Wow, it’s really communicate with canbus from user apps?
May be there is way to read can data, like we can see in develop mode
Sdese2000 said:
Wow, it’s really communicate with canbus from user apps?
May be there is way to read can data, like we can see in develop mode
Click to expand...
Click to collapse
To be clear, I only have access to whatever the canbox has already decoded (and the radio has accepted), at least on my vehicle, thee is a lot more CAN traffic that is ignored. What CAN data do you see in develop mode? I am not aware of this.
AxesofEvil said:
Sorry, maybe that was inappropriate. I guess I was thinking about ways to give users access to the canbox data since Tasker doesn't seem able to hook into services this way. One use case would be direct access to steering wheel buttons from the canbox (my understanding is that in some cases FwHwOneKey can't handle canbus related buttons...maybe I'm wrong). Or, perhaps there isn't really any use at all for this info to trigger user applications.
Click to expand...
Click to collapse
Not inappropiate at all. I just didn't get what you meant.
And yes: The BT like commands are still a big misunderstanding (for me that is). I think that could very well be a combi of activity, canbus and "something else"
But as my unit doesn't have buttons anymore, and neither my previous one, I don't spend time on my own app anymore.
AxesofEvil said:
To be clear, I only have access to whatever the canbox has already decoded (and the radio has accepted), at least on my vehicle, thee is a lot more CAN traffic that is ignored. What CAN data do you see in develop mode? I am not aware of this.
Click to expand...
Click to collapse
In Head Unit settings there is trigger, if turn on it, can logs will appear on the screen.
If found some code in com/syu/util/DebugViev.jave that provide it
Spoiler
package com.syu.util;
import android.content.Context;
import android.graphics.Canvas;
import android.graphics.Paint;
import android.support.p000v4.internal.view.SupportMenu;
import android.view.View;
import android.view.WindowManager;
import java.util.Locale;
public class DebugView extends View {
private int CELL_HEIGHT = 35;
int[] COLOR = {SupportMenu.CATEGORY_MASK, -1, -16711936, -256, -16776961};
private final int MAX = 16;
private final int TEXT_SIZE = 23;
/* access modifiers changed from: private */
public int[] mColors = new int[16];
/* access modifiers changed from: private */
public int mCount;
private boolean mDbg = false;
/* access modifiers changed from: private */
public int mLastIndex;
private WindowManager.LayoutParams mLp = ToolkitApp.buildOverlayLayoutParams(-1, -1);
/* access modifiers changed from: private */
public int mMsgCnt;
/* access modifiers changed from: private */
public String[] mMsgs = new String[16];
private Paint mPaint = new Paint();
public DebugView(Context context) {
super(context);
init();
}
private void init() {
this.mPaint.setAntiAlias(true);
this.mPaint.setTextSize(23.0f);
this.mPaint.setColor(-1);
}
public void setDbg(boolean flag) {
this.mDbg = flag;
}
public boolean isDbg() {
return this.mDbg;
}
public WindowManager.LayoutParams getWindowLayoutParams() {
return this.mLp;
}
public void msg(String msg) {
if (this.mDbg && msg != null) {
HandlerUI.getInstance().post(new MessageHelper(msg));
}
}
public void msg2(String msg) {
if (this.mDbg && msg != null) {
HandlerUI.getInstance().post(new MessageHelper(msg));
}
}
public void msgHex(String str, byte[] data, int start, int length) {
if (this.mDbg && data != null) {
if (data.length - start < length) {
length = data.length - start;
}
String msg = String.valueOf(str) + " * ";
for (int i = 0; i < length; i++) {
String c = Integer.toHexString(data[start + i] & 255).toUpperCase(Locale.CHINA);
if (c.length() < 2) {
c = "0" + c;
}
msg = String.valueOf(msg) + c + " ";
}
HandlerUI.getInstance().post(new MessageHelper(msg));
}
}
public void msgHex(String str, int[] data, int start, int length) {
if (this.mDbg && data != null) {
if (data.length - start < length) {
length = data.length - start;
}
String msg = String.valueOf(str) + " * ";
for (int i = 0; i < length; i++) {
String c = Integer.toHexString(data[start + i] & 255).toUpperCase(Locale.CHINA);
if (c.length() < 2) {
c = "0" + c;
}
msg = String.valueOf(msg) + c + " ";
}
HandlerUI.getInstance().post(new MessageHelper(msg));
}
}
private class MessageHelper implements Runnable {
private String mMessage;
public MessageHelper(String msg) {
this.mMessage = msg;
}
public void run() {
DebugView debugView = DebugView.this;
debugView.mLastIndex = debugView.mLastIndex + 1;
DebugView debugView2 = DebugView.this;
debugView2.mCount = debugView2.mCount + 1;
if (DebugView.this.mLastIndex > 15) {
DebugView.this.mLastIndex = 0;
}
if (DebugView.this.mCount > 16) {
DebugView.this.mCount = 16;
}
DebugView debugView3 = DebugView.this;
debugView3.mMsgCnt = debugView3.mMsgCnt + 1;
DebugView.this.mMsgs[DebugView.this.mLastIndex] = String.format("%06d @ %s", new Object[]{Integer.valueOf(DebugView.this.mMsgCnt), this.mMessage});
DebugView.this.mColors[DebugView.this.mLastIndex] = DebugView.this.COLOR[DebugView.this.mLastIndex % DebugView.this.COLOR.length];
DebugView.this.invalidate();
}
}
/* access modifiers changed from: protected */
public void onDraw(Canvas canvas) {
if (this.mCount != 0) {
int count = this.mCount;
int firstIndex = (this.mLastIndex - count) + 1;
if (firstIndex < 0) {
firstIndex += 16;
}
if (firstIndex + count > 16) {
int rightCount = 16 - firstIndex;
int leftCount = count - rightCount;
for (int i = 0; i < rightCount; i++) {
int index = firstIndex + i;
this.mPaint.setColor(this.mColors[index]);
canvas.drawText(this.mMsgs[index], (float) 5, (float) ((i + 1) * this.CELL_HEIGHT), this.mPaint);
}
for (int i2 = 0; i2 < leftCount; i2++) {
this.mPaint.setColor(this.mColors[i2]);
canvas.drawText(this.mMsgs[i2], (float) 5, (float) ((rightCount + i2 + 1) * this.CELL_HEIGHT), this.mPaint);
}
return;
}
for (int i3 = 0; i3 < count; i3++) {
int index2 = firstIndex + i3;
this.mPaint.setColor(this.mColors[index2]);
canvas.drawText(this.mMsgs[index2], (float) 5, (float) ((i3 + 1) * this.CELL_HEIGHT), this.mPaint);
}
}
}
}
I have updated the OP with a link to the GitHub library (here). The library is not really meant to be used standalone, but instead to be incorporated into other projects. I haven't posted the code for the logger as there is still quite a bit more to do on that side.
The library repo does include an example application which will simply log every message received to the screen/logfile (in /Downloads). It is very inefficient since it just blindly asks for every possible ID regardless of whether it is actually available for a given CanBox or not, but is meant to give a quick idea of what data is available and a short example of how to use the library. The latest compiled APK can be found here: https://github.com/AxesOfEvil/FYTCanbusMonitor/releases
I found this interesting tidbit today:
It seems to be that arbitrary commands can be sent to the canbus through the radio via sys.ms by calling ToolkitDev.writeMcu(0xE3, PID, data-len, data0, data1, ...) (where data can be 1-8 bytes).
Edit: ToolkitDev.writeMcu(0xE3, ...) seems to write commands to the canbox module. As I don't have the source for teh module, I'm not sure how it handles these commands, but they don't g out verbatim on the canbus itself.
There is also ToolkitDev.writeCanbusDirect, but this may send commands via an OBDII dongle...Edit: this seems to just directly send raw commands to the CanBox. It is similar to the above but requires manually calculating the entire packet (including checksum)
I have not found a way to pass arbitrary data from an external app through an intent to allow other apps to send arbitrary canbus commands, but with a hacked syu.ms, it probably means I can eliminate the DynAudio AMP control box I had to make to get my audio working. And that with more hacking, it may be possible to send GPS directions and music info to the HUD.
AxesofEvil said:
The CanBox ID is specified by ID=1000.
The low 16 bits appear to specify the canbox type, and the upper 16bits seem to represent the car make/model. This mapping happens in syu.ms.module.canbus.HandlerCanbus with the name mapping in module.canbus.FinalCanbus
Here is an example of the IDs for (some) Reise RZS CanBox to give an idea of what type of data is available:
Code:
U_CUR_OIL_EXPEND 0
U_MISC_BEGIN 0
U_LOW_OIL_WARN 1
U_LOW_BATTERY_WARN 2
U_LIFE_BELT_WARN 3
U_CLEAN_FLUIT_WARN 4
U_HANDLE_BRAKE_WARN 5
U_RESIDUAL_OIL 6
U_BATTERY_VOLTAGE 7
U_DRIVE_MILE 8
U_PARK 9
U_RADAR_MUTE 10
U_CUR_SPEED 11
U_ENGINE_SPEED 12
U_OUT_TEMP 13
U_AIR_BEGIN 14
U_AIR_POWER 14
U_MISC_END 14
U_AIR_BIG_WIND_LIGHT 15
U_AIR_LITTLE_WIND_LIGHT 16
U_AIR_AC 17
U_AIR_MAX 18
U_AIR_CYCLE 19
U_AIR_DUAL 20
U_AIR_REAR 21
U_AIR_BLOW_UP 22
U_AIR_BLOW_BODY 23
U_AIR_SHOW 24
U_AIR_BLOW_FOOT 25
U_AIR_WIND_LEVEL 26
U_AIR_TEMP_LEFT 27
U_AIR_TEMP_RIGHT 28
U_AIR_AQS 29
U_AIR_SEAT_HEAT_LEFT 30
U_AIR_REAR_LOCK 31
U_AIR_AC_MAX 32
U_AIR_SEAT_HEAT_RIGHT 33
U_AIR_TEMP_OUT 34
U_AIR_AUTO 35
U_AIR_END 36
U_DOOR_BEGIN 37
U_DOOR_ENGINE 37
U_DOOR_FL 38
U_DOOR_FR 39
U_DOOR_RL 40
U_DOOR_RR 41
U_DOOR_BACK 42
U_DOOR_END 43
U_AIR_FRONT 44
U_AIR_BLOW_MODE 45
U_CNT_MAX 46
Click to expand...
Click to collapse
Where did these IDs come from? Did you find one for Illumination/Headlights?
The IDs came out of the source code for 190000000_com.syu.canbus.apk
The IDs are canbox and probably vehicle specific, so such info may be available, but you need to identify exactly what you are looking for.
Use JadX or BytecodeViewer or a similar application to analyze the apk file above, and look in app/src/main/java/module/canbus for the appropriate Canbox for your vehicle