Hi,
I was wondering how accurate Quadrant and other benchmarks are when it comes to evaluating ROM performance.
Across all of the custom ROMs I've flashed, most of them have led to lower Quadrants (around 200 points lower) as compared to the stock ROM.
Some reading suggests that Quadrant scores can be easily spoofed, but nevertheless, how can I interpret this result?
That's the problem. They are easily faked. The only real way is by real world usage.
Wayne Tech S-III
Related
My Vibrant took a tumble onto an uneven parking lot last weekend and busted the screen. As much as I like my Vibrant, and since I have to replace it, I've been trying to decide whether to spend a little more and upgrade to a better phone or not.
I'm looking at Quadrant scores around, and the Nexus S is showing around 1600, while my Vibrant shows a score of 951. I was under the impression that the two phones were running very similar hardware, so I don't understand seeing a 68% increase in benchmark scores. Does TouchWiz really eat up that much of the phone's power, or is it something else?
jfleegle said:
My Vibrant took a tumble onto an uneven parking lot last weekend and busted the screen. As much as I like my Vibrant, and since I have to replace it, I've been trying to decide whether to spend a little more and upgrade to a better phone or not.
I'm looking at Quadrant scores around, and the Nexus S is showing around 1600, while my Vibrant shows a score of 951. I was under the impression that the two phones were running very similar hardware, so I don't understand seeing a 68% increase in benchmark scores. Does TouchWiz really eat up that much of the phone's power, or is it something else?
Click to expand...
Click to collapse
Its because the Vibrant uses a different RFS file system while Nexis S uses ext4 which is a whole lot better and faster.
Sent from my HTC Glacier using XDA Premium App
jfleegle said:
Arrow
Why is there such a difference in Quadrant over Vibrant?Arrow
Click to expand...
Click to collapse
becasue 2.3 JIT is optimized for hummingbird were 2.2 JIT isn't
JIT will drastically improve CPU performance
posted this a while back...my vibrant on 2.3 ignore I/O score in green and note the CPU score in blue
http://forum.xda-developers.com/showthread.php?t=879662
here is vibrant on 2.2 again ignore the I/O score and look at CPU
http://www.eskwadrat.com/vibrant/CAP201008311008.jpg
mackster248 said:
Its because the Vibrant uses a different RFS file system while Nexis S uses ext4 which is a whole lot better and faster.
Click to expand...
Click to collapse
that isn't true.
I had a vibrant and had voodoo lag fix that converts the file system to ext4 it only helps with i/o score
If you use the paid version on quadrant you will see the boost is in the CPU score ...Google optimized JIT in 2.3 for hummingbird CPU
vibrant will score pretty much the same as NS once it gets 2.3
So, after playing around with several different roms I have a better idea of my own likes and dislikes, but I came across an interesting phenomenon. When switching from InsertCoin 3.3.1 to Pyramid 3D 8.0.0, I felt that P3D was faster (or at least as fast) as InsertCoin. When I checked the benchmark scores in CF-bench, P3D scored MUCH lower than InsertCoin.
Does anyone know why a rom might seem faster, but actually perform worse on a benchmark?
the overall UI can be fast with lower benchmark scores.
But try heavy 3D apps, like N64oid or Tegra2 games, most of the time the fastest ROM in benchmarks will be faster for 3D rendering.
Also true for apps that need time to render something, like for example Photaf that needs some time to render a panoramic photo, etc
I mean in a ROM, the UI can be well optimized, fast, but 3D can be slow, or vice versi
Hey Guy's,
I just purchased the Infuse 4G and retired my Xperia X10. In real world use the Infuse feels much snappier and faster then my Xperia X10, however i noticed that it benches much lower scores in Linpack despite having a faster 1.2ghz processor vs 1Ghz on the X10.. Why is that?
On my Infuse 4G i'm getting about 17mflops, on my experia i would get between 34 and 37.. what gives? Does this have anything to do with the JIT implementation on this phone, or is this benchmark just to artificial to mean anything? Are you guy's seeing higher numbers on custom roms? I'm currently stock besides being rooted with Lagfix enabled on Rogers with GB 2.3.3.
I know that Quadrant scores don't really mean anything, but i thought the Linpack floating point operations were more of a real world brute strength test.. am i wrong?
DaMeatMan said:
Hey Guy's,
I just purchased the Infuse 4G and retired my Xperia X10. In real world use the Infuse feels much snappier and faster then my Xperia X10, however i noticed that it benches much lower scores in Linpack.. Why is that?
On my Infuse 4G i'm getting about 17mflops, on my experia i would get between 34 and 37.. what gives? Does this have anything to do with the JIT implementation on this phone, or is this benchmark just to artificial to mean anything? Are you guy's seeing higher numbers on custom roms? I'm currently stock besides being rooted with Lagfix enabled on Rogers with GB 2.3.3.
I know that Quadrant scores don't really mean anything, but i thought the Linpack floating point operations were more of a real world brute strength test.. am i wrong?
Click to expand...
Click to collapse
ehhhh i dont think they play a real role.......i mean i have had Quads in the 1000's and my phone be super snappy.....then i have had quads in the 2500's and my phone actually lag more. and Linpack is just your phone calculating FLoating points which is not a real test to me.......but i get linpacks around 27 with Cm7 and Miui
and Quads of 3500 and up.
but in the end, i dont really care with the results of those test are.
To me the biggest test is actual real world use, and how snappy the phone navigates menus, smooth scrolling, and launching apps. In that regard i'm extremely happy with the Infuse 4G. I was just curious though to see what might be going on behind the scenes in terms of affecting the Linpack scores.
How do you like CM7 and MIUI, and how does it compare in your eyes to the stock ROM in terms of real world use feel, smoothness etc..? I'm keeping mine pretty much stock for now since to my knowledge there are still issues with bluetooth and possibly wifi on the current custom rom's.. or has this changed recently?
If anyone else has theories on why this may be happening please feel free to chime in.
DaMeatMan said:
To me the biggest test is actual real world use, and how snappy the phone navigates menus, smooth scrolling, and launching apps. In that regard i'm extremely happy with the Infuse 4G. I was just curious though to see what might be going on behind the scenes in terms of affecting the Linpack scores.
How do you like CM7 and MIUI, and how does it compare in your eyes to the stock ROM in terms of real world use feel, smoothness etc..? I'm keeping mine pretty much stock for now since to my knowledge there are still issues with bluetooth and possibly wifi on the current custom rom's.. or has this changed recently?
If anyone else has theories on why this may be happening please feel free to chime in.
Click to expand...
Click to collapse
Only Non functioning things are Bluetooth and HDMI....i have had NO ISSUES at all with WiFi.
other do though, it may just be that i have a good router. not saying theirs are bad.....but who knows what different brands and set ups may be playing a role in the Wifi issues.
I like them Both.....both are Super fast. both are customizable(MiUi more so)
i can do stock....i have went back to stock for a week or so. and i just kept getting that itch to FLASH....lol
Thanks for the feedback! No Bluetooth is currently a deal breaker for me since i use it often in the car to play my music.
If that issue get's resolved you can be sure i'll be flashing the same day!
Linpack measures floating point, since floating point is processed in a dedicated part of the processor it had no relevance to something more common and CPU intensive like video decoding. Different processors have different strengths and floating point is not one of them on the hummingbird. Yet the hummingbird is very good at decoding video for a single core, as well as being great at open gl for a processor from 18 months ago.
Linpack gives a single point of data but many are needed to truly measure performance. It's nothing to worry about since you probably won't be crunching large data sets on you phone that use floating point.
As i'm sure most are aware, note3 and others locks the system on a high performance mode when it runs predefined benchmark apps like http://arstechnica.com/gadgets/2013...rking-adjustments-inflate-scores-by-up-to-20/ explains in good detail.
So ofc this makes the note3 seem a bit more powerful than is actual real usage conditions, and, since benchmarks will be the deciding factor for many ppl (and there's loads of benchmarks results and pics all over the place), that's a lot like cheating the consumer for profit...
So in my view, all the benchmarks that use standard benchmark apps are invalid and shouldn't really be taken in consideration for when a person is deciding to buy a phone.
The good thing is, like ars shows, this high performance mode for bench apps can be circumvented by changing it's name.
I didn't really search much around, but i was wondering if there's any place with a collection of benchmark apps with the name changed so that they're not affected by the cheat, and maybe the app's creators should take that into consideration and do some fix/workaround to prevent the cheating...
I don't get why people call it cheating. I feel that benchmarking apps are supposed to ramp up all cores to 100% and then only start the test. Only this will truly show the full potential of the device.
Samsung is only increasing the cpu clock to its max while running benchmark apps and there is no overclocking involved. This does not mean they are cheating.
system.img said:
I don't get why people call it cheating. I feel that benchmarking apps are supposed to ramp up all cores to 100% and then only start the test. Only this will truly show the full potential of the device.
Samsung is only increasing the cpu clock to its max while running benchmark apps and there is no overclocking involved. This does not mean they are cheating.
Click to expand...
Click to collapse
did you really understand what is going on? by just changing a name they get lower score. maybe if samsung allowed us to insert app names to this list then it might be ok but this way it is just cheating because i will not see the performance that a benchmark shows me in a game.
Did you buy any phone based on the benchmark score?
Kisses99 said:
Did you buy any phone based on the benchmark score?
Click to expand...
Click to collapse
it is not important if anybody looks benchmarks before buying phone. it is still cheating and we are talking about cheating.
kromosto said:
did you really understand what is going on? by just changing a name they get lower score. maybe if samsung allowed us to insert app names to this list then it might be ok but this way it is just cheating because i will not see the performance that a benchmark shows me in a game.
Click to expand...
Click to collapse
The benchmark app by itself is not maxing the CPU speed before taking the benchmark, which the developer should code it to do.
Samsung is just maxing out the CPU for the highest score possible on the hardware. If the game is optimised right, it will give the actual performance the benchmarks indicate as the hardware is more than capable of handling it.
Elusivo said:
As i'm sure most are aware, note3 and others locks the system on a high performance mode when it runs predefined benchmark apps like http://arstechnica.com/gadgets/2013...rking-adjustments-inflate-scores-by-up-to-20/ explains in good detail.
So ofc this makes the note3 seem a bit more powerful than is actual real usage conditions, and, since benchmarks will be the deciding factor for many ppl (and there's loads of benchmarks results and pics all over the place), that's a lot like cheating the consumer for profit...
So in my view, all the benchmarks that use standard benchmark apps are invalid and shouldn't really be taken in consideration for when a person is deciding to buy a phone.
The good thing is, like ars shows, this high performance mode for bench apps can be circumvented by changing it's name.
I didn't really search much around, but i was wondering if there's any place with a collection of benchmark apps with the name changed so that they're not affected by the cheat, and maybe the app's creators should take that into consideration and do some fix/workaround to prevent the cheating...
Click to expand...
Click to collapse
How profound. Search is your friend.
http://forum.xda-developers.com/showthread.php?t=2465518
Big freaking deal. I disable power saving on my laptop so it runs at 100% all the time. Plus, we've seen impressive results from Anomaly 2 and Epic Citadel game engine benchmarks that don't disable power saving.
Honestly, who cares. Most consumers don't look at benchmarks when buying phones. Benchmark consumers probably account for maybe 2% of sales. The majority of people who benchmark out of that 2% do so in order to test differences when removing bloat or changing roms, and not to gauge real world performance off of the shelf.
Samsung is one of many companies that do this benchmark manipulation. Since most companies do this, I see it as a non issue. It's not as if they are producing false results, they are only making there products run at there best for certain programs. How can we be surprised or be offended?
Here is an article testing all the major android phone manufacturers and all but Moto and Nexus phones manipulate benchmark results.
Android Benchmarking
Mod Edit
Duplicate thread is duplicate
Thread closed
malybru
Forum Moderator
With the last ota patch, antutu scores much lesser~62000. The original nougat returned ~65000.
Antutu scores may not matter in real life, but the difference in scores is quite vast.
"Much lesser" and "only" 62K?? I thought you get 40K to be much lesser. Don't rely on these benchmarks, it doesn't affect the real world usage.
sanyasi420 said:
With the last ota patch, antutu scores much lesser~62000. The original nougat returned ~65000.
Antutu scores may not matter in real life, but the difference in scores is quite vast.
Click to expand...
Click to collapse
it depends on how much your CPU is tired.. if your CPU is exhausted working heavily.. it results in lower and lower and further lower benchmarks