DxOMark results are...

[etherial]

... a waste of time! (IMO!)

I always had my suspicions about these DxOMark tests and have read various people talking them up (recently Nikon fan-boys about the D800 score) and also people saying they were rubbish. (Note this is NOT a brand war thread!)

Well today I've gained some experience to make my mind up. I normally shoot with a 7D but recently I forgot to take it to a herding event and borrowed a Nikon D5100 to take some shots. (the best camera you have is the one in your hand and at least I got some shots!!) I've just edited those photos and I'm blown away by how washed out and poor the photos were. They all required almost a 1 stop bump in exposure and a strong contrast tone curve to be applied and then further pushing with highlight and shadow sliders. I really struggled with getting a decent looking end result without breaking those pixels. I am amazed at just how poor the files were out of the camera. Examples are below, sadly I don't have a 7D photo from the same conditions to compare, but I have shot there with my 7D on different days and been happy with the results. Of course the lens will have a significant bearing on this as well, but again, my experience with my old 450D and kit lens was far better that these. (Again I'm not trying to bag the 5100, just trying to illustrate the point about the test scores).

So out of interest I jumped onto the DxO scores for the 7D and the D5100 and was amazed to see that the D5100 scores massively higher than the 7D. I can honestly say that these results are total bollocks! The results should be the exact opposite in my opinion. These results may give them a number in a lab but in a real life situation, it couldn't be further to the truth. To take it a step further I compared the D5100 to the new Canon 5D3. OMG the D5100 scores just one point lower!! Now this of course is a very unscientific test, and the glass will make a difference, but the comparison of these scores is simply staggering.

Comparison at DxOMark here.

I'm blown away by this. I was looking at these results for the 5D3 a couple of weeks ago trying to get a gauge of what sort of difference I would see from my 7D to the 5D3 I'm thinking of upgrading to. After seeing this today, I will be deleting the DxOMark from my bookmarks. It is useless. The real world is where it matters.


This is one example from the day, all the photos had very similar setting changes in Lightroom. This thread could easily double as a "why you MUST post process your photographs" thread!

Original jpg straight out of the camera cropped mildly to improve composition, shot in Aperture priority, ISO400 f5.6 1/800s, pattern metering. The histogram is bunched up in the middle and no-where near the blacks or whites.




Lightroom edited, Strong Tone curve applied, Exposure +0.9, Contrast -5, Highlights -78, Shadows -54, whites +2, blacks -71, clarity +25, vibrance +5, saturation -15, and some sharpening. I'm still not happy with the hue of the red, but I think the result is reasonable. Overall I think I've managed to push this image as far as I'd dare, I'm not real impressed with some of the banding in some areas that has resulted.

[Eberbachl]

I have extensive experience with Nikons and have even owned a D5100. You can very easily get outstanding images straight out of the camera.

If I went shooting with a borrowed 7D I'd probably produce sub-par images, but I wouldn't blame the camera as I know full well it's capable of stunning results.

[etherial]

As I said this IS NOT a Nikon bashing thread. You have missed my point entirely.

[Lance B]

I think what DxOMark results are eluding to is the potential of the IQ that can be achieved from a specific camera, which is exactly what you have shown with your post processing.

Just because you have to work at an image from a camera doesn't mean that it is bad, nor that DxOMark's test results are wrong.

I think you are being a little oversimplistic with your test methodology.

[etherial]

I agree it is a rather unscientific test, but think about this; they rate the 5D3 and the D5100 only 1 point different, are you suggesting they are capable of similar results? I can't imagine getting an image straight out of the camera from a 5D3 like the first one above. If that were the case why bother getting better bodies, just invest in glass, which reinforces my thoughts that such ratings are a waste of time.

[Tommo1965]

i think you might have the camera set to standard image in Jpeg..that gives it a washed out look IMO..try Vivid next time...then you will get a image similar to the edited version of your shot ..all raw shots will have the washed out look too if you shoot with matrix metering and let the camera do the work...or exposure compensate for a better look straight from the camera


so as far as DXO..not sure that relevant in this case..more that fact that your not used to the Nikon and have it set different than what your used too

[ameerat42]

Etherial. Much as I hayte to poke fingers at people's experiences, this just sounds to me like an experience that totally failed to meet your expectations.

As you presented the situation, it doesn't prove anything. It's all anecdotal.

I don't give a pair of owl's hoots about DxO marks either, but I will say that what you presented is hardly on a par with whatever testing regime they (DXO-ers) based their conclusions on.

Much of your dissertation reads like a powerful negative reaction to something. What settings were on the camera? Standard everythings?

Basically, it's a different camera from what you're used to. I would say, expect some different results, and when you get them, try to interpret them a little more constructively than just the general waste of time that you have reported.

Please don't feel offended.
Am.

[Eberbachl]

As I said this IS NOT a Nikon bashing thread. You have missed my point entirely.

No, not at all... you're implying that the D5100 does not produce nice jpegs out of the camera.

The fact of the matter is that you have not adjusted it's settings correctly to produce the sort of image you require. It's simply a case of driver error. The D5100 can produce sharp contrasty and nicely saturated images with very little effort if it's set up correctly.

[Eberbachl]

Additionally, the specific area in the DxOMark comparison where the D5100 really caned the 7D is in low light performance. Hardly a relevant factor in the image you posted. The D5100 beat it in dynamic range where it's done a good job here (plenty of detail in the shadows and highlights), and only slightly beat it in colour depth.

There's nothing in the image above that actually disputes DxOMark's findings.

[Tannin]

DXO has been a well-known very poor joke for a long, long time. Mic, I don't pay any attention to speak of to your bad experience with a D5100. I bet you that once I got to know it I could produce excellent results with it. What's more, I bet that you could too. Or any of us here. It may or may not be slightly better or worse than the equivalent Canon (a 550D, I guess) but any difference will be very small.

But when you point out that DXO has just produced yet another absolute shocker of a result, yet another "scientific" finding so steeped in absurdity that it was probably originally rejected by the Japanese Whaling Publicity Office on the grounds that it was too ridiculous .... well, why would anyone be surprised?

DXO: the software which "proved" that the D200 had better high-ISO noise ability than a 5D Mark 1.

The DXO heritage: it started out completely useless and has maintained the tradition perfectly right up to the present day.

[Eberbachl]

DXO has been a well-known very poor joke for a long, long time. Mic, I don't pay any attention to speak of to your bad experience with a D5100. I bet you that once I got to know it I could produce excellent results with it. What's more, I bet that you could too. Or any of us here. It may or may not be slightly better or worse than the equivalent Canon (a 550D, I guess) but any difference will be very small.

But when you point out that DXO has just produced yet another absolute shocker of a result, yet another "scientific" finding so steeped in absurdity that it was probably originally rejected by the Japanese Whaling Publicity Office on the grounds that it was too ridiculous .... well, why would anyone be surprised?

DXO: the software which "proved" that the D200 had better high-ISO noise ability than a 5D Mark 1.

The DXO heritage: it started out completely useless and has maintained the tradition perfectly right up to the present day.

I don't want to be seen to be defending DxOMark here, as I never look at their scores and know little about their testing methodology.

Your comment that they rate the D200 higher than the 5D MkI in terms of low light performance intrigued me however as it seems ridiculous that they would do so, so I looked up the scores.

They rate the Nikon D200 at 583 ISO

They rate the 5D MkI at 1368 ISO (2.3 times better)

Seems to me that they rate the 5D MkI significantly higher (as they should) than the D200 in terms of low light performance.

[etherial]

OK well for those that see this as a Nikon bashing then fair enough, lets make it one!

Regardless of how good or bad the sensor is, if I as an reasonably experienced amateur photographer can't pick up this camera and get good shots out of the camera in a couple of hours, then the camera is poor (again IMO!). Furthermore, no matter how good a sensor is supposed to be if the camera can't process them to give me a reasonable result on basic settings without digging through menus or going away from standard picture styles, in my mind it is a poor camera. There will be many that will be disappointed with the results if that is a representative example of what to expect from it. I am not typically a brand fan-boy and I respect Nikon for producing some great products that I've considered strongly for various reasons.

I looked back through my Canon 450D photos with a kit lens and out of the camera jpgs are much more pleasing and easier to work with than these were. Look back at my photo of them month winner from years ago, that photo was taken with the 450 and kit lens on my very first day with it when I knew nothing! And the result out of the camera was pretty good. If these photos above are representative of what a D5100 produces, well I made the right choice way back then.

At the end of the day, my point still stands, lab tests like this that put the likes of a 5D3 on a overall par with a 5100 or in this case put a 5100 well ahead of a 7D are a waste of time because as I've demonstrated either through fault of the camera, or fault of my ability to use the camera in the real world the results will be wildly different. Even if you choose to believe the results regarding the sensor, there is much more to a camera and the resultant images it produces than just the sensor.

Just my opinion! Flame suit on!

[Eberbachl]

No body is making it a Nikon bashing thread... simply pointing out that if you don't succeed in adjusting the camera to get the results you wanted (that which the camera is quite capable of) it doesn't mean that the DxO Mark testing results are any less accurate. Only that you haven't optimised the camera correctly to suit your needs.

The discussion is brand neutral in my opinion, the same would go for any camera.

[Eberbachl]

Regarding the 5D MkIII Vs D5100 comparision, they say that the D5100 has a higher dynamic range, the 5D MkIII has much better low light ability, and the 5D MkIII slightly better colour depth. They're not saying anything much else about the cameras other than that. Do you dispute those specific results?

[etherial]

Regarding the 5D MkIII Vs D5100 comparision, they say that the D5100 has a higher dynamic range, the 5D MkIII has much better low light ability, and the 5D MkIII slightly better colour depth. They're not saying anything much else about the cameras other than that. Do you dispute those specific results?

I've not used a 5D3 so I can't really comment on the specifics of each rating. But the fact that overall they are rated only 1 point different tells me something is wrong with either their testing method or their score weightings. What I would say though is that they rate the 5100 better than the 7D by some margin in all areas which based on my experience I find staggering.

[Eberbachl]

I agree with the notion that the image out of the D5100 above is sub-par, but none of the issues with it relate to colour depth, dynamic range, or low light performance (DxO Mark's tests) so I wonder why based on that you would dispute their findings about the 7D Vs the D5100.

The 7D and D5100 cameras are both beaut pieces of gear and capable of excellent results when driven well. I know I wouldn't be able to drive a 7D nearly as well as a D5100 so my images from a D5100 are likely to be far superior than my images from a 7D. That in itself doesn't mean much except than I'm better at driving a Nikon than a Canon.

[etherial]

I agree with the notion that the image out of the D5100 above is sub-par, but none of the issues with it relate to colour depth, dynamic range, or low light performance (DxO Mark's tests) so I wonder why based on that you would dispute their findings about the 7D Vs the D5100.

Interesting. So if the image is poor, but for reasons not related to those that they test, then my only conclusion that there is something lacking from their testing regime which reinforces the title of my thread?

The 7D and D5100 cameras are both beaut pieces of gear and capable of excellent results when driven well. I know I wouldn't be able to drive a 7D nearly as well as a D5100 so my images from a D5100 are likely to be far superior than my images from a 7D. That in itself doesn't mean much except than I'm better at driving a Nikon than a Canon.

I bet you I could hand you my 7D and you would get good results from it easily. Hell I stick it in Aperture priority, auto ISO and give it to my wife who's photography abilities are surpassed by her iPhone and she still manages to take reasonable photos with it!

[Eberbachl]

It seemed to me that you were arguing that DxOMark's results were not accurate based on the image you presented. My point was that your image didn't do anything to dispute DxOMark's findings. Are DxOMark's tests useful? Maybe not, maybe so. I think it depends on whether you understand what they're testing and whether or not those tests are useful to you. Personally I never look at them. I'd rather look at photos than lab tests. I just haven't found anything here to dispute their accuracy - that's all.



Getting great photos from any camera depends on your ability to adjust the settings to taste. I don't know what any of the jpeg processing controls in the D5100 you used were set to, but I agree they were bland. I used to get great jpegs out of my D5100, and my wife could too.

[Tommo1965]

OK well for those that see this as a Nikon bashing then fair enough, lets make it one!

Regardless of how good or bad the sensor is, if I as an reasonably experienced amateur photographer can't pick up this camera and get good shots out of the camera in a couple of hours, then the camera is poor (again IMO!). Furthermore, no matter how good a sensor is supposed to be if the camera can't process them to give me a reasonable result on basic settings without digging through menus or going away from standard picture styles, in my mind it is a poor camera. There will be many that will be disappointed with the results if that is a representative example of what to expect from it. I am not typically a brand fan-boy and I respect Nikon for producing some great products that I've considered strongly for various reasons.

I looked back through my Canon 450D photos with a kit lens and out of the camera jpgs are much more pleasing and easier to work with than these were. Look back at my photo of them month winner from years ago, that photo was taken with the 450 and kit lens on my very first day with it when I knew nothing! And the result out of the camera was pretty good. If these photos above are representative of what a D5100 produces, well I made the right choice way back then.

At the end of the day, my point still stands, lab tests like this that put the likes of a 5D3 on a overall par with a 5100 or in this case put a 5100 well ahead of a 7D are a waste of time because as I've demonstrated either through fault of the camera, or fault of my ability to use the camera in the real world the results will be wildly different. Even if you choose to believe the results regarding the sensor, there is much more to a camera and the resultant images it produces than just the sensor.

Just my opinion! Flame suit on!

LMAO..sounds like your the only one here making it a Nikon bashing thread...and thats what you really want.....to say Canon is better....LOL:..and for you it probably is

[etherial]

LOL, I didn't want to make it one, my point really was directed more at the DxOMark ratings, but I think everyone took it as me having a dip at the 5100.

So yes, while it wasn't the point of the thread as I don't generally like to get into brand war discussions as they are pointless, but you're right, I didn't like it!