Are comparison videos biased or just misunderstood? [Update 2]

by Nick Doerr { Jan 16th 2008 at 6:00PM }

We don't normally like to get into the game of pointing fingers, but we were alerted to an interesting tip regarding the infamous "comparison videos" used by website GameTrailers. They're a good site and we respect them. They put up new video content promptly, and [insert more compliments here]. However. A lot of debate is going on currently over their comparison videos which are being touted as heavily biased against the PS3.

Straight from GameTrailer's mouth regarding the videos: "We don't create these videos for the fanboys. That's not the point. We create them so that people who own both systems can be a little more informed on which version to buy. It's the fanboys that turn the comments section for these videos into a debacle. As for using RGB or adjusting the white levels on the PS3, that completely defeats the point of these videos. We play both systems on default settings with the same cabling. If we were to adjust the settings on the PS3 to make it look better then we'd be sacrificing the integrity of the video. It's not the other way around."

So what they're saying is that they aren't taking advantage of the PS3's full-HD capabilities. It's built for that. But they aren't using it that way in their videos. Either crank both up to their maximum potential or get out, we say. Feel free to leave your thoughts as well.

[Thanks, Christian!]

[Update 1: Took out an uninformed sentence regarding Xbox video output. We apologize for the inaccuracy.]
[Update 2: Gametrailers offers another explanation here.]

Tags: game-trailers, gametrailers

Reader Comments (Page 1 of 2)

Pulse-Wave @ Jan 16th 2008 6:08PM

So they DON'T want to maximize the PS3's built-in capabilities? Odd, that.

Mike Jenkins @ Jan 17th 2008 10:56PM

Are 90% of gamers going to change their settings every time they play a different console, if they touch their settings at all? NO! Most people don't bother messing with settings they just want to plug something in, turn it on, and start playing. This is why their comparisons are best for informing gamers as to which version to buy because most gamers use the default settings anyway. Wouldn't you be pissed if you saw that the GameTrailers comparison made the PS3 version look as good as the 360 version, then when you buy the PS3 game and go to play it it looks like shit on your TV? I would! I'm glad they use the default settings because it's an honest comparison between the two systems.

Jojo Renteria @ Jan 17th 2008 4:55PM

and by NOT utilizing the full 1080p videos they ARE HURTING the ps3 version.

They dont have to adjust black/contrast/white levels but at least use the damn HDMI cable and a 1080p tv, if not then you are seriously gimping not only the PS3 versions of games but some of the 360s as well.....

Laurens Holst @ Jan 16th 2008 10:26PM

Afaik an HDMI connection is encrypted, so I’m not sure you can record from that.

MattyD @ Jan 16th 2008 6:11PM

Most definitely needs to be using full potential for a valid comparison...

Why would you want to play the game at the scaled back version otherwise.

Haiddasalami @ Jan 16th 2008 7:46PM

Its like playing a crysis on low with a high end system.

miguel @ Jan 16th 2008 6:11PM

I agree, it is completely unfair to compare a game without showing the best quality available. Isn't that the point of the comparisons to show how good the game can be?

Limanima @ Jan 16th 2008 6:15PM

I totally agree with you. They should compare both system at their optimum configurations, with the color/brightness/contrast to best fit the system.
Those comparisons are plain silly...

Elsnerma @ Jan 16th 2008 6:16PM

Agreed. Both systems should be compared with the best picture they can possibly put out. Fanboyism aside, that's the way you should compare ANYTHING really. That's like comparing two guitars straight out of the box without taking the time to tune them first.

AL7AIR @ Jan 16th 2008 6:57PM

Nice analogy and I second that :) I own both systems and had several opportunities to test multi-platform games and demos on the same screen, and with the right output settings on the console and perfectly matched input settings on the television the differences are next to nothing, especially when it comes to the "washed out" pictures shown in way to many comparison videos. Sure those settings aren't helping poor ports to a better frame rate and anti aliasing, but then again who's seriously comparing those titles to judge system power? Or better yet the fun you'll have when playing a game :)

AL7AIR @ Jan 16th 2008 7:00PM

XanthouS @ Jan 16th 2008 6:17PM

Man, that's really silly. Biased or not, they're putting out incorrect information and cutting off the PS3 at the knees.

If you're going to use the same settings/cabling, at least compare a PS3 to the Elite so they output via HDMI...

Hashbrown_Hunter @ Jan 16th 2008 7:04PM

Exactly. They should max both consoles out and compare them, or state that the comparisons take place on default settings in the video.

Fiefdom @ Jan 16th 2008 6:17PM

I agree with the position on the matter your post implies you hold. Perhaps the videos would be more informative and worthwhile if they displayed the differences between a PS3 game on default settings with RGB cables and the same game with the best picture the PS3 can produce. That would allow for a more accurate basis of comparison for any PS3 owner with HD.

Fabricating a random and exaggerated analogy, one could more easily relate to this concern by imagining the 360 as only outputting in black and white and therefore using a black and white TV in defiance of the PS3's support of color. As I stated, the analogy is grossly exaggerated and is not to be taken in a literal sense at all but the idea behind it is relevant.

PETEJOZ @ Jan 16th 2008 6:23PM

I am blocking Gametrailers so I never visit the site again.

wicko @ Jan 16th 2008 6:27PM

Definitely biased. Take for example the 360's issues they had previously with VGA, games looked washed out. So they added an option to fix that, and people use it. Its funny how they nitpick about cabling and whatnot. If you're going to go out of your way to use identical parts, why wouldn't you use identical colour settings and brightness settings? By not doing that, all that you gain is the knowledge that default settings should ALWAYS be tweaked. This is something that should be done regardless.

I find myself going out of my way to ignore the brightness difference and possible washout so I can see the real differences that count: Texturing, lighting, enabled features, shaders, etc.

joey @ Jan 16th 2008 6:27PM

I'm confused, why do they say that the 360 only outputs to 480p? Doesn't it output upto 1080p, even without HDMI?

katz @ Jan 16th 2008 7:09PM

No, 1080p is only achieved through HDMI.
they can however produce 1080i and 720p over the 5-plug component cables.
and the default may be 480(i/p) for NTSC regions, but it's 576(i/p) for PAL regions (meaning most of the world).
oh, and all of the above goes for both consoles.
the difference is that only the ps3 was designed with 1080p in mind, and the 360 who got HDMI over a year and a half after launch can't give true 1080p games, because they won't run on those with older consoles. that is if any of the old ones haven't gone to RRoD boot-camp yet.

Severe_Gamer @ Jan 16th 2008 7:53PM

That is not entirely true. Most TVs are incapable of taking a 1080p input through component even though the 360 CAN output a 1080p signal via component. For example, the Sony KDS-A3000 actually has 3 1080p component inputs.

That said, the 360 is a system built around 720p. So even though the system is outputting a 1080p signal, it is really only an upscaled 720p signal. Most 360 games read 1080p on the back, but that is just a straight-up lie since it is upscaling the game up to 1080p.

Finally, I agree with everything everyone is saying. If they are going to compare systems, they shouldn't dumb down the more powerful system until it looks like the weaker one. If nothing else, they should post their test settings and hardware with each video (much as with videocard benchmarking).

tomoguisuru @ Jan 16th 2008 6:28PM

It's true, take for example the PS3 version of the Orange Box. The issues that they showed aren't even really there. I own the game on my PS3 and have never seen any of the issues they showed. ( and yes, I'm running everything in HD including audio)

I think that they are doing everything they can to put down the PS3 version. Personally, I have nothing against the 360 but when people "rig" the comparison, that's when I have a problem.

If you want people to take your opinions seriously, don't try to fake them out.

never$$hort @ Jan 16th 2008 6:29PM

The shitty thing is that they aren't the only site using these unfair comparision methods (hint: Gamespot)... I was actually banned indefinately last year from Gamespot for complaining to them that thier comparision shots were totally unfair and that if they were to do a fair test they needed to max the Ps3 out via HDMI and calibrate the display settings... Seriously who in thier right mind could say that this fair?? Its not our fault the 360s crippled and maybe if more people saw REAL COMPARISIONS on these sites, they would realize this and ditch that ugly hunk of junk..

Joe @ Jan 16th 2008 6:29PM

480p? Huh?
All Xbox 360's output 1080p max.

Wow Nick... 'tis a shame.

ollebro @ Jan 16th 2008 6:32PM

noo, whiteout an HDMI you cant get full HD.

Joe @ Jan 16th 2008 6:34PM

All 360's have HDMI.

Joe @ Jan 16th 2008 6:50PM

All neew 360 SKUs output a 1080p max display and come with HDMI output.

And with component cables, they output at 720p/1080i

In fact, most games available for both PS3 and 360 show up as 1080p on the 360 and only 720p on the PS3

Not sure where this 480p stuff came from though.

upz @ Jan 16th 2008 6:50PM

The old 360s don't have HDMI, but you can still get an HD signal through component cables. I ran Gears at 720p through component and it still looks pretty damn nice. So yeah, Nick's a bit misinformed, but his point is still right on. If we go by GT's philosophy, we might as well output both through composite cables at 480i and call it a day.

Nick Doerr @ Jan 16th 2008 6:56PM

Heh, I don't have a 360, but I have used component cables before and it only moved me into 480p territory. I assumed it was the same with the console (someone said something about an update to the system allowing 1080p upscaling? Not the same, but close enough), but I guess not. It was probably my TV (at the time it was a crappy TV) that couldn't push past 480p.

I've since eliminated that sentence because, I'll be cursed to Hades, one slip gets focused on while the larger and more "important" issues are ignored.

Tim Parsons @ Jan 16th 2008 7:42PM

360 made b4 the ELITE came out DONT have HDMI,. HDMI is NOT needed for 1080P component supports 1080P FULLHD

Mike Jenkins @ Jan 17th 2008 10:57PM

That's completely untrue. With the 360 you can output 1080p over component, VGA, and HDMI. I don't know about the PS3, but i think you can do 1080p over component too. Whether your TV/projector/monitor accepts a 1080p input over component or VGA is a different story.

BLee @ Jan 16th 2008 6:36PM

hrm.. my 360 doesnt have hdmi..

Dan @ Jan 16th 2008 6:39PM

GT is saying because of the HDMI anti-piracy its impossible to make recordings, which makes sense then to be forced to use component cabling. Everyone is attacking GT mainly because they feel if they did the reviews using HDMI and enabling Full range RGB + Super white that the comparison would be a equal footing for the ps3, however those options require the console to be using HDMI, but you can't blame GT for the whole issue with HDMI not being able to be recorded.

I'm sure they don't mean to bash the PS3 as it seems. In the ends its more about everyone misunderstand the issue and attacking them before really knowing why they are doing the reviews as is.

Hell I love my PS3 waay more then my 360, anyone owning would know the difference between the two. Why they have to attack GT for a small issue is beyond me :-/

Joey @ Jan 16th 2008 6:42PM

No, not all 360's have HDMI, but they do come with a component cable, and they do output at a max resolution up to 1080p (with latest software updates). So if they are using an Elite (or any new 360 within the past 6 months) to test vs a PS3 - then "same cables" means HDMI. Anything older they are probably using Component on 360 but I'm not sure if there is component on PS3

Popfrogs @ Jan 16th 2008 7:04PM

Yeah, there's component on PS3 with the old PS2 or new PS3 A/V Multi cables. Still it doesn't make much sense to make videos with the RGB Full option off since most of us HDTV owners need it on and notice a big difference in contrast/brightness with it.

myself @ Jan 16th 2008 6:47PM

check as many comparison videos u want they are biased.
there games i play and look on the videos and lack the vibrant colors and quality textures as i watch.
Hell sometimes it looks as a ps2 using composite on a SDTV....

A-Troll-Named-Slickback (Yes the A Troll Named part, and yes every time!) @ Jan 16th 2008 6:49PM

I commented on this when they introduced RGB limited and full... under the name PSN:ummfada here on PS3FANBOY... that "washed-out look" is not true!

Andrew @ Jan 16th 2008 6:50PM

Wow, I know the name of the site is ps3FANBOY but do you really need to spread misinformation?

360 at 480p, what?

They're talking about using the same cables to ensure a fair test, hence not adjusting colour and brightness settings from their system defaults.

Tranando @ Jan 16th 2008 6:49PM

Only a few 360's have HDMI, all others are stuck at 720p/1080i, unless you have one of the 3 sets on the market that does 1080p over VGA.

The 360 was never designed for 1080p, hence why only a couple of games actually do it. 1080p was forced on 360, whn Microsoft discovered PS3 could do it (remember the claims from Microsoft that said it would be impossible for PS3 tro do games in 1080p...)

Either way, Gametrailers got caught out, and are pretending they are not really biased, by falling back on the "default settings" excuse. The sad fact it, the 360 settings that GT use, and the PS3 ones are crippled. All PS3 owners know this, and everything always looks better than the GT videos, and this is why.

I feel sorry for the idiots who actually believe ANY internet blog can be unbiased these days. Joystiq, Kotaku, GameTrailers, IGN, 1UP, they are all paid off by sponsors money. That's buisness unfortunatly.

Ignore the reviews, play the PSN demos, and decide for yourself. Orange box on PS3 is proof of this, the game itself plays fine, the claimed slowdowns, and the faked video are done by manipulating the QuickSave feature.

Doug @ Jan 16th 2008 7:12PM

I'm not sure how MS was "forced" to do 1080p, but does it matter why anyone adds it or not? It does it, and has done it for 21 months now. Both systems shipped without full graphics capability enabled (remember how PS3 games written for 720p [read: most] had to run at 480p on 1080i TVs until a software update?)

Comparing the same game on both platforms, in most cases the 360 renders at a slightly higher resolution (or turns on AA where the PS3 turns it off). The output resolution is nearly always separate from the rendered resolution. Most games on both platforms render at 720p (or thereabouts) and then scale to whatever resolution you have set in the menu, be that 480i, 480p, 720p, 1080i, or 1080p.

Point is, for the comparison to be fair, they either have to go with the native settings of each console, or tweak both for the best they can do. If they decide to tweak one but not the other, comparisons are obviously going to be biased. I personally think they came up with a good compromise... not limiting each console to the cabling it came with (which would be Composite for the PS3), and not tweaking either one for its best output settings.

Tim Parsons @ Jan 16th 2008 7:46PM

WRONG!! the 360 Premiums made b4 HDMI was put in still display full 1080P over COMPONENT,. I know because I rented a 360 console when Halo3 came out and I played it in 1080P

NoK610 @ Jan 16th 2008 6:51PM

I don't understand how gametrailers is being biased. They did say they are using the same cabling as you stated. Also, gametrailers is not only showing PQ, but they are comparing other things such as load times. I don't think they are being biased.

Joe @ Jan 16th 2008 6:53PM

And if you REALLY want to break it down and compare apples to apples, the 360 comes with COMPONENT/HD cables (720p/1080i) where the PS3 comes with COMPOSITE (480i) cables.

RatBoy @ Jan 16th 2008 6:53PM

I must defend GT (not because I want too but because I can see their point).

Their philosophy is fine and dandy if you assume (as it seems they do) that the purpose of the comparison videos is for the Joe Average Consumer who is just playing games straight out of the box with the minimum (read "no") effort to optimize and might not (read "probably doesn't") have the set up to run either system at their highest settings. With that audience in mind the purpose is not to compare the systems in ideal conditions, rather to compare the system-specific performance of a given title at a sort of baseline.

All that being said I would LOVE to see some comparisons at peak settings for both systems.

Murry @ Jan 16th 2008 6:53PM

Its a crock, thats for sure. I mean your going to claim that its less honest to.. i dunno. configure the machines correctly before putting them on display? when i see a comparison, I damn sure expect both systems to be setup to deliver the best picture they can. /thats/ a comparison. Not using full RGB or properly adjusting the gamma before hand defeats the purpose because at that point its not even doing justice to the GAME let alone the system or the viewers.

Jason @ Jan 16th 2008 6:56PM

I would compare their testing methods to racing two bicycles, a single gear-ratio BMX bike and a Tour de France race bike, and only allowing the Tour de France bike use first gear and then saying the BMX bike is faster. Shameful.

Joey @ Jan 16th 2008 6:57PM

Not even all games on PS3 are IN 1080p - but 720p. Uncharted and Ratchet and Clank both output at 720p, the box even describes them as such. I have a 1080p LCD but why is my PS3 limited to 720p, but my 360 upscales and outputs to 1080p on my television?

Bottom line is, rather than saying they use the same settings, they should outline what those settings are...too many variables as to what some consider 'defaults' - they should say:

Assassins Creed Comparison on a 40gb PS3 outputting via HDMI to the same television as Assassins Creed playing on an elite (or 360 with HDMI)....then we'd know what the deal is. Either way...multiplatform games, it comes down to personal choice on which console you own (even if you own both, different features vary) - Buy what you want to buy.

Tony @ Jan 16th 2008 9:34PM

Really? Probably is your tv then. For me it was like looking at an out of focus picture and a focused picture.

Doug @ Jan 16th 2008 6:58PM

Wow... a lot of misinformation here...

The PS3 has component cables available. That's how I have mine hooked up. If I were to go HDMI I would have to switch cables between my TV and projector constantly, and I don't want to do that since HDMI is kind of a fragile connector. By using component I can use a component video distribution amplifier to send video to both devices simultaneously. And I really don't mind because the difference in image quality between component and HDMI is for all intents and purposes negligible, despite what anyone tells you.

All 360s are 1080p capable, even if they don't have the HDMI port. The very first software update for the 360 added 1080p capability via component. All Elite 360s have HDMI, and all other 360s produced since about June/July 2006 also have HDMI.