Log in

No account? Create an account
ss - oscar

My SQL Dump

MySQL musings by a self professed MySQL Geek

Previous Entry Share Next Entry
Why is everybody so steamed about a benchmark anyway?
ss - oscar
There has been a lot of discussion in the blogosphere recently about benchmarks, and I can't help but notice that a lot of the comments and opinions are harsh, sometimes downright mean, and alas uninformed.

Why do people care so much about a benchmark. IT IS JUST NUMBERS PEOPLE.

Seriously (and this isn't directed at any one person in particular):

Who doesn't like to play "my thing is bigger than your thing" every once in awhile, but you don't have to take it so personally. If a vendor wants to waste money running a huge benchmark for little profit - WHO CARES? There must be some audience for the benchmark results, but if you aren't it, why bother screaming at the top of your lungs that the benchmark is ludicrous, or useless or ineffective. Just ignore it.  You obviously are not the intended audience.  If you are the intended audience and you still don't care, then go play another game.  You've obviously tired of the industry.

People spend way too much time arguing about unimportant things, particularly when they don't understand the underlying technical issues to the fullest extent possible.

I personally want to say:
I think ParAccell's TPC-H result is impressive. Running the TPC-H queries at 30TB is an ENGINEERING MARVEL, regardless of the amount of tech that was necessary to produce the result. Period. If your database can run 30TB TPC-H, then fine, feel free to criticize the 30TB results and BEST THEM, otherwise, give some fucking credit where credit is due. dudes. seriously.

fine print:
I work for Kickfire, who has published audited TPC-H benchmark results on the Kickfire MySQL Appliance.  
My opinions here don't represent Kickfire.  This is my blog, with my own opinions.

  • 1

well, it's pretty hard for me to find out what benchmark and what comments you are referring to, so i can't judge that really. But there is a very easy answer why people care about benchmarks.

First of all, there is the vendor's selling point. Vendors that advertise "speed" or "performance" as a benefit of their product should generally feel an internal urge to prove how much faster their product is as compared to their competitor's products. In other words, "speed" is apparently part of the vendor's value proposition, so customers that think they need speed are going to expect exactly that. Now speed and performance are complex attributes, but benchmarks are apparently the best way to measure them. So Vendors that want to sell fast products are going to end up publishing benchmarks.

Car analogies shouldn't be hard to find.

Another way of looking at it is to consider what would happen if we didn't have benchmarks. All the customers would have is marketing brochures, and all the sales persons would have is hopefully good faith from their prospects. This way of doing business is backwards - sure, commercial relationships are built on trust, but trust needs to be gained by facts. By facts I do not mean "absolute truth", but facts in the sense that science produces facts: claimed true statements that at the very least shows the potential of being objectively (independently) verified or falsified.

Your statement about "IT IS JUST NUMBERS PEOPLE" makes me chuckle - I mean, seriously, if you accept that there is no simple way to gauge performance, and that the least worst thing we have is benchmarks, then surely, the numbers is what it is all about. You can't just dismiss numbers if you don't like them, and pass them in evidence if you do. That is to say, if the numbers really wouldn't matter then you might as well not do the benchmarks either.

"why bother screaming at the top of your lungs that the benchmark is ludicrous, or useless or ineffective."

Well, that seems obvious too. I mean, if benchmarks are the accepted tools to measure performance, and you are in the business of making products that claim to have an edge in performance, then surely the value of the benchmarking methodology as a whole declines when it turns out customers can't really trust benchmarks. So if I were in that line of business, and my competitors are successful with rigged or unobjective benchmarks, then surely I won't stand for it. At least I would make an effort to point out any flaws, and possibly launch my own benchmark. Or better still, have a third party do them if it helps objectivity.

Kind regards,


In particular, the benchmark I'm talking about is the TPC-H benchmark, and in to be further specific ParAccel's 30TB TPC-H results recently posted on the TPC-H website.

Merv Adrian posted a blog entry about the results, and how he thinks it will improve ParAccel's market position. Curt Monash spectacularly disagreed and he made what I felt were somewhat disparaging remarks about Kickfire, so I chimed in on the comments.

That said, discussion went back and forth on twitter, where I tried to justify the benchmark. I pointed out that the Kickfire appliance performs significantly better on the benchmark using only 3U of equipment as Oracle running on 3 RACKS (240U) of hardware. Using 650W of power instead of 18000W! And their hardware is 10x more expensive. We kick ass at the TPC-H and are rightly proud of that fact. Compare our hardware to the competitors.

And it isn't as if we can just run TPC-H quickly. We obviously didn't set out to write a database engine that can only run TPC-H. We've set the bar higher than ever before for small form factor, green database computing, and as soon as people start recognizing that, the better.

What I'm concerned about is the bashing that is going on, basically calling us fools for trying to point out how ridiculously fast the appliance really is.

The SQL Chip is radically different than any other database technology introduced since the relational database was first implemented. The benchmark results PROVE that.

Curt says on twitter:
CurtMonash: .@wkernochan Notwithstanding Kickfire's justified plaint "Our TPC-H is so awesome if proves SOMETHING", I think TPC-Hs should just die.

I'd prefer him to really look at our results and see that they demonstrate a clear difference between our technology and what else is out there in the market. Our results our absolutely stunning given the hardware footprint difference.

Our results clearly demonstrate the advantage of our approach. What he should be doing is prodding other vendors to try to come up with some way to match our results. Power isn't cheap, and it comes at a high price to the climate. Using an order of magnitude less power than the nearest competition should alone win us accolades from the mountain tops.

But, to get back to my original point, they are, in the end JUST NUMBERS. Feel free to ignore them if you want, or think they are silly, but don't attack a company for an accomplishment like the 30TB TPC-H.


thanks for providing links - this makes it much easier for us to see what this is about.

Concerning Curt - och aye, what can I say. Whatever he does, he never fails to piss someone off - I know I have been! For some reason, those debates are rarely based on arguments - I leave any conjecture about the underlying reason up to the reader. Let's just say, since you're working for Kickfire, I guess the best thing to do is to be prepared to get pissed off by him in the future. Just don't let it get to you - it costs way too much time and energy. If you must engage in an argument with Curt, don't be tempted to rant, stick to sensible objective arguments and just keep asking him if he can please do the same.

"But, to get back to my original point, they are, in the end JUST NUMBERS. Feel free to ignore them if you want"

Really - I keep disagreeing here. What if a scientist or a mathematician would say such a thing, would you still trust their research?

Also, I've been working on my own benchmark as well, which I want to release as open source. It is vendor neutral, does not use highly complex SQL, is scalable and can accurately judge various aspects of RDBMS performance. It is kind of like a sysbench for OLAP. It has only two tunables, but it can be used to test all kinds of different things, such as partitioning, hash or bitmap indexing etc.

I can't wait to try it on all kinds of databases, and using it to help me tune them the best way possible. Thats what I call fun. I'm a database geek.

  • 1