Vendors Face-off on Benchmark Approach

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

A major showdown between two top storage vendors could lead to a new era of competitive benchmarks, and a sea change in how the performance evaluation tool is viewed in the industry.

It all started last year when Network Appliance decided to benchmark its FAS3040 modular storage system and EMC’s CLARiiON CX3 Model 40 under the auspices of the Storage Performance Council’s benchmark program. In the past vendors have benchmarked just their own products.

The nonprofit SPC defines, standardizes, and promotes storage system benchmarks to disseminate objective, verifiable performance data. It created its industry-standard performance benchmark SPC-1 in 2001.

The unusual move to include a competitor’s product didn’t hit any news reels as the council is under no legal obligation to notify vendors of such activity or announce such council member decisions, SPC administrator Walter Baker told InternetNews.com.

[cob:Related Articles]The Council’s lone stipulation to such practices is that vendors not skirt legal stipulations that may preclude a product from being benchmarked without vendor authorization, Baker said. According to the SPC, NetApp followed the rules and requirements in conducting the testing and completing the SPC’s auditing mandates.

While EMC isn’t saying NetApp broke any laws, it clearly believes it has stepped way out of bounds in publishing its evaluation results.

EMC’s Global Marketing CTO, Chuck Hollis, describes the SPC benchmark program and NetApp’s activity as pure “bench marketing.

“Benchmarking, true objective testing, is very important to us. We spend a stupid amount of money on research and testing. What this is [the NetApp test results] is free entertainment based on cheap marketing tactics,” Hollis told InternetNews.com.

NetApp ran the two SAN products through SPC’s SPC-1 with and without snapshots technology enabled. Snapshots are used by enterprises to provide continuous protection and smart copies of data.

According to the published results NetApp beat the CLARiiON in performance. According to SPC guidelines, EMC has 60 days to challenge or protest the benchmark results. As of Friday, the SPC said it had not received any challenge notification.

The benchmark results aren’t impressing most industry analysts for several reasons. Some cited the fact that an objective party did not oversee the testing. Another expert says a one-time evaluation effort done typically holds little validity.

Charles King, principal analyst at Pund-IT, told InternetNews.com it’s pretty much a matter of “he said, she said,” since vendor-sponsored tests bring lots of “wiggle room” into play.

“In a one-time [benchmark] case it’s more of a vendor rubbing another vendor’s face in the dirt. But if this isn’t a one-time deal it could present an interesting situation,” said King.

And that’s where the showdown looms. NetApp is gung-ho to keep testing its products against EMC using the SPC benchmark program. “We will certainly do it again. This was not done in isolation,” Val Bervocici, NetApp’s chief technology architect, told InternetNews.com.

“EMC was ratcheting up erroneous performance information about our products and making claims about its own products that we knew were not correct. They called our performance into question and so we decided to put the products to the test to show customers which performs best,” said Bervocici.

“We proved that our product doesn’t take any performance hits when Snapshot is in play while it obviously impacts the EMC product performance. The competition [EMC] forced this benchmark. We just want to put the right information out there to customers,” he added.

Hollis, for his part, is also concerned about getting customers correct information. He’s worried enterprises not familiar with storage technologies may not be able to decipher between true benchmarks and what he calls a marketing gimmick.

“We’re concerned about the customers who might not know better. There’s an ethical line here and EMC will not cross it. What we’re telling them [new customers] is we’ll run a benchmark test for you that relates to your actual environment to show you what the real performance factor is with our products,” Hollis said.

King, as well as Illuminata analyst John Webster, are concerned with the SPC’s role in the benchmarking activity and what the Council’s intentions are with allowing such competitive testing.

“My issue is whether or not we can regard it as unbiased and independent. In spite of the fact that we have here the unprecedented situation where vendor X does a benchmark run on a competing vendor’s product, the SPC made no public statement with regard to the operative policy under which this benchmark was published,” Webster wrote on his blog. EMC is a client of Illuminata but the vendor says it had nothing to do with Webster’s blog content.

In response Baker says the benchmarking process was open and transparent. “They clearly are not understanding what we’re trying to do. There is no downside to being completely open and that’s what we’ve been.”

King wonders why the SPC would allow itself to be “dragged” into such a unique situation. “The SPC may have just opened the doors to a new era that could change the landscape of benchmarking as we know it,” he said.

This article was first published on InternetNews.com.