Measuring Linux and Open Source

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

How do you measure success?

This question is often asked in the Linux business and development communities, with various answers given: “the number of servers in the enterprise,” “the number of desktop deployments,” “or the number of new bugs found in Vista today.”

Measuring success is a matter of definition, and therefore it can be a hard thing to pin down. And, it seems, the challenges of measurement go a lot deeper than something as subjective as “success.”

For something as objective as measuring, say, processor I/O, one would expect the answer to be very clear-cut and well defined. You have processor A, you have a certain job to perform in X amount of time, and you have a stopwatch (metaphorically speaking). Go. The process of measuring what seems to be such a straightforward process, however, is no easy task.

As Linux continues to move into the enterprise data center, its power and speed are often touted as being stronger and faster compared to other operating systems. But how much faster? And what numbers are used to determine its performance? These are questions that many inside the Linux community are trying to answer, as they explore the various ways and means of measuring computer performance.

One decades-old organization has already explored the paths of computer measurement, and is hoping to lend Linux the benefits of the trails it has already blazed.

The Computer Measurement Group (CMG) got its start 33 years ago, when mainframe engineers at IBM began the arduous task of defining once and for all how performance on IBM’s big iron could be measured and compared with other mainframes. Today, the not-for-profit organization has broadened its scope to include the measurement and management of all computer systems, not just mainframes.

The processes that CMG has helped to define and measure are no longer limited to strictly hardware numbers. According to Dr. Michael Salsburg, Chair of CMG’s Marketing Committee, the organization has turned to the measurement and management of IT services.

“We’re defining IT services in terms of business processes,” Salsburg explained. Such processes could include how long does it take to sell a book on a bookstore Web site, or adjudicate a claim on an insurance company’s online portal? This process orientation allows the CMG to set benchmarks that can be used across industry verticals.

The CMG, given its mainframe origins, is not well known in other technology circles. This is something Salsburg and the CMG wants to change, particularly in the open source community.

Right now, Salsburg maintains, “management tools in Linux are pretty basic, compared to Windows and the mainframes.” What’s troublesome to Salsburg is that if there is any progress made in open source management or measurement tools, a lot of it is redundant work.

“People are rediscovering what was being discovered 20 years ago,” according to Salsburg, who also is the chief architect for Unisys Systems and Technologies.

Because of this, the CMG is beginning an active campaign to reach out the open source community and assist them in developing measurement and management techniques that have served the CMG well in the past. The CMG has recently opened the contents of its Proceedings to non-CMG members, and sometime this summer plans to launch a new Web magazine called “Measure IT” that will allow the general public to tap into what Salsburg describes as CMG’s “wealth of knowledge.”

CMG is also making broader announcements regarding its 33rd annual conference, taking place from December 2 -7 in San Diego, CA at the Manchester Grand Hyatt.

The information that the CMG plans to disseminate is free of charge and free of restrictions. Indeed, the CMG has long been a believer in open, patent-less techniques and algorithms, well before the term “open source” was coined in 1998. Now, developers in all technology communities will have access to the organization’s testing tools.

Salsburg is hoping that the contribution CMG will be making to the open source community will in essence become the performance community for open source. With much of the performance tests and management tools already figured out by the CMG, such a collaboration could be another way to measure success for Linux.

This article was first published on LinuxPlanet.com.