Sourceforge's Mirrored Content Gets a Hand

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

With over 180,000 open source projects in its repository, SourceForge.net has a lot of files it needs to track and serve.

It can become a challenge to manage all those downloads. For the most part, Sourceforge relies on a system of globally distributed mirror sites that help to deliver open source application files and programs to users.

Now, the mirror system itself is gaining a new mirror in the US thanks to a donation from network vendor CDNetworks. The mirror will help Sourceforge manage its open source delivery efforts, and the growth of those efforts.

The new mirror brings the total number of Sourceforge mirrors globally to 18 and comes at a time when content providers are increasingly looking to content delivery networks like CDNetworks and its chief competitor Akamai to reliably distribute and deliver content.

Beyond that, CDNetworks is staying coy on costs involved.

“The value is purely expressed in Gigabytes used because we don’t have an exact knowledge of what the volume will be,” Robert J. Gribnau, vice president of sales, CDNetworks US and Europe, told InternetNews.com.

“If you use the market rate times the volume that will provide a value. It’s material enough that I can’t share the usage numbers.”

Gribnau noted that current market rates hover between 20 and 25 cents per gigabyte. SourceForge VP Jay Seirmarco told InternetNews.com that on a typical day the mirror network serves 3 million downloads, though that’s a number that can spike upwards. Seirmarco noted that on Wednesday of this week the SourceForge mirror network served 3.9 million downloads totaling 30.1 Terabytes (define) of data.

SourceForge has two types of mirror sites in its delivery network, partial and full. The CDNetworks deployment is a full mirror and represents approximately 8 TB of open source data.

Seirmarco explained that SourceForge directs traffic to its mirror network by way of a redirector. The redirector does an IP lookup and figures out where someone is located and then autoselects a mirror based on the address. A user can also select a mirror manually if you have a preference.

Though CDNetworks has a network of 65 nodes around the world, the new SourceForge mirror will be co-located at one of CDNetworks US nodes. Gribnau noted that SourceForge will get dedicated infrastructure and will not be shared across the broader global network. He added that CDNetworks already has one SourceForge mirror operational in Korea, so the US mirror represents and expansion of the donation.

As to why CDNetworks is donating the network capacity, it’s simply a matter of giving back to open source and investing in open source. Gribnau commented that CDNetworks benefits from open source and uses it in its own infrastructure.

One of the key open source tools used by both CDNetworks and SourceForge is the rsync application that helps to synchronize multiple nodes in a distributed computing environment.

“In some countries it is non trivial to have a mirror with many TeraBytes of storage and just to sync that is a challenge,” Seirmarco said.

He added that SourceForge must synchronize multiple-terabytes of data across the mirror with no errors and its something they do reliably day after day.

This article was first published on InternetNews.com.