4Gbps over your Network, without the Cost of Fibre

Original Article Date: 2009-06-01

A number of my customers in the digital content creation sector having been looking at ways of getting higher speeds across their networks without spending the vast sums necessary to invest in fibre or 10-gigabit technologies.

Well thanks to a clever little network interface card from Intel, you can aggregate commodity-priced gigabit ethernet cabling and switching to achieve 4-gigabits per second to each workstation. This is significant since it allows most standard and HD video to be streamed across the network, enabling centralised storage of video files, and no more hassle of moving files around on removable hard drives.

Furthermore, you can make potentially hundreds of terabytes of data storage available across this network using a combination of a standard file server box plus direct attached storage nodes, up to 256 drives per disk controller.

Here's how it's all laid out:

The Key Ingredient: The Intel 4-port Gigabit NIC

It's the special network interface card that's the enabler of this whole architecture. Intel produce a number of different quad-port gigabit Ethernet NICs which support link aggregation . As its name implies link aggregation (designated officially as IEEE 802.3ad) enables 2 or more gigabit ports on the controller to be aggregated or combined, to form a single 2, 3 or 4-gigabit virtual port as seen by the operating system and other devices on the network. Link aggregation is not exclusive to these Intel controllers, but is an enterprise feature that does not natively come with gigabit ports supplied onboard many workstation and server boards. So you may not be able to do this without getting one of these special controllers.

At the time of writing, Intel currently have two quad-port 1-gig controllers supporting link aggregation. These are:

E1G44ET - Intel® Gigabit ET Quad Port Server Adapter - PCI-Express*4 Full-Height and Low-Profile (latest model with full virtualization support)
PWLA8494GT - Intel® PRO/1000 GT Quad Port Server Adapter - PCI-X Full-Height (older model for legacy workstations and servers with no PCI-Express slots)

These controllers provide link aggregation at the driver level, which is then enabled using configuration software that comes bundled. Newer operating systems will allow you to aggregate separate gigabit links also, but the usefulness of doing this with only two onboard links is obviously more limited, and performance is likely to be reduced compared to a driver-level implementation.

Whilst the focus on this article is on 4Gbps, it should be noted that link aggregation can be extended across two NICs, to provide up to an 8Gbps virtual port to your workstation!

Cabling and Switching

Cabling of this solution is as easy as regular gigabit LAN, since it's using the same standard - CAT5E or CAT6 cables with RJ-45 connectors to the length required to run from your workstations to your switch. The only difference is you'll have a lot of cables to run - 4 per workstation.

The switch can be any gigabit switch that supports link aggregation (look for IEEE 802.3ad support). All major manufacturers produce models with this feature supported.

Because gigabit switches using RJ-45 connectors are the most common variety of switches available, prices stay low - you're in the mass-market. And it's this part of this high-speed network that saves you the most money when comparing to alternative solutions such as fibre-channel or 10-gig, as the switches in these latter technologies typically cost upwards of $10,000. Expect to pay no more than $1,000 for a commodity gigabit switch.

The Server

The specification for the server is whatever you want it to be. As long as a total of 2 PCI-Express*4 or PCI-X slots are available to host the two quad-port NICs required, then there is no particular stipulation on what is needed. You can even use an existing server if you want.

Two 4-gigabit NICs are specified for this example, to give you 8Gbps, which should be sufficient for delivering 4Gbps to a small workstation group. If you're running a larger set of workstations, or you anticipate that each workstation will need its own dedicated 4Gbps connection to the server, then you'll need to add more NICs. Your only limitation is the number of PCI-X or PCI-Express slots available in the server.

Onto storage issues - if you're integrating your storage directly on the server, then obviously you'll need a good number of hotswap drive bays to facilitate plenty of storage headroom.

If you're going with direct attached storage (see next section), then there is no such requirement. However, for this option, you'll need an additional PCI-Express*8 slot to house the external RAID controller.

Adding Additional Storage, Simply

Imagine these all too common scenarios - you need more network storage but either your server has run out of hard drive bays, or you're using a server chassis that simply doesn't have very many. But you need terabytes of storage to host your large media files on. Well, you can either buy a new server with plenty of drive bays or, more easily, add Direct Attached Storage (DAS) to your existing server.

Adding DAS to an existing server is as simple as adding a new external RAID controller (such as the Adaptec 5085), and then plugging it into any chassis housing up to 24 hotswap bays and a SAS (Serial Attached SCSI) backplane.

Serial Attached SCSI Direct Attached Storage is perhaps the simplest, cheapest and most exciting new storage technology to emerge in the last decade. SAS DAS has the following features and advantages:

External SAS RAID Controllers are made by all leading RAID manufacturers, including Adaptec, LSI and Intel, and retail at less than $1,000.
The external connection from the controller to the disk box uses an industry standard 4-port shielded SAS cable, which allows up to 12Gbps.
Each SAS backplane is intelligent, and multiplies the incoming ports to the number of drives connected to each backplane (up to 24 drives per chassis, typically).
Each SAS backplane has output ports that allow daisy-chaining of further storage boxes, thus a huge number of drives can be supported from a single controller (up to 256 depending upon the controller).
SAS backplanes and RAID controllers support both SAS and SATA hard drives. SAS provides better speed, a richer diagnostic and utility feature set, and higher reliability, whilst SATA offers huge storage potential at a much lower cost.
Some storage chasses have dual SAS backplanes, that allow two incoming connections, thus giving your DAS a staggering 24Gbps bandwidth.
When drives are added to an external DAS box via the external controller, they appear directly in the RAID controller configuration, allowing you to set up a RAID, and local linux or Windows volumes as if the drives were inside the server.

Summary

This example shows the power of combining two inexpensive "commodity" technologies - gigabit ethernet and SAS to deliver terabytes of flexible storage via a 4Gbps network to your small business workstation group.

The price of the above solution will vary, depending upon how much storage you need, and what specification your server would be. But the networking components alone come to less than $5,000. With fibre or 10-gigabit, you'll be lucky to get one or two special ports added to your switch, for that sum, without even allowing for the price of the required NICs.

Combining the low cost 4Gbps network with the flexibility and low cost of SAS Direct Attached Storage can provide to your studio the ease of a centralized network store that has no practical limit in size, while at the same time providing the necessary bandwidth to stream video without lag directly to each workstation.

If you'd like more information or to discuss designing a network storage solution customized to your individual situation, just let me know.

Best regards,

Ben Ranson
Chief Systems Engineer
Electronics Nexus
http://elnexus.com
ben@elnexus.com
1-877-773-5366