A Round-Up of External PC Interconnects
Original Article Date: 2007-11-27
In this issue, I thought I'd do a round-up of the ways in which two computers, or
a computer and a device, can talk to each other. Before you think I'm going to get
into a treatise on networking, think again. Networking is a very large subject in
itself - with its complexity driven mainly through the software concepts and applications
that accompany
it.
Rather, this article focuses on what different hardware architectures are out
there currently which allow you to send data from one machine to another, and how
quickly that data can go.
Why Connect?
Do you remember when all you needed was a solitary PC? At that time it perhaps never
occurred to you that it might be useful to connect it to other computers or devices
(except, maybe, a printer via that special "parallel port"). Well this changed with the explosive growth of the internet,
and the embedding into most peoples' minds that interconnecting computers was something
desirable.
But when you break down exactly why it is desirable to have computers connected,
it essentially comes down to one basic - the accessing of data that is located away
from the PC. When broken down a little more into purpose, it can be seen that there
are three basic types of interconnect:
- PC to PC
- PC to External Device
- PC to External Storage Device (obviously a specialization of item
[2], but it is
so important and widespread that it warrants its own category).
These three purposes have evolved respectively into the following technologies
today:
- Ethernet
- USB, Firewire
- SAS, Fibre-Channel , eSATA
There is, of course, some cross-over between these purposes and
technologies. For
instance, Firewire and USB are commonly used to link a PC to an external storage device, and fibre-channel is often used to connect compute nodes in blade servers
and clusters. But by and large,
and perhaps because of marketability of products as much as anything, the different
technologies have settled into their specific purpose markets.
In all three cases, the need to have our PC talking to something outside itself
is today clearer than it's ever been. E-mail, internet and home and office networking
are now seemingly essential parts of our lives. Printers, modems, PDAs,
even flashlights can now be connected to our PC, and the list of devices grows each
year. And lastly, the concern of data backup and storage flexibility have caused
an explosion in growth of external hard drives and increased modularization of enterprise
storage.
But how did these needs evolve into the current sets of technologies we have today?
The answer lies in the old "functions monitors structure" philosophy. If you know
the function (or requirement) of something, you can determine what structure, or
design, would logically follow. In the case of computer connectivity, we are concerned primarily
with the following functional needs:
- Highest Speed
- Lowest
Cost
- Longest
Range
- Ease of Connectivity
In developing and marketing connectivity solutions, vendors have had to make continual
trade-offs between these four requirements. Thankfully, over the years, newer technologies
have emerged that have raised the bar on all performance related functionality,
whilst being sold for an overall lower cost.
In assessing the current choices on connectivity, let's take a look at each of the
technology groups in turn.
PC to PC: Ethernet
During the 1990s, Ethernet emerged as a the clear winner amongst a number of competing
network technologies being developed and marketed up to that
time. Ethernet is something
we're all familiar with, and is the standard on virtually all of the world's wired
(as opposed to wireless) networks today.
It has evolved from its original speed of 10Mbps (10 mega bits per second) over coaxial cable and BNC ("bayonet")
connectors through 100Mbps over twisted-pair and
RJ45 connectors, to a 1Gbps standard
which is now becoming standard amongst new equipment.
A summary of the current standards of ethernet is as follows:
10BASE-T - "Ethernet"
100BASE-T - "Fast Ethernet" (from 1995)
1000BASE-T - "Gigabit Ethernet" (from 2000)
BASE-T is the current standard form factor using RJ45 connectors over twisted-pair Copper cabling (Category 5, 5e or 6) and has approximately 100m range.
100BASE-T is currently the most common form of networking out there, although 1000BASE-T
is now the standard speed of onboard networking ports on all server and workstation
boards. Gigabit switches are replacing 100Mbps switches in most corporate environments
at this time. So it is expected that gigabit LAN will be become
the standard for the next 5-7 years.
10 Gigabit Ethernet
Whilst most of us are still on 100Mbps and are slowly transitioning to gigabit ,
there is yet an even faster standard out there - 10Gbps or "10 Gigabit".
This is still only used
for specialist networks, such as blade servers and compute clusters, where high-bandwidth
communications are necessary to share RAM and CPU loading between the nodes.
10 Gigabit Ethernet is currently realised by the following form factors:
10GBASE-CX4 - up to 15m range - lowest cost per 10Gb port - typically
$900 per adapter. Application: intra-rack
usage (e.g. clusters, blades)
10GBASE-SR - up to 300m range (using 850nm multi-mode fibre) - typically $3,000 per adapter. Application: intra-building
usage (high speed LAN)
10GBASE-LR - up to 10km range (using 1310nm single-mode fibre) - typically $5,000 per adapter. Application: inter-building and office use (high
speed WAN)
10GBASE-T is still to be realised and has yet to be standardised
by the IEEE, but if 10Gb is possible on standard Cat 5,5e,6 cabling, it is likely
to become the next standard for all wired networking in 5-7 years. Watch this
space.
PC to Device: USB and Firewire
Once the domain of the old RS232 Serial and Parallel Printer cables, the task of
allowing PCs to talk to devices has been almost entirely replaced by two standards:
Universal Serial Bus ("USB") and IEEE1394 ("Firewire").
The market segment is largely dominated now by USB, since that technology was an
open standard developed by a forum of different manufacturers, and was cheaper to
incorporate into devices and onboard chipsets, since it did not incur often prohibitive
royalty fees payable to Apple, who unilaterally developed and marketed Firewire.
That said, Firewire is a superior technology, and provides faster real-world throughput than USB (in comparing 400Mbps IEEE1394a with 480Mbps USB2.0), making it more suitable
for high-speed devices such as external hard drives. This performance advantage
is further enhanced with Firewire 800 (IEEE1394b).
Both standards currently have a 5m range on standard copper cabling.
The following summarizes the two competing standards:
Firewire
IEEE1394a - 400Mbps
IEEE1394b - 800Mbps
IEEE1394c - 3.2Gbps (standard published in June 2007)
USB
USB
1.0 - 12Mbps
USB
2.0 - 480Mbps
USB
3.0 - 4.8Gbps (in development, products expected 2009/2010)
For regular devices such as printers, modems, PDAs, etc., USB2.0 will continue to
dominate. Despite its lower performance, it will also continue
to outstrip Firewire
in the external hard drive market, due to its lower cost and ubiquitous nature (all
PCs have USB ports included, most don't have Firewire).
PC to Storage: SAS, Fibre-Channel and eSATA
Whilst Ethernet has become the standard for LANs, the storage interconnect market,
which requires a potentially much higher bandwidth, still has a number of competing
technologies. Perhaps the main reason for this is due to the lack of maturity in
the market. Back in the early 90s, Ethernet was still fighting it out amongst other
standards for PC connectivity. But today, external storage is still evolving. The
growth of media usage (images, audio,
and particularly video) amongst internet users has led to an explosion of storage
needs amongst data providers in the enterprise space, as well as home users needing
bigger drives outside their PC for storage and backup.
For the Desktop -
eSATA
The need for external storage on the desktop is largely being met by USB and Firewire
drives, although a case can be made for the new eSATA standard
for such drives. eSATA simply stands for External SATA, and is an extension of this
protocol to allow a SATA drive in an external enclosure to connect directly to a
desktop PC.
With a 3Gbps bandwidth, compared with 800Mbps for Firewire(1394b)
and
480Mbps for USB2.0, faster single drives that can deliver 80MB/s burst data rates
are not bottlenecked by the interconnect. Additionally, as eSATA is a native SATA
interface (the only difference from SATA to eSATA is really a voltage issue, to
boost signal strength down the cable), real-world performance on single drives
is much better, as the signal does not have to be converted from SATA to USB or
Firewire protocols.
There is a 2m range limit on cabling for native eSATA architecture, or 1m range for constructs using a passive PCI I/O
bracket which connects to a standard SATA port on the mainboard.
Fibre Channel
The need for high speed external storage in enterprise environments led to the development
of the Fibre Channel protocol in the late 1990s. With the technology was born the
concept of a SAN (Storage Area Network), allowing separation of
compute boxes and storage boxes and to improve flexibility and modularity of the different
components in an enterprise computing and data storage system (such as Microsoft
Hotmail's server and storage farm). With no competition, Fibre Channel soon became
the standard for server-to-disk connectivity.
Granted, there is the potential for using a traditional LAN to separate out compute
and storage boxes (see my previous article on Network Attached Storage),
but Fibre Channel could utilise data rates faster than gigabit LAN, and required
only low-latency Fibre switching between the host bus adapter and the disks.
Starting at 1Gbps line rates, Fibre Channel's speed has increased to real-world
HBAs supporting 4Gbps over dual ports (i.e. 8Gbps per HBA). Standards
are in place for this line rate to increase to 10Gbps.
Fibre Channel has great flexibility. It can be set up as point-to-point (e.g. HBA
direct to each disk), as Arbitrated Loop (ring type daisy-chain), or Switched (like
in a traditional LAN). An additional plus-point of Fibre Channel is its long
range. Depending on the type of fibre used, the range of Fibre Channel
can vary from 70m to 50km.
The downside to this feature-set however, is price. A dual-port 4Gbps Fibre Channel
HBA costs a minimum of $1,600. Compare this price point to SAS HBAs, as I will discuss
next, and then you might see why Fibre Channel's reign over enterprise storage is threatened.
SAS
SAS stands for Serial Attached SCSI and is a revolutionary revamp of that venerable
technology. You can read more on this new standard in my
previous article on SAS, but essentially it is a point-to-point enterprise
disk transmission protocol, allowing a SAS HBA to connect directly to SAS or SATA
drives within the same box or, using external cabling and a SAS "expander backplane",
to link multiple external storage boxes to the compute box via the SAS HBA.
An external SAS HBA will typically cost around $800, and may feature two ports,
each carrying four 3Gbps SAS channels. That's a theoretical 24Gbps for just
$800! Compare this with the Fibre Channel HBA delivering a total of
8Gbps for $1,600, and you can see why SAS stands to seriously challenge Fibre Channel
in the enterprise storage market!
The only downside of SAS over Fibre Channel is range. With an 8m cable limit
(using SFF 8088 "Mini-SAS" or SFF 8470 "Infiniband" copper cabling), SAS
is intended for use in connecting storage to compute boxes within the same rack
or server room, whereas Fibre Channel can connect to different buildings or different
sites. That said, the vast majority of users will be content with this limitation,
as most just need to hook up boxes within the same rack.
Summary
I hope this overview of PC interconnects has been of some use. There does appear
to be a bewildering array of competing methods out there when it comes to connecting
a server or PC to something else. But once you break down the currently available
technologies into four functional needs (price, speed, range and ease-of-use), you
can make a decision on purchasing equipment that will be most suited to your needs.
On the desktop, USB will, in my opinion, continue to dominate PC
to device and external drive connectivity, despite the technological superiority
of eSATA and Firewire.
In looking at enterprise solutions, my personal view is that I would see no reason
to recommend Fibre Channel over SAS (for CPU to Disk applications),
or 10 Gigabit Ethernet CX4 (for CPU to CPU applications). This
is simply due to price considerations. Fibre Channel is pricey, although it does
have the advantage of longer range, and I have yet to evaluate whether there is
a significant difference in latency between these competing interconnects. But if,
like the majority of users, you're concerned with connecting boxes within the rack,
there is no need to go to fibre, as you can use the low cost, high bandwidth "copper"
equivalents.
Best regards,
Ben Ranson
Chief Systems Engineer