Systems designers are always pressing for faster and faster real-time performance, and data acquisition technology continues to evolve to meet that need. Currently available high-performance digitizers perform in the range of 2 to 5 GS/sec, and some new instruments are featuring 7 GS/sec digitizers for transient capture.
Often, advances in one area of technology put pressure on others, and in this case these faster digitizers have created a corresponding need for faster data recording to capture the digitized results in real-time. Streaming data recorders are currently capable of data rates in the range of 720 MB/sec, which is quite fast but still means a single recorder is incapable of recording all the data from one of these advanced digitizers.
With some architectural insight and utilization of advances in serial FPDP interfaces, recording engines, switching, and storage arrays, however, an ultra-wideband streaming data recorder can be created to keep up with these advanced digitizers.
Faster Serial FPDP
The first step in creating an ultra-wideband recorder is to get data off the digitizer, calling for a fast, scalable interface. Serial Front Panel Data Port (FPDP, ANSI/VITA 17.1) has become a de-facto standard in sensor signal processing applications, for two reasons: it is fast, and it simplifies cabling and allows for longer cable lengths than parallel connections.
Serial FPDP builds on Fibre Channel links to remove the limitation of driving a parallel cable called for in the original parallel FPDP interface specification. A pair of optical fibers carries serial FPDP data from a few kilometers up to tens of kilometers of distance depending on exact configuration, and the high quality optical connection also has the benefit of improving bandwidth.
First generation serial FPDP products ran at 1.062 GB/sec, but link speeds continue to improve. Using a current 2.5 Gb/sec link, one of the faster serial FPDP implementations claims support for sustained rates of 247MB/sec using standard 8b/10b encoding with only 1% overhead for headers. As Fibre Channel speeds increase to 10 Gb/sec, serial FPDP transfer bandwidths will continue to increase.
But today, even with these speeds our high speed 2 to 5 GS/sec digitizer of interest still outputs data faster than a single serial FPDP interface can handle. Solving this problem requires the first architectural insight: splitting a single data stream from the digitizer into several serial FPDP channels. Digitizers have been designed using FPGA IP that stripes data from the high-speed A/D converter out to numerous RocketIO ports using serial FPDP formats. In a 2 GS/sec digitizer, 12 of these links can carry the load using just 167MB/s per link. Faster digitizers could scale and implement more links.
With the data digitized and striped out to a set of serial FPDP ports, the front-end challenge is solved and the data moves to multiple recording engines for capture.
More Recording Horsepower
Recording engines generally grab data from a source and tag and format it for a mass storage device. Since digitized data is readily placed on serial FPDP links, recording engines based on quad serial FPDP inputs are called for.
While serial FPDP has advantages, it’s not the only interface out there. With that in mind, the latest recording engines have been designed to interface to a wider range of data sources by basing their inputs on PMCs, such that the I/O capability can be changed to meet the mission and scaled as technology advances. Using the latest generation quad serial FPDP PMC such as the VMETRO SFM, a single recording engine can handle four of the striped data channels from the digitizer. Bumping the interface from PMC to XMC increases the available I/O bandwidth into the recorder engine even farther.
Once the data is onboard, fast onboard processing using a Power Architecture processor augmented by an FPGA performs the tagging and formatting functions to prepare data to move into the storage area network (SAN).
Today’s fourth generation recording engine technology, such as the VMETRO M6000, is using XMCs with x8 PCI Express interfaces coupled with the high performance onboard processing, and has moved the bar for streaming real-time performance from around 90 MB/sec a few years ago to a current figure of 720 MB/sec. These recorders also feature GigE interfaces to facilitate management tasks.
High speed switching
SAN technology has also advanced with the use of 4Gbps Fibre Channel. While Fibre Channel offers real advantages in terms of the possible SAN topologies, without a good switch implementation many of those topologies are hard to physically realize.
A new generation of SAN switch platforms, such as the QLogic SANbox series, can help in constructing a scalable storage area network. The SANbox offers features such as auto-detecting Fibre Channel ports, high-speed 10 Gbps links for expanding units in a stack, configuration and zoning wizards, and a non-blocking switch architecture for high performance.
By implementing a fast fabric switch, the SAN can connect a set of data recording engines to a set of storage arrays, creating a very large, scalable solution. This also allows for very high reliability recording with redundancy built into the SAN, and virtually limitless storage.
Streaming Storage Arrays
Several elements have gone into making faster streaming storage arrays, but two are particularly noteworthy: FibreChannel disk interfaces, and the use of RAID technology.
Fibre Channel (FC) hard disks are enterprise-class drives with 1.2 million hour MTBFs. A single disk drive delivers 80 MB/sec of performance all the way to the inner tracks. They again take advantage of the lightweight, long distance cabling FC offers, and using the FC arbitrated loop, over 100 drives can be attached to a single FC port. Disks are dual ported, and both ports can be used simultaneously. This means real-time recording tasks can use one set of ports, and analysis functions can be carried out with a workstation operating on the second ports.
In addition, RAID configurations are possible. Besides the dual porting and multiple drive per port capability mentioned, queues of disk groups can be established to boost performance and allow uninterrupted recording and analysis on a larger scale. Many recorders can share a single disk group, or a single recorder can be routed to multiple disk groups. The configuration possibilities are only limited by physical space and cost considerations, but FC can scale from the smallest 73GB drive to the largest of configurations measured in petabytes.
Putting It All Together
With these four items in place, an ultra-wideband streaming recorder can be constructed as shown in the opening illustration. Clearly seen are architectural elements of the fast digitizer with its outputs striped to serial FPDP ports, multiple high speed recorder engines connected to an array of storage, a Fibre Channel switch connecting all the data to an array of analysis workstations, and a single SBC that provides timing and control of the entire system.
This particular configuration was set up to capture data from a 2 GS/sec digitizer. Note the data is split into 12 serial FPDP channels, which are then routed to three recorder engines using the quad serial FPDP PMCs mentioned. This system configuration is improved over one just a couple years old, which required six recorder engines to do the same job.
Since every situation of this scale is different, the exact implementation and performance of the recording environment will vary. The main points of this architecture and its ability to scale to meet the need are very clear:
- Digitizers continue to improve in speed,
- Striping the digitizer output sets the stage for scalability,
- Recording engines can be ganged,
- Fibre Channel simplifies cabling and improves speed and function systemwide,
- SAN topologies help scale storage capacity and function.
Stretching Isn’t Difficult
This recording architecture continues to grow in capability as the technology pieces improve. Faster digitizers, better FPGAs, higher speed Fiber Channel links, faster disk drives, better processing on recording engines, and improved switching will continue to improve speeds possible with this type of architecture. Constructing a system to record high-speed data sources isn’t nearly as difficult as it might seem at first glance.