The emergence of distributed, extreme-scale science applications is generating significant challenges for data transfer. Ideally, high-performance data transfer should reach terabit/s throughput to make full use of the underlying networks and provide real-time and deadline-bound data transfer.
Although significant improvements have been made in the area of bulk data transfer, currently available data transfer tools and services cannot successfully meet these challenges, for the following reasons. Existing data transfer tools and services lack a data-transfer-centric approach to seamlessly and effectively integrate and coordinate the various entities in an end-to-end data transfer loop. Existing data transfer tools and services lack effective mechanisms to minimize cross-interference between data transfers. These tools and services also are oblivious to user requirements.
BigData Express is a schedulable, predictable data transfer service that provides a data-transfer-centric architecture to seamlessly integrate and effectively coordinate the various resources in an end-to-end data transfer loop. A time-constraint-based scheduler can schedule data transfer tasks and an admission control mechanism provides guaranteed resources for admitted data transfer tasks. A rate control mechanism improves data transfer schedulability and reduces cross-interference between data transfers.