Multiple streams of UDT and HpFP protocols for high-bandwidth remote storage system in long fat network

Abstract

Large-scale distributed file system is an important technology for big data sciences. Several works so far have succeeded in high-bandwidth data I/O on multi-point to multi-point remote storage systems. However, for easy use by many scientific researchers, high-bandwidth remote storage tools working on a single client computer are preferable. As for high-bandwidth multi-point to single-point remote storage system, there is a study to introduce a high-bandwidth virtual remote storage (HbVRS) tool using a distributed file system (Gfarm) and a UDP-based data transfer protocol (UDT) in long fat network (LFN). In this paper, we introduce a novel transport protocol, named high-performance and flexible protocol (HpFP), to show high throughputs for the HbVRS in LFNs with packet loss. We examine basic performances of multiple streams of the HpFP and the UDT on a laboratory experiment simulating an international LFN. In the presence of packet losses (0.5%), the HpFP shows more than 9.3 Gbps network throughput at any parallelization numbers from 1 to 12, while the maximum throughput by the UDT is limited in 1.4 Gbps with 12 parallelization.