The Quest for Bandwidth Estimation Techniques for large-scale Distributed Systems

In recent years the research community has developed many techniques to estimate the end-to-end available bandwidth of an Internet path. This important metric can be potentially exploited to optimize the performance of several distributed systems and, even, to improve the effectiveness of the congestion control mechanism of TCP. Thus, it has been suggested that some existing estimation techniques could be used for this purpose. However, existing tools were not designed for large-scale deployments and were mostly validated in controlled settings, considering only one measurement running at a time. In this paper, we argue that current tools, while offering good estimates when used alone, might not work in large-scale systems where several estimations severely interfere with each other. We analyze the properties of the measurement paradigms employed today and discuss their functioning, study their overhead and analyze their interference. Our testbed results show that current techniques are insufficient as they are. Finally, we will discuss and propose some principles that should be taken into account for including available bandwidth measurements in large-scale distributed systems.