A testbed Framework for Large Scale EmpiricalEvaluation of Real-Time DASH in Wireless Scenarios

We kindly ask that should you mention DASHbed, or use our code, in your publication, that you would reference the following paper:

D. Raca, Y. Sani, C.J. Sreenan and J.J. Quinlan, "DASHbed: a testbed Framework for Large Scale Empirical Evaluation of Real-Time DASH in Wireless Scenarios". In Proceedings of the ACM Multimedia Systems Conference (MMSys 2019), Amherst, MA, USA. June 18-21 2019. [CORA] [ACM]

Darijo Raca, Dr. Yusuf Sani and Prof. Cormac Sreenan presenting DASHbed at MMSys 2019 in Amherst, MA, USA.

Darijo, yusuf and Cormac presenting DASHbed at MMSys 2019


Abstract: Recent years have witnessed an explosion of multimedia traffic carried over the Internet. Video-on-demand and live streaming services are the most dominant services. To ensure growth, many streaming providers have invested considerable time and effort to keep pace with ever-increasing users demand for better quality and stall abolition. HTTP adaptive streaming (HAS) are at the core of every major streaming provider. Recent years have seen sustained development in HAS algorithms. However, to compare their proposed solution, researchers need to create a framework and numerous state-of-the-art algorithms. Often, these frameworks lack flexibility and scalability, covering only a limited set of scenarios. To fill this gap, in this work we propose DASHbed, a highly customizable real-time framework for testing HAS algorithms in a wireless environment. Due to its low memory requirement, DASHbed offers a means of running large-scale experiments with a hundred competing players.

Finally, we supplement the proposed framework with a dataset consisting of results for five HAS algorithms tested in various evaluated scenarios. The dataset showcases the abilities of DASHbed and presents the adaptation metrics per segment in the generated content (such as switches, buffer-level, P.1203 values, delivery rate, stall duration, etc.), which can be used as a baseline when researchers compare it to the output of their proposed algorithm. 

DASHbed: a testbed Framework for Large Scale Empirical Evaluation of Real-Time DASH in Wireless Scenarios

We provide instructions for setting testbed. We use Vagrant for managing virtual machines.

Link: Instructions Download

Note: Alternatively, you can set up DASHbed using physical machines. You follow the instructions in the link above, and download the server and client code below:

Link: DASHbed (source code)  Download

Link: Modified dashc client video player (source code)  Download

DATASET: Complementary dataset generated with DASHbed 

Link: DASHbed Dataset Download


As an example, we randomly extract one bw trace and show the following scenarios:

  • one algortihm per bw trace, 300 seconds, one run, two segments durations (2,4 seconds)
  • five competing clients with different streaming algortihms, 300 seconds, one run, two segment durations (2,4 seconds)
  • four competing clients with same streaming algortihm, 300 seconds, one run, 8-second segment duration

2-, 4- and 8-second logs can be viewed/downloaded from the following link: Download

Figure 1 shows the P.1203 QoE for all five algortihms in a single case scenario (4 seconds segment duration)

dashbed fig1

Figure 2 and 3 show  P.1203 QoE for all five algortihms in a competing case scenario for 2 and 4 seconds segment durations, respectively

dashbed fig 2

dashbed fig 3

Figure 4 shows P.1203 QoE for four competing clients with same algortihm (logistic) (8 seconds segment duration)

dashbed fig 4

This work has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/IA/1892.


Mobile and Internet Systems Laboratory

Department of Computer Science, Western Gateway Building, University College Cork, Western Road, Cork, Ireland.