Designing a Highly Reliable and Scalable Content Distribution System

KyoungSoo Park (Princeton University)

Friday, March 21th, 2008
10 am - SENSQ 5317

Refreshments at 9:30 a.m

Hosted by

Abstract

With the explosive growth of content delivery demand on the Internet, content distribution networks(CDNs) are gaining more and more importance in scalably reaching millions of users. However, most commercial CDNs are centrally controlled, making them harder to deploy in unreliable environments. Likewise, they are optimized for small average transfer sizes, making them ill-suited for increasingly popular large-file workloads such as videos and online software distribution. In this talk, I will describe a number of techniques which allow regular HTTP CDNs to be fully self-organizing and highly reliable even on unreliable platforms, such as PlanetLab. I show how unilateral monitoring, proactive failure detection/avoidance, and a fast and predictable DNS lookup system dramatically improve service availability. I will also describe how to scalably serve large files without modifying the server and client. Our design uses a unstructured mesh over a structured overlay and a probabilistic single-hop forwarding system instead of the more traditional multi-hop approach. Also, it builds on the infrastructure for small-file CDNs instead of developing a new.

Compared to the best previous research system, our system, CoBlitz, reduces the bandwidth consumption at the origin server by a factor of 7, while still improving client download speed by a factor of 3. It also beats widely-deployed systems like BitTorrent by 27-48% even in the worst-case scenario, and by 55-86% in a fair comparison. CoBlitz is deployed on PlanetLab and has been providing CDN service to the public for over three years. It currently is one of the most popular services on PlanetLab and delivers several terabytes of content every day.

Biography of Speaker

KyoungSoo Park is a research staff member in the computer science department at Princeton University. He received his Ph.D. from Princeton under Professor Vivek Pai, and his B.S. from Seoul National University in South Korea. His research interests broadly focus on distributed and networked systems, operating systems and networking. He is currently leading a number of Co* projects including CoDeeN and CoBlitz content distribution networks which serve several TBs of content on a daily basis. In 2007, he co-founded CoBlitz Inc., which provides customized CDN solutions to ISPs and Telecommunication companies.