Princeton S* Network Systems

Together with Arvind Krishnamurthy, I’ll be chairing this year’s International Workshop on Peer-to-Peer Systems (IPTPS).  The workshop was started in 2002, which coincided both with the popularization of P2P file sharing (Napster, KaZaA) and the introduction of distributed hash tables (DHTs) from several different research groups.

Eight years later, P2P file sharing is still going strong (now through BitTorrent), while the previously-academic DHTs have found their way into real use.  DHTs now form the decentralized lookup structures for file sharing services—in the form of so-called “trackerless” BitTorrent—with the DHT in the Vuze service comprising more than a million concurrent users.  (As an aside, I’m proud to note that Vuze’s DHT is based on Kademlia, which was proposed by one of my officemates in grad school, Petar Maymounkov.)

These self-organizing systems have also found their way into the datacenter.  One notable example is the storage system, Dynamo, that forms the basis for Amazon’s shopping cart and other back-end  applications.  Or Facebook’s Cassandra, used for its Inbox search.  Or the rest of the key-value stores that do automated partitioning.  And we are starting to see these techniques being proposed for scaling enterprise networks as well.  With that in mind, we wanted to broaden the scope of this year’s IPTPS to include topics relating to self-organizing and self-managing distributed systems, even those running in single administrative domains.

We also plan to have a demo session at this year’s IPTPS to highlight developed and deployed systems.  The workshop will be collocated with NSDI in San Jose, so will be especially convenient for those in the Bay Area.  We welcome submissions (both paper and demos) from researchers, developers, and hackers.  If you don’t want to write a paper, come show off your running P2P system.

Paper submissions are due Friday, December 18, 2009.  More information can be found at http://www.usenix.org/event/iptps10/cfp/.

Continue Reading

So let’s start with the first of seven lessons from CoralCDN’s deployment:

• How all published distributed hash table (DHT) algorithms are susceptible to race conditions and routing errors for non-transitive network connectivity, and what can be done to mitigate these problems.

Some challenges with deploying DHTs

CoralCDN’s primary goal was to enable websites to survive spikes in traffic.  We can see examples of such so-called flash crowds through CoralCDN: The figure on the left shows a spike to Coralized slashdot.org URLs that occurred in mid-2005.  Requests grew from nothing to … Continue Reading

In this post, I describe the architecture and mechanisms of CoralCDN at a high-level. This is meant to provide some of the background necessary for some of our experiences and lessons with operating the system.

System Overview

CoralCDN is composed of three main parts: (1) a network of cooperative HTTP proxies that handle users’ requests, (2) a network of DNS nameservers for .nyud.net that map clients to nearby CoralCDN HTTP proxies, and (3) the underlying Coral indexing infrastructure and clustering machinery on which the first two applications are built.  You’ll find that I refer to the entire system as “CoralCDN”, but the … Continue Reading

We’ve recently been working hard on Firecoral – a browser-based, peer-to-peer content distribution network for web caching. I’ll be presenting a short talk on Firecoral at the 8th International Workshop on Peer-to-Peer Systems (IPTPS) on April 21st in Boston, MA.

Peer-to-peer content distribution has been inarguably successful for large file distribution (e.g. BitTorrent), but P2P services have been restricted to stand-alone applications, not transparently incorporated into Web browsing and seamlessly running over HTTP. CoralCDN has served as a web content distribution network for the past five years, but its deployment has been limited to PlanetLab and demand quickly … Continue Reading