Researchers Built Their Own ISP to Fix the Internet– A Decade Later, It’s Still Running
When a major service provider like Amazon or Cloudflare goes offline because of a routing error, as has happened in recent weeks, network engineers have to fix a problem they can’t see. Somewhere in the Internet’s routing system, traffic is being sent to the wrong place—maybe halfway across the world. But studying these failures used to be impossible. You couldn’t experiment with Internet routing without being an Internet service provider (ISP) yourself.
The solution? Build your own testbed ISP.
In 2014, Nick Feamster and his colleagues published “PEERING: An AS for Us“, describing a testbed that let researchers experiment with the Internet’s routing infrastructure for the first time. That work recently received the Test of Time Award from the ACM SIGCOMM Internet Measurement Conference. (The term “AS” in the paper’s title refers to “Autonomous System”, which is a technical term for an independently operated network like an ISP).
The project started under GENI (Global Environment for Network Innovations), an NSF-funded program with an ambitious goal: design a future Internet architecture. GENI built infrastructure to let researchers experiment with networking at scale. The system that eventually became PEERING was called Transit Portal, infrastructure that Feamster and his Ph.D. student Valas Valancius developed at Georgia Tech to give researchers access to interdomain routing.
“The GENI and FIND programs had a vision of letting researchers actually experiment with Internet infrastructure and architectures, not just simulate it,” Feamster said. “I had this idea that it should be possible to experiment with Internet routing on the real Internet. Transit Portal was our way of making that a reality. Actually getting it working—building the control plane, the safety mechanisms, the whole system—was a huge design and implementation lift. Valas really got it off the ground.”
Transit Portal leveraged Internet Exchange Points (IXPs), the points in the Internet where networks connect to exchange traffic directly. By connecting to other service providers directly at IXPs, the testbed could peer with hundreds of networks and announce routes just like a real ISP. But the Transit Portal also needed safeguards. Servers sat between researchers and the live network, filtering routes and enforcing policies so a bad experiment couldn’t break anything.
“The challenge wasn’t just technical,” Feamster said. “We had to convince operators at major IXPs like AMS-IX in Amsterdam to let us plug academic research infrastructure into their production network. That required trust, and we had to earn it.”
After the GENI program ended, Transit Portal might have disappeared like so many research prototypes do when funding ends and students graduate. Instead, Professor Ethan Katz-Bassett and his team at USC, and later at Columbia, took over leadership of the project and continue to maintain it to this day. They expanded it to more IXPs, maintained the infrastructure, developed new capabilities, and continue to actively support the research community’s use of the testbed.
“PEERING has flourished because of Ethan and his team,” Feamster said. “They’ve kept it running for more than a decade, expanded it, and supported the research community. That sustained effort—deploying at IXPs, maintaining the system, making it possible for so many researchers to actually use it—that’s why the work has stood the test of time. They deserve enormous credit. This work has stood the test of time and remains one of the main lasting, impactful legacies of the GENI program, in large part due to Ethan’s ongoing dedication to the project, as well as many other students and researchers who have continued to extend and support the work.”
The testbed enabled research that wasn’t possible before: systems for repairing routing failures, prototypes for software-defined Internet exchanges, and tools for diagnosing path changes. It’s still being used today. Transit Portal, and ultimately PEERING, demonstrated that the Internet’s core could be studied experimentally, not just observed from the outside. In some sense, this is the lasting legacy of the work, as well—that Internet researchers can improve the Internet by building systems that interact with the live network, going beyond the simulation-based experiments that were the de facto mode of operation prior to Transit Portal and GENI.
“The impact goes beyond just the testbed itself,” Feamster said. “We demonstrated that it was possible to do empirical research on the Internet’s routing infrastructure. Prior to Transit Portal and PEERING, the predominant mode of understanding Internet routing was through simulation or theory, both of which had significant shortcomings when translated to practice. We showed that it was possible to conduct experimental research on critical infrastructure if you build the right guardrails. That principle applies to everything I work on to this day, whether it is network security and privacy, performance monitoring and inference, or machine learning models for networks. Ultimately, lasting impact in this field requires building and measuring deployed systems.”
Feamster continues this style of research at UChicago’s Network Operations and Internet Security Lab, focusing on AI- and data-driven security, privacy, and auditing in networked systems.
