Internet, a Network of Networks

Internet is a network of networks. It’s mind-boggling how data packets containing requests and responses travel over the internet. How a content request moves from one’s Wi-Fi router (or cell tower) > Tier 3 provider ¹> Tier 2 provider ¹> Tier 1 provider ¹> (maybe) internet exchange > content provider (including CDNs) and reverse, traversing hundreds and thousands of miles across continents through undersea and on land fiber optics cables to reach a data center (or someone’s home) server to serve the requested content. This symphony is done in conjugation with multiple content providers, network transit providers, data centers and internet exchanges. Internet is truly a network transcending national boundaries.

Exchanging traffic is the underlying fabric which makes the internet tick. No one provider can reach every single device connected to the internet through its own network. Network switch needs to happen somewhere from requester’s network to provider’s network. That’s why it’s decentralized.

Smaller (Tier 2 and Tier 3) network pay bigger, carrier grade (Tier 1) networks for transport or transit as it is called, of their network traffic to reach far off networks on the internet. Carrier grade networks have optical fiber networks for selling and company usage. These are often called internet backbones as they carry the bulk of international internet traffic.

On the other hand, peering is a settlement free arrangement with no money exchanging hands between networks which have equal benefits from the arrangement, like Tier 1 networks can exchange traffic with other Tier 1 networks in either direction or between ISPs which need content for their user and content providers which want their content to reach end user. Peering generally happens in Internet exchanges (or internet exchange point). More peering, through internet exchange help ISPs to take data directly from content providers leading to less reliance on transit providers leading to significant cost saving, reduced latency and at times improved redundancy. Internet exchanges provide a net positive impact to the internet connectivity in the region.

I’m writing this while listening to a YouTube music video. The video probably is received over a private peering arrangement between Google and my internet service provider, Reliance Jio; considering the traffic Reliance Jio produce and Google’s low 1 Gbps traffic requirement for private peering. This blog probably would also be served in the same manner as GitLab pages (which hosts it), makes use of Google Cloud. If it was hosted on say, Hetzner Online’s server, it would reach someone through one or more of the listed networks interconnect arrangements.

India earlier used international bandwidth and transit providers to access the internet through European or Singapore internet exchanges and data centers. Higher demands and low latency requirements brought more internet exchanges to India like DE-CIX, Extreme-IX, Kolkata IX and the oldest of the lot, National Internet Exchange of India or NIXI, which in turn has localized quite a bit of internet. NIXI had this weird rules which disallowed content providers (like Google, Netflix) and CDNs (like Akamai, Cloudflare etc), which provide bulk of the internet traffic as well as charging for bandwidth in x minus y model, under which ISPs paid for downloaded data minus uploaded data, as opposed to just renting bandwidth. This was something unheard of internet exchange space, which only charge flat port fees. Thankfully, both these rules were done away with in 2019. Private players (DE-CIX, Extreme-IX) have largely taken the lead in this space. Even touching 1 Tbps traffic ^{2 3} on some days in comparison to NIXI’s max of less than 200 Gbps ⁴, largely due to the fact content providers have shied away from peering at NIXI.

Netflix is a different case which needs special mention. In addition to peering via internet exchanges, they also provide Open Connect Appliances boxes to ISPs with 5+ Gbps of Netflix traffic. These are sorts of local caches (usually with 100 TB storage), filled with popular local content, that’s directly embedded in ISPs network. This way Netflix content isn’t served over the usual internet but directly from this box inside ISPs network, significantly reducing bandwidth loads, cost and latency.

Peering, internet exchanges, submarine cables makes the network that powers the internet. Why should only data centers and servers take all the credit ;)