Transport Network Guide

Internet Infrastructure: Gateways, Exchange Points, CDNs, and Deep Packet Inspection

12 min read

Internet Infrastructure: Gateways, Exchange Points, CDNs, and Deep Packet Inspection

A Tier-1 service provider’s outward-facing edge is far more than a single uplink to “the internet.” It comprises a layered system of border routers, peering fabrics, embedded caches, and inspection engines that collectively determine how traffic enters, exits, and traverses the provider’s network [16]. This chapter covers the four pillars of that system: the Internet Gateway, Internet Exchange Points, content delivery networks, and deep packet inspection [16].

Content

The Internet Gateway

The conceptual “PE to Internet” link in simplified SP diagrams is, in practice, an entire subnetwork variously called the Internet Gateway (IGW), Border Network, or Peering Edge. It houses multiple border routers, each specialised for a distinct class of external connectivity:

  • Transit edge — connects to paid upstream providers.
  • Peering edge — terminates settlement-free private peering sessions.
  • IXP edge — faces internet exchange fabrics.
  • CDN edge — interfaces with embedded content-provider caches.
  • DDoS/security edge — scrubbing and filtering layers that sit in front of production traffic.

External Connection Categories

Service providers maintain four broad categories of external connectivity:

CategoryRelationshipCost ModelScale
TransitSP pays upstream carrier$0.50–2.00/Mbps (95th pct)Full internet reachability
Private PeeringBilateral, settlement-freeZero (symmetric traffic)100G–400G dedicated fibre
IXP PeeringShared fabric, many peersPort fee to IXP operatorHundreds of networks per port
CDN/Content EdgeSP hosts provider cachesFree hardware from content provider95%+ of video served locally

Major global transit providers:

ProviderASNNotes
LumenAS 3356Largest global transit provider
CogentAS 174Price-competitive, dense peering
ArelionAS 1299Formerly Telia Carrier
GTTAS 3257Enterprise-focused
TataAS 6453Strong Asia/Middle East presence

IGW Router Requirements

Border routers at the IGW carry the full global BGP routing table (approximately 1.05 million IPv4 and ~220,000 IPv6 prefixes (May 2026; see https://bgp.potaroo.net)) [11], which demands high-memory platforms (16 GB or more). These routers enforce complex inbound and outbound policy through RPKI validation [2, 5], prefix lists, and BGP communities [1, 10]. A typical Tier-1 SP deploys four to eight border routers per major POP, connected via 100G or 400G links, with at least 2N or N+1 redundancy [1, 16]. The IGW is geographically distributed across multiple POPs to provide resilience and traffic-engineering flexibility [16]. Operators committed to the routing-security baseline implement the recommendations of BCP 194 [1] and the MANRS actions [10] — prefix filtering, anti-spoofing, coordination, and global validation — with origin validation on EBGP sessions consuming RPKI ROA state [3, 5] delivered from a relying-party cache via the RTR protocol [4]. Path validation, where deployed, uses BGPsec [6].

Internet Exchange Points

Before IXPs existed, traffic between two ISPs in the same country could traverse an ocean and back, adding over 100 ms of latency and incurring unnecessary transit fees [16]. An Internet Exchange Point (IXP) solves this by providing a shared Layer 2 switching fabric in a neutral datacentre where many networks interconnect directly [16].

Physical Architecture

The IXP operator maintains a large Ethernet switching fabric. Each participating network runs a physical cable from its router to a switch port on the fabric. The entire fabric forms a single broadcast domain, typically addressed from a single /24 IPv4 subnet and a /64 IPv6 subnet. Every participant receives an IP address from the peering LAN and establishes eBGP sessions with other participants over that shared medium.

graph TD
    ISPA["ISP A"]
    ISPB["ISP B"]
    ISPC["ISP C"]
    GOOG["Google AS15169"]
    IXP["IXP Switching Fabric<br/>192.0.2.0/24"]
    RS["Route Server"]

    ISPA --- IXP
    ISPB --- IXP
    ISPC --- IXP
    GOOG --- IXP
    IXP --- RS

Four networks connect to a central IXP switching fabric. The Route Server acts as a central point for multilateral route distribution among all connected participants.

A single 100G port at a major IXP like DE-CIX Frankfurt grants potential access to over 1,000 connected networks.

Bilateral vs. Multilateral Peering

Two models exist for establishing sessions at an IXP:

  • Bilateral peering requires a separate BGP session with each desired peer. This gives maximum control over routing policy but scales linearly: n peers means n sessions to configure and maintain.
  • Multilateral peering uses a route server operated by the IXP. A single BGP session to the route server yields routes from every other network that also peers with it, turning one session into connectivity with hundreds of networks.

Notable IXPs

IXPPeak throughputConnected networks
DE-CIX Frankfurt~18 Tbps peak (late 2025; see https://www.de-cix.net for current statistics) [12]1,000+
AMS-IX Amsterdam~10 Tbps [13]900+
LINX London~7 Tbps950+
Equinix IXDistributed30+ cities globally

Regional IXPs serve a critical role in keeping domestic traffic local. The Cairo Internet Exchange (CAIX) ensures that traffic between Egyptian operators like Orange Egypt and Vodafone Egypt stays in-country rather than routing through Marseille or Amsterdam — avoiding 100 ms of unnecessary latency over a 10 km distance. In Saudi Arabia, STC, Mobily, and Zain peer at the Saudi Arabia Internet Exchange for domestic content, with regional traffic flowing through Equinix IX Dubai or UAE-IX.

On-Net Traffic

When two customers of the same SP communicate, traffic remains entirely within the provider’s network. Every PE router learns via iBGP which customer prefixes are reachable at which other PEs. A packet from Customer A (connected at PE2) destined for Customer B (connected at PE4) is forwarded directly across the MPLS core without ever touching an IXP, transit provider, or external peer.

The larger the SP’s footprint, the greater the proportion of traffic that stays on-net. Large Tier-1 providers typically see 30—50% of their total traffic remain internal.

Hot-potato vs. cold-potato routing. When traffic must cross multiple autonomous systems, the sending AS can hand it off at the nearest exit point (hot-potato routing, which minimises the sender’s cost) or carry it across its own backbone to an exit point closer to the destination (cold-potato routing, which improves latency for the receiver). Most SPs default to hot-potato; large content providers sometimes negotiate cold-potato arrangements for better end-user experience.

International Points of Presence

A Point of Presence (POP) is a physical location — typically a cage or suite in a carrier-neutral datacentre — where the SP deploys routing and switching equipment. Tier-1 SPs distribute POPs globally for several reasons: reaching major peering venues (peering at DE-CIX requires physical equipment in Frankfurt), serving multinational customers, terminating subsea cables, and optimising latency.

Key carrier-hotel facilities where SPs concentrate equipment:

  • Equinix LD8 (London) — UK and European hub
  • Interxion MRS1 (Marseille) — Mediterranean subsea cable landing
  • Equinix SG1 (Singapore) — Asia-Pacific gateway
  • 60 Hudson Street (New York) — US East Coast peering
  • Equinix LA1 (Los Angeles) — Asia-to-US gateway

A Tier-1 backbone typically interconnects 30—100 POPs via 400G long-haul fibre. Subsea cable systems such as SEA-ME-WE 6 (Singapore to Middle East to Western Europe), 2Africa (45,000 km circumnavigating Africa), MAREA (Virginia Beach to Bilbao), Dunant, and Grace Hopper form the intercontinental links. Cable cuts from anchors, earthquakes, or conflict (such as the 2024 Red Sea incidents) cause dramatic regional latency spikes.

Content Delivery Networks and SP Integration

A CDN caches popular content as close to end users as possible, eliminating the need for every request to traverse long-haul paths back to origin servers. CDNs and SPs have a symbiotic relationship: the CDN gains proximity to subscribers, while the SP offloads expensive transit traffic.

Embedded Cache Programmes

Major content providers ship free hardware appliances to qualifying SPs:

  • Netflix Open Connect Appliances (OCAs): 1U/2U servers that pre-fill overnight (typically 02:00—06:00 local time) with content predicted to be popular the following day. During peak hours, over 95% of Netflix streams are served from these local caches.
  • Google Global Cache (GGC): serves YouTube, Google Play downloads, and software updates.
  • Meta FNA (Facebook Network Appliance): handles Facebook, Instagram, and WhatsApp video traffic.

Commercial CDNs such as Akamai, Cloudflare, and Fastly generally operate their own globally distributed edge POPs rather than embedding inside SP networks. Cloudflare alone operates in 330+ cities (2026), placing 95% of internet users within 50 ms of a cache node.

BGP Integration of Embedded Caches

Embedded caches integrate into the SP’s routing fabric through a standard pattern:

  1. The cache appliance receives a private IP address within the SP’s infrastructure.
  2. It peers via eBGP with nearby PE routers using a private ASN.
  3. The cache announces the content provider’s public IP prefixes (for example, Netflix’s address blocks).
  4. The SP’s routing policy prefers these locally-originated routes through higher LOCAL_PREF or better metrics.
  5. The content provider’s intelligent DNS steers end-user requests to the local cache by resolving hostnames (such as netflix.com) to the cache’s address.

Modern Traffic Composition

Video accounts for 40—60% of all SP traffic. Over 70% of total traffic is now served from CDN caches, either on-net or at nearby IXPs. Transit has shifted from the primary forwarding path to a fallback used mainly for long-tail and uncached content.

Deep Packet Inspection

Deep Packet Inspection (DPI) examines packet payloads — not just Layer 3/4 headers — to identify, classify, or manipulate traffic based on its content or application-layer characteristics.

SP Use Cases for DPI

  • Traffic classification and QoS: identifying application types (gaming, video, peer-to-peer) to apply differentiated forwarding treatment. A common example is deprioritising BitTorrent during peak hours.
  • Lawful interception (LI): regulatory mandates require SPs to deliver targeted traffic to law enforcement agencies. The governing standards are ETSI TS 102 232 in Europe [14] and CALEA in the United States; in 5G mobile networks LI architecture and functions are specified by 3GPP TS 33.127 [15].
  • Content filtering: regulatory requirements such as UK age verification or filtering regimes in the Middle East and Asia.
  • Billing and zero-rating: identifying specific application traffic (for example, “WhatsApp free” plans) to exclude it from data caps.
  • Subscriber-aware policy enforcement: in mobile networks, binding packets to individual subscriber identities and enforcing per-plan rules in real time.
  • DDoS detection: recognising abnormal traffic patterns and diverting suspect flows to scrubbing centres.

DPI Vendor Landscape

  • Sandvine — historically the dominant SP DPI vendor; controversial due to documented deployment in government censorship systems.
  • Cisco SCE — Cisco’s legacy DPI platform.
  • Huawei SIG — widely deployed across Asian and emerging-market SPs.
  • Allot, Procera — smaller specialist vendors.
  • Nokia Deepfield — analytics-oriented, augmenting NetFlow/IPFIX telemetry with DPI-derived classification.

Ethical and Political Dimensions

DPI is a dual-use technology. The same engine that classifies gaming traffic for QoS can identify VPN tunnels for blocking or political content for censorship. Sandvine equipment has been documented in use by governments in Belarus, Egypt, and Turkey to throttle VPNs and suppress access to opposition websites.

The Encryption Shift

stateDiagram-v2
    [*] --> HTTP: Pre-2015
    HTTP --> HTTPS: TLS adoption
    HTTPS --> TLS13: TLS 1.3 (2018)
    TLS13 --> ECH: Encrypted Client Hello
    ECH --> QUIC: HTTP/3 / QUIC

    state HTTP {
        note right of HTTP: Full payload visible to DPI
    }
    state QUIC {
        note right of QUIC: All metadata encrypted<br/>DPI must use ML/behavioral
    }

With over 95% of web traffic now encrypted via HTTPS and TLS 1.3 [7], traditional content-level DPI is effectively obsolete. Encrypted Client Hello (ECH) replaces ESNI; see the IETF TLS WG for current RFC status (draft-ietf-tls-esni progressed through 2024–2026). ECH bootstrap parameters are carried in the DNS via SVCB/HTTPS resource records [9], and ECH hides even the target hostname from middleboxes [7]. QUIC (HTTP/3) further encapsulates transport metadata inside the encrypted envelope, removing yet another inspection surface [8].

DPI and Ethics

DPI is a dual-use technology. The same engine that classifies gaming traffic for QoS can identify VPN tunnels for blocking or political content for censorship. Sandvine equipment has been documented in government use in Belarus, Egypt, and Turkey.

The industry has responded with behavioural and ML-based classification: analysing flow-level metadata such as packet sizes, inter-arrival times, and session durations, then applying machine learning models to infer the application with high accuracy despite full encryption.

End-to-End Walkthrough: Video Delivery in Practice

To illustrate how these components work together, consider a home user in Riyadh streaming YouTube:

flowchart TD
    User["User Device<br/>Riyadh"] -->|"HTTPS"| Access["SP Access Network"]
    Access --> PE["PE Router"]
    PE --> DPI["DPI Engine<br/>Classify: Video → QoS mark"]
    DPI --> Decision{{"Cache hit?"}}
    Decision -->|"Yes (95%+)"| GGC["Google Global Cache<br/>(on-net, same DC)"]
    Decision -->|"No"| Escalate["Escalate to IGW"]
    Escalate --> IXPPath{{"Best path?"}}
    IXPPath -->|"Option 1"| IXP["IXP peering<br/>Equinix Dubai"]
    IXPPath -->|"Option 2"| Direct["Direct Google POP"]
    IXPPath -->|"Option 3"| Transit["Transit provider<br/>(last resort)"]
    GGC --> Stream["Video stream<br/>to user"]
    IXP --> Backfill["Retrieve + backfill GGC"]
    Direct --> Backfill
    Transit --> Backfill
    Backfill --> Stream

    style GGC fill:#dfd,stroke:#333
    style Transit fill:#fdd,stroke:#333

Key Insight

This flow demonstrates the deeply collaborative nature of modern internet infrastructure: SPs, IXPs, CDNs, and content providers have built an interlocking system of peering, caching, and traffic engineering that makes sub-second global content delivery appear effortless.

This flow demonstrates the deeply collaborative nature of modern internet infrastructure: SPs, IXPs, CDNs, and content providers have built an interlocking system of peering, caching, and traffic engineering that makes sub-second global content delivery appear effortless.

See Also

  • Tier-1 Service Provider Architecture and L3VPN (SP-Architecture chapter 01) — core SP topology, PE/P router roles, and MPLS core that underlies the IGW’s internal forwarding
  • Service Provider Services: DIA, L2VPN, VPLS, and EVPN (SP-Architecture chapter 02) — external-facing services (DIA, VPLS, EVPN) that ride the same border infrastructure described here
  • BGP Fundamentals (BGP book) — BGP protocol mechanics referenced throughout this chapter (eBGP sessions, LOCAL_PREF, communities, RPKI)

References

  1. RFC 7454 / BCP 194 — BGP Operations and Security. J. Durand, I. Pepelnjak, G. Doering, IETF, February 2015. https://www.rfc-editor.org/rfc/rfc7454
  2. RFC 6480 — An Infrastructure to Support Secure Internet Routing (RPKI architecture). M. Lepinski, S. Kent, IETF, February 2012. https://www.rfc-editor.org/rfc/rfc6480
  3. RFC 6482 — A Profile for Route Origin Authorizations (ROAs). M. Lepinski et al., IETF, February 2012. https://www.rfc-editor.org/rfc/rfc6482
  4. RFC 8210 — The Resource Public Key Infrastructure (RPKI) to Router Protocol, Version 1. R. Bush, R. Austein, IETF, September 2017. https://www.rfc-editor.org/rfc/rfc8210
  5. RFC 6811 — BGP Prefix Origin Validation. P. Mohapatra et al., IETF, January 2013. https://www.rfc-editor.org/rfc/rfc6811
  6. RFC 8205 — BGPsec Protocol Specification. M. Lepinski (Ed.), K. Sriram (Ed.), IETF, September 2017. https://www.rfc-editor.org/rfc/rfc8205
  7. RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3. E. Rescorla, IETF, August 2018. https://www.rfc-editor.org/rfc/rfc8446
  8. RFC 9000 — QUIC: A UDP-Based Multiplexed and Secure Transport. J. Iyengar, M. Thomson (Eds.), IETF, May 2021. https://www.rfc-editor.org/rfc/rfc9000
  9. RFC 9460 — Service Binding and Parameter Specification via the DNS (SVCB and HTTPS Resource Records). B. Schwartz, M. Bishop, E. Nygren, IETF, November 2023. https://www.rfc-editor.org/rfc/rfc9460
  10. MANRS — Mutually Agreed Norms for Routing Security. https://www.manrs.org/
  11. CIDR Report — Active BGP entries (FIB). https://www.cidr-report.org/as2.0/
  12. DE-CIX — Traffic statistics. https://www.de-cix.net/en/locations/frankfurt/statistics
  13. AMS-IX — Traffic statistics. https://www.ams-ix.net/ams/documentation/total-stats
  14. ETSI TS 102 232 series — Lawful Interception (LI); Handover Interface and Service-Specific Details.
  15. 3GPP TS 33.127 — Lawful Interception (LI) architecture and functions (5G).
  16. L. Peterson, B. Davie, Computer Networks: A Systems Approach, 6th ed., Morgan Kaufmann, 2021.

Graph View

Backlinks