Hardware Infrastructure Global Viewpoint – January 2019

Understanding Leaf-Spine Network Topology

The concept of a leaf-spine IP network design is well understood for data centers. Video engineers traditionally have relied on the any-to-any concept based on video routers. Here is a brief tutorial on how leaf-spine design works.

In my last article on system design, Implementing PTP aka SMPTE ST 2110-10", we talked about building the IT infrastructure to support SMPTE 2110-XX. The term COTS, aka Commercial Off the Shelf, is used to describe the networking devices needed to build the infrastructure for SMPTE ST2110. The statement you can use COTS is vague, misleading and it does not address the common misnomers of using COTS to define the technology that involves a myriad of devices that can be acquired Commercially and Off the Shelf. In this Part 2, I will go over each of the definitions of COTS.

The Idea that any PC will do is misleading. It should be stated that any PC will do, as long as its server class, has 48 cores, 96GB of memory, a 10G network port and 2 Petabytes of storage. It is unlikely that such a system is just Off The Shelf.

The term also applies to network planning. COTS switches and routers are to be used, as long as they have a 40-100Gb/s backbone with 10Gb/s to the endpoints and are PTP enabled. Something most networks are not designed to manage. But these devices are just your everyday COTS network device - right?

Now Time to get Serious

There is a new network design philosophy that was developed to address the latency and arbitrary path routing issues in a COTS IP network. As we reviewed in my PTP – Precision Time Protocol article, see link above, broadcast is not the only industry being challenged by this technology. 

As the load demands and traffic on IP networks have increased, mission critical services are being challenged. The need to optimize network performance and introduce traffic shaping to manage the network to support these services has become a high priority. In the COTS world of networking, this introduced various control and management parameters that can be administered in the network. Some of these are:

  • Network Time Protocol (NTP) brings a layer of clock synchronization to the IP network, keeping the network and all connected devices on the same time reference within milliseconds of UTC– sort of. NTP is provided by a separate server that has a full time link to one of the time standard providers, time.gov is one. NTP is then distributed across the network to devices enabled to receive and lock to network time.
  • Shortest Path Routing – this limits the number of intermediate paths a signal takes going from a source device to a destination device.
  • Spanning Tree (STP) & Rapid Spanning Tree Protocol (RSTP) – This is to prevent loops and any packet broadcasting that results from them. Spanning tree reroutes IP traffic if a link fails. As the name suggests, spanning tree looks at the network and finds the best path between any two network connected devices.
  • Quality of Service (QoS) – This actually prioritizes packets on the network. For example, prioritizing router commands, which are small, is more important than media which is big and could overwhelm the router command and cause delay. Setting QoS helps the little guys get to their destination when traveling with the big guys.

Introducing Leaf-Spine Network Architecture

The transition to networked based media adds a new layer of complication to COTS network devices and network topology design. Allow me to introduce leaf and spine network architecture. In my previous ST2110 article we looked at PTP and how important network design is and how its changing. The ST2022, ST2110 suite, PTP and AMWA NMOS family of IS-xx recommendations have different network requirements. The new network topology is a more structured architecture with the core router, aka master switch (layer 3), and introduces boundary and transparent switches. If you recall from the PTP article and guilt by association includes NMOS, we discussed master, ordinary, boundary and transparent clocks.

That’s one aspect of the leaf-spine network architecture. The goal is to create a fast, predictable, scalable and efficient architecture to support the high performance demands placed on networks now handling massive amounts of data or media.

Leaf-spine is a two-layer network topology composed of leaf switches and spine switches. Leaf-spine's topology is useful where network traffic moves more east-west than north-south. The topology is composed of:

Leaf switches (to which servers and storage connect)

Spine switches (to which leaf switches connect).

Leaf switches mesh into the spine, forming the access layer that delivers network connection points for servers.

Basic architecture of leaf-spine networks. Click to enlarge.

Basic architecture of leaf-spine networks. Click to enlarge.

In a leaf-spine design, the network is managed by Layer 3 routing. In this architecture, all devices are the same number of network segments away from each other and contain a predictable and consistent amount of delay or latency over the network and between devices. This new topology has two layers, the leaf layer and spine layer.

The leaf layer consists of switches that connect to endpoint devices like servers, firewalls, load balancers, and edge routers. The spine layer is the backbone of the network and every leaf switch is interconnected with every spine switch.

In leaf-spine there are a few new and exciting configurations. In addition to Spanning Tree Protocol (STP) leaf-spine introduces:

  • Equal-Cost Multi-Pathing (ECMP) This allows all switch connections to be utilized at the same time. It is a method of load balancing and a routing strategy where packet movement to a single destination (end=point device) can occur over multiple best paths and avoid loops in the network.
  • Transparent Interconnection of Lots of Links (TRILL) is the IETF specification for enabling multipathing TRILL, which is undergoing IETF’s standardization process and is meant to replace the spanning tree protocol (STP). STP, which was created to prevent bridge loops, only allows one path between network switches or ports. As networks have grown and become more complex due to technologies like convergence and virtualization, STP has proved to be inefficient. TRILL unlike STP incorporates a knowledge of the entire network and uses that information to support Layer 2 multipathing. This can reduce latency and improve overall network bandwidth utilization. By getting rid of STP, network managers will be able to migrate Virtual Machines (VMs) across the network. There will also be more bandwidth available for media intensive applications.
  • Intermediate System-to-Intermediate System (IS-IS) an intermediate system refers to a router, as opposed to an End System (ES), which refers to a node .IS-IS is one of the most commonly used routing protocols. ES-IS protocols allow routers and nodes to identify each other. This is a Layer 3 routing protocol to Layer 2 devices.
  • Shortest Path Bridging (SPB) Shortest Path Bridging is the IEEE specification for enabling multipath routing in the network. This is intended to simplify the creation and configuration of networks, while enabling multipath routing. SPB is similar in concept to TRILL in that it is meant to replace the spanning tree protocol STP. SPB has some differences from TRILL. One of these is that SPB network routes are symmetric, meaning that the route from one point to another is the same going back. This allows shortest path bridging to use some of the existing network management and monitoring technologies.

What Does This all Mean?

I know it’s a lot to digest. Leaf-spine network topology is the next gen network design and is being adopted in enterprise data centers plus it’s necessary and a requirement in building the IP network for SMPTE ST2110/PTP and AMWA NMOS. Leaf-spine addresses the latency issues in three tier networks and offers new ways to optimize the network for media.  A lot of good information is available to help broadcast and production engineers better understand leaf-spine architecture and many offer guidelines on how to configure such a network.

As we continue the transition to IP, network design is a critical component.

Editor Note:Gary Olson has written a book on IP technology, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”, which is available from major book sellers.

Commenting is not available in this channel entry.

Related Editorial Content

Implementing PTP aka SMPTE ST 2110-10

There is an ongoing discussion in the industry of how to maintain consistent timecode and sync/genlock between systems and devices in facilities and in the field. As the transition to IP moves forward there is a need to maintain…

Surviving the Early Adoption to IP

Broadcasters now find themselves on one side of a digital gap. On one side we have SDI, the other side IP. Engineers want to reach the far shore, but do so without sacrificing their investment in current SDI solutions. This…

Your Guide to Understanding IP

See that hill up ahead? It’s not a hill, it’s Mt Everest and your job is to conquer that mountain. Rendered into familiar industry vernacular, you, video engineer, are charged with building an IT-centric facility. A SMPTE standard was…