You may also be interested in:

Full-scale streaming has a clear premise – we already know what scale to support on the video streaming delivery network. So far, most broadcasters have worked on consumption-based CDN models because streaming usage, and the peaks and troughs of capacity usage, have either been low (i.e., too low to commit to capacity models), too high (i.e., infrequent high peaks of viewers) and/or unpredictable. But at full-scale streaming we know what capacity usage to expect every day at a national level, and it will be fairly consistent and predictable.

An Edge server can be quite accurately sized from a capacity perspective. While the mix of traffic being served makes a difference to the maximum streaming egress performance that can be managed, we can at least have a minimum level of throughput for one Edge server on which to base a system’s committed delivery capacity. Recently published speed tests by Varnish (early 2023) and Broadpeak (late 2022) pointed to 2RU servers streaming at 1.1 Tbps and 750 Gbps respectively. Each server had different CPUs. These tests chose traffic that would allow the server to be pushed to the limit. In the real world, cache hit rates, live vs. VOD, VOD variety, origin connectivity, video bitrates requested, streaming protocols, etc. all make a difference. A conservative base level of throughput we can use for capacity planning is 50 Gbps per 1RU of Edge server.

What about redundancy? Many broadcast systems have redundancy built in. At full-scale streaming, we need to be clear what happens in the event of delivery capacity failures. Building out the overall delivery capacity for concentrated regional-level viewing probably gives sufficient capacity redundancy at a national level. Even if the platform has enough streaming capacity for an entire national population, most of the time this level won’t be reached at the same time so the system naturally contains server-level redundancy. If PoPs fail, redirecting traffic to other PoPs to serve the extra traffic will need to be designed in. If an entire content delivery network goes down, clear disaster-recovery options must be available. In streaming, a worst-case scenario should be that the bits and bytes must be delivered farther away from the viewer, which risks latency and buffering but most viewers could receive their content. Even if significant video delivery capacity is unavailable, or if the risk of delivery issues is high, then reducing the bitrate per viewer to allow the video to get through is a useful strategy. And if network failures occur that are outside the purview of the video streaming delivery network, then there can even be options to switch from fixed-line broadband to mobile or vice versa.

Location of Edges is also critical to full-scale streaming. Choosing the right number of Edge locations is a discussion point between content providers and individual ISPs. ISPs know their network topology, their congestion points, their customer bases, and their customer demand for data including video. When ISPs work with wholesale or national-level access network providers (e.g., in the UK, most ISPs work with either BT Wholesale or BT Openreach) then the ISPs co-locate their broadband infrastructure inside buildings owned or operated by the access network provider. For full-scale streaming, that real-estate map is the focus of the targeted Edge deployments. Picking the locations which provide the best mix of performance, cost and overall system efficiency is the right starting-point.

A National-level Architecture

The UK is the example in this article. Other telephony/broadband environments around the world look very similar.  In mobile-first economies, like India, the infrastructure is different. But in any country where telegram and then telephony services evolved over a century ago and are moving from copper to fiber networks, the UK example below is a good reference point.

As noted previously, the Edge scales with the audience. So where is the audience in the UK? Wherever people live, with major population centers as shown in Fig 1.

The UK’s broadband network is distributed around the country primarily via the same real estate locations as the original telephone system. British Telecom’s Openreach is the primary access network operator in the country and operates the telephone exchange buildings and the infrastructure that connects exchanges with homes and businesses. BT Openreach has 5,600 exchanges in the UK. About 2000 of them serve 65% of the population, or about 43 million people. It’s clear from the map of the UK that those 2000 are in the purple, red, orange, yellow, and some of the green areas. The final one-third of the population is served by the remaining 3,600 exchanges. The Greater London metropolitan area alone has 176 telephone exchanges, given the density of the population (c. 9.5 million people). The Greater Manchester and Liverpool areas have 44 and 40 Exchange buildings respectively, that serve a combined 4 million people.

In the list of 2000 exchanges, some of the exchanges are serving much smaller areas like Daventry in the Midlands that has about 30,000 inhabitants and 10 exchanges. The average number of people served per exchange in this list of 2000 exchanges is 22,500 people. The maximum average number of people served per urban area is 62,955 and the minimum average is 1,054.

But part of the transformation of streaming video delivery relates to the point that BT Openreach is currently working towards reducing the number of exchanges from 5,600 to 1,000. Why? Because FTTP (fiber-to-the-premises) is, by design, a more centralized system. This is because fiber transports light, while copper is electrical. Distances of network connections with fiber can be much, much longer and still retain excellent performance. Not only should fiber networks have better performance, but due to the longer network links they can have a reduced real estate footprint, which also reduces the amount of network intervention and maintenance overheads. This separate article with input from BT Openreach from the beginning of their decade-long FTTP transformation program covers this subject in much more detail.

This more centralized access network is built for data, agnostic to what that data is for. Video is just one of the uses of this network, although we know from Cisco’s various State of the Internet reports that about 80% of all internet traffic is video. A CDN is built to move content more quickly from its source to the users. It is the fast lane of the internet highway. But CDNs often stop either before the ISP network begins, or they are embedded in anywhere from 2 to 50/60 ISP locations in a country like the UK. Netflix is reported to have 100 locations in the UK, but they are the giant of the streamer CDN group.

A Potential Video Edge Blueprint In The UK

Video delivery is at a key inflection point as streaming for broadcasters moves from 10-20% of total content delivery towards 50% of total content delivery in the next 5-6 years. 100% streaming delivery is the end-game, but it’s not yet clear when that time will come.

Sustainability of streaming delivery is a key requirement for any future deployment. Dedicating a network to video delivery and then using it for other video processing tasks or putting it into “power-down” mode at off-peak times could be the right balance between most-efficient video performance to each viewer as we move from c. 20-30 Tbps of special peak bandwidth consumption towards 200-300 Tbps of regular peak bandwidth consumption. This 10x increase assumes 10 Mbps average bitrate which is fine for our current “2D flat-screen” viewing method. But if immersive viewing formats become popular then the video delivery network capacity will need to be increased. Dedicating the platform to video also creates an opportunity for a unified and aggregated content delivery environment, reducing both the number of servers required and the amount of content movement required.

Let’s imagine that we deploy the Edge capacity much more deeply in the UK’s broadband networks. The Edge would need to be a specialized consumer video delivery service, and therefore probably outside the realms of a product offered by a regulated access network operator (e.g., BT Openreach) or wholesale broadband network operators (e.g., BT Wholesale). But the deployment itself should be in PoPs that are highly distributed. The 1000 BT Openreach exchanges that form the basis of the UK’s FTTP network could be the final deployment location of the Edge servers. They could support content delivery for all the major broadcasters and streamers, and they could be set up to serve the same content to consumers using different broadband service providers.

This ability to serve content from a single Edge to consumers on multiple ISP networks is a “reversed origination” concept, and it could be the most efficient model for full-scale streaming video delivery. To explain, CDNs today often deliver content for multiple content providers from Origins that interface to many internet service providers, often through an intermediate cache layer or Origin Shield layer. If we use this same method and apply it to distributed Edge servers in the 1000 locations, we could have an aggregated video delivery platform that interfaces upstream to multiple ISPs and downstream to singular access network operators. In the UK, imagine an Edge server with BT, Sky, TalkTalk, and many other ISPs upstream, with BT Openreach – the access network operator - downstream.

If we connect a single unified Edge to multiple ISP networks at this entry-point into the access networks, single pieces of content like a live stream or a VOD asset can be brought to the Edge by a request from a consumer of one ISP, but then reused by consumers on other ISP networks. In principle, rather than each ISP building out their own deep Edge network to reach the necessary scale for their customer base (and then perhaps shrinking it if their customer base falls), the capacity can be scaled and managed for both the ISPs and the Content Providers by a Video Edge Network provider.

This design is technically feasible.  To implement it will require focused collaboration between ISPs and some type of Video Edge Network service provider that would aggregate this part of the internet’s traffic on behalf of the main video streamers in the market.

If we follow the scenario that we build Edge streaming capacity for best possible regional-level viewing, which would future-proof the capacity, then in the UK we would build about 660 Tbps of capacity that could be shared across all content providers and all ISPs. In the USA we would need 3,300 Tbps, in Germany 830 Tbps, in Argentina 450 Tbps, and in Finland 55 Tbps.

660 Tbps is enough capacity to deliver 10 Mbps streams per viewer for 66 million concurrent viewers. In a unicast-centric delivery model 660 Tbps means 13,000 1RU servers which, if deployed across 1000 BT Openreach PoPs, means 13 servers per PoP. Add a 1RU network switch and a 1RU firewall server, and we have 15RU per PoP, or about one-third of a standard rack. For a platform that could stream at 10Mbps to everyone in the country, or at 30Mbps to the prime-time audience of about one-third of the population, that’s not a big and scary footprint.

And if we have servers that stream at 100 Gbps then we reduce the server count to 6,000 servers. Or if the network topology means that it is more efficient to deploy Video Edge servers in 200 Exchanges instead of 1000 Exchanges, then we reduce the number of physical locations that need to be managed by 500%. Additionally, if server use can be balanced out in a way that extends its working life from a typical 5 years to 7-8 years then we can reduce the carbon footprint associated with server manufacturing and deployment.

There are various ways to design the deployment to optimize system-wide cost and efficiency, but the basic requirement is that it can deliver consistently good quality video at consistently low latency to daily prime-time audiences and occasional super-large audiences, and it can do this while being highly efficient and cost-effective to operate and maintain. Keeping the system simple by reusing internet protocols and implementing standard servers that are de-tuned, energy-optimized, and deployed close to the consumers can achieve these objectives.

So, how much capacity and how many servers (roughly) would need to be deployed in some of the UK’s locations like Greater London, Greater Manchester & Liverpool, and, as an example of a smaller urban area, Daventry? Note that this same approach could be applied to other population centers around the world with broadband infrastructure.

A first assumption to make is how many Exchange buildings are likely to be present in each location (an “Exchange building” is an original telephony building, housing telecommunications infrastructure). Without an exact and confirmed BT Openreach map, we can simply apply a proportional reduction in Exchange count based on the planned total reduction from 5600 exchanges to 1000 exchanges (i.e., 18% of current buildings will remain). 

This approach seems to make good sense in terms of total infrastructure deployed for video streaming at full-scale. But there are three big hurdles to overcome.

First is how to deploy the servers this far inside the ISP’s current networks. Each ISP has its own policies for on-net server deployment. To find the most efficient video delivery model needs a partnership approach between Streamers and ISPs which is based on a shared vision of full-scale streaming delivery that achieves broadcast-grade standards for excellent reach, reliability and QoE to consumers, and significantly reduces the load on the current ISP core network infrastructure. At this moment in time, things like the Fair Share debate in the EU are taking a different approach to this situation, looking for extra investment in network capacity to support the already large, but growing, streaming traffic levels. But while that moves money from one party to another, it doesn’t describe how that delivers an ultra-efficient and sustainable network for full-scale video streaming.

The second point is how to manage the CDN service to drive it towards this ultra-efficient model. Leaving it open to standard market forces may not yield the target results quickly. How much will we overbuild capacity and duplicate effort in a race to provide this service to the fast-growing Streamers? A pre-requisite to achieving efficiency is for the major streamers to work together on their delivery capacity requirements – given that streamers share audiences at different times of the day/week/month, then how can they reuse the same capacity between them like they do today in DTT broadcast networks? It is worth considering that the aggregation of streamer traffic towards the ISPs would optimize the capacity deployments because the primary driver of capacity will be total audience size, which is determined independently of an individual streamer or individual ISP or any CDN working in the market. So, as broadband market shares ebb and flow, and as total audience size adjusts from hour to hour, and as new viewing formats enter into the market, and as the traditional broadcast infrastructure is largely replaced by streaming video infrastructure, an industry-level unified Edge platform can be right-sized and right-featured to fit exactly what is required for the market. And while this may sound like a heavily regulated, monopolistic and even nationalistic approach, which could be reasonably argued to go against the normally efficient principles of free market competition, the point is that the final solution should be focused on sustainability and efficiency, however that is best achieved.

The third point is how to ensure the platform is secure. Redundancy is one element of providing secure infrastructure, which was highlighted in the previous article. We can probably obtain sufficient redundancy within the overall system by deploying capacity that enables reaching each person in the country, which would enable delivery to continue even if significant elements of the platform became unavailable. But at full-scale streaming, the network will need a world-class ability to withstand cyber-attacks. The video delivery network will need to be treated like a piece of critical national infrastructure, just like existing telco networks and broadcast networks. The Video Edge Network Provider must focus heavily on this specific aspect of platform management on behalf of all its media customers and their governmental, societal, and commercial stakeholders.

Partnering For Full-scale Streaming

Fragmentation can often be the enemy of efficiency because it duplicates effort and can leave resources idle. Deep levels of efficiency require industry-level partnership with the most demanding shared objectives. The Media industry on the one hand and the Telecommunications Industry on the other hand can form a partnership to work together to define the best deployment model that works for everyone and gets us to an environmentally and economically sustainable operating model more quickly. 

To repeat a point made earlier, we are talking about the 80% of internet traffic that is made up of video which can be treated differently from the other types of data in the remaining 20%. Some businesses already operate in this media/telecommunications middle-ground, providing specialized network services to the Media Industry, aggregating their content delivery and ensuring all the content reaches consumers. Some of these network services already include the terrestrial broadcast networks, which will need to be carefully phased out over time as DTT shifts to OTT. These service providers know what full-scale video delivery looks like, and some are adapting quickly to a world where video delivery changes from 90:10 DTT:OTT into an OTT-centric model. They can manage the transition pathway for the different technology platforms to decrease in DTT and increase in OTT, specifically to ensure video can be efficiently and effectively managed.

These service providers have the capability and business interests to build out our full-scale streaming platforms of the future. Under the hood of their services there is opportunity for new technologies to innovate, combining CDNs, peer-to-peer networking, multicast, WebRTC, DVB-I / ATSC 4.0, and even AI and blockchain. All of these technologies may have a role in different ways in different environments, but the complexity can be abstracted and managed to make things more efficient for both the media industry and the telecommunication industry.

When we talk about full-scale streaming, hopefully we can agree on a shared vision of an optimally efficient network, that provides us with all our forms of video in the most environmentally and economically sustainable manner possible.

How Long Will This Take?

This might sound like a very long road ahead but building out a full-scale network will not take as long as you might think. This is not the “design, manufacture, build from scratch” process of a highly complicated new system. This is a new deployment of proven software and hardware technologies (i.e., CDN technologies) in real-estate that is largely already in place (i.e., telco networks and “Exchange buildings”).

The main part of the deployment time is the network architecture planning to ensure the connectivity between the distributed Edges and from Edges to Origins is correctly defined, and this requires aligning the vested interests in the media and telecommunications markets to reach a shared vision.  Even with this planning process, the deployment is not a big-bang moment in time. The Edge capacity can and should expand over time.

As streaming audiences shift from 10% of total video viewership to 20%, 50%, 80% and maybe 100%, the network can be expanded accordingly. Aligning Video Edge network expansion with telco fiber roll-outs and broadcaster streaming plans will create a clear roadmap towards full-scale streaming.

With the right shared vision and political will, it is conceivable that the next 5-10 years will bear witness to this industry transformation towards a Video Edge Blueprint that meets all our objectives.