ST 2110-10 - Timing & Sessions

ST 2110 is built on a foundation provided by the systems layer described in Part 10. That describes how streams are announced and discovered by receivers and how they are synchronized precisely when each essence stream is delivered separately. Latency is reduced by pre-empting packet losses with redundant error correcting information in the transport streams instead of using the slower TCP transport.

Part 10 covers the timing and synchronization of the elementary streams. It describes these core parts of the system architecture:

The AMWA NMOS specifications augment ST 2110 with application level protocols and metadata which are integral to deploying your ST 2110 installation. They are all relevant and freely available to download. NMOS and other useful standards are described in the appendices.

Precision Time Protocol (PTP)

Any system must embody some latency when transmitting material between a source and destination. ST 2110 is designed to minimize the delay. It relies on Precision Time Protocol (PTP) to synchronize timing and control signals. Part 10 describes the interaction between RTP streaming and PTP timing synchronization.

The Precision Time Protocol (PTP) is fundamental to making this all work and is derived from the previously existing IEEE 1588 standard that was developed more than 10 years before the ST 2110 standard.

It is designed to synchronize events within a Local Area Network (LAN) but can be deployed more widely.

PTP is more accurate than the NTP (Network Time Protocol) protocol which is used to synchronize system clocks and can resolve intervals under 10 milliseconds. PTP can resolve events to less than a microsecond accuracy (in the region of 100 nanoseconds). This is sufficiently accurate for synchronizing whole video frames but not for individual audio samples.

PTP provides time services where GPS receivers are too expensive to deploy everywhere or where the GPS signal is shielded and cannot be received.

In normal situations, the PTP time values are based on the same epoch as UNIX time. An epoch describes the earliest date in the where the tick count is zero. In this case it is January the 1st, 1970.

PTP In Context

Timing accuracy to fractions of a microsecond are possible for PTP within a LAN. If we consider some of the data flows within an ST 2110 architecture, these are the resolutions of accuracy required to achieve real-time event delivery.

The audio sample clock events occur more frequently than PTP can resolve but this is fine because PTP regulates the timing for RTP packets that each contain many audio samples. The receiving player needs to generate its own internal sample clock for playing back the samples.

There is a latency limit calculation that multiplies the sample clock rate by the number of samples in a packet to derive the maximum delivery time allowed for RTP to transmit its payloads. Failure to arrive in time will silence the audio.

When converting an SDI stream to the ST 2110 elementary streams, time stamps are derived from the frame information in the original source SDI feed and stored in each RTP packet. The streams might travel by different routes that have different latency but they carry their timestamps with them.

PTP At The Receiving End

Synchronization of the incoming streams happens at the receiver and PTP ensures that the streams are synchronized regardless of which route was used. Late packet arrivals would imply that the minimum latency is governed by the slowest traffic. PTP would reassemble the packets into the correct sequence but the sequence as a whole might run late.

Read the annex in ST2022-7 which describes offsetting the streams to maintain synchronization.

These are the major versions of PTP.

PTP Versions

PTP has undergone some changes since it was first introduced in 2002. The evolution is necessary to keep up with changes in the way that network technologies are evolving. The internet as a whole is always in a constant state of flux requiring attention to how protocols are deployed.

A collection of amendments identified by letter suffixes have been published to update the core PTP standard. More amendments are in progress for later delivery. An additional part is expected to describe stateless operation of PTP. As well as introducing technical changes or additions, most amendments also include errata that correct typographical or technical mistakes in the core standard.

The IEEE PTP standard is available here to purchase and download:

Synchronization Between Sites

The Joint Taskforce on Network Media (JTNM) suggests these key criteria regarding synchronization are important:

• Accurate timing and synchronization within a single site is mandatory.
• Provided an incoming feed from an external source can be re-synchronized to the reference clock the individual sites do not need to be so closely time locked with each other.
• Clients synchronize to a reference clock by exchanging messages and measuring the propagation time for the round trip. The observed journey duration indicates an offset from the reference time.
• The local clock is adjusted to take that transmission latency into account. These assumptions determine the reliability of the local clock.
• The transmission time is constant over any period of time.
• The transmission time is the same in both directions.
• Both endpoints can accurately determine the message departure and arrival times.
• If different time sources are used at each end of a connection, then an offset may need to be measured using a calibration mechanism. Choosing the same reference time source at each end is the optimal approach.

Integrating OB Units

There is an integration challenge with live feeds from an Outside Broadcast (OB). Provided the feed is continuous and glitch-free, we need not worry about the internal implementation. Frame accurate synchronization for source switching is necessary but should introduce less than a frame of delay. Likewise, the playout system may need to buffer files that it fetches from the asset store so it can stream them out without interruption. Everything else downstream to the receiving devices works as it always has.

Time synchronization between an OB unit and the HQ system can be facilitated by using an external time source such as GPS. This would be available everywhere and locking everyone to GPS ensures everything is based on the same reference time source. Cheaper GPS receivers should be able to resolve to 100ns accuracy while more expensive solutions can resolve to smaller intervals if they are stationary and receiving a clear signal.

Be aware that there are plans for new technologies to replace GPS that are already in hand and there are alternative services available depending on your location.

There have also been instances of GPS being jammed and interfered with, causing a loss of signal. Developing a back-up time source might be useful in case of problems. You may already be doing this if you are news-gathering in a conflict zone.

Relevant Standards (PTP)

These are the standards are relevant to PTP in the context of an ST 2110 system:

Session Announcement Protocol (SAP)

SAP is a protocol for advertising session details to all locally reachable nodes. SAP senders periodically transmit SDP descriptions to port number 9875 at a well-known multicast address. These are retransmitted often enough that latecomers can also receive them in a timely manner.

A well-known multicast address is a special IP address on the LAN which really means every available node that is currently listening. So every connected system will receive a connection request on its own port number 9875 at the same time. Because this is managed by the router, the sender only needs to send one message to this special address.

The SDP description is carried as a payload in a Session Announcement Protocol (SAP) packet. Here is an example SDP payload taken from Annex B of the ST 2110-10 standard.

SDP announces new streams as they are deployed. Nodes receiving these announcements can act on them to acquire the details of the remote stream if they qualify for the invitation.

Despite it appearing complex, this is a simple format to construct. Some of the data items will already have been defined as configuration settings and will be the same for all sessions. Others are unique to this announcement.

Session Initiation Protocol (SIP)

SIP is a protocol similar to HTTP and SMTP. Because it is text-based, interoperability and integration with other sub-systems is straightforward. It is used to initiate an SDP session that describes the transmittal of media elements in an ST 2110 architecture.

When setting up an SDP session, the payload of a SIP message carries an SDP data unit, that describes the media format, codec and media communication protocol for the session.

Session Description Protocol (SDP)

The Session Description Protocol (SDP) announces streams and services that are vended by server-nodes in your network. A client-node discovers these and can access them as a live-feed source using the details contained in the SDP message.

Automatic discoverability is provided by carrying SDP messages via the Session Announcement Protocol.

Session configuration is established by carrying a descriptive SDP message via the Session Initiation Protocol.

Refer to Section 8 and Annex B of the ST 2110-10 standard for details of how SDP is used in the ST 2110 environment. This is based on RFC documents published by the IETF. Those documents have been revised several times and the most recent version replaces all of the earlier revisions even if ST 2110 does not track the changes:

SDP Parameters

An SDP-based signaling protocol is defined for technical metadata necessary to receive and interpret the streams. ST 2110-20 section 7 describes this SDP metadata which is transmitted separately to the essence. Refer also to RFC 4566. Additional SDP metadata is described in ST 2110-21 as it applies to traffic shaping. This metadata applies to all samples, rows, fields, and frames in a stream.

The metadata is delivered in name-value pairs or just as names which are treated as flags having a default value associated with them:

Here are a few examples. Some of these are mandatory and others depend on the format being delivered. Read section 7 of the standard for details:

Observe that the height and width parameters have a range already capable of dealing with 32K video.

The SDP parameters relating to traffic shaping are:

Real Time Transport Protocol (RTP) Streams

ST-2110 content is delivered with Real Time transport Protocol (RTP) streams. These were originally described in RFC documents and have been revised and extended many times since they were first introduced. RTP is extensible. Alter the behavior to suit your needs. The streams are all locked to a common timing reference defined by PTP and time-stamped with markers at the beginning of each video frame.

ST 2110 is based on the RTP protocol as originally defined by the IETF in RFC 3550 with the further caveat (mentioned in ST 2110-10) that it should conform to the profile extensions defined in RFC 3551 to describe the audio and video payloads. The ST 2110-41 standard for delivering metadata describes RTP packets that only need to conform to RFC 3550. The payload description in ST 2110-41 is much simpler and makes no mention of RFC 3551. It refers the reader back ST 2110-10 for further details of the packet size limits.

RFC documents are frequently superseded by later documents with different numbers that enhance their protocols. The RFC 3550 and 3551 documents do have enhancements published as later RFCs that update and extend their capabilities. In order to remain fully compliant with ST 2110, continue to stick with the older versions described therein. At some point, the SMPTE ST documents should be updated to refer to those newer RFCs if the enhancements are relevant.

RFC documents are easy to find and download to read and understand the fine detail.

There is a later secure version of RTP described as SRTP. At the time of writing, security issues have not been addressed at this level in ST 2110. That is likely to become an important working group topic although boundary security should deter external intruders for the time being. SRTP might be useful for managing internal security permissions and authorized access to resources. This is a topic that will be addressed in the ST 2138 control plane standard.

RTP Packet Structures

The RTP packet structure is described in detail in section 6. Refer to IETF RFC 3550 to see this in context. The transmission is affected by traffic shaping considerations covered in ST 2110-21. The Virtual Receiver Buffer model was covered in an earlier chapter. To recap, the senders will be one of these three types:

• Type N - Narrow Senders.
• Type NL - Narrow Linear Senders.
• Type W - Wide Senders.

They are also further described in Appendix A of VSF TR-05.

Recall that for ST 2110-20, only the active picture area data is transmitted. Audio and Ancillary data are delivered separately.

The RTP payload format is described by IETF RFC 4175.

Some signaling information directly relevant to the essence is carried with it in the RTP packets. The packets are timestamped so the essence can be reconstructed reliably from an ordered sequence of packets. Other supporting metadata is transmitted separately as Session Description Protocol (SDP) streams.

The start and end of each progressive frame or interlaced field are marked in the RTP packet header. The receiver can detect these boundary markers and initiate a picture reconstruction. These must be interpreted carefully when dealing with progressive vs interlaced content as their behavior is context dependent.

For progressive video frames, the Field ID marker acts as a Frame Start indicator. The last packet carries an End of Frame marker:

The structure of an interlaced frame is more complex and the two fields are marked individually:

Restoring the field dominance correctly is important to avoid introducing jitter when the streams are processed back into video frames.

Progressive Segmented Video (PsF) transports the picture as if it were interlaced but the picture can be reconstructed as a progressive image in the receiver. This allows progressive material to be processed through workflows that only understand interlaced formats.

The RTP payload is a collection of Pixel Group (pgroup) data structures. A pgroup is a run of pixels managed as a block. It allows picture areas to be defined as non-rectangular shapes. This is possibly the most complex part of ST 2110-20. Read about the RTP payload in section 6.2 then read TR-05 for additional insights and example calculations. The construction of a pgroup is somewhat complex and depends on the chroma-sampling model being used.

The header of the payload describes where in the picture rectangle the pgroup should be placed. The line number and offset describes a pixel accurate location. This is most often used to insert a channel-identifying graphic. Using a pgroup significantly reduces the amount of data being transmitted if the stream only contains a sub-raster (overlay) with most of its picture area unoccupied.

Relevant Standards (RTP)

These documents are useful when studying the RTP packet structures:

Forward Error Correction (FEC)

The RTP streams are delivered using UDP transports. These have low latency but can suffer from packet loss and sequence order scrambling. Buffering and retransmission solves that when you use TCP transports, but that introduces significant latency.

ST 2110 anticipates the potential packet loss by adding redundant data to facilitate the reconstruction of missing packets. This technique is called Forward Error Correction (FEC). It does add a small latency overhead because the entire FEC stream needs to be received and processed to potentially recover lost media packets.

There are two different levels of protection offered by FEC based on a row and column arrangement that organizes the packets into a two-dimensional grid:

• Level A - Column only protection.
• Level B - Column and row protection.

Sometimes the word ‘Datagram’ is used. In the context of ST 2110 and IP, this is interchangeable with the term ‘Packet’.

Read ST 2022 Part 5 for a detailed explanation of how FEC works.

The following standards are relevant to understanding FEC and how it fits into the RTP context:

The XOR Operator Used In FEC Calculations

Let’s digress for a moment to explore and understand the Boolean Algebra Exclusive OR operator. This is used in Boolean logic expressions in a similar way to the arithmetic operators in mathematical expressions.

The FEC packets are created by combining the payload in the media packets using the XOR logical operator. The XOR operator is used because applying it once reverses all the bits in the source that correspond to a 1 bit in the XOR mask. A second application reverses them back to the original state with no loss of data.

At the receiving client, the FEC is combined using XOR in a bitwise fashion with all the incoming media packets in the same way. If any single packet was dropped, the result will be the content of that packet. The sequence number on the media packets should be monitored to indicate if a recoverable dropout has occurred. Recovery of a missing payload packet is only possible if ALL of the FEC packets have arrived intact. If there is a single gap in the payload sequence, it can be repaired with the computed FEC result.

FEC Protection - Level A

Level A protection is a one-dimensional arrangement with a sequence of media packets aggregated to create a single FEC packet. The length of this sequence is defined by your configuration settings. The media packets are transmitted to a designated IP port at the client node. The FEC data is transmitted separately to a separate but nearby port. This technique is resilient to the loss of a single packet.

FEC Protection - Level B

Level B protection works on a two dimensional grid of media packets with a FEC packet separately created for each column and row. The FEC is calculated for a column in the same way as Level A. The columns are stacked in rows based on the configured column length. If the column length is set to 100 packets then every 100th packet will start a new column. This is more robust and the FEC overhead is slightly larger. There are a variety of different ways to arrange these columns. ST 2022 Part 5 discusses the advantages of offsetting the columns.

FEC Stream Arrangement

The FEC packets are transmitted separately to the media in their own RTP stream. This allows them to use an RFC 3550 compliant packet header without alterations to add FEC protection to the essence streams. The column and row FEC streams are transmitted separately so that they can have their own sequence numbers.

RFC 3550 mandates that a media stream is delivered to an evenly numbered port (N) at the destination IP address. The supporting column based FEC packets for a Level A connection will be sent to port number N+2 at the same address. For a level B connection, the column packets will be sent to the same relative port (N+2) and the row FEC packets will be sent to port number N+4. Therefore, each incoming stream can potentially require three evenly numbered IP ports to be allocated at the receiving client.

The receiver processes the media stream to check the packet sequence numbers and reconstruct any missing packets by combining the incoming media and FEC packets on the associated streams.

FEC Ranges & Limits

Because the RTP stream has timestamps that indicate the boundary between one video frame and another, it makes sense to base the size of your columns and rows on those markers. Then, you may add extra layers of protection to request retransmission of entire frames if FEC provides insufficient protection on its own.

The number of rows and columns that can be protected with FEC is described in ST 2022 Part 6 (see sub-section 7).

For a Level A scheme, between 4 and 255 media packets can be aggregated for FEC. A longer run has less FEC overhead but is more likely to lose more than one packet. The resilience of your network will indicate how you can tune this value in the configuration to match the expectation of a single packet loss.

A Level B scheme can have the same column length of between 4 and 255 packets. The row FEC can accommodate between 4 and 1020 columns. The RTP header sequence number will wrap around when it reaches the maximum count. Therefore, the aggregated number of media packets should be less than that to avoid sequencing errors.

The row and column limits are related. Multiplying them together should yield a value less than the permitted maximum (FEC max). The protection column indicates the duration of an outage that can be repaired.

3G is a bandwidth definition and should not be confused with mobile telecoms standards.

Read ST 2110-10 First

It is important to read ST 2110 Part 10 before the rest of the standard. The background information in the supporting documents from other sources will also help you understand how the systems layer works.

Beware that ST 2110 sometimes mandates earlier versions of external standards. Nevertheless, it is still important to be conversant with the latest versions in case ST 2110 is revised to accommodate them.

These Appendix articles contain additional information you may find useful: