Related content:

In a traditional broadcasting environment, media assets (sound, video, timed-text and metadata) are implicitly synchronized in Real-Time. When the assets are delivered as separate elemental streams through an IP environment, timing is asynchronous and there are no guarantees regarding timing accuracy without intervention. Packets may arrive out of sequence or not at all unless you use additional layers such as TCP to buffer and handshake the incoming data stream. Even then, the timing no longer resembles real-time. Countermeasures must be applied at the destination to resynchronize the assets. This needs careful planning and design with additional timing metadata being inserted at the transmitting end. Latency will still introduce a delay regardless of how well the assets are resynchronized.

Timing Domains

Timing of media assets is complex. On the one hand, we are care about micro-timing with PTP to ensure frame accurate switching and syncing of sources. On the other hand, timecode for subtitle texts, user navigation and essence metadata can be less precise. Date values become important when we move assets to our archives.

These are the principal domains where timing must be carefully managed:

• Precision timing - Important when synchronizing multiple element streams. Also, important when converting the time-base from 50 to 60 FPS or vice versa.
• Sample accurate - Purely monophonic audio is rare so there will be at least two synchronized sound channels. Sample accuracy is important to avoid phase related distortions.
• Frame accurate - Individual video frames must be synchronized with accompanying audio samples.
• Timecode offset - Synchronization within the asset is needed for timed-text subtitles, chapter marks and other interactive event-triggering metadata.
• Origin time values - Indicate the start of the asset. This is used for broadcast scheduling and as a reference timestamp for timecode relative offsets.
• Date values - Recorded in the metadata for archiving. Content creation dates, modifications and other relevant date information is necessary to maintain content stores and archives.

Video assets may include multiple synchronized picture streams for alternative viewing angles, stereoscopic images and augmented or virtual reality scenes.

Unless the cameras were reliably synchronized to each other for multiple camera angles, some phase offsets for the individual frames might occur. This would be visible if high speed cameras are used to capture motion that is played back at normal frame rates. It might be important for stereoscopic and VR use to avoid unnecessary eyestrain.

Surround-sound systems require 6 channels for 5.1 and 8 channels for 7.1 formats. 8K TV services (NHK) require more than 20 channels of audio to be delivered in phase accurate synchronization.

Reference Time Sources

In order to lock everything in perfect synchronization, a reliable source of reference time signals is needed. There are a variety of solutions to this. Analogue technology could survive some degree of uncertainty and the receiving equipment could generate synchronizing signals internally. This is not feasible with digital systems delivered via IP and an external reference is necessary.

Reference time values are embedded with the transmitted assets and then reconciled at the destination. The receiver can acquire an accurate clock value from the same reference source that the transmission system used. Here are the important reference signals and specifications that are commonly used:

SMPTE Timecode

A long time ago, SMPTE standardized the concept of timecodes. Refer to ST 12 for details of the classic timecode specification. Timecodes are used in film, video and audio production. They can be encoded in a variety of different formats depending on the medium:

UTC Timestamps & NTP

UTC time is used in computing systems. Each day is exactly 86400 seconds long. The Network Time Protocol (NTP) will synchronize the internal system clocks to UTC. NTP achieves an accuracy of approximately 10 milliseconds over the Internet and 1 millisecond on a Local Area Network (LAN). NTP can operate using a client-server or peer-to-peer model to exchange messages for synchronization. The messages are sent using UDP protocol on port number 123. This must be left open in the firewall to avoid blocking the time-synchronizing messages.

The UTC timestamp can be represented in a variety of human readable formats. These are defined by ISO and RFC standards. Incoming textual metadata containing date-time values must be parsed to extract the UTC timestamp value. It is much more useful to store the numeric UTC value in a database instead of the human readable date string. Relative time and date/time arithmetic is then easy to calculate. It is also more compact. Here are some example date-time-string formats:

IEE 1588 - Precision Time Protocol (PTP)

PTP is most useful when endpoints must be closely synchronized without access to GPS timing signals.

PTP version 1 was originally introduced in 2002. The later version 2.0 was released in 2008. Because version 2 is incompatible with version 1, they cannot co-exist on the same network. Version 2.0 introduces the concept of profiles and adds support for IPv6 addressing. Profiles simplify the configuration of services. Version 2.1 is upwards compatible with version 2.0 and adds these features:

• Security
• Robustness
• Monitoring

The standard was completed in 2018 and published in 2019. References to the standard may describe either date.

The profiles introduced at version 2 have been developed for many industry specializations. This sub-set is particularly relevant to broadcasters.

PTP resolution is accurate to less than a microsecond within a Local Area Network (LAN). Transporting messages to other networks will transit through a switch or router. The propagation delay may affect the performance.

PTP is based on the same Epoch as UNIX Time which has a reference starting date of 01-January-1970. UNIX Time is based on UTC values and will be affected by Leap-Seconds. PTP uses International Atomic Time (TAI) which is unaffected by leap-seconds. Be very careful when reconciling PTP time values against timestamps from other sources because they could be inconsistent.

Observe the PTP traffic with network analysis tools such as Wireshark. This is a useful tool for recording network traffic. It can store the entire session in a file for later detailed analysis.

Timing Accuracy & Resolution

The resolution of various time references and the system times on different platforms affects the accuracy of any media synchronization. Here are some examples:

Accessing these time values is also dependent on the programming language and style of execution you use. Applications running in interpreted languages will not be able to match the accuracy of compiled code. Scripting languages such as Python & Perl do have embedded support for more accurate time measurement but this escapes out of the interpreted context to run compiled code extensions. An appropriate expression we use in computing circles is:

“Your mileage may vary.”

Time-Linear vs. Time-Non-Linear

The relationship between audio and video is traditionally maintained in a time-linear fashion which keeps them closely synchronized. This exists in a Real-Time context. Processing must also be carried out in real time with minimum latency introduced. Ingest the media into a non-linear editing tool (NLE) where faster processing is available and synchronization is maintained within the application and regenerated on export.

In an IP based infrastructure, the only places where real-time delivery is mandated are at the ingest (camera and audio sampling hardware) and play out to the viewing public. Everything else can be operated on in a Time-Non-Linear fashion provided the necessary metadata carries precisely timestamped markers.

In a broadcast context, the more you distribute the processes involved, the harder it is to maintain timing accuracy and synchronization. With the introduction of IP based infrastructure, the television content is being moved from a Time-Linear environment to a Time-Non-Linear environment. The transportation of the content is no longer time synchronous. Accurate synchronization is reconstructed as the content is received back into a Time-Linear environment.

The transition from Time-Linear to Time-Non-Linear and back again happens at the cloud compute resource edge where content is uploaded or downloaded. Content moving around within the cloud compute resource or from cloud compute resource to cloud compute resource will remain in a Time-Non-Linear format.

The Video Services Forum (VSF) has published a useful Technical Recommendation TR-11 which describes Signal Transport and Timing Considerations for Ground-Cloud-Cloud-Ground workflows.

Remember, cloud compute resource can be in a private (on or off prem) data center or a public cloud service.

This covers three transitions from one environment to another:

• From your core systems up to a cloud compute architecture.
• Exchanges between several separate cloud compute systems.
• Downloading content from the cloud compute resource, back to your core systems.

TR-11 refers to other VSF published material which is all freely available. Acquire a complete set of their documentation for your reference library.

Moving content to a cloud resource and back again is guaranteed to introduce some latency. This is caused by coding requirements and bandwidth limitations during transport. Buffering at the receiving end is necessary to reconstruct a time-linear output but introduces more latency.

VSF recommends these solutions:

• H.264 low latency over MP2 Transport Streams, via VSF TR-06 (RIST).
• JPEG XS as described in TR-07, via VSF TR-06 (RIST).
• JPEG XS as described in TR-08, managed as described in VSF TR-09.

The TR11 document has expectations of timing accuracy being within 0.005%.

A real-world timestamp needs to be established for the start of the media as it enters the cloud environment. The timestamps may be expressed in different ways depending on the source. Applying an offset allows for latency if it is predictable. The timecode may be embedded within the incoming signal or carried in a separate metadata stream:

• ST 2110-xx incoming media should adhere to ST 2110-10 (version 2022). This uses an RTP timestamp on the incoming stream.
• ST 2038 coded VANC data carries an Ancillary Time Code (ATC) extracted from the ST 291 stream.
• ST 2110-40 uses the same ATC value.
• In the H.264 stream, the Supplemental Enhancement Information (SEI) carries a timecode value.

Carefully review VSF TR-11 because it goes into great depth about how to implement the upload and download process whilst preserving the timing information.

The GCCF API

The Video Services Forum provides sample code to illustrate how an API is implemented to insert media into a cloud compute based service and retrieve it at the receiving end. This is still a work in progress but is available as open-source from their Git repository.

The API is called the GCCF and integrates your workflow with the content in a web-based service so it can be integrated into your dashboard support more easily.

GCCF can be used with ST 2110 based workflows. The content buffering is designed for compatibility with ST 2110 conventions. Timing information is carried in a Media Element Header structure. The essence data is presented as separate elements as defined in ST 2110:

• Video buffer.
• Audio buffer.
• ANC metadata.

In addition to the send and receive mechanisms, the API also provides support for NMOS interactions. An IS-04 Node API is available and connection management conforms to IS-05 as described in AMWA BCP-004-01.

GCCF also vends a time service based on TAI values (International Atomic Time). These are protected from discontinuities caused by Leap-Seconds. The timestamps are accurate to within 50 microseconds. Phase relationships are also important and the timestamps are regulated to within plus or minus 100 milliseconds of their expected arrival times. The values should continuously increment for each successive read.

There may be some discontinuities where time apparently goes backwards at startup or when corrective events arrive to synchronize everything back to the reference time.

Although it is not yet finished, GCCF would be a good starting point for experimenting with the design of a cloud compute interface from your workflow.

Things That Go ‘Bump’ In The Night

There are a few important points to consider in the context of precise time values. Things can go wrong for very obscure reasons. Traffic in an IP based network moves very rapidly. During a 1 second outage of timing reference signals, many hundreds or even thousands of events and transactions might occur. If these need to be time stamped and logged, serious problems can happen later on downstream if the correct ordering of these events is compromised.

There are multiple reference time sources derived in a variety of different ways. They are somewhat coordinated but there are significant discrepancies between them. 

You must use the same reference source throughout your architecture.

Here are some potential issues that could crop up. Some of these might seem unlikely and very far off but they are important:

Related Standards, Acronyms & Guidelines

The documents and acronyms listed below are relevant to time related services in an IP based broadcast context:

Conclusion

There are subtle aspects related to timing reliability in media systems. If the correct approach is used these problems can be avoided altogether. Using Atomic Time (TIA) based regimes is helpful. Storing timestamps in a machine-readable format is also a good idea and avoids time zone and daylight savings issues where sub-optimal date-time parsers are deployed for timestamp conversion.

All programming languages have date-time support. Most of these are based on conventions developed a long time ago in the ANSI Standard C-Language. They are accessed via the Standard Library.

Web server middleware platforms such as PHP are based on the same library and function names with some influences also from the BASH shell scripting environment. Familiarity with C-Language and BASH will accelerate your understanding of PHP.

The GCCF API will prove very useful as a foundation for developing cloud compute integration with your existing workflow dashboards.