What Are Media Containers?

Choosing the right media container file facilitates ease of production and deployment. Some container types allow multiple time-synchronized assets to be embedded inside a single file. The various codecs in common use and their symbiotic relationship to the media container files are fundamental for packaging content effectively for storage or delivery.

Some containers describe an essence format that is integral to the packaging mechanism. Many of them are based on Matroška and RIFF as their foundation but having come from different regimes they are incompatible with one another at the reading and parsing level even though internally they are very similar. Containers also carry descriptive metadata to facilitate the production.

Why Do We Need Container Files?

Media storage container files are essential for managing and automating workflows. Streaming architectures such as ST 2110 are useful in specific situations but permanent storage is always necessary. Each container format is designed for a specific task:

• RAW Essence – Individual essence files simplify the workflow process. The workflow must maintain the synchronization and run length, but can apply effects and other modifications more easily. This facilitates automation.
• Production – Production files carry uncompressed data for the content editing and creation process. Many different data types are stored in these files. Audio and video are the most obvious, but additional metadata that describes visual effects, 3D models, textures and associated timed text must all be stored in a structured fashion.
• Distribution – File containers for distribution are optimized for streaming to client player applications and devices.
• Archiving – Archival formats benefit from additional metadata for the supporting materials stored with them. It may be expedient to collect scanned images and textual documents into
  a single container. A Zip archive might be appropriate. Proprietary and application specific file types are not recommended. Simple, uncompressed sampled audio and individual video frames are good. Plain text files with markup to add styling are more likely to be readable in 500 years’ time than a highly structured word processor file.
• Legacy – Because archive and library collections tend to gather assets over a very long time, vintage content will be stored using older formats. These should be carefully documented. Older formats should be up-converted to a higher quality storage container to avoid losing access. Once the assets have been ingested, this can be automated to run on a scheduled basis.

What Do We Need To Carry?

Modern containers can carry multiple video streams and multiple alternative audio tracks for language, accessibility support and surround-sound. Other kinds of media can be stored and synchronized with the video and audio for subtitles and interactive presentations.

Containers Vs. Codecs

Confusion sometimes arises because a few containers have the same name as a codec stored inside them. The AVC/H.264 video codec has several alternative names which adds to the uncertainty. Currently the most popular codecs for end-user deployment are:

• AV1.
• AVC.
• HEVC.
• MPEG-2.
• VP9.

These can all be carried in the ISO standard MP4 container along with anything else that has been registered as a compatible media type. Virtually anything can be stored in a Matroška container file.

You may see containers described as being lossless but the containers themselves do not affect the video quality. They are simply a means of packaging the output of a codec. It is the codec that might be lossy and not the container!

Compatible Combinations

Certain combinations of video and audio codecs are frequently used together. If you defy the conventions and use an unusual combination, your viewers may only see the video but not hear the sound, or vice-versa. The container type dictates your available choice of audio and video codecs. 

Common Container Files

There are many alternative container formats to choose from. Some are designed for simple carriage of a specific kind of media essence. Others are general purpose packaging formats but can also be used for media. There is a small sub-set of formats that you will likely encounter in day-to-day use.

• AAF – Advanced Authoring Format.
• AES31 – Portable audio content.
• AIFF – Audio Interchange Files.
• ASF – Advanced System Format.
• AVI – Audio Video Interleave.
• IMF – Described by SMPTE ST 2067.
• Matroška – The basis of many other open formats.
• MPEG-4 – Based on ISOBMFF.
• MXF – Material eXchange Format.
• PDF – Portable Document Format.
• QTFF – QuickTime File Format.
• WAV – Waveform Audio Files.
• WebM – Web Media Files.
• Zip – Generic archive packaging (also used as the basis of some document types).

The most versatile container formats are the ISO Base Media Format Files (ISOBMFF) also known as MP4 (MPEG) and the MKV (Matroška) format.  These offer the widest range of codec compatibility and support across multiple platforms. The MP4 format in particular is well supported in all browsers and many other kinds of devices. The AVI file format is almost as versatile but lacks support for some modern codecs.

Analyzing Container Files

An archiving system with historical content will likely contain some ancient formats in addition to the more familiar and commonplace modern containers. You need a range of generic tools to operate on them. It is important to understand their internal structure. That equips you to design more powerful workflow processes to interact with them. Here are some toolkit ideas:

• Examining the internals of video, audio & image file formats.
• Hex dumping files for analysis.
• Disassembly tools to inspect the internal structure in detail.
• Validation tools to look for incorrectly formatted content.
• QA checking tools to validate that the file has all the metadata required for distribution.
• Reading the headers to extract metadata.
• Internal metadata vs file system modification time stamping.
• Identifying unlabeled files.
• Forensics and tracing the provenance of the content.
• Verifying the authenticity of content.
• Detecting AI generated content.
• Command line tools to extract metadata to steer shell scripts.
• Modifying the file content from the command-line.

Identifying Unlabeled Files

The forensic and verification processes will increasingly rely on technologies such as C2PA which can embed metadata and watermarks to authenticate content.

In a UNIX environment the File Magic tools are useful for identifying unlabeled files with missing or incorrect file extensions. Alternatively, use the od command to dump out the first few bytes of a file for inspection. Files often have recognizable tags at the beginning which identify the content type. The File magic tools use this to identify the file type.

If containers have been created correctly, the file extension should identify the content. In some cases, the essence carried inside the container may need to be inspected to determine what it is when the file extension is ambiguous.

File Name Extensions

File name extensions identify the contents of a file provided the correct one is used. Some containers have multiple alternative file extensions for subtle variations of their content. Audio only and video content often have different file extensions. Some file types have synonyms that identify the container type with more than one extension, but the content is identical for both extension values. One of these will be the default or preferred file name extension.

Selecting an appropriate handler inside your software application can use a coding technique called a dispatcher. This examines the value of the file extension and builds a switch/case tree to select the right code:

• Always select the last item from the filename splitting process. This avoids spurious cases where the filename might have multiple period characters embedded. That is more commonplace than you would expect.
• Convert the file extension to lower case to avoid the need for multiple switching routes for upper and lower case variants. Some file systems behave differently regarding the sensitivity and preservation of upper/lower case naming.
• Always implement a default generic handler in your switch trees for unmatched extensions.

Here is a pseudo code example that is easily translated into the language of your choice:

These are the file name extensions for container types you may encounter. There are others that are more obscure and may crop up in rare circumstances. Some of these are simple containers for a single element or media object. Others can carry multiple elements which are synchronized for presentation in a player:

Product Design & Coding Quality

We just contemplated a small coding example that implements a switch tree dispatcher. Coding is not only a specialist craft skill but somewhat of an art that is improved by experience and diligence.

After 10,000 hours of practice you will become adept and highly skilled at any endeavor you apply yourself to. Aspire to become a JEDI Master.

Choosing the right algorithm is important. The wrong one leads to excessive code needed to make it do what you wanted.

Reduce the number of lines of code without obfuscating it. I guarantee that six weeks later even the programmer who wrote it will struggle to understand what they composed. Less code = less maintenance overhead.

Skilled developers work in several coding languages at the same time. For example, this is a very powerful combination for high performance web server back-end development.

Selecting an appropriate language is also vital. Languages are not all created equal even if they do support many of the same programming constructs. Choosing the wrong language because it is the only one you are familiar with will lead to serious problems later on.

Libraries of code obtained from open-source projects are powerful time savers. Open source apps often have command-line interfaces that can empower your designs.

Develop a mindset that always builds robust behavior into your product design and coding to avoid the need for maintenance fixes later. It may take a little bit longer but it will save time, trouble and money. You also avoid the damage to your brand image and reputation caused by fixing bugs in public view. That just demonstrates ineptitude and insults your customers. Always go the extra mile so your customers won’t have to.

Abbreviated Container Type Names

These are not file extensions but abbreviations that are often used to describe media containers in documentation.

Format Owners & Creation Dates

New container formats are developed from time to time. It is argued that the later ones are more useful because we would expect them to inherit features from what went before. The Matroška container is probably the most versatile and has been around for some time. The more recent Adobe Flash video containers are now obsoleted so later doesn’t always mean better.

Patent Rights & Intellectual Property Issues

Patent and royalty issues may determine your choice of container. Some licenses cover patented technologies which are provided free of charge while others levy a fee on all content created with them.

The MPEG standards are usually patent encumbered and require license fees to be paid. The Matroška format (MKV) is license free and whilst the WebM profile might prefer to use codecs built with patented technology, they are carried in a royalty free container.

Note that there may be royalties on the video codecs used to code the video stored in a container but not on the container formats that carry the essence. Since container formats are standardized separately, any patent encumbrance could apply to either, both or neither.

Media Type Identifiers

These Media Type identifiers are used when delivering content over a network connection, usually to a web browser but not exclusively. Any application can use them to identify specific or generic formats.

These values are sometimes described as MIME types. The name is inherited from when they were first invented to identify individual parts of a mail message so that attachments could be included. It is appropriate to use that terminology when they are described in the context of a mail message forming part of the title of an RFC document.

Most of these media type identifiers are registered with the IANA but a few are not. One web site that claims to list all the available media types declares that there are nearly 3700 different kinds, but even that list is incomplete. There are other identifiers used in broadcasting that are not included in that collection.

The Media Type identifier format is in two parts. The first is the general category and the second is the unique type. The same unique suffix type may be cited in several categories to describe different contexts and container contents. For example:

Generic Containers

These containers can be used for a variety of different media:

Audio Media Type Identifiers

Some identifiers are defined to describe audio-only content. Audio can be carried on its own in the video types, but it is not recommended to deliver video in the audio-only containers to avoid confusing the player logic:

Image Media Identifiers

These are a few of the more popular image types. There are many more. Some of them support a variant that carries a sequence of images that is presented as a very short looping movie.

Video Media Identifiers

Video encapsulation should use containers that are labelled as such. It is often permitted to carry audio with no video streams but you should confirm that with the specifications. Video content has a display rectangle where sub-titles may be displayed. If there is no video rectangle the player may struggle to support them:

Relevant Standards

Where a standard comprises multiple separate documents that are not enumerated, the date for the latest part or amendment is used to define the vintage of the whole standard:

Choose Your Containers Carefully

Storing media in container files must be an entirely lossless process even if the payload might have been compressed with a lossy codec.

Worldwide Standards Convergence

The SMPTE IMF file format is an industry standard preferred by many organizations that need to share content. Whilst there are alternatives that may be technically superior, some of them are proprietary.

The creation of world-wide standards and supporting advisory documents – such as SMPTE TSP 2121 which describes the Interoperable Master Format for use in DPP applications using Apple ProRes coding – moves us closer to content that can be deployed in multiple countries with little or no post-processing required.

If this level of collaboration continues, the future does look very encouraging, but it will depend on agreed standards to make it all work.

These Appendix articles contain additional information you may find useful: