Meta What?

Metadata is data about data. The content is described as essence data. Metadata describes any properties of that data which are not directly involved in the management of the essence. Here are some properties of a simple audio file with an indication of whether they are essence or metadata.

Metadata is the glue that connects all of the media assets to each other. Here is where it affects your systems and processes:

• Workflow process pipeline steering.
• Content searching in the library.
• Creative process support.
• End-user playback experience.
• Container management.
• Revenue generation.
• Sharing and exchange with other organizations.
• Deployment.
• Scheduling.

Metadata is vital to managing your collection of assets in an archive or librarian system. It describes the relationships between different media assets so they can be managed as sub-sets owned by a particular program.

Metadata steers the production processes and broadcast delivery. Each stage is enabled by specific kinds of metadata.

During playback, metadata driven signals trigger remote actions to synchronously present supporting assets alongside the viewer in the client player.

Structured metadata is often converted into XML or JSON formats for transmission via a web server. Unpacking those formats is well supported in web browsers with JavaScript.

Metadata Quality

Reliability and performance are impacted by the quality of your supporting metadata.

Media asset collections are curated with a content management system. This typically describes the workflow from ingest to asset storage ready for playout. Although that is the core part of the production process, the asset exists well outside of that context and the metadata system should reflect the entire life cycle.

Poor quality or missing metadata will compromise any activity that depends on it and persuading operational teams to enter metadata has always been problematic.

Useful metadata can be created at the conceptual stage of a project and more insights can be gathered from the consumer’s playback device to analyze their level of engagement with the content and any advertising that you include with it. The user experience can also be steered by metadata that is created during pre-production. These aspects are rarely supported by a content management system. Embracing the entire life cycle facilitates significant time and money saving automation.

What Is Metadata?

Video and audio recordings are described as Essence Data. They are the embodiment of the content in a tangible form.

Anything else that is not directly controlling of the playback process is Metadata.

Timed sub-title text is a special case that carries human readable text as essence data. It can also carry control signals, chapter marks and other descriptive information which is very much in the realm of metadata. Timed text looks a lot like metadata but it really is an essence format.

Metadata helps to make the best use of your content assets. Ideally, it should be developed in parallel with the essence format design.

Metadata Standards

There are very few metadata standards and they only describe the core assets. This is sufficient for a basic content storage system. There are no metadata standards for workflow process control or lifecycle management. There are proprietary systems that do this, but they are not open-sourced.

Our primary focus is on the lifecycle of media assets for creating broadcast or streamed media. No single schema will cover everything. For full coverage, multiple schemas are necessary but these can be linked using Relational Database Management System (RDBMS) techniques.

It may be useful to look outside the broadcasting media walled garden and consider content-specific standards. If a natural history documentary program is being created, then knowing about metadata standards such as the Access to Biological Collections Data (ABCD) schema is helpful. A program about space exploration might make use of a resource database built with the Astronomy Visualization Metadata (AVM) schema.

By incorporating external knowledge-based metadata into the workflow, additional checks and balances can be applied automatically. For example, nomenclature and terminology used in subtitles can be reconciled against subject specialist resources and errors highlighted for correction before deployment.

There are a few metadata standards directly relevant to digital content management amongst the many others that address a large range of topics.

The Digital Curation Centre (DCC) provides a useful list of metadata schemas with descriptions and links to resources:

Amongst others, DCC describes these schemas which are particularly relevant when building digital media asset management solutions:

The DublinCore Metadata Initiative

For a long time, the DublinCore Metadata Initiative (DCMI) has been the most well-known and often quoted example of a metadata standard. The EBU has adopted DublinCore and enhanced it to create their own metadata standard (EBUCore). Public Service Broadcasters in America have derived PBCore from DCMI. Likewise, the Audio Engineering Society have based AESCore on DCMI too. Parts of the DCMI specification are also used in the EPUB standard for eBooks.

Start with DCMI for your own metadata designs because it encapsulates a description of the fundamentals. These are covered by high-level element types originally defined in 1995. In 2000 the standard was revised to allow extensibility and controlled vocabularies. Many more properties were added in 2008 and the metadata terms were redefined as Resource Description Framework (RDF) properties. Additional inherited RDF classes are used to define domains and range limits for the DCMI properties. RDF is a W3C standard and this helps with deployment on the Internet.

Alternatives To DCMI

Whilst DCMI is a useful standard, many additional properties may need to be recorded at each stage of a workflow. Other schemas might suggest enhancements to the foundational DCMI:

Proprietary Designs

Although it has never been published as a standard, one of the most complete schemas was inside Apple’s long discontinued Final Cut Server product. This was originally based on the Proximity Artbox content server. Final Cut Server described over 1200 different properties belonging to a media asset. Used selectively, they are axes on which the media library could be filtered and searched. This was a very sophisticated media librarian.

Some properties described deeply technical aspects of the footage such as shutter and aperture settings. Others pertained to the ingest of movie film including the text burned into the edge of the film substrate.

Clearly the aperture and lens settings metadata will be very useful when working on the footage to add visual effects.

Interestingly, some of these obscure metadata properties are also still present in the macOS Spotlight search engine and can be selected for use in your own searches from the Finder:

Outside-in Design Process

Study all the available resources and design your own schema. This is not a trivial exercise and requires competent database design skills as well as a robust knowledge of media internals.

Reverse the design process for your content workflow by documenting the user-facing application first. The metadata and database model are then governed by the requirements of the client player.

Work backwards so that each process in your workflow defines the metadata to be collected by the preceding process steps. This reveals parts of the design that require further refinement. Discovering omissions early creates a more reliable product. It needs less maintenance and exhibits fewer problems for the end-user.

Visualize the database table design in an Entity Relationship Diagram (ERD) to work out where tables are related to one another. Make the connections which define the joins in your SQL queries.

Object Oriented Programming (OOP) is recommended and leads to a collection of Object Modelling Diagrams derived from the ERD.

These should also describe the state changes your model undergoes and any transaction sequences for messaging and communications protocol design.

The Object Management Group has created the Unified Modelling Language (UML) as a standardized approach to building object models. At the time of writing the current version is 2.5.1 and an earlier version was adopted as ISO 19501. Although UML is declining in popularity, it still has some useful ideas to offer.

Develop your own modelling conventions based on previously published work. Document them and apply them consistently throughout the project.

Complete the metadata and database model design before writing any code to shorten the development time.

Metadata Categories

A program proceeds through the workflow and is eventually scheduled for delivery to the consumer. Metadata describes the production processes, the commercial and revenue generating aspects, scheduling and delivery.

Creating a single database table to satisfy the requirements for all of these would lead to an unwieldy design with hundreds of redundant columns. Many columns are only needed for part of the lifecycle.

Good business logic design separates each stage of production into its own table. Connect them together with RDBMS joins during a SQL query.

This is a spine and role record model. Attach role records to the spine using their unique record ID values as foreign keys only when you need them. Some of them might not exist until late in the creative or deployment process. Be careful not to duplicate any data in more than one role. 

The requirements are different for each part of a media item lifecycle. Each stage is represented by a different role:

International Standard Audio-visual Number

This works like the ISBN numbers used to register books or ISSN numbers used for describing newspapers and magazines in search systems. It facilitates the distribution of your content when it is released for viewing. It also uniquely identifies a program when searching the archives.

The ISAN numbers are defined by the ISO 15706 standard.

Register your media with an ISAN agent to get a unique ID number for each episode so that individual programs can be found. Do this during the pre-production stage of your project, long before any content is created.

Your registration facilitates the collection of your copyright licensing revenues and recording music permissions for the audio content of your production. This process is evolving and growing in popularity and if you cannot locate a registration agency for your own country, there is alternative coverage via the ISAN organization.

Publish your ISAN number by including it in the end-credits crawl.

Hints, Tips, Pitfalls & Gotchas

Metadata is sometimes described as a Cinderella technology. It is often under-appreciated, neglected and if it is dealt with at all, it is often as an afterthought. Neglecting the metadata gathering and quality assurance tasks ultimately leads to trouble sooner or later.

Here are a few suggestions of what can go wrong based on real life project examples:

• Last minute agonies – Metadata design is sometimes left until very late in the project. Likewise, documentation is often a last-minute exercise and is sometimes not done at all. This has serious consequences for your project and compromises the maintenance later on. Conceptually, the documentation is another kind of metadata, perhaps in a less structured format.
• Wasted effort – Only capture information that is genuinely required to avoid wasting effort and resources.
• Potential data loss – The media business exists as separate silos of expertise. The kind of metadata collected at each stage may only apply during that part of the lifecycle but it might affect things much later. The importance of the metadata being collected is not always apparent at the point of collection. It is often not captured because the operators at that point in the lifecycle have no use for it themselves.
• Collaboration – The on-location shooting team cannot know what the deployment team will need without their help and advice. Collaboration and information sharing is necessary to keep everyone informed. Publish a manifest of metadata that is needed with actions directed to the team that can collect it most efficiently.
• Too much detail – Avoid developing the modelling diagrams down to a fine grained level. They are helpful when looking at the bigger picture but can hinder development progress if they are overused.
• Not enough detail – If the metadata includes the film grain size (possibly based on the film stock and developing techniques used), an optimal scanning resolution and dynamic range can be chosen when digitizing old movie film. Smaller grain sizes give better resolution but are less sensitive to light. Adjust the brightness, contrast and dynamic range automatically to compensate.
• Keep all the documents – Interactive presentations or extras on a DVD product rely on behind the scenes material being gathered and logged during the shoot. Retain all the unused footage, paperwork and notes.
• Huge benefits for free – A natural history unit was able to create a six-part series using footage that had not been previously broadcast. This saved money by not needing to send out a film crew and was only possible because the footage was accurately logged and marked as unused.
• Silly mistakes – A news package editor substituted ‘nice looking’ background scenery from an earlier project. They did not know that the GPS metadata would be transcribed into a footer in the client player showing the location of the shot. In the middle of a high-profile news item, the strap-line suddenly described the location as being somewhere else entirely. The news outlet’s credibility was damaged because the editor was unaware of the downstream processes. Check and clean the metadata in the finished product and ensure everyone fully understands the pipeline where it will be routed.
• Proving prior art – If cases of copyright infringement, plagiarism or IP ownership crop up after the program has been aired, you need to be able to prove you did the due-diligence for including material from other sources or that your content was created first. Lack of evidence leads to lost legal cases.

There Can Be Only One

One of the fundamental rules in database design and software development in general is that any data value should only ever be defined just ONCE! There are many programming techniques that help you define constant values in code using a symbolic name. Databases are optimized to relate tables to each other so the same source data cells can be used in different contexts.

That singular definition of the value should be inherited everywhere it is needed. The moment you define any data values a second time, you have introduced a potential maintenance nightmare.

It is possible to build code that picks up these unique values from other domains and programming paradigms.

Automated Metadata Extraction

With the orders of magnitude increase in footage being acquired, it is no longer humanly possible to scan and log everything manually. Technology is evolving to fill the gap. Automation can extract such metadata as is embedded in the footage by modern cameras. A second pass analyzes the content for semantic meaning. A lot of useful information can be deduced automatically:

• Coding format, frame rate, image size, video standards (PAL/NTSC/Interlace/Progressive/pull-down).
• Facial recognition makes educated guesses about the identity of people in the frame.
• Voice recognition can back that up to reduce uncertainty.
• Scene detection can record useful bookmarks where cuts occur.
• Background music recognition can potentially identify rights and licensing issues.
• Where previously finished programs are being ingested, OCR techniques can extract burned in text, slates, title screens and end-credits.
• Copyright ownership and creation date can be extracted from the end credits. Sometimes that text is very small and moving quickly. A human eyeball can discriminate this and read the text but OCR may struggle.
• Sub-title streams can be extracted and stored as VTT files.
• Geographic location detection when there is no camera GPS data is possible with reverse image searching techniques.

Off-air recorded programs should contain embedded EPG metadata. This can be parsed with a data detector to infer various properties. The conventions of EPG data are generally followed very well although there are occasional uncertainties, formatting oddities, typographical and spelling errors and outright mistakes. I wrote a data detector a few years ago and extracted these properties in an off-air recording system:

Reconciling this against a database of known broadcast schedules (see BBC Genome) provided additional metadata.

A useful metadata feed into the asset management database is possible without human intervention. Manual inspection and moderation is important for quality assurance until it is proven to be accurate and reliable.

Document Centric Vs. Application Centric Design

If you conceive your workflow based on individual apps, you are limited by the file formats they support. Working out a streamlined path through the workflow becomes more difficult.

Turn the problem inside out by making the workflow document-centric rather than application-centric. Then find apps and tools that can operate on those document types. This simplifies the interchange between different parts of your workflow. The concept is illustrated here with a simple Rich Text Format document.

Problems are sometimes easier to solve when they are translated into a different domain.

Metadata & Data Models

Each metadata standard serves a different discipline. No single standard covers everything and some schemas are informal. Metadata needs to:

• Describe content.
• Identify assets.
• Locate assets within the storage.
• Index the assets.
• Describe technical processes operating on the content.
• Provide administrative support.
• Control access permissions.
• Manage version control.
• Log access and usage.
• Track changes.
• Manage long-term archives.
• Facilitate conservation.

These involve the use of many different standards. Here are some that you might find useful. Use their profiles to constrain the range of values:

These are starting points when designing your own data model to steer the workflow. More standards are described at the Digital Curation Centre (DCC) web site.

Relevant Standards

This is a list of the most relevant and helpful standards. Not all of these are for media applications. They may contain fragments of useful information to help you design a metadata model.

Metadata Is Vital To Your Success

Streaming services require significantly more metadata than traditional broadcast ever required. It becomes increasingly important as content is shared between organizations.

Previously, ‘job knowledge’ and scribbled notes were sufficient when programs were expected to air just once and never be seen again. With on-demand viewing, that is no longer the case. Viewers only find content if the metadata accurately describes it.

Content is now available indefinitely and the so called ‘long-tail-economy’ ensures that it can continue to generate valuable revenue long after it has been aired. This can only work to your benefit if you have maintained the necessary metadata to license and re-distribute your content. Repurposing to new platforms that do not yet exist is also only possible if the supporting metadata allows it.

Traditional broadcasting lacked a communications channel back to the service provider so monitoring analytical data was conducted on an estimated best-efforts basis. Modern streaming platforms can very accurately measure the viewing statistics and report back instantly. This can be fed back to the content store as metadata to identify content that is more popular.

These Appendix articles contain additional information you may find useful: