This article is part of our growing series on Standards. There is an overview of all 26 articles in Part 1 -  An Introduction To Standards.

Buying a vendor asset management solution is a pretty safe and common solution but it doesn’t mean you don’t need to understand or put effort into molding the system to your requirements. Even the most flexible products will require extra work to customize them to your exact needs. Gaining a better fundamental understanding of how the system works can help you achieve more effective systems.

For those seeking to build their own systems, this article advocates challenging convention to see what benefits some more lateral approaches might bring.

There are useful metadata standards for the data model and components used to build workflows but the controlling framework is not easily standardized. Use open-standards if possible and proprietary standards when necessary.

File Based Workflows

Processing media files is quite straightforward. Deploying a Content Management System (CMS) to keep track of the media solves problems but it also introduces some complexity. The CMS has to perform several tasks:

• Creation and maintenance of metadata.
• Ingest and tagging of new content.
• Physical storage management with redundancy in case of loss.
• Library and archive curation.
• Search and retrieval.
• Automated background processes to create multiple alternatively formatted versions for deployment.
• Workflow orchestration, scheduling and control.

We must be diligent (and possibly somewhat draconian) when it comes to adding metadata at the point of ingest for this to work reliably.

Metadata & Data Models

Each metadata standard serves a different discipline. No single standard covers everything and some schemas are informal. Metadata needs to:

• Describe content.
• Identify assets.
• Locate assets within the storage.
• Index the assets.
• Describe technical processes operating on the content.
• Provide administrative support.
• Control access permissions.
• Manage version control.
• Log access and usage.
• Track changes.
• Manage long-term archives.
• Facilitate preservation and conservation.

These involve the use of many different standards. Here are some that you might find useful. Use their profiles to constrain the range of values:

These are starting points when designing your own data model to steer the workflow. More standards are described at the Digital Curation Centre (DCC) web site.

Spine & Role Pattern

Repurpose the RDBMS spine and role pattern to invoke optional schemas as needed to support your workflow without having unreasonably wide database tables.

The spine record describes where the media is located but it is not swamped by irrelevant data. Additional properties are joined using relational links only when they are needed. The camera and lens details facilitate visual effects work. The shooting location helps when searching for related clips. Collect relevant properties together into a single role only when they are used together.

Research the use of Object-Oriented Patterns to discover some additional robust and well-conceived designs to repurpose for your own needs. These are well documented with much helpful guidance.

Learn From Other Disciplines - Semantic Searching

Scientific Visualisation includes graphical animations of weather patterns and other huge data sources that have too much information for a human to grasp. Animation provides 'time' as an additional axis to display dynamic content. Translating a problem into a different domain reveals an alternative analytical solution. This is an elegant and often simple approach which is fascinating and now informs all of my engineering designs.

The IN-SPIRE system was developed by Pacific Northwest Labs in the mid 1990s. It revolutionized the analysis of textual information and presents the deductions graphically. It was the precursor to what we now think of as Artificial Intelligence.

Moving things to a different domain allows us to deploy radically different algorithms. Arrange database search output onto an X-Y grid using two results criteria for the axes. Then apply spatial-geometric techniques to find related items.

IN-SPIRE uses this technique to take an existing document and parse it to create filters that locate a collection of similar documents. This goes way beyond simple keyword searching. I think we could use AI techniques like this to implement video searching tools that are hugely powerful.

The IN-SPIRE system encourages us to develop more sophisticated processing of our search results to analyze textual information embedded inside our media.

Document-centric Vs. Application-centric Design

If you conceive your workflow based on individual apps, you are limited by the file formats they support. Working out a streamlined path through the workflow becomes more difficult.

Turn the problem inside out by making the workflow document-centric rather than application-centric. Then find apps and tools that can operate on those document types. This simplifies the interchange between different parts of your workflow. The concept is illustrated here with a simple text document.

Controlling Tools & Software Apps With Scripts

When the media repository contains many thousands of items, a manual solution is no longer practical. Automation is best implemented as script-driven command-line processes.

Many applications are built on a foundation library that supports a screen-based interaction and a script-driven command-line. In fact, there may be many levels of scripting (like an onion skin). Carefully studying the internals of your applications reveals this functionality:

These apps support command-line-scripted control:

Bridging Control Between Different Environments

If scripting is not directly available at the command-line, there may be a way to bridge across and call it indirectly. Learning how to do this is another example of moving a problem to a different domain. Remote execution or triggering via service calls facilitates cross-platform integration.

In the 1990's the Showtime network automatically created interstitial stings and trails by steering an AppleScript-driven Media 100 video editor from an Excel Spreadsheet.

This opens up your workflow processing choices to utilise apps that could only ever be run manually on a Windows or macOS platform. Now they are accessible from your main workflow orchestration running on Linux.

Workflow Orchestration, Scheduling & Control

Build the workflow from a collection of processes running in a distributed fashion across multiple machines. Those processes implement these activities:

• Movers.
• Converters.
• Renderers.
• Filters.
• Front-ends to hardware.

Avoid hard-coding the behaviors and use a data-driven approach and present the control surface in a web-based user interface. A PHP scripted back-end creates data files to initiate tasks and forward the necessary controls. Command-line driven tools wrapped in shell scripts then acquire the parameters to steer those tasks.

Here are some of the high-level constructs we use to build these processes:

Micro-service Architectures

Construct micro-service architectures from small component modules which are easily virtualized on cloud-based systems. They follow Object Oriented principles and only communicate via a well-defined API with their inner workings concealed. Data is often passed in a JSON structure but this is not mandatory.

The API for a micro-service can be implemented in a variety of ways:

A micro-service responder could be built with inexpensive devices such as a Raspberry Pi single board computer. These can access real-world hardware using Internet of Things (IoT) principles. This makes remotely operated steerable cameras, lights, auto-scripters and other hardware accessible to your workflow.

Conclusion

Think differently by moving problems to a different domain so they can be implemented with less work. Less code equals easier maintenance and therefore fewer things to go wrong. Apply radically different domain-specific algorithmic techniques. Micro-services help to reduce complexity. Bridging techniques allow you to exploit previously unreachable features. Searches can be made much smarter with quite simple extensions. Build your workflow with small re-usable functional blocks.

These Appendix articles contain additional information you may find useful: