Passion, Planning, and Duct Tape

Live TV field production of one-off sports events is an open invitation for surprises of all kinds. The show must go on and it’s the worst possible time for drama. It’s a good fit for passionate engineers who thrive on winning and enjoy alternating bursts of despair and adrenaline. The more adrenaline a field sports production generates, the greater the thrill. Like a magic act, the trick is keeping production secrets hidden from view.

Engineering a live TV field production is a four-step process. Step one is the idea, or the "treatment." Step two is to identify all the necessary elements to turn the idea into television, and conduct a technical on-site survey to access, identify, plan and test for potential problems. Test and verify early. Excuses will not be accepted later.

Step three occurs when the producer, director, crew and gear turn the location and event into a live television production stage and the show is backhauled and broadcast. Step four is to pack up and return the physical condition of the gear and production spaces to as if it never happened.

Every location provides its unique challenges. All have good, bad and ugly features. I recently engineered and directed live 3-hour broadcast coverage of the Offshore Powerboat Association World Championships in Englewood FL. It was backhauled by satellite and broadcast live by ABC affiliate WWSB in Sarasota. FL. The production had plenty of good, and we found effective ways to conceal the bad and ugly.

Basic Live TV

Full throttle action began moments before the broadcast went live. The event was front-loaded with its fastest powerboats first to grab viewers.

Full throttle action began moments before the broadcast went live. The event was front-loaded with its fastest powerboats first to grab viewers.

You never get a second chance to make a first impression. Viewers will decide to tune in or tune out during the first couple of minutes. Fortunately, the technology was in place for a great race on live TV and the race needed to be directed for live TV.

TV broadcasting revolves around a clock where program-on and program-off times are rock-solid. The live broadcast must hit the ground running the moment it becomes live. Action needs to pause or be paused during commercials. That's what replays are for. If the organizers know approximate commercial break times, they can help the show flow. The best shows come together and are more fun when everyone is on the same page.

Most races usually build up to higher speed classes, but we wanted to lead the broadcast with the highest speeds. The Offshore Powerboat Association race organizers agreed and TV viewers were instantly gripped by non-stop high-speed action, from the open through the credit roll. Action pauses were few and short. Racing officially ended about five minutes before the TV show's scheduled end. Was it due to planning or luck? Yes!

Site Surveys and Planning

A site survey should identify, fix or find work-arounds for difficult problems before the trucks and crews ever arrive. Among other things, a site survey must verifying parking for production and/or satellite vehicles, test camera angles, consider sunlight issues and test microwave paths.

The survey includes locating the electrical panels, confirming adequate HVAC, and fully testing the Internet connection with a video stream containing motion and audio lip-sync cues. It is also the time to identify risks and develop fallback Plans B, C, D and E. Our plan E was streaming iPhone video from the balcony with live ad-lib commercials. Some viewers might not even notice a difference, but everyone would recoil at the dreaded trouble graphic for Plan F.

Plan F graphic. Plans B, C, D and E are to avoid Plan F.

Plan F graphic. Plans B, C, D and E are to avoid Plan F.

The crew and gear arrived at the temporary Englewood studio venue 6 days early. The plan was to be completely ready a couple of days before the event and relax up to the show. That’s always the plan but it never quite works out that way.

On location, cell service was spotty and so was local Internet service. We’d been warned about local Internet service and predictions were right. We ordered a static IP business connection from the local cable company. It worked, but it wouldn’t pass our streaming video to the TV station for a satellite backup, or to Ustream. We soon discovered we couldn't stream outside our local node.

It was beyond a simple port forwarding problem. Two days of visits from three cable technicians talking and trying couldn’t fix it.

Before and after shots of the rental condo TV studio with a perfect view of the race course.

Before and after shots of the rental condo TV studio with a perfect view of the race course.

Cable techs are trained to pull cables and set up dynamic residential IPs. The techs told us our problem was fairly common in their system and unresolved. Engineers at the ISP’s offices claimed the problem wasn’t with their system. We thought we were in real trouble. Then mysteriously, the static IP and port forwarding problems fixed themselves overnight the day before the broadcast.

Signal paths

In addition to ISP issues, the show relied on a couple of iffy microwave signal paths. During the site survey, the operator of the 70’ bucket truck hired to raise our microwave antennas wouldn’t ride in the bucket above about 40’. The test signal was okay, but not strong enough for reliable broadcasting. We hoped the extra 30’ would fix it, but hedged our bet on the 5.8 GHz Wi-Fi backhaul with two 900 MHz radio systems and 16 dB gain antennas.

This 16 dB gain 900 MHz Wi-Fi system was Plan B.

This 16 dB gain 900 MHz Wi-Fi system was Plan B.

The 900 MHz band has approximately 26 MHz bandwidth, about enough for a couple of cameras if there isn’t any interference. The high-gain antennas would burn through palm tree leaves and limit interference. The 5.8 GHz band has 555 MHz more available bandwidth, and 23 non-overlapping channels sufficient for video transport, but a 5.8 GHz signal won’t burn through much that you can’t visually see through.

It turned out that at the full 70’ boom height, our 5.8 GHz Wi-Fi signals were near saturation. The 900 MHz systems were put away for future use, so we could take advantage of the broader 5.8 GHz bandwidth and use our highest antenna point for both the source and a repeater. Over the course of three days of racing, streaming and broadcasting, the Wi-Fi radios and Matrox Maevex encoders and decoders didn’t miss a frame. Ubiquiti statistics indicated we lost a grand total of one packet, on one night.

Another pleasant surprise was learning that the Matrox Maevex units not only encoded and decoded an IP stream, but could also transcode. The NewTek TriCaster and Panasonic cameras were set to 720p, as was ABC affiliate WWSB. With a couple of changed settings, the Maevex system readily transcoded 1080p Apple iPhone HDMI output to 720p without missing a frame. The decoded iPhone contained recorded video from an earlier event parade and was ingested as a live HDMI clip.

A 70’ boom raised the 5.8 GHz Wi-Fi radios above signal obstructions.

A 70’ boom raised the 5.8 GHz Wi-Fi radios above signal obstructions.

We had our own side-race going in the studio. Two Panasonic AG-AC160PJ video cameras were setup on the race course. One was on a balcony outside the temporary studio. It connected to the NewTek TriCaster production switcher with a copper HDMI cable.

The other was more than a mile up the beach, where the camera and encoder were atop a 20’ scaffold, near the 70’ bucket truck. The Wi-Fi radio was mounted in the bucket. A lengthy Cat 5 cable from the encoder was buried in the sand from the scaffold to the truck, and connected the Wi-Fi radio in the bucket.

In the studio, a Ubiquiti radio output plugged into a 1 Gb switch and provided two streams. One Maevex decoder was set to receive the scaffold camera stream and provided HDMI out to the TriCaster. A second Maevex selected and decoded the encoded stream from a GoPro camera on a static shot in the pits via another Ubiquiti radio. That decoder also got its signal from the 1Gb switch and its HDMI output plugged into the TriCaster.

 The private LAN was a virtual production router.

The private LAN was a virtual production router.

Side-by-side, shot-to- shot, any picture quality difference or anomalies between the wired and the matching Wi-Fi camera were negligible. Overall delay of the Wi-Fi, encoding and decoding was also negligible. We never had an opportunity to compare matching copper and Wi-Fi camera shots prior to the live event. The similar high quality signals turned out to be one of the most pleasing technical surprises of the production.

Laptop CG

The TriCaster is a wonderful device, as it comes with LiveText, the CG program. bundled with it. The problem is you can’t run both simultaneously. You can however, buy a second copy of LiveText, load it on a laptop and make that device the pre-production and live CG creation edit and control point.

A Cat 5 connection to the LAN brings the laptop up on a TriCaster Net input. In our case, the operator brought in a brand new laptop, loaded with pre-produced graphics and custom templates. Surprise! It didn’t have a Cat 5 connector. Fortunately, the local Staples store had a USB-to-Cat 5 adaptor and it worked. Another bullet dodged.

Graphics were served from the kitchen counter.

Graphics were served from the kitchen counter.

The new laptop running LiveText was set to a hot downstream key on the TriCaster. The CG operator on the network controlled the downstream key on/off. This allowed the CG team to gather and feed real-time stats to the CG operator, who uniquely knew when status graphics were ready to key.

There was no particular need to coordinate with the TD. When stats were updated, they instantly appeared in a reserved area on the screen as the TD watched. Everyone was satisfied.

Solutions, S - XXXL

We used one AC160 camera to pre-record short interviews with boat drivers and owners on an internal SD card. The interviews were played back in real time on the AC160 and ingested in the TriCaster as live HDMI.

Interviews were recorded on a SD card and ingested as live HDMI.

Interviews were recorded on a SD card and ingested as live HDMI.

A fixed GoPro camera was located in the pits, aimed at a crane moving race boats from trailers to the water. The GoPro’s live output was HD, but 4:3. Zoom controls in a TriCaster M/E channel made it appear to be native 16:9. It’s the magic of TV and our little secret. Who would know? The pictures were beautiful.

There were the little impossible-to-plan-for surprises that made for some special fun. For example, the camera operator on the beach scaffold accidentally dropped his glasses over the side, half-way through the live production. Suddenly he had difficulty focusing. To overcome the problem, he set his lens focus to infinity, didn’t zoom in too tight and the show went on.

The author shows how a length of red duct tape strap helped keep production communications together.

The author shows how a length of red duct tape strap helped keep production communications together.

Perhaps the greatest challenge of the WWSB live broadcast came during the middle of it. While directing and switching, my headset mic kept dropping to my throat and the droop progressively worsened. Camera operators working in the high-noise environment of full-throttle racing engines couldn’t hear me. To be heard, I held a push to talk button with one hand and held the headset mic near my mouth with the other. It was impossible to switch. I was out of hands.

It was a gaffer tape production on a duct tape budget, I’m the engineer and this was an emergency. There wasn’t time to split hairs. A nearby roll of duct tape fixed the droopy mic using eye glasses for support. Communications resumed and we moved on.

Every 1280x720 frame of high speed action was frozen with a 1/2000 shutter. The shutter was turned off when the camera was on talent.

Every 1280x720 frame of high speed action was frozen with a 1/2000 shutter. The shutter was turned off when the camera was on talent.

Ups and downs

A 5th floor suite in the Englewood Beach WannaB Inn was the site of the studio and control room. The view was excellent but we soon became concerned about beach sand everywhere, including the tile floors in the room. We thought the sand might cause a problem in HDMI connectors. That didn’t happen, but a different unanticipated sand problem did occur. Sand jammed the elevator doors the morning of the broadcast, and parts weren’t available until the next day.

The broken elevator wasn’t a problem until teardown. Fortuntely, the WannaB Inn sent some healthy surfers to carry our gear down the stairs to a van. We packed up and everyone got some exercise on the stairs that day. Step four was complete and we all collectively exhaled. The production never varied from Plan A.

The day after the broadcast, the crew received a thank you email from WWSB VP and GM Jeffrey S. Benninghoff that started out with “All I have to say is Wow!!!!” He concluded “This was a first class production which was definitely network quality.” I expect the half-life of the glow from that compliment to be about a month. Maybe more. It was an exciting show, we were proud of it and the station VP/GM validated our efforts.

The bottom line for remote production is this: Don’t forget the duct tape, always have alternate plans ready and Good Luck. Problems are minimized and benefits peak when everyone involved understands what it takes to make a live TV production successful.

You might also like...

An Introduction To Network Observability

The more complex and intricate IP networks and cloud infrastructures become, the greater the potential for unwelcome dynamics in the system, and the greater the need for rich, reliable, real-time data about performance and error rates.

Essential Guide: Location Sound Recording

This Essential Guide examines the delicate and diverse art of capturing audio on location, across a range of different types of film and television production. A group of seasoned professionals discuss their art and the how it can dramatically elevate…

What Are The Long-Term Implications Of AI For Broadcast?

We’ve all witnessed its phenomenal growth recently. The question is: how do we manage the process of adopting and adjusting to AI in the broadcasting industry? This article is more about our approach than specific examples of AI integration;…

Next-Gen 5G Contribution: Part 2 - MEC & The Disruptive Potential Of 5G

The migration of the core network functionality of 5G to virtualized or cloud-native infrastructure opens up new capabilities like MEC which have the potential to disrupt current approaches to remote production contribution networks.

Designing IP Broadcast Systems: Addressing & Packet Delivery

How layer-3 and layer-2 addresses work together to deliver data link layer packets and frames across networks to improve efficiency and reduce congestion.