The offshore powerboats were fast. Our private Wi-Fi video transport network was faster.
IP video over Wi-Fi provided a solid new signal transport solution at a fraction of most wireless link costs. Wi-Fi was easy. The challenge was finding available devices to convert camera HDMI into IP for Wi-Fi transport, and decoders to convert IP back to HDMI for the production switcher.
Part 1 of this story introduced the uninvited co-producer for the big powerboat TV production, Murphy’s law. He spoke up well ahead of the show and apparently just stepped out for coffee. Making all brand new gear work together was the pre-show challenge and everything was working together. Audio and video checked perfect with the affiliates. Its noon! Our flashy new graphics come up on the return video from the TV stations. In less than 60 seconds, Technical Difficulty stills were on the TV video. It was a record nobody wanted. Who wants to sink at a boat race?
Careers flashed before eyes as we found and quickly fixed a loss of program audio. From that tenuous takeoff, quality quickly ascended to what we had hoped for. It was a muddy road to two great broadcast days of sunny new sparkle, amazing technical feats and racing excitement.
Cell service around the hilly Lake of the Ozarks region is not like the city. The cell system handles normal traffic for about 10,000 locals, but add 50,000 visiting boat race spectators with cell phone cameras and a spectacular accident on the race course, and the local cell tower can turn into digital putty. It happened at last year’s Lake Shootout race TV coverage. About a minute after the later-fatal accident occurred, bonded cellular camera-to-studio backhaul pictures began turning into blocks. It took a couple of hours while spectators presumably uploaded their videos for the local cell tower to recover bandwidth and stabilize.
It’s also difficult to direct camera operators at any event when the director and announcers are five seconds behind the outside world. With that kind of delay and high-Decibel powerboats, little is usually said on the PL. Camera operators are pre-directed and TD switching is more like watching TV with a remote control. Camera ops know not to make any movements they wouldn’t want to see on TV. Five seconds before the accident appeared on studio screens we heard distorted screaming in the PL. Four, three, two, one. Then we saw it. Not good. Not good at all.
The production continues to use station-provided Dejero and TVU Networks systems set with 10 second delays for program backhaul to affiliates via a healthy Internet connection. The delay creates plenty of time for buffering and 10 seconds doesn’t matter so long as production timing is offset for it. But, after the overloaded cell phone site experience and frustrating delay issues it was time to change the way we backhaul field cameras.
Its a science experiment
The Rocket M5 Base Station gathered signals from the Nanobeams at the camera positions and plugged into an in-studio switch for de-multiplexing and distribution to individual Matrox Maevex decoders. The decoders' HDMI outputs connected directly to the TriCaster Mini's HDMI inputs.
Determined to find a more reliable method to transport multiple camera signals across a couple of miles of water, we decided to try something entirely new and different that didn’t cost thousands of dollars per camera. A local fiber and cable expert friend recommended 5 GHz Wi-Fi gear from Ubiquiti. Think of Wi-Fi as a Cat5 cable, he said.
The Ubiquiti gear was powerful, consumer-level affordable and sold on Amazon. We used a Ubiquiti Networks Nanobeam MR NBE-M5-19 High Performance 19 dBi airMAX Bridge at each camera position. A Ubiquiti Rocket M5 5GHz airMAX BaseStation with AMO-5G10 2x2 MIMO Omni Antenna was mounted on a pole was at our receive site on the control room side of the lake. The complete multi-camera RF system cost about $700. All we needed was a switch, encoders and decoders.
Short of buying some of the new crop of field cameras with built-in streaming, we needed stand-alone encoders and decoders that fit our minimal budget. Visits to relevant NAB exhibits in search of such gear often resulted in similar responses from vendors. You're trying to use what? To do what? For how much? Clearly, we were pioneering.
Few affordable encoders and fewer stand-alone decoders were available. Most were rack-mounted, over powered and over our budget. The selection we could afford was between slim and none. Matrox Graphics offered a unique encoder and decoder product package called the Maevex Series H.264 Encoders and Decoders that looked like it would work within budget. We couldn’t find anyone who had actually used Maevex for a live multi-camera broadcast with a Wi-Fi system, but our fiber expert friend kept reminding us to just think of Wi-Fi as Cat 5.
"Think of Wi-Fi as Cat 5" said Jim Petropoulos with Co-Mo Connect.
One Matrox Maevex encoder/decoder pair was ordered to test with Wi-Fi. It was set up in the radio station office.
At one one of the hall, HDMI from a GoPro was plugged into the encoder in. The encoder IP out plugged into the Ubiquiti Nanobeam Cat 5 connection. At the other end of the hall, the Rocket receiver plugged into the same switch as the decoder and later decoders.
A video screen connected to the decoder HDMI out showed a perfect picture, no dropped frames and minimal delay. Bingo.
The receiver test was setup on a deck next to the studio. Laptops monitored the Maevex encoder and decoder and the Ubiquiti Rocket spectrum analyzer displays.
We packed up and took the Nanobeam and encoder to the farthest camera distance on the race course, approximately two miles, and set up a temporary receive site near the race studio with the Rocket, a switch, a decoder and a monitor. Bingo again. More Maevex and Nanobeams were ordered for the other cameras.
The systems were delivered when the area was under deluge by prolonged record rainfall. There wasn’t enough rain-free time to check simultaneous signals from all the actual camera locations at the event receive site until two days before the broadcast. Excitement and adrenaline were racing with sunshine and high expectations. In theory it should work.
A sensitive signal strength meter (left) is built into the Nanobeam for aiming. There's not much else to do than watch it work.
Skies eventually cleared and luck found us. Cameras and Wi-Fi systems were setup on-site in relatively short order and nearly everything was working flawlessly. There was ample signal strength and bandwidth from each site.
The Nanobeams and Rocket airMAX BaseStation have built-in signal strength LEDs for aiming and polarization adjustments. They provided all the necessary scientific and objective set up tools and then became transparent.
A powerful AirView spectrum analyzer comes bundled with the Rocket.
The Rocket BaseStation has a built in 5GHz band spectrum analyzer observable with included AirView software. It finds and identifies other 5GHz devices and tracks open frequencies. For a short while, you can watch the system negotiate and automatically setup the best RF choices and connections.
With some expert hand-holding from Matrox, we found the firmware on the encoders and decoders was one version behind the latest control software. Once the versions were synchronized, everyone exhaled and commenced to be amazed until it was all torn down following the Sunday show.
Camera backhaul was rock-solid for the duration, except when an event volunteer unplugged the receiver’s AC power extension cord before airtime. No harm, no foul. Luck was back on our side.
The Maevex decoders were stood on their sides to stay cool.
The Matrox Maevex encoders and decoders were virtual appliances. The Maevex control screen displayed current data rates and offered many control handles including dynamic or static IP addresses.
During the production, the Maevex control screen displayed relatively stable bandwidths of 10-20 MHz from each Maevex encoder over the Wi-Fi linked private network.
Once set, the Maevex units were as transparent and stable as the Wi-Fi with the real-time performance of both displayed on laptops. The new NewTek TriCaster Mini HD 4i liked the decoded HDMI video.
One never knows for certain until one sees video for oneself, and there were more than a few new variables saying hello for the first time between the lens and Program out.
Digital artifacts or dropped frames broadcast during our two, three-hour live broadcasts were rare. Camera-to-switcher delay was a fraction of a second. It was truly amazing and fulfilled our requirements. We could afford it, and it was close enough to real time to get bring the camera operators and studio back in sync.
The camera on an observation tower next to the studio was fiber-linked to the TriCaster Mini.
Our local fiber friend donated several hundred feet of fiber to backhaul another camera from a nearby tower platform to the studio. It was about a 150’ run, some flown, the rest on the ground.
An Advanced Fiber MC2-K-P-2-D fiber TX and RX system was an incredibly simple. Its output was SDI, automatically converted to HDMI in an Ensemble Designs NXT-410 Clean HDMI Router. The NXT 410 HDMI out was connected to the fourth TriCaster HDMI input, shared with several lesser used NXT 410 sources such as the studio camera. The fiber system was an out-of-the-box, rock-solid solution, but it couldn't get through this production without a story.
The tower camera operator preparing for the second day couldn’t see the green ‘on’ light on the fiber TX. In fact, the sun was so bright the green light was invisible. He thought he lost power and unplugged his camera, waiting for the green fiber TX light to indicate power.
Back in the studio the green light was lit on the RX, indicating it was on and connected to the working TX, but there was no SDI output. It was time to visit the tower. With some shade the green light on the tower TX appeared. The camera was switched back on and the panic switch turned off.
After overcoming pre-production obstacles large and small, some associates were still feeling the bad vibes that seemed attached to the production. One was a local photographer with the best drone video system in town, a new tricked-out DJI Inspire 1.
His job was to get some footage of a big meet-the-drivers street party the night before the race. He set his GPS to fly at 100’ altitude about 0.4 miles up the street, turn around and fly home. When he flew to the far end of his route, he lost his drone. Return-to-home didn’t work. It was dark. He thoroughly searched where he thought it might have crashed or gone and found nothing.
Early the next morning, he brought a buddy who owned a DJI Phantom. The Phantom was programmed with a similar flight path to look for the Inspire. Guess what. When the Phantom got to where it was supposed to turn around it disappeared, just like the Inspire. A hike to the far end of the street revealed two drones stuck high in the branches of a tall oak tree, apparently about 110’ tall. A tree-service bucket truck was called in to retrieve the drones. No more than egos were appreciatively damaged and video from the drone aired as planned.
20th Century square
By sign-off of the broadcast on day two, the crew was as elated and proud as they were exhausted. We thought we won the battles and the war. Little did we know that our high-five victory parade was about to be rained on. While tearing down and packing up, we began hearing some complaints. Several cameras were sponsored and the logos and graphics were cut in half. Camera sponsors were unhappy. What?
Off-air pictures were perfect. The programs were leaving broadcast transmitters okay.
All graphics were prepared within the safe limit boundries NewTek’s Live Text provides and everything looked fine to everyone who watched over-the-air. It was a good broadcast. GMs and managers of the TV affiliate stations complemented us. Recordings of the off-air broadcasts and screenshots confirmed 16:9 transmission.
It turned out that not all cable systems and satellites passed along our affiliate stations’ full HD signal. Some, including local provider Charter Cable, converted our affiliate’s 16:9 transmission to 4:3 center-cut SD. Bye-bye corner logos. Most sponsors were good sports and paid anyway because the money went to charity. But really, what an embarrassment.
Some cable and satellite systems stole our side panels and resolution. Note to MVPDs. Check your calendars. Its half past 2015.
A modern broadcast engineer’s story can no longer end with “It was leaving here okay.” Perfectly good ATSC DTV signals pass through myriad devices and preference settings beyond the control of broadcast engineers before most viewers without antennas can see pictures.
Who in the MVPD the chain knows who is deciding what or if anyone is thinking at all? Anyone? Its one of many digital challenges for early 21st Century TV broadcasters.
It was a lot easier back in yesteryear to tell complaining viewers "Its leaving here okay. You need to call a TV repairman" and be done with it.
Systems are supposed to be transparent. We thought the 20th century was over. We were wrong and hope the 4:3 incident was Murphy and his bad vibes’ way of saying so long. It was an appropriate “Gotcha” kink-in-the-pants to mark to the end of a high-speed roller-coaster learning experience that was some of the most extreme television fun of my career.
You might also like...
Planning for any kind of live TV broadcasting starts with a ‘what-if?’ list. What if the power source fails? What if a key production person gets sick or hurt? What if broadband internet access becomes unstable? What are the chances for…
Most live remote outside broadcasts are thoroughly planned by producers and directors who are often too busy to consider potential equipment problems. Technology is an engineering responsibility. Engineers must be ready for any circumstances that threaten to take the show…
Live TV production may not be the best fit for perfectionists who can’t recognize ‘good enough’ and move on. Live TV has no patience, no second chances and can never be late. Every live shot is a first impression.
NDI (Network Device Interface) is a free protocol for Video over IP, developed by NewTek. The key word is “free.”
For the past year an international group of technology companies, funded by the European Union (EU), has been looking into the use of 5G technology to streamline live and studio production in the hopes of distributing more content to (and…