When more than 100,000 people swarm a remote location loaded with photo and video opportunities and a cellphone infrastructure designed for a small fraction of that number, some cellular data customers are going to become frustrated.
In addition to being a member of The Broadcast Bridge editorial team, I'm also the engineer in charge of live broadcast TV coverage of several world-class offshore powerboat races every year. Typical race venues are often the permanent home to a few thousand people, a few line-of-sight cell towers and mostly water. However, powerboat races often attract more than 100,000 spectators and many of these visitors expect to be able to upload photos and videos during the event.
One result is that these live TV powerboat racing productions encounter cellular phone and data service problems race days. The normally adequate service always becomes unstable when the fans show up and cell service can slow to a crawl or even crash if something spectacular occurs and everyone wants to post images to their social networks.
Wireless data Speedtests at the same on-site studio location the day before the event on left, during the event on right.
Service dives when the crowds arrive
The most recent powerboat race at Missouri’s Lake of the Ozarks, LakeRace 2017, was no exception. It once again demonstrated what happens to local cell service when car-and boat-loads of people arrive ready to have fun and take pictures.
While the event was in progress, uploading a single cellphone photo to Facebook took more than 5 minutes. Even voice phone calls required extra patience. The local cable company's neighborhood internet service node was similarly choked.
At big breaking unscheduled news events, unsuspecting cell networks and wired ISPs can quickly become so overloaded the systems cannot reliably pass broadcast-quality video streams.
The larger the unscheduled event, the more likely the local public data networks will choke and stumble. Yet, adequate service at large newsworthy events, typically scheduled well in advance, do not have such problems. Why?
Rapid Deployment Units
Cellular companies have mobile units designed to provide temporary cell service for emergencies and disasters. Such units are known by several names: Rapid Deployment Unit (RDU), Cell on Wheels (COW), or Cell on Truck (COT). This story will refer to these units as RDUs.
When not busy at emergencies and disasters, RDUs can also supplement local cell service at major sports events and large national conventions. RDUs are regularly seen parked near the Las Vegas NAB convention center to support the 100,000+ attendees. RDUs pop-up at many events where event organizers expect high-capacity crowds.
After watching local cellphone data service predictably descend to a few kB/sec at the umpteenth powerboat race, it seemed like a good idea to reach out to the big cell ISPs with some questions about how broadcasters might help prevent overloading local cell services.
An AT&T RDU in use at a local disaster site.
Initially, I hoped to learn what factors might prompt a cellular ISP to dispatch a Rapid Deployment Unit to support an unusually large scheduled event? Is there a phone number an organization or local government can call to request temporary additional service if the cellular network might be overloaded? How much lead time is necessary to have the additional equipment installed and configured?
No easy answers
The response to these questions revealed no simple answer, or no answers at all. Editorial requests were emailed to media contacts at AT&T, Verizon and Sprint. Only AT&T responded.
AT&T media spokesperson Meghan Danks explained that the process is too unique to handle at the national level. There is no single point of RDU contact at AT&T. There is no 1-800 number to call or special email address. The process is always initiated locally.
Danks said “We use a number of factors to determine deployment of resources to support temporary events. These include input from local network teams, assessments of our area network and requests from event organizers. Organizations looking for support from us can contact our local community teams in their area. The person to contact depends on the market.” She emphasized that event organizers need to initiate the contact.
Our live TV powerboat racing events are a notorious cell service ground rod. Given our previous experience, event organizers plan to contact local cellular ISPs in the market this week to request RDU support. Standby for the results.
A real-world solution
The day and evening before this year's LakeRace 2017, my team streamed test video and audio to YouTube. We monitored the return link in the studio and on cell phones where the program feed remained clear and stable.
Internet service the next morning was a different story altogether. The YouTube feed was scheduled to begin streaming at 10 a.m. with live broadcasting starting at 11 a.m. About 7 a.m. the test YouTube stream began to stutter and fail, and it appeared to be an issue at our end. It was time to reboot everything and then reboot it all again.
Meantime, also in the on-site studio, creating live content for the live stream and imminent live TV broadcast window was the main focus of the staff. The program was being uplinked to Galaxy 17, via the satellite truck of the lead affiliate, Gray Television stations KY3/KSPR in Springfield, MO. Other affiliates tuned to the same Galaxy 17 transponder for their live show feed. Live closed-captioning data was inserted in the satellite truck. The live captioning was created by a person off-site, connected by two dedicated hardwired phone lines.
Viewers enjoying over-the-air broadcasts of the event were never affected by the local ISP meltdown.
The sat feed was reliable and stable as expected. It was tested at affiliates Friday afternoon and was up for test at 10 am on race day. Unfortunately, uplinking the same program content to YouTube from on-site studio was struggling.
By 9 am, local wired internet service from the local cable company had turned to mush. Plan B was to stream to YouTube via the cell system with a personal hotspot. The YouTube stream was created in a Matrox Monarch, and it needed 5MB of upload bandwidth. A Speedtest the day before indicated 7mB upload speed on AT&T cellular.
The need for a Plan C became obvious after rebooting the cable modem, the Monarch, or streaming directly from the NewTek TriCaster made no difference. That’s when running a Speedtest on the local wired internet and available wireless internet revealed both to be somewhere between Morse code and a 28kB dial-up modem. The YouTube stream had gone from pretty pictures to herky-jerky to flat-lined. Plan C was to ask the KSPR satellite truck operators if they had any ideas.
The Vibroplex Bug was a huge speed tweak for ham radio Morse code operators, and it epitomized everything analog.
Fortunately, KY3 meteorologist Kevin Lightly was on the scene and he immediately thought of a Plan C that resuscitated the lifeless YouTube upload stream. His idea was to take the live satellite feed content as it appeared on the router in his Springfield studio Weather Center, and then upload the stream to YouTube using a Weather Center computer. Uploading to YouTube wasn’t unusual for weather photos and videos, and all he needed was the Live LakeRace steam YouTube user name and password. We couldn’t get it to him fast enough.
About 10:50 a.m. the KY3 Weather Center logged on as LakeRace and the highly-publicized 10 a.m. YouTube stream suddenly reappeared on the internet, rock-solid. The YouTube connection problem was resolved, the “the show goes on” promise was fulfilled and all the sponsor’s ads and logos appeared on screens worldwide along with exciting multi-camera live race coverage, as promised. Sweet.
Broadcast engineering theorems confirmed:
1) The internet is wonderful—until it stops working.
2) A satellite remains the most stable, portable and private non line-of-sight live content backhaul technology for TV broadcasters.
3) Phone companies want to help, but they need to be brought into the scheduled event loop well ahead of time.
4) The best TV friends are those who step up and help when you’re live and out of ideas.
A recording of the LakeRace 2017 live stream continues to live on YouTube, including the first hour of hiccups. I'm not the internet guy. The program feed leaving the switcher was okay the whole time.
Related Editorial Content
3-part Series: Live powerboat racing production. How digital delay became an invisible advantage. Most sanctioned offshore power boat races are held in the ocean, generally at some distance from most landlocked spectators. The multi-million dollar offshore powerboat racing circuit has…
The Lake of the Ozarks is at the center of powerboat racing action every year, on two separate weekends, each covered with two consecutive days of live outside broadcasting. One event is the Lake Race Invitational; the other is the…
You never know who is watching, OR Making money by giving.