ripeflow
← Field notes

What breaks in a live stream, and which of it you can engineer around

6 minute read

Live videoReliability

Every live stream we have ever run has had something go wrong in it. That is not pessimism, it is the base rate. A live production is a chain of maybe fifteen components, most of them consumer grade, several of them held together by a cable somebody has walked over four hundred times, and all of it has to work simultaneously for two hours with an audience watching.

The useful question is which of those failures the audience sees. Here is the list in the order we worry about it, which is not the order most people worry about it.

The operator

The most common failure in a live stream is a person. Wrong scene at the wrong moment. A microphone left muted through the opening remarks. A laptop lid closed during the break. Somebody plugging a phone charger into the wrong strip.

Nobody sells you a fix for this because there is nothing to sell. What actually helps is unglamorous: a written run sheet with the cues in order, one person whose only job is the stream, a rehearsal at the real start time with the real lights on, and a program monitor showing what is going out rather than what the operator thinks is going out. Most of the disasters we have been called in to review were a person doing a reasonable thing with bad information about what was live.

The connection is the one that ends the stream rather than degrading it, and it is the one you have least control over. A single business line drops. A venue's guest wifi has forty people on it by the second speaker. A cellular modem that tested fine at ten in the morning is sharing a tower with a stadium by seven.

Here is the part we would rather say early than have you find out later: a stream cannot be more reliable than the connection under it. No mixer, no encoder, no cloud service fixes a dead link. What you can do is stop depending on one, which means bonding several connections of different kinds so that the failure of any one is a change in available bandwidth rather than the end of the broadcast. That is a real fix and it is the only real fix. It is also why the connection comes up in every live video conversation we have.

Under test, with a link killed mid stream, LiveboxMix held 30 frames a second with zero disconnects and recovered in 2 seconds. Read the condition carefully: that is a link killed while another one was available. It is a measurement of what the mixer does when the redundancy you paid for is called on. It is not a claim that anything survives losing every connection at once, because nothing does.

The cameras

Cameras rarely die. What they do is change their minds. Autofocus hunts when somebody walks between the lens and the subject. Auto exposure ramps when a door opens and daylight comes in. Auto white balance shifts the skin tones halfway through a keynote. A viewer reads all three as bad production, and every one of them is a setting.

Lock focus, lock exposure, lock white balance, and set them at the real light level with the real people in the room. It takes twenty minutes and removes more visible problems than any equipment purchase.

The actual camera failure, when it happens, is a power supply or a loose connector rather than the body. Which is the next item.

The cables

The cable is the most likely component to fail and the cheapest to make redundant, so it is strange how rarely anybody does. HDMI has no lock and works its way out of a socket over a couple of hours of vibration. SDI locks, which is most of why broadcast uses it. The run that fails is the one crossing the doorway, every time, and it fails during the part nobody rehearsed.

Tape the runs, use locking connectors where the format has them, carry a spare of every cable in the chain, and label both ends. None of that is engineering. All of it is the difference between a ten second gap and a dead show.

The encoder

An encoder fails in two ways. It overheats, usually because it is a small fanless box that somebody put in a road case with the lid closed, and it throttles until the frame rate falls apart. Or it is a laptop, and a laptop is a general purpose computer with opinions: an update dialog, a sleep timer, a notification with somebody's personal messages on it appearing over the program feed.

Give it air. If it is a laptop, take it off the network it updates from, turn off sleep, turn off notifications, and use it for nothing else that day.

The power

Power failures at a venue are rarer than the other five and worse when they happen. The proportionate answer is a small uninterruptible supply on the encoder, the switch and the router, which are the three things whose reboot cycle is long enough to end a stream. Cameras and lights coming back in ten seconds is a visible glitch. An encoder coming back in ninety seconds with no stream key loaded is over.

Do not try to put the whole rig on battery. Put the brain on battery and let the limbs blink.

What survives what

Sort those six by whether a well built mixer can absorb the failure.

A camera dropping out, a cable pulling loose, one source going black: the program feed continues on another source and the audience sees a cut. That is what a multi source mixer is for and it is the case that works.

One uplink of several failing: the stream continues at lower available bandwidth and recovers. That is the measurement above.

The encoder overheating, the venue losing power to the rack, every connection going down at once, the operator switching to the wrong scene: the audience sees all of it. No amount of engineering downstream fixes an upstream that stopped.

The answer

Roughly half of what breaks in a live stream is absorbable by a mixer that expects failure, and you should have one. The other half is process: locked camera settings, taped cables, a run sheet, a rehearsal, one person who is not also giving a speech, and a battery under the encoder.

And the thing under all of it is the connection. If your venue has one line into it, that line is your single point of failure and everything above is decoration. Fix that first, then buy the mixer.

Tell us what is taking your team's time.

Thirty minutes with an engineer, free. You leave with a written page on what to automate first, and a written quote if you want us to build it.