Building Plaudere: Syncing Live Content

The main idea behind Plaudere was to make it possible for several people to stream together, in real time, with very little delay. The website allows one person, the streamer, to create live content for an audience. They can use pre recorded materials and devices like a camera and microphone from a computer, tablet, or mobile phone. However, allowing many streamers to go live at the same time on any platform brings two key difficulties:

• There is always a delay from when a video or audio stream is created until the audience or other streamers receive it. This delay is affected by the device used to record, the internet speed (bandwidth), and how powerful the server is at sending the streams quickly.
• The more streamers connected, the more server power is needed. The server must mix the videos or audios in time, change formats from different browsers, and still send everything to the audience and other streamers as fast as possible.

For a typical video or audio call, the most important thing is to keep the delay very small, usually around 500 to 1000 milliseconds (half a second to a second). This allows people to react instantly to each other. In calls, exact timing is not important. However, for performances that need precise timing, like music, if the delay is more than 30 milliseconds, the performance will sound out of sync for both the musicians and the audience. This makes it impossible for music or other timed performances.

Because of these problems, controlling the delay and the server's power to send streams quickly is very hard. There are special desktop applications that try to achieve this. However, the greater the distance between streamers, the more delay there will be. This is true even if device delays and server power are perfectly managed, because the speed of data on the internet varies and takes more than 30 milliseconds for connections over long distances.

Plaudere’s Approach for Collaborative Streaming

The main idea for Plaudere was to allow easy real time streaming collaboration. To start, this means having two streamers performing together, using common devices like a computer or mobile phone, headphones, and a normal internet connection through a website. However, as mentioned, the delay from devices and the server’s power to send media parts make it impossible for them to hear each other perfectly and create a joint stream with instant feedback. This would need a lot of server power and special audio and video devices for the streamers.

Therefore, a simpler solution was found: the first streamer performs without hearing the second streamer. The first streamer leaves parts in their performance to be completed by the second streamer. This works well for a song sung by two singers with clear timing and visual guide if available. The second streamer can react to the first streamer because the second streamer can hear the first, but not the other way around. This reduces the problem of the first streamer not being able to react naturally. The second streamer knows exactly how the final mix will sound to the audience, and the audience will hear both streamers in sync, as if they were performing in the same room. Of course, this means the performance needs to be predictable. For jamming or improvising music, it is difficult for the second streamer to fill the parts left by the first if there is no visual guide available during the performance.

The limitation of this approach is that it is not suitable for all types of streams, like conference calls or live interviews. However, it is very useful for streamers who share the same timing during a planned performance. It creates the feeling for the audience that the performers are in the same physical space, creating an atmosphere of a live and synced collaboration between both streamers.

How it Works Technically

To make this work technically, the website must have very clear options to choose between a single stream streamer or a collaborative stream.

• For a single user stream on the current website, the streamer clicks an option to transmit. They choose the devices they want to use, like a camera and microphone, or select media content for the performance, such as a karaoke video or a background music track. They set a device delay to ensure the system can mix all media from devices and tracks without any issues, and then they start streaming to the audience. Behind the scenes, the streamer is creating small parts of media that the server receives and then sends to the audience, adapting to internet speed and delay limits.
• For a collaborative user stream on the current website, the first streamer first creates a background stream, using a device like a camera or microphone, and/or a video or audio track. This follows the same idea as the single user stream. The key difference is that the second streamer can add the first streamer’s stream as a video or audio background. By choosing their camera or microphone and setting the correct delay, both streams can be mixed into one single stream to be sent to the audience. Internally, both streams are created, but once the second streamer confirms their part, the first stream is redirected from the audience to the second streamer. Then, the mixed stream of both performers is sent to the audience, creating a seamless experience.

Challenges in Building This System

Creating a website that allows this kind of collaboration brings many challenges.

The first challenge is keeping track of the streaming state at every moment as the first and second streamers set up their collaboration. The website must always know if the first streamer is producing content, if the second is setting up the collaboration, if both streams are being mixed by the second streamer, and so on. In the current setup, there is a mix of theoretical states (for example, when a streamer is setting up their stream) and physical states (when second streamers are receiving a background stream from the first streamer). All these must be considered to define the current status and prevent issues like two “first streamers” or two “second streamers” performing at the same time.

Another challenge is how the audience receives the mixed stream. The system must always know if only the background stream from the first streamer is playing, or if the second streamer is also active, so that the redirection works correctly. However, if for any reason the second streamer stops sending media parts to the server, there is a waiting time before the server redirects the stream back to only the first streamer. This ensures the streaming continues without interruption.

In terms of mixing video and audio, using Canvas (a way to draw graphics on a webpage) and the Web Audio API (a tool for processing audio) with multiple nodes and destinations makes this possible. However, this creates complex situations that the system must handle, including showing error messages and notifications.

A Promising Future

Finally, the fact that with relatively simple equipment from the streamer, such as a basic website, a built in camera or microphone, and an average bandwidth, two streamers can collaborate online for an audience is a very useful approach. This is especially true for streamers who live far apart but still want to create and share live media production together. While this system can be improved a lot, as a first step, it shows that the future of Plaudere is full of promise, new challenges, solutions, and collaboration.

Joe Esteves