How virtual advertising is created in real-time for sporting events.
You cant even believe your own eyes anymore as virtual advertising enters real life
Found this technology interesting so I wanted to do a little deep dive into it and see if I could make any heads or tails of it.
It is commonly known that major sports clubs bring in large amounts of money by selling broadcasts right. In 2019 the FC Barcelona racked in €840.8 million from its combined commercial activity of which 35% or €298.1m come from licensing broadcast right. On the flip side, the broadcasters have to pay billions for all broadcast licenses and the situation has worsened for them with the introduction of alternative channels like youtube and others. This creates pressure for holders of broadcast licenses to the recipe as much money as possible. not until recently has there been a way to maximize the TV ads revenue.
With increased computational power it became possible to digitally replace all stadium side LCD panels. This enabled the broadcaster to segment the advertising revenue per country to which it streams the events.
There is a lot of discussions online on how this is made as the information is proprietary and there is not that much information about it. It looks like there are several different approaches but there are two that stuck out as most.
The first and least technical is where the display’s refresh rate is linked to a camera shutter. It was developed by Swiss company Appario Global Solutions (AGS) and it calls it Dynamic Content Multiplication (DMC). If you look at the display as normal, you only see one piece of content, but other full-screen content can be hidden on this display at the same time. Professional cameras can capture one or more sets of images, while the eye only sees one content. This only became possible when global shutters became a reality on digital cameras eliminating the rolling shutter effect. 1
Global shutters enable cameras to read the whole sensor in one go, which helps prevent issues with rolling shutters like warping of fast-moving objects. It also enables Stadion LCD screens to refresh screens with different images at high speed making them unperceivable to the human eye but a camera that is synced to that refresh rate could see only that image. A P10 LED display from leyarled states in its specification that it is able to refresh its display over 1000 times a second.
The process, if we wanted to show a video to stadium attendees and a different video to viewers at home, would go something like this. If we assume we have a video where every 5th frame is replaced with a frame from a different source and assume that the frame rate of the video is 150 fps and shutter speed of 1/30 the camera will record only the video frames from that different source and not the original video. The viewers on the stands will see the original video while viewers at home will see a different video.
An example: Let's assume 150 frames per second. Show the viewers eye picture 1 to 4 from a related sequence, but picture 5 shows the first picture of a completely different sequence 2. Pictures 6 to 9 then belong again to sequence 1, picture 10 to sequence 2 - and so on and on. The eye receives 120 images per second from sequence 1 and 30 images from sequence 2 within one second. However, because of the persistence of visions, the human viewer only sees sequence 1, sequence 2 remains invisible because human perception is too sluggish to view the very brief "sneaked in" pictures.
Due to the high refresh rates of LED screens, it is possible to insert “up to 4 different video streams” into the LED panels.
What makes this technique powerful is its simplicity and robustness. There is no need for additional camera equipment. No major post-production is necessary to make the alternative content visible, as it is generated directly in the existing camera equipment. This method also creates real reflections in objects, unlike any other composing technique.
A few downsides are that it is not an overlay and is linked to LED screens which restrict its use to them, unlike the next method. The other is that for every stream of video that is encoded on the LED panels you need another camera with a different shutter speed.
The second method is much more technical and computationally intensive
This method uses invisible, to the human eye, infrared light that the camera with a smaller IR filter can see and can then be digitally removed. This creates a problem with the picture being too sensitive and creates a pink tinge which can be also digitally adjusted.
For this method to work the LED screens need to be modified and a 4th pixel added. This 4th pixel is IR LED that acts just like a standard green screen does. The production of such screens is not hard as production lines for RGBW where W is for the pixel that emits only white light can be replaced with IR pixel. But still creates an additional cost of purchasing or installing all-new LED panels on all venues.
How this works, in principle as very little information is known and is proprietary so this is just an educated guess.
Viewers in the stands see Vodafone ad while the camera sees additional infrared light that is rather different than any normal color and creates a pinkish tint over the LED panel. Because this pinkish tint is so different than other colors on the screen and if we combine it with other information from the camera as its position, zoom level, and others it is possible to replace the pink with any other image we want. In essence, what this does is create a pink “green screen”. By keying the pink color and replacing it the viewers at home would see the Continental ad.
Of course, it is more complex than explained but the idea is there. It also helps that LED panels are all box-shaped and have a fixed position so with little setup and a lot of math this approach also seems workable.
The downsides are many but so are the upsides. With the need to replace or somehow augment the existing panels, maybe by adding IR backlighting to existing panels or covering them in some transparent to normal light but reflective to IR light material, this approach could become expensive fast. But then you don’t only get access to 4 additional streams of advertising but almost infinite as any broadcaster can selectively add advertising. In the case of cable channels, this could be done per city not just country making this technology rather appealing despite its rather high upfront cost.
Then again if possible ad revenue is taken into account the cost of such a system fades rather quickly. For that matter, any of these systems easily recoup the cost of introduction and we will see them being used more often.
https://www.film-tv-video.de/event/2021/05/18/neue-technologie-fuer-variable-bandenwerbung-in-stadien/