Monitoring a multichannel playout infrastructure

Vue régieThe LoriotPro BE (Broadcast Edition) software dedicated to the monitoring of the audio and video infrastructure was implemented within the new playout of Canal + Overseas  that is currently operated by the company Ericsson Broadcast Service France. This multi-channels playout is in charge of providing about 30 channels including Canal + channels toward the operators of telecommunication in charge of the routing of the TV channels up to the destination countries.

The French company LUTEUS, software publisher and longtime partner on similar projects with Ericsson Broadcast Service France was chosen for supervision of the broadcast infrastructure.

The LoriotPro BE monitoring software, thanks to its internal architecture of multithreading processing, supports the collection and display of thousands of indicators of presence and quality of audio and video signals. It also supports monitoring of the network infrastructure for data flow and the video servers and storage servers. Operators are informed in real time by audible and visual alarms; furthermore all events are mostly stored in history files for offline analysis.

The performance figures: To control the distribution and quality of video and audio streams, LoriotPro refreshed 2300 indicators of operation and / or performance. Each indicator is the result of the collection and correlation of multiple SNMP objects on 6 chassis Kaleido mosaics of Grass Valley and 60 chassis IQ Modular Advanced Digital Snell (Snell & Wilcox). The collection period is from 1 to 5 seconds depending of the criticality of the indicator. Modular IQ 60 are equipped on average of 6 cards on which between 2 to 15 SNMP objects are collected. Add to this the 147 indicators for monitoring the IPTV streams and a hundred for monitoring LAN. To ensure these collections in the time constraints, 150 threads are sharing the workload.

A frequent problem for playout operators in such context is the monitoring of many channels simultaneously. Any absence of video or audio signals or frozen image (black, freeze) must be identified in seconds to trigger the necessary corrective actions. The challenge therefore is to inform operators in the second following the occurrence of a fault and then help them quickly and accurately identify the cause.

As for the audio and video streams, the numerous mosaics, Miranda Kaleido already provide visual alarms, but the simultaneous monitoring of a large number of screens, up to 16 channels on the same screen between SD (Simple Definition) and HD (Hight definition) video and only for the control of outputs is rather difficult. LoriotPro intervenes here by centralizing all information provided by the Miranda Kaleido equipment in a synthetic dashboard that clearly displays anomalies. Blinking object and sound messages attract the attention of the operator on the monitroing screen in case of change of status, absence of signal or loss of quality. For this, the power of the internal multitasking engine LoriotPro is put to use to achieve the collection by SNMP status indicators of audio and video signals present on the mosaics, but also indicators of active sources (ChangeOver) managed by IQHC3193 cards from SNELL. It is about 300 SNMP objects that are thus collected every second, analyzed and treated to provide a rapid update of centralized monitoring screen.Supervision Departs
A set of dashboards are available for operators and allow them to navigate rapidly between views of correlated status down to detailed views for each TV channel. The main display shows the overall audio and video status of all channels, by simply clicking a clearer view displays the type of anomaly (loss of video signal frozen video signal or black image) for the video part and for each audio channel status derived from a correlation of possible problem. As in the main view, depending on the severity of the anomaly on the SD or HD output signal, corresponding red and orange colors are used.

LoriotPro processing speed is achieved by the use of an internal database containing structures which are called Global Objects that stores the values of SNMP data collected or the values of correlated data. Contrôle départs vidéo/audioYou will find on this site many explanations on their principles of operation and use. The functional principle of the whole, as shown on the image on right side, the monitored equipments, on the right side, the dashboard and amidst this Global Objects database and audits programs (thread) in charge of SNMP data collections. The dashboard (Active View) performs requests to the database of Global Objects in order to update the function blocks and status color.

The engineers in charge of the technical maintenance have access to detailed dashboard with the status of the components constituting the audio and video processing chain. The synoptic below shows the architecture used for processing chain of audio and video signal, it may exist variants as required in SD and HD signals. As can be seen, the doubling of equipment and routes within the signal path, but also for the outputs to the broadcast operators, ensures a very high availability. However, it is always necessary to know the status of all video processing modules like the failover modules and which outputs are currently ON AIR. The broadcast operator is in need of constant flows on the four outputs provided to it by the playout.

For supervision, the most detailed visuals like this image on the left were therefore used to rapidly identify the source of anomaly in the signal processing chain. Visual use an internal service called LoriotPro Active View. Synoptique ChannelThe Active View allow to create a large variety of dashboard like in Microsoft Visio but with dynamic graphic elements (color change) based on SNMP collected values. It is therefore easy on an existing schema to add dynamic status indicators based on the working condition of equipment, in this environment mainly by the audio and video processing cards (glue) SNELL. On the Active View indicators show the presence of the SDI signal in all intermediate stages of treatment and the current active output by the switches position Normal / Backup of departures 4 HD and SD.

The method selected to manage the configurations, uses descriptive files of the equipments and of the SNMP objects to collect. Everything is done to easily add or change the collection settings and make the solution scalable. For example, here is a sample configuration file for SNELL equipment channel number 13:

Let's go deeper in the technical aspects. The yellow highlighted area concerns a card SNELL IQSDA32-3G, a distribution amplifier of 7 HD SDI outputs. This card is physically located in slot 13 of the IQ Modular chassis from SNELL and relates to a SDI video stream on its Normal path (non-backup). Collected on this card, two SNMP objects that respectively indicates whether there is an input SDI signal and what type of signal it is (HD, SD, 3G). Dedicated processes (Audit in LUA script) are then used for the collection of these SNMP objects, the storage within the Global Object database and for display on the detail view.SNELL IQSDA32
The four outputs a channel to broadcasting operators, two main and two backup can either be powered by one of four sources (HD / Main, HD / Backup, SD / Main, SD / Backup). To ensure a seamless switching of the SDI source, cards SNELL IQHCO31 are used. Based on detecting presence of the input signal and / or error signal, the automatic changeover (auto changeover) is function of programmable conditions. The manual switching is always possible with GPI or RollTrack. Monitoring these cards is based on several SNMP objects, listed below:
iqhco31setuploggingmasterstate"]="OK" --change over Master state : 
iqhco31setuploggingbackupstate"]="OK" --change over backup state : 
iqhco31setuploggingoutput1"]="MASTER" --Cmd 104 State 0=backup 1 =master : 
iqhco31setuploggingrulesstate"]="WARN:Off" --"ACTIVE:MASTER" --WARN:Off --*

Supervision of video servers, storage servers and network infrastructure is also performed by LoriotPro. LoriotPro sends SNMP collection from different equipments and stores them in dedicated Global Object (internal base). Are then generated result files in XML and / or JSON directly accessed remotely via the LoriotPro WEB server. Ericsson then uses this data on the client side (WEB browser) for creating visual HTML.

In conclusion, LoriotPro demonstrates with this implementation, its ability to adapt to severe operating constraints both in terms of volume and in terms of collection speed, near from real time. Furthermore, the configuration flexibility provided by the use of the integrated LUA scripting language allows fast maintenance and update operations of this changing environment.