Challenge

How to extract more information and knowledge from the audiovisual contents of the Digital Government Congress and how to make them more accessible to users.

Current issue

The conferences, seminars, workshops and other informative activities organized by the AOC to promote the digital transformation and innovation of the Catalan public administrations generate a large amount of audiovisual content that is not always used well to extract information and knowledge.

This fact has been aggravated by the pandemic that has forced us to change our habits, routines and ways of relating overnight, so that, in a short time, the number of activities in line (meetings, conferences and congresses) has multiplied exponentially and now we have a huge amount of videos, voice notes and texts that we don't have time to process.

In addition to the pandemic, there are other factors that have accelerated the digitization process of events and activities:

  • Ease of access to meetings, the comfort and the reduction of organization costs thanks to a much simpler logistics.
  • Technological advances that make the online user experience better and better, allowing accurate audience analysis and data control.
  • The reduction of carbon emissions as a result of avoiding the displacements of the attendees.

So much so that in 2021 the Digital Government Congress organized by the AOC every two years, it went from being 100% face-to-face to a hybrid format, with presenters and speakers on stage and a virtual audience.

We immediately realized that a simple "Video on Demand" platform, such as YouTube, would not be sufficient to take advantage of the large amount of audiovisual content that would be generated before, during and after the congress, nor to make the more accessible knowledge. There was also a challenge which was the collection of data from users and their interests based on their browsing, in order to better understand their needs and expectations and to be able to redesign the content program for future events.

So, we set out to find out if there was any tool or market service capable of processing and analyzing videos, voice notes and texts, using artificial intelligence techniques, to extract information and knowledge and make - the most accessible.

Proposed solution

The applied solution came from Omnios, a Catalan technological startup born in Barcelona in 2019 that has developed an intelligent information processing service that makes large amounts of audiovisual content accessible; Winnow by Omnios: The intelligent Video On Demand platform.

What is Winnow?

A smart video platform that helps you navigate your media content. The videos and presentations contained therein are pre-processed using artificial intelligence techniques such as voice and person recognition, text mining and content categorization; in order to extract knowledge from it and be able to establish relationships between the concepts discussed.

Main functionalities:

The platform offers:

  • Smart maps of knowledge

    When a video is uploaded, the system automatically generates interactive maps. These maps are the extraction of the content and can be used both to compare what each speaker has said, and to navigate by comparing the videos and topics. The size of the circle basically depends on the frequency and importance of the concept with respect to the video, and the arrows mean a direct relationship between speakers and concepts. Maps can be explored according to Keywords | Brands or entities | Locations | people

What are Winnow Knowledge Maps?

Interactive graphs that help understand the knowledge Winnow automatically extracts from the content it processes (videos and presentations). The system understands how different papers relate to each other, finds links between concepts and people, and explores new and deeper connections.

It also offers:

  • Un smart search engine to be able to search, through keywords, the exact moment where a specific topic is discussed, thus saving idle time searching in parts of the video that are not of interest.
  • Speech recognition of speakers: Using artificial intelligence, the platform is able to recognize who is speaking at any given moment and this allows the speech to be segmented and each piece assigned to a specific person.
  • Download the transcript of the video in different languages. La plataforma té la capacitat d'entendre el vídeo o nota de veu i permet descarregar tota la transcripció en català, castellà i anglès, en format de document o bé mitjançant subtítols.
  • Classification intelligent of all videos and automatic "tagging".
  • Content and user analytics (by Google Analytics): they can be used to get to know users better based on their navigation; understand what they ask and want.

Proof of concept and pilot

The AOC Consortium launched a pilot project to test the advantages and benefits of AI applied to the audiovisual content of the CGD2021, while processing the resulting videos and presentations from the congress with the Winnow service.

A total of 162 contents were processed, including videos of the sessions (80) and supporting documents of the presentations in pdf format.

On 25/11/2021, access to the Winnow service was enabled through the CGD 2021 platform which was operational for 6 months.

Results

During the time that the pilot has been operational they have accessed the platform 167 unique users, a modest number considering that the number of people registered for the congress was 2.000.

El number of contents viewed has been from 354.

El chart 1 shows how, once access to the Winnow platform was advertised, 150 unique users used it on the first day, reaching a total of 874 visits in the first three months.

Chart 1

Al chart 2 we can see how the average time that users spend browsing the platform is approximately 4 minutes. With this information we can confirm that the goal of optimizing search time in a video as much as possible and reducing idle time has been achieved, since users go directly to the video they are interested in and manage to go to the point containing the relevant information quickly.

Chart 2

El chart 3 shows the number of users who have used the platform's search bar.

Chart 3

The most searched keywords through the smart search bar have changed over time. During the first 3 months of the pilot the most searched word was "artificial intelligence". At the end of the pilot the most searched word was "data".

conclusions

The metrics obtained demonstrate that the application of artificial intelligence to the multimedia content of an event allows:

  • Facilitate navigation within the contents and for users to find what they are looking for quickly
  • Automate summaries within audiovisual content
  • Collect user data to analyze their behavior and understand their interests

However, the platform where the content is hosted must also be easily accessible for everyone, whether they are people who want to watch a specific video again or those who were unable to attend the congress and want to nor an "intelligent" viewing.

In our case, access to the Winnow platform had to be logged (at the request of Omnios, the developer startup) and we believe that this discouraged the use of the service because people do not remember passwords, even less if they know what the contents are available on Youtube, an easy-to-access website that everyone knows.

Another handicap that we encountered when extracting knowledge from the audiovisual content of the congress is the language. The majority language of the congress is Catalan, but the Winnow platform is still in the process of improving the natural language interpretation algorithms of this language. We have been able to verify that the dictionaries and taxonomies of English and Spanish are much more developed.

Therefore, the pilot experience with the Winnow service has been positive, but the lack of filming in Catalan and the little use that has been made of the service has not allowed us to collect enough data from the users to make a comprehensive analysis of their behavior in order to understand their needs and interests and be able to redesign the content program of the next Digital Government Congress 2023.

Status of the project

Pilot completed

More information