VID-AI: VIDeo feAture detectIon and retrieval

Visual threats are increasingly present in modern life and are starting to have large negative impact. Visual threats include all forms of manipulated images, videos and sophisticated tools, of which deepfakes are currently most common and pose the largest threat. Deeptrace B.V. – known as Sensity - is world leader in research and development of technology to detect such visual threats. It intercepts visual threats on a daily basis and monitors when new malicious content is released online. To determine whether a video or image file has been altered with, Sensity employs its state-of-the-art algorithms, detects even the tiniest (malicious) alteration and stores this in its internal database. Due to the quickly advancing field of visual threats production and the increasing easiness of using these tools, companies worldwide are rushing to develop technology that is better able at searching, detecting and preventing visual threat presence on the internet. Video, by its very nature, contains a lot of redundant information. There is much repetition among video frames, and much of the pixels in each frame are irrelevant for the meaning of the video. This is a universal challenge for the industry. Sensity currently employs a range of neural networks to detect certain ‘semantics’, i.e. infor-mation about a video that says something about the nature of the video. This way it can detect some visual threats. To future-proof the technology, it wants to developing advanced video embedding and retrieval technol-ogy. This will enable learning any relevant feature of a video, whether it is an object, a scene, an action, a se-ries of events – the idea is that using this information allows to detect certain flaws in the video so that it can be labelled a visual threat. Being able to retrieve (query) videos in any database based on a semantic or similarity search opens the door for uncountable applications. In case of visual threats, it would allow for querying of a certain video and quickly identify any video that looks very alike (which might be a deepfake) and it would aid immensely in classifying the applied visual threat production technology. This will for the first time make it possi-ble to gain more insight in the source of the visual threat, actually combatting the problem and defending people and companies from this ever growing threat. Sensity is an expert in computer vision and AI R&D and is one of the leading visual threat detection companies in the world. However, it has less experience with developing an advanced data and training pipeline or retrieval algorithms to realize video embedding and retrieval. This prompted the collaboration with BrainCreators. Brain-Creators’ expertise is to analyse any type of static data using its state-of-the-art data augmentation pipeline. Though it does not support dynamic video data yet, it is certain that its award winning BrainMatter technology will enable a unique approach mostly by leveraging its proprietary data annotation and feature extraction technology. These features lay the technological groundwork for developing a more effective and streamlined embedding, learning and retrieval approach. Successful completion of this project will result in Sensity being able to commercialize the first visual threat de-tection platform which is built on the latest video embedding technology and will be the only platform to support video retrieval. This functionality will be demonstrated with security firm PCS Security in this project. BrainCrea-tors will be able to integrate this technology in its BrainMatter platform, further positioning it as the most innova-tive enterprise AI solution provider. It has already received widespread interest from current and potential client in this kind of technology to, for example, clean up large video datasets, enable live (security) video analysis, monitor busy locations, greatly improve road inspection, etc. At least one of these use cases will be demonstrat-ed with SPIE or Advantech in this project, with the option to demonstrate additional use cases for BrainCreator’s own costs.