Deep Learning Based Advertisement Replacement on Dynamic Background Videos

Abstract

Advertising (AD) on video backgrounds reduces the disturbing of audiences. However, integrating virtual AD on videos’ dynamic backgrounds is difficult. The object detection method cannot follow the background moving correctly, so that the AD replacement shows flicking on the video. Furthermore, the target background object may move behind the virtual logo and foreground objects. The virtual AD should be inserted behind these things. To overcome those challenges, this research integrates multiple Deep Neural Networks (DNN). First, object detection DNN identified object locations. Thus, tracking of these locations was done through an optical flow DNN. Moreover, an image in-painting DNN reconstructed the blocked objects, which helps the detection approach. Based on the detection and tracking, the virtual AD is pasted on the video, then object segmentation is utilized to put foreground objects back on top of the virtual AD. The experimental results show that, in the dynamic background scenario, the AD replacement has a sensitivity of approximately 81.1%, a specificity of at least 92.57% and a successful rate of more than 83.3%. This means that, in most cases, the virtual AD can be integrated into the appropriate position on the moving background.