NBO-SAGAN: Namib Beetle Optimized Self-Attention GAN for Cross-Modal Short Video Generation with Semantic Decoupling

Abstract

The rapid growth of short video platforms has fueled interest in automated short video generation, a task that demands high fidelity, temporal coherence, and semantic alignment with input data such as text, audio, or static images. This task poses important issues since to the requirement for accurate semantic understanding, spatial-temporal alignment, and realistic visual synthesis. In this research, a novel deep learning-based cross-modal short video generation framework is developed to address these challenges effectively. The Microsoft Research Video to Text (MSRVTT) dataset is gathered from the Kaggle, has 10,000 video clips from 20 categories undergoes extensive preprocessing, tokenization, Audio Modality using Mel-Frequency Cepstral Coefficients (MFCCs), Video Modality using Temporal-Spatial Synchronization and Normalization (TSSN). Then feature extraction is performed with a semantic decoupling module that utilizes encoders to extract and disentangle high-level semantic components, such as object appearance, motion dynamics, background context, and emotional tone from the input. A Namib Beetle Optimized Self-Attention based Generative input in various Adversarial Networks (NBO-SAGAN) is employed to generate short videos and to enhance fine details and correct visual artifacts. Experimental evaluations using Python shows that the NBO-SAGAN approach outperforms traditional methods Frechet Video Distance (FVD) of 230.74 and Kernel Video Distance (KVD) of 12.58 highlighting its effectiveness for expressive cross-modal video generation. This integrated methodology effectively combines modeling to produce controllable, expressive, and visually rich cross-modal video content.