Mind Video

Mind-Video is an advanced AI tool designed to reconstruct high-quality videos from brain activity data obtained through continuous functional magnetic resonance imaging (fMRI) scans. This innovation builds upon the groundwork laid by Mind-Vis, a previous fMRI-image reconstruction project. Mind-Video addresses the intriguing challenge of translating non-invasive brain recordings into seamless video experiences through a two-module process. The initial module employs unsupervised learning with masked brain modeling and spatiotemporal attention to capture general visual fMRI features. It then refines these features through multimodal contrastive learning, using an annotated dataset. In the second module, these learned features undergo fine-tuning via co-training with an augmented stable diffusion model, specialized in generating videos guided by fMRI data. The tool's flexibility lies in its adaptable pipeline, consisting of an fMRI encoder and an augmented stable diffusion model, trained separately and fine-tuned together. This approach enables the encoder to progressively learn brain features, resulting in videos with exceptional semantic accuracy, including lifelike motions and scene dynamics, surpassing previous state-of-the-art methods. Moreover, an analysis of the attention mechanisms in the transformers decoding fMRI data underscores the pivotal role of the visual cortex in processing visual spatiotemporal information. The tool utilizes data from the Human Connectome Project and acknowledges the invaluable contributions of collaborators and supporters to its development.