Reconstructing Videos from Mouse Brain Activity

Researchers have developed a method to recreate short video clips using only data on brain activity recorded from mice. This approach made it possible to reconstruct the visual scenes that the animals observed. The work, conducted at University College London (UCL), opens new opportunities for studying how the brain transforms visual signals into internal representations of the surrounding world.

A New Approach to Studying Visual Information Processing

An article published in the journal eLife describes how these findings can contribute to a deeper understanding of how the brain processes visual information. In the future, this method will allow scientists to compare how different species perceive the same environment.

Previous studies on humans used techniques where participants were shown images and movies while their brain activity was recorded using MRI. Researchers then attempted to reconstruct the visual information, down to individual pixels, from this data. In the new study, recordings were made from individual brain cells in mice, providing a more detailed picture of how visual information is encoded in the brain. Using data from the visual cortex, the team was able to create high-quality reconstructions of the videos shown to the animals.

Technology and Reconstruction Algorithms

To reconstruct the films, the team used a dynamic model of neural coding, originally developed for the Sensorium 2023 competition. This model predicts the responses of individual neurons when mice watch movies and takes into account additional parameters such as animal movements and changes in pupil diameter.

The UCL researchers refined this approach using the same dataset. First, they calculated how neurons would respond to a blank screen and compared these predictions with actual activity during movie viewing. Brain activity was measured using a microscopic technique that identifies active cells by local increases in calcium levels.

The algorithm gradually adjusted the pixels of an initially blank film, using the difference between predicted and measured activity, so that the reconstructed video increasingly resembled the original shown to the mouse.

Method Validation and Future Prospects

After training the model, an additional test was conducted: brain activity was recorded while a mouse watched a video clip that had not been used for training. Using only this data, the system was able to reconstruct a 10-second film similar to the new video.

To assess the quality of the reconstructions, the researchers used a pixel correlation method, comparing corresponding pixels in the original and reconstructed films. The analysis showed only minor temporal differences between the two videos. The study notes that there is potential to improve image resolution and expand the scope of visual scenes that can be reconstructed.

In the future, the researchers plan to collect data that will enable the creation of sharper reconstructions and cover a larger portion of what animals see. The method will also be used to study differences between physical reality and the brain’s internal representation.

Significance of the Research

Vision is not a simple recording process like a camera. The brain constantly interprets, filters, and modifies incoming sensory information. Understanding where and how these changes occur can reveal important principles of perception. This research demonstrates that the brain’s internal representation of the world differs from objective reality, and these differences reflect the unique ways the brain processes information.