Real-Time MindEye

Hi all this is special project -- we've been working on it for several months with prominent academics including Dr. Ken Norman (Princeton), Dr. Jonathan Pillow (Princeton), Dr. Ben Cowley (CSHL), and Dr. Kendrick Kay (Univ. of Minnesota). We're now at a point where I'd like to request help from y'all. If you can complete any of the tasks outlined below then you'll be invited to be a co-author on subsequent publication(s) resulting from this project. I will be adding additional tasks as the project progresses, I will alert everyone in the MedARC Discord working group after any such status updates.

Huggingface repository with data + code: https://huggingface.co/datasets/pscotti/rtmindeye

Communication / Questions

If you have any questions/comments regarding this project please publicly post in the MedARC Discord Real-Time MindEye thread: https://discord.com/channels/1025299671226265621/1261354849857962155

Project Goals

Collect our own 3-tesla fMRI dataset of people looking at images (2 subjects collected so far; sub-001 has 1000 samples and is shared in the huggingface repo)
Replicate successful MindEye results on our own data (currently in progress and are the basis for the tasks I'm requesting help with)
Simulate a real-time MindEye preprocessing + analysis chain to find the best real-time compatible approach (potentially with the help of active learning)
Implement that approach at Princeton MRI center to achieve world's first real-time reconstruction of perception

Aims

In order to achieve real-time reconstruction we first need to replicate the results of our MindEye2 paper, where we showed that you can get reasonable retrieval and reconstruction results with just an hour or so of fMRI data. The original dataset used for MindEye is the Natural Scenes Dataset, which is a 7-tesla very high-quality dataset of 30-40 hours of scanning per participant. It remains an open question whether we can achieve similar results with a new 3-tesla dataset, which would be an important achievement because there are not many 7-tesla MRIs in the world—most MRI machines are 3-tesla—and so making this technique practically useful would necessitate generalizing models to 3-tesla.

I am sharing our initial (preprocessed) 3-tesla data from sub-001 that I collected at the Princeton Neuroscience Institute. I am also sharing the code to analyze and run MindEye on it, where our subject looked at 1000 individual images following a scanning protocol almost identical to the Natural Scenes Dataset. You may need to read the MindEye papers and Natural Scenes Dataset papers for context when tackling these tasks.

MindEye2: https://arxiv.org/abs/2403.11207

Natural Scenes Dataset: https://www.nature.com/articles/s41593-021-00962-x

Natural Scenes Dataset Manual: https://cvnlab.slite.page/p/channel/CPyFRAyDYpxdkPK6YbB5R1/notes/M3ZvPmfgU3

Status Updates

July 17 2024: MindEye is now working on our Princeton datasets!

Open Tasks