How Well is Human Attention Preserved in fMRI-Based VisualBrain Decoding?

Abstract

Visual brain decoding (VBD) seeks to reconstruct visual stimuli from neural signals, yet current evaluation methods primarily focus on pixel-level or semantic fidelity, often neglecting how humans perceive these reconstructions. Since VBD models are intended for human-centric applications - such as clinical diagnosis and brain-computer interfaces - perceptual plausibility and interpretability are crucial. From a human-computer interaction (HCI) perspective, we propose assessing VBD outputs using saliency prediction models to simulate visual attention. By comparing saliency maps of original images and their reconstructions, we evaluate how well attention-relevant features are preserved. Our findings reveal perceptual biases in current VBD models across different image categories. We also suggest ways to incorporate saliency-based cues into the decoding process to improve interpretability. This approach offers a scalable, objective complement to user studies, bridging neuroscience, computer vision, and HCI. © 2025 Copyright held by the owner/author(s).

Publication
CHItaly 2025 - Proceedings of the 16th Biannual Conference of the Italian SIGCHI Chapter

Add the full text or supplementary notes for the publication here using Markdown formatting.