Thresholds in fMRI studies: Reliable for single subjects?

Many studies have investigated test-retest reliability of active voxel classification for fMRI, which is increasingly important for emerging clinical applications. The implicit impact of voxel-wise thresholding on this type of reliability has previously been under-appreciated. This has had two detrimental effects: (1) reliability studies use different fixed thresholds, making comparison of results is challenging; (2) typical studies do not assess reliability at the individual level, which could provide information for selecting activation thresholds. To show the limitations of traditional fixed-threshold approaches, we investigated the threshold dependence of fMRI reliability measures, with the goal of developing an automated threshold selection routine. For this purpose, we demonstrated threshold dependence of both novel (ROC-reliability or ROC-r) and established (Rombouts overlap or R_R) reliability measures. Both methods rely minimally on statistical assumptions, and provide a data-driven summary of the threshold-reliability relationship. We applied these methods to data from eight subjects performing a simple finger tapping task across repeated fMRI sessions. We showed that the reliability measures varied dramatically with threshold. This variation depended strongly on the individual tested. Finally, we demonstrated novel procedures using ROC-r and overlap analysis to optimize thresholds on a case-by-case basis. Ultimately, a method to determine robust individual-level activation maps represents a critical advance for fMRI as a diagnostic tool.