TY - JOUR
T1 - How well do covariates perform when adjusting for sampling bias in online COVID-19 research? Insights from multiverse analyses
AU - iCARE Study Team
AU - Joyal-Desmarais, Keven
AU - Stojanovic, Jovana
AU - Kennedy, Eric B.
AU - Enticott, Joanne C.
AU - Boucher, Vincent Gosselin
AU - Vo, Hung
AU - Košir, Urška
AU - Lavoie, Kim L.
AU - Bacon, Simon L.
AU - Vally, Zahir
AU - Granana, Nora
AU - Losada, Analía Verónica
AU - Boyle, Jacqueline
AU - Shawon, Shajedur Rahman
AU - Dawadi, Shrinkhala
AU - Teede, Helena
AU - Kautzky-Willer, Alexandra
AU - Dash, Arobindu
AU - Cornelio, Marilia Estevam
AU - Karsten, Marlus
AU - Matte, Darlan Lauricio
AU - Reichert, Felipe
AU - Abou-Setta, Ahmed
AU - Aaron, Shawn
AU - Alberga, Angela
AU - Barnett, Tracie
AU - Barone, Silvana
AU - Bélanger-Gravel, Ariane
AU - Bernard, Sarah
AU - Birch, Lisa Maureen
AU - Bondy, Susan
AU - Booij, Linda
AU - Da Silva, Roxane Borgès
AU - Bourbeau, Jean
AU - Burns, Rachel
AU - Campbell, Tavis
AU - Carlson, Linda
AU - Charbonneau, Étienne
AU - Corace, Kim
AU - Drouin, Olivier
AU - Ducharme, Francine
AU - Farhadloo, Mohsen
AU - Falk, Carl
AU - Fleet, Richard
AU - Fournier, Michel
AU - Garber, Gary
AU - Gauvin, Lise
AU - Gordon, Jennifer
AU - Grad, Roland
AU - Vallis, Michael
N1 - Funding Information:
This work was supported by the Canadian Institutes of Health Research (CIHR: MM1-174903; MS3-173099; SMC-151518); the Canada Research Chairs Program (950-232522, Chair holder: Dr. Kim L. Lavoie); the Fonds de recherche du Québec—santé (FRQ-S: 251618; 34757); the Fonds de recherche du Québec – Société et culture (FRQSC: 2019-SE1-252541); and the Ministère de l'Économie et de l’Innovation du Québec (2020-2022-COVID-19-PSOv2a-51754). Study sponsors had no role in conducting the research.
Publisher Copyright:
© 2022, Springer Nature B.V.
PY - 2022
Y1 - 2022
N2 - COVID-19 research has relied heavily on convenience-based samples, which—though often necessary—are susceptible to important sampling biases. We begin with a theoretical overview and introduction to the dynamics that underlie sampling bias. We then empirically examine sampling bias in online COVID-19 surveys and evaluate the degree to which common statistical adjustments for demographic covariates successfully attenuate such bias. This registered study analysed responses to identical questions from three convenience and three largely representative samples (total N = 13,731) collected online in Canada within the International COVID-19 Awareness and Responses Evaluation Study (www.icarestudy.com). We compared samples on 11 behavioural and psychological outcomes (e.g., adherence to COVID-19 prevention measures, vaccine intentions) across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates (e.g., sex, age, education, income, ethnicity) impacted sampling discrepancies on these outcomes. Significant discrepancies emerged between samples on 73% of outcomes. Participants in the convenience samples held more positive thoughts towards and engaged in more COVID-19 prevention behaviours. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. No covariate performed reliably well. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviours being studied than would samples drawn using more representative means. Adjusting results for demographic covariates frequently increased rather than decreased bias, suggesting that researchers should be cautious when interpreting adjusted findings. Using multiverse-style analyses as extended sensitivity analyses is recommended.
AB - COVID-19 research has relied heavily on convenience-based samples, which—though often necessary—are susceptible to important sampling biases. We begin with a theoretical overview and introduction to the dynamics that underlie sampling bias. We then empirically examine sampling bias in online COVID-19 surveys and evaluate the degree to which common statistical adjustments for demographic covariates successfully attenuate such bias. This registered study analysed responses to identical questions from three convenience and three largely representative samples (total N = 13,731) collected online in Canada within the International COVID-19 Awareness and Responses Evaluation Study (www.icarestudy.com). We compared samples on 11 behavioural and psychological outcomes (e.g., adherence to COVID-19 prevention measures, vaccine intentions) across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates (e.g., sex, age, education, income, ethnicity) impacted sampling discrepancies on these outcomes. Significant discrepancies emerged between samples on 73% of outcomes. Participants in the convenience samples held more positive thoughts towards and engaged in more COVID-19 prevention behaviours. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. No covariate performed reliably well. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviours being studied than would samples drawn using more representative means. Adjusting results for demographic covariates frequently increased rather than decreased bias, suggesting that researchers should be cautious when interpreting adjusted findings. Using multiverse-style analyses as extended sensitivity analyses is recommended.
UR - http://www.scopus.com/inward/record.url?scp=85141539697&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85141539697&partnerID=8YFLogxK
U2 - 10.1007/s10654-022-00932-y
DO - 10.1007/s10654-022-00932-y
M3 - Article
C2 - 36335560
AN - SCOPUS:85141539697
SN - 0393-2990
JO - European Journal of Epidemiology
JF - European Journal of Epidemiology
ER -
