Prof. Johnson’s Online Guide to State-of-the-Science (and -Art) Systematic Reviews

•  Overviews of the Systematic Review Process

1. Johnson, B. T. (2021). Toward a more transparent, rigorous, and generative psychology. Psychological Bulletin, 147(1), 1-15. DOI: 10.1037/bul0000317
2. Johnson, B. T., & Hennessy, E. A. (2019). Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses. Social Science & Medicine, 233, 237-251. DOI: 10.1016/j.socscimed.2019.05.035
3. Johnson, B. T., & Eagly, A. H. (2014). Meta-analysis of social-personality psychological research. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (2^nd Ed., pp. 675-707). London:  Cambridge University Press. [Download Link]
4. Overview of meta-analytic reviews (updated version of a talk Blair T. Johnson and Emily A. Hennessy originally presented at the CDC in April 2021). [Link 71:11]

• The Meta-Review Phase of Research: Do You Have a Good Topic?

1. Helpful flow chart (here) for deciding whether a review concept should be pursued.
2. Hennessy, E. A., Johnson, B. T., & Keenan, C. (2019). Best practice guidelines and essential methodological steps to conduct rigorous and systematic meta‐reviews. Applied Psychology: Health and Well‐Being, 11(3), 353-381. [article]
3. Hennessy, E. A., & Johnson, B. T. (2020). Examining overlap of included studies in meta‐reviews: Guidance for using the corrected covered area index. Research Synthesis Methods, 11(1), 134-145. [article]
4. Pescatello, L. S., Hennessy, E. A., Page, W., Craft, L., Katzmarzyk, P., Fish, A., & Johnson, B. T. (2021). Best practices for meta-reviews in physical activity and health research. Journal of Physical Activity & Health. DOI: 10.1123/jpah.2021-0243
5. Overview of meta-reviews (updated version of a talk Emily A. Hennessy and Blair T. Johnson presented at the CDC in May 2021). [Link 31:42]
6. Why focus purely and only on randomized controlled trials (RCTs?) Why uncontrolled trials might prove truly important. [Link 20:27]

• Literature Search

1. How do you define a topic? Use TOPICS+M (vs. PICO, PICOT, PICOTS, etc.), brief video [Link; 1:42]
2. ISSG Search Filters Resource
3. litsearchr is an R package to facilitate quasi-automatic searches for systematic reviews. There’s a video tutorial here on YouTube (of the first version). The article introducing listsearchr is here. This software can automatically write Boolean search strings in over 50 languages, with stemming support for English. It’s a partial solution to the problem of mono-language bias in literature searches.
4. How many studies will I find? [Link 20:32]

• Coding Studies

1. Coding traditional study dimensions. [Email Prof. Johnson if you want access to this video.]
2. Incorporating spatiotemporal dimensions (e.g., community- or nation-level dimensions).

• Estimating Effect Sizes

1. Guide to Effect Sizes and Confidence Intervals (Jané et al., 2024)
2. Campbell Collaboration (David B. Wilson) Effect Size Calculator
3. Calculating effect sizes, David B. Wilson
4. Complexities in Effect Size Calculation: Using p-values and deducing pooled standard deviations

• Analyzing Effect Sizes
• Meta-Analytic Models of Effect Sizes: Overview
• Heterogeneity and the moving constant technique. [Plus article here]
• Spatiotemporal Meta-Analysis [plus article here]
• Testing and adjusting for publication bias in meta-analysis
• Publication Bias in Meta-Analysis (and Other Vexing Issues) 
• Meta-Analysis of Dependent Sizes—Robust Variance Estimation with clubSandwich (James Pustejovsky)
• Chapter 11: Undertaking network meta-analyses
• metafor project (in R)
• Meta-Analysis via Linear (Mixed-Effects) Models — rma.uni
• Generating meta-analytic statistics in R
• How to perform a meta-analysis in R
• Methodological Quality
• Shea B. J., Reeves, B. C., et al. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ
• Higgins, J. P., Savović, J., Page, M. J., Elbers, R. G., & Sterne, J. A. (2019). Assessing risk of bias in a randomized trial. Cochrane handbook for systematic reviews of interventions, 6, 205-228. [link]
• The Role of Methodological Quality in Studies and Meta-Analyses [plus article here]
• Presentation of Systematic Reviews and Meta-Analyses
• Shiny app to create a PRISMA flow diagram
• What is the PRISMA guideline and what’s new in the 2020 guideline?
• APA’s Meta-Analysis Reporting Standards (MARS)
• External Resource Lists

1. HubMeta’s related links
2. Harvard’s Systematic Reviews and Meta Analysis page

Notes. Prof. Johnson’s YouTube channel is here. These resources compiled as of Monday, 04 February 2024. Please email suggestions to Prof. Johnson.

Synonyms and related terms: Meta-Analysis. Research Synthesis. Quantitative Review. Meta-Synthesis. Meta-Review. “Umbrella Review”. “Overview”. Meta-Regression. Big Data. Generalizable Knowledge.