This dissertation investigates how metascientific approaches can enhancethe quality and reliability of evidence syntheses in educational psychology.Prompted by the replication crisis, widespread questionable researchpractices, and the growing dependence on systematic reviews and metaanalysesin education, this work critically examines current researchstandards and advances innovative solutions rooted in open science. Study I evaluates the methodological validity and reproducibility of theinfluential research synthesis Visible Learning by John Hattie. The studyreveals several methodological flaws that contest the assumptions of thefindings and the failure of being able to reproduce the statistics serves as awarning example of the presence of the replication crisis. Study II evaluates the risk of bias and transparency in systematic reviewsconducted in educational psychology. Alarmingly, most includedsystematic reviews were judged as high risk of bias and across the entiresample, there was a lack of data sharing, preregistered protocols, andreproducible primary research data. Study III is a proof of concept of a registered report in educationalpsychology, the study aims to investigate the evidence of a writingintervention by conducting a systematic review. By adhering to the stateof-the-art conducting standards in systematic reviews, this protocol coversall aspects needed to produce reliable evidence as well as beingreproducible. In Study IV, an innovative open-source Community-Augmented Meta-Analysis combined with a database is developed. The study presentssolutions to several well-known problems in systematic reviews byallowing the research community to update, store, calculate, and shareeducational interventional data in a convenient way. The findings of the included studies highlight significant gaps in researchrigor and transparency, underscoring the necessity of fundamental changeto adhere to current standards and modern research practices. By incorporating methodological tools such as preregistration, openscience, risk of bias assessments and FAIR data principles, this dissertationcalls for a paradigm shift in the synthesis and application of evidence ineducational psychology. Ultimately, it seeks to promote more trustworthy,transparent, and impactful research to better inform educational policy andpractice.