Main

Sixty years ago, Rachael Carson brought the damaging effects of dichlorodiphenyltrichloroethane (DDT, Chemical Abstracts Service Registry Number (CAS): 50-29-3) and other organochlorine pesticides to light in her seminal book, Silent Spring. She described a range of negative impacts of organochlorine pesticides on wildlife, the environment and humans. Carson further emphasized the alarming persistence of organochlorine pesticides and their propensity to bioaccumulate in both the environment and within living organisms.

Silent Spring’s exposé of the negative impacts of organochlorine pesticides spurred a remarkable shift in public opinion towards pesticide usage. This shift in opinion eventually catalysed the emergence of the pro-environmental movement and rapid growth in primary literature investigating organochlorine pesticide impacts1. The publication of Silent Spring and the subsequent research kickstarted pivotal policy changes, eventually resulting in the formation of the US Environmental Protection Agency and the widespread banning of many organochlorine pesticides.

As the primary research on organochlorine pesticides grew, it naturally spurred a subsequent wave of secondary research. This secondary research often took the form of meta-analyses: that is, the quantitative syntheses of research results2. At their best, meta-analyses can be a powerful tool to reconcile conflicting outcomes and direct future research and can effectively complement primary research to inform policy decisions. However, at their worst, they can be misleading and riddled with subjective bias while projecting the illusion of objective authority3.

Meta-analyses are frequently used to elicit the impacts of organochlorine pesticides, but their methodological quality remains uncertain. Uncertainty regarding methodological quality is worrisome because some environmental policy decisions are influenced by the conclusions of meta-analyses4. Consequently, the weaknesses of existing meta-analyses may be overlooked and may misinform policy decisions. Furthermore, poor-quality methodologies in meta-analyses can mistakenly depict weak evidence as strong evidence, hindering future research. Critical appraisal tools such as the Collaboration for Environmental Evidence Synthesis Appraisal Tool (hereon, CEESAT v2.1) can address these issues by helping researchers identify the methodological quality and rigour of meta-analyses5. In turn, appraisal tools can be valuable for policymakers and the research community to identify poor methodological quality in meta-analyses.

The concerns regarding meta-analyses on organochlorine pesticides extend beyond methodological issues. This is because the characteristics of the primary studies used in meta-analyses, including which pesticides and subjects were examined and whether key ecological and ecotoxicological factors were synthesized, remain largely unknown. The lack of clarity regarding the included study characteristics could misinform policy decisions in areas where policy implementation is necessary. Concurrently, the fragmented evidence presents a challenge for future research, as the limitations in our current understanding remain unclear. To effectively address this last issue, one can employ a systematic review map (that is, a systematic evidence map of secondary literature) to identify study characteristics included in meta-analyses6. By mapping evidence included in meta-analyses, systematic review maps allow researchers to identify limitations in large and multidisciplinary research topics, which is essential to consolidate the past 60 years of organochlorine pesticide research since Silent Spring.

Given the highlighted concerns, we aimed to critically appraise and systematically map existing meta-analyses on the impacts of organochlorine pesticides. First, we assessed the methodological quality of meta-analyses. We then quantified which policy documents have cited the included meta-analysis and investigated whether the methodological quality of meta-analyses differed between those cited in policy documents and those not. Second, we identified the central research themes regarding characteristics of the primary literature that includes pesticides, subjects and impacts. Furthermore, we investigated whether these important ecological and ecotoxicological factors were included in the analysis (for example, in meta-regression models or subgroup analysis). To augment the critical appraisal and systematic map of meta-analyses, we integrated a bibliometric analysis under the ‘research weaving’ framework7. This enabled us to delineate global research geography and identify the key collaboration networks among countries, continents and research disciplines, providing a holistic view of the research focused on meta-analysis on organochlorine pesticides.

Results

Search and general time trends

The primary aim of this study was to investigate the methodological quality and study characteristics in meta-analyses investigating the impacts of organochlorine pesticides. To locate existing studies, we conducted a systematic literature search. This initial literature search was completed on six scientific literature databases: Scopus, Web of Science Core Collection, PubMed, ScienceDirect, the Cochrane Library and Bielefeld Academic Search Engine (see Supplementary Information 1 for full search strings). We then supplemented the scientific literature search with a backwards/forwards citation search using relevant umbrella reviews. Ultimately, our scientific literature search yielded a total of 3,439 unique records. To screen for relevant studies, we implemented a two-step process. First, we screened titles, abstracts and keywords, resulting in 344 articles meeting our predefined eligibility criteria. Second, we screened full texts. Following the full-text screening, we included 105 meta-analyses representing a body of 3,911 primary studies in our systematic map (Supplementary Fig. 3). We have provided a list of all studies rejected at full-text screening in Supplementary Data 1.

The earliest found meta-analysis fulfilling our eligibility criteria was published in 1993 (ref. 8). However, it was not until 2006 that meta-analyses became consistently published. The most productive years in terms of the number of articles published were 2014, 2016 and 2021, each of which yielded more than 10 meta-analyses (Fig. 1a,b). We found that a total of 227 policy documents cited the included meta-analyses, and the total number of policy citations is increasing over time (Fig. 1c,d). Furthermore, we found that policies focused on health (n = 121), agriculture and food (n = 22) and toxicological reports (n = 34) were most likely to cite the included meta-analyses (Fig. 1c,d). Clearly, despite the impacts of organochlorine pesticides being recognized for more than 60 years, it is only in the past two decades that meta-analyses have become commonplace in this research field and thus cited in policy documents.

Fig. 1: Characteristics of the dataset.
figure 1

a, Bar chart showing the annual number of meta-analyses synthesizing research on the impacts of organochlorine pesticides, categorized by different subjects of exposure. b, Area graph showing the cumulative time trends of meta-analyses synthesizing research on the impacts of organochlorine pesticides, categorized by different subjects of exposure. c, Bar chart showing the annual number of policy citations of the included meta-analysis analyses synthesizing research on the impacts of organochlorine pesticides, categorized by policy topics. d, Area graph showing the cumulative time trends of policy citations of the included meta-analysis analyses synthesizing research on the impacts of organochlorine pesticides, categorized by policy topics.

Full size image

Methodological evaluation

To indicate the methodological quality of meta-analyses on the impacts of organochlorine pesticides, we critically appraised 83 out of 105 relevant meta-analyses using the CEESAT v2.1 (ref. 5). The remaining 22 meta-analyses were unsuitable for critical appraisal using CEESAT v2.1 because they were meta-analyses between multiple databases (not primary papers) or without systematic review. To enhance the utility of CEESAT v2.1 to appraise the methodological quality of meta-analyses effectively, we surveyed the reporting of an additional four methodological items not currently appraised in CEESAT v2.1 (that is, publication bias, heterogeneity, sensitivity analyses and the use of reporting guidelines).

Overall, for each critical appraisal item, the included meta-analyses received the lowest score (represented in red) or the second-lowest score (represented in amber) in 83.4% of cases, indicating that low-quality methodologies are prevalent in meta-analyses investigating the impact of organochlorine pesticides (Fig. 2a). Furthermore, we investigated whether methodological quality differed between those cited in policy documents and those not. We found that meta-analyses were cited in policy documents irrespective of methodological quality (multinomial generalized linear model: Z = −0.0417, standard error (s.e.) = 0.3423, P value = 0.903) (Fig. 2b). This is a notable concern as it highlights that poor-quality meta-analyses are used in policy documents and are likely contributing to policy-making.

Fig. 2: Methodological assessment using CEESAT.
figure 2

The methodological and reporting quality of meta-analyses according to CEESAT v2.1 (ref. 5). Scores are represented by the following colours: gold represents the highest (best) score, green is the second-highest score, amber is the second-lowest score, and red is the lowest (worst) score. The total counts of studies allocated to each score are shown in each bar. All CEESAT v2.1 items, along with our interpretation, are provided in Supplementary Data 1. a, CEESAT scores for 83 assessed meta-analyses. b, CEESAT scores for meta-analyses not cited in policy documents (left) and those cited in policy documents (right).

Full size image

Concerning specific areas of methodologies in meta-analyses, we found that items related to data extraction (CEESAT items 5.1, 5.2, 6.1, 6.2 and 6.3) remain a notable area for improvement, with red scores being received in 44.3% of cases. Conversely, literature searching (CEESAT items 3.1 and 3.2) received the fewest red scores (6.6%), showing an area of relative methodological strength. However, we found that across all methodological areas assessed by CEESAT v2.1, second-highest scores (represented in green; 10.7%) and highest scores (represented in gold; 5.9%) remained scarce. This finding is consistent with other reports that poor-quality methodologies in meta-analysis are common in environmental science9,10. For complete details on the results of each CEESAT v2.1 item, please see Supplementary Information 1.

To extend the insights on the methodological quality, we surveyed methodological items for meta-analyses not appraised in CEESAT v2.1 (refer to Supplementary Data 1 for a comprehensive list of extracted methodological items). This survey focused on the reporting of publication bias (also known as risk of bias due to missing evidence), heterogeneity, sensitivity analyses and the use of reporting guidelines. Additionally, we provide an indication of the literature databases, analysis software, effect sizes, risk-of-biases tests and visualization techniques used within relevant meta-analyses in Supplementary Information 1.

In the appraised meta-analyses, 37.3% of studies did not report publication bias test results (n = 31) (Fig. 3a). This high proportion is a notable concern given that publication bias can alter the results of a meta-analysis11. Importantly, when publication bias is present and not addressed, meta-analytic conclusions are undermined and could mislead policymakers and the scientific community12.

Fig. 3: Additional methodological evaluation.
figure 3

Bar plots showing the counts (and percentages) of meta-analyses investigating the impacts of organochlorine pesticides according to main types of publication bias tests used (a), main types of data heterogeneity assessments used (b), main types of sensitivity analyses used (c) and main types of reporting guidelines used (d). Note that some meta-analyses may contribute to multiple types of approaches. MOOSE, Meta-analysis of Observational Studies in Epidemiology; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Full size image

Next, we found that heterogeneity was explored in 85.5% of appraised meta-analyses (Fig. 3b). This is a noted area of strength in the literature because exploring heterogeneity enables authors to quantify the inconsistency in effect size estimates. We emphasize that measuring heterogeneity is essential to understanding and correctly interpreting the overall mean effect13. If future authors find heterogeneity among effect size estimates, we encourage them to investigate sources of heterogeneity using meta-regression models12.

Also, we found that 37.3% (n = 31) of the meta-analyses reported sensitivity analyses (Fig. 3c) (a different analysis from publication bias and within-study risk-of-bias assessments, which are sometimes considered sensitivity analyses14. We assert that omitting sensitivity analyses comes at a cost to the methodological quality and reliability of meta-analyses. This is because sensitivity analyses enable authors to explore the robustness of meta-analysis results by conducting additional analyses, such as analysing the data with an alternative model, omitting a study or accounting for outlier effects and rerunning the model14.

Last, we investigated the use of reporting and conduct guidelines. We discovered that 51% of the surveyed meta-analyses followed a reporting or conduct guideline (n = 41) (Fig. 3d). Notably, we found that meta-analyses following a guideline had higher methodological quality compared to meta-analyses that did not follow a guideline (multinomial generalized linear model: Z = 5.18, s.e. = 0.4656, P value < 0.001). This is primarily because guidelines and checklists provide minimum reporting or conduct standards. Moreover, for meta-analyses that followed a reporting or conduct guideline, 10.5% included a relevant checklist in the supplementary material (n = 4). We found that, despite their uptake in other disciplines15, reporting guidelines remain underutilized in meta-analyses on the impacts of organochlorine pesticides and methodological quality is increased when reporting guidelines are used.

Taken together, we demonstrated that poor-quality methodologies are prevalent in the assessed meta-analyses (Fig. 2a). Also, other important elements of a robust meta-analyses are commonly not reported (Fig. 3). These findings underscore the need for enhanced methodological and reporting quality in future meta-analyses. We address these needs with methodological recommendations in the section ‘Recommendations to improve methodological quality’ below.

Characteristics of included primary studies

We characterized primary studies synthesized in the included meta-analyses to find gaps and clusters of the synthesized evidence. We considered the characteristics that are underrepresented in the existing meta-analyses as gaps and the ones that are common as clusters.

We found that the most frequently synthesized organochlorine pesticides were pooled DDT isomers (CAS: 50-29-3, n = 36, 43.4%), p,p’-dichlorodiphenyldichloroethylene (CAS: 72-55-9, n = 21, 20.3%), pooled DDE isomers (CAS: 72-55-9, n = 20, 19.2%) and lindane, also called gamma-hexachlorocyclohexane (CAS: 58-89-9, n = 20, 19.2%) (Fig. 4). Overall, 14 organochlorine pesticides were included in 10 or more meta-analyses. However, despite widespread coverage of many pesticides, 19.2% of meta-analyses did not report the chemical classification of the pesticides in the synthesis (n = 20). This is a notable concern, as poor chemical classification introduces ambiguity and makes it more difficult for research to effectively inform evidence-based policy on specific pesticides. Additionally, we found that 100% (n = 105) of meta-analyses included ecotoxicological relevant factors such as pesticide type, duration of exposure or concentration of exposure as moderators in a meta-regression. This is a highlighted strength of the evidence base, which features how important ecotoxicological factors influence results.

Fig. 4: Contents of meta-analyses.
figure 4

Bubble heat map displaying the number of times each of the top eight pesticides was included in meta-analyses and their studied impact categories. HCH, hexachlorocyclohexane.

Full size image

In terms of subjects and impacts measured, we found that 76.2% of meta-analyses focused on humans (n = 80). Here, carcinogenic effects (n = 35, 33.3%), neurological effects (n = 14, 13.3%) and endocrine disruption (n = 14, 13.3%), were the most frequently investigated (Fig. 3 and Supplementary Information 2). Thus, human-focused research is a distinct cluster of knowledge in the evidence base. In contrast, 16.2% of meta-analyses focused on the impacts of organochlorine pesticides on wildlife (n = 17) (Supplementary Information 2). This is a notable gap given that organochlorine pesticides have been described in primary literature to have both direct and indirect impacts on birds, fish, amphibians, mammals and insects16, providing ample scope for meta-analyses in ecotoxicology. Furthermore, we found that 100% (n = 105) of meta-analyses included ecologically relevant factors such as environment/habitat type, species exposed (if wildlife) or life stage of the exposure group. Similar to ecotoxicological characteristics, this is a strength of the evidence base, highlighting the importance of ecological factors in influencing results. Future directions for meta-analyses based on gaps in study characteristics are provided in the section ‘Future opportunities’.

Global research geography and collaborations

Our bibliometric analysis was conducted on an exported bibliometric file from Scopus, which included 100 of the 105 relevant meta-analyses. We found that the most productive countries of affiliation of first authors in the evidence base were China (n = 17, 17%), the United States of America (n = 11, 11%), Belgium (n = 7, 7%), Canada (n = 6, 6%) and France (n = 6, 6%) (Fig. 5 and Supplementary Information). These findings highlight that most research is led by developed countries, with limited studies led by Southeast Asia, Africa and Eastern Europe (Fig. 5). In addition to poor geographical coverage, international co-authorships remain scarce, with 59% (n = 59) of meta-analyses having all authors affiliated with a single country (Supplementary Information). The lack of research output and collaboration efforts with developing countries is concerning, particularly because many developing countries continue to use organochlorine pesticides for agricultural pest control and to combat vector-borne diseases17.

Fig. 5: Authorship and collaboration network.
figure 5

a, Heat map of the world showing the country-level counts for first authors’ country of affiliation of meta-analyses investigating the impacts of organochlorine pesticides. Grey indicates no publications affiliated with a given country in our dataset. Map generated according to ggplot2 function map_data using Mercator projection. b, Collaboration plot by meta-analyses authors’ continent of affiliation. Lines originate from one author’s continent and connect to the continent affiliated with a collaborating author. The portion of the circumference for each continent corresponds to how many authors are affiliated with that continent. Purple, authors affiliated with North America; orange, authors affiliated with Europe; green, authors affiliated with Asia; yellow, authors affiliated with other continents (Africa, Australasia and South America).

Full size image

To foster more research from developing countries and promote international co-authorships in research, numerous strategies have been proposed. For example, journals and institutions could incentivize international collaboration18. Similarly, researchers could adopt open science initiatives such as the sharing of code, data and methods19. By integrating research from less developed countries and promoting broader international collaborations, a more inclusive and comprehensive understanding of pesticide impacts can be achieved. This integration is crucial for developing globally relevant policies for organochlorine pesticide use.

Discussion and recommendations

Recommendations to improve methodological quality

In light of the identified methodological limitations, as well as gaps and clusters of synthesized evidence within meta-analyses exploring the effects of organochlorine pesticides, we offer recommendations to address these shortcomings in the literature.

Our survey indicated that potential publication bias was not reported in 31 (37.3%) meta-analyses within the evidence base. Among the meta-analyses examining the impacts of publication bias, Egger’s regression was the most-used methodology (n = 44, 66.7%; Fig. 3a). Additionally, the funnel plot was the most frequently used visualization technique (n = 56, 67.5%; Supplementary Fig. 13). Although widely used, Egger’s regression and funnel plots are often not appropriate as they cannot handle heterogeneity and, more importantly, cannot account for non-independence between effect size estimates20. To combat these limitations, we recommend leveraging recent methodological developments, such as implementing a multilevel meta-regression approach to Egger’s regression11. This approach can be extended to account for time-lag bias (that is, a decline in the size of effect sizes over time21), which is seldom considered in the literature (n = 0 in our dataset).

Next, we showed that assessment of (within-)study risk of bias (that is, critical appraisal of primary studies) remains relatively scarce in the literature (n = 42, 50.6%). Among those meta-analyses that reported a measure of within-study risk of bias, the Newcastle–Ottawa scale was used most frequently (n = 21, 50.0%). The Newcastle–Ottawa Scale was developed to assess the quality of non-randomized controlled studies in medicine, which may limit its applicability to environmental sciences. For instance, it is not well-suited for evaluating pesticide exposure experimental studies on wildlife due to the omission of important steps in experimental design, as it does not include details such as the species or life stage of exposure22. Furthermore, we found that these risk-of-bias tools were rarely included in the analysis (n = 7, 8.4%). To enhance the usefulness of risk-of-bias assessments, we recommend developing more tailored tools for specific scenarios in environmental science23 and incorporating the results of these assessments into statistical analyses.

Unfortunately, we discovered that meta-analyses synthesizing evidence on the impacts of organochlorine pesticides commonly do not report a sensitivity analysis (meaning sensitivity analysis excluding publication bias and within-study risk-of-bias assessments) (n = 52, 62.7%). The most widely used sensitivity analysis methodology was the leave-one-out analysis, in which each effect size is systematically excluded one by one, and meta-analytic models are rerun to investigate how the resulting overall effect size estimates are altered (n = 26, 66.7%). Notably, we propose that sensitivity analyses can be extended to highlight the consequences of violating assumptions of statistical or methodological non-independence14, helping to mitigate a widespread issue in environmental science meta-analysis12. Hence, sensitivity analysis can extend beyond investigating how individual studies impact meta-analytic results to shed light on the broader implications of methodological decisions.

We learned that guidelines for reporting and conducting meta-analyses are underused in the evidence base (n = 38, 45.8%). We argue that this underuse is a leading cause of the overall poor methodological and reporting quality overserved in meta-analyses synthesizing evidence on the impacts of organochlorine pesticides, as shown by the difference in the CEESAT v2.1 scores between those meta-analyses reporting the use of a guideline and those not. Consequently, we recommend that future meta-analyses consider following reporting guidelines such as PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)24,25 and ROSES (RepOrting standards for Systematic Evidence Syntheses)26 and conduct guidelines such as COSTER (recomendations for the conduct of systematic reviews in toxicology and environmental health research)27 to increase reporting and methodological quality.

Future opportunities

Primary studies on organochlorine pesticides have been described to impact a range of non-human animal taxa16. Yet meta-analyses on this topic remain scarce (n = 16, 15.7%). Research synthesis approaches using meta-analyses can investigate the role of important ecotoxicological and ecological factors in pollution research. For example, they can test whether phylogeny influences sensitivity to organochlorine pesticides. Although multispecies experiments can also be conducted, it is usually not possible to explore pesticide impacts on large numbers of species across many taxonomic groups due to ethical concerns and the resources available. To overcome this constraint and study how phylogeny moderates the impacts of organochlorine pesticides, meta-analytic models can incorporate phylogenetic relatedness when aggregating evidence from existing primary studies.

Study limitations and additional opportunities

Although our systematic review map provides several valuable insights, we acknowledge potential limitations stemming from the conduct of the literature search and data extraction. We recognize that our search was solely conducted in English, which may introduce language bias. This limitation could contribute to the geographical biases observed in bibliometric analyses28. Our work can be extended in the future to investigate global research output and collaboration efforts in languages other than English. Additionally, we acknowledge that other critical appraisal tools may give different insights than CEESAT v2.1. Thus, using or developing alternative critical appraisal tools can be considered in future work on this topic. Finally, we acknowledge that the Altmetric and Plumx platforms capture a limited range of policy documents. Therefore, we are likely to underestimate the potential impact of meta-analyses on policy documents.

Concluding remarks

Our systematic map, critical appraisal and bibliometric analysis of meta-analyses on the effects of organochlorine pesticides found that the literature has grown since Silent Spring’s publication to include 105 meta-analyses of 3,911 primary studies. Furthermore, we found that meta-analyses on organochlorine pesticides have been cited in 227 policy documents. The collated list makes these meta-analyses easier for policymakers and the environmental science community to find. By highlighting issues with methodological quality and research patterns, we have indicated directions for future evidence synthesis on this topic. Our bibliometric analysis showed a geographical bias in global research output, with a limited number of meta-analyses from developing countries, which could be addressed by fostering greater international collaboration and skills transfer.

Methods

We adhered to the RepOrting standards for Systematic Evidence Syntheses for systematic map reports26, adapting it for mapping meta-analyses. We preregistered our work with PROCEED (PROCEED-22-00043). Our full search and coding strategy can be found in Supplementary Information 1 and Supplementary Data 1, respectively. We provide author contributions within the methodology section using the MeRIT approach29.

Deviations from preregistration

We adhered to our preregistration (PROCEED-22-00043) as closely as possible with five minor modifications implemented. First, our initial plan was to employ CEESAT v1.0 for the critical appraisal component of our study. However, after deliberation, we decided to use CEESAT v2.1 (ref. 5). This revised version was deemed to provide a more robust and comprehensive assessment of the methodological quality and rigour in meta-analyses. Second, our data extraction process was refined. Whereas our original intention was to note whether a study had used a reporting guideline such as PRISMA 202025, we expanded this to code whether the study explicitly reported the application of the guideline or just presented the process flowchart. These two items were considered as two additional points in our analysis. We also added the following additional variables to enhance the insights from our study: (1) ecotox_confound, which refers to whether the meta-analysis provides ecotoxicological factors in the analysis; (2) eco_confound, which refers to whether the meta-analysis provides ecological factors; and (3) confound_analysis, which refers to whether the meta-analysis incorporates confounds into the analysis identified through risk-of-bias assessments. Third, we gathered the Web of Science Journal Citation Category for each study. Fourth, we additionally coded a general classification of the impact category investigated in relation to organochlorine pesticide exposure. Fifth, we extracted the policy document citations using Plumx (https://www.elsevier.com/en-au/insights/metrics/plumx) and Altmetric (https://www.altmetric.com/) data of all included meta-analyses to find out the policy influence of the included literature. We also categorized each policy by the country/region of origin and what area the policy was directed to (for example, agriculture, health, chemical profiling). This enabled us to compare methodological quality between studies that were cited in policy documents at least once and those that were not. Finally, our initial proposal was to use the bibliometrix package30 for bibliometric analysis. However, to enhance our research, we supplemented the bibliometrix package output by also performing bibliometric analysis using VOSviewer31.

Searching procedure

K.M. conducted a systematic literature search on five published literature databases: Scopus, ISI Web of Science Core Collection, PubMED, Cochrane Library and ScienceDirect. All searches were conducted on 4 August 2022 (accessed via the University of New South Wales, Sydney). Our search strategy comprised two groups of keywords: (1) terms describing organochlorine pesticides, including ‘aldrin’, ‘endrin’ and ‘endosulfan’, alongside their relevant abbreviations; and (2) terms related to meta-analysis, including ‘evidence synthesis’, ‘global analysis’, and ‘meta-review’. Complete details of all used search strings can be found in Supplementary Information 1.

K.M. vetted the sensitivity of our search strings against a set of ten pertinent benchmark papers32,33,34,35,36,37,38,39,40,41. In addition, we performed backwards and forwards citation searches using a set of relevant umbrella reviews42,43,44,45,46. To further expand our search, we also explored the grey literature using the Bielefeld Academic Search Engine, focusing on academic theses. Full details of the benchmark studies and the backwards/forwards citation searches are provided in Supplementary Information.

Screening process

We conducted abstract and full-text screening using Rayyan QCRI47. The screening was carried out in accordance with our PECOST framework (Supplementary Table 1) and screening decision trees (Supplementary Figs. 1 and 2). To minimize potential biases, every article underwent independent review by at least two examiners (K.M. screened 100% of the articles, and L.R., C.W. and M.L. each screened 33% of the articles). Any conflicts arising during the review process were initially addressed through discussion. In cases where disagreements persisted, an independent mediator (S.N.) was engaged to facilitate a resolution. Initial screening conflict rates between reviewers were established during a series of pilot screens and were documented in the registration (PROCEED-22-00043). All studies rejected during the full-text screening stage, along with the reason for exclusion are listed in Supplementary Data 1.

Data extraction

We manually extracted data in five steps. First, we extracted bibliometric information such as author, publication year, digital object identifier, journal and a unique study ID. We also extracted study methodology details, including the literature databases used, effect size type and how they tested for publication bias. Second, we extracted details about the organochlorine pesticides that were synthesized in each of the included meta-analyses. Third, we extracted information on the study subjects in each meta-analysis: specifically, whether the focus was on the impacts of organochlorine pesticides on humans, the environment or non-human animals. Fourth, we extracted information regarding the impact types investigated in relation to organochlorine pesticide exposure. Fifth, we then extracted all policy citations for each of the included meta-analyses. All the data extraction was conducted by K.M., with C.W., L.R. and M.L. cross-checking 7% of studies each (21% of data were cross-checked). Any conflicts between reviewers were resolved through discussion, with a mediator present if conflict persisted (S.N.). Supplementary Data 1 provides a complete data extraction strategy and all data descriptions (that is, metadata). Furthermore, all extracted data are provided in an external GitHub repository: https://github.com/KyleMorrison99/organochlorineSRM_analysis.

Critical appraisal

To assess the methodological quality of included meta-analyses, we used CEESAT v2.1 (ref. 5). K.M. conducted the appraisal for all relevant meta-analyses (with no authorship involvement in any of the assessed meta-analyses), and C.W., L.R. and M.L. cross-checked 7% of extractions each (excluding any articles they authored). We note that it was not possible to conduct a critical appraisal of all included meta-analyses because some meta-analyses did not synthesize evidence across multiple primary studies, so that many items of CEESAT were not applicable in such cases. This excluded 22 meta-analyses from the critical appraisal. We conducted the critical appraisal on 83 of the remaining meta-analyses. We then compared the methodological quality of meta-analyses that were cited in policy documents at least once and those that were not. Supplementary Data 1 includes all CEESAT v2.1 items and our interpretation of each item.

Bibliometric analysis

K.M. downloaded bibliometric information from Scopus on 20 March 2023 using the digital object identifier of each of the included meta-analyses. We used the bibliometric software VOSviewer31 to complete the bibliometric analysis. The network construction method used was bibliometric coupling, and the count method selected was ‘full counting’ (that is, all bibliometric coupling links weighted the same). The units of the analysis were document, source, author, organization and country. For each of the created networks, we filtered for the largest set of connected units. K.M. completed all bibliometric analyses, which were cross-checked by Y.Y.

Data analysis

K.M. conducted data analyses (cross-checked by Y.Y.) and created figures in R Statistical Environment version 4.2.1 (ref. 48) using RStudio build 576 (ref. 49). To compare methodological quality between meta-analyses cited in policy and those not, we used the clm function in the nominal package50. To create visualizations, we used circlize version 0.4.15 (ref. 50) and ggplot2 version 3.4.1 (ref. 51). All code is provided within a GitHub repository: https://github.com/KyleMorrison99/organochlorineSRM_analysis.

Declaration of generative AI and AI-assisted technologies

During preparation of this work, the authors used Generative AI: GPT 4.0 by OpenAI. This was used to enhance the structure, clarity and readability of the manuscript. GPT 4.0 was also used to annotate code with comments. The authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.