Introduction

Globally, a growing number of programs seek to enhance the use of scientific and technological information within ecosystems of evidence, policy, and practice (Oliver et al., 2022). In the United States alone, more than 170 programs aim to train scientists and engineers—hereafter referred to as “researchers”—in policy engagement (Akerlof et al., 2025), reflecting a sharp increase in number over the past decade. These programs are hosted by various institutions and take diverse forms, including scientific society training sessions, Capitol Hill visits, academic degrees and certificates, placements in government offices and boundary-spanning organizations, and participation in science policy student groups. The common denominator across all these programs is that they seek to build researchers’ capacity for policy engagement. Yet, globally, few of these programs are evaluated with respect to the outcomes they set out to achieve, such as their learning objectives. Of the 253 activities to build researcher skills that Oliver et al. (2022) identified worldwide, only 4% had been evaluated. As a result, scholars have concluded that “research-policy engagement is under-theorized and under-evidenced, with new activity outstripping research capacity to conceptualize and assess these efforts” (Hopkins et al., 2021, 352). We seek to advance the development of evaluation frameworks by identifying relevant learning outcomes from academic and professional fields—such as science communication and public affairs—and ascertaining the extent to which existing programs, aimed at building researchers’ capacity for policy engagement, align with them. To do so, we conducted a study of outcomes and how they have been evaluated among science policy academic programs (n = 6), student organizations (n = 5), and fellowships (n = 2) in the Commonwealth of Virginia, a state with among the highest number and diversity of such programs in the United States (U.S.) (Akerlof et al., 2025). With two or more programs across each of the three categories, the state of Virginia thus serves as an opportune exploratory case study for investigating the outcomes prioritized by highly varied initiatives. Twenty-eight states and the District of Columbia have been identified as having one or more of these types of programs. Virginia is unique in having the highest number of such programs in the nation. We refer to these initiatives collectively as “science policy programs” because of their applied nature, seeking to facilitate researchers’ engagement in policy processes, and their focus on both policy for science and science for policy (Brooks, 1964). Formal degree programs in science, technology, and innovation policy (Guston and Sarewitz, 2007; Lane et al., 2011) may or may not deliver these types of professional training and experiential opportunities in which researchers serve as a focal audience. For a review of the current landscape of U.S. programs see Akerlof et al. (2025). While there are few evaluations of programs that engage researchers in policy, there are even fewer academic studies of the programs themselves and their goals like that of Oliver et al. (2022), which focused primarily on cataloging a comprehensive sample of programs in the United Kingdom (U.K.). To our knowledge, little research has been conducted in the U.S. on training programs that engage researchers in policy (Akerlof et al., 2025). Developing an evaluation requires understanding each program’s goals, from prioritization of audiences to intervention outcomes (Goldman and Schmalz, 2006; McLaughlin and Jordan, 2015). For example, the central components of a model developed to evaluate a U.K. policy engagement program include the capacity of participating individuals, the institutions and systems in which they are embedded, and associated actions and outcomes (Mäkelä et al., 2024). Whether participants are able to successfully demonstrate learning outcomes after training (Kraiger et al., 1993) represents just one type of potential outcome that might be of interest for program leaders. Without an understanding of the full range of desired program goals, designing appropriate evaluations will remain challenging. Further, programs will find it difficult to identify and leverage relevant knowledge that has been aggregated over time by adjacent professional and academic fields.

Training researchers to engage in policy inherently involves boundary spanning between the research and policy communities (Sarewitz and Pielke, 2007). Knowledge about these communities and their interconnections is the purview of various fields of professional practice and academic disciplines that have developed their own distinct literatures, including prioritization of learning outcomes in educational programs and training. For example, in recent years, science communication scholars have focused on ways to train individual researchers to engage with the general public and various other stakeholders, including policymakers (Besley et al., 2020; Dudo et al., 2021). Public affairs as a field addresses policy engagement from the perspective of the interests of the organizations that its practitioners represent (Timmermans, 2020; De Bruycker and McLoughlin, 2021). Lastly, an emerging multi- and interdisciplinary field specifically aimed at training researchers to engage in policy has begun to develop measures and methods to evaluate what constitutes effectiveness (Crowley, Scott, Long, Green, Giray et al., 2021; Crowley, Scott, Long, Green, Israel et al., 2021; Long et al., 2021). In this exploratory study, we seek to elucidate how these literatures could inform the design and evaluation of researchers’ training in policy, and vice versa. We describe how current science policy initiatives in one state are conceptualizing program and learning outcomes and compare them with the approaches taken by the aforementioned academic and professional fields. We pose three research questions:

RQ1: What program outcomes—particularly participant learning outcomes—are the foci of the diverse range of science policy initiatives, including academic programs, student organizations, and fellowships, in the state of Virginia?

RQ2: How do these learning outcomes relate to educational and training recommendations from academic and professional fields?

RQ3: What types of outcome evaluations are Virginia’s science policy programs conducting?

Literature review

In conducting a review of relevant literature, we focused on academic and professional fields that have as their purview the education and training of individuals, whether in an individual or organizational capacity, to engage in policy and politics. This restricted our search to those fields that define themselves as largely serving to educate practitioners, as opposed to generating academic research. We first identified relevant fields based on co-author expertise and a literature search and subsequently conducted keyword searches to locate articles on learning outcomes and evaluation metrics. We identified three broad domains: (1) science communication training, (2) public affairs and lobbying education, and (3) an emerging multidisciplinary scholarship on training researchers to engage in policy. Below, we review the literature from each of these domains to identify which learning outcomes and associated correlates they describe as important for training researchers for policy engagement. Examples of the measures these fields have developed to assess these constructs are included in Tables A1–A3 of the Supplementary Materials.

Science communication training

Globally, post-World War II discourses about the relationship between science and society increasingly have emphasized the expectation that government funding for research should lead to specific public gains (Guston, 2000), including improved economic growth, workforce education, industrial innovation, and decision-making. In the U.S., for example, the National Science Foundation’s (NSF) Directorate for Technology, Innovation and Partnerships (TIP) describes its mission as advancing “use-inspired” research (NSF, 2024) that both generates new knowledge and has immediate societal application (Stokes, 1997). As wider audiences participate in the production and use of science, and decision-makers and other stakeholders increasingly expect that scientific and technical information for decision-making is understandable, the demand for scientists and engineers to communicate their work with lay audiences has spurred the creation of science communication training opportunities (Newman, 2019). In the U.S. alone, Muindi and Luray (2023) identified more than 330 training opportunities for scientists in public engagement, 60% of which were categorized as science communication. Notably, 40% of the public engagement trainings were coded as focusing on policy and advocacy. Indeed, one of the primary stated goals for science communication, according to the National Academies of Sciences, Engineering, and Medicine, is “to influence people’s opinions, behavior, and policy preferences” (NASEM, 2017, 18). These goals align with the disciplinary roots of science communication in strategic communication (Besley and Dudo, 2022), which has been described as “the purposeful use of communication by an organization to fulfill its mission” (Hallahan et al., 2007, 3).

While science communication training has been accused of lacking a theoretically supported and evidence-based approach (NASEM, 2017), in recent decades, a scholarship has emerged to illuminate what outcomes trainers and researchers are trying to achieve through these efforts and to assess the extent to which they are attaining those goals (Besley et al., 2020; Rodgers et al., 2020; Dudo et al., 2021). As a result, the field of science communication has begun to develop typologies and associated measures of learning outcomes—such as knowledge and skills—that may be relevant to training researchers to engage in policy as well. We summarize these below. Baram-Tsabari and Lewenstein (2017) mapped six learning goals for science communication training that align with the educational literature’s focus on three domains—cognitive, affect, and behavior (Kraiger et al., 1993; David and Baram-Tsabari, 2019)—that are also often the foci when evaluating individual participant outcomes (W.K. Kellogg Foundation, 2004; J. D. Kirkpatrick and Kirkpatrick, 2016). Their six strands—adapted from Bell et al. (2009)—include affective responses, content knowledge, methods, reflection, participation, and identity. As described by David and Baram-Tsabari (2019), these strands can be incorporated into a four-level evaluation model, aligned with the Kirkpatrick training program assessment methodology to measure reaction, learning, behavior, and results (D. L. Kirkpatrick, 1967; J. D. Kirkpatrick and Kirkpatrick, 2016). Reaction entails how participants perceive the program, such as their views regarding the experience and attitudes; learning equates to the acquisition of new content knowledge and skills; and behavior refers to the extent to which these new practices are implemented following training. Finally, results constitute “the entirety of the outcomes and evaluates them in light of the program’s initial aims and goals” (David and Baram-Tsabari, 2019, 180). The authors do not recommend specific evaluation measures or attempt to summarize what knowledge, skills, or affective responses could potentially fall under these goals, but other scholars have attempted to do so.

A challenge in developing measures to assess whether learning outcomes have been met is that there are many potential contexts in which they will inevitably vary. While “know your audience” is a longstanding informal adage, it also holds true in the development of more formal strategic communication objectives (Besley and Dudo, 2022), implying that the skills, knowledge, and approach needed by the communicator may differ considerably whether they are conducting a media interview, giving a talk to a local organization, working with a community to co-produce new knowledge, or testifying to Congress. Aurbach et al. (2019) reviewed the science communication literature to identify foundational communication skills that are broadly applicable regardless of differences in communicators, contexts, and audiences. Each of their nine categories represent a set of skills (Table 1) ranging from identifying the researchers’ overarching goals to potential audiences in messaging, crafting narratives, visual design, nonverbal communication, writing, and engaging in dialog. Subsets of these skills also largely map to Baram-Tsabari and Lewenstein’s (2013) learning outcome typology and methodology for assessing science communication writing (Table 1; Table A-1, Supplementary Materials).

Table 1 Outcomes and correlates from science communication training.
Full size table

Attempts to evaluate outcomes from specific training events have been met with mixed success (Rubega et al., 2021; Capers et al., 2022), demonstrating the difficulty both in designing and implementing measures that accurately capture the learning outcomes of interest. In a series of studies, Rubega et al. (2021) and Capers et al. (2022) tested communicator speech clarity, engagement, and credibility using external audiences as evaluators (Table 1; Table A-1, Supplementary Materials). While these studies did not find pre- and post-training effects in the audience evaluations, they did find reduced use of jargon (Capers et al., 2022).

Rodgers and colleagues developed the science communication training effectiveness (SCTE) scale (2020) to assess a broader range of knowledge, affect, and behavioral variables (Table 1; Table A-1, Supplementary Materials). In implementing the scale pre- and post-training, they were successfully able to identify significant intervention effects on science communication self-efficacy, oral presentation self-confidence, and science communication knowledge. They also found that participants’ attitudes and motivations during training had a greater impact on the intervention’s outcomes than those that participants had at the outset of the experience (Akin et al., 2021). The question of what motivates scientists to engage with lay audiences has been the subject of a number of studies (Poliakoff and Webb, 2007; Besley, 2015; Besley et al., 2018) and led to the development of self-efficacy and outcome expectations scales (Peterman et al., 2017) for the purpose of conducting training assessments (Table 1; Table A-1, Supplementary Materials), although to our knowledge, no results from the use of these assessments have been published.

Public affairs education and lobbying

While the focus of science communication training is on preparing researchers to strategically engage with lay audiences, including decision-makers, to achieve their individual public engagement goals, the multidisciplinary field of public affairs takes an organizational- and systems-level perspective. Further, it specifically focuses on influencing policy processes, drawing from the academic disciplines of organization and management science, political science, public administration, policy analysis, and communication (Timmermans, 2020). While the field does not focus on training scientists and engineers per se, the extent to which most policy issues currently involve scientific and technical expertise (Fischer, 2009) suggests that research experts would be a relevant audience. As John F. Kennedy wrote: “Lobbyists are in many cases expert technicians and capable of explaining complex and difficult subjects in a clear, understandable fashion” (1956). Over the last few decades, articles in the Journal of Public Affairs, Journal of Public Affairs Education, and Interest Groups and Advocacy have explored how to design educational curricula to train public affairs professionals and lobbyists (Newcomer and Allen, 2010; Griffin and Thurber, 2015; Holyoke et al., 2015; Powell and Saint-Germain, 2016; Timmermans, 2020). Accreditation standards from the Network of Schools of Public Policy, Affairs, and Administration (NASPAA) mandate that master’s programs address five domain competencies for student learning (2023, 7):

  1. (1)

    the ability to lead and manage in the public interest;

  2. (2)

    to participate in, and contribute to, the policy process;

  3. (3)

    to analyze, synthesize, think critically, solve problems and make evidence-informed decisions in a complex and dynamic environment;

  4. (4)

    to articulate, apply, and advance a public service perspective;

  5. (5)

    to communicate and interact productively and in culturally responsive ways with a diverse and changing workforce and society at large.

The requirement by NASPAA that programs conduct evaluation has sparked models for assessing public affairs education and learning outcomes (Newcomer and Allen, 2010). Newcomer and Allen identified short-term outcomes from classroom learning and field experiences, such as the acquisition of new knowledge and skills, and subsequent use in employment. In their theoretical model, these short-term outcomes are moderated by enabling student characteristics such as self-efficacy and reflection that result in longer-term public service outcomes. But the authors did not present standardized metrics for assessing these outcomes. Instead, programs have taken a diversity of approaches, ranging from reviews of student performance on assignments and exams to surveys and student ratings of instruction (Williams, 2002; Piskulich and Peat, 2014; Powell and Saint-Germain, 2016). Jennings (2019) pointed to the difficulty of assessing public affairs competencies and outcomes—“a challenging, complex, and messy affair” (p 15)—as the reason that few formal studies have been conducted. He recommended looking to the field of public education as a model for more sophisticated analyses. Among the few formal studies described in the literature are some that focus on external engagement components of public affairs education. For example, Sprague and Percy (2014) sought to assess practicum experiences at Stanford University in which students work in small groups to conduct policy analyses for a local government or nonprofit. In a survey of five years of classes, conducted post-graduation, the authors asked graduates to rate their skills pre- and post-practicum, and for their subsequent usefulness. Alternately, to assess service-learning programs, Levesque-Bristol and Richards (2014) created a short form of the Public Affairs Scale measuring civic learning, which they define as community engagement, cultural competence, and ethical leadership. Similar to science communication, in which defining goals and objectives are foundational skills (Aurbach et al., 2019), public affairs and lobbying also take a strategic approach (Fleisher, 2005; Griffin and Thurber, 2015; Holyoke et al., 2015; Timmermans, 2020). Many of the learning outcomes relate to the need to collect and analyze information in planning a course of action, such as considering policy issue dimensions, stakeholders, governance institutions, communication media and content, and ethical and legal considerations (Table 2). As Wippersberg et al. (2015) stated, students should be able to “analyze politics and policies, and align both with corporate goals” (p. 58). Scientists and engineers in industry and government may easily recognize the need to understand their organization’s interests—and rules—when conducting policy engagement, but even if academic researchers are not as cognizant, they can face implications, depending on their institution’s rules and legal strictures (CSLDF, 2021; CSLDF, 2022). As such, the public affairs institutional-level perspective has broad application to all researchers seeking to engage in policy. Timmermans (2020) describes this learning outcome as understanding the extent to which issues present not just risks, but also opportunities for organizations. Lastly, the public affairs literature recognizes that policy engagement is a team sport that entails working and communicating within a broad system of other institutions and stakeholders, and their associated networks. As a result, public affairs learning outcomes include relationship and coalition building, ethics, communication and message development, and an understanding of governance institutions and policy processes (Table 2). These skill sets align with calls for improvements to evidence-based policymaking research and processes, in which scholars have increasingly argued for taking a systems perspective (National Research Council, 2012).

Table 2 Outcomes and correlates from public affairs education.
Full size table

Training researchers to engage in policy

A third area of literature, aligned with the highly multidisciplinary Use of Research Evidence field (URE) (Farley-Ripple et al., 2020), focuses on training researchers to engage in policy as a component of evidence-based policymaking processes. Recommendations for the curricular content of these trainings include: combining direct instruction with experiential learning to improve researchers’ knowledge and skills in policy-making processes; conducting policy-relevant research; analyzing policy; building relationships; communication; understanding how policymakers use research; voter attitudes and ideologies; researcher preferences for varying types of engagement; and knowledge of lobbying regulations (Scott et al., 2019; Crowley, Scott, Long, Green, Israel et al., 2021). While a few such programs have publicly shared their internal program evaluations (Alberts et al., 2018; Bankston et al., 2023), to our knowledge, only one—Penn State’s Research-to-Policy Collaboration model—has conducted experimental evaluations of the program’s impact on both researchers and legislative offices (Crowley, Scott, Long, Green, Giray et al., 2021; Crowley, Scott, Long, Green, Israel et al., 2021).

Two constructs that frequently appear in the literature are: (1) differing types of evidence use in policymaking contexts (e.g., instrumental, conceptual, strategic, tactical, imposed) (Scott et al., 2019; Crowley, Scott, Long, Green, Giray et al., 2021; Crowley, Scott, Long, Green, Israel et al., 2021; Long et al., 2021); and (2) the varying roles that researchers can take in engaging in policy (Steel et al., 2000; Steel et al., 2004; Singh et al., 2014). Each of the constructs describes an important dimension of the contexts in which researchers and decision-makers interact. Crowley, Scott, Long, Green, Israel, et al. (2021) found that congressional offices that participated in the program with trained researchers were not only more likely to value the idea of using research to inform how policies are understood, but also increased their use of research as assessed through the analysis of legislative texts. Researchers who participated in the program also demonstrated greater involvement in supporting congressional policymakers’ conceptual and imposed use of evidence; in the case of the latter, when policymakers were required or requested to do so (Crowley, Scott, Long, Green, Giray, et al., 2021). Measures of researchers’ preferences for various possible roles engaging in policy—whether as simply reporting study information, working closely with policymakers to integrate information in policy, or as issue advocates—have not to our knowledge been included in assessments of training. But these types of survey questions have been included in descriptive studies of attitudes toward engagement (Singh et al., 2014) and have been discussed within the context of developing training curricula (Scott et al., 2019). Measuring behavior changes post-intervention—including both policy engagement on the part of researchers and evidence use by policymakers—has often been the outcome of most interest, but typically relies on self-reports (Rocha, 2000; Alberts et al., 2018; Crowley, Scott, Long, Green, Giray et al., 2021). Other constructs of interest related to whether researchers choose to engage in policy and their experiences in doing so often include: competence perceptions or self-efficacy, outcome expectations or response efficacy, and perceived social norms (Table 3). However, it is important to note that, likely due to the highly multidisciplinary nature of this literature and lack of theoretical foundations, the terms used to describe and measure constructs are highly variable.

Table 3 Outcomes and correlates from training researchers to engage in policy.
Full size table

Comparative themes across literatures

Many learning outcomes appear across two of the literatures; however, only one appears across all three of the articles we have highlighted here: communication skills (Table 4). Furthermore, the fields of public affairs and “training researchers to engage” emphasize skills in building relationships as an important distinct additional dimension. These two fields also underscore the importance of teaching knowledge and skills related to policy processes, and policy research and analysis. Other areas of consensus across at least two of each of the literatures include: self-efficacy and outcome expectations as a correlate (all three literatures), social norms as a correlate (science communication/training researchers), an understanding of legal requirements (public affairs/training researchers), and taking a strategic approach (science communication/public affairs).

Table 4 Key themes across the three literatures; bolded themes appear in two or more of them.
Full size table

Methods

This exploratory case study employs a multiple-case design (Yin, 2008), in which we compare academic theory regarding programmatic outcomes to those of current initiatives. We identified 13 science policy programs in Virginia through a national database that we developed in a parallel study (Akerlof et al., 2025). The methodology for database development is described in the Supplementary Materials (Table B). The study instruments and data are available at https://osf.io/2srkd/. Our respondents include 12 of the 13 program leaders, each of whom participated in a pre-survey and interview between March 7th and April 26th of 2024; one of the program leads chose not to participate. George Mason University’s (GMU’s) Institutional Review Board approved the study protocols [#2135839-2]. Study participants were invited to serve as co-authors on subsequent research publications and were rewarded with a $50 gift card.

Logic model development

The study respondents participated in a two-stage process to develop logic models that represent their respective programs’ audiences, activities, outcomes, and long-term impacts (7 years+) (Supplementary Materials, Tables C, D). This paper focuses on program outcomes, defined as occurring within 1–6 years post completion (W.K. Kellogg Foundation, 2004), as identified for specific audiences that were prioritized as either primary or secondary. An online pre-survey collected information to generate the initial logic model (see template under instruments at https://osf.io/2srkd/). During the second stage of the study, research team members interviewed the respondents, reviewing their logic models, adding and correcting details, and prompting them to rate the importance of the outcomes to their program (1–5, very unimportant to very important) and to qualitatively describe the rationale for their rating. We also asked respondents during the interviews about their evaluation of program outcomes.

Variable coding and weighting

We developed exploratory typologies for audiences and outcomes by inductively coding the logic model data for each program into a small number of umbrella categories that were further subdivided to reflect secondary-level details (Saldaña, 2021) (Supplementary Materials, Tables E, F). We combined the frequency and importance rating of each code to generate learning outcome scores. The scores were generated for each of the three types of programs—student organizations, fellowships, and academic degrees or certificates—and each focal learning outcome category (e.g., skills, knowledge). To calculate them, we summed the importance ratings for all instances in which a specific code was mentioned by program leaders, divided by the summed ratings for all codes, as aligned with Sankey methodology (Riehmann et al., 2005). The resulting score, a percentage, illustrates the relative emphasis ascribed to each coded learning outcome from the full contingent of those described by program leaders. In instances where we compare between the three program types, we have multiplied each outcome importance rating by the inverse of program frequency in the sample to offset differences in sample representation. We use Sankey figures to illustrate connections and flow across sets of variables (Riehmann et al., 2005), in this case, relationships between audience, audience prioritization, and learning outcomes for each of the program types.

Respondent characteristics

Six institutions host the 13 science policy programs in Virginia (Fig. 1), three of which are universities designated by Carnegie as R1, or “very high research activity” (American Council on Education, 2024): Virginia Tech (VT), GMU, and the University of Virginia (UVA). The other three institutions include: Virginia Institute of Marine Science (VIMS)/William and Mary, Virginia Sea Grant (VASG), and the Virginia Academy of Science, Engineering and Medicine (VASEM). Among the dozen program leaders interviewed, four lead student organizations, two run fellowship programs, and six head academic programs. The academic programs and student organizations are located at universities (VT, GMU, UVA, VIMS), while the two fellowship programs are hosted by VASG and VASEM. Descriptions of each program are available in the Supplementary Materials (Tables G, H, I).

Fig. 1
figure 1

The Commonwealth of Virginia is host to 13 science policy programs.

Full size image

The program leaders include a lawyer (n = 1), doctoral students (n = 5), and doctorate degree holders six or more years past their degree (n = 6). Of those currently engaged in or graduated from a PhD program (n = 11), eight program leaders are from the physical sciences, life sciences, and engineering (their academic disciplines include astronomy, biochemistry, biology, marine science, and biomedical engineering). The remaining three are from the social sciences, with disciplinary expertise in public policy and administration. The sample of program leaders is split with equal representation of men and women. Most self-identified as white (n = 11) and one person as Asian.

Results

Virginia’s science policy programs seek to achieve a broad range of outcomes (Fig. 2), encompassing changes at the level of individual participants’ knowledge, skills, and behavior; more systemic changes to the relationships between researchers and decision-makers; and institutional changes in academia and government. These program outcomes can be divided into nine categories. The outcomes with the largest proportion of program-adjusted ratings are participant skills (percent outcome scores, 25%) and knowledge (21%), both of which we focus on in this paper. Changes to institutions and relationships represented relatively smaller proportions of adjusted summed outcomes scores (government policy processes, 6%; academic institutions, 6%; and researchers’ engagement in policy, 5%). Program leads conceived of these outcomes as indirect effects of programmatic and participant support (13%), behaviors, attitudes, and beliefs (10%), and workforce development (14%).

Fig. 2: Science policy program outcomes.
figure 2

Virginia’s science policy program leaders describe their desired outcomes as straddling academic institutions, government policy processes, and broader societal impacts.

Full size image

Desired science policy learning outcomes: participant knowledge and skills

While each program rated all of the knowledge and skills outcomes for their participants in their logic models as somewhat or very important (i.e., 4 or 5 on a 5-point scale), there was some variability across programs in terms of how they ranked each outcome in importance. In the following sections, we individually analyze the knowledge and skills outcome ratings by each program type, along with the reasons that respondents gave for prioritizing these outcomes. Sankey figures demonstrate how the learning outcomes relate to specific programmatic audiences and are linked to interactive dashboards.

Academic degrees, certificates, and training programs (n = 6)

Academic program leaders cited the greatest range of specific desired knowledge and skills outcomes compared to student organizations and fellowships: 11 knowledge outcomes (average score, 4.6) and 16 skills outcomes (average score, 4.6). Many of these programs focus on graduate students. In analyzing knowledge and skills learning outcomes independently for associated audiences and their prioritization, almost half of the total importance scores for each, as rated by program leads, were associated with graduate students (percent learning outcome scores for this audience, 47% knowledge; 43% skills) (Fig. 3). The six other audiences included undergraduate students (percent learning outcome scores, 15% knowledge; 17% skills), postdoctoral scholars (11%; 12%), academic faculty members (11%; 10%), students (general) (11%; 14%), nonacademic researchers (3%; 2%), and returning professionals (3%; 2%).

Fig. 3: Academic certificate/degree/training programs (n = 6).
figure 3

The Sankey diagrams reflect frequency x importance rating scores for: A knowledge outcomes, B skills. *Click on these links to access interactive dashboards.

Full size image

The specific types of knowledge with the highest ratings were basics of the policy process (percent knowledge outcomes scores, 23%) and basics of science policy (18%). By way of explanation, one respondent stated:

How do we move the large ship [STEM-H] … redirecting that large ship more towards an appreciation for the complexities that happen in policy processes beyond the scientific and technical.

Science policy is often divided into two categories: science for policy and policy for science (Brooks, 1964). Science for policy/using research evidence (percent knowledge outcome scores, 19%) was among the top three knowledge areas, while policy for science, including funding science, received a small percentage of the total ratings (3%). Exploration of the tensions between the roles of experts, decision-makers, and stakeholders in representative democracies similarly received smaller percentages of the importance ratings and were categorized as: stakeholder processes (5%); politics, values, and science (3%); roles of scientists and engineers in policy (3%); and decision-making roles (3%). The top priorities in terms of skills were communication and policy, categorized as: general communication (percent skills outcome scores, 14%), science communication and translation (12%), policy design (12%), and policy analysis (11%). As one program lead noted:

We’ve actually surveyed alumni to hear what skills they need to be successful in these types of positions. … It’s communication skills, really broadly defined from oral to writing, technical writing, etc.

It has been argued that advocacy is a necessary skill for scientists and engineers (Runkle and Frankel, 2012). However, it ranked low relative to the other skills prioritized by Virginia’s science policy programs (2%). One respondent did, however, strongly argue for its inclusion:

I think we do disservice to academics [by] not preparing them to be able to advocate for research funding, especially federal research funding—going up and being able to talk to staffers and policymakers or when you have secretaries and stuff come down to universities to hear how all the research funding is actually being used.

Student organizations (n = 4)

Virginia’s science policy student organizations focus on a smaller range of specific knowledge and skills outcomes than the academic programs—6 knowledge and 7 skills outcomes with average ratings of 4.7 and 4.8. However, they are similarly broad in their prioritization of policy knowledge and communication skills and also focus largely on graduate student learning outcomes. A majority of importance scores for knowledge and skills outcomes are associated with graduate audiences (percent learning outcome scores, 63% knowledge; 71%, skills) (Fig. 4). Indeed, their only other audiences with specific learning outcomes are undergraduate students (37% knowledge; 25% skills) and staff (4% skills). Student organization leaders described the same three sets of knowledge outcomes with the highest percentage of ratings as academic program leaders: basics of science policy (percent knowledge outcome scores, 39%), basics of the policy process (20%), and science for policy/use of research evidence (14%). As one respondent stated:

So, I think we find it really important in addition to give students a baseline knowledge of science policy for them also to go outside the confines of our programming capabilities and learn more about the field from other sources.

Fig. 4: Student organizations (n = 4).
figure 4

The Sankey diagrams reflect frequency x importance rating scores for: A knowledge outcomes, B skills. *Click on these links to access interactive dashboards.

Full size image

However, unlike those leading academic programs, they equated policy for science/science funding (14%) in importance ratings with science for policy (also 14%). Understanding the roles of scientists and engineers in engaging in policy (7%) and the roles of various types of decision-makers (6%) rounds out the knowledge outcomes identified by the student groups. The skills that student organization leaders said are important are those of communication and engagement. The highest rating percentages for specific skills include general policy and decision-maker engagement (percent skills outcome scores, 30%), science communication and translation (21%), general communication (17%), policy memo writing (17%), and op-ed writing (8%). Networking (4%) and general skills training and professional development (4%) complete the list. Said a student organization leader:

So the main mission is to catalyze the engagement of students in policy making. And how would you do [that]? … the skills that we are developing here [are] the communication skills such as both writing and oral skills, such as a policy memo, op-ed … these activities feed directly into the success of the mission.

Fellowships and government placements (n = 2)

The two fellowship programs in the Commonwealth of Virginia defined knowledge and skills learning outcomes solely for their primary audiences: graduate students and postdoctoral scholars. Graduate students, as in the other types of programs, serve as the predominant audience of interest in regard to the learning outcomes (percent learning outcome scores, 61% knowledge; 68% skills) (Fig. 5). The range of knowledge and skills described by program leaders is about the same as for student organizations: 5 knowledge outcomes (average score, 4.8), and 6 skills (average score, 4.6). The knowledge that the fellowship program leaders said is most important for participants to learn is about local and state governance institutions (percent knowledge outcome scores, 40%), which was not identified as an outcome for either of the other two types of programs. The program leads also prioritized knowledge of decision-maker roles (21%) and current policy issues (16%); current policy issues was also not a category cited by the other programs. Basics of the policy process (17%) and of government (6%) are more closely aligned with the learning outcomes described by academic programs and student organizations. One of the fellowship’s coordinators said:

So from my past experience with the fellowship, there are a lot of people who don’t know how the government works and especially don’t know how the legislative process works. And, if we want people working in this space, that’s been the one feedback that we’ve really gotten from policymakers is that scientists and engineers don’t speak the same language that they do. So, we want to see that the fellows are learning how the process works.

Fig. 5: Government placement/fellowship programs (n = 2).
figure 5

The Sankey diagrams reflect frequency x importance rating scores for: A knowledge outcomes, B skills. *Click on these links to access interactive dashboards.

Full size image

Like the other program types, communication is the most important among the skills identified: science communication and translation (percent skills learning outcomes, 36%). Policy memo writing (23%) was also described by the fellowship program leaders as a specific form of communication that is valuable for those seeking to engage in science policy. Other specific skills emphasized by program leads included time management (13%), networking (5%), and tracking legislation (4%). A respondent stated:

We want to be able to give them skills that will help them in their career and, hopefully, help them in a public policy career later on.

Comparison of Virginia science policy program learning outcomes to professional and academic literatures

Overall, the learning outcomes cited by Virginia’s science policy programs leaders generally resemble those in the literature, with a focus on teaching knowledge and skills in communication and policy processes. However, there are some differences. Learning outcomes in the literature (Tables 1–4), which were not cited by the Commonwealth’s program leads, include knowledge of relevant legal restrictions around policy engagement and lobbying (public affairs/training researchers); defining strategic goals and objectives (science communication/public affairs); and self-efficacy and outcome expectations as correlates (all three literatures). On the flip side, program leads emphasized some outcomes not addressed by the literature:

  • understanding science policy, and differentiating between science for policy and policy for science;

  • skills to write specific policy products (e.g., policy memos and op-eds); and

  • knowledge of how values, politics, and science intersect.

The desired learning outcomes—skills and knowledge—described by the three groups of program leaders do not resemble those from any one particular set of literature, but rather an amalgamation. Perhaps unsurprisingly, some of the constructs most closely tied to disciplinary literatures—such as evidence use typologies (Nutley et al., 2007) and the six strands of informal learning (Baram-Tsabari and Lewenstein, 2017)—are not cited as programmatically important, while general concepts like communication and policy processes are.

Evaluation of outcomes

Half of the respondents—in equal numbers across each of the program types (3 academic programs, 2 student organizations, 1 fellowship)—said that they perform some type of outcome assessment. In many cases across all three program types, the assessments consist of surveys that simply track information about who is participating and gain feedback about their experiences. But academic programs at Virginia Tech and University of Virginia have developed tools to conduct more in-depth evaluations of learning outcomes. In the case of the Science, Technology, and Engineering in Policy (STEP) graduate certificate at VT, some of the participants are engaged as part of an NSF Research Traineeship. A program evaluator has been conducting yearly surveys of participants, asking them to assess the course in regard to the learning objectives, reflect on how the content may integrate into their work, and contemplate how it has changed their thinking. Participants in one of the STEP program courses are also asked to construct mental models of the policy process at the beginning and end of the semester in order to interrogate how the experience has changed their understanding of governance in practice and the roles of expertise therein. In contrast to STEP, UVA’s PhD Plus program deploys an in-depth pre- and post-survey protocol with qualitative and quantitative measures to assess the university’s professional development offerings in science policy, including an annual Science Policy Bootcamp in January, spring semester policy trip to Washington, D.C., and a summer science policy skill-based workshop. The survey questions focus on the participants’ current knowledge and interest in science policy before and after attending the January bootcamp (4-point scale from “know very little about science policy and am just curious” to “very interested in a career in science policy”) and an assessment of whether participation in the bootcamp has improved their ability to incorporate science-policy perspectives into their research. Additionally, the pre- and post-surveys include a series of questions (5-point scale from “not at all confident” to “highly confident”) aimed at measuring the change in their self-assessed level of confidence following skill-based workshops, including their ability to identify legislative priorities of their representative, outline a policy-focused message for a nontechnical audience, write a memo for a science-policy audience, and communicate their message in clear, engaging language that avoids jargon. Debrief meetings focus on how the participating graduate students and postdoctoral researchers will use the knowledge gained in their research and career planning, how they will use the professional skills and science policy experiences they want to gain after participating in each type of training, and which career pathways and opportunities they are exploring. These data are used internally at UVA to inform and refine existing PhD Plus program offerings and to develop more advanced opportunities for graduate students and postdoctoral researchers who intend to pursue a career in science policy or aim to incorporate policy-informed perspectives into their future research.

Discussion

We identified three bodies of literature that establish curricular and learning outcomes for education programs designed to facilitate researcher engagement with policymakers and their participation in policy processes: science communication training, public affairs and lobbying, and training researchers to engage in policy within the field of URE (see Tables 1–3). These areas of scholarship and Virginia’s science policy programs strongly emphasize knowledge and skills learning outcomes, which are focused largely on graduate student audiences and, in some cases, postdoctoral researchers. Indeed, among the logic model-derived categories of outcomes coded for programs in the state, participant skills (25%) and knowledge (21%) ranked highest in importance when the scores are weighted to counterbalance differences in the relative number of each program type. Other outcomes identified include programmatic and participant support (13% total weighted importance ratings), workforce development (14%), participant behaviors, attitudes, and beliefs (10%), and impacts on government policy processes (6%), academic institutions (6%), and researchers’ engagement in policy (5%).

The multidisciplinary academic and professional scholarship on engaging effectively in policy processes suggests that curricula should broadly cover communication and policy processes (Tables 1–3), which largely aligns with the knowledge and skills learning outcomes established for science policy programs in the Commonwealth of Virginia. The importance of understanding the basics of the policy process was among the top three knowledge learning outcomes for all types of Virginia’s science policy programs (percent knowledge outcome scores, 23% for academic programs, 20% for student organizations, and 17% for fellowships). Similarly, either general communication, or science communication and translation, ranked as one of the top two skills for each program category (percent skills outcome scores, 14% for academic programs, 21% for student organizations, and 32% for fellowships). Yet, there are differences between the program types in their emphasis on specific knowledge and skills. Academic programs have the widest range of types of cited knowledge (11) and skills outcomes (16) among the three. Furthermore, while both academic programs and student organizations rate the importance of broad knowledge of science policy, the policy process, and science for policy highest, fellowships focus primarily on local and state governance institutions. There are also discrepancies between the results of our literature review and the outcomes prioritized by program leads. For example, both the public affairs literature and scholarly studies focused on training researchers to engage in policy emphasize relevant legal foundations and restrictions, while these concerns are not emphasized in Virginia’s programs. Organizations like the Climate Science Legal Defense Fund have tried to make this type of information more readily available to researchers through pocket manuals that can be used by any scientist (CSLDF, 2021, 2022). Furthermore, scholarship from the science communication and public affairs fields argues for a strategic approach in determining goals and objectives for engagement at the outset. These types of learning outcomes are also absent from the Commonwealth’s programs. While we would anticipate differences in prioritized knowledge and skills between academic, student, and fellowship programs due to their leadership, structure, and function, it is less clear why there might be differences broadly between academic literature and the learning outcomes of current science policy programs. We speculate on a few potential factors:

  • studies in the academic literature typically focus on specific program designs, especially in training scientists to engage in policy scholarship, that may have different audiences and logics than the Virginia programs, with correspondingly distinct skills and knowledge needs, and potential legal considerations;

  • and program leaders, often with backgrounds in the natural sciences, may be less familiar with the literature from these social and policy sciences.

Building better connections between these academic literatures and practice will not necessarily reduce these differences. Instead, we hope to inspire cross-learning that facilitates their mutual benefit and growth.

Limitations

This case study is exploratory in nature, focusing on a dozen of one state’s science policy programs. The process may, nonetheless, serve as a model for studying programs across the U.S. and beyond. In the future, researchers should explore larger samples for the development of more robust measures rooted in the initial categorization of program outcomes provided in this paper. A second potential limitation is that, because these programs largely lack written documentation for logic models or an equivalent, it is not clear to what extent the ideas expressed during the pre-survey and interviews by program leaders have become ingrained as programmatic knowledge and a guide to implementation. It is important to recognize that program leadership can change, and successors may not always adopt their predecessors’ viewpoints or goals. Each program leader may bring different priorities, though these are likely to generally align with the group’s overall historical mission.

Future directions for research and practice

Perhaps the most valuable benefit of the professional and academic literature illuminated in this study is its contribution to building theoretically informed evaluation models and measures (Tables A-1, A-2, A-3, Supplementary Materials). Without evaluation information on the impacts of these programs on their participants and the broader science-policy ecosystem, it will remain difficult to learn from other existing programs, engage in adaptive management, and demonstrate to program funders that they are achieving desired impacts. While half of the dozen science policy programs in Virginia included in the study have collected some data for assessment purposes, we found that only two have conducted in-depth evaluations of program outcomes. The rapidly evolving evaluation literature in adjacent fields may contribute over time to the development of promising practices using a set of common criteria, regardless of program type, length, organization administering it, or other variables. The literature review revealed that the multi-disciplinarity of fields that focus on policymaker and policy engagement is likely hindering the development of universal theoretical models and constructs. Each of the fields’ perspectives could contribute unique theoretical insights about these processes, improved research methods, and better evaluation. However, in order to do so, they need a unified framework—perhaps similar to Ostrom’s (2009) general framework for analyzing the sustainability of social-ecological systems—that facilitates interdisciplinary research and insights through the use of a common conceptual model and associated terms. Without such a framework, it will be difficult to establish theoretically informed metrics and assessment tools for these programs.

Addressing programmatic challenges

Finding the time and resources needed to engage in outcomes assessments is a challenge for policy programs that are relatively new and resource-constrained, though in somewhat varying ways based on the type of program. For example, while the VT STEP academic program has a relatively robust evaluation process due to its associations with an NSF-funded project, the team still struggles to conduct a comprehensive evaluation and to use those findings to update curricula. In contrast, the student group at VIMS and William and Mary found outcome assessments difficult to implement due to variable attendance and inability to capture student attention after an event when they have other obligations. In turn, the leadership of the Commonwealth Coastal Marine Policy Fellowship has wrestled with ways to develop a standardized set of outcomes when each fellow has a unique experience based on the state agency where they are placed and the project on which they work. Furthermore, fellows enter the program with different goals (i.e., obtain a job in a state agency, gain policy experience, network, etc.) that shape their experiences and outcomes. Research-to-practice partnerships are needed to build programmatic capacity for data collection and analyzes that build on existing theoretical models. In addition to an appropriate framework, program leads will need support to engage in such evaluations, both through institutional encouragement for incremental improvements of such programs through repeated data collection and program evaluation, and by providing financial backing for the leaders of these programs and outside evaluators to analyze, interpret, and implement any changes deemed necessary based on the assessment results.

Conclusion

The more than 170 programs across the U.S. seeking to engage researchers in policy invest significant resources in doing so. This study finds that programs in the Commonwealth of Virginia align with the research literature recommendations in terms of the basic components of what these curricula should entail: communication skills and knowledge of the policy process. Constructs from these studies can be used as a starting place for the development of evaluation metrics and for the design and sustainability of similar programs in other U.S. states. However, there remains no overarching theoretical basis or set of instruments by which to conduct research and evaluate these programs nationwide. Establishing a common conceptual framework of constructs and their relationships will allow researchers and practitioners to better collaborate to move the field forward.