As highlighted by the OECD Innovation Strategy, better measurement of innovation and its impact on economic growth, sustainability and inclusiveness is key to fulfilling the promise of better co-ordinated innovation policies in the digital era.
OECD Oslo Manual, 2018
The social contributions of research activities have become more and more important in the rapidly changing innovation environment. Despite the fact that industrial commercialization of research results constitutes one of the most essential drivers for innovation and competitiveness, most generally used social impact evaluation criteria do not include clear metrics involving research commercialization possibilities. In a similar manner, principles regarding sustainable development have been largely omitted from the impact criteria. This paper considers the "broader impacts criteria" (BIC) model developed for social impact evaluation in the National Science Foundation in United States. We propose extensions to the BIC criteria related to commercialization and sustainable development viewpoints on impact evaluation. This paper also considers a newly introduced extension to BIC, called "inclusion-immediacy criteria" (IIC). Based on it, we propose an extended version of the model that aims to additionally evaluate the impact of research from a commercialization point of view.
Academic institutions are nowadays considered as important drivers for national economies, since they are expected to spur innovations and thus stimulate economic growth (Weckowska, 2015; Rajalo & Vadi, 2017). These benefits may be delivered by means of industry utilizing the results of the research made in universities. For this reason, governments and national innovation policy makers are actively promoting the establishment and development of collaborative networks between universities and society, represented by industry, other private sector actors, the public sector, and non-profit organizations (Morlacchi & Martin, 2009; Perkmann et al., 2013; Rajalo & Vadi, 2017). In this way, national innovation policies nowadays often emphasize a so-called “third mission of universities”, which means that in addition to the fundamental goals of higher education and academic research, universities are also expected to make social contributions. These social contributions may include collaborative knowledge creation, transfer, and exchange between universities and external partners (Pennacchio, 2016). Policy makers are thus increasingly expecting publicly funded research not only to produce scientific and scholarly results, but also to enable clear social impacts.
From the viewpoint of attempting to commercialize academic results, social impact now plays a central role. In this context, successful commercialization requires industrial firms to be able to absorb critical knowledge from universities, and together with university partners create new knowledge that may be critical for their future innovation and new product development (Kunttu & Neuvo, 2018, 2020). It has nevertheless been argued that the results of academic research seldom yield specific inventions or industrial products (D’Este & Patel 2007), while at the same time it has been difficult to empirically evaluate and show the direct impact of university collaboration on industrial innovation (Laursen & Salter, 2004).
In recent decades, methods and tools have been developed to evaluate the social impact of university/academic research. Since 1997, the National Science Foundation (NSF) in the United States of America has used specific evaluation criteria to assess the contribution of research to society by requiring grant applicants to discuss ways their research will have broader impacts on society. Since then, the so-called “broader impacts criteria” (BIC) has become a standard policy tool for the NSF to show policy makers and the public that government-funded research is useful from a social point of view. The BIC model requires researchers to show that their research makes a social contribution in terms of educational outreach or broad dissemination of results (also to non-academic audiences). The BIC model also highlights the collaboration with external stakeholders.
Woodson et al. (2021) have recently proposed an extension to BIC called “inclusion-immediacy criteria” (IIC). This model aims to determine how research impacts are distributed across various social groups. In the European Union’s research funding instruments, the principles of “responsible research and innovation” (RRI) are used to evaluate the social contributions of research projects. RRI is used to evaluate the social impact of the research process by emphasizing viewpoints that involve ethical acceptability and ecological sustainability.
The existing general impact criteria such as BIC target evaluating the social impact of scientific research in general. Thus, these criteria do not consider impacts related to commercializing or developing innovations, which is often crucial to making the results of academic research useful for commercial products. In addition, the currently used version of BIC does not have links to sustainable development, which is considered one of the main themes in RRI. Likewise, the principles of sustainable development are strongly emphasized by the United Nations and its related organizations (Griggs, 2013).
In this paper, we discuss how the BIC model could be extended to better cover various aspects of commercialization and sustainable development. We also propose an extended version of the IIC model, which focuses on business and user viewpoints, both essential regarding commercialization. Both proposed extensions can be regarded as examples on how the currently used impact and evaluation criteria can be modified to meet the needs of commercializing academic research results.
The rest of the paper is organized as follows. Section 2 gives an overview of current assessments involving “social impact” and introduces the current BIC and IIC models. In section 3, we present our extended versions of these two criteria, and in section 4 discuss the contribution and conclusions of the proposed methods.
Literature Insights Assessing the Social Impact of Research
The priority of governments and innovation policy makers is to support research that not only increase scientific knowledge, but also contributes to society. For this reason, policy makers and other actors operating in the innovation policy domain have created policy tools that seek to evaluate the social impact of publicly funded research. In recent decades, connections between academic science and social impact have been emphasized in the national science policies, for example, in the USA and Europe. Behind this development are steady calls from politicians to make academic research more accountable, transparent, and applied, which has resulted in increasing demands from the public for demonstrable returns on investment in science (Woodson et al., 2021).
Broader impact criteria
In the USA, the National Science Foundation (NSF) introduced the broader impacts criteria (BIC) in 1997 to ensure that grant proposals would take the social impact of their research into account (Davis & Laas, 2014). BIC’s European counterpart, Responsible Research and Innovation (RRI) was introduced in 2011, and has now been adopted into the European Commission’s Horizon 2020 strategy. RRI has been defined as “a transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view on the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society)” (Burget et al., 2015). When comparing the BIC and RRI, it seems that BIC is concerned with more peripheral aspects of research that include, for example, inclusion and participation of disadvantaged groups, outreach activities and utilization of results. RRI, on the other hand, seeks to understand fundamental aspects about how research is conducted, including sustainability and equality considerations (Davis & Laas, 2014).
With pressure to increase the social contribution of science, both BIC and RRI have been relatively widely adopted as parts of impact evaluation systems. For example, both RRI and BIC offer guidelines on quite a general level, and there have been difficulties turning their general principles into specific guidelines (Davis & Laas, 2014). Debates have taken place about the importance and value of the BIC and RRI as evaluation methods (Woodson et al. 2021). Thus, there have been pressures to develop the current criteria to better respond to the needs of evaluating the wide practical social impact of academic research. Commercializing research results represents one essential part of these impacts (Perkmann et al., 2014). It also has been problematic that while a BIC-based evaluation is usually conducted, nevertheless the social impacts are often not systematically evaluated after the completion of the research projects (Woodson, 2021). Table 1 presents the BIC criteria that are used by evaluators of NSF grant proposals. The evaluators are asked to evaluate the social impact of the proposed projects based on these criteria.
Table 1. BIC Criteria, adapted from Roberts (2009)
Immediacy and inclusion
In recent years, there have been proposals to consider the immediacy of research. In Woodson et al. (2021), immediacy is defined as the inherent nature of broader impact activities relative to the research. Immediacy can be divided into three categories: intrinsic, direct, and extrinsic immediacy. Intrinsic immediacy refers to research activities that are central to the focus of a research project. Consequently, the goals of research projects that have intrinsic immediacy are directly related to the social impacts. Direct immediacy means impacts that are achieved while conducting the research, even if they are not the actual goals of the project. For example, training younger researchers and graduate students in education belong to this category. The third category is extrinsic immediacy. This means that project activities are separated from actual research, wherein the researchers make contributions to society through outreach activities such as school visits, public lectures, or newspaper columns. Table 2 summarizes the three levels of immediacy.
Table 2. Levels of Immediacy, adapted from (Woodson, 2021)
Woodson (2021) has further developed this evaluation scheme by adding a new dimension “inclusion”, to better understand the role of various users of new innovations, including marginalized groups. Inclusion determines the types of people that benefit from the research. It is divided into three categories: universal, advantaged/status quo, and inclusive. Universal inclusion means that the innovation is targeted for everyone, independent of their status, and that everyone benefits from its results. For example, research related to minimizing the effects of climate change would belong to this category. Innovations in the second category primarily target advantaged groups, who can afford products based on innovation. New technological solutions for consumer electronics often belong to this category. These innovations may eventually also benefit marginalized groups, but only after being redesigned, or after advantaged groups have fully benefited from the innovation. The third category of inclusive innovations are those that are designed to help marginalized communities directly. Marginalization may be based on poverty or belonging to an underrepresented minority group, as well as a person’s dialect, gender identification, or religion. Thus, research initiatives with inclusive impacts may include, for example, the participation of women in scientific fields where they are underrepresented, or developing new pedagogical methods for children with special needs. Table 3 summarizes the levels of inclusion.
Table 3. Levels of Inclusion, adapted from (Woodson, 2021)
In their newly proposed model: “Inclusion-Immediacy Criteria, IIC”, for assessing the social impact of research, Woodson et al. (2021) combined the immediacy and inclusion criteria. They suggest this combination complements the current BIC model by better determining how research impacts are distributed across social groups.
In this paper, we extend the existing versions of BIC and IIC to highlight two distinctive themes: commercialization and sustainability. The reasons behind these proposed extensions are to underline the commercialization potential of research, and ensure that research takes into consideration demands related to sustainable development. Our research process started by making a review of the existing literature on impact evaluation with special emphasis on social impact.
We also benchmarked the most widely used evaluation approaches in the United States and Europe, namely BIC and RRI. These approaches were presented on quite a general level, and we identified several “blind spots”, that is, relevant areas that were not considered in the evaluations. Especially in the BIC approach, we found topics related to sustainable development and commercialization to be blind spots, since BIC had nothing related to sustainable development, and commercialization came across as quite vague.
Finally, we examined how these two topics could best be integrated with existing evaluation approaches. This led us to decide that the most straightforward way to do this would be to propose two new sections to the BIC model. In the following sections, we describe the practical process in detail.
Extending BIC and IIC models to cover commercialization and sustainability
Extending the BIC model with commercialization and sustainable development viewpoints
After an extensive preparation and inclusion process, the UN member states adopted 17 Sustainable Development Goals (SDGs) in 2015 (Griggs, 2013). These goals were based on an urgent call for action by all countries - developed and developing - in a global partnership towards peace and prosperity for people and the planet. The goals recognize that ending poverty and other deprivations must go hand-in-hand with strategies that improve health and education, reduce inequalities, and spur economic growth - all while tackling climate change and working to preserve our oceans and forests.
The European RRI framework relies strongly on these general UN goals in its principles for evaluating the social impact research. The current form of BIC has no dimensions related to sustainable development, despite the fact that demand for impacts related to green thinking, sustainability, wicked global problems, resilience, and equality are emphasized by the UN and many other global actors. For this reason, we propose two additional criteria to BIC, as presented in Table 4.
Table 4. Proposed new BIC criteria related to sustainability.
Commercializing innovations based on scientific research has been considered as one of the most essential social impacts of academic work. This is because commercialization and business development constitute immediate and measurable market impacts based on the results of academic research (Markman et al., 2008; Perkmann et al., 2013). The commercialization process is directly linked to developing new products or services. This creates commercial relevance for the firm that develop their businesses based on innovation. This means that the industrial interests to utilize research results are directly related to commercialization (Markman et al., 2008). Because the current form of BIC does not include criteria related to business development or the commercialization of results, we propose two additional criteria for commercializing, as shown in Table 5.
Table 5. Proposed new BIC criteria related to commercialization
Developing a commercialization viewpoint to IIC model
As described earlier, the newly introduced model, “Inclusion-Immediacy Criteria” (IIC), was designed for assessing the social impact of research by combining criteria related to immediacy and inclusion (Woodson et al., 2021). The purpose of the IIC model is to complement the currently available BIC model by better determining how the impacts of research are distributed across social groups (see Table 2). In this paper we present an extended IIC criteria (EIIC) that focuses more on widespread social impacts involving three levels: universal, project stakeholder, and user. Based on this proposed model, social impacts related to practical usage and commercialization can be viewed and discussed within a wide scope, and evaluated in a systematic manner. In the EIIC model, the levels of immediacy remain the same as in the original IIC, whereas the modified inclusion levels of EIIC are summarized in Table 6.
Table 6. The levels of inclusion in the original IIC (Woodson, 2021) and in the proposed EIIC model.
The EIIC model can be presented as a 3x3 matrix, as shown in Table 7, where the rows represent inclusivity, while the columns represent immediacy. The first row is used to express the universal impact of research. It considers how research is important to everyone and how anyone can benefit from research. The intrinsic level of universal impact describes what direct goals of the project benefit everyone. For example, research that is directly related to reducing carbon emissions belongs to this area, since everyone benefits from having a cleaner environment. The direct level is used to describe the universal benefits achieved while conducting research (even if they are not a part of the project goals). For example, if a research project focuses on developing low-emission engine technologies, then a better environment is not a direct goal of the research, even while the results of the research could contribute to improving the environment. The extrinsic level of universal impacts includes those that are separate from the actual research. Outreach activities like school visits or podcasts to wide audiences belong in this category.
The second row describes how the project benefits stakeholders. The stakeholders include both direct and indirect project stakeholders, such as funders, research communities, companies, public sector actors, ecosystems, etc. The impacts also concern business . The intrinsic level of stakeholder impact describes how a project’s goals benefit the project’s stakeholders, who can be regarded as customers of the research. The direct level of stakeholder impact expresses how the project’s stakeholders may benefit from the research outside of its initial goals. For example, if a project develops new algorithms for an IT company (intrinsic result), a direct result might be having a competent R&D workforce trained during the project period, which can potentially be recruited by the customer company. The extrinsic level of stakeholder impact includes impacts that are separate from the actual research. For example, presentations given by researchers at events organized by stakeholder organizations belong in this category.
The third row in the EIIC matrix describes the impact of the research outcomes to potential users. This category considers how the end-users of research results, or the individuals somehow affected by the research results, are involved. As on previous rows, intrinsic impacts are direct outcomes from an end-user’s point of view, and the direct impacts tell what kinds of benefits can occur while doing research. For example, research related to accessible and easy user interfaces causes intrinsic impacts to individual users. On the other hand, developing face recognition technologies aiming at new biometric solutions does not directly aim at consumer solutions, though the results may potentially end up in consumer photography applications. Extrinsic activities in the user category may be, for example, related to involving users or consumers in testing and evaluating newly developed innovations.
Table 7. Proposal of the extended IIC model (EIIC)
Discussion and Conclusion
As academic research makes contributions to society in many ways, it is important for funding bodies, policy makers and a general audience to understand the value of its social contribution. We considered this topic related to how the currently available impact evaluation approaches could better support commercializing the results of research. By developing dedicated evaluation approaches that emphasize commercialization, new possibilities arise to encourage the research community to pay more attention in considering applications for potential industrial usage of their research results.
In this paper, we also explored how sustainable development has a central role in the European RRI evaluation methodology. However, despite its growing importance, the principles of sustainability are absent in the American BIC evaluation concept. While the BIC criteria provide a good basis for assessing the social impact of funding proposals, they still lack important areas from the viewpoints of sustainable development and commercialization, both of which have become important themes for the social contribution of research. Because European RRI relies quite heavily on the principles of sustainable development, it would be beneficial also to BIC-related activities to somehow include these topics in impact evaluation considerations. However, we highlight that the extension proposals presented in this paper are just example ideas about how the current criteria can be extended based on the individual needs and focuses of the funding calls. It is up to the funding bodies and impact evaluators to decide how the extensions could serve the goal of the impact evaluation in the best possible manner, according to the case of each individual funding call.
As a second main topic in this paper, we presented an example of extending the newly introduced IIC model. As the IIC model has a practical two-dimensional structure for impact considerations in the inclusion-immediacy domain, it can also be utilized for considerations that focus on practical impacts related to commercialization. In our EIIC approach, we consider inclusivity on three levels: universal, stakeholder, and user. Especially the stakeholder and user levels are important from the practical commercialization point of view, since they measure what kinds of real benefits the research provides to direct project stakeholders, and what the benefits are for end-users of the innovations developed.
There is no single evaluation model that would fit all kinds of research. For this reason, we find it important to emphasize that the existing models can be modified, and that the models presented in this paper could work as examples of evaluation models for research with potential for commercialization in the short-term. The models for evaluating research impact tend to steer the planning of research projects, and thus effect the researchers’ mindsets regarding their research activities. In this manner, new evaluation models emphasizing commercialization viewpoints may encourage researchers to consider practical viewpoints involving their research from business perspectives. This activity can foster an innovation mindset among university researchers, while facilitating interactions between academia and industry.
This research is a part of research project: “Evolving innovation space, RDI policies and impact evaluation (ELVIS)”, funded by Business Finland (Grant 33346/31/2020).
D’Este, P., & Patel, P. 2007. University-Industry Linkages in the UK: What are the factors underlying the variety of interactions with industry? Research Policy, 36(9): 1295-1313. DOI: https://doi.org/10.1016/j.respol.2007.05.002
Davis, M., & Laas, K. 2014. ‘Broader Impacts’ or ‘Responsible Research and Innovation’? A Comparison of Two Criteria for Funding Research in Science and Engineering. Science and Engineering Ethics, 20: 963-983. DOI: https://doi.org/10.1007/s11948-013-9480-1
Griggs, D., Stafford-Smith, M., Gaffney, O. et al. 2013 Sustainable Development Goals for People and Planet. Nature, 495: 305-307. DOI: https://doi.org/10.1038/495305a
Kunttu, L., & Neuvo, Y. 2018. Balancing Learning and Knowledge Protection in University-Industry Collaborations. The Learning Organization, 26(2): 190-204. DOI: https://doi.org/10.1108/TLO-06-2018-0103.
Kunttu, L., & Neuvo, Y. 2020. The Role of Academics, Users, and Customers in Industrial Product Development. Technology Innovation Management Review, 10(3): 60-69. DOI: https://doi.org/10.22215/timreview/1337
Laursen, K., & Salter, A. 2004. Searching High and Low: What types of firms use universities as a source of innovation? Research Policy, 33(8): 1201-1215. DOI: https://doi.org/10.1016/j.respol.2004.07.004
Markman, G.D., Siegel, D.S., & Wright, M. 2008. Research and Technology Commercialization. Journal of Management Studies, 45(8): 1401-1423. DOI: https://doi.org/10.1111/j.1467-6486.2008.00803.x
Morlacchi, P., & Martin, B.R. 2009. Emerging Challenges for Science, Technology and Innovation Policy Research: A reflexive overview. Research Policy, 38(4): 571-582. DOI: ttps://doi.org/10.1016/j.respol.2009.01.021
Pennacchio, A.B.L. 2016. University Knowledge and Firm Innovation: evidence from European countries. The Journal of Technology Transfer, 41: 730-752. DOI: https://doi.org/10.1007/s10961-015-9408-9
Perkmann, M., Tartari, V., McKelvey, M., Autio, E., Broström, A., et al. 2013. Academic Engagement and Commercialisation: A review of the literature on university-industry relations. Research Policy, 42(2): 423-442. DOI: https://doi.org/10.1016/j.respol.2012.09.007
Rajalo, S., & Vadi, M. 2017. University-Industry Innovation Collaboration: Reconceptualization. Technovation, 62-63(April): 42-54. DOI: https://doi.org/10.1016/j.technovation.2017.04.003
Roberts, M.R. 2009. Realizing Societal Benefit from Academic Research: Analysis of the National Science Foundation’s Broader Impacts Criterion Realizing Societal Benefit from Academic Research: Analysis of the National Science Foundation’s Broader Impacts Criterion. Social Epistemology, 23(3-4): 199-219. DOI: https://doi.org/10.1080/02691720903364035
Weckowska, D.M. 2015. Learning in University Technology Transfer Offices: Transactions-focused and relations-focused approaches to commercialization of academic research. Technovation, 41: 62-74. DOI: https://doi.org/10.1016/j.technovation.2014.11.003
Woodson, T.S., Hoffmann, E., & Boutilier, S. 2021. Technovation Evaluating the NSF broader impacts with the Inclusion-Immediacy Criterion : A retrospective analysis of nanotechnology grants. Technovation, 101, 102210. DOI: https://doi.org/10.1016/j.technovation.2020.102210