October 2013 Download this article as a PDFAbstract

This article provides an overview of how colleges and polytechnic institutes are fostering innovation literacy via support for business innovation, and it outlines models for measuring innovation literacy for improved downstream innovation and productivity in industry. The article demonstrates how we can innovate innovation by taking a specific, proactive, and instrumental approach to fostering business innovation and skills acquisition gained through applied research work experience by students as part of their college education. This approach is being used by George Brown College in developing a framework for measuring this innovation potential with a long-term, outcomes-based analysis.


The past decade has seen increasing investment in applied research in Canadian colleges. A primary output of applied research conducted in the college system is the impact on highly qualified and skilled personnel (HQSP): the students and graduates of programs that engage in applied research with industry partners. Although HQSP can be considered an input to business innovation, it is important to understand how HQSP are also important outputs of the college system. HQSP engaged in applied research at a college gain useful skills as an outcome of this experience. The ultimate outcome of this experience is increased business innovation. To understand how skills are acquired through student engagement in applied research and the potential downstream impact on firms, we must articulate the kinds of skills and activities that result in applied research activities in colleges.

A key program supporting business innovation in Canada is the College and Community Innovation Program (CCIP), which was formally instituted with the 2007 science and technology strategy (Industry Canada, 2007). The CCIP has two objectives: i) increased R&D and innovation capacity by local firms in a college's catchment area and ii) increased capacity of colleges to engage local firms in applied research. Although there are several significant government funding initiatives that focus on business innovation and applied research activities in colleges (e.g., FedDev Ontario’s Applied Research and Commercialization Initiative; Alberta Innovation Vouchers from Alberta Innovates Technology Futures; and Quebec's support for College Centres for the Transfer of Technologies), this article will focus on the CCIP as a program that is generally representative of programs that aim to enhance college capabilities in support of business innovation.

There is a two-fold benefit arising from colleges conducting applied research with industry partners: i) industry gains access to talent, facilities, markets, networks, and capital, along with support to launch new products and services into the marketplace, and ii) engaging students in applied research fosters innovation skills (i.e., “innovation literacy”) in graduates, thereby increasing the innovation potential of the workforce. In order to understand how these two benefits can be measured, this article provides an overview of the applied research activities undertaken by colleges, how these activities relate to innovation skills, how these skills might be measured in students and graduates, and what subsequent impacts graduates can have on firm-level innovation performance. A logic model is presented that articulates the relationships between activities, skills, and measurement. Such measurement is necessary for the college system to show impact on improved downstream innovation and productivity in industry as part of accountability for innovation programs as linked to education.

Applied Research in Colleges and Polytechnic Institutes

The college system is funded explicitly to engage in applied research with local firms. Applied research is distinguished from basic research in that it is oriented almost exclusively toward commercialization and practical outputs, such as the development of prototypes and the market entry of new products and services. Applied research in colleges is linked mostly to supporting small- to medium-sized enterprises (SMEs), because these comprise the majority of Canadian businesses. Funding for applied research in colleges is linked to engaging firms and requires these firms to match money spent on the activities. Canada's longstanding lag in industry R&D spending provides part of the impetus for applied research by colleges: industry partners are required to match CCIP funding, thereby promoting industry investment in R&D.

Colleges engaged in applied research support a range of services offered to firms as part of helping these firms to commercialize new products and services. The scope of funded, applied research services that colleges offer to firms demonstrates the kinds of activities firms require for innovation and commercialization. As per the two CCIP objectives described earlier, colleges offer these services to business to support innovation focused in their region. Importantly, these applied research services relate directly to the development of innovation skills in graduates; this is a key outcome that is designed to create and foster a resilient regional innovation capacity in local industry. Colleges help firms innovate while giving students innovation skills.

Both the Association of Canadian Community Colleges and Polytechnics Canada track metrics regarding applied R&D activities. The activities are linked to capacity development and the provision of applied R&D services. In terms of activities to support firms, Box 1 shows the applied research metrics (or "capabilities") collected annually by Polytechnics Canada to define the types of activities undertaken as part of applied research with industry partners.

Box 1. Inventory metrics for applied research, as collected by Polytechnics Canada

  • Proof of concept
  • Intellectual property registration
  • Feasibility study
  • Market identification/research
  • Proof of commercial concept
  • Application identification
  • Technology development / application development
  • Modelling/simulation
  • Prototype development
  • Field testing / technology verification / alpha testing
  • Product enhancement
  • Beta testing
  • Cost avoidance
  • Manufacturing process design and development
  • Commercial scale-up design
  • Certification (products, processes, and services)
  • Mass production
  • Market navigation
  • Marketing assistance
  • Technology adoption assistance (adoption of product/process by consumers)

Over the course of the past year, a team led by George Brown College in Toronto, Canada, has developed an online resource intended to enable firms to locate an applied research service provider in the college and polytechnic system. Called the Public-Private Partnership in R&D (P3RD), the tool used the activity metrics from the Association of Canadian Community Colleges and Polytechnics Canada as a basis to define the types of services firms need to support applied research and commercialization. The P3RD team used the North American Industry Classification System (NAICS) to orient applied R&D services to particular industrial contexts. We used the inventory metrics for applied research to describe R&D services firms access so that the P3RD system could adopt a user-centred firm perspective. The NAICS was used to delineate the type of industry accessing the site. This enabled us to design the site to support a firm working in a particular industry sector (as defined by NAICS) who needed a certain service, as shown in Box 2.

Box 2. Activities and R&D services offered by colleges, as represented in the P3RD application

Use a business service

  • Write a business plan
  • Clarify a product, process, or service
  • Develop a human resource practice
  • Develop a prototype

Develop a digital technology

  • Perform data analysis
  • Develop an academic technology program
  • Collaborate remotely
  • Design and develop a manufacturing process

Develop machining

  • Work with precision machines
  • Perform quality assurance

Get marketing advice

  • Perform market navigation
  • Study cost avoidance
  • Get marketing advice
  •  Increase adoption of a product
  • Map my value stream
  • Write a sales pitch

Develop products or services

  • Build a model or simulation
  • Develop and enhance a service
  • Conduct field testing
  • Enhance a product
  • Beta test
  • Develop a product
  • Develop a rapid prototype
  • Make a 3D drawing
  • Make packaging
  • Design for commercial scale-up
  • Design tools
  • Develop a new technology or application
  • Develop application identification

 Work in manufacturing and production

  • Investigate mass production
  • Layout a manufacturing plant
  • Plan a process or production
  • Build robots
  • Develop an inventory control system
  • Study sensory evaluation
  • Work in computer numerical control

The activities listed in Box 2 reflect the kinds of applied R&D services offered by colleges in which students are engaged as part of their experiential learning. The list is directly related to the types of skill outcomes associated with college programs. In other words, each applied research service offered to support firms relates to the kinds of skills students will acquire by performing these services. Students work on applied research as part of their program curricula. The demonstration or performance of these skills can be part of their core curriculum or ancillary to it. By specifically linking skills acquired and demonstrated through the performance of applied research conducted for firms, students are encouraged to embrace and understand innovation as it is practiced at the firm level.

The learning outcomes associated with college education are augmented by experiential learning, leading to greater innovation capacity; the hypothesis is that this will lead to greater innovation capacity in the economy in which our graduates eventually will work. These activities should therefore have outcomes and impact in firms, both those that partner with colleges on applied R&D, and those that employ graduates who have experience with providing applied R&D as part of their vocational training. Accordingly, these activities represent the practice of skills relevant to innovation in firms.

When students gain experience with supporting innovation through applied research, they gain "innovation literacy", which is “the ability to think creatively, evaluate, and apply problem-solving skills to diverse and intangible issues within industrial problems and multidisciplinary contexts” (Luke, 2009). Innovation literacy includes “research, development, problem solving, leadership, and entrepreneurial skills, along with the ability to recognize innovation in work contexts” (Luke, 2011). Innovation literacy encompasses the essential employability skills that students acquire through their work on applied research projects with partner firms. Innovation literacy is an amalgam of skills that encompasses the cognitive, psychomotor, and affective domains of learning; it is the ability to engage in the types of business innovation activities outlined in Box 2.

From Innovation Skills to Firm-Level Innovation

In colleges, applied research is focused on business innovation as an outcome. This outcome presents a measurement challenge insofar as the demonstration of impact of college-supplied R&D services will be affected by many variables outside of the college system's span of control, making direct causation or attribution difficult. Despite this challenge, we can still undertake efforts to link the provision of applied R&D services to the ultimate success of firms in terms of innovation capacity. A useful way to do this will be through college graduates, who can be proxies for our measurement of outcomes, as measured by the acquisition of innovation skills.

Skills acquisition, demonstration, and deployment are well defined areas of research and practice. The Ontario college system articulates essential employability skills as attributes of a college education. The OECD Innovation Strategy (Organisation for Economic Co-operation and Development, 2010) articulates the connection between affective domain skills and the hard and soft skills as key to enabling innovation. These essential employability skills and those skills articulated by the OECD are foundational to innovation literacy. However, there is work to be done in terms of ensuring that both graduates and employers see the links between innovation skills and their effect on productivity in the economy (Dwyer and Luke, 2012). Of key importance is the application of skills into workplace settings, with a focus on the capacity to engage in innovation and entrepreneurship activities, thereby ensuring that students see the link between applied research activities with firms and skill acquisition as being directly related to supporting firm-level innovation.

Given that the explicit mandate for colleges in Ontario is to ensure that graduates are prepared for the workforce (Ontario Ministry of Attorney General, 2003), our discussion of skills and innovation literacy must be grounded in how these skills and attributes may affect downstream economic performance. This performance will need to be measured both in the firms that partner with colleges on applied research as well as in the graduates themselves. Both of these settings present their own measurement challenges.

There is a compounding variable in the readiness of a college to engage in applied R&D with partners. There are therefore two intertwined variables with each needing discrete treatment: i) the college system's readiness for and effectiveness at applied R&D, and ii) the acquisition of innovation skills in students and graduates. Together, these variables have a downstream effect on firms. The readiness of the college system to engage in applied research is beyond the scope of this article, but what follows is a measurement framework for linking applied research services to skill acquisition and demonstration, with consideration given to downstream impacts on firm-level innovation.

A combination of performance measures and client feedback is seen by evaluation experts as optimal for ensuring quality, productivity, and return on investment (Kahn and McGourty, 2009). There is promising potential for statistical analysis of the link between end-user outcomes and college activities that relates to measuring outcomes of applied research conducted with firms. Given the nature of the activities undertaken within the scope of the CCIP, the evaluation of applied research should focus on practical outcomes (United States National Research Council, 1999). An outcomes management framework is necessary for us to effectively demonstrate the value of applied research in colleges to Canada’s science and technology enterprise. Key audiences include (van den Berg, 2012):

  • Government (funds provider)
  • Agency (funds delivery, investment choices)
  • University, college research and innovation administration
  • Students
  • Firms

Each of these audiences requires specific measures and instruments. All need a consistent focus on outcomes.

There is a well-developed framework for linking objectives, activities, and outcomes: the logic model. Logic models are well-established frameworks for evaluation, and they are used widely within the Government of Canada. For example, the Treasury Board of Canada Secretariat publishes detailed descriptions of their approach to performance management, including the use of logic models. A program logic model can aid us in defining and articulating the links between the activities, outputs, and outcomes associated with applied research and the acquisition of skills by students. Such links are important to all audiences listed above. Tracking outcomes against stated objectives lets us link inputs to outputs and ultimately outcomes and impacts.

The following components of a logic model are used to measure progress towards a desired objective over time:

  1. Purpose: the strategic aim/objective of change/effect to be measured
  2. Inputs: the parameters/projects in which we invest to effect change over time
  3. Activities and participants: what is done and who does it
  4. Outcomes/impacts: what happens over time. These objectives are typically structured as immediate, intermediate, and ultimate outcomes. The ultimate outcomes are directly related to the purpose.

These elements of a logic model provide a tool for linking the applied R&D activities supported by colleges for firms and the skills students gain as a result of engaging in this form of experiential learning. A key aspect of this approach is that it allows us to test – over time – the acquisition and demonstration of skills by students, through to downstream innovation support in firms after the students have graduated. Thus, this approach allows us to develop a logic model for training highly qualified and skilled personnel (HQSP) through applied research, as shown in Table 1.

Table 1. A logic model for training highly qualified and skilled personnel (HQSP) through applied research


Foster innovation literacy in students while improving firm-level innovation






Faculty with interest in applied research

Funding to support student engagement in applied research

Projects and placements with firms

Business development to support firm engagement

Projects and innovation support needed

Students: GISAT (pre/post project engagement measurement)

Firms: TO Next survey


Project involvement through curricula, paid work, internship

Project engagement and support of students' learning

Activities are defined with the taxonomy (above) as per applied research services offered by colleges


Students produce products, prototypes, services for firm partners, as linked to learning outcomes

Products, prototypes, services created for market entry

Skills articulated (students) against output metrics (firms) collected as part of applied research activity

Immediate Outcomes

Students gain innovation literacy skills; can articulate these and demonstrate proficiency

Firms commercialize new products or services

Students: Awareness of skills acquisition; Number of students engaged in applied research

Firms: Successful market entry of new product or service

Intermediate Outcomes

Graduates gain jobs as a result of gaining innovation literacy skills

Firms that employ (or startups started with) students with applied research experience have greater innovation capacity

Students: Number of jobs and startups

Firms: Number of employees hired

Ultimate Outcomes

Graduates are more likely to start a company or work on innovation activities within a company

Partner firms innovate, conduct R&D, and bring new products and services to market

Students: GISAT measured over time, as linked to employer innovation performance

Firms: greater likelihood of participating in innovation and enhancement of the Canadian innovation system


Measurement components

There are several components to our logic model for measuring the acquisition of innovation literacy skills and their deployment in firms post-graduation. The components presented below are from work conducted to date on measuring innovation literacy in students and graduates:

  1. An innovation skills measurement tool: The Conference Board of Canada’s General Innovation Skills Aptitude Test (GISAT) offers a tool to measure the acquisition of skills and their application in firms. The Conference Board of Canada has recently updated their Innovation Skills Profile, which provides a basis for understanding the kinds of skills relevant to fostering innovation in firms and how these are related to applied research activities as noted above.
  2. General and college-specific key performance indicators (KPIs): The provincially mandated KPI survey offers a rich dataset that lets us examine how students and graduates feel about the acquisition of the skills that comprise innovation literacy. We can use the KPI survey questions that are relevant to skills acquisition to link to innovation literacy skills acquisition as measured by the GISAT. Whereas the GISAT can be deployed specifically to students engaged in applied research, the KPIs provide a context or baseline against which to measure the general population. In addition to the KPIs as provincially mandated, colleges are allowed to put in five college-specific questions asked only of their students and graduates. George Brown College has included questions that ask students and graduates if they participated in an “Applied research project/course project with industry” and if so, what is their level of satisfaction with the experience. We also ask this of employers.
  3. Toronto Next survey: In October 2012, George Brown College released the results of a survey of Greater Toronto Area firms and their understanding of innovation and productivity, and the inputs required for these. Toronto Next: Return on Innovation (2012) gave us several key insights into how firms in the Greater Toronto Area value productivity while not necessarily valuing the inputs required for it: innovation skills, skills training, R&D, and investments in new equipment and technology. Elements of the Toronto Next survey will be redeployed to partner firms who are engaged with colleges in applied research. The resulting data will enable us to gain an understanding of a firm’s level of interest in and understanding of the innovation inputs that lead to productivity outputs.

The logic model for training HQSP through applied research requires the collection of metrics from both within and outside the colleges. Internal measurement enables a view of the form and function of the applied research system in colleges. External measurement allows us to track the outcomes and potential impacts the activities lead to, outside of the colleges themselves and in the larger community. For colleges, each set of measurements has its own purpose. The internal measurements provide a view toward program delivery, standardization of applied research services and student experience and the acquisition of innovation skills; the external measurements ensure that outcomes and objectives as stated by the CCIP are met and that performance can be managed accordingly. Thus, although there is consonance between both external and internal views of the logic model, and there will be useful overlap in the instruments we use to ascertain value and return on investment, it is important to measure indicators relevant to both internal and external perspectives. Certainly, the internal outcomes relate to the ability to deliver on the external outcomes. Both are necessary; neither alone is sufficient. We invite discussion and debate on the most useful and useable way to measure the acquisition of innovation in graduates and how best to demonstrate the results to external audiences.

Why It Is Important to Measure the Impact of Applied Research in Colleges

Measuring the effectiveness of the applied research system in colleges is relatively new: it was formally supported through the CCIP starting in 2007. However, as yet, there are no externally valid and reliable data to show impact. A report by the Council of Canadian Academies (CCA), titled “The State of Science and Technology in Canada” (2012), provides “a thorough analysis of the scientific disciplines and technological applications where Canada excels in a global context. It also identifies Canada’s science and technology strengths, regional specializations, and emerging research areas”. The expert panel that prepared this report was charged with ascertaining Canada's strengths in both basic and applied research. Colleges were included given the focus on federal funding of applied research in the college system since the publication of the Industry Canada's strategy report: "Mobilizing Science and Technology to Canada’s Advantage" (2007). The CCA's 2012 report was a thorough and in-depth, evidence-based analysis of Canada’s science and technology capacity. The measures for basic research are well defined and resulted in strong observations about research excellence. The measures for applied research, however, were limited, particularly when it comes to technology development (CCA, 2012). And, although the college system was included in the report, the data for applied research conducted in colleges are not well defined:

               "Canada’s colleges and polytechnics have been undertaking an increasing amount of applied S&T [science and technology] in recent years, often in cooperation with local businesses. Due to the growing importance of this activity to their overall role in the Canadian higher education landscape, colleges and polytechnics are now actively monitoring and recording many metrics related to applied S&T outputs. […] Most of these other sources of data on applied R&D activity in Canada’s higher education sector and public research organizations are not broken down by the field or type of research. As well, in many cases, data are available only for specific institutions, sectors, or regions, and are not available consistently across the country. As a result, while general statistics of this kind may illuminate certain facts about Canada’s applied R&D strengths in specific institutional settings, their piecemeal nature precludes a systematic identification of Canada’s research and technology strengths. The Panel thus concludes that there remains a need for more systematic and detailed data collection of metrics related to applied research and technology development activity in Canada." (CCA, 2012)

The challenge faced by the college system, then, is to come up with measures that can be used for future assessments. These measures will need: i) to be easily and consistently applied and collected across the country; ii) to be segregated by industry; iii) to focus on impact and outcomes (i.e., not just activities); and iv) to stand externally as viable measures of success. It will thus be increasingly important to link applied research activities to student learning outcomes given the growth trajectory of applied research in colleges. In so doing, we can innovate innovation by taking a specific, proactive, and instrumental approach to fostering business innovation and skills acquisition gained through work experience in applied research.


There are two key outputs for colleges engaging in applied research: i) the support of firm-level innovation and ii) the training of highly qualified and skilled personnel, who gain innovation skills ancillary to their program outcomes. As noted above, these skills are directly related to the provision of applied R&D services to firms. The applied research services offered by colleges offer a strong platform on which to base innovation skills, and measuring these skills over time is strongly related to the success of the college system and its ability to provide for the innovation capacity of Canada. The learning outcomes associated with college education, as augmented by experiential learning such as applied research, will lead to greater innovation capacity in partner firms, as well as those firms that employ graduates equipped with innovation literacy.


This article has benefitted from the input of many colleagues and reviewers, including Ken Doyle, Bert van den Berg, Kevin Stolarick, Suzanne Dwyer, and Dawn Davidson.

Share this article:

Cite this article:

Rate This Content: 
No votes have been cast yet. Have your say!

Keywords: applied research, business innovation, college, experiential learning, innovation literacy, logic model, polytechnic, skills, student research