When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts advanced to the stage of science.
William Thomson, 1st Baron Kelvin (1824–1907)
Mathematical physicist and engineer
Current research indicates that the idea evaluation processes of many firms are ad hoc or intuitive, with very few firms having defined methods. We propose a new approach to select the best ideas to pursue amidst different probable versions of the future. In support of "front end of innovation" processes, the approach emphasizes the formation of requirements for any idea that can be prioritized and measured against possible future worlds. This approach is currently conceptual; future work will develop the approach into a methodology that can be tested using real-world problems. This article will be relevant to those who are exploring novel methods and approaches to selecting the best idea within their particular domains.
Idea generation appears to be a simple task, and in many ways, it is. Anyone can have a new idea that fundamentally changes a technology, a society, or the world at large. Equally, anyone can have an idea that offers no value. Without acting on these ideas, how can we know whether an idea will change the world or just be another failed attempt to reach an objective? The reality is that we cannot. Instead, this article begins to conceptualize a method to evaluate ideas across a set of varying alternative futures or possible worlds.
Idea generation is the first step in virtually all innovation processes. Companies are formed and new products are made because a seemingly great idea was identified. Publishers may consider hundreds of manuscripts, few of which, after further development, make it to print. Organizations developing new branding may create dozens of alternatives and select the top ideas to refine. Software or technology firms developing a new product may propose many ideas before selecting one for production. Generating ideas that lead to innovation processes plays a critical role in a firm’s success.
Even as organizations put more resources into the innovation process, 80-90% of new product launches fail (Görs et al., 2012). Apple, a technology company, uses their own innovation framework (Apple’s New Product Process) that consists of the following front-end activities: i) ideation, ii) product start-up, iii) prototyping, iv) and group evaluation (Busche, 2014). However, even with this innovation process, Apple arguably has not produced any disruptive technology since the iPod. The Apple Watch, the 12-inch MacBook, and the iPad Pro have been described as “products without purpose“ (Wilcox, 2016).
There is no shortage of novel ideas at IBM, which has an annual R&D budget of $6 billion and generated 6,180 U.S. patents in 2011 alone. To decide which ideas should be taken through their innovation process, they follow a series of steps that can be distilled to the following:
- Generate ideas and use collaborative tools to solicit feedback from stakeholders.
- Take the top ideas and establish compelling customer offerings.
- Evaluate the ideas deemed worth piloting.
- Run a pilot project.
- Start production.
They also use idea marketplaces such as online open innovation "jams". Additionally, they created an innovation lab that has funded 25 projects to the prototype stage from 400 ideas submitted (Quitzau, 2013). Yet, even with these processes, IBM has experienced ongoing decline in new product growth over the last 16 quarters (Fox, 2016).
Why do such established and resource-rich companies continue to struggle turning innovations into successes? To answer this question, we focus on the front end of innovation activities (FEoI), which can be separated into three operations (Kempe et al., 2012):
- Enrichment: developing raw ideas
- Evaluation: estimating the likelihood that the ideas can reach their goals
- Selection: choosing an idea to execute
Research has already identified that FEoI activities are the most important stages of innovation processes and that successful outcomes are linked to the quality of the FEoI (Cooper, 1988; Elerud-Tryde et al., 2011). Generating new ideas, selecting the best ones, and taking them through the innovation process has considerable failure rates and high costs (Buyukozkan & Feyzioglu, 2004). The disparity between innovations that win and those that lose in the market are predominantly due to the quality differences in front end of innovation activities (Stevanovic et al., 2012). The existing literature is silent on the evaluation of the ideas, especially when there is a large set that must be assessed. If an evaluation methodology can be established to deal with the complexity of determining the likelihood of an idea reaching its objective then, selecting and executing the best idea will lead to an increase in successes.
Despite the work that has been done to improve back end of innovation processes and execution, innovation success rates have not increased in any measurable way. Companies and organizations continue to encounter the same dilemma: they require a reliable and repeatable idea-evaluation method for selecting ideas.
According to William Thomson's words in the opening quotation to this article, until an idea can be measured and expressed numerically, our knowledge on the subject will be limited. Innovation is still a hubristic process. Until the difficult work of figuring out how to measure and express this complex process numerically is done, deep understanding cannot occur and major improvements cannot be made. This is not to imply that a fully quantifiable solution exists. However, we aim to redress the commonly accepted notion that the front end of innovation is mysterious and cannot be managed. We seek a quantifiable mechanism to sharpen a very hazy idea-evaluation process.
The study of innovation is still in its infancy precisely because there has been a lack of research in quantifying idea evaluation (Elerud-Tryde & Soonvald, 2011). Measurement theory, the study of assigning numbers to objects and phenomena (Roberts, 1985), as it relates to innovation management, has focused on the back end’s output performance (Adams et al., 2006). This is due to the fact that back-end processes (i.e., production and marketing) are easily identified and organized with budgets, personnel, and repeatable processes. Studies by Kettunen, Ilomäki, and Kalliokoski, (2007) and Kim and Wilemon (2002) indicate that the FEoI has generally been left unstructured and uncontrolled whereas the back end has been structured. Many practitioners rely on heuristics and tacit knowledge to evaluate ideas because the necessary components are difficult to quantify. Most people have a tendency to place disproportionate weight on specific pieces of information they use to govern their thought process (Bonabeau, 2003). The human brain is known for injecting familiar patterns into new situations even when they are inappropriate (Bonabeau, 2003). Once this occurs, a bias is formed and new information is adjusted to reflect what the person believes to be true (Loosemore, 2013). For example, when estimating the cost of a new design innovation, people assume that the cost will be the same as an older design that was similar. The reason why this approach does not work is because, when innovation processes are executed, there is no guarantee that the same agents (decision makers) involved will interpret the variables the same way every time. Eschewing this notion allows visionaries such as Elon Musk to redefine industries. Elon did not assume that the cost of building a rocket would be the same as rockets that came before him. Looking at the design of rockets from first principles allowed Musk to drastically decrease the cost to build and deliver cargo more effectively than established companies such as Boeing and Lockheed Martin (Anderson, 2013).
Over the last twenty years, the semantics of innovation has become progressively abstract as the word and its signifiers become increasingly intangible and unbounded (Moldaschl, 2010). We have not been able to improve FEoI techniques in any measurable way because there is little empirical evidence to guide academics or organizations towards activities that contribute positively to front-end performance (Markham, 2013). We lack a theory that can predict the outcome of an innovation cycle or determine which ideas are better suited towards specific future outcomes thereby reducing the risk of the innovative process. Current approaches in rating an idea are generally motivated by specific situations (e.g., how to cut wait times for buses during rush hour times) (Kudrowitz & Wallace, 2013). By introducing an approach that treats idea evaluation as a quantifiable process, we hope to contribute some ideas that may lead to a deepening of knowledge and providing practical and effective tools for practitioners to achieve greater success in selecting ideas for innovation.
Idea evaluation and selection is a problem that takes place in the present. Implementing all ideas to determine the best one is impractical and it remains impossible to select ideas from the future; thus hindsight is not a viable decision-making tool. Instead, we can attempt to predict the future and assess how well a given idea will perform within it. Given the number of variables at play, it is better to come up with a limited set of possible worlds against which to assess the quality of ideas. We are creating an idea evaluation method that analyzes possible worlds to select the idea with the highest likelihood of successfully reaching its objectives.
To represent a possible world, its characteristics must first be defined. A characteristic is composed of a property value pair tied to a particular world. It asserts that a property has a specific value within that world. Given a set of worlds, each one will differ by at least one characteristic from every other world. We can represent a future world as a set of properties extracted from general categories. For example, the environment is a general category, climate is a property of the world, and temperature, precipitation, humidity, and UV indexes are sub-properties of climate. These properties remain incomplete without a viable way to distinguish them among different worlds. Nearly every version of the future will have a climate, therefore, in order to characterize them, a value must be associated with these properties.
The next step is to determine the requirements of the idea. These requirements will be a set of world characteristics that maximizes the idea’s chance of success. The idea’s requirement will be divided into two parts: the characteristic's numerical value and the characteristic's importance to the idea.
Finally, we need a method to measure the world’s ability to satisfy the idea(s). To do this, we use an artificial construct from mathematics called a utility. A utility is a numerical value used to represent the amount of benefit that is achieved through the implementation of an idea. A world better suited for a particular idea will allow that idea to yield a higher utility than that of a world that does not meet the idea’s requirements. Utility is used to allow the evaluator to measure the likelihood that the idea will reach its objective. All ideas have a corresponding world where their implementation is best suited, though; the probabilities of those worlds existing are independent to the idea.
As the value of the characteristics change from one world to the next their proximity to the requirements also change. For example, suppose that a certain idea has the highest likelihood of success if the prime business rate has a value of 2.5%; one of the idea's requirements will be that the prime business rate is 2.5%. This implies that the idea's utility is maximized in a world with a prime business rate of 2.5%. However, it is quite possible that other worlds with less than ideal values can return a positive utility, even if the returned utility is less than optimal. If no conditions are minimally met, the utility becomes 0. Consider a world where the interest rate is 3% instead of 2.5% as stipulated by the requirement. It is likely that the idea will still be useful. Figure 1 shows this representation and depicts the utility of the idea as the interest rate value changes.
The goal is to select the idea that has the highest utility across the most probable worlds. Just because an idea may be great, the conditions necessary for its successful outcome is based on how the future unfolds, and that is independent of the idea. For example, the iPod was successful because consumers were moving away from portable CD devices. Because the likelihood of consumers making the shift away from portable CD devices was greater than the likelihood that consumers would continue using portable CD devices, the iPod was the best product for Apple to pursue. The unfolding of different futures has associated probabilities, and we want to select the idea that performs best (has the highest utility) in the most probable versions of the future.
Figure 1. The utility value of an idea across characteristic values
In the next section, we illustrate the key aspects of this approach through an example scenario.
Example Scenario: Innovation in the Oil and Gas Industry
In the present world of low oil prices, recovery of heavy crude oil and bitumen from the Canadian Oil Sands based on steam-assisted gravity drainage (SAGD) is no longer economically viable (Findlay, 2016). Even when oil prices were high, SAGD was only applied in the best quality oil sand reservoirs, leaving lesser quality reservoirs commercially unviable. Due to the further decline of oil prices, new high-tech ideas for in situ recovery are necessary to maintain profitable operation.
In this example scenario, let us suppose that an engineer introduces Idea A. The profitability of Idea A is dependent on the following requirements:
- The prime business rates remain at 2.5%.
- The price of oil remains above $25 a barrel.
- Oil sand production is not banned by the government due to environmental concerns (Binary choice 1/0).
Another engineer introduces Idea B. Idea B is dependent on the following requirements:
- The cost of steel drops to US$280/tonne.
- The price of oil reaches at least $46 a barrel.
- Surface oil sands can still be found (probability range).
In order to select either Idea A or Idea B, four steps need to take place:
- We compute the probability of each possible world occurring based on the likelihood of the characteristics being true. Given that we do not know what the future will be, we establish a set of possible worlds and determine their associated probabilities of being realized. To do this, we use the following equation to calculate the chance of each of the worlds characteristics occurring according to the variables defined in Table 1:
Table 1. Variables and definitions used to formulate the possible future worlds
For simplicity, we will assume that only one future version of the world is possible: either it will satisfy idea A or it will satisfy idea B. In reality, there will be a number of worlds to evaluate (Figure 2), though the exact number will differ from problem to problem.
Figure 2. Visual representation describing worlds to evaluate
To calculate the probability of each characteristic being realized we will have to make predictions from the available data or use predictions from a trusted source. In this example we use data from the World Bank. According to the World Bank Group’s Commodity Markets Outlook (2016) the price of oil is estimated to be $41/barrel in 2016 (World Bank, 2016). According to the Bank of Canada prime business rates will remain at 2.7% (Bank of Canada, 2016), and it is very unlikely that oil sand production will be banned in the near or distant future. The price of steel is expected to be US$365/tonne (World Bank, 2016), and the oil sands that remain are considered difficult oil. This information shows that the most probable possible world has the following characteristics:
- Oil is $41/barrel
- Prime business rates are 2.7%
- Oil sand production continues but is difficult (non-surface)
- Steel is US$365/tonne
- Oil sand production is not banned
Therefore, the world with these characteristics is most likely to occur. In practice, probabilities ( ) would be assigned to each characteristic to determine the probability of the world occurring.
- We compute the utility of an idea in a particular world as a function of the requirement and its associated world characteristic weighted to the probability of that particular world occurring:
To determine the utility of the idea within a particular world, we will have to create a function that can compare the requirements to the characteristics of the world and return a numerical value based on how well the world satisfies the requirements. Each requirement is weighted by importance such that the utility reflects the ideas hierarchy of needs.
Based on the information from step 1, there is a higher likelihood that each requirement of idea A will be met in the most probable world. This yields a higher utility than idea B given that the world necessary for idea B to be successful has a lower probability of being true. The utility will be described by a numerical value. The higher the utility, the higher the value.
- We find the expected value of the ideas utility by summating all of the utilities from each possible world:
After calculating the utility of each idea in each possible world, we will determine the ideas expected utility, which will give us the expected performance of the idea based on the possible worlds that may occur. This is an important step because we do not know which world will realize. By calculating the expected utility, we will determine the best idea to select based on the range of possible futures most likely to occur.
- Finally, we select the idea. Because there was only one world in this example, the expected utility is the same as the calculated utility for that world. Idea A is most suitable for the possible world most likely to occur. Thus, idea A has the highest utility and should be selected ahead of idea B.
The oil sand example simplified the utility function by only considering five requirements. In reality, an idea has many requirements and the utility is determined by the totality of those requirements being satisfied. Most of the literature on idea evaluation that we found considers 4–8 criteria before making a decision (e.g., Girotra et al., 2010; Gutiérrez et al., 2009; Martinsuo & Poskela, 2011). A major part of our future research will be spent identifying a methodology to determine the criteria necessary to form the correct amount of requirements necessary to calculate an ideas utility.
Using our example, the price of oil per barrel should remain greater than $25 in order for idea A to be successful, however, this is only the second of three requirements. If one requirement is not satisfied, the idea's utility will be reduced. If the world’s characteristics are unable to meet any of the idea’s requirements, such as is the case with idea B, the idea's utility within that particular world tends towards zero. Figure 3 illustrates how an idea's utility, when tied to numerous requirements, responds to different world characteristics.
Figure 3. Example of the individual and collective effect of a single requirement on the utility of an idea
Many of the variables that are used to model the likelihood of characteristics being true are stochastic in nature. Though not everything is quantifiable, everything has probable outcomes. It is not necessary to model every aspect of the future to determine the necessary requirements for success.
Through our future research, we will create software that takes input from the decision maker and performs the necessary calculations to determine the utilities of the ideas. This is how our approach will be easily used by any decision maker to evaluate and select ideas within any domain.
In the early part of this century, attention has focused on exploiting ideas to generate innovations (Dooley & O’Sullivan, 2001). Significantly less attention has been paid to identifying the best ideas (Rindasu & Mihajlovic, 2008). Our approach is best suited for ideas being applied to innovation processes that are tied to specific objectives, such as radical (breakthrough) innovation, transformational (disruptive) innovation, market creation, and competitor disruption. Each one of these innovation types requires a detailed understanding of both the idea and the sort of world it will be applied to. Ideas and subsequent worlds that are better defined allow for better data collection because the decision maker knows what it is they are looking for. This approach may find itself useful in the generation of ideas by improving the quality of the best ideas by having the decision maker consider what ideas are likely to work well in some future world.
Given the novelty of this approach within the context of front end of innovation activities and the innovation process in general, we do not yet have direct evidence that this approach is feasible. However, available research indicates that people typically consider too few factors in forecasting and therefore, unfortunately, often produce rather simplistic analyses of possible outcomes (Dörner & Schaub, 1994). By evaluating ideas through a possible-world framework, we aim to enhance FEoI activities enough to reduce the number of failed innovation projects.
We present this approach at an early stage of development to encourage practitioners and decision makers to consider how calculating an ideas utility as an expected value across future worlds can lead to a rigorous approach in front end of innovation activities to improve the success rate of innovation processes. We also seek to generate, discuss and debate the best way to refine and test this approach, and to build relationships with organizations that wish to use their experiences and data as early users of our method. We are currently working with the innovation department at The Hospital for Sick Children in Toronto, Canada, to conduct historical recreations and analyses to develop our methodology. Using historical empirical data on the end-to-end innovation activities undertaken by the hospital to differentiate successful ideas from failed ideas will allow the performance of our function to be tested by finding the deviation of real-world behaviour with the expected results predicted by it. Of course, we will not know what the real future looks like until it arrives, so front end of innovation activities cannot benefit from hindsight directly. However, hindsight from past innovation activities can play an important role in developing methodologies to help organizations select the most promising ideas and in improving our ability to predict the future worlds in which those ideas will come to fruition.
Anderson, C. 2013. Rethinking Public–Private Space Travel. Space Policy, 29(4): 266–271.
Arrow, K. 1962. Economic Welfare and the Allocation of Resources for Invention. In The Rate and Direction of Inventive Activity: Economic and Social Factors: 609–626. Princeton, NJ: Princeton University Press.
Bank of Canada. 2016. Daily Digest. Bank of Canada, July 20, 2016. Accessed August 3, 2016:
Bonabeau, E. 2003. Don’t Trust Your Gut. Harvard Business Review, 81(5): 116–123.
Busche, L. 2014. The Skeptic’s Guide to Low-Fidelity Prototyping. Smashing Magazine, October 6, 2014. Accessed August 1, 2016:
Buyukozkan, G., & Feyzioglu, O. 2004. A New Approach Based on Soft Computing to Accelerate the Selection of New Product Ideas. Computers in Industry, 54(2): 151–167.
Cooper, R. G. 1988. The New Product Process: A Decision Guide for Managers. Journal of Marketing Management, 3(3): 238–255.
Dooley, L., & O’Sullivan, D. 2001. Structuring Innovation: A Conceptual Model and Implementation Methodology. Enterprise and Innovation Management Studies, 2(3): 177–194.
Dörner, D., & Schaub, H. 1994. Errors in Planning and Decision Making and the Nature of Human Information Processing. Applied Psychology, 43(4): 433–453.
Elerud-Tryde, A., & Soonvald, A. 2011. Proposing a Framework for Evaluating and Selecting Ideas in the FEI: A Case Study of Volvo Cars. Master of Science Thesis, Department of Technology Management and Economics. Chalmers University of Technology Göteborg, Sweden.
Findlay, J. P. 2016. The Future of the Canadian Oil Sands: Growth Potential of a Unique Resource Amidst Regulation, Egress, Cost, and Price Uncertainty. Oxford: Oxford Institute for Energy Studies.
Fox, J. 2016. IBM Is Not a Growth Company. Bloomberg View, April 19, 2016. Accessed August 1, 2016:
Girotra, K., Terwiesch, C., & Ulrich, K. T. 2010. Idea Generation and the Quality of the Best Idea. Management Science, 56(4): 591–605.
Görs, J., Horton, G., & Kempe, N. 2012. A Collaborative Algorithm for Computer-Supported Idea Selection in the Front End of Innovation. In Proceedings of the 45th Hawaii International Conference on System Science (HICSS): 217–226. Washington, DC: IEEE.
Gutiérrez, E., Kihlander, I., & Eriksson, J. 2009. What’s a Good Idea? Understanding Evaluation and Selection of New Product Ideas. In Proceedings of ICED 09, the 17th International Conference on Engineering Design, Vol. 3, Design Organization and Management, Palo Alto, CA, USA, August 24–27, 2009.
Kempe, N., Horton, G., Buchholz, R., & Görs, J. 2012. An Optimal Algorithm for Raw Idea Selection under Uncertainty. In Proceedings of the 45th Hawaii International Conference on System Science (HICSS): 237–246. Washington, DC: IEEE.
Kettunen, J., Ilomäki, S.-K., & Kalliokoski, P. 2007. Making Sense of Innovation Management. Helsinki: Teknologiainfo Teknova.
Kim, J., & Wilemon, D. 2002. Strategic Issues in Managing Innovation’s Fuzzy Front-End. European Journal of Innovation Management, 5(1): 27–39.
Kudrowitz, B., & Wallace, D. 2013. Assessing the Quality of Ideas from Prolific, Early-Stage Product Ideation. Journal of Engineering Design, 24(2): 120–139.
Loosemore, M. 2013. Innovation, Strategy and Risk in Construction: Turning Serendipity into Capability. London: Routledge.
Markham, S. K. 2013. The Impact of Front-End Innovation Activities on Product Performance. Journal of Product Innovation Management, 30(Suppl 1): 77–92.
Martinsuo, M., & Poskela, J. 2011. Use of Evaluation Criteria and Innovation Performance in the Front End of Innovation. Journal of Product Innovation Management, 28(6): 896–914.
Moldaschl, M. 2010. Why Innovation Theories Make No Sense. Papers and Preprints of the Department of Innovation Research and Sustainable Resource Management (BWL IX). No. 9/2010. Chemnitz, Germany: Chemnitz University of Technology.
Quitzau, A. 2013. How IBM Innovates: The Front End of Innovation in IBM. Presentation given December 2013. Accessed August 1, 2016:
Rindasu, V. C., & Mihajlovic, I. 2008. Idea Management for Organisational Innovation. Analele Universităţii Eftimie Murgu Reşiţa, XV(1): 398–404.
Stevanovic, M., Marjanovic, D., & Štorga, M. 2012. Decision Support System for Idea Selection. In Proceedings of DESIGN 2012, the 12th International Design Conference, May 21–24, 2012, Dubrovnik, Croatia.
Wilcox, J. 2016. Why Apple's Future Failure Is Certain. Beta News, January 20, 2016. Accessed August 1, 2016:
World Bank. 2016. Commodity Markets Outlook: April 2016. Washington, DC: World Bank.
Keywords: FEoI, front end of innovation, idea selection, innovation, multi-criteria decision making, possible worlds, real options, stochastic models