Data Science techniques are often used to gain insights. Insight is the understanding of cause and effect within a specific context. This is often based on the identification of relationships and behaviors using modelling to simplify a situation or scenario. Insight is gained when non-obvious approaches explain previously unexplained phenomena in a new and repeatable fashion.
Insight can lead to behavioural recognition, but the knowledge does not come from directly within the problem space. This applies to both organisations that use insight analytics to understand the external motivation and behaviour of customers, and customers that are able to find internal meaning and belonging in their interactions with organisations.
Psychological Insight
Two clusters of problems, those solvable by insight and those not requiring insight to solve, have been observed [1]. An individual’s cognitive flexibility, fluency, and vocabulary ability are predictive of performance on insight problems, but not on non-insight problems. Insight gives illumination and understanding by perceiving the nature of the problem from knowing that certain things are causes in certain scenarios. This information can be arrived at by combinations of introspection, acute observation, deduction, discernment, and perception.
Psychological insight is the act or result of understanding the inner nature of things or of seeing intuitively, and occurs when a solution to a problem presents itself quickly and without warning. It is the sudden discovery of the correct solution following incorrect attempts based on trial and error, sometimes described as a ‘Eureka!’ moment. Solutions via insight have been proven to be more accurate than non-insight solutions [2].
Human intelligence derives insight [3]. Specifically that insight involves three different processes (selective encoding, combination, and comparison) applied to problems. Selective encoding is the process of focusing attention on ideas relevant to a solution, while ignoring features that are irrelevant. Selective combination is the process of combining the information previously deemed relevant. Finally, selective comparison is the use of past experience with problems and solutions that are applicable to the current problem and solution.
Computational Insight
Mathematical insight processing has four stages [5]. First is preparation and framing to solve a problem; Second is incubating the problem (modelling), which encompasses trial-and-error; thirdly, insight occurs, and the solution is illuminated; and finally, the verification of the solution to the problem is experienced.
Raw computing power speeds up the trial and error process by factoring out the irrelevant facets of the problem, and condensing down the multi-dimensional problem space into one in which the human experience processes of insight can be used to work on the solution. Computers are used in the modelling process, and verification of the model is achieved by using statistical methods to assess the correlation, covariance and goodness of fit to the real world. Information systems are also invaluable in quickly assessing the reliability, validity and repeatability of findings.
A Data Science Process
Data Science is a problem solving exercise and there are two thought processes in play [4]. The first involves logical and analytical thought processes based on reason (cognitive computation), while the second involves intuitive and automatic processes based on experience (psychological reasoning). Research has demonstrated that insight probably involves both processes with second process being more influential.
Combining the relative strengths of computer logic processing with those of human experience suggests a process similar to the one below:
Summary
Although data science is driven by the application of computing power and statistical learning models to the transactions and interactions of online environments, the insights gained from such endeavours are initiated, shaped, gained and curated by the analysts, scientists, business people and engineers that participate in such initiatives.
Should we develop our people or do we develop our technology to gain insight from data science? Both are essential to the processes of gaining insight, and this suggests that an information system that is optimised to process for insight should be based on the concept of social machines, using the fluid intelligence and raw processing that computing systems do best, blended with the question framing and domain wisdom of experienced personnel.
References
[1]: Gilhooly, K.J. and Murphy, P., 2005. Differentiating insight from non-insight problems. Thinking & Reasoning, 11(3), pp.279-302.
[2]: Salvi, C., Bricolo, E., Kounios, J., Bowden, E. and Beeman, M., 2016. Insight solutions are correct more often than analytic solutions. Thinking & reasoning, 22(4), pp.443-460.
[3]: Davidson, J.E. and Sternberg, R.J., 1984. The role of insight in intellectual giftedness. Gifted child quarterly, 28(2), pp.58-64.
[4]: Lin, W.L., Hsu, K.Y., Chen, H.C. and Wang, J.W., 2012. The relations of gender and personality traits on different creativities: A dual-process theory account. Psychology of Aesthetics, Creativity, and the Arts, 6(2), p.112.
[5]: Hadamard, J., 1954. An essay on the psychology of invention in the mathematical field. Courier Corporation.
Information with Insight 