Too much data and not enough insight, a common complaint amongst many organisations these days. Recent events here in the UK have shown that people can interpret the same data in very different ways, and that predictive insights from raw data can be misleading.
Organisations often find themselves struggling to cope with the influx of data and often fail to translate the data they receive into actionable trustworthy insights.
So how should your organisation use analytics to organise and empower decision making?
Here is my take on Action from Data based on the DIKW pyramid described here:
https://en.wikipedia.org/wiki/DIKW_Pyramid
Raw data is transformed into actionable insights through the application of context. This process happens at several levels through the information pyramid.
Data
Data points are often described as the objective, immutable values of things that are being measured. On their own they represent simple statement of fact conveyed as a message, a signal or a symbol. A data point may be incorrect, but it is still factual.
The problem with data is that it is increasing exponentially in volume and arrives with increased speed, in a bewildering variety of formats and of varying quality.
Storage is only part of the problem, it needs to be processed before it becomes useful for analysis.
Information
Data is translated into information by adding additional descriptions to it.
As an example a database would typically be used to add context through the use of a Schema. This schema would provide a description of the data and assign a data type to the data.
The data point has now been described as an information item because it is data that has been placed in a context of units and it has been named.
The schema may also impart additional descriptions (assuming we are storing the data in some kind of structured store). The data point could be further described which gives more context about the single data value.
Not all databases hold information schemas directly, schemas are applied at read time, field descriptions are sometimes held separately, or as a result of their column order in (say) a comma separated file.
Information is merely data that has been described to give meaningful values. So far, so straightforward.
Managing all this information into knowledge, and from knowledge into insight and action is where the fun part starts. By fun, I mean that this is where the ‘real’ work of analytic interpretation is done.
Knowledge
Knowledge is defined as ‘awareness or familiarity gained by experience of a situation’. Knowledge can be created as a result of processing or procedures.
- Knowledge Processing
In order to convert this information into knowledge the information must be related to some context in a way that provides expertise within a particular domain. As an example, our information needs to be processed to give it meaning and to allow comparison between other values in order to be fully understood.
To do this we need to ascertain the following:
- Identify the goal of the analysis. What purpose is the knowledge gained in the exercise to be used for? In order to do this it is important to seek the input of customers, business domain experts, subject matter experts, industry insights and highlight available datasets that will indicate the direction that the research will take.
- Identify the information that is to be used in the research effort. Relate the information to a time series, product grouping, customer and/or a geography. Knowledge of how items change across these dimensions is important as many business questions can usually be resolved to Product-Market(Customer)-Time questions.
- The rise of big data and analytics has broadened our access to knowledge by allowing automated systems to discover, describe and present many more data sets into the decision making jigsaw.
- Store and aggregate the data at the lowest grain of detail that is possible. This allows it to be aggregated up if less detailed analysis is required.
- Ensure the information values are consistent with the domain. This will take the form of cleansing, validating, mastering and verifying data to ensure that analysis is not skewed or influenced by incorrect values.
- Identify any calculations that need to be made to place our information points in context and uncover any dependencies between the information points. e.g Unit Price = Sum(Raw Material Cost), Profit= Sum(Sale Price-Unit Price).
- Search for relationships. Knowledge management systems, databases or spreadsheets can be used to provide comparative relationships so that the information points can be built up and compared with or analysed alongside information from other domains. The relationships between information points and datasets give the analysis new vectors from which to explore the domain in question.
Combining information points with other related information points tells us something new about whether this value was typical, or consistent across other information sets, and allows the analyst to propose and draw conclusions about the areas of interest.
- Procedural Knowledge
Knowledge creation and interpretation is based on synthesizing the processed information, business process and the procedural knowledge and logic used in organisations.
This kind of knowledge, however, is subjective, sometimes elusive to describe and is based on the knowledge of the experiences and environment within which the analyst is working. This is often embedded in what organisational personnel ‘know’ and have documented about the way the business works. This is also why many analysts are subject matter experts within the business.
Although it is possible to utilise machine learning and managed repositories to populate and process organisational knowledge the same ‘version of the truth’ can result in widely different knowledge interpretations, especially when this grounded information is used to project forward to making predictions.
That is, knowledge presents A version of the truth, not necessarily THE single version of truth offered up by the data.
Individuals and organisations use knowledge that has been discussed and justified with others to present a co-ordinated viewpoint that appears to be a true representation of the information. Knowledge is about application, that is, it involves the ‘know-how’ of taking actions based on data and information.
This synthesis between the processed information and the procedural information is the point at which interpretations of an identical information set can diverge, because it is dependent upon the subjective interpretation of the researcher.
Insight and Action
The ‘How’ of the knowledge is applied to support a world view of ‘Why’ such a pattern is seen. Often referred to as Wisdom, for this article I have chosen to subdivide this into Insights and Actions.
Insights provide a framework within which the knowledge gained has promoted an understanding of how the problem domain works. Analysts and business leaders are able to see why the information that has been collated describes the patterns that it does, and form an opinion that interprets and (to a certain extent can predict) what might happen in future.
Businesses will be particularly keen to use the insights gained and the trend knowledge that has been discovered to build a shared understanding into pricing models, business strategies or projections of sales and costs.
The organisation uses knowledge to provide insight by taking into account ethical, political, regulatory and strategic constraints and considerations to frame the possible reasons for the information distributions.
This understanding can be used to rule in or rule out the various actions that could be taken, allow further research to be undertaken, or be assimilated into the strategic direction that the business wishes to pursue. Insight helps the organisation to understand the problem domain and helps decision makers have the confidence to decide which actions to take.
Summary
The information pyramid is a simple and useful visualisation tool for assessing the effectiveness of business analytics information systems.
It provides a viewpoint with which to assess the maturity and effectiveness of automated and manual data collection and interpretation and recognises that the layers of the pyramid are qualitatively different from each other.
It is the role of technologists in the business to empower users to access the data by ensuring that it is gathered, accessible, protected, cleansed and persisted in formats so that the business is able to mix, match, consume and understand to support the processes and knowledge management that underpin business decision making.
It is the role of the analysts in an organisation to research, explore and frame the information points into knowledge and insights that are relevant to the problem domain. Good analysts will stretch the understanding and knowledge of how their organisation and environment works and should always be seeking innovative ways in which to blend the information across information sets to build new insights.
It is the role of business leaders to utilise information capabilities to fashion the action and strategy of their organisations. Context is all in a business and it still remains with business process owners to apply judgement in how they evaluate the output of analytics towards coherent, relevant, agile and valuable decisions.
The actions resulting from the insights will be framed by not only the insight but on the constraints within which business choices are being taken. It is the role of business owners to utilise the knowledge gained to critically analyse whether action is needed, and which actions and directions to pursue.
Feedback loops are present within the pyramid whereby changes at each level affect all of the layers beneath it. Taking action will result in new or changed insight, knowledge, information and data. The dynamics of the information continuum ensure that the interpretation is never fully realised, and business leaders need the ability to interpret and re-interpret the data, information and knowledge to inform future actions.
Information with Insight 