From Data To Data Insights
Companies are betting big on data, and are seeing payoffs from these investments. According to MIT Researcher Yang Lee, 65% of Chief Data Officer positions have been established within only the past 3 years. It’s clear that we’ve entered a new era where there is a strong desire to convert data into data insights. We’re often asked to help teams gather insights from data, and recognize the importance of alignment between business and engineering teams. This is important for two major reasons. The first reason is that business team adoption is a key factor in determining whether data insights will be valuable to the organization. The second reason is that understanding business goals and keeping them in sharp focus is critical in order for data scientists, engineers and analysts to design and scope projects, and make sure the desired result is achieved in the end.
Because the number of questions we could ask about data from beginning to end are infinite (and often interesting), a focus on goals prevents engineers from ‘flapping in the wind’. This allows teams to complete projects more quickly and iterate over problems faster. It also creates an important barrier between what we’re trying to do and how to do it. A friend recently told me that some of her managers pulled her into their office and told her ‘We need big data. How do we get big data?’. This approach of putting the solution ahead of the problem is precisely what we are aiming to avoid.
Maximizing Benefits / Minimizing Problems
Developing data insights often requires multiple iterations, essentially while minimizing a problem (ex: people leaving a website) while maximizing a benefit (ex: more clicks, purchases, etc..). Each iteration, we hope, will get us closer to maximizing the benefit of whatever we are trying to do. At the end of each iteration, we may have different questions and the goals can change, and that’s okay. But it underscores the further need to document and make the goals crystal clear.
The main focus here is the ‘whatever we are trying to do’ — in order to do this the engineering and business teams must be aligned on the context of what the problem is, and the desired end goal.
Outlining the Goals
In Max Shron’s Thinking With Data, he introduces the acronym CONVO to help outline some of these important up front steps: Context, Needs, Vision, Outcome. Context: understand how the findings will be used, who the decision maker is and generally, why they want to know this information. Needs: what are we trying to solve, ex: A/B testing to demonstrate that one is more effective than another, or how effective is a given campaign? Vision: what will our project look like when we’re done? Here, we take a cue from graphic designers and create mock ups, graphs, or statements that look like what the final result will look like, and the business owners can look at them and demonstrably say, yes that’s what I want. Outcome: how the insights will be used, and how the value will be measured.
There are several variations on “The Data Science Hierarchy of Needs”, this one has a similar emphasis on goals (though many omit it completely in favor of things like tools, data cleaning and visualization).
These are fundamental ideas, but we’ve seen enough cases where they are not identified up front and result in frustration and confusion at almost every step of the process.
Using the Right Tools
When the goals are clearly outlined, it’s possible to make decisions about which tools are best to solve the problem. Time, performance, and size components can help dictate more tangible things like which database to use, whether a cluster is needed or the analysis can be performed on a desktop running R or python, whether the process runs in realtime or is scheduled to run overnight, use the data warehouse or the production database, NOSQL/Sql. Is it a one time discovery or is it ongoing?
Today we have more data and better tools, making it the perfect time to derive valuable insights from data like never before. Carefully planning and staying focused on goals and outcomes is critical to success in data insights.
Find out how Pythian can help you maximize your business outcomes, and leverage your data with Data Enablement.
Thinking With Data by Max Shron
Succeeding With Data Published by NewVantage Partners