“Big Data” is big right now. It offers the promise of turning vast amounts of data into profits. Often, however, this promise is not fulfilled, because in order to reap the benefits of big data, you first have to know what the right data is. And that is a question that no AI can yet answer.
Not every business is ready for big data
Not every business is ready for big data. In fact, quite a few of those businesses that have tried to create a big data system have not attained the benefits that they were expecting from doing so.
The core problem is that people in business and people in data science have deep expertise in their respective areas, but not a lot of knowledge about the other. Imagine you were a CEO wanting to hire someone to build your data pipelines filling up a data lake and lay some visualizations on top of it. Would you know how to define the deliverables in a way to help your business? And if you were an individual tasked with building such a thing, how would you know what the company needed?
First, we need to differentiate between big data used for external purposes, like customer predictive analyses or data-as-product, and internal purposes, like managing the company. In this article, I’ll focus on the latter.
Of course, a CEO and data scientist can sit down together and chat to come up with a plan, but this is where things often break down. The vast majority of managers don’t really know how the work of individuals rolls up to the overall success metrics for the company. So when they ask a data scientist to build a big data system, they often just ask for what they currently do to be automated.
…automation of the status quo is bound to have gaps
If a company has not thought about what it is trying to accomplish, the metrics that track that, and how those metrics translate down into the organization to every individual, such automation of the status quo is bound to have gaps. As one plant manager put it before embarking on this task, “Management here is like a lot of people mowing a lawn. Everyone mows their strip really well, but no one is mowing in between the strips, and that grass is up to our waists!”
Metrics that grow organically in an organization are very unlikely to have any unified purpose behind them. One manager tracks what is important to her while another manager in a similar job tracks something entirely different. There might be value to one, both, or neither of those metrics as far as the company is concerned. Add a host of other managers, and you have a lot of metrics that are being tracked, and that could be automated and dumped into a data lake, but the probability that they have much to do with what the company really needs to be managed is vanishingly small.
The first step when embarking on a big data initiative has nothing to do with big data…
The first step when embarking on a big data initiative has nothing to do with big data, but with what the right data is for the company and its purposes. I show a method to do this in my new book Galileo’s Telescope. Once metrics to measure the success of the company are in place, and once these metrics have been translated down to individuals, you understand what data are needed internally.
I need to be clear about what I mean when I say “translate” the metrics. A common error in big data is to take results for the company, say a profit measure, and push that down into every management level. The problem is that as you get farther and farther from the top level of the company, it gets harder and harder to figure out just how that metric applies to you and what you can do to affect it. This translation happens at every level, and it means that profitability at one level becomes something like a production rate or preventive maintenance compliance at another. If I am a maintenance manager, I probably don’t know the exact formula as to how preventive maintenance affects profit, but I know it does and that I should manage it.
…even best-guess metrics…will lead to undeniable improvements
After translating company objectives into individual success metrics, you may find many can be done semi-manually now, and you don’t have to wait for a big data system at all. My experience is that if you get even best-guess metrics up and running across a company you will start learning massively important things about your own business that will lead to undeniable improvements. Bringing a big data system into that is much more likely to pay big dividends.
Another common error when building a big data system is to give managers at a higher level too much detail and the ability to drill down. If you roll up all the metrics in an area and deliver them to the manager, they will be overwhelmed with the number of metrics. Even worse, they might be tempted to manage using those metrics, skipping intervening levels of managers to directly contact Billy Bob on third shift to ask them why they scrapped lot 123456. This dis-empowers all those managers, indicates that you don’t trust them to manage their area, and has been codified in the language of business as “micromanagement.” Managers should only have visibility to their direct reports’ metrics, and no further.
Management allocates responsibility to those who are in the best position to do something about it
I know, I know, managers really want that ability to drill down. But this is a fundamental misunderstanding of what management is about. Management is supposed to allocate responsibility to those who are in the best position to do something about it. If your managers have the right data, you can trust them to react to those data before you need to, otherwise, why did you hire them? It is only when something rises to the scale of your level of management that you should need to react to it — something that your managers can’t see from their position. Anything less is their job, and if you try to do their job as well as your own, you end up failing in both.
Another reason big data is not for everyone is that much of business happens on a human scale. Even Amazon needs humans to make, buy, load, and deliver their products. Local process improvements might use big data to identify opportunities, but humans (at least for the near future) are the ones who have to prioritize, allocate resources for, and perform experiments and pilot ways of doing things better. Similarly, initiatives documented on a strategic plan don’t happen at computer speed. They happen over months or longer as employees put together teams, identify potential fixes, pilot potential solutions and implement these countermeasures. These are all “little data” questions that a big data system will not make happen.
These are all “little data” questions that a big data system will not make happen
In the end, big data can provide very valuable information for a business, but only when that business knows what their business is trying to accomplish and what that means for their employees. Bringing big data into a business that has no need for big data, or doesn’t already have a set of integrated metrics in place is most likely going to be a very expensive blunder.
