Terabytes of Truth

4 min read

With a growing volume of channels, content producers and social bubbles, it’s increasingly difficult to discern what’s true from what’s ‘fake’. The behemoths of media, news and politics are struggling to maintain reputation amid post-truth noise. The fear that entire countries are succumbing into irrationality seems justified. Interestingly, there is little discusion about how, and if, this massive culture shift bears any relation to enterprise BI, data, reporting and decision-making. How do companies decide in the post-truth world?

‘Truth’ is one of the sacred words of the Western world. It’s the pinnacle of science. It’s what pulled us from the Dark Ages. It’s the guide that separates us from delusion, madness and folly. It’s the basis of mutual trust. It’s what we wish our politicians respected. It’s the compass that the noble person follows. Or so the collective narrative goes.

So to say there’s something rotten in the Kingdom of Truth is discomfortingly close to blasphemy. To say it on national TV was easily my toughest interview moment so far. „But if all models are wrong, as you say, where’s hope that anyone will reach truth?“ asked the interviewer. And these are my two endless seconds to think how to say „there is none.“

My perspective is skewed. I work with big data. No matter how loaded that term has become, it means a really big pile of information. For instance, for a retail customer, we analyse 4 million business hypotheses a day. For Kiwi.com, it’s 50 million daily search requests. We literally deal with terabytes on a daily basis. And in a very significant sense, all that data is true. It’s a factual record of past transactions. And it’s not even that detailed.

This changes one’s perspective. For years now, the question „what’s true“ has not had much value for me. In the world of business intelligence, there’s an absolute over-abundance of truth.

So, for the truth-seekers out there, this is good news. Today’s technology is producing more truth than a human will ever be able to process, by orders of magnitude. There’s enough for everyone. In fact, we have reached the point of a hyperinflation of truth, Zimbabwe-style.

By definition, in a hyper-inflationary state, truth is losing relative value.

And there are layers and layers of truth within the data. Our typical enterprise customer logs in 30 000 sales transactions per day. Enriched with the basic product and customer information, the company receives some 360 000 new pieces of information every day, and we’re talking only about sales here. If we start analysing this daily volume, that is, looking for patterns and relationships within those 360 000 pieces of information, we are facing a space of…

…potential insights. That’s approximately the number of all grains of sand on Earth. (And we’re still talking about one company, one day). And that’s just sales data, which is relatively sparse. With IoT and its millions of sensors logging minutiae truths every few seconds, these numbers go through the roof again.

Who Else Has Noticed?

In the meantime, the world at large is discussing truth decay. The “fake news” phenomenon has escalated into a complete Donald Trump’s Fake News Awards. Post-truth became the Oxford Dictionaries Word of the Year in 2016. And Cambridge Analytica proved that a data-driven campaign can create fantastical realities. Twice.

It can at times feel like an intellectual apocalypse. As a breakdown of the venerable, time-honoured navigation by truth. A mortal threat to democracy, the Western way of life, and, above all, those highly skilled in Oxbridge-style debating. An uprising of the uneducated, who are unable to grasp truth, but ferociously vote with their like button.

The dramatic, fear-infused, post-truth debate has a strong parallel with what’s happening in the world of business intelligence.

Back To Business

The sheer (and exponentially rising) volume of available information says loud and clear: We no longer have a scarcity of truth. We have a life-threatening, disorienting over-abundance.

The business effect is managers receiving 70-page PDFs every Monday morning, full of charts and 80+ KPIs. Hundreds of reports that no-one looks at. Dashboards full of useless noise. Data lakes turning into data swamps. Business analysts drowning under requests. Business users spending hours drilling and or ignoring the reports altogether. “Decreased productivity” is a euphemism. We hear it in virtually every company we go to – and almost invariably on the third meeting when the ice melts a little.

Current business intelligence tools are not built for an avalanche of data.

The deeper effect is this:

When dealing with a million new pieces of information per day, the mere fact that something is true no longer constitutes a sufficient selection criterion.

Not only has so much data diminishing value. It’s the value of truth – as a navigation device – that’s vanished. It’s gone forever, just as the value of speaking Latin, clock towers and arranged marriages.

Brave New World

So how is business done post-truth? How does one analyse 50 million search requests? They are all equally true. But what’s important?

Precisely. What’s important? That’s the question that dominates my work with data. It’s also the question that fuels the development of Stories. We passionately feel that the ability to reduce information and surface only what matters most—perhaps 0.0002% of the source data—is a critical piece of technology for the enterprises of today, and the governments, societies and people of tomorrow. (More on how we do it can be found here or here.)

It’s also an uncommonly empowering question. Compared to “what’s true?”, the answer to “what’s important?” is highly subjective. It depends on the business strategy, personal preference, experience and beliefs. It encourages choice and self-reflection. Answering it is a journey of self-expression, creativity and freedom. It’s not a regression into the dark ages of a pre-scientific world, as the public debate often implies. Rather, the post-truth decision-making is a brave quest toward ourselves. Embarking on it equates to the search for meaning.

The parallels between a personal quest, business intelligence, startups, enterprise management and the political debate will keep growing closer. All these domains are, in their own space and pace, shifting toward decision-making in an over-informed, post-truth, post-hierarchical, fluid, dynamic and experiential world. And, as a direct effect of the growing volumes of data, the new decision-making will have the following traits:

  • The fundamental post-truth dilemma is not ‘true’ or ‘false’. It’s ‘useful’ or ‘useless’. That’s where the cognitive cost is the highest.
  • It will need to be realized through a synergy of human and artificial intelligence.
  • It will be highly subjective, dependent on business strategy, personal preference, experience, beliefs and meaning.
  • It will push us toward becoming ourselves.

My personal answer to what’s important is ‘building AI that helps people decide’. That’s the macro level. On the micro level, I’ll have to find the answer 1000 times over througout 2018. As Yuval Atsmon told me recently:

“In a $75 trillion world economy, there are many, many ways to build a $100 million business.”

Indeed. And in that search, in those decisions, the ancient icon of ‘truth’ will not play such a large role. Let it burn. And let’s do what matters.