The machine’s search for meaning

38 sec read

In March last year, I had an opportunity to speak at Pycon 2017 in Bratislava. I leveraged this opportunity to explain how are we reinventing the world of nowadays Business Intelligence tools that are aimed on Automated and Augmented Analytics.

Roughly 80 % of analyst’s work is to search for meaning in data. Unfortunately, 95 % of signals found are pure noise. This noise is a tremendous waste of time and potential of one of the smartest people in the company.
To address these issues, we designed and developed python engine that leverages 20+ analytics libraries and encompasses standard analyst’s processes. It can find all critical business anomalies in data, faster and more precisely than any human analyst would do. It also understands the context and can correctly add a right explanation to every situation. In my talk, I explained overall system architecture, its orchestration via Airflow and libraries we use including pandas, scikit-learn or networkx. I also discussed the biggest issues we faced to make our Stories engine fast, configurable and applicable to any business data.