My name is Yali and I'm a data guy
I hope to share my thoughts on the rapidly evolving data space here
My name is Yali. I’m a data guy. I first got into data at university - where I had the chance to study a lot of different sciences (Physics, Chemistry, Physiology, Experimental Psychology) and then study the History and Philosophy of Science. The history in particular really blew my mind in terms of how whole disciplines have emerged over time around different styles of reasoning, and different ways of using data to reason.
When I left I became a Consultant - because that felt like a good place to learn how to use data to help businesses reason about things and use data to create value. I was very lucky early on in my career, working on a project with a manufacturer of complicated industrial goods, to use data that had been created for one purpose: recording the delivery schedules for different manufacturing plants, for an entirely different purpose: measuring the accuracy of the planning process. That analysis showed that actual planning cycles were significantly shorter than management had previously believed. It led to a wholesale restructuring of the organization. And so very early on I got to see the transformational impact of data, on a commercial context, with my own eyes. It was a totally different experience from anything I’d seen in the academic realm beforehand.
In the years that followed I got to learn how to use data to support companies developing strategies as well as improve operations.
In 2011 and 2012 I was very excited about what we now call behavioral data: the iPhone had triggered a wave of innovation in apps that permeated every part of our lives and that meant there were opportunities to create data that described how people behave: how they fall in love on dating sites, collaborate with coworkers to solve problems in productivity apps, learn on education platforms and spend their free time consuming media and playing games: data that was both broad (could describe millions of people) and deep (it described for each person exactly what she or he were doing, second-by-second and minute-by-minute). That, and the possibilities that big data frameworks like Hadoop and AWS services like EMR led me to cofound, with Alex Dean, Snowplow in 2012: an open source platform to create behavioral data at scale - one that any organization could use to create their own data. (At the time, organizations relied on tools like Google Analytics and Adobe, which prescribed a very narrow set of use cases on the data, and created all sorts of barriers to organizations to innovate on their own data.) Our goal was to set organizations - and data folk within them - free to innovate.
Ten years on the data space is unrecognizable. But it has only grown more exciting, there is so much more possibility and so much more innovation than I could have imagined back in 2012. Even better, being at Snowplow gave me the opportunity to work with a whole host of innovators in the data space and see first hand how advances in technology have been seized on to make fundamental changes to the way we are as human beings and the way organizations conduct themselves. It is incredibly good fun to think ahead, to the next 10 years, and speculate how they will evolve.
My intention is to use this substack to share and debate some of those thoughts: and widen the conversations I get to have around data, from the people I get to work with directly, to a much broader set of people like you, out there in the world.