Follow the science

Follow the science is a phrase repeated endlessly by politicians, institutions and advocates of various causes of late. This to the point where the phrase itself has become more of a motto than anything else.

But what exactly does it mean to follow the science?

This blog outlines in simple terms how science and academia gathers knowledge and how you can use this process as a best practice to make evidence based decisions in your own business.

The evidence pyramid.

Although there are many different formulations of the evidence pyramid, often tailored to the specifics of the field they are addressing, they all have common traits.

At the bottom of the evidence hierarchy you typically find anecdotal evidence in various forms, like expert opinion, single case studies etc.

This data is great for generating hypotheses’, but does not provide strong evidence, and is therefore situated at the bottom of the hierarchy.

Edging towards the top you’ll find randomised test and control trials.

These are experiments in a controlled setting where participants are randomly assigned to either a test group which is exposed to some sort of intervention, or a control group which is not.

The randomised component makes sure that any difference in performance between the groups can only be explained by the intervention.

Finally, at the top you find systematic reviews and meta analysis.

These are trying to find patterns of results across the studies conducted and represent the gold standard of scientific efforts because this is where you typically find actionable insights.

Gathering evidence in your business.

The evidence pyramid gives a rough guide. 

Below follows two simple tips as to how to incorporate these best practices from the academic community in your business, and ensure that your business is ‘following the science’.

First, embed experimentation into the way you govern 
your business. 

Whilst having domain knowledge experts in your business is extremely valuable, unless projections and projects are rigorously tested through experimentation and numbers, they should only be counted as anecdotal evidence.

Experimentation is a great way to ensure your business is data driven, can challenge cognitive biases and help decision makers with valuable insights.

Second, set up an analysis database. 

As mentioned earlier, the holy grail of scientific discovery comes from the patterns observed across different studies.

The only way to perform systematic reviews or meta analysis is by having access to the analysis done previously on the topic.

Therefore, make sure that you document the results and other parameters of any analysis you do in your business and gather them in a single database.

Future posts will outline in more detail how to structure your company’s data to be research ready, how to design a database for historical analysis, and how to perform simple meta-analysis of your experiments.

I hope you enjoyed reading this blog post. If you want us to just do your data analytics for you, click here.

The power of experimentation

Experiments, and in particular the randomised test control trial, is a great way to use data and analytics to drive decisions. This particular form of analysis has driven forward knowledge in the scientific community in diverse fields such as neuroscience, experimental psychology and medicine.

Here are three reasons as to why you should incorporate randomised experiments as a key tool in your business analytics strategy.

It’s easy to understand how it works.

A randomised test control trial typically takes a set of entities (like people, stores, products etc.) and randomly assigns them into two groups;

one test group and one control group.

Then the test group is exposed to some intervention, whilst the control group is not intervened with. The analyst then measures the performance of the test group when exposed to the intervention on whatever measurement is relevant, and compares that to the control group.

If the performance on the key metric of the test group is significantly above the control group the intervention has had a positive effect, whilst if it is below that of the control group the intervention has had a negative effect.

It’s easy to understand how it breaks.

The main reason why a test control trial may “break” is due to the two samples not being random.

By break I mean that the results we see cannot be trusted.

If the difference between the groups can be explained by something apart from the intervention then the model “breaks” in that it is not giving us accurate estimates of the effect of our intervention.

The good thing about this is that it is as much of a common sense exercise as it is a technical one. After you have assigned your test and control group – play the devil’s advocate and ask yourself if there are any reasons why we would expect these groups to differ. If yes, then you may want to think about reassigning or redesigning your experiment.

It’s easy to perform.

Because at its core, all we are doing when analyzing the results of a randomised test control trial is to compare the mean value of two groups, it is something everyone can do as long as you have some basic excel skills as a minimum.

In addition, all statistical softwares as well as open source languages like R and Python will have libraries to analyze experimental data in just a few lines of codes.

To summarise…

the randomised control trial is robust and best practice in a wide array of fields in academia.

It is easy to understand the results and limitations of the analysis, and is easy to implement. This makes the randomised control trial a great tool to incorporate in your business data strategy.

As a side note, I did gloss over the concept of statistical significance when discussing the analysis of experiments.

I hope you enjoyed reading this blog post. If you want us to just do your data analytics for you, click here.