# Thoughts on Analytics

- Link: https://www.solmesz.com/articles-en/thoughts-on-analytics
- Published: 2019-10-09T20:00:58-03:00
- Author: Sol Mesz

[_Este post también está en español._](https://www.solmesz.com/blog/notas-sobre-analytics)

Preparing my talk “Full loop analytics framework” for ILA 19 in Medellin, which 
as the title implies is focused on analytics, I’ve been reading a lot on the subject.

Here’s a small compendium of a few things that I found interesting.

## **Context is key**

In his article “[Observations on Data, Metrics and Goals](https://uxdesign.cc/observations-on-data-metrics-goals-204a02654850)”,
Dan Hill  says “_Goals are three things: a metric, a value and a moment in time.
When a goal feels wrong, it´s usually missing one of these_”. For instance, a goal
could be “increase sales by 10% in the next 3 months”.

**I would add a fourth component: context.** This is key in order make sense of 
the goal. Following the example above, is 10% realistic? Is it too much or too little?
If the growth of sales month to month, for the last 3 months has been 1%, then increasing
it by 10% is not realistic. Having context for a metric helps you see the big picture
and helps you have the right conversations.

## **Hope vs reality**

He also says: “Explain variance to forecast, not variance to hope”. And I thought
it was a great observation, especially when considering the previous point.

The problem is, even though forecasts should be based on historic trends and other
business/market data, often times goals are based on hope rather than forecast. 
Following the previous example, it may be that the 10% increment that was set as
a goal may effectively be in the forecast, but that said forecast was created based
more hope than reality.

And if we take into account the context component of the goal, we can see that this
10% is not only not realistic in terms of forecast, but also not realistic in terms
of reality. Again, adding context to metrics helps to understand (and explain) results
as well as to have the right conversations.

…

From Jack Martin’s presentation (former Head of Product at Roomgo) “[How to identify relevant KPIs](https://www.youtube.com/watch?v=bt7vLU64-8Q)”
for Product School I also found a few key points I wanted to highlight.

## **Funnels: focus on abandonment**

When analyzing a funnel don’t just focus on those who completed the process. Rather,
look at those who fell off because that’s where you’ll get the most valuable information.

The value that those users can bring include:

 * Their business potential: the users that fell through already wanted to convert,
   so we have higher chances of helping them and higher power of influence over 
   those who never came to our site or started the process.
 * They hold valuable information: they can help us uncover problems with our existing
   product or flow so we can make improvements for these and future users.

## **A holistic view is key**

Focusing on short term goals, which are usually results oriented, can lead us to
making the wrong decisions.

For instance, if we have a goal to increase the CTR to product pages, and we do 
so by eliminating some product information from the homepage so visitors are forced
to go to the product page to get the information they need.

We may increase the CTR with this tactic but at the expense of creating friction
for the users, which now have a harder time finding the information they are looking
for to make a purchase decision, and at the risk of them not doing the effort to
find it and therefore abandoning the process. If on top of that our competition 
does have product information easily accessible, we have an even higher risk of 
losing the sale.

So we may increase the CTR in the immediate term, but since this comes at the expense
of the user experience we may be hurting sales in the short to mid term.

In short, if we don’t have a holistic view we may mistake the tree for the forest,
and not realize that one small improvement in a specific part of the process is 
hurting the entire system.

## **KPIs and Business questions**

This point is quite obvious but often overlooked: all metrics must answer a business
question, otherwise you don’t know how to interpret the KPIs.

For instance, an increase in visits to the help pages of your site, is good or bad?
If we analyze the KPIs in isolation, we may think it is a bad thing because it means
more people are having problems with our product.

But if we have a goal of having more people use digital channels over the call center,
then an increase in visits to the help pages is a good thing, as long as the volume
or length of calls decreases.

In other words, it is important to peg KPIs to business questions and not analyze
them in isolation because we may arrive at the wrong conclusions.

## **Cohorts**

When we make changes it is important to use cohorts when measuring the impact because
if we use aggregate data we may the wrong results.

For instance, if we make changes to the checkout flow we should make a cohort with
the existing users (those who made a purchase in the past) and another one with 
new users when measuring performance.

This is important, because existing users may have trouble with the new flow at 
first, while new ones may find it easy to interact with. So if we put both users
in the same group when analysing results we may not get accurate results.