I know what you’re thinking. 

You saw the words “data” and “science” in the same sentence and couldn’t click fast enough. 

Or, if you’re like most people, you had flashbacks to difficult high-school tests and that unique, exquisite blend of boredom and frustration that it seems only spreadsheets can provide. 

But what if I really did know what you were thinking? What if I could use data to understand patterns in behaviour, and reliably predict what you were going to do next?

That’s what this article is about: forecasting, and how it can be used in digital marketing optimisation.

After reading this, you’ll hopefully have a decent understanding of how this process works – a big bonus here is that you’ll be able to talk about it and have everyone hanging on your every word at your next family gathering.

First, let’s define “data science” properly. 

Data science is simply the techniques and principles we use to slice, dice and understand data. In marketing, this could be customer data, sales data, website usage data, campaign data, and so on. 

Forecasting can be achieved by pulling all this data together and building a mathematical equation that explains patterns and predicts what’s going to happen next.

Your data might seem like one set of information, but if you look at it the right way, you’ll find patterns within it.

Asking a well-defined question 

Before we create the actual forecasting model, we need to tackle the first and most important step of the data science workflow: we must understand that data science is an iterative process.

In practical terms, this means taking time to understand the purpose of the model (i.e. the business requirements it’s going to fulfil), and accepting that it’ll take some time to refine it.

As highlighted in the Cross Industry Standard Process for Data Mining (CRISP-DM), we will need to loop back and refine our project, depending on what we learn from the data along the way. This is crucial to gain a firm grasp on what the forecast will be used for. 

What do we compare against? 

OK, so we want to use the forecast as a sort of future benchmark to track our efforts against. But this poses a problem if the dataset we’re using to train our algorithm contains one-off, non-recurring outliers.

Fortunately, we can pre-emptively correct for these red herrings by weighting against them based on the probability that they occur; if they’re extremely uncommon and wildly out of range, we can tell our model to ignore them.

Understanding the data 

If we want to keep building a prediction model that could drive value to a business, we need to be able to learn from all the data we’re recording, including outliers or changes that might appear when we switch our tactics.

This involves us taking a deep dive into whatever datasets you have available, to understand the feasibility of the proposed solution.

For example, if you want to learn who your audience is (and who would respond well to your campaigns), you may be inclined to use your demographic data to create a logistic regression forecast that would predict which category users fall in (who wouldn’t?!).

But what if your customers’ specific demographic attributes weren’t varied enough? You wouldn’t be able to differentiate between potential converters and non-converters. In this case, demographics wouldn’t be enough to create our predictive model.

This might seem like an obvious conclusion, but if we’d skipped this essential data exploration step, we could’ve wasted a lot of time creating a model that wouldn’t have had any value to the client.  

Finding the right data

By looping through the steps of business understanding, data understanding and data preparation, we can understand whether we need to change the business requirements depending on the availability of data and feasibility of proposed solutions.

While we can forecast just about anything, such as sales or other offline or online actions, these forecasts are based on past time periods, user actions and/or user attributes. This means that we must have access to lots of rich data to build predicative models.

We could consider forecasting the decrease in revenue if we scaled back our advertising budget. We could create a forecast for any variable that causes a high risk of uncertainty, specific to our business. Or we could measure the impact of total digital spend on our client’s business goals.

A vs. B

Take into consideration the consumer journey specific to your business and try to picture the influencing factors that lead them to a conversion. This would help you understand what datasets may be combined or used to extract the features needed to train the predictive model.

To start with, you could try to identify and incorporate as many of the following elements as possible in your forecasting model:

  1. Users who convert due to habit (i.e. without the need for any marketing)
  2. People who are acquired through promotions (i.e. pricing)
  3. Audiences who are influenced by word of mouth (i.e. social proof/sentiment analysis)
  4. Seasonality and economic factors that influence leads for your industry vertical

To deal with the complexity of creating the predictive model, we have to assess the best approach and can choose between a) machine learning; and b) a statistical model.

Although the accuracy of machine learning models is higher, it’s often hard to interpret why certain factors influence the result or prediction. A better choice is often to use statistical approaches, which to lead to more descriptive results.

Note: There are other factors such as error rates, how long it takes the model to run, volume of data, and costs, which should also be taken into consideration before deciding. But that’s another article within itself.

Accounting for outside factors

The way things usually go, at some point you’ll be faced with an unexpected challenge (e.g. the website is undergoing significant changes).

A simple way to allow our predictive model to differentiate between the different versions of the website would be to introduce a  binary variable – a process called “one-hot encoding” – that would help us quickly identify whether the metrics were from the old or new version of the site.

Binarization can help differentiate one portion of your data from another.

For companies that spend a lot on paid media, knowing what to expect can be a huge asset. We can predict the expected the number of form submissions, clicks on a Store Locator button, or phone calls – all measured against how much is spent on each channel.

For example, to achieve a goal number of phone calls through paid media, a client might adjust their budget based on what the forecast would tell them to expect during a certain period of the year.

But rather than using a traditional time-series-based statistical model to calculate seasonality, we could first break down all KPIs by channels, use that dataset to train different machine learning algorithms, and finally choose the best performing model as the predictive model.

Think laterally!

Most companies have a paid marketing program, and often would want to predict sales or conversions. The biggest challenge in forecasting can be availability of data. How do we predict our sales when we are working with scarce data?

Well, one way to overcome this challenge would be to make a forecast of a proxy or other indicator, that is forecasting a factor that affects our conversions, rather than conversions themselves.

This could mean looking at major events at a stadium in order to predict traffic numbers, rather than looking at historical traffic numbers themselves.

This method allows us to use publicly available data to predict the market and economic factors that influence conversions for our industry, thus allowing us to indirectly forecast the required spend or budget for our digital marketing efforts.

As you can hopefully tell by now, data science is more about having a logical process than about PhD-level maths.

Watch this space – next time, I’ll dive into some other exciting aspects of data and analytics.

Rishabh Sarangi

Rishabh Sarangi is a Data & Analytics Executive at iProspect Brisbane. With a background in Commerce and Information Systems, Rish is also deeply interested in machine learning and automation, and how to harness them to drive bottom-line performance for clients.