Experimentation Marketing Methodology Explained

Experimentation Marketing Methodology Explained

The modern business utilizes precise data to approximate success more often these days. Marketing is becoming increasingly similar to classical science than psychological gambling. The world is becoming more complex and gives more and more opportunities for growth.

In this article, I want to share the methodology of the approach to marketing, which will make it up-to-date, and therefore effective and controllable.

1. Set up precise and specific goals

Any development begins with setting the exact goal. This is important because businessmen tend to view the whole picture, which makes the goal meaningless. Yes, the foundation of any business is making money and an increase of their amount, but you will not achieve this result if you do not decomposite the main goal into small ones – achievable, unambiguous tasks.

Let’s imagine that you have a digital product and you want to increase the daily number of active users. But it is too ambiguous to be the goal. After you decide to think about retaining existing users, as long as it does not increase the daily number of active users, but it is too vague for the purpose. You think further and decide to help users use more functions of the site because your analytics showed that the more features the client tried, the more often they returned to it. In-depth testing of product functionality by users leads to retention, which leads to an increase in the daily number of users. It turns out that your goal is to increase the use of product features by the new users by 2 times.

Raw goal: to get more money by increasing the number of daily active users.
Smart goal: to increase the use of system functionality by new users by 2 times.

Many executives and marketers experience great difficulties in the process of narrowing a common goal to a specific one. There is a way to understand that the goal is still not accurate enough: if you have not gotten to a task that can be completed once and for all, then your goal is still too extensive.

The mind map of coming to a smart goal will look like this:

Experimentation Marketing Methodology Explained

Can you ever say that you’ve finished increasing your company’s profits? No! So it is not a goal. Will you be able to stop increasing the number of active users on some day or stop the retention process? Also, negative – not a goal either. But informing all your users about the features of the product is the task we were looking for. After it, you can proceed to another task.

2. Use the right levers

Each company has strengths or means that can be used as a lever to grow. If you make a little effort but got a great result, it means you have found the right lever.

You need to decide what to do first: send letters to your customers or add a “What’s New” section. If you already have an email database of your customers and a mailing system (for example, MailChimp), you can create an email and send it to all users within 1 day. Looks like a promising lever. Creating a new section on the website and adding it to the menu will take a lot more time and human resources: creating a prototype, section design, a bit of layout and coding… All this doesn’t look like the right road.

The principal of the lever makes the choice for you. You just have to send the newsletter. Another company in a similar situation may make a different choice, but only if the lever looks different.

Plans, goals, and objectives, which are created and ranked in a vacuum, without taking into account the strengths of the company, are erroneous.

3. Run the experiment

You decided on the goal, choose the right lever, and, of course, did not forget about the analytics. Now it’s time to start the experiment! A few things to keep in mind during the launch:

Write down all your assumptions/hypotheses

Before starting the experiment, write down your best suggestions about what will happen when you start. Will a new newsletter show better CTOR indicators than a regular newsletter? Why? How much will this letter increase user engagement with the site next month? Will this indicator increase by 2 times with one campaign or need several of them?

“Why write something down and waste time if I can just send letters and find out what will happen?” – if you think this way, you’re missing one important thing about testing hypotheses. The list of hypotheses is are very useful because it does not allow you to distort the past and helps tracking down the history of thinking, doing and evolving the way of thinking.

Imagine that you wrote down the following: “CTOR will decrease if we add another email to our workflow and will start sending two newsletters a week instead of one”. Then you run the experiment and observe an increase in CTOR. If the hypothesis was not written down, the history wouldn’t tell that originally you were not targeting CTOR increase, but checking the opposite thing! This will influence your and your team approach to email marketing.

The hypothesis history forces you to be honest to the end. And now you can explore and objectively discern why your hypothesis was wrong, even though it turned out a success in the end. You can conclude that some customers want to know more about the company. The importance of this insight cannot be overestimated.

The idea of forming a hypothesis makes marketing more similar to science than to the familiar marketing of the 20th century. And this is good.

Be realistic when it comes to resources

  1. First, the entire team must be notified of upcoming experiments in order to be prepared for any setbacks that will occur.
  2. Secondly, you should know when you are extremely limited in resources and take this into account when planning experiments. If on Fridays the server of your site is already on the verge of falling, then you should not initiate an event that will add 30% more traffic on this day.
  3. Thirdly, if you need a certain amount of time to prepare an experiment, the Hofstadter’s law will be useful:

“It always takes longer than you expect, even when you take into account Hofstadter’s Law”.

Do not worry about bad results

There is a phenomenon that I personally have experienced countless times, and this is a hard belief that any experiment I am working on will change everything for the better. The experiment I am doing now seems to be the obvious solvation of the company’s problems.


Failures are normal. That is why each target must be attacked from different flanks. Most attacks just won’t work.


There is no such thing as bad data. Even if the experiment failed, you undoubtedly collected a lot of information about the product and users, which can be used in future experiments. Thomas Edison failed more than 1000 times trying to create his own light bulb. When asked about this, Edison allegedly said: “I was mistaken 1000 times. I have successfully discovered 1000 ways NOT to make a light bulb. ”You can learn from success, and you can learn from failures. You stop learning only when you give up.

4. Optimize

Experiments should be optimized. This is not the thing you did once and forgot. Customize the experiments! Restart the experiments! You can abandon the experiment only when it is reasonable, but not because you are tired of it.

Use focus group

The focus group is needed whenever you have an opportunity for it. Using it, you can track the influence of the external environment and/or events that are not subject to your company.

You send a letter to 80% of customers and track how it will affect their engagement. According to the data, it turns out that the letter not only did not increase but reduced the engagement. But this could not have been caused by the mailing itself, but by anything else. With the focus group, you will be able to clear out that the headline wording sounds spammy while the body text is engaging. This insight contradicts the main conclusion of the experiment. So it needs to be repeated with the less spammy headline.

Use A/B testing

A/B testing is used by top marketers for a reason. It may seem to you that you know what the subject of the letter should be for people to open it, but A/B testing will show the truth. You may think that you know which landing page is better for landing traffic in order to precisely increase the engagement of customers, but A/B testing will show the truth. There are very few methods in marketing with the same value.

5. When to stop

I usually do not give up the experiment until my lever is weaker than I had planned. Another case for stopping the experiment: I cannot logically imagine an experiment that gives results without an excessive amount of resources.

6. Repeat

Now it’s time to get down to a new experiment or to an optimized version of one of the previous experiments and perform all the steps again. If you work according to the system described above, then success for you will be a by-product of your perseverance and not an unexpected coincidence.

Share on facebook
Share on twitter
Share on linkedin
Share on pinterest
Share on pocket