If you are a fan of Shark Tank, you would surely be familiar with the question that every shark loves to ask “How much has your startup grown since last year?”. Growth is never an easy process, especially for a startup with rather limited resources, but there is still unrelenting pressure for every startup to grow as fast and sustainably as they can. In “The Lean Startup”, Eric Ries has developed a methodology with the same name to combat this problem – he proposes adopting a combination of business-hypothesis-driven experimentation, iterative product releases and validated learning. This is similar to how scientists progress with their research – they conduct experiments based on hypotheses. Two notable examples of Ries’s approach are Dropbox’s two-side referral program and Buffer’s acquisition of Digg Digg.
As every experiment needs to follow a clear process, a successful growth experiment includes indispensable elements including: a well-defined objective, a hypothesis, and a structured experimental design. In this post, we will assess the significance of growth experiments and identify the deciding elements of a successful one. We will also offer you a growth experiment template we have used to work with our clients.
The importance of growth experiments
A growth experiment is a cheap and organic way to boost company’s numbers, compared to traditional initiatives that require significant planning and financial resources. Although many businesses have been successful with certain growth hacks, it is still important that you experiment with your hacks to actually evaluate whether a growth tactic works in favor of your company’s characteristics.
Growth experiments are very popular among startups not for arbitrary reasons. The most important rationale is that throughout this growth experimentation process, companies can have a comprehensive overview about their business and are thus able to set a more structured growth objective and plan a well-designed experiment to achieve business goal. In contrary to random trials that might be used by startups with limited resources, those growth experiments provide a systematic way of assessing which factor of the funnel that the startup should focus on.
Furthermore, growth experiments are usually quicker to implement than traditional growth methods. Traditional methods (such as advertising and PR) usually consume plenty of resources, take a long time to be implemented and to be assessed in terms of business impact. Instead, with growth experiments, startups’ marketing executives can decide which growth experiments to prioritize first based on the available resource constraints of the company. Those growth experiments can vary from one week to six months. During that period, companies can use their tracking metrics to analyze whether their goal has been reached and make corresponding adjustments to improve their current and future experiments.
Lastly, those experiments are typically low-risk. Similarly to how marketers can choose experiment intervals, they can also choose growth tactics depending on their available resources, for example, the number of full-time marketers, the technology and the financial resources the company is processing. Therefore, even though the experiments might not come out as expected, it will not be of a big loss to the startup itself. Additionally, those controlled experiments can help marketers assess whether the introduction of the tactic is the only factor leading to the observed results. For example, in using A/B test, controlled experiments isolate the impact of the product change while controlling for external factors such as the day of the week, or the season those experiments are conducted.
However, it is also important to emphasize that growth is not one trick that is implemented in a short interval of time and can result in hyper growth for every hack. Businesses might be affected by the myth that many startup success stories such as Airbnb’s Craigslist integration are based solely on one hack. Instead, growth is a constant process of experiment.
How to build a structured growth experiment
Before implementing any growth experiments, marketers need to ensure that the following elements are set out and agreed within the team: objectives, hypothesis and experimental design.
The objective of an experiment is what you are actually trying to figure out by conducting this experiment. Objectives should be tied to some of your business’s KPIs or derived from them. There is little point in conducting a growth experiment without having KPIs to track the experiment’s performance. Furthermore, objectives should also be succinctly defined so that marketers would not be distracted mid-way through the experiment, and the whole team can work towards a shared goal. One question you can ask to determine your growth experiment’s objective is “If you were given 40 days to improve one aspect of your startup, what would you focus on?”. Examples of clear objectives are:
- Increase signup conversion by simplifying the subscription form
- Improve email open rate by sending high-quality information
In contrast, some pitfalls of objectives can be too vague (Increase signup conversion) or not tied to any KPIs (Improve retention).
Hypothesis
The second element is a hypothesis (what you think will happen by conducting this experiment). To effectively measure improvements, you need to come up with a concrete outcome-based prediction that you can easily test. It should ideally include the metric you want to move and your guess at how much you expect the metric to move by. This is where you are advised to also include any assumptions you make. It is because this will help your company understand why an experiment works, and whether it can be duplicated or scaled up in the future.
The template of how to set a hypothesis is: “If successful, I predict (the metric you are testing) will increase by (% or units of the metric you are testing) because (initial assumption). To have a benchmark on the % or units change of the metric you are testing, you can look at past experiments of other businesses, or your startup’s metrics. Some good examples of a clear hypothesis are:
- By only including necessary signup information, which are: Name, Email and Company, we expect the email subscription rate to increase by 10%. It is because we assume that people do not want to share too much information about themselves on a website, and do not want to waste time on this.
- By attaching videos and ebooks related to growth hacks, we expect the open rates to increase by 15%. We assume that people would be attracted to content of high substance, compared to emails just repeating services that the company offers.
In contrast, “Signup increase” and “Email deliverability increase” are too ill-defined.
Experiment Design
Once you have identified your growth experiment’s objective and hypothesis, you can move on to design your experiment. As mentioned above, objective and hypothesis should both be closely linked to KPIs, so the first step of this stage is to identify KPIs you would measure on. For example, it could be the subscription number or subscription rate. Next, you can decide what method you would use to conduct the experiment.
The most popular method is A/B testing, where you compare the result observed from the control group (without the change) and that of the treatment group (benefitting from the change). Another frequently-used method is to compare the performance pre and post-treatment; however, since many factors might have changed during this period, we can get biased results. The third step is to list out the resources required to conduct the experiment, from human resources, financial resources to required cooperation with other brands or influencers.
Before you start, you need to ensure that your conversion funnel is tracked from the beginning to end, to see where people are dropping out of your conversion funnel and why it happens like that. After you have identified above elements, you can run the experiment for the previously-set time frame. Once the experiment ends, you can use various tools, such as Google Analytics, Crazy Egg, Compete, Google Website Optimizer, Optimizely, KISSmetrics, ClickTale and Facebook Insights to assess whether your hypothesis is true.
If it is, you can consider doubling-down on the experiment, or see if it provides more insights to other experiments. Otherwise, you can explore why the experiment does not come out the way you expect it to be, and continue refining the experiment until you see some positive results. It is also recommended that you wait for the data to be statistically significant before rejecting the result completely.
However, the experiment might not end here. There are some other questions you might want to answer before moving on to a new experiment; for instance, the long-term effect of the change, or any mistakes you could have made. Types of confounding mistakes could be that another experiment is being tested with the same user set, a big event happens at that time, the sample size is too small, and too many variables between the control group and treatment groups are tested. If none of the issues are at place, we can be more confident in the validity of the experiment.
Examples of successful growth experiments
To illustrate above points, we will assess two successful cases of growth experiments
Wistia (video-hosting platform) – Getting video into new user’s account
- Context: Growth team in Wistia, after analysing the data, noticed that many users who borrowed a video from Wistia just to complete onboarding step actually viewed the video.
- Objective: Therefore, Wistia’s Growth team wanted to use this video space to educate users about how to use various Wistia tools, for example changing the color of the video bar. By doing so, they hope that users will be more familiar with their tools.
- Hypothesis: By changing the old video to another one that educates users about features of Wistia, it is predicted that the onboarding metrics (such as feature adoption and feature usage) will increase by 15%. This example of Wistia highlights the importance of identifying your assumption in the hypothesis. For example, the growth team would not have been able to pinpoint the reason behind their success in the growth experiment had they not pre-thought of possible assumptions they need to care about. Once you cannot identify the reason for the experiment result, it is challenging for you to repeat the tactic and still observe similar outcome. Suppose that Wistia had not been aware that their increase in onboarding metrics was possibly due to either appealing thumbnail or attractively educational content, they would have concluded wrong information from the experiment, and would not have been successful in repeating this video with different thumbnail (or different video content).
- Experimental design: They compared the onboarding metrics of the control group (50% of the users watching the old video), and the treatment group (the remaining users watching the new educated video). They observed that the difference in onboarding metrics is 15%, and concluded that this is an effective change to their customer retention process
Airbnb – Referral program
- Context: Airbnb implemented a referral program called Referrals 1.0 in 2011. Although it was not an unequivocal success, there were still some positive impacts that the team can leverage on. Therefore, Airbnb team looked at some levers that could impact their referral program, such as Monthly Active Users Sending Invites, Invitees per Inviter, and Conversion Rate to New User.
- Objective: Airbnb’s objective for Referrals 2.0 is to improve its revenue produced from referrals.
- Hypothesis: By improving on above growth levers, Airbnb expected from 20 to 90 time improvement in revenue produced from referrals. This hypothesis is based on the assumption that people go by recommendations from friends and family (according to Nielsen report, 84% of people do so). It is also based on the observation that people would not want to be taken advantage of.
- Experiment Design: Airbnb used A/B testing emails promoting the referral program to existing users, to check which call to action would attract them to begin sharing referral invites to their friends. One is “Invite Your Friends, Get $25”, and the other is “Give Your Friend $25 to Travel”. The referral is sent over email, and with the referrer’s picture put in the center to increase social trust. The former emphasized self-interest, while the latter highlighted altruism. The result was that globally, the latter performed better and was therefore used in the future referral program. Overall, referrals increased bookings by 25% in many markets. The initiative looked promising and gave Airbnb more insights on what they can improve on such as being more explicit about what Airbnb does on the referring email (because some people were confused when they received the email) and looking further into the impacts of influencers and celebrities into the virality of the referral program.
In short, it is important that every startup should have a defined template before implementing any growth experiment. We have also attached a growth experiment template so you can construct experiment in a more structured way. The document is structured in a similar way as the outline above, and includes space where you can elaborate on the results and future learnings from the experiment. Before you conduct a growth experiment, it is recommended you fill in every part of the document and anticipate how the experiment will be carried out in real life.
Download the Growth Experiment Template here: [download id=”11055″]
About Fast Forward Growth Services
In the past few years, Fast Forward Advisors has used this structure to achieve faster growth for our clients. To read more, please click on Grow:Up and ezCloud. ezCloud is a SaaS startup of management products for the hospitality industry: ezFolio (4-5* hotel management software), ezCloudhotel (cloud-based hotel management software) and ezCMS (cloud-based channel management system). After we conducted structured growth experiments for ezCloud, the signup rates increased by 300% and CAC reduced by 70%. We walked them through the process of choosing important metrics, and kept track of above metrics. After that, we worked with ezCloud team every week to together run growth experiments with A/B testing in order to improve their performance on Adwords and Facebook Ads campaigns.
Do you have issues experimenting growth at your startup? We would love to hear more at [email protected]. We are always looking to improve our content so that we can best serve our readers – please share with us your thoughts on our fundraising methodology right below in our comment section or Facebook. We look forward to hearing from you!
