Why Focal Wants You to Fail Fast and Frequently- Using A/B Testing to Improve Results
Hospitality revenue management experts sum up the discipline as the ability to market the right product, to the right customer, at the right time, at the right price. At times, this definition is expanded to include using the right channel by savvy revenue optimization experts who understand the importance of net profitability.
However, exactly how is it that we come to determine whether we have truly optimized each component of the revenue management equation?
All too often, our revenue optimization strategies are based on the prior year’s tactics plus some incremental improvement intended to further enhance results. Each year, we run an incrementally better marketing campaign to an incrementally better customer, at an incrementally better time, at an incrementally better price. And at times, we drive bookings through incrementally better channels.
As a result, hoteliers continue to cede market share to disruptive competitors. In fact, in 2017 the hotel industry grew RevPAR by just 3%. For comparison purposes, Airbnb grew revenue by over 64% while on its way to obtaining a valuation greater than that of Hilton and IHG combined.
What if there was a better answer– a way to leapfrog the incremental in pursuit of the exponential? We’re not talking 2-3% inflationary gains, but rather career making, game changing, rock star performance gains.
But wait, there’s a catch. What if I told you that in order to achieve such gains you’d have to make mistakes? A lot of mistakes. Often small ones– but occasionally– mistakes so big that you wonder what you were thinking. What if I told you that along with those mistakes will come a deeper understanding of your hotel and competitive landscape?
“I can’t take the risk,” you might tell me. “My GM and owners will kill me if my RevPAR index shows anything less than 100 on a given day. In fact, my GM calls me into the office each time our rates fall more than $10 below our top competitor.”
And so you’d carry on with your incremental improvements to last year’s incremental results, ensuring sustained incremental growth. “But I’m slightly outpacing inflation and the comp set,” you might tell me, “and our owners haven’t been upset about our STR in 6 months.”
“That’s great,” I might tell you. “Sounds like you are poised to receive your incremental cost of living salary adjustment next year, and perhaps even an incremental performance bonus.”
“Well that’s disappointing,” you’d tell me, “shouldn’t I be rewarded to a greater degree for driving growth?” But incremental begets incremental, and results in marginal career opportunities and compensation increases.
Are you ready to make mistakes yet? Good.
The secret to outsized gains? A/B Testing– the process through which you determine the best strategy for anything through experimentation. The process through which you’ll make a lot of mistakes, but will also achieve great success. By following the 8 steps below, you’ll be well on your way to learning the lessons that lead to exponential growth.
8 Keys to A/B Testing Success
1. Determine Desired Impact-
What is the intended result of this test? Perhaps you are trying to improve website conversion, sell a particular room type more frequently, or determine the pricing elasticity for your hotel as you increase or decrease your BAR rates in relation to your competitive set.
2. Establish Control Factors and Benchmark Metrics-
Determine the period of time to target, along with any additional control factors.
Example- Perhaps you are attempting to determine whether aggressive pricing 30-45 days in advance of arrival will allow you to capture higher total occupancy levels during your summer season.
Decide how the efficacy of each strategy will be measured and ensure your systems are set up to track the key metrics associated with your A/B tests.
Example- Measure the percentage of room nights for a given period booked 30-45 days in advance as a total of all room nights consumed for that period, as well as any variance in total occupancy levels compared to a control period. Perhaps it is also important to measure the impact on total revenue and/or RevPAR to ensure that you are not simply driving occupancy at the expense of profitability.
Determine relative sample size needed to ensure statistical significance.
Example- Limiting the sample date range to one weekend will likely lead to a flawed analysis as the potential for impacts outside of the testing controls (such as special events or group displacement) will be high.
Subscribe to our Newsletter!
Like what you see? Submit your contact information below to bypass the website and have fresh content updates delivered directly to your inbox.
3. Develop a Hypothesis-
In order to avoid confirmation bias and remain receptive to observing unintended impacts, refrain from guessing what the outcome of the test will be.
4. Isolate Testing Variables-
Ensure that causal factors can be isolated to the greatest degree possible during the period measured to prevent spurious correlations.
Homogenize the testing period by ensuring both strategies are tested simultaneously over periods sharing the same demand patterns. Consider day(s) of week, seasonality, special events, group compression, and any other relevant factors.
Refrain from introducing additional variables during the testing period. For example, when measuring the impact of pricing more aggressively 30-45 days out, you wouldn’t want to also launch an advanced purchase sale for the same timeframe.
Randomize test samples to the greatest degree possible. For example, ensure pricing strategy changes are tested across as many channels as possible.
5. Implement Test-
Continue to monitor strategies throughout the A/B testing period to ensure consistency. For example, did the competitive set lower their rates during a similar booking window in an attempt to counter the effect of your aggressive pricing strategy?
6. Measure Test Results-
Consider immediate vs. longer term results.
Example- You may see variances in booking pace for the 30-45 day window immediately, while calculating the percentage of total occupancy provided by the bookings during this period will require that the consumption period has actualized.
Keep an open mind. It is possible, and in fact likely, that the A/B test may have produced impacts you didn’t originally anticipate. If you thought something would happen and it did, you haven’t learned much. If you thought something would happen and it didn’t, you’ve gained invaluable insight that you can apply to drive revenue or prevent mistakes in the future.
Example- Perhaps booking pace saw little variance, but ADR for rooms booked during this period actually went up as guests booked premium room types due to a perceived value in relation to the competitive set.
Determine whether a given dimension had an outsized impact on total results.
Example- Perhaps the Retail market segment or voice and website channels experienced the greatest increase in bookings while production for other segments or channels actually declined.
Compare any revenue increases against any expenses incurred in order to assess profitability.
7. Digest Findings and Share Results-
Eliminate bias. View your data as if you had no prior understanding of your hotel or market.
Be honest. It’s ok if the scenarios in your A/B test didn’t produce the desired impact. Mistakes lead to learning, and learning leads to $$$.
Retain records using a knowledge management platform so you can reference them at a later date when determining strategy. Knowledge isn’t valuable until shared and applied.
8. Tweak and Test Again-
If your strategy improved results, great! It should become your baseline, the new “A” strategy. However, it is important to continue testing new strategies to ensure continuous improvement.
If your strategy did not improve results, keep your current “A” strategy consistent and test a new “B” strategy.
Don’t succumb to “sunk strategy fallacy.” From time to time, revisit prior A/B tests to ensure results are still applicable. At times you will retest later and the results will change due to any number of factors. This is particularly relevant when your competition updates their strategies in order to mirror your success.
If the above sounds like a lot of work, it is. If it sounds scary, it can be. However, testing new strategies is the only way to break through the cycle of incrementalism.