Be careful when you read Airbnb analyses
Be careful when you read Airbnb analyses
06 NOVEMBER 2015 7:14 AM

Airbnb keeps its data close to the vest. Any third-party analysis is a best-guess proposition. 

“Torture the data long enough, and it will tell you anything.”
I’ve heard that quote often during industry conferences and presentations. After all, ours is a business sector where analytics and performance benchmarking drive key decision making on a minute-to-minute basis. 
While tongue-in-cheek, it’s nonetheless a good reminder that sifting through data can be just as treacherous as compiling it in the first place. 
That’s especially true in an age when many of us are drowning in a sea of information. There’s often too much of it, leaving even the careful reader to grasp for a lifesaver in the form of an eye-catching headline or succinct article summary. 
When’s the last time you read an entire report or analysis cover to cover? As a reporter, it’s my job to do so, but even I have difficulty finding enough time in the day. 
I found the time this week, however, when I saw news that the overall impact of Airbnb in New York City in one 12-month period was estimated at $2.1 billion. That’s a number worthy of anyone’s full attention. 
The finding was published in “Airbnb and impacts on the New York City lodging market and economy,” conducted by HVS and commissioned by the Hotel Association of New York City. Presented on the second page of a letter outlining the summary findings, author Rodney Clough broke it down further:
Airbnb demand in New York City resulted in a “direct loss” of hotel revenue in the amount of $451 million. (The remaining tally accounts for lost revenue for ancillary services, such as food and beverage, as well as lost construction activity and tax revenues.) 
To make it a bit easier to wrap my arms around in this column, I’m going to focus only on that $451 million number. It’s HUGE. But is it VALID? In other words, does that number measure what it was supposed to measure? 
Here are three reasons why I have some doubts: 
1. The $451-million tally accounts for all Airbnb room inventory. While private rooms or entire residences compete more directly with hotel rooms, the general assumption is that shared rooms, couches, air mattresses, etc. do not. 
Interestingly, the report spends five pages saying as much. “The product type of Airbnb available inventory is important to consider in the overall impact analysis, as not every listing represents a unit that would necessarily compete with the New York City hotel industry,” it reads. 
For instance, 4% of NYC’s Airbnb listing inventory comprises “non-real beds” (e.g. couches). Analyzing the inventory in terms of room type, 4% of the inventory is for a “shared room.” 
I asked the report author to share the “direct loss” revenue number by room type, which was not included in the original report. He did so at the permission of the Hotel Association of New York City. Here’s the updated table: 
With this new data, the actual revenue loss to hotels would not appear to exceed $439.6 million. (Total revenue excluding shared rooms.)  But that’s if you assume all those Airbnb guests would have otherwise booked with a hotel, which brings me to point No. 2 …
2. You can’t assume that all Airbnb customers would have otherwise booked at a hotel. The inventory discussion above shows the immediate flaw with that way of thinking, although it extends deeper. 
There are many in the hotel industry who think Airbnb generates a lot of incremental demand. Put another way, it serves a niche travel segment that would not have booked with hotels regardless. 
Hilton Worldwide Holdings’ CEO, Chris Nassetta, explained it this way on his company’s most recent earnings call:
“I think it is a good business and will be around. I just think it is serving a different kind of need. It’s a different business segment onto itself. And the segments that we are serving largely will remain separate and distinct from that.”
3. Finally, the $451-million revenue figure comes from Airdna, a data collection firm. Airdna’s data might be accurate. If not 100% accurate, it might be directional indicative of trends in Airbnb supply and market rates. Or it might be way, way off. 
The problem is: I have no way of knowing, as we've not independently verified their data. 
Airbnb is privately held and keeps its info close to the vest. That’s why any analysis of its impact is a best-guess proposition. 
That includes some of the content we’ve published of late. That’s why we make efforts to be as transparent as possible with the methodology and caveat the hell out of it. 
Which brings me back to the Airbnb/NYC report—or any reports, studies and analyses, for that matter. Be careful how you sift through the data. Headlines and summary findings don’t always tell the full story. Or the best story. 
When I asked the HVS report’s author why he choose to use in the summary findings that $451-million revenue figure—a number that he calls into question only a few pages later—he said he used the total inventory figures at the bequest of the client, the Hotel Association of New York City. 
When I asked the association why it made that request, spokesperson Lisa Linden told me, “We stand by the study. We hired an esteemed firm to conduct it.” 
(For the record, the NYC-area hoteliers we reached out to said they’ve felt only a modest impact from Airbnb.) 
The industry is clamoring for data about Airbnb and its peers in the sharing-economy space.  That means a lot of people will rush to provide it. 
Just be careful what you read. That includes everything on Hotel News Now as well. Sound off in the comments when you have questions or concerns. Better yet, reach out to the editor who wrote the story.  
Remember, just because the data is telling you something doesn’t mean it’s telling you the right something. 
Now on to the usual stuff …
What’s making me happy this week?
Forget the hotel industry. Nothing can make me happier than 70-degree weather in our home base of Cleveland in November. (Usually the month is a dour spectacle of rain, snow and the slow, steady, unavoidable creep of winter.) 
Stat of the week
$3.9 billion: Amount online travel agency Expedia Inc. has agreed to pay to acquire vacation rental service HomeAway. 
I spent a lot of time this week thinking of how hotel companies might invest in the sharing-economy space. It was sparked mainly by this comment from Hilton’s Nassetta, also made during the company’s earnings call: 
“If you believe … that (Airbnb’s) sort of developing into its own segment, one of the big hotel companies could say that’s a segment we want to serve and we don’t serve, and we don’t think it cannibalizes any of the rest of our business, which I’d happen to believe is true and we want to have that segment.”
That’s why news of the Expedia buy caught me a bit off guard. It shouldn’t have. The OTAs have shown a willingness not only to engage in sizeable M&A activity but also to jump more quickly into uncharted waters. 
Your move, Priceline. 
Quote of the week
“We've, for some time, looked at this whole sharing-economy dynamic as a broad consumer issue and the consumer behavioral change, and we've always been drawn towards it, not sort of away from it. Because we feel like we need to learn from what we're seeing evolve in the market and how consumers think and how they behave.” 
Mark Hoplamazian, president and CEO of Hyatt Hotels Corporation, on Tuesday during an earnings call with analysts.
Hyatt has emerged as a fascinating case study amid this alternative-accommodations ascension. While other public hotel companies are keeping their distance, Hyatt dove in with an undisclosed investment in Onefinestay, a home rental service that focused on upscale residences in key gateway markets such as New York City, Paris and London. 
Reader comment of the week
“This study does not address the overall economic impact at all. Then to estimate a revenue number for Airbnb and make the assumption that this figure is a direct loss to the lodging industry... wow that’s reckless. Same issue on the supply side; to cook up a value for the development it would take to create equivalent hotel rooms to match Airbnb supply and say that the hotel industry missed out on that dollar amount is simply laughable. With supply growth what it is in New York relative to other major markets, I think anyone would be hard pressed to say Airbnb has had any negative impact on development, not to mention that is it physically impossible to build that many hotel rooms and ridiculous to suggest that this could have happened. So many glaring methodology errors here leading to misleading data points.”
Reader “Alex” after reading the news release announcing the findings of HVS’ Airbnb impact study of New York City.
The news release sparked some of the longest comments I’ve seen in quite a while. Most of them were quite critical. 
Email Patrick Mayock or find him on Twitter.
The opinions expressed in this column do not necessarily reflect the opinions of Hotel News Now or its parent company, STR and its affiliated companies. Bloggers published on this site are given the freedom to express views that may be controversial, but our goal is to provoke thought and constructive discussion within our reader community. Please feel free to comment or contact an editor with any questions or concerns.


  • Lionel November 6, 2015 4:35 AM

    The "reader" who made the comment about development obviously has not read the report or is a shill for airbnb and btw - they have loads of astroturf surfers given that they have more money than there is water in the pacific . The fact is that people are staying in major cities like Paris, NYC and SF because their rooms are significantly cheaper - unlike in non-dense areas

  • zelman November 6, 2015 5:18 AM

    Though you say that Airbnb keeps its data close to the vest, it is interesting that they recently launched a campaign in San Francisco (since pulled because of negative reaction) touting their $12M in room tax contributions to the city. It's pretty signficant that they offered up that number.

  • TomAirdna November 6, 2015 5:54 AM

    We at Airdna are very happy with our Airbnb numbers for New York. We provided raw data that we have meticulously gathered and are quite happy with our methodology and invite curious readers to contact us. I wish Patrick had prior to writing this! That said, we had nothing to do with the conclusions of this report and the calculation methodologies.

  • Airdna? November 8, 2015 3:34 AM

    While Tom of Airdna may be "very happy" with their Airbnb numbers for New York, has anyone yet heard an explanation of how they capture their data? The mere act of publication does not validate Airdna's data.

  • ntsearles November 12, 2015 6:47 PM

    In sales one has to recognize who is and who is not your client.

Comments that include blatant advertisements or links to products or company websites will be removed to avoid instances of spam. Also, comments that include profanity, lewdness, personal attacks, solicitations or advertising, or other similarly inappropriate or offensive comments or material will be removed from the site. You are fully responsible for the content you post. The opinions expressed in comments do not necessarily reflect the opinions of Hotel News Now or its parent company, STR and its affiliated companies. Please report any violations to our editorial staff.