"The home sharing platform at Airbnb is a two sided marketplace for hosts to rent out their spaces, referred to as listings, to be booked by prospective guests from all around the world. A typical booking starts with the guest issuing a search at airbnb.com for homes available in a particular geographic location. The task of search ranking is to respond to the guest with an ordered list of a handful of listings from the thousands available in the inventory".
"The search ranking model under discussion is part of an ecosystem of models, all of which contribute towards deciding which listings to present to the guest. These include models that predict the likelihood of a host accepting the guest’s request for booking, models that predict the probability the guest will rate the on trip... It is common for guests to do multiple searches, clicking through some of the listings to view their details page. Successful sessions end with the guest booking a listing."
"Typical configuration of the network: an input layer with a total of 195 features after expanding categorical features to embeddings, feeding the first hidden layer with 127 fully connected ReLUs, and then the second hidden layer with 83 fully connected ReLUs. Inspired by NN [neural net] architectures the new model was an attempt to combine the strengths of all three models. For the FM [factorization machine] model we took the final prediction as a feature into the NN. From the GBDT [gradient boosted decision tree] model, we took the index of the leaf node activated per tree as a categorical feature.
"The features feeding the DNN were mostly simple properties of the listings such as price, amenities, historical booking count, etc, fed directly with minimal feature engineering. The searches performed by a guest and their interactions are logged. While training, a new model has access to the logs of the current and previous models used in production. The new model is trained to learn a scoring function that assigns impressions of the booked listings in the logs at as high a rank as possible... The new model is then tested online in an A/B testing framework to see if it can achieve a statistically significant increase in conversions compared to the current model."
Over time we realized that moving to deep learning is not a drop-in model replacement at all; rather it’s about scaling the system. As a result, it required rethinking the entire system surrounding the model. Conned to smaller scales, models like GBDT are arguably at par in performance and easier to handle, and we continue to use them for focused medium sized problems. So would we recommend deep learning to others? That would be a wholehearted Yes. And it’s not only because of the strong gains in the online performance of the model. Part of it has to do with how deep learning has transformed our roadmap ahead. Earlier the focus was largely on feature engineering, but after the move to deep learning, trying to do better math on the features manually has lost its luster. This has freed us up to investigate problems at a higher level, like how can we improve our optimization objective, and are we accurately representing all our users? As a side note, while DNNs have achieved human level performance on certain image applications, it is very hard for us to judge where we stand for a similar comparison. Part of the problem is that it’s unclear how to define human level performance. Going through the logs, it’s quite challenging for us to identify which listing was booked. We find no objective truth in the logs, only tradeoffs highly conditional upon the budget and tastes of the guest which remain mostly hidden.