WSDM Cup 2023: Unbiased Learning & Pre-training for Web Search

Learning to Rank (LTR), aiming to measure documents' relevance w.r.t. queries, is a popular research topic in information retrieval with huge practical usage in web search engines, e-commerce, and multiple different streaming services. With the vogue of deep learning, the heavy burden of data annotation drives the academia and industry communities to the study of learning to rank using implicit user feedback or pre-training language model (PLM) with self-supervised learning. However, directly optimizing the model with click data results in unsatisfied performance due to the strong bias on implicit user feedback, such as position bias, trust bias, and click necessary bias. Unbiased learning to rank (ULTR) is then proposed for debiasing user feedback with counterfactual learning algorithms. However, real-world user feedback can be more complex than synthetic feedback generated with specific user behavior assumptions like position-dependent click model and ULTR algorithms with good performance on synthetic datasets may not show consistently good performance in the real-world scenario. Furthermore, it is nontrivial to directly apply the recent advancements in PLMs to web-scale search engine systems since explicitly capturing the comprehensive relevance between queries and documents is crucial to the ranking task. However, existing pre-training objectives, either sequence-based tasks (e.g., masked token prediction) or sentence pair-based tasks (e.g., permuted language modeling), learn contextual representations based on the intra/inter-sentence coherence relationship, which cannot be straightforwardly adapted to model the query-document relevance relations. Therefore, in this competition, we focus on unbiased learning to rank and pre-training for web search under real long-tail user feedback dataset from Baidu Search (Baidu is the biggest Chinese search engine with 6.32 million monthly active users that has a great ambition and responsibility to promote the technique development in the community).