SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
paper available at https://arxiv.org/pdf/2207.02578
code available at https://github.com/microsoft/unilm/tree/master/simlm
Paper abstract
In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval.
It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training.
We use a replaced language modeling objective, which is inspired by ELECTRA,
to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning.
SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries.
We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings.
Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost.
Results on MS-MARCO passage ranking task
Model | dev MRR@10 | dev R@50 | dev R@1k | TREC DL 2019 nDCG@10 | TREC DL 2020 nDCG@10 |
---|---|---|---|---|---|
RocketQAv2 | 38.8 | 86.2 | 98.1 | – | – |
coCondenser | 38.2 | 86.5 | 98.4 | 71.7 | 68.4 |
ColBERTv2 | 39.7 | 86.8 | 98.4 | – | – |
SimLM (this model) | 41.1 | 87.8 | 98.7 | 71.4 | 69.7 |
收录说明:
1、本网页并非 intfloat/simlm-base-msmarco-finetuned 官网网址页面,此页面内容编录于互联网,只作展示之用;2、如果有与 intfloat/simlm-base-msmarco-finetuned 相关业务事宜,请访问其网站并获取联系方式;3、本站与 intfloat/simlm-base-msmarco-finetuned 无任何关系,对于 intfloat/simlm-base-msmarco-finetuned 网站中的信息,请用户谨慎辨识其真伪。4、本站收录 intfloat/simlm-base-msmarco-finetuned 时,此站内容访问正常,如遇跳转非法网站,有可能此网站被非法入侵或者已更换新网址,导致旧网址被非法使用,5、如果你是网站站长或者负责人,不想被收录请邮件删除:i-hu#Foxmail.com (#换@)
前往AI网址导航
2、本站所有文章、图片、资源等如果未标明原创,均为收集自互联网公开资源;分享的图片、资源、视频等,出镜模特均为成年女性正常写真内容,版权归原作者所有,仅作为个人学习、研究以及欣赏!如有涉及下载请24小时内删除;
3、如果您发现本站上有侵犯您的权益的作品,请与我们取得联系,我们会及时修改、删除并致以最深的歉意。邮箱: i-hu#(#换@)foxmail.com