malteos/scincl

SciNCL

SciNCL is a pre-trained BERT language model to generate document-level embeddings of research papers.
It uses the citation graph neighborhood to generate samples for contrastive learning.
Prior to the contrastive training, the model is initialized with weights from scibert-scivocab-uncased.
The underlying citation embeddings are trained on the S2ORC citation graph.
Paper: Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (EMNLP 2022 paper).
Code: https://github.com/malteos/scincl
PubMedNCL: Working with biomedical papers? Try PubMedNCL.

How to use the pretrained model

from transformers import AutoTokenizer, AutoModel<br /> # load model and tokenizer<br /> tokenizer = AutoTokenizer.from_pretrained('malteos/scincl')<br /> model = AutoModel.from_pretrained('malteos/scincl')<br /> papers = [{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'},<br /> {'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}]<br /> # concatenate title and abstract with [SEP] token<br /> title_abs = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers]<br /> # preprocess the input<br /> inputs = tokenizer(title_abs, padding=True, truncation=True, return_tensors="pt", max_length=512)<br /> # inference<br /> result = model(**inputs)<br /> # take the first token ([CLS] token) in the batch as the embedding<br /> embeddings = result.last_hidden_state[:, 0, :]<br />

Triplet Mining Parameters