Model card for CLAP
Model card for CLAP: Contrastive Language-Audio Pretraining
Table of Contents
- TL;DR
- Model Details
- Usage
- Uses
- Citation
TL;DR
The abstract of the paper states that:
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.
Usage
You can use this model for zero shot audio classification or extracting audio and/or textual features.
Uses
Perform zero-shot audio classification
Using pipeline
from datasets import load_dataset<br /> from transformers import pipeline<br /> dataset = load_dataset("ashraq/esc50")<br /> audio = dataset["train"]["audio"][-1]["array"]<br /> audio_classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-fused")<br /> output = audio_classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])<br /> print(output)<br /> >>> [{"score": 0.999, "label": "Sound of a dog"}, {"score": 0.001, "label": "Sound of vaccum cleaner"}]<br />
Run the model:
You can also get the audio and text embeddings using ClapModel
Run the model on CPU:
from datasets import load_dataset<br /> from transformers import ClapModel, ClapProcessor<br /> librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")<br /> audio_sample = librispeech_dummy[0]<br /> model = ClapModel.from_pretrained("laion/clap-htsat-fused")<br /> processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")<br /> inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt")<br /> audio_embed = model.get_audio_features(**inputs)<br />
Run the model on GPU:
from datasets import load_dataset<br /> from transformers import ClapModel, ClapProcessor<br /> librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")<br /> audio_sample = librispeech_dummy[0]<br /> model = ClapModel.from_pretrained("laion/clap-htsat-fused").to(0)<br /> processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")<br /> inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt").to(0)<br /> audio_embed = model.get_audio_features(**inputs)<br />
Citation
If you are using this model for your work, please consider citing the original paper:
@misc{https://doi.org/10.48550/arxiv.2211.06687,<br /> doi = {10.48550/ARXIV.2211.06687},<br /> url = {https://arxiv.org/abs/2211.06687},<br /> author = {Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},<br /> keywords = {Sound (cs.SD), Audio and Speech Processing (eess.AS), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering},<br /> title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},<br /> publisher = {arXiv},<br /> year = {2022},<br /> copyright = {Creative Commons Attribution 4.0 International}<br /> }<br />
收录说明:
1、本网页并非 laion/clap-htsat-fused 官网网址页面,此页面内容编录于互联网,只作展示之用;2、如果有与 laion/clap-htsat-fused 相关业务事宜,请访问其网站并获取联系方式;3、本站与 laion/clap-htsat-fused 无任何关系,对于 laion/clap-htsat-fused 网站中的信息,请用户谨慎辨识其真伪。4、本站收录 laion/clap-htsat-fused 时,此站内容访问正常,如遇跳转非法网站,有可能此网站被非法入侵或者已更换新网址,导致旧网址被非法使用,5、如果你是网站站长或者负责人,不想被收录请邮件删除:i-hu#Foxmail.com (#换@)
前往AI网址导航
声明:本文来自AIGC网址导航投稿,不代表微浪网立场,版权归原作者所有,欢迎分享本文,转载请保留出处!