BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Table of Contents

Title: BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Authors: Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, Jaewoo Kang

Published: Jan 25 2019

Link: https://arxiv.org/abs/1901.08746

Summary (Generated by Microsoft Copilot):

Introduction:

  • Biomedical text mining is crucial due to the rapid growth of biomedical literature. BioBERT is introduced to adapt BERT for biomedical text mining.

Challenges:

  • General NLP models perform poorly on biomedical texts due to different word distributions.

Methods:

  • BioBERT is pre-trained on large-scale biomedical corpora and fine-tuned on specific tasks like NER, RE, and QA.

Novelties:

  • BioBERT is the first BERT-based model pre-trained specifically for the biomedical domain.

Results:

  • BioBERT significantly outperforms BERT and other models in biomedical NER, RE, and QA tasks.

Performances:

  • Achieves higher F1 and MRR scores across various datasets compared to state-of-the-art models.

Limitations:

  • Computationally intensive pre-training process.

Discussion:

  • Pre-training on biomedical corpora is essential for effective biomedical text mining. Future versions will include domain-specific vocabulary.

BioBERT has been pre-trained on large-scale biomedical corpora, including PubMed. Specifically, the model was trained on:

  1. PubMed Abstracts: This dataset includes around 4.5 billion words from PubMed abstracts.

  2. PMC Full-Text Articles: This dataset comprises approximately 13.5 billion words from full-text articles in PubMed Central (PMC).

These extensive datasets help BioBERT understand and process biomedical texts more effectively, making it highly suitable for tasks like named entity recognition, relation extraction, and question answering in the biomedical domain.