---
language: en
license: apache-2.0
datasets:
  - bookcorpus
  - wikipedia
tags:
  - fill-mask
library_name: transformers
---
This model was derived from the bert-base-uncased checkpoint by replacing the GELU with ReLU activation function and continued pre-training to adapt it to the change of the activation function.