Fig. 5: The pre-training and fine-tuning of BERT model in this study.

The chemBERT model was pre-trained on the SciBERT model (designed for scientific texts) with technical documents and Wikipedia from the Chemical Industry. The chemBERT model was then finetuned using corrosion-related text and was used to fill the mask in six sentences.