Prompt Tuning or Fine-Tuning

Investigating Relational Knowledge in Pre-Trained Language Models.

authored by

Leandra Fichtel, Jan-Christoph Kalo, Wolf-Tilo Balke

Abstract

Extracting relational knowledge from large pre-trained language models by a cloze-style sentence serving as a query has shown promising results. In particular, language models can be queried similar to knowledge graphs. The performance of the relational fact extraction task depends significantly on the query sentence, also known under the term prompt. Tuning these prompts has shown to increase the precision on standard language models by a maximum of around 12% points. However, usually large amounts of data in the form of existing knowledge graph facts and large text corpora are needed to train the required additional model. In this work, we propose using a completely different approach: Instead of spending resources on training an additional model, we simply perform an adaptive fine-tuning of the pre-trained language model on the standard fill-mask task using a small training dataset of existing facts from a knowledge graph. We investigate the differences between complex prompting techniques and adaptive fine-tuning in an extensive evaluation. Remarkably, adaptive fine-tuning outperforms all baselines, even by using significantly fewer training facts. Additionally, we analyze the transfer learning capabilities of this adapted language model by training on a restricted set of relations to show that even fewer training relations are needed to achieve high knowledge extraction quality.

External Organisation(s)

Technische Universität Braunschweig
Vrije Universiteit

Type

Conference contribution

No. of pages

15

Publication date

2021

Publication status

Published

Peer reviewed

Yes

Electronic version(s)

https://doi.org/10.24432/C5RC75 (Access: Open)