Semantic-guided attention and adaptive gating for document-level relation extraction

Abstract

In natural language processing, document-level relation extraction is a complex task that aims to predict the relationships among entities by capturing contextual interactions from an unstructured document. Existing graph- and transformer-based models capture long-range relational facts across sentences. However, they still cannot fully exploit the semantic information from multiple interactive sentences, resulting in the exclusion of influential sentences for related entities. To address this problem, a novel Semantic-guided Attention and Adaptively Gated (SAAG) model is developed for document-level relation extraction. First, a semantic-guided attention module is designed to guide sentence representation by assigning different attention scores to different words. The multihead attention mechanism is then used to capture the attention of different subspaces further to generate a document context representation. Finally, the SAAG model exploits the semantic information by leveraging a gating mechanism that can dynamically distinguish between local and global contexts. The experimental results demonstrate that the SAAG model outperforms previous models on two public datasets.

Introduction

The purpose of document-level relation extraction (DocRE) is to extract all possible semantic relations among multiple entities from a given original text. It is widely used in downstream applications, such as knowledge graph construction¹, recommendation systems², and question answering³. Early research mainly focused on sentence-level relation extraction. However, most relations can only be expressed through multiple sentences and cannot be extracted by sentence-level relation extraction. Therefore, researchers have recently extended their efforts to DocRE.

Many researchers have adopted graph-based neural networks⁴, heuristic structures⁵ , or attention mechanisms⁶ for DocRE. Graph-based neural-network models can construct a document to connect different nodes and aggregate their information, and pretrained models can encode the input document into contextual representations and obtain state-of-the-art performance. However, previous models have focused on integrating sentence or document contextual information to enhance different granularity information representations while ignoring the effect of different words on contextual semantics in sentences. As Fig. 1 shows an example from document-level relation extraction dataset (DocRED)⁷, we need to predict the inter-sentence relation of “country” between the entity “Monticello High School” (red font) and the entity “U.S.” (blue font). Accurate entity representations facilitate relational judgments. For the subject entity “Monticello High School” in sentence 4, we observe that different words contribute unequally to the subject entity. Specifically, “Illinois” contributes more to the subject entity than other words. Likewise, the word “Galesburg” contributes more than the word “1886”. Hence, identifying critical words that contribute to semantic information is essential for solving relation extraction. In this study, to address the issue of semantic information ambiguity, semantic-guided attention techniques are employed that not only focus on the semantic information of the sentences containing the entities but also encompass the semantic information of the entire document. Furthermore, to distinguish more accurately the semantic information required for different entity representations, an adaptive gating mechanism is introduced.

Fig. 1

An example adapted from the DocRED. Different entities are distinguished by different colored font.

Combining the semantic-guided attention technique and adaptive gating technique produced the following contributions:

• Semantic-guided attention is developed, which can capture critical words that determine crucial semantic information in sentences.

• A novel mechanism is designed for information aggregation. It is an adaptive gating mechanism for dynamically fusing semantic sentence features and document topic information.

• Experiments demonstrate that the SAAG model is more competitive than existing models on two public DocRE databases. We also performed thorough analyses to show that our model is well interpretable.

Related work

In recent years, most models have focused on document graph construction and path-reasoning mechanisms in DocRE. Zeng et al.⁸ introduced double graphs with different granularities to solve the problems of complex interactions and path reasoning. Huang et al.⁹ constructed a relational knowledge graph to infer the unknown relations to learn the known knowledge. Zeng et al.¹⁰ designed a new mention-level graph by adding a document node and mention node and introducing a logical reasoning module to address logical reasoning problems more pertinently. Zhou et al.¹¹ proposed adaptive thresholding and localized context pooling techniques to address multiple entity pairs and relation types, respectively. Xu et al.¹² proposed a structured self-attention network to achieve structural and contextual reasoning simultaneously under the guidance of a model. Xie et al.¹³ proposed an evidence-enhanced framework to make the framework focus more on the import sentences by extracting and fusing evidence. Yu et al.¹⁴ introduced the RSMAN model to obtain flexible entity representations by incorporating attentive features. Yuan et al.¹⁵ proposed an optimization method based on knowledge relation reasoning that performs global reasoning on relational prompts through constraints from semantic segmentation. Han et al.¹⁶ proposed two co-occurrence prediction subtasks from coarse-grained and fine-grained perspectives to capture relational relevance. Then, the learned relation-aware embeddings are used to guide the extraction of related facts. Moreover, previous studies have employed attention mechanisms to model the semantic information of mentions or key sentences^17,18; however, none of these methods considers that each contribution of the words that constitute a sentence is different. If the same attention score is used indiscriminately, it affects the accuracy of sentence representation and the performance of DocRE.

Unlike previous research, in this study, an attempt is made to address this problem by developing a novel Semantic-Guided Attention and Adaptive Gated(SAAG) model to enhance DocRE performance. As shown in Fig. 2, SAAG consists of two fundamental techniques to achieve semantic enhancement and adaptive entity representation: (1) a semantic-guided attention technique for incorporating word features to generate more-focused sentence representation and (2) an adaptive gating technique for better distinguishing between intersentence relation extraction and intrasentence relation extraction.

Fig. 2

The overall architecture of Semantic-guided Attention and Adaptive Gated (SAAG) model. First, the document is input into the encoder. Then, the semantic-guided attention technique is used to obtain the sentence semantic feature representation and the document context representation. Next, the adaptive gating technique is employed to derive the semantic feature information representation. Finally, this representation is combined with the semantic feature representation and the relative distance representation, and the relation probability of the entity is obtained through a classifier.

Model

This section will provide the overall architecture of the Semantic-guided Attention and Adaptive Gated (SAAG) model. As shown in Fig. 2, in the following sections, we propose the encoding module in “Encoding module”, then detail the introduced SAAG model in “Semantic-guided attention and adaptive gatingmodule”, and finally, introduce the classification module in “Classification module”.

Encoding module

A given document $D=\{w_1,w_2,w_3,\ldots ,w_n\}$ can be segmented into n words, where $w_i$ denotes the i-th word in the document, and a context encoder is used to convert the document into a sequence of contextualized representations. Because entity types can enrich word representations and further narrow down the predicted entity relation, entity type embeddings ${E_t}({t_i})$ are concatenated with word embeddings ${E_w}({w_i})$. As for the entity type, following Yao et al.⁷, we introduce six entity types that cover a variety of entity types, including PER (Person), ORG (Organization), LOC (Location), NUM (Number), TIME (Time) and MISC (Other types). We get the following meaningful semantic information embedding representation by concatenation two embeddings:

$${x_{i}} = [{E_{w}} ({w_{i}} );{E_{t}} {(t_{i}} )]$$

(1)

Then, we feed the result of Eq. (1) into the encoder to obtain hidden state vector sequences for each word:

$$\begin{aligned} H=[{h_1},{h_2},\ldots ,{h_n}] = Encoder([{x_1},{x_2},\ldots ,{x_n}]) \end{aligned}$$

(2)

where encoder is BiLSTM or BERT.

Finally, it is necessary to obtain entity representations. Because an entity is expressed by one or many mentions, it is necessary to aggregate the mention information to calculate the entity representation to which it belongs. Furthermore, each mention comprises one or more words; therefore, one first must calculate the mention representation. Specifically, for entity $e_i$, assuming that the set of mentions is $E({e_i})$ by this entity, any mention $m_j$ in the mention set contains the start and end positions from the s-th to the t-th word. Because logsumexp pooling can aggregate semantic feature information well, it is used to represent the mention $m_j$:

$$\begin{aligned} h_{m_{j}}=\log \sum _{j=s}^{t} exp({h_j}) \end{aligned}$$

(3)

For the entity $e_i$ representation, we use the similar calculation method:

$$\begin{aligned} e_{i}=\log {\sum _{j=1}^{N}}\, {m_{j}\in E_{({e_i})}} exp(h_{m_{j}}) \end{aligned}$$

(4)

Semantic-guided attention and adaptive gating module

An SAAG mechanism is designed to aggregate sentence semantic information and dynamically manage intra-sentence relations or inter-sentence relations. In the following, the two technologies are introduced in detail.

Semantic-guided attention

For a target entity, the contribution of each word in a sentence to its prediction relation is different, especially for the prediction of intrasentence entity relations. Therefore, using the attention mechanism¹⁹ to guide sentence semantics by assigning different attention scores to different words is considered. Previous methods^20,21,22 only considered words when calculating references and entities without considering the impact of each word on the target entity or only considered the impact of the sentence where the entity is located on the entity. Therefore, this method only roughly calculates the semantic feature information of a sentence; however, each word in the sentence has an impact on the entity.

More-important words are assigned higher attention scores to determine the effect of each word on the target entity in the sentence. Multilayer perceptron attention²³ is used to guide the semantic sentence representation. Specifically, first, to better tie the words and entities in the sentence together, it is necessary to concatenate entity $e_i$ and word $w_{k}^{i}$ and then apply the softmax function to obtain the attention weight of each word in the sentence on the entity. Finally, the weighted sum of all words in the $S_m$ sentence for entity $e_i$ is obtained. This is similar to the case of $S_{n}^j$.

The more important words are assigned more attention scores to get the effect of each word on the target entity in the sentence. We apply MLP attention²³ to guide the semantic sentence representation. Specifically, first, in order to better tie the words and entities in the sentence together, we need to concatenate the entity $e_i$ and the word $w_{k}^{i}$, then we apply the softmax function to get the attention weight of each word in the sentence on the entity. Finally, we can obtain the weighted sum of all words in the $S_m$ sentence for entity $e_i$. Similar for $S_{n}^j$.

Here, assume that there are p words in the m-th sentence, we get the representation of the sentence $S_m$ where the entity $e_i$ is located:

$$\begin{aligned} \hat{h_k^{i}}= & [e_i;w_k^{i}] \end{aligned}$$

(5)

$$\begin{aligned} \omega _{k}^{i}= & \sigma (W_1\hat{h_k^{i}}+b_1) \end{aligned}$$

(6)

$$\begin{aligned} {\mu _k^{i}}= & \frac{exp(\omega _{k}^{i})}{\sum _{k=1}^{p}exp(\omega _{k}^{i})} \end{aligned}$$

(7)

$$\begin{aligned} S_{m}^i= & \sum _{k=1}^{p} \mu _k^{i} w_{k}^{i} \end{aligned}$$

(8)

where [ ; ] is concatenation operation. $\sigma (\cdot )$ is sigmoid function. $W_1$ and $b_1$ are learnable parameter and bias parameter, $\mu _k^{i}$ is the attention score of $\hat{h_k^{i}}$.

By concatenating $S_{m}^i$ and $S_{n}^j$, we can get the sentence semantic representation of entity pair $(e_i,e_j)$:

$$\begin{aligned} S_{(i,j)}=[S_{m}^i;S_{n}^j] \end{aligned}$$

(9)

In addition, to match textual contexts and filter irrelevant semantic information for target entity pairs $(e_i,e_j)$, the multihead attention mechanism is used to capture the attention of different subspaces to the target entity pairs further, and an entity-pair-aware attention strategy is introduced to generate document context representation $c_{(i,j)}$:

$$\begin{aligned} c_{(i,j)}= & \sum _{d=1}^{D}(A_i^d \odot A_j^d) \end{aligned}$$

(10)

$$\begin{aligned} C_{(i,j)}= & H^T\cdot softmax \left( \frac{c _{(i,j)}}{1^{T}{c _{(i,j)}}} \right) \end{aligned}$$

(11)

where $A_i$ and $A_j$ respectively are the attention weight of all words on entity $e_i$ and $e_j$, D is the number of multi-head, H is the entire document embedding representation, $\odot$ denotes matrix multiplication.

Adaptive gating

Based on the work of Yuan et al.²², an adaptive gated mechanism is designed to fuse sentence and context feature representations dynamically. Therefore, the proposed model can choose semantic sentence information to solve the intrasentence relation extraction task and enrich document context information to solve the intersentence relation extraction task. This technique can better capture local–global contextual information and promote the performance of DocRE.

Gating mechanism is used to adjust adaptively the representation of sentence sematic and document context representation as Eqs. (12) and (13) for targer entity pairs $(e_i,e_j)$:

$$\begin{aligned} g_{(i,j)}= & \sigma (W_g([S_{(i,j)};C_{(i,j)}])+b_g) \end{aligned}$$

(12)

$$\begin{aligned} G_{(i,j)}= & g_{(i,j)}\odot C_{(i,j)}+(1-g_{(i,j)}) \odot S_{(i,j)} \end{aligned}$$

(13)

where $\sigma (\cdot )$ denotes sigmoid function, $W_g$ and $b_g$ are learnable parameter and bias parameter, $\odot$ is element-wise multiplication, $G_{(i,j)}$ is semantic feature information of entity pair $(e_i,e_j)$.

Classification module

In the DocRE task, the relation extraction problem is a multilabel classification task. Specifically, for the entity pair $(e_i,e_j)$, the entity representation is concatenated with the feature information representations and relative distance representation to form the final representation of the target entity:

$$\begin{aligned} \mathbf {e_{(i,j)}}=[e_{(i,j)};G_{(i,j)};D_{(i,j)}] \end{aligned}$$

(14)

where $D_{(i,j)}$ is the relative distance representation from two entities of the first mentions in the document.

At last, we use a sigmoid function to calculate the probability of each relation r:

$$\begin{aligned} y_r=sigmoid(W_{r}\mathbf {e_{(i,j)}}+b_r) \end{aligned}$$

(15)

where $W_r$, $b_r$ is learnable parameter and bias vector.

We deploy binary cross-entropy loss to train our model, and we apply Adam optimizer²⁴ to optimize it during training. The loss function is reformulated as follows:

$$\begin{aligned} \mathcal {L}=-\sum _{r=1}^{R}\left( p_r\log \left( y_r\right) +\left( 1-p_r\right) \log \left( 1-y_r\right) \right) \end{aligned}$$

(16)

where R is the number of pre-defined relation type, $p_r \in \{0, 1\}$ is the ground truth value.

Experiment

Dataset

The proposed SAAG method is evaluated on two public DocRE datasets: (1) DocRED: The DocRE dataset created by Yao et al.⁷ is a large-scale dataset for DocRE that includes 3053/1000/1000 documents for training, development, and testing, respectively, with a total of 96 relation types. (2) CDR: The chemical disease reactions dataset introduced by Li et al.²⁵ is a widely used DocRE in the biomedical domain, which contains 1500 documents split into three equal-sized documents for training, development, and testing. Dissimilar to the DocRED, the CDR focuses on only one relation, “chemical-disease” with manually annotated.

Compared models

Our SAAG model compares with the following classic and competitive methods on two public DocRE datasets, including three types: Sequence-Based Models. These models leveraged various neural architectures to encode the given document into a hidden state sequence, including convolution neural networks (CNN)⁷, long short-term memory (LSTM)⁷ and bidirectional long short-term memory (BiLSTM)⁷. (2) Graph-based Models. The model converts the given document into a graph structure to explicitly tackle long-distance dependencies, including EOG⁴, GAIN⁸, HeterGSAN²⁶. (3) BERT-based Models. These picked up pre-trained models such as BERT²⁷ to implicitly capture long-distance dependencies to improve predicting task , including ATLOP$_{bert}$¹¹, SSAN$_{bert}$¹², DISCO$_{bert}$²¹, MRN$_{bert}$²⁸, EIDER$_{bert}$¹³, AFLKD$_{bert}$²⁹, GRACR$_{bert}$¹⁷, CPT-R$_{bert}$¹⁵, DREEAM$_{bert}$¹⁸, DRETC$_{bert}$³⁰, Correl$_{bert}$¹⁶, DRE-EA$_{bert}$³¹.

Implementation details

The proposed SAAG model is evaluated based on Hugging Face Transformers³² under PyTorch³³, as word embedding and encoder, following most of the experiment settings^5,7. SAAG$_{glove}$ applies 100-dimension Glove³⁴ and 128-dimension BiLSTM³⁵ as word embedding and encoder. SAAG$_{bert}$ uses Uncased BERT-Base as an encoder with a learning rate of 1e−5, and the word embeddings of Uncased BERT-Base are mapped to 128-dimension space by a linear layer. For CDR, we apply SciBERT³⁶ as the encoder. Early stopping is considered based on the dev set’s maximum F1 value, corresponding epoch, and threshold $\theta$. We adopt Adam²⁴ as the model optimiser for 200 epochs with a weight decay of 0.0001. All hyper-parameters are tuned, referring to the printed results on the dev set.

In addition to the widely used F1 values for DocRED, another evaluation indicator, Ign F1, is introduced. Yao et al.⁷ proposed Ign F1 for calculation with triples sharing in the training set removed. This approach is helpful for performance cheating leaks from the training set.

Experiments and results

DocRED results

Comparative experiments are conducted with three different model types using DocRED. Detailed experimental results are presented in Table 1. Among the models that do not use BERT as an encoder, SAAG with GloVe obtained 58.63 F1 in the test and outperformed the best model, MRN, by 0.17 points. This validates the necessity of incorporating sentence semantic features and document topic information into DocRED.

Table 1 Result comparison on DocRED.

The SAAG model, which uses BERT as an encoder, surpassed nearly all the models in terms of the F1 value and is only slightly behind the most competitive models. This is primarily because of the demonstrated strength of the DRE-EA model in evidence reasoning. However, the proposed model does not incorporate reasoning techniques, which results in a performance disadvantage compared with the DRE-EA model in relation extraction tasks that require reasoning. Additionally, among the compared models, the DREEAM model uses an attention mechanism that focuses on evidence sentences. Similarly, the EIDER model extracts and enhances evidence sentences to find supporting sentences for entity relations; however, it does not achieve better performance in DocRE. This is primarily because the attention mechanism used in this model focuses on evidence sentences, whereas the evidence for extracting intersentential entity relationships often comes from the entire document. Some models that use reasoning approaches for relation extraction, such as the GRACR and CPT-RI models, rely on entity-level reasoning. This results in larger modeling granularity, which limits their ability to capture finer-grained semantics. In contrast, the SAAG model focuses on the influence of words on entities, leading to a better performance in DocRE. Consequently, the performance of these models in this area is relatively low.

CDR results

Table 2 shows the detailed results of the test set for the CDR dataset. The proposed SAAG model obtains the best results. SAAG$_{SciBert}$ exceeds the best EIDER model by 0.29 F1. This suggests that SAAG has more substantial prediction power in DocRE tasks. This results from the two techniques of semantic-guided attention and adaptive gating. Combining them can further enhance the performance in relation extraction, even in many biomedical texts.

Table 2 Result comparison on CDR test set.

Ablation study

To analyze the effect of the proposed SAAG model, an additional ablation study is conducted on the DocRED dev set when the dependency of each module is excluded in SAAG. Table 3 lists the results. that the following can be observed: (1) Sentence semantic representation, document context representation, and adaptive gating mechanisms are essential components of the SAAG model. When each of these modules was removed from the SAAG model, the F1 values decreased by 2.53, 1.15, and 1.07, respectively, indicating that they played different roles in the SAAG model. (2) F1 dropped sharply by 2.53 on the dev set when sentence semantic representations are removed. This result is consistent with the fact that nearly 60% of the relations must be extracted from intrasentence relations in DocRED. This shows that the proposed sentence semantic representation module can address intrasentence relations more effectively. (3) F1 decreased by 1.15 on the dev set when document context representations are removed, which shows that document context representation plays a crucial role in capturing document-aware features. (4) F1 decreased by 1.07 on the dev set when the adaptive gated mechanism is removed, from which one can conclude that the adaptive gated mechanism provides an effective technique to adjust sentence semantic information and document context information dynamically and address the intrasentence and intersentence relation problem in DocRE in a more targeted manner.

Table 3 Performance of SAAG model with different components.

Intra- and inter-sentence relation extraction

The results of the ablation experiments demonstrate that that semantic and document context information has been specially designed to solve the problem of intra-sentence and inter-sentence relation extraction through an adaptive gate mechanism. Next, comprehensive experiments are conducted to verify further the performance of the model in intrasentence and intersentence relation extraction on the DocRED dev set.

The problem of distinguishing whether an entity pair has an intra-sentence or inter-sentence relationship must be addressed. When the relations of an entity pair can be predicted from a single sentence, the entity pair expresses intrasentence relations. Otherwise, it expresses intersentence relations. Unlike intrasentence relations, intersentence relations usually require complex interactions across multiple sentences throughout a document.

Figure 3 shows the intra-F1 and inter-F1 results for the dev set. One can observe that SAAG performs better than the previous models. For example, compared with the MRN model, the SAAG model improves the Intra-F1 value by 2.50 and the Inter-F1 value by 0.98. In addition, experiments on intrasentence and intersentence relation extraction show that, when the model enhances the performances of both intrasentence and intersentence relation extraction, the performance of DocRE improves. Thus, these experimental results validate the effectiveness of the two key components, semantic-guided attention and adaptive gating, in addressing intrasentence and intersentence relations in the proposed model.

Fig. 3

Intra- and inter-sentence performance (F1 value) of the models on the DocRED dev set.

Case study

Figure 4 shows the relation extraction performance of the SSAN$_{bert}$¹², EIDER$_{bert}$³², and SAAG models for instances from DocRED. The instance contains six sentences, in which both SSAN and EIDER can recognize the intrasentence relations of (Life in Color, United States) and (EDM, United States), as well as the intersentence relation of (Life in Color, SFX Entertainment). However, the SSAN and EIDER did not extract the relation between the entities “Life in Color Festival” and “United States”. If models can utilize the information of the entity “Life in Color” in the triplet (Life in Color, United States, P127), they can understand better the semantic information of the entity “Life in Color Festival”. EIDER focuses too much on the evidence reasoning chain; however, in this case, there is no evidence chain that can be reached (Life in Color Festival, Life in Color), only scattered and sparsely related reasoning information. Therefore, the advantages of EIDER cannot be utilized effectively. However, the SAAG model proposed focuses more on entity representation. The model not only constructs the connection between “Life in Color Festival” and “Life in Color” through the attention mechanism but also gives a more detailed weight division for words in the sentence. Therefore, the SAAG model provides a more effective solution for extracting the relation between the entity pair (Life in Color Festival, United States).

Fig. 4

A case study for SAAG model and two other models on DocRED.

However, it is evident that the proposed SAAG model did not correctly extract the relation between the entities “Dayglow” and “United States”, nor that between “Miami” and “United States”. In contrast, the EIDER model successfully captured the relation between “Dayglow” and “United States” by leveraging the advantages of evidence chains. It is necessary to investigate carefully the reasons for this. First, regarding the relation between “Dayglow” and “United States”, the SAAG model effectively recognizes the relation between “Life in Color” and “United States”, as well as “Life in Color” and “Dayglow”. However, it struggles with the relation mediated by the bridging entity “Life in Color”. Because “Life in Color” is mentioned most frequently and forms the longest relation chain, the semantic-guided attention technique in SAAG assigns more weight to “Life in Color”, resulting in less attention being allocated to the “Dayglow” entity within the same sentence. This leads to a lack of semantic connection when attempting to identify the relation between “Dayglow” and “United States”. Second, concerning the relation between “Miami” and “United States”, the SAAG model has certain shortcomings in common-sense reasoning methods. Furthermore, it allocated less attention to “Miami” in the same sentence as “Life in Color”, which resulted in failure to extract the relation between these entities accurately.

Conclusion

The SAAG model is developed specifically to address the issue of semantic ambiguity between entities in DocRE. The model utilizes semantic guidance techniques to model different representations of intrasentence and intersentence entities. In addition, it employs an adaptive gating mechanism to distinguish between intrasentence and intersentence entity pairs dynamically. However, this study still has somewhat coarse granularity in defining the relationships between entities.

Planned future work will focus on different reasoning paths^37,38 between entities, further refining the distinctions between intrasentence and intersentence entities to establish more-detailed entity pair relations, thereby enhancing the performance of DocRE.