Crisis impact extraction language model using transformer-based synergy via transfer learning

To enhance society's resilience to various disasters, quick acquisition of reliable, crisis-related information on social or economic losses is critical. Extracting crisis impact text data from social media and news articles has been widely utilized using various Natural Language Processing (NLP) technologies. However, the low fidelity and limited number of entities in the crisis impact text data present challenges in effectively utilizing this information for crisis management. Furthermore, novel language models are competitively developed in various applications, which require vast data to train the model. In the crisis management field, the scarcity of data and irregular structure of context have bottlenecked the development of domain-adaptive language models, limiting the enrichment of textual data. To address these challenges, we propose a novel language model that integrates state-of-the-art transformer-based synergy using Bidirectional Encoder Representations from Transformers (BERT) and Generative Pretrained Transformers (GPT) by transfer learning, CIE-BERG model (Crisis Impact Extraction model with BERT-GPT synergy). First, the GPT model is employed to generate a text corpus and annotate a custom entity dataset to train and fine-tune our pretrained model. Second, a domain-adaptive transfer learning process for crisis management, including the Entity-Aware Masked Language Model (EAMLM) and Causal Contextual Attention (CCA), is developed, integrating the strengths of transformer's text generation and contextual comprehension. Finally, the developed model is fine-tuned to perform high-fidelity crisis impact extraction across diverse custom entity recognition tasks. Our CIE-BERG model outperformed other Language Models (LMs) on both synthetic and real data evaluation, which revealed significant advancements in crisis impact information extraction tasks.