Multimodal Transfer Learning for Crisis Response: A Unified DeBERTa–ResNet Framework on CrisisMMD

Authors

  • Haffiz Ud Din Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author https://orcid.org/0009-0005-3377-8454
  • Musharraf Ali Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author
  • Anzal Hussain Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author
  • Akmal Hussain Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author
  • Syed Qamar Abbas Kazmi Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author
  • Hannan Hussain Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author
  • Syed Hassan Abbas Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author
  • Sohail Ahmed Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan Author

DOI:

https://doi.org/10.57041/8fpqd649

Keywords:

Multimodal deep learning, transfer learning, crisis informatics, DeBERTa, ResNet, social media, disaster response, CrisisMMD

Abstract

Social media platforms generate large volumes of multimodal content during natural disasters that can support situational awareness and humanitarian decision-making. However, effectively analyzing both textual and visual information remains challenging due to weak cross-modal alignment, severe class imbalance, noisy user-generated data, and limited labeled samples. This paper presents a unified multimodal transfer-learning framework that combines a frozen DeBERTa-v3-base text encoder with a frozen ResNet-50 image encoder through a simple late-fusion strategy for crisis-related classification. The architecture projects the unimodal features into a common 256-dimensional space, concatenates them, and feeds the resulting representation into a lightweight multi-layer perceptron classifier. The framework is evaluated on all three official tasks of the CrisisMMD v2.0 dataset. Experimental results show an accuracy of 0.76 (macro-F1 = 0.74) on the informativeness task, 0.43 (macro-F1 = 0.39) on the humanitarian categories task, and 0.69 (macro-F1 = 0.64) on the damage severity task. The model performs relatively well on classes with distinctive multimodal signatures, while residual confusion remains on semantically overlapping or visually subtle categories. For the damage severity task, which is primarily image-based, the multimodal model achieves essentially the same performance as an image-only configuration, indicating limited contribution from text. The study also acknowledges that formal text-only and image-only ablation experiments under the same protocol were not conducted, so the precise benefit of fusion cannot be rigorously quantified. Overall, the work demonstrates that a computationally efficient frozen late-fusion architecture can serve as a practical and reproducible reference point for multimodal crisis informatics under the resource constraints typical of real-world disaster response.

Author Biography

  • Haffiz Ud Din, Department of Research and Development, Gilgit-Baltistan First Research Center, Binary HUB, Gilgit-Baltistan, Pakistan

    I’m an AI/DS practitioner committed to building trustworthy, end-to-end machine learning systems that create real-world impact. My focus is on clear problem framing, careful data curation, robust modeling, and rigorous evaluation—paired with reproducibility, ethics, and continuous learning.

    Areas I’m passionate about include medical AI and clinical decision support, perception and tracking for autonomous systems, multimodal learning with geospatial context, and scientific ML such as protein classification. I also enjoy applying ML to systems analytics and performance modeling. Beyond building models, I value mentoring, collaborative research, and communicating results with clarity.

    I’m motivated by meaningful problems, disciplined execution, and steady improvement. Open to collaborations and roles where rigorous ML, careful experimentation, and responsible deployment are core values.

Downloads

Published

2026-11-13

How to Cite

Multimodal Transfer Learning for Crisis Response: A Unified DeBERTa–ResNet Framework on CrisisMMD. (2026). International Journal of Emerging Engineering and Technology, 5(1-1), 60-71. https://doi.org/10.57041/8fpqd649

Similar Articles

1-10 of 34

You may also start an advanced similarity search for this article.