Hybrid Deep Learning Framework for Comprehensive Depression Assessment through facial Images and Text Data Integration

Abstract

In this research, a hybrid model for reviewing depression severity by calculating Beck Depression Inventory (BDI) scores through the integration of facial image analysis and sentiment analysis of textual data. This comprehensive approach leverages deep neural techniques to improve the accuracy of mental health diagnostics. The image dataset comprises facial expressions labeled with corresponding BDI scores, while the text dataset, stored in CSV format, includes written responses labeled with sentiment or BDI scores. Preprocessing steps for the image data involve resizing, grayscale conversion, and normalization, ensuring uniformity and improved feature extraction. For text data, preprocessing includes cleaning, tokenization, stopword removal, and sequencing, making the text suitable for model input. Conv2D (Conv2D) are used for facial image classification to predict various emotional states, while Natural Language Processing (NLP) techniques are used for sentiment analysis of text. The fusion algorithm integrates the outputs from both the image and text models by extracting, concatenating, and processing features through dense layers to predict the final BDI score. This hybrid approach leverages the strengths of both data types, providing a more robust and nuanced assessment of depression severity. Our results demonstrate that combining visual and textual information significantly enhances the predictive accuracy of BDI scores, offering a promising tool for more precise mental health diagnostics and personalized treatment plans. This work contributes to the advancement of AI-driven mental health assessment, aiming to improve early detection and intervention for individuals suffering from depression.