Integrated Sensor Fusion and Multi-Modal Hierarchical Neural Network for Activity Recognition in Lower Limb Prosthetics
Main Article Content
Abstract
Activity recognition plays pivotal role in enhancing functionality for prosthetic devices, ensuring seamless integration with users' movements. However, the complexity arises from the diverse data sources, including acceleration, angular velocity, joint angles, orientation, electromyography (EMG), and marker data, necessitating a robust approach to overcome challenges in information integration. The primary challenge lies in the effective utilization of multiple sensor modalities, each with unique characteristics and potential noise sources. The proposed solution addresses this by employing advanced sensor fusion techniques, such as Kalman filtering, during data collection. Synchronization and resampling ensure temporal consistency, while noise reduction techniques, such as low-pass filters, mitigate signal distortions. To further refine the process, a hybrid optimization-based feature selection Adaptive Step Size in Marine Predators Algorithm (ASSMPA) is introduced, focusing on marker data features. ASSMPA synergizes Marine Predators Algorithm (MPA) and Pathfinder Algorithm (PFA) for optimal feature selection in marine predator pathfinding tasks. The Feature Fusion step integrates attention mechanisms to dynamically weigh the significance of different sensor modalities during the fusion process. This strategic fusion enhances the overall performance of the Multi-Modal Hierarchical Neural Network (MMHNN). The proposed model is implemented using Python.
Article Details
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
References
Slemenšek, J., Fister, I., Geršak, J., Bratina, B., van Midden, V.M., Pirtošek, Z. and Šafarič, R., 2023. Human gait activity recognition machine learning methods. Sensors, 23(2), p.745.
Xu, G., Wan, Q., Deng, W., Guo, T. and Cheng, J., 2022. Smart-Sleeve: A wearable textile pressure sensor array for human activity recognition. Sensors, 22(5), p.1702.
Li, C., Li, G., Jiang, G., Chen, D. and Liu, H., 2020. Surface EMG data aggregation processing for intelligent prosthetic action recognition. Neural Computing and Applications, 32, pp.16795-16806.
Cao, T., Liu, D., Wang, Q., Bai, O. and Sun, J., 2020. Surface Electromyography-Based Action Recognition and Manipulator Control. applied sciences, 10(17), p.5823.
Cui, J.W., Li, Z.G., Du, H., Yan, B.Y. and Lu, P.D., 2022. Recognition of Upper Limb Action Intention Based on IMU. Sensors, 22(5), p.1954.
Hussain, T., Iqbal, N., Maqbool, H.F., Khan, M., Awad, M.I. and Dehghani-Sanij, A.A., 2020. Intent based recognition of walking and ramp activities for amputee using sEMG based lower limb prostheses. Biocybernetics and biomedical engineering, 40(3), pp.1110-1123.
Vijayvargiya, A., Gupta, V., Kumar, R., Dey, N. and Tavares, J.M.R., 2021. A hybrid WD-EEMD sEMG feature extraction technique for lower limb activity recognition. IEEE Sensors Journal, 21(18), pp.20431-20439.
Su, B.Y., Wang, J., Liu, S.Q., Sheng, M., Jiang, J. and Xiang, K., 2019. A CNN-based method for intent recognition using inertial measurement units and intelligent lower limb prosthesis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(5), pp.1032-1042.
Cheng, S., Bolívar-Nieto, E. and Gregg, R.D., 2021. Real-time activity recognition with instantaneous characteristic features of thigh kinematics. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29, pp.1827-1837.
Vijayvargiya, A., Khimraj, Kumar, R. and Dey, N., 2021. Voting-based 1D CNN model for human lower limb activity recognition using sEMG signal. Physical and Engineering Sciences in Medicine, 44, pp.1297-1309.
Zhang, K., Xiong, C., Zhang, W., Liu, H., Lai, D., Rong, Y. and Fu, C., 2019. Environmental features recognition for lower limb prostheses toward predictive walking. IEEE transactions on neural systems and rehabilitation engineering, 27(3), pp.465-476.
Iqbal, N., Khan, T., Khan, M., Hussain, T., Hameed, T. and Bukhari, S.A.C., 2021. Neuromechanical signal-based parallel and scalable model for lower limb movement recognition. IEEE Sensors Journal, 21(14), pp.16213-16221.
Luan, Y., Shi, Y., Wu, W., Liu, Z., Chang, H. and Cheng, J., 2021. Har-semg: A dataset for human activity recognition on lower-limb semg. Knowledge and Information Systems, 63, pp.2791-2814.
Peng, F., Zhang, C., Xu, B., Li, J., Wang, Z. and Su, H., 2020. Locomotion prediction for lower limb prostheses in complex environments via sEMG and inertial sensors. Complexity, 2020, pp.1-12.
Wang, J., Cao, D., Wang, J. and Liu, C., 2021. Action recognition of lower limbs based on surface electromyography weighted feature method. Sensors, 21(18), p.6147.
Li, X., Liu, Z., Gao, X. and Zhang, J., 2020. Bicycling Phase Recognition for Lower Limb Amputees Using Support Vector Machine Optimized by Particle Swarm Optimization. Sensors, 20(22), p.6533.
Wang, T., Liu, N., Su, Z. and Li, C., 2019. A new time–frequency feature extraction method for action detection on artificial knee by fractional fourier transform. Micromachines, 10(5), p.333.
Pergolini, A., Livolsi, C., Trigili, E., Chen, B., Giovacchini, F., Forner-Cordero, A., Crea, S. and Vitiello, N., 2022. Real-time locomotion recognition algorithm for an active pelvis orthosis to assist lower-limb amputees. IEEE Robotics and Automation Letters, 7(3), pp.7487-7494.
Y. Lv, J. Xu, H. Fang, X. Zhang and Q. Wang, "Data-Mined Continuous Hip-Knee Coordination Mapping with Motion Lag for Lower-Limb Prosthesis Control," in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 30, pp. 1557-1566, 2022, doi: 10.1109/TNSRE.2022.3179978.
J. Kim, N. Colabianchi, J. Wensman and D. H. Gates, "Wearable Sensors Quantify Mobility in People with Lower Limb Amputation During Daily Life," in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 6, pp. 1282-1291, June 2020, doi: 10.1109/TNSRE.2020.2990824.
N. H. D. Nordin, A. G. A. Muthalif, M. K. M. Razali, A. Ali and A. M. Salem, "Development and Implementation of Energy-Efficient Magnetorheological Fluid Bypass Damper for Prosthetics Limbs Using a Fuzzy-Logic Controller," in IEEE Access, vol. 10, pp. 18978-18987, 2022, doi: 10.1109/ACCESS.2022.3149893.
M. Hu, Y. He, G. Hisano, H. Hobara and T. Kobayashi, "Coordination of Lower Limb During Gait in Individuals with Unilateral Transfemoral Amputation," in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 3835-3843, 2023, doi: 10.1109/TNSRE.2023.3316749.
S. S. Mishra, A. T, B. K. Panigrahi and D. Joshi, "Neuromechanical Model-Based Corrective Torque Estimation During Weight Shifting in Lower Limb Amputees," in IEEE Transactions on Automation Science and Engineering, vol. 20, no. 4, pp. 2350-2366, Oct. 2023, doi: 10.1109/TASE.2022.3210569.
R. Stolyarov, M. Carney and H. Herr, "Accurate Heuristic Terrain Prediction in Powered Lower-Limb Prostheses Using Onboard Sensors," in IEEE Transactions on Biomedical Engineering, vol. 68, no. 2, pp. 384-392, Feb. 2021, doi: 10.1109/TBME.2020.2994152.
Vijayvargiya, A., Singh, B., Kumar, R., Desai, U. and Hemanth, J., 2022. Hybrid Deep Learning Approaches for sEMG Signal-Based Lower Limb Activity Recognition. Mathematical Problems in Engineering, 2022.
Dataset collected from: “https://data.mendeley.com/datasets/k5y9jkx87y/1”, dated 29/11/2023.