Isolated sign language recognition through integrating pose data and motion history images
Yazarlar (2)
Dr. Öğr. Üyesi Ali AKDAĞ
Tokat Gaziosmanpaşa Üniversitesi, Türkiye
Konya Teknik Üniversitesi, Türkiye
| Makale Türü |
Özgün Makale
|
| Makale Alt Türü | SSCI, AHCI, SCI, SCI-Exp dergilerinde yayınlanan tam makale |
| Dergi Adı | Peerj Computer Science |
| Dergi ISSN | 2376-5992 Wos Dergi Scopus Dergi |
| Dergi Tarandığı Indeksler | SCI-Expanded |
| Dergi Grubu | Q2 |
| Makale Dili | Türkçe |
| Basım Tarihi | 05-2024 |
| Cilt No | 10 |
| Sayı | 1 |
| DOI Numarası | 10.7717/PEERJ-CS.2054 |
| Makale Linki | http://dx.doi.org/10.7717/peerj-cs.2054 |
| Özet |
| This article presents an innovative approach for the task of isolated sign language recognition (SLR); this approach centers on the integration of pose data with motion history images (MHIs) derived from these data. Our research combines spatial information obtained from body, hand, and face poses with the comprehensive details provided by three-channel MHI data concerning the temporal dynamics of the sign. Particularly, our developed finger pose-based MHI (FP-MHI) feature significantly enhances the recognition success, capturing the nuances of finger movements and gestures, unlike existing approaches in SLR. This feature improves the accuracy and reliability of SLR systems by more accurately capturing the fine details and richness of sign language. Additionally, we enhance the overall model accuracy by predicting missing pose data through linear interpolation. Our study, based on the randomized leaky rectified linear unit (RReLU) enhanced ResNet-18 model, successfully handles the interaction between manual and non-manual features through the fusion of extracted features and classification with a support vector machine (SVM). This innovative integration demonstrates competitive and superior results compared to current methodologies in the field of SLR across various datasets, including BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL, in our experiments. |
| Anahtar Kelimeler |
| Deep learning | Feature fusion | Motion history image | Sign language recognition |