Arabic Sign Language Recognition: A Multimodal Systematic Review, Taxonomy, and Benchmark Recommendations

Lyth Khaled Shbailat, Faiza Charfi, Anis Mezghani

Abstract


Over 430 million people worldwide experience disabling hearing loss, highlighting urgent needs for accessible communication technologies. This paper presents a systematic review of ArSL recognition covering 123 primary studies and 27 datasets (2000–mid-2025) across sensor- and vision-based modalities. We introduce a unified processing framework (data acquisition → preprocessing/segmentation → feature extraction → temporal modeling → post-processing) and a two-part taxonomy (by capture mechanism and by task: alphabet, isolated-word, continuous). Using consolidated comparison tables, we quantify trends: image-level CNNs routinely exceed ~94–99% on internal alphabet splits (e.g., ArSL2018), while signer-independent isolated-word and continuous experiments commonly show substantially lower generalization (examples in the text report signer-independent rates as low as ~64–75%). We identify critical gaps — a scarcity of large multimodal continuous corpora, limited backhand/orientation coverage, and underdeveloped signer-independent continuous recognition — and propose prioritized research directions and benchmark practices (canonical signer-independent splits, per-class metrics, and multimodal leaderboards) to accelerate reproducible progress and practical deployments.

Full Text:

PDF

References


J. N. Pérez, “Juan pablo bonet, reducción de las letras y arte para enseñar a hablar a los mudos,” RIFOP: Revista interuniversitaria de formación del profesorado: continuación de la antigua Revista de Escuelas Normales, no. 14, pp. 252–253, 1992.

N. E. Groce, Everyone here spoke sign language: Hereditary deafness on Martha’s Vineyard. Harvard University Press, 1985.

L. Ray, “The abbe’de l’epee,” American Annals of the Deaf and Dumb, vol. 1, no. 2, pp. 69–76, 1848.

M. A. Abdel-Fattah, “Arabic sign language: a perspective,” Journal of deaf studies and deaf education, vol. 10, no. 2, pp. 212–221, 2005.

B. S. Wilson and D. L. Tucci, “Addressing the global burden of hearing loss,” The Lancet, vol. 397, no. 10278, pp. 945–947, 2021.

T. Aujeszky and M. Eid, “A gesture recogintion architecture for Arabic sign language communication system,” Multimed. Tools Appl., vol. 75, no. 14, pp. 8493–8511, 2015.

A. RaySarkar, G. Sanyal, and S. Majumder, “Hand Gesture Recognition Systems: A Survey,” Int. J. Comput. Appl., vol. 71, no. 15, pp. 25–37, 2013.

M. J. Cheok, Z. Omar, and M. H. Jaward, “A review of hand gesture and sign language recognition techniques,” Int. J. Mach. Learn. Cybern., vol. 10, no. 1, pp. 131–153, 2019.

Y. Ying Wu and T. S. Huang, “Hand modeling, analysis and recognition,” IEEE Signal Process. Mag., vol. 18, no. 3, pp. 51–60, May 2001.

Ahmed, A. M., Alez, R. A., Taha, M., & Tharwat, G. (2016). Automatic translation of Arabic sign to Arabic text (ATASAT) system. Journal of Computer Science and Information Technology, 6, 109122.

R. M. Duwairi and Z. A. Halloush, “Automatic recognition of arabic alphabets sign language using deep learning.” International Journal of Electrical & Computer Engineering (2088-8708), vol. 12, no. 3, 2022.

M. Hassan, K. Assaleh, and T. Shanableh, “Multiple proposals for continuous Arabic sign language recognition,” Sensing and Imaging, vol. 20, no. 1, p. 4, 2019.

M. Mohandes, J. Liu, and M. Deriche, “A survey of image-based arabic sign language recognition,” in 2014 IEEE 11th International Multi-Conference on Systems, Signals & Devices (SSD14). IEEE, 2014, pp. 1–4.

M. Mohandes, S. Aliyu, and M. Deriche, “Arabic sign language recognition using the leap motion controller,” in 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE). IEEE, 2014, pp. 960–965.

C.-H. Chuan, E. Regina, and C. Guardino, “American sign language recognition using leap motion sensor,” in 2014 13th International Conference on Machine Learning and Applications. IEEE, 2014, pp. 541–544.

K. S. Abhishek, L. C. F. Qubeley, and D. Ho, “Glove-based hand gesture recognition sign language translator using capacitive touch sensor,” in 2016 IEEE international conference on electron devices and solid-state circuits (EDSSC). IEEE, 2016, pp. 334–337.

P. Molchanov, S. Gupta, K. Kim, and J. Kautz, “Hand gesture recognition with 3d convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2015, pp. 1–7.

R. Hartanto, A. Susanto, and P. I. Santosa, “Real time static hand gesture recognition system prototype for indonesian sign language,” in 2014 6th International Conference on Information Technology and Electrical Engineering (ICITEE). IEEE, 2014, pp. 1–6.

T.-Y. Pan, L.-Y. Lo, C.-W. Yeh, J.-W. Li, H.-T. Liu, and M.-C. Hu, “Real-time sign language recognition in complex background scene based on a hierarchical clustering classification method,” in 2016 IEEE second international conference on multimedia big data (BigMM). IEEE, 2016, pp. 64–67.

R. Yang, S. Sarkar, and B. Loeding, “Enhanced level building algorithm for the movement epenthesis problem in sign language recognition,” in 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2007, pp. 1 8.

——, “Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming,” IEEE transactions on pattern analysis and machine intelligence, vol. 32, no. 3, pp. 462– 477, 2009.

Ahmed, A. M., et al. (2017). Automatic Translation of Arabic Sign To Arabic Text (ATASAT) System. Computer Science & Information Technology, 109.

Saleh, Y., & Issa, G. (2020). Arabic sign language recognition through deep neural networks fine-tuning.

Hasasneh, A. (2020). Arabic Sign Language Characters Recognition Based on A Deep Learning Approach and a Simple Linear Classifier. Jordanian Journal of Computers and Information Technology, 6(3).

Kamruzzaman, M. (2020). Arabic sign language recognition and generating Arabic speech using convolutional neural network. Wireless Communications and Mobile Computing, 2020.

Tharwat, G., et al. (2021). Arabic sign language recognition system for alphabets using machine learning techniques. Journal of Electrical and Computer Engineering, 2021, 1-17.

Alani, A. A., & Cosma, G. (2021). ArSL-CNN: a convolutional neural network for Arabic sign language gesture recognition. Indonesian journal of electrical engineering and computer science, 22.

Alawwad, R. A., et al. (2021). Arabic sign language recognition using faster R-CNN. International Journal of Advanced Computer Science and Applications, 12(3).

Azhar, N. A. N., et al. (2022). Development of Mobile Application for Arabic Sign Language based on Android Studio Software. JOURNAL OF ALGEBRAIC STATISTICS, 13(3), 3152-3160.

R. M. Duwairi and Z. A. Halloush, “Automatic recognition of arabic alphabets sign language using deep learning.” International Journal of Electrical & Computer Engineering (2088-8708), vol. 12, no. 3, 2022

M. Zakariah, Y. A. Alotaibi, D. Koundal, Y. Guo, and M. Mamun Elahi, “Sign language recognition for Arabic alphabets using transfer learning technique,” Computational Intelligence and Neuroscience, vol. 2022, no. 1, p. 4567989, 2022.

M. M. Balaha, S. El-Kady, H. M. Balaha, M. Salama, E. Emad, M. Hassan, and M. M. Saafan, “A vision-based deep learning approach for independent-users arabic sign language interpretation,” Multimedia Tools and Applications, vol. 82, no. 5, pp. 6807–6826, 2023

M. A. Bencherif, M. Algabri, M. A. Mekhtiche, M. Faisal, M. Alsulaiman, H. Mathkour, M. Al-Hammadi, and H. Ghaleb, “Arabic sign language recognition system using 2d hands and body skeleton data,” IEEE Access, vol. 9, pp. 59 612–59 627, 2021.

A. Alnuaim, M. Zakariah, W. A. Hatamleh, H. Tarazi, V. Tripathi, and E. T. Amoatey, “Human-computer interaction with hand gesture recognition using resnet and mobilenet,” Computational Intelligence and Neuroscience, vol. 2022, no. 1, p. 8777355, 2022.

A. A. Alani and G. Cosma, “Arsl-cnn: a convolutional neural network for arabic sign language gesture recognition,” Indonesian journal of electrical engineering and computer science, vol. 22, 2021.

E. Aldhahri, R. Aljuhani, A. Alfaidi, B. Alshehri, H. Alwadei, N. Aljojo, A. Alshutayri, and A. Almazroi, “Arabic sign language recognition using convolutional neural network and mobilenet,” Arabian Journal for Science and Engineering, vol. 48, no. 2, pp. 2147–2154, 2023.

B. A. Dabwan and M. E. Jadhav, “A deep learning based recognition system for yemeni sign language,” in 2021 International Conference of Modern Trends in Information and Communication Technology Industry (MTICTI). IEEE, 2021, pp. 1–5.

I. Hmida and N. B. Romdhane, “Arabic sign language recognition algorithm based

on deep learning for smart cities,” in The 3rd International Conference on Distributed Sensing and Intelligent Systems (ICDSIS 2022), vol. 2022. IET, 2022, pp. 119–127.

H. I. Mohammed and J. Waleed, “Hand gesture recognition using a convolutional neural network for arabic sign language,” in AIP Conference Proceedings, vol. 2475, no. 1. AIP Publishing, 2023.

Al Ahmadi, S., Mohammad, F., & Al Dawsari, H. (2024). Efficient YOLO-based deep learning model for Arabic sign language recognition. Journal of Disability Research, *3*(4), 20240051. https://doi.org/10.1016/j.dhjo.2024.20240051

Batnasan, G., Gochoo, M., Otgonbold, M.-E., Alnajjar, F., & Shih, T. K. (2022). ArSL21L: Arabic sign language letter dataset benchmarking and an educational avatar for metaverse applications. In 2022 IEEE Global Engineering Education Conference (EDUCON) (pp. 1814–1821). IEEE.

Attia, N. F., Ahmed, M. T. F. S., & Alshewimy, M. A. M. (n.d.). Improved deep learning model based real-time recognition of Arabic sign language. [Unpublished manuscript or in preparation].

Alamri, F. S., Rehman, A., Abdullahi, S. B., & Saba, T. (2024). Intelligent real-life key-pixel image detection system for early Arabic sign language learners. PeerJ Computer Science, *10*, e2063. https://doi.org/10.7717/peerj-cs.2063

Aiouez, S., Hamitouche, A., Belmadoui, M. S., Belattar, K., & Souami, F. (2022). Real-time Arabic sign language recognition based on YOLOv5. In IMPROVE (pp. 17–25).

Mazen, F., & Ezz-Eldin, M. (2024). A novel image-based Arabic hand gestures recognition approach using YOLOv7 and ArSL21L. Fayoum University Journal of Engineering, *7*(1), 40–48.

Gochoo, M., Batnasan, G., Ahmed, A. A., Otgonbold, M.-E., Alnajjar, F., Shih, T. K., Tan, T.-H., & Wee, L. K. (2023). Fine-tuning vision transformer for Arabic sign language video recognition on augmented small-scale dataset. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 2880–2885). IEEE.

Mohamed, M. M. (2020). Automatic system for Arabic sign language recognition and translation to spoken one. International Journal, *9*(5), 7140–7148.

Sarhan, N. A., El-Sonbaty, Y., & Youssef, S. M. (2015). HMM-based Arabic sign language recognition using Kinect. In 2015 Tenth International Conference on Digital Information Management (ICDIM) (pp. 169–174). IEEE.

Assaleh, K., Shanableh, T., Fanaswala, M., Amin, F., & Bajaj, H. (2010). Continuous Arabic sign language recognition in user dependent mode. Journal of Intelligent Learning Systems and Applications, *2*(1), 19. https://doi.org/10.4236/jilsa.2010.21003

Mosa, D. T., Nasef, N. A., Lotfy, M. A., Abohany, A. A., Essa, R. M., & Salem, A. (2023). A real-time Arabic avatar for deaf–mute community using attention mechanism. Neural Computing and Applications, *35*(29), 21709–21723. https://doi.org/10.1007/s00521-023-08936-9

Boufelfel, R. (2024). An Arabic sign language recognition system for word-level generation and translation [Doctoral dissertation, University of Guelma].

Almaazmi, M., Elkadi, S., Elsayed, L., Salman, L., & Shanableh, T. (2024). Motion images with positioning information and deep learning for continuous Arabic sign language recognition in signer dependent and independent modes. IEEE Access. Advance online publication.

Shohieb, S. M., Elminir, H. K., & Riad, A. M. (2015). Signsworld atlas; a benchmark Arabic sign language database. Journal of King Saud University-Computer and Information Sciences, *27*(1), 68–76. https://doi.org/10.1016/j.jksuci.2014.03.005

Briones Cerquín, A. D., Tumay Guevara, J. A., & Ovalle, C. (2025). Mobile application for continuous recognition and classification of sign language images through deep learning. International Journal of Interactive Mobile Technologies, *19*(7).

ElBadawy, M., et al. (2017). Arabic sign language recognition with 3d convolutional neural networks. Paper presented at the 2017 Eighth international conference on intelligent computing and information systems (ICICIS).

Luqman, H., & Mahmoud, S. A. (2017). Transform-based Arabic sign language recognition. Procedia Computer Science, 117, 2-9.

Ibrahim, N. B., et al. (2017). An Automatic Arabic Sign Language Recognition System (ArSLRS). Journal of King Saud UniversityComputer and Information Sciences.

Abdel-Samie, A.-G. A.-R., et al. (2018). Arabic sign language recognition using kinect sensor. Research Journal of Applied Sciences, Engineering and Technology, 15(2), 57-67.

Elpeltagy, M., et al. (2018). Multi‐modality‐based Arabic sign language recognition. IET Computer Vision, 12(7), 1031-1039.

Hisham, B., & Hamouda, A. (2021). Arabic sign language recognition using Ada-Boosting based on a leap motion controller. International Journal of Information Technology, 13, 1221-1234.

Almasre, M. A., & Al-Nuaim, H. (2020). A comparison of Arabic sign language dynamic gesture recognition models. Heliyon, 6(3).

Mohamed, M. M. (2020). Automatic system for Arabic sign language recognition and translation to spoken one. International Journal, 9(5).

Aly, S., & Aly, W. (2020). DeepArSLR: A novel signer-independent deep learning framework for isolated arabic sign language gestures recognition. IEEE Access, 8, 83199-83212.

Luqman, H., & El-Alfy, E.-S. M. (2021). Towards hybrid multimodal manual and non-manual Arabic sign language recognition: MArSL database and pilot study. Electronics, 10(14), 1739.

Abdul, W., et al. (2021). Intelligent real-time Arabic sign language classification using attention-based inception and BiLSTM. Computers and Electrical Engineering, 95, 107395.

Boukdir, A., et al. Isolated video-based Arabic sign language recognition using convolutional and recursive neural networks. Arabian Journal for Science and Engineering, 1-13.

Balaha, M. M., et al. (2023). A vision-based deep learning approach for independent-users Arabic sign language interpretation. Multimedia Tools and Applications, 82(5), 6807-6826.

Alyami, S., et al. (2023). Isolated Arabic Sign Language Recognition Using A Transformer-based Model and Landmark Keypoints. ACM Transactions on Asian and Low-Resource Language Information Processing.

Assaleh, K., et al. (2010). Continuous Arabic sign language recognition in user dependent mode.

Tolba, M., et al. (2012). A proposed graph matching technique for Arabic sign language continuous sentences recognition. Paper presented at the Informatics and Systems (INFOS), 2012 8th International Conference on.

Tolba, M. F., et al. (2013). Arabic sign language continuous sentences recognition using PCNN and graph matching. Neural Computing and Applications, 23(3-4), 999-1010.

Sidig, A. a. I., et al. (2018). Arabic sign language recognition using optical flow-based features and HMM. Paper presented at the Recent Trends in Information and Communication Technology: Proceedings of the 2nd International Conference of Reliable Information and Communication Technology (IRICT 2017).

LUQMAN, H., & ELALFY, E. (2022). Utilizing motion and spatial features for sign language gesture recognition using cascaded CNN and LSTM models. Turkish Journal of Electrical Engineering and Computer Sciences, 30(7), 2508 2525.

Luqman, H., & Mahmoud, S. A. (2020). A machine translation system from Arabic sign language to Arabic. Universal Access in the Information Society, 19(4), 891-904.

Luqman, H. (2023). ArabSign: A Multi-modality Dataset and Benchmark for Continuous Arabic Sign Language Recognition. Paper presented at the 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG).

Tharwat, G., et al. (2021). Arabic sign language recognition system for alphabets using machine learning techniques. Journal of Electrical and Computer Engineering, 2021, 1-17.

Elons, A. S., et al. (2013). A proposed PCNN features quality optimization technique for pose-invariant 3D Arabic sign language recognition. Applied Soft Computing, 13(4), 1646-1660.

ElBadawy, M., et al. (2017). Arabic sign language recognition with 3d convolutional neural networks. Paper presented at the 2017 Eighth international conference on intelligent computing and information systems (ICICIS).

Sidig, A. A. I., et al. (2021). KArSL: Arabic sign language database. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 20(1), 1-19.

Nagi, J., et al. (2011). Max-pooling convolutional neural networks for vision-based hand gesture recognition. Paper presented at the 2011 IEEE international conference on signal and image processing applications (ICSIPA), Kuala Lumpur, Malaysia.

Assaleh, K., et al. (2012). Low complexity classification system for glove-based arabic sign language recognition. Paper presented at the Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar, November 12-15, 2012, Proceedings, Part III 19.

Mohandes, M., et al. (2014). Arabic sign language recognition using the leap motion controller. Paper presented at the 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE).

Shohieb, S. M., et al. (2015). Signsworld atlas; a benchmark Arabic sign language database. Journal of King Saud University-Computer and Information Sciences, 27(1), 68-76.

Ahmed, A. M., et al. (2016). Automatic translation of Arabic sign to Arabic text (ATASAT) system. Journal of Computer Science and Information Technology, 6, 109-122.

Alzohairi, R., et al. (2018). Image based arabic sign language recognition system. International Journal of Advanced Computer Science and Applications (IJACSA), 9(3), 185-194.

Ibrahim, N. B., et al. (2018). An automatic Arabic sign language recognition system (ArSLRS). Journal of King Saud UniversityComputer and Information Sciences, 30(4), 470-477.

Hayani, S., et al. (2019). Arab sign language recognition with convolutional neural networks. Pape presented at the 2019 International Conference of Computer Science and Renewable Energies (ICCSRE), Agadir, Morocco.

Latif, G., et al. (2019). ArASL: Arabic alphabets sign language dataset. Data in brief, 23, 103777.

Alnahhas, A., et al. (2020). Enhancing the recognition of Arabic sign language by using deep learning and leap motion controller. Int. J. Sci. Technol. Res, 9, 1865-1870.

Ismail, M. H., et al. (2021). Static hand gesture recognition of Arabic sign language by using deep CNNs. Indonesian Journal of Electrical Engineering and Computer Science, 24(1), 178-188.

Al-Barham, M., et al. (2023). RGB Arabic Alphabets Sign Language Dataset. arXiv preprint arXiv:2301.11932.

S. Bilal, R. Akmeliawati, A. A. Shafie, and M. J. E. Salami, “Hidden Markov model for human to computer interaction: a study on human hand gesture recognition,” Artif. Intell. Rev., vol. 40, no. 4, pp. 495–516, 2011.

P. K. Pisharady and M. Saerbeck, “Recent methods and databases in vision-based hand gesture recognition: A review,” Comput. Vis. Image Underst., vol. 141, pp. 152–165, 2015.

N. El-Bendary, H. M. Zawbaa, M. S. Daoud, A. E. Hassanien, and K. Nakamatsu, “ArSLAT: Arabic Sign Language Alphabets Translator,” in 2010 International Conference on Computer Information Systems and Industrial Management Applications, CISIM 2010, 2010, pp. 590–595.

M. A. Abdel-Fattah, “Arabic Sign Language: A Perspective,” J. Deaf Stud. Deaf Educ., vol. 10, no. 2, pp. 212–221, 2005.

A. Almohimeed, M. Wald, and R. I. Damper, “Arabic Text to Arabic Sign Language Translation System for the Deaf and Hearing-Impaired Community,” in Proceedings of the Second Workshop on Speech and Language Processing for Assistive Technologies, 2011, pp. 101–109.

E. Aghajari and D. Gharpure, “Real Time Vision-Based Hand Gesture Recognition for Robotic Application,” Int. J. Adv. Res. Comput. Sci. Softw. Eng., vol. 4, no. 3, pp. 2277–128, 2014.

E. Stergiopoulou and N. Papamarkos, “Hand gesture recognition using a neural network shape fitting technique,” Eng. Appl. Artif. Intell., vol. 22, no. 8, pp. 1141–1158, Dec. 2009.

S. Kausar and M. Y. Javed, “A Survey on Sign Language Recognition,” in Frontiers of Information Technology (FIT), 2011, pp. 95–98.

H. Hasan and S. Abdul-Kareem, “Human–computer interaction using vision-based hand gesture recognition systems: a survey,” Neural Comput. Appl., vol. 25, no. 2, pp. 251–261, 2014.

Y. Zhou, G. Jiang, and Y. Lin, “A novel finger and hand pose estimation technique for real-time hand gesture recognition,” Pattern Recognit., vol. 49, pp. 102–114, 2016.

K. Oka, Y. Sato, and H. Koike, “Real-time fingertip tracking and gesture recognition,” IEEE Comput. Graph. Appl., vol. 22, no. 6, pp. 64–71, 2002.

J. Triesch and C. Von Der Malsburg, “A system for person-independent hand posture recognition against complex backgrounds,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 12, pp. 1449– 1453, 2001.

M. a Berbar, H. M. Kelash, and A. a Kandeel, “Faces and Facial Features Detection in Color Images,” in Geometric Modeling and Imaging – New Trends (GMAI06), 2006, pp. 209–214.

Ahmed, A. M., et al. (2017). Automatic Translation of Arabic Sign To Arabic Text (ATASAT) System. Computer Science & Information Technology, 109.

Tharwat, G., et al. (2021). Arabic sign language recognition system for alphabets using machine learning techniques. Journal of Electrical and Computer Engineering, 2021, 1-17.

Tubaiz, N., et al. (2015). Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Transactions on Human-Machine Systems, 45(4), 526-533.

Assaleh, K., & Al-Rousan, M. (2005). Recognition of Arabic sign language alphabet using polynomial classifiers. EURASIP Journal on Advances in Signal Processing, 2005(13), 507614.

Tharwat, A., et al. (2015). Sift-based arabic sign language recognition system. Paper presented at the Afro european conference for industrial advancement.

Bauer, B., & Hienz, H. (2000). Relevant features for video-based continuous sign language recognition. Paper presented at the Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on.

Abdel-Wahab, M. S., et al. (2006). Arabic sign language recognition using neural network and graph matching techniques. Paper presented at the Proceedings of the 6th WSEAS International Conference on Applied Informatics and Communications.

Sidig, A. A. I., et al. (2021). KArSL: Arabic sign language database. ACM Transactions on Asian and Low Resource Language Information Processing (TALLIP), 20(1), 1-19.

G. Latif, J. Alghazo, N. Mohammad, and R. Alghazo, “Communicating with the deaf and hard of hearing through automatic arabic sign language translator,” in Journal of Physics: Conference Series, vol. 1962, no. 1. IOP Publishing, 2021, p. 012055.

N. M. Alharthi and S. M. Alzahrani, “Vision transformers and transfer learning approaches for arabic sign language recognition,” Applied Sciences, vol. 13, no. 21, p. 11625, 2023.




DOI: https://doi.org/10.31449/inf.v49i25.10879

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.