ENG
INA
90th Doctoral Promotion
Promovendus I Dewa Made Bayu Atmaja Darmawan succeeded in his doctoral dissertation exam and became the 90th PhD of the Study Program of Doctoral Engineering Science (PSDIT), Faculty of Engineering, Udayana University. The dissertation titled SIBI Sign Language Translation : Multi-Channel Recognition of Facial Expressions and Gesture Movements in Total Communication received a cumlaude predicate.
Dr. I Dewa Made Bayu Atmaja Darmawan explained that Sign language is the primary communication technique for individuals who are deaf (Deaf). This form of communication relies on the principle of total communication, which combines hand gestures (manual cues) with supporting components (non-manual cues) such as facial expressions, lip movements, and body dynamics. In the context of sentences, continuous sign language forms a sequence of complex movements that must be transformed into Indonesian text through an effective translation system.
This study implements temporal segmentation to separate continuous signs into isolated units. This process uses the dense optical flow method, which detects sign boundaries based on pixel movements between frames. After segmentation, frame selection is applied to reduce the number of frames entering the recognition process. For this purpose, a Motion-Adaptive Frame Selection Optical flow-Based (MAFS-OF) method is developed, which adapts to the characteristics of the movement. MAFS-OF selects key frames based on motion magnitude to filter important information while reducing visual redundancy.
The key frames are used in two sign recognition approaches: a Modified STMC and VGG11+ConvLSTM, which are compared in terms of accuracy and computational efficiency. This study also integrates facial expression recognition to identify the signer’s emotions by modifying the feature extraction stage to recognize non-manual cues. The outputs of both channels—namely, glosses (morphemes resulting from sign recognition) and emotion predictions—are constructed into Indonesian sentences using the Transformer T5 model, which has been detuned to align with the linguistic structure of the Indonesian language. The main focus of this research is the translation of Indonesian sign language (SIBI) through the development of a comprehensive dataset from 22 signers, encompassing both hand gestures and facial expressions. Optical flow-based temporal segmentation successfully separates continuous signs with satisfactory accuracy. Frame selection using MAFS-OF reduces the number of frames by up to 43% without degrading performance. The VGG11+ConvLSTM architecture shows competitive performance compared to the Modified STMC based on F1-Score and Word Error Rate (WER). The application of the Transformer T5 for translation that combines manual and non-manual signs also yields positive results, as measured by BLEU, ROUGE-L, and METEOR scores, although still limited by the scope of the available data.
The promoter, co-promoters I and II, are Prof. Ir. Linawati, M.Eng.Sc., PhD., IPU, Dr. Ir. Gede Sukadarmika, ST, M.Sc. IPU and Prof. Ir. Ni Made Ary Esta Dewi Wirastuti, ST, M.Sc, Ph.D., IPU., respectively.
FACULTY OF ENGINEERING