The proposed project SMILE-II aims to research and build advanced technology for sign language learning. More precisely, the proposed project builds on the groundwork laid down by the SNSF Sinergia project SMILE, which dealt with assessment of the manual activity of Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) in isolated signs produced by early learners and L2 learners. SMILE-II will extend this technology to continuous sign language assessment including both manual and non-manual components of signs so that a DSGS learner’s sentence-level production can be assessed in an automatic manner.
The proposed goal is faced with several challenges: (a) lack of DSGS resources; (b) continuous sign language can contain both manual information and non-manual information and almost all work on recognition to date has focused on manual information only. Furthermore, continuous sign language recognition and continuous sign language production are still open research problems; (c) lack of standardized
instruments for DSGS assessment; and (d) lack of automatic methods to assess continuous sign productions at both the manual and non-manual level.
To achieve the goal of the project, SMILE-II brings together researchers from different fields, namely, sign language linguistics, sign language assessment, sign language technology, and spoken language technology, across four different institutions, namely, the Idiap Research Institute (Martigny, Switzerland), the Interkantonale Hochschule für Heilpädagogik Zürich (HfH), the University of Zurich, and the University of Surrey (UK).
The planned research is organized into four interdependent sub-projects:
1. Sub-project 1 focuses on continuous DSGS resource creation (including longitudinal DSGS learner data) and linguistic analysis.
2. Sub-project 2 focuses on developing methods to extract manual and non-manual information from continuous signing and developing deep learning-based sign language recognition and sign language production approaches.
3. Sub-project 3 focuses on developing instruments for sign language assessment aligned to the Common European Framework of Reference for Languages (CEFR).
4. Sub-project 4 focuses on tools for automatic sign language assessment based on the data (Sub-project 1), technology (Sub-project 2), and instruments (Sub-project 3).
The SMILE-II project will involve not only experienced and internationally known researchers in their respective fields, but also young hearing and Deaf team members. The results of the project are expected to have an impact in the Deaf community beyond DSGS, the sign language linguistics community, the sign language technology community, and the sign language education community.