Mobile devices (i.e., smartphone and tablets) are becoming a common tool to support learning since primary school. Indeed, many didactic applications are already available on online stores. Most of these applications engage the child by immersing the educational purpose of the software within an entertaining environment, often in the form of a game with sophisticated
graphic and interaction. Unfortunately, this makes most of these applications inaccessible to visually impaired children.
In our research we focus on the design, development and testing of accessible applications for mobile devices to support the education of people with visual impairments.
Supporting Math learning in primary school with MathMelodies
MathMelodies is an iPad application that supports primary school children in learning Mathematics. MathMelodies has been designed and implemented to be enjoyable by both visually impaired and sighted children. The software has been first developed as a university prototype and then, thanks to a crowdfunding campaign, engineered and
distributed as a commercial application. It is available for free on iPad from the iTunes Store.
MathMelodies is accessible and entertaining to sighted and BVI (blind or visually impaired) children and presents a fantasy tale in which the child solves math exercises to advance the story (13 types of exercises at different difficulty levels). The image below shows an example of a counting exercise.
Since its release, the app was downloaded more than 50,000 times worldwide, and in particular from English speaking countries and Italy.
Developing assistive application with React Native
Cross-platform developing techniques have been attracting lot of attention in the last years, especially in the field of mobile application, because they enable the developers to code apps in a single programming language and then to deploy on different platforms (e.g. iOS and Android). One well-known framework for cross-platform development is
React Native. While this framework presents some features to support accessibility to blind or visually impaired people, to the best of our knowledge the accessibility of applications developed with this framework has not been systematically investigated.
In our research we developed MathMelodies 2, an app prototype that builds on MathMelodies and that was developed with React Native to run on both iOS and Android smartphones and tablets. Based on our experience we can conclude that, while basic accessibility features are supported by React Native, one advanced capability, which is required by MathMelodies 2, is not supported. Thus, it is necessary to write additional native accessibility components.
Acquiring literacy skills with WordMelodies
WordMelodies is a cross-platform prototype aimed at supporting primary school children in acquiring and practicing basic literacy skills. It includes 8 different types of exercises. In the following examples, upon tapping on the speaker symbol, the target word is read. Instead, upon tapping on an animal icon, its call is played.
Drag&Drop based exercises
Multiple choice answer exercises
- Niccolò Cantù, Mattia Ducci, Dragan Ahmetovic, Cristian Bernareggi, Sergio Mascetti. MathMelodies 2: a Mobile Assistive Application for People with Visual Impairments Developed with React Native. In proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). 2018.
- Dragan Ahmetovic, Valeria Alampi, Cristian Bernareggi, Andrea Gerino, Sergio Mascetti. Math Melodies: Supporting Visually Impaired Primary School Students in Learning Math. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work (p. 26). 2017, ACM.
- Cristian Bernareggi, Andrea Gerino and Sergio Mascetti. Accessible Mathematics on Touchscreen Devices: New Opportunities for People with Visual Impairments. In the Proceedings of the 3rd International Workshop on Digitization and E-Inclusion in Mathematics and Science 2016 (DEIMS2016). 2016.
- Andrea Gerino, Nicolò Alabastro, Cristian Bernareggi, Dragan Ahmetovic, Sergio Mascetti, MathMelodies: inclusive design of a didactic game to practice Mathematics. In proceedings of the 14th International Conference on Computers Helping People with Special Needs (ICCHP). Springer, 2014.