Abstract

Mobile devices (i.e., smartphone and tablets) are becoming a common tool to support learning since primary school. Indeed, many didactic applications are already available on online stores. Most of these applications engage the child by immersing the educational purpose of the software within an entertaining environment, often in the form of a game with sophisticated graphic and interaction. Unfortunately, this makes most of these applications inaccessible to visually impaired children.

In our research, we focus on the design, development and testing of accessible applications for mobile devices to support the education of people with visual impairments.

Supporting Math learning in primary school with MathMelodies

MathMelodies is an iPad application that supports primary school children in learning Mathematics. MathMelodies has been designed and implemented to be enjoyable by both visually impaired and sighted children. The software has been first developed as a university prototype and then, thanks to a crowdfunding campaign, engineered and distributed as a commercial application. It is available for free on iPad from the iTunes Store and for Android devices on Google play store.

MathMelodies is accessible and entertaining to sighted and BVI (blind or visually impaired) children and presents a fantasy tale in which the child solves math exercises to advance the story (13 types of exercises at different difficulty levels). The image below shows an example of a counting exercise.

Since its release, the app was downloaded more than 50,000 times worldwide, and in particular from English speaking countries and Italy.

Developing assistive applications with Cross-Platform developing tools

Cross-platform developing techniques have been attracting lot of attention in the last years, especially in the field of mobile applications, because they enable the developers to code apps in a single programming language and then to deploy on different platforms (e.g. iOS and Android). One well-known framework for cross-platform development is React Native.  While this framework presents some features to support accessibility to blind or visually impaired people, to the best of our knowledge the accessibility of applications developed with this framework has not been systematically investigated.

In our research we developed MathMelodies 2, an app prototype that builds on MathMelodies and that was developed with React Native to run on both iOS and Android smartphones and tablets. Based on our experience we can conclude that, while basic accessibility features are supported by React Native, one advanced capability, which is required by MathMelodies 2, is not supported. Thus, it is necessary to write additional native accessibility components.

Acquiring literacy skills with WordMelodies

WordMelodies is a cross-platform prototype aimed at supporting primary school children in acquiring and practicing basic literacy skills. It includes 8 different types of exercises. In the following examples, upon tapping on the speaker symbol, the target word is read. Instead, upon tapping on an animal icon, its call is played.

Drag&Drop based exercises

Multiple choice answer exercises

Multimodal exploration of mathematical function graphs

When sighted people observe a mathematical function graph they can grasp the overall trend (e.g., it is an increasing function, it is always above the x-axis) and focus on details (e.g., the function passes by <0, 1>). Conveying this information without sight is a challenge. We developed and evaluated a web-based platform-independent approach that uses sonification and multimodal proprioceptive exploration to quickly convey the overall trend of a function graph and that allows the user to explore details of the function with text-to-speech.

A demo of the system is available online.

Bibliography

• Bernareggi, Cristian, Dragan Ahmetovic, and Sergio Mascetti. "μGraph: Haptic Exploration and Editing of 3D Chemical Diagrams." The 21st International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 2019.
• Ahmetovic, Dragan, et al. "Multimodal Exploration of Mathematical Function Graphs with AudioFunctions. web." Proceedings of the 16th Web For All 2019 Personalization-Personalizing the Web. ACM, 2019.
• Ahmetovic, Dragan, et al. "AudioFunctions. web: Multimodal Exploration of Mathematical Function Graphs." Proceedings of the 16th Web For All 2019 Personalization-Personalizing the Web. ACM, 2019.
• Mascetti, Sergio, et al. "WordMelodies: Supporting Children with Visual Impairment in Learning Literacy." International ACM SIGACCESS. ACM, 2019.
• Cantù, Niccolò, et al. "MathMelodies 2: a Mobile Assistive Application for People with Visual Impairments Developed with React Native." Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 2018.
• Mascetti, Sergio, et al. "On the evaluation of novel sonification techniques for non-visual shape exploration." ACM Transactions on Accessible Computing (TACCESS) 9.4 (2017): 13.
• Ahmetovic, Dragan, et al. "Math melodies: Supporting visually impaired primary school students in learning math." Proceedings of the 14th Web for All Conference on The Future of Accessible Work. ACM, 2017.
• Bernareggi, Cristian, Andrea Gerino, and Sergio Mascetti. "Accessible Mathematics on Touchscreen Devices: New Opportunities for People with Visual Impairments." DEIMS. 2016
• Gerino, Andrea, et al. "Towards large scale evaluation of novel sonification techniques for non visual shape exploration." Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. ACM, 2015.
• Gerino, Andrea, et al. "Eyes-free exploration of shapes with invisible puzzle." Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. ACM, 2015.
• Gerino, Andrea, et al. "Mathmelodies: inclusive design of a didactic game to practice mathematics." International Conference on Computers for Handicapped Persons. Springer, Cham, 2014.