Applications supporting people with Autism Spectrum Disorder
Since it is proved on various studies that the most autistic people tend to use visuals along with the technological devices (Behrmann, Thomas, & Humphreys, 2006; Grandin, 2009; Hayes, Hirano, Marcu, Monibi, Nguyen, & Yeganyan, 2010; Mechling & Savidge, 2011; Gonzalez, Martin, Minshew, & Behrmann, 2013; Bradshaw, 2013; Radwan & Cataltepe, 2016), many organizations and developers work on visual supports for autistic people that can be helpful during their daily lives. There are hundreds of applications are available on different platforms for autistic people, nevertheless, most of them are developed for visual schedules to assist completing daily tasks. Also, many of the apps have 2D interfaces used with smartphones, tablets or computers. The applications reviewed in this study are chosen carefully to give a general information about the interfaces used for different actions.
If we look for AR applications for autistic people, there are a couple of important inventions through the years.
Autism Glass Project
In 2013, Google published a video showing a Google Glass project for autistic children (Ayres, Mechling, and Sansosti, 2013). Google Glass can scan the environment and show the information present in the place using Augmented Reality through the glasses of the device to the user. Now in 2018, the project seems to be still in development process under the name of Autism Glass Project at Stanford Medicine. It is a behavioral therapy project, involving families of the autistic children in the use of the application. The settings are changed through and Android application by the family or the caretaker (Figure.2.2).
Figure 2.2: Autism Glass Project at Stanford Medicine (URL-3).
Empowered Brain
In 2017, a company Brain Power announced an app called Empowered Brain built for Google Glass and other Android glasses that can help autistic people to understand the emotional reactions of people around them (Palladino, 2017). It is a gamified application with colorful smiley icons and true or false games. In time they improved the app and added more features to assist autistic people on their daily routines, and it is available to purchase online (URL-4).
Sensory Customization Application
After Apple released ARKit in 2017, autism educator Craig Smith mentioned the possibilities of AR technology for autism on his article (URL-5). One of his ideas is to adapt the traditional visual support tools like visual schedules into AR via ARKit. Visual schedules are one of the essential visual support tools for autistic people. It provides a list of things to do like how to brush teeth or pour a glass of milk. AR can place virtual layers on real objects in the space and show to autistic people what to do with that object. According to Smith, AR can also help for visual focus by highlighting the crucial tools in a room especially in a school environment. Thus, autistic people can focus on the tools they needed for the task on their schedule. He also suggests an application for sensory customization (Figure.2.3). The autistic people with sensory problems can customize the places with light adjustments.
Figure 2.3: Sensory Customization Application Suggestion by Smith for ARKit (URL-5).
There are also some VR applications developed for autistic people mostly for therapy purposes. Although there have been around since 1996, the bulky VR gadgets back that time was not convenient for the studies in autism. After the release of Oculus in 2012, the number of studies started to rise again (Nunner, 2018).
Blue Room
A project called “Blue Room” aims to help autistic children to overcome their various phobias in VR (Maskey, Lowry, Rodgers, McConachie, & Parr, 2014). The application does not require using a VR goggle. It consists of a room with projected 360-degree image on the all four walls. A professional, needs to assist to autistic child in the room to control the application via an iPad (Figure.2.4).
Figure 2.4: Blue Room (Maskey, Lowry, Rodgers, et.al., 2014).
FloreoVR
Another VR project named FloreoVR focuses on assisting autistic children with joint attention, imitation, gestures, and social skills (URL-6). In a VR setting, different scenarios shown to children like sitting on a train, being in a concert to prepare them for confusing situations. The application requires an iPhone 7 or higher, a VR headset, and an iPad. The scenarios are managed by a specialist or the parents from the iPad. Also, a calming experience called Xylophone is included in the application (Figure.2.5), which can be managed by the user himself/herself. In the experience, the music plays automatically when the user moves his head.
Figure 2.5: A screenshot from Xylophone application.
Driving App
In 2015, a VR application was developed at Vanderbilt University to help autistic people to learn how to drive (Wade, 2015). The application can present different driving scenarios to the users while keeping them engaged in the learning process (Figure.2.6).
Figure 2.6: The driving perspective from Pilot Study (Wade, 2015).
Scenarios for Social Situations
Although it is not a developed VR app, a study was conducted in 2013 to help autistic people in different social situations (Kandalaft, Didehbani, Krawczyk, et al., 2013). They created real-world scenarios like a job interview in Second Life that is a virtual world (VW), a2nd let autistic people experience them, and observe their reactions in VW (Figure.2.7). The result shows that the VR environment helps autistic children to improve social cognitive skills in a safe environment.
Figure 2.7: VW screenshot of an interview scenario (Kandalaft, Didehbani, Krawczk, et.al., 2013).
Since MR technology and devices are still in the development process, there are just five applications listed on Microsoft Store for autistic people.
Talk for me
Talk for me application is one of the available apps for HoloLens. It can also be used across different devices like Xbox One, Windows computers, tablets, and phones (URL-7). The aim of talk for me is to assist people with ASD or any people with speech difficulties. Although it can be used via HoloLens, the interface is still 2D (Figure.2.8).
Figure 2.8: The screenshots of Talk for me taken via HoloLens Device Portal.
Visual Timers
The second application is called Visual Timers. It allows users to see the time passing with the help of visuals. In this way, the user can stay calm and patient, while s/ he is waiting for a task to be completed (URL-8). It is also available on PC and Windows mobile devices. The interface of the application is designed as 2D, and the boundaries of the interface is exceeding the screen size, so the user has to modify the screen to see all the content in the interface (Figure.2.9). There is no spatial interaction available. Moreover, there are advertisements on the interface, which can cause distraction.
Figure 2.9: The screenshots of Visual Timers taken via HoloLens Device Portal.
WeFeel
WeFeel is a therapy application which tracks the mental situations of the users like a therapist (URL-9). Except the HoloLens device, it is available on Windows PCs and mobile devices. WeFeel has also 2D interface like the other autism applications on HoloLens and provide a screen-like experience (Figure.2.10).
Figure 2.10: The screenshots of WeFeel taken via HoloLens Device Portal.
TalkTablet - AAC/Speech
Another application on HoloLens is TalkTablet - AAC/Speech. It is an Augmentative and Alternative Communication solution not for autistic people but people who have speech and communication problems (URL-10). TalkTablet is available on various platforms including PCs, android phones and iPhones, iPad and Kindle. It has a 2D interface with customization options (Figure.2.11).
Figure 2.11: The screenshots of TalkTablet – AAC/Speech (URL-10).
AuThink pro
The other application AuThink pro has not been working on HoloLens for a while due to some software problems (URL-11). But the PC version AuThink lite is available for the use. It is an educational application with various tests like matching colors and pairing items for autistic children to develop skills (URL-12). The interface of the application is 2D and same on each platform including HoloLens (Figure.2.12).
Figure 2.12: The screenshots taken from the PC version AuThink Lite.
Apart from AR and VR apps, there are many mobile apps for autistic people on the market.
AutisMate
AutisMate is an augmentative and alternative communication app available for iPad users, aims to help autistic people to communicate better (URL-13). It has 2D interface containing the pictures of the rooms in the house (Figure.2.13). The room pictures are customizable and can be changed with the pictures of the users home. The application needs to be set up by an adult before use. Various tags can be added to the pictures for giving information about the objects in that room, or the tasks can be done with that objects. The general interaction method is tapping on the objects and tags. Sound feedback is used mostly in the application.
Figure 2.13: The interface of Autismate (URL-13).
InnerVoice
InnerVoice app assists non-verbal autistic people with communication problems using visual icons and avatars (URL-14). It is available for iPad and iPhone users. The interface is formed by avatars in 2D (Figure.2.14). Avatars are customizable and replaced with the photo of the user. After the mouth of the avatar is marked, the avatar can talk and express emotions. There are so many texts and buttons in the interface. The user can interact with a tap, hold and drag actions.
Figure 2.14: The interface of InnerVoice (URL-14).
Kinder Tangram
Kinder Tangram is another app available on Apple app store which is an educational app helping not only autistic children but all to build houses with colors, stickers and various other elements (URL-15). The 2D interface is colorful and consisting of puzzles at different levels and art tools to modify the completed puzzle (Figure.2.15). There is little text used in the interface; there are mostly visuals and icons.
Figure 2.15: The interface of Kinder Tangram.
iPrompts
iPrompts is a visual schedule app, providing autistic people visual support, available on most of the app stores including Apple, Android, and Amazon app stores (URL-16). It has a simple 2D interface mostly graphics (Figure.2.16). The visual schedules are formed as a checklist with timer and choice prompts. Nevertheless, there is still no connection to the real places of the user.
Figure 2.16: The interface of iPrompts.
Proloquo2Go
Proloquo2Go is another application that can be used on iPhone, iPad, and also Apple Watch (URL-17). It has a 2D interface consists of matching symbols and texts (Figure.2.17). It aims to help autistic children to express themselves with the help of visuals. The button colors and texts are customizable in the application.
Figure 2.17: The interface of Proloquo2Go (URL-17).
iDress for weather
iDress for weather is developed for iPhone and iPad only in 2D (URL-18). The interface shows a scene with clothes hang in a garden (Figure.2.18). The type of clothes is changing related to the weather, so the user can learn what to wear based on the weather condition. The app is not only for autistic people but everyone. Nevertheless, it is one of the apps listed in autism applications list on several autism institutions. The interface is colorful and fun, but even though it is associated with the environment, it is the same 2D graphics for each location.
Figure 2.18: The interface of iDress or weather (URL-18).
Model Me Going Places 2
Model Me Going Places 2 can be downloaded from Apple App Store to iPhone and iPad (URL-19). It has a 2D interface showing the pictures of some places like doctor, mall, and restaurant as a button (Figure.2.19). The user can choose one of them and watch a video showing what to do in those places. Since there are not so many options, the app is easy-to-use; but lacks the interaction with the real places.
Figure 2.19: The interface of Model Me Going Places 2 (URL-19).
After the exploration of the applications for autistic people on different platforms, it is concluded that there are no applications with spatial interfaces are present for autistic people on HoloLens platform. Thus, the proposed spatial interface in this study will fill in this space.