Using Kinect in Special Schools for Pupils with Severe Learning Difficulties

kinect fountain.jpg


Return to the Gesture Based Technology Homepage here.
Click on the news page to see updates specific to using the Kinect.

Impact Evidence:

The results of our 2012-2014 impact evidence are online- please go here to see videos, photographs and case studies of pupils using the Kinect.

Kinect Room Set Up:


Here is how Trinity Fields have set up their Kinect Room in their special needs school.

Kinect One / Kinect V2:


Kinect One for Xbox is out and we are currently assessing Fantasia and the Kinect Sports game with pupils. Kinect Version 2 for Windows is also just out and we have started using programs in school. (Jan 2015)

Why use the Kinect?


Gestural Based Technology- like touchscreens, iPads, interactive projectors and the Kinect are becoming a very promising tool in SEN education. They give some pupils the chance to effect their own environments themselves. Many of our pupils have problems with fine motor skills, movement and self awareness. Often they can be passive, withdrawn and unable to interact with their environment in meaningful ways. Gesture technologies enable some of our pupils to do things that they simply could not do before. They do not need to speak, use a keyboard, mouse or even a switch to interact and have an effect on their environment- they just have to move (in whatever way they are able to) and explore.
The Kinect is a motion sensing camera with a microphone built in, it's also cheap, used worldwide and plugs into most computers. Artists and computer designers have been using it as a tool for a couple of years using open source software. These developments and the Kinect interactive community have direct relevance to our settings. We're not talking numeracy and literacy skills here but encouraging creativity, movement, engagement and exploration. The Kinect plugs into a PC or laptop.

Xbox 360 console and Kinect.
This is an easy way to start. If you can get hold of a normal Xbox and a Kinect then you should download 'Kinect Party' from the Xbox Live store, it's about £8. It has 35 or so different interactive scenes that are mainly exploratory, open and 'playful'. Choose the right ones for your students and it will increase their willingness to move and explore- but a word of warning for pupils with SLD- the program is quite visually 'messy'- there's a lot going on and although it can track wheelchair users it's not 100% at this, but it is worth it as an easy thing to start off with- try it out for yourself. There are some very, very good interactive scenes on this. I would certainly recommend it- if only for getting your more able pupils to start sharing and communicating with each other as it is a superb motivator!Also Fruit Ninja for the Xbox is a great activity for the more physically able.
kinect party.png
Kinect Party
fruit-ninja-kinect_1000.0.jpg
Fruit Ninja



What to do with Kinect and computers.


Setting Up.
1- Buy a Xbox Kinect sensor!- Around £90 from Amazon, you'll need just the Xbox Kinect Sensor, not the Xbox games console. This Kinect will have a power plug and a USB that you can plug into your PC, no software comes with it as that is where 'Drivers' come in. Note- NOT the Xbox One/ Kinect One version.
NOTE: If you already have a Kinect from an Xbox package you can use it but you will need to purchase a Power supply unit and a USB adapter for it first (around 10 pounds).
DO NOT get the Kinect for Windows (Costs around £200), as yet I've not found anything useful to run this with for our pupils.
What computer? Kinect programs work better on new PC's like Windows 7/Vista/MACS. So far most of these programs are for PC's, two for the MAC and two are have both versions. Laptops are more useful as they are portable and you're probably going to be moving them around a lot. The Mac programs work without driver installation and so are easier to 'plug and play'. PC programs will require driver installation but it's not too tricky.
Xbox-360-Kinect-Standalone.png
Kinect plug.jpg
Kinect PC power plug

Explanation- Drivers

This Kinect was meant for the Xbox, not the PC, so it needs a program layer to enable it to 'talk' to the program you use, this is the driver program. Typically each program uses a different driver. Once installed on a PC it will appear in your control panel under - system- hardware- device manager. On the Mac it is easier as there are no drivers to install for the four programs here.

Projection/Screens

As the Kinect works on visuals you're going to need a projector, smartboard or some other large screen, a big TV screen is just about enough. Get the right leads for linking the laptop or PC to the projection device first! At Trinity Fields we have back projected onto a fabric screen in the hall, used it with Interactive Smartboards and also used a normal projector projecting onto a blank wall (see photos below).
Hopefully you will have sensory rooms and most of the equipment you need somewhere in school already, it's then a case of putting them all together and downloading some programs!

Extension Leads
The Kinect will stretch to about 3 metres from your computer with it's own leads. If you want to go further you will need a USB extension cable. The advice is to use 5m long powered USB cables, not just regular ones. Also 2 x 5m extensions work together (10m total) but 3 (15m total) doesn't. Don't buy 10m cables as they also won't work. Buy a decent make as well- not too cheap.


Programs


When you're ready to go with the hardware you need some software! These are the ones we have tried so far, I've had all of them working in school so if you come across problems there is a solution. Somantics, Reactickles and Pictogram Room were designed for special needs pupils, the rest are art and interactive programs written by individuals and kindly released as open source programs on the internet.

Somantics

somantics.pngScreen Shot 2013-02-09 at 15.18.05.png

The perfect place to start! Download the FREE version for PC or MAC from www.somantics.org. The PC version needs the drivers installed- there should be a readme file which explains this or there's also a .pdf instruction file with pictures on site too! It uses Xbox NUI drivers which come with the download in the zip file. Only seems to work on Windows 7, XP is troublesome.
This program is awesome and has ten different applications that respond to body movement, you can paint with your body, make sparkles appear around you and explore different graphic reactions to movement. They were designed for ASD pupils in mind, but work for any SLD pupils who have good vision, they encourage pupils to move, create and explore their physical movements. Put the pupils in front of it and let them explore- there is no 'task' or right or wrong answer to this, it is about encouraging movement and exploration of the self and the interactions it creates. It is very good, and the whole project is way ahead of everyone else in the SEN world in doing this.
There is also a free iPad app that works on the same principles which can be hooked up to a larger screen if you don't have a Kinect.
It was designed by Dr Wendy Keay-Bright at Cardiff Metropolitan University for use with students with Autistic Spectrum Disorder. The project is part of The Centre for Applied Research in Inclusive Art and Design or CARIAD (which means 'love' in Welsh). This is the link for a recent overview of the project:
http://www.visualperformanceanalysis.com/industry-blogs/centre-applied-research-inclusive-arts-design/

See the evidence page for a Postgraduate study using Somantics. It also uses whole body tracking, not just hands and feet- so anyone who can look at themselves on a wall and move (however slightly) can interact with the system so it is suitable for more able pupils to PMLD pupils.

Somability


This is Somantics 2! It is out for PC and Kinect and consists of another set of programs that use the Kinect that encourage movement and interaction for pupils with PMLD and SLD. Follow the download link and instructions on the Somability page.

somability.png

Reactickles


Sort of a sister project to Somantics. Go to www.reactickles.org for the links for the FREE PC or MAC version downloads. Works more with sound than with the Kinect- turning vocal sounds into graphics.

Visikord
Go to www.visikord.com and download the free trial version. Install it first, then plug in the kinect and it should run. Works best on Windows 7/Vista newer computers. The latest version (September 2014) has ability to trigger sounds when you touch side bars too, trigger graphics when you touch dropping icons and also a great 'quiet' background visuals mode.
Hand Powers!
Hand Powers!



The Visikord system is aimed at nightclubs and DJ's. It uses full body tracking together with great visual effects such as 'hand-powers' to turn body movement into superb visual graphics. Although not designed for our setting it does work well, it shows the users body in various guises on screen, you extend out your arms and get these amazing visuals from your hands too. It really encourages movement in pupils. The maker Davor Magdic in Santa Barbara has been very interested in the responses we've had from our pupils.
The most noticeable reactions to the system have been from our low functioning Autistic pupils. They react well to the way that Visikord is like a ’virtual mirror’ and the system encourages them to move because their movement then has an immediate visual effect, an instant feedback to them. A young autistic girl really slowed her movements down and watched intently as the graphic outline of her body altered in response to her movement. The young man who was involved in the Somantics trial has also responded well to it, altering his repetitive movements to interact with the system and to activate the ‘effect bars’ at the side of one of the settings. Students with a higher level of ASD are also captivated to explore the cause and effects of the software. There is also a full version, still cheap compared to educational software prices. Here are four photos of a hard to engage P5 level student with ASD using the program in our ICT room. Notice how blurry the photos are- as it was hard to catch a still pose!

v2.JPG
Pupil using handpowers- shot taken from laptop screen
v1.JPG
ASD pupil and visikord

v3.JPGv4.JPG
It also encourages movement in all pupils, and as importantantly it is great fun! This is one of our pupils with Down's Syndrome using it back-projected in the hall as part of a PE lesson. It mirrors the body with a graphical outline and so is very intriguing to many pupils, it can also be used by pairs of pupils.
external image downsyndromestudent-300x197.jpg


Wiggle- Mac


The product of a Gesture Based Technology PLC meeting workshop with CARIAD interactive's Pete Hellicar. Some of our members designed this interactive Kinect program with rhythm in mind and it was coded pretty much on the same day by Neil Mendoza. It is available here as a free download for Mac computers only.


The premise is that when you move the lines wiggle and set off the music clip- when you stop the line goes still and the music stops as well. Works best with one user but will pick up two or more. Also uses blob tracking so will pick up wheelchairs with no problem. The photos don't show the middle line wiggling but it does! Nice program for stop/start interaction with a great music element too. Instructions on the bottom- space bar to alter distance/ kinect angle, F to fullscreen.
Wiggle screenshot.pngWiggle controls.png

Z-Vector
This program from Delicode is currently a free download in BETA from the Z-Vector website (Sept 2013). It is available for PC and MAC. It is meant for live visual performances and you can adjust what the body creates when it moves in front of the screen. Thankfully there are also a few examples that you can find here they are .png pictures which you just drag and drop into the program to create different effects. Here are some of them:
zvector 1.jpgzvector 2.jpgzvector 3.jpg

So all the above are from the same program- and they're all the pre-made examples, so it's really easy to use. Here's one of our pupils using it from the floor- its tracking ability is very good- with a little movement you get a body track and a good visual response.
web kinect july.JPG


Snowbells - PC
From Binaura. Go to this link.
Download the zip file of Snowbells, install it on the computers main drive, unzip it and if you have the Visikord drivers installed as well it will work with those- so you can switch between the two, otherwise there is a readme file with instructions to follow.
Snowflakes fall from the sky, when they come into contact with a body they make chime and tinkling sounds and gently bounce off. The body appears as a white shape and it can track mulitiple users, make sure you have a speaker plugged in. This works for visually impaired pupils too and will pick up pupils in wheelchairs as well as more mobile pupils, as long as they move initially it will continue to track their body outline. Most pupils respond to this program well although the three twinkle sounds are quite limited- there is a definate need for more music based kinect responsive systems for our visually impaired pupils.
0-1.jpg


Noise Ink - Mac

From the lovely Trent Brooks who's helped me a lot! Watch this short clip: www.vimeo.com/21691884 This is an AWESOME! graphic visual movement interactive program. It only works on Mac computers however. It can be downloaded for free from this link.
https://github.com/trentbrooks/Noise-Ink
This is plug and play, set the depth with a few key presses and you are away, move around to create lovely line visuals that do their own little thing while they are waiting for you when you are still! Be aware your Mac computer requires an adaptor lead if you are plugging it into a standard smartboard or projector.
https://github.com/trentbrooks/Noise-Ink/raw/master/screenshot1.jpg
https://github.com/trentbrooks/Noise-Ink/raw/master/screenshot1.jpg
noise ink.JPG

Fluid Wall

Austin Hines, a computer programmer from Texas made this for PC's:
https://code.google.com/p/fluidwall/
Watch the video while you're there! This is a virtual 'wall' that when you break through it gives misty and spiky graphic visualisations and the image of the users body as an outline. It also tracks multiple users.
Download and unzip on your computer. There is a readme file, print it off and follow the instructions very carefully. There's two different drivers. It should install okay but there are instructions to help if it doesn't.
Run the program, make a note of the key controls too in the first window- you can alter the graphic type, depth of the 'wall', colour of the user etc. This is a really nice interactive art program- if you can get it to work. It also works best on fancy 64 bit computers- but on my 32 bit it is okay. Works better in darker rooms and the 'wall' aspect means pupils can explore the depth aspect- sticking their legs and arms through it for instance.

fluid wall.JPGfluidwall.JPG
ASD pupil using Fluidwall in ICT rooom on the spiky line setting (number 2). Note the two kinects- one runs Somantics from a PC, the other runs a guest program from a laptop- this enables easy switching between the two in a short session. There are also two seperate projectors on a high shelf at the back.

Pictogram Room
From Spain! Currently in Spanish, but it's easy enough to use. Made for pupils with autism. Not so creative but it has lots of body awareness, problem solving and symbol interaction functions on it. For Windows 7 and upwards PC's only. A free download too from:
http://fundacionorange.es/fundacionorange/en/project/pictogram.html
It's a plug and play program too, so it automically installs the drivers (although nothing else on this page will work alongside it!) It is more for specific tasks than the other programs so not so open and interactive, I'd say suits P6-NC1 pupils who need some physical activities and body awareness tasks. There's LOTS to it though- 40 activities based on the body and posture and it's worth trying. There is an English version out soon hopefully but there are picture clues too which are self explanatory and students don't need to respond to the language themselves. There's musical interaction too.
It is more task based and the tasks are graded from quite easy to follow directions to difficult body poses. I tried it and was suprised to find one of my ASD pupils couldn't shake his head! Nor could he shake one leg without perching on a desk for stability as seen in a picture below! So it really does pick up movement and body awareness issues up that you perhaps hadn't noticed before.

P7160094.JPGP7160099.JPG


Kinect2Scratch: Music, Movement and Fun

Kinect2Scratch is a really amazing free program developed by Stephen Howell, a computer science lecturer from Ireland.

Scratch is a really popular programming language that makes it easy to create and share your own games, music, art and more. It is free and was developed by the Lifelong Kindergarten Group at the MIT Media Lab.

Stephen has made it possible to have Scratch and Kinect work together. You can download it from here http://scratch.saorog.com/ . At the moment we have only been using the sample games that Stephen has made himself. These are also available to download from Stephen's website. We have used the Music Generator and The Skeleton and I have to say that the children love it.

It does however need a fast laptop. When I first downloaded it onto a class laptop there was a time lag between my movement and what was happening on screen. I then downloaded it onto a faster laptop and it worked perfectly. Originally I thought it wouldn't work with Somantics on the same laptop but we have figured out a way to switch between the two programmes with a minimum of fuss.

There are a few pics on the school website of a pupil using it: http://www.littleangelsspecialschool.com/ict

Conclusion

Like most things these Kinect sessions take time to prepare for and set up although hopefully when Kinect V2 apps start going up on the Windows app store in late 2014/ early 2015 things will get easier.With a little bit of space and a small outlay in money an interactive area can be set up to try to engage our pupils and help their interactions, creativity and movement. Different sessions and different programs produce different responses and these responses also evolve over time. Some pupils love it and take to it straight away, others need some encouragement and patience, others aren't bothered with it at all, but in my experience that is true of everything! What is important is that these Kinect programs have no real parallel in our schools, they use movement to create visuals and sound without needing the pupils to use equipment or to learn any specific skills first. They are natural and intuitive.It is hard for people who do not understand the complex needs of our pupils to understand the significance of enabling and motivating some of our pupils to interact meaningfully with anything in their environment. The more enabling technologies we use the more chance we have of finding something that will increase motivation, creativity, interaction and movement when other traditional methods have failed.The Kinect is not replacing these other methods but adding a whole new level of opportunity to the tools at our disposal. I am certain that given the resourcefulness and inventiveness of the special needs teaching community worldwide and the support of the business and technical community that these opportunities will multiply and flourish.
Further Links:

Ceri Williams in Torfaen also has tried the kinect, his website has a guide to using Open NI and Processing: http://primaryparper.wordpress.com/

Tags: Special Needs Education, SEN, SLD, PMLD, Kinect, Autism, ASD, special needs school.