Traditional assessment methods fall short when trying to measure experiential learning. A multiple-choice test is not a helpful barometer of how well a participant knows how to use Photoshop, for instance. The field of Learning Analytics is making great headway in assessing experiential learning, but the current methods are not practical when teaching a brief, two-hour workshop.
Tracking mouse movements can tell us a great deal about how a participant is interacting with a piece of software. By aggregating all participant’s mouse movements, and tying that visual data to an audio recording of the instruction, we can greatly improve our teaching. We can tell when participants are distracted or lost, when they are falling behind. MACAW stands for Mouse Analytics for Computer Assisted Workshops, and it is a method for data collection, sorting, visualization, and analysis.
The presentation (NMC Summer Conference 2013) in the video below provides more background information on the idea, as well as some ideas for future development.
The MACAW Method
The method is a work-in-progress, and we will continually be making improvements to it and documenting them here. The goal is to simplify the method to make it faster for the instructor to set up and gather useful data.
Current Method, in brief:
- Install IOGraph* on all the computers being used. Note: IOGraph uses Java, and in my experience using Macs, it does not seem happy with Java Runtime Enviroments (JREs) other than version 6. To check your Java Version, open terminal and type java -version. Hopefully, it is "1.6.0_x". If not, feel free to contact me for help.
- Set up a networked drive and mount that drive to all the computers being used. Create a directory structure with a folder for each computer.
- Write a script to make IOGraph work in the background and take a fresh capture of mouse movements every X seconds. Have the script tell IOGraph to save the PNGs to the appropriate folder on the networked drive. (Contact me for help with scripting. I have examples, but don't want to share them here, since they are specific to my environment.)
- Start an audio recording that will pick up the instructor’s voice.
- Set the script in motion on all the computers and teach the workshop.
- When finished teaching, pull up the directory structure, and start data analysis.
- Look for patterns in the user’s mouse movements. All the PNGs will be time-stamped so it will not be too hard to match them up to the audio recording.
- To more easily see all the data at once, put everything into one giant image sequence in Photoshop, open the Animation timeline and hit play (this will require significant computing power, but is the easiest way to quickly see what is happening).
- After you identify patterns worth investigating, try color-coding the data, replace the white background with a screenshot of the application you were teaching, and finally match up the timestamp on the visual data to the timecode on the audio recording.
If you are interested in MACAW, and want more information, please contact Nico Carver at nico [at] udel [dot] edu.
*Disclaimer: IOGraph is a free piece of software, developed and owned by Anatoly Zenkov & Andrey Shipilov, all rights reserved. We are neither affiliated with the developers of IOGraph nor responsible for their content. The MACAW method is free to use, modify, and share (with attribution) under a CC-BY 3.0 US License.