Gaming Technology Helps MS Patients

Microsoft and global healthcare company Novartis have joined forces against multiple sclerosis (MS), a disabling disease that affects the immune and central nervous systems of millions of people worldwide.

A new tool called Assess MS, to be tested soon in clinical trials, uses Microsoft’s Kinect gaming system to monitor and evaluate a patient’s movements to determine if the disease is progressing or not.

Assessing whether a patient’s symptoms are stabilizing or getting worse is complicated, and Novartis and other healthcare companies have been searching for years for a consistent way to quantify whether treatments being developed for MS are working.

With MS, symptoms might progress with heartbreaking speed while in others they may show up slowly, erratically, and over a period of many years.

“One of the most difficult things about MS is the uncertainty of it,” said Cecily Morrison, a researcher working on the project in Microsoft’s Cambridge (UK) research lab.

In standard tests to quantify the progress of multiple sclerosis, doctors typically ask patients to touch their nose or sit with their arms outstretched. Doctors watch the patients and then use a rating scale to determine how strong the patients’ symptoms are.

The problem? Doctors are only human, and despite all their best efforts to standardize the MS test, in the end it is subjective. The researchers found that when a group of doctors are shown the same patient doing the same movement, some may interpret it as a “1” on the rating scale, while others will say it’s a “2.” Even when the same doctor is shown the same movement on two different days, that doctor may give that patient a different rating.

“The clinicians that we worked with really care about their patients. They really want what’s best for them, and even the best neurologist will admit that when they use these rating scales, it’s pretty coarse-grained,” said Abigail Sellen, a principal researcher in Microsoft’s Human Experience and Design group in Cambridge. “They know that there’s a lot of variability, even in their own judgments, over time.”

When Microsoft released the Kinect system for playing Xbox video games about five years ago, researchers at Novartis thought the computer vision and machine learning technology inside the Kinect might help doctors get a more consistent reading of how a patient performed on each of the tests. A uniform measurement of MS systems could also speed up the process of getting the right treatments to patients.

Microsoft researchers agreed to Novartis’ idea, knowing that machine learning would be ideal for a project like Assess MS because, as the computer vision system captures more recordings of patient movements, it can deliver more consistent results showing the disease’s progression.

“It was clear that this was a very ambitious project,” said Peter Kontschieder, a postdoctoral researcher at Microsoft who has built many of the machine learning algorithms used in the Assess MS project.

In a typical machine learning scenario, an experiment starts with a huge amount of data, such as many pictures of trees. The computer is given those pictures and, using a machine learning algorithm, it creates a model that recognizes what a tree looks like. Then, the next time it sees a picture of a tree — even if it wasn’t one of the pictures it was originally shown — it has learned to identify the image as a tree.

For the Assess MS project, the first challenge was gathering data.

In order to get an accurate view of how an MS patient was faring, researchers needed to figure out a way to make the Kinect depth camera, which is designed to recognize the sweeping gestures of playing a living-room video game, pick up on subtle movements such as the sway of a patient’s torso.

That required the researchers to come up with entirely new algorithms that would work behind the scenes with the Kinect’s depth camera and that would both recognize the patients’ body parts in a more nuanced way and provide a more precise representation of how the patient was doing on each of the tests.

Even once they had created those algorithms from scratch, the machine learning experts still knew they wouldn’t be able to collect as much data as they would ideally want to have to create a machine learning model. That’s because they were gathering these types of recordings for the first time, and they would be limited in how they could use the data by stringent patient privacy protections.

There was a third challenge as well. Unlike simple image recognition, in which you can say one picture is a tree and another is a car, this project required the researchers to grapple with nuanced, subjective data showing even slight disease progression.

All of these challenges required new strategies for labeling data consistently, so that they could create a strong model for quantifying the results accurately.

Another element of the project involved designing the system to fit into a real exam room, which might be small or crowded or have things like chairs in the way. It also had to be easy for doctors and nurses to operate.

After years of working on design and algorithms, the researchers say they have developed a proof of concept, using a limited number of patients, so they know the system works in principle. The next step is to test Assess MS in practice, so they can see how it works with a larger number of patients.

Ultimately, the researchers hope Novartis and other pharmaceutical companies can use Assess MS to speed up clinical trials for treatments of multiple sclerosis, and perhaps, eventually, for other, similar diseases as well.

“Novartis is leveraging digital technologies to transform patient care and drug development,” said Vas Narasimhan, a Harvard-trained physician who is now global head of development at Novartis Pharmaceuticals.

Optical and photonics technologies — and the people who work with them — have brought tangible social, environmental, health, and economic gains to humanity.

Whether by bringing inexpensive and efficient alternative energy to rural and developing areas, ensuring the safety of our food, or enabling instant communications, researchers, engineers, and industry professionals have advanced light-based research and technologies for the betterment of the human condition.

Do you have a story to contribute or a question about how optics and photonics technologies benefit humanity?

Write to us at spieprofessional@spie.org.

Read more articles and blog posts celebrating the many ways that optics and photonics are applied in creating a better world.

• Have a question or comment about this article? Write to us at spieprofessional@spie.org.
• To receive a print copy of SPIE Professional, the SPIE member magazine, become an SPIE member.