Building a cyber-physical fall detection system for seniors

The population of senior citizens is on the rise, and the number staying at home longer is increasing. However, a leading source of injury for seniors is falling (over a third of seniors older than 65 fall each year¹), with a large number of serious injuries or even death. Seniors living at home, often by themselves, are at high risk for delays in receiving necessary medical help. We have developed a Senior Collapse Detection (SCD) system as a domestic cyber-physical system to monitor for and detect falls.² SCD communicates with the senior, and in cases of no response or an affirmative response, the system has an emergency response protocol. SCD could help seniors get more immediate attention when they do have falls, and as such could possibly prevent death and reduce the severity of long-term injury.

The system uses the Microsoft Kinect sensor to gather 3D vision data on which skeleton tracking is performed and used to monitor the senior in their home. We have developed a series of fall detection algorithms using concepts from physical therapy to detect potential fall events. Figure 1 shows the SCD system at work, with skeleton and 2D vision data streams for demonstration purposes. Figure 2 shows a fall detection event, and the triggered emergency response protocol. The Kinect sensor supports audio input through a microphone array, and has a supporting speech recognition application programming interface. Combined with synthesized speech, the SCD will follow emergency protocol and ‘speak’ with the senior, asking them questions like, “Have you fallen?” and “Do you need assistance?” Their answers inform its emergency response actions. Synthesized speech is also used to inform the senior of actions, for instance, by stating, “Emergency contact has been made.”

Figure 1. The Senior Collapse Detection (SCD) system detects a skeleton in an image.

Figure 2. SCD detecting a fall. Emergency response includes email contact and audio output.

Detecting falls is not an easy task. Simply enumerating all of the different ways a person could fall was the subject of a survey from Mubashir,³ and we have further identified a number of additional fall types not in the survey. Each fall type has certain characteristics that our computer vision algorithms are coded to detect. These detection algorithms have to contend with the nearly constant difficulty of missing data, for instance, parts of a person's body hidden by their orientation to the sensor or by standing behind a piece of furniture. In addition, processing 3D depth data into skeleton tracking is not easy, resulting in joint locations that are by no means perfect, and sometimes completely erroneous. Our application tracks users as they move freely about their home. We have used the vast research on fall indication factors^4–8 in SCD's fall detection components of head motion fall detection, floor prone detection, and horizontal detection.

Significant acceleration of the head is a strong indicator of falling,^5–8 and this is monitored in the skeleton data by the head motion fall detection component. The floor prone detection component uses the common-sense notion that most falls end with the senior prone on the floor. A senior is classed as on the floor if a significant number of skeleton joints are within a small distance of the calculated 3D ground plane. Finally, the horizontal detector component detects relatively horizontally positioned skeletons and checks the distance of the individual's head to the floor. As with other ‘medically inclined’ systems, we favor false positives over missed fall events. For example, in our system, the movements an individual makes when moving to lie down on the floor can be detected as a fall. By calculating the distance from various joints to the floor, our detection algorithms are able to detect real fall events while ignoring fall-like normal daily activities. A great advantage of our system is that it is autonomous and no devices are worn by the senior.

Raw skeleton data can be unstable. Consequently, SCD has the option to use reliability metrics for each detected joint position and angle based on comparison to physical models. These models are derived from physical therapy data, including range-of-motion parameters.⁹ Currently, SCD uses a single model, but we are examining the possibility of matching the individuals detected in the video frame to the most appropriate physical therapy data based on age, height, and other physical properties.

The SCD algorithms have performed well with our recorded data of simulated falls. The next step will be to test SCD in a real senior home environment.