Vision-based patient wellness monitoring using facial cues

• Main Author: Sathyanarayana, Supriya
• Other Authors: Thambipillai Srikanthan
• Format: Theses and Dissertations
• Language: English
• Published: 2017
• Subjects: DRNTU::Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: http://hdl.handle.net/10356/72179
• Institution: Nanyang Technological University

Patient monitoring systems (PMS) are gaining importance considering the increasing demand for wellness monitoring at affordable costs. Vision-based PMS using CMOS cameras are being increasingly explored due to their low-cost and passive sensing capabilities. In this thesis, computationally efficient techniques for extracting facial features such as eyes, mouth and brow furrows as well as an integrated framework for a vision based PMS have been proposed. Noting that eyebrow is a stable facial feature, a compute-efficient technique for eyebrow detection has been proposed to help localize other facial features. An iterative thresholding method was proposed for the efficient extraction of the eyebrow edges. Evaluation on standard databases shows a high detection rate of 96% and robustness to variation in ethnicities, expressions, illumination and presence of partial occlusions. Computational savings of 35% was achieved compared to a state-of-the-art method. Next, eyebrows were relied upon as anchor points for the efficient detection of eyeball and eye-state. The proposed technique for eyeball detection relies on the local intensity variations within the eye while the eyeball position together with inter-feature distances were used to classify eye state as - open, closed or partially-open. The proposed eye state detection method achieves a recall and precision of 91% and 96% respectively, when evaluated on the proposed WellCam database and results in significant cost savings compared to existing techniques. The temporal analysis of eye state and eyeball position was then performed for the efficient extraction of the wellness indicators: eye blink, eye state over time and eyeball movement. The average mean accuracy of blink detection when evaluated on two standard databases was 98.8%. Eye-state-over-time and eyeball movement were evaluated using the proposed WellCam database. The eye-state-over-time achieves a recall and precision of 98.2% and 97% respectively. The technique proposed for the robust detection of the mouth state as open and closed relies on the efficient detection of the upper and lower lip positions and the analysis of the region between them. In addition, computations were restricted to vertical cross-sections of the mouth for drastically collapsing the complexity, leading to significant computational savings compared to existing techniques. The technique achieves an accuracy as high as 95% when evaluated on standard databases and is shown to be invariant to changes in facial expressions and ethnicities. Temporal analysis of mouth state has enabled the extraction of the wellness indicators such as ‘mouth kept open’, ‘yawning’ and ‘talking’. Evaluation of the yawning detection method on the YawDD database confirms a perfect detection rate (i.e. 100%). Noting that the existing techniques for detecting brow furrow features typically rely on a compute-intensive detection of brow lowering actions, a technique to directly extract and analyse brow furrows has been proposed in this thesis. A selective method for extracting brow furrow edges against surrounding noise was also incorporated to improve the average precision and recall to 92% and 91% respectively. In order to lower the compute complexity of the conventional face detection technique deployed in this work, a search space reduction technique has been proposed to limit the regions being investigated. The proposed method relies on the head and shoulder curves, which were extracted using a block based analysis for improved compute efficiency. An average reduction in search space of 70% is achieved. The computational complexity is reduced by 34%, if a worst case of 50% reduction in search space is considered, compared to the entire image. The wellness indicators such as eye state, eyeball movement, blink, mouth kept open, yawning, talking and brow furrows were finally integrated into a framework to determine the wellness state to be one of - asleep, awake, drowsy, inactive and discomfort. Past history of the wellness state over a period of time was also incorporated to generate a wellness profile, which is then used to assess the condition of the patient relative to an initial state. A confidence measure, based on the intensity and persistence of the wellness indicators, has been incorporated to accommodate situations in which not all relevant wellness indicators were present. The system is also capable of generating triggers in the event the patient's state drifts towards a threshold set by the doctor for notifications of improvement or worsening of patient’s condition. Finally, the major contributions in this thesis have paved way for propelling further research in this emerging area of interest, and for realizing an affordable vision-based patient wellness monitoring that can eventually lead to mass volume adoption.