Electronic Thesis and Dissertation Repository

Thesis Format

Integrated Article

Degree

Master of Engineering Science

Program

Electrical and Computer Engineering

Collaborative Specialization

Artificial Intelligence

Supervisor

Roy Eagleson

Abstract

The recent pandemic has impeded patients with hand injuries from connecting in person with their therapists. To address this challenge and improve hand telerehabilitation, we propose two computer vision-based technologies, photogrammetry and augmented reality as alternative and affordable solutions for visualization and remote monitoring of hand trauma without costly equipment. In this thesis, we extend the application of 3D rendering and virtual reality-based user interface to hand therapy. We compare the performance of four popular photogrammetry software in reconstructing a 3D model of a synthetic human hand from videos captured through a smartphone. The visual quality, reconstruction time and geometric accuracy of output model meshes are compared. Reality Capture produces the best result, with output mesh having the least error of 1mm and a total reconstruction time of 15 minutes. We developed an augmented reality app using MediaPipe algorithms that extract hand key points, finger joint coordinates and angles in real-time from hand images or live stream media. We conducted a study to investigate its input variability and validity as a reliable tool for remote assessment of finger range of motion. The intraclass correlation coefficient between DIGITS and in-person measurement obtained is 0.767- 0.81 for finger extension and 0.958–0.857 for finger flexion. Finally, we develop and surveyed the usability of a mobile application that collects patient data medical history, self-reported pain levels and hand 3D models and transfer them to therapists. These technologies can improve hand telerehabilitation, aid clinicians in monitoring hand conditions remotely and make decisions on appropriate therapy, medication, and hand orthoses.

Summary for Lay Audience

Patients with postoperative hand surgeries and upper extremities issues need regular rehabilitation under a specialist’s supervision.COVID-19 has impacted the healthcare delivery system making it difficult for patients to visit a therapist in person leading to the emergence of tele-rehabilitation services. Telerehabilitation is a branch of telemedicine referring to the delivery of rehabilitation services over telecommunication networks. However, remote hand therapy faces challenges in reducing the reliance on in-person evaluation and assessment. The technology and treatment workflow developed in this project provide a way to address those challenges. For proper delivery of hand therapy, therapists require accurate information about the patient’s hand condition. A complete 360∘ scanning of the hand is essential for therapists to understand the hand deformities and the severity of trauma. Most professional 3D Scanners and sensor-based rehabilitation devices are costly, non-portable and require technical knowledge to operate. Our project investigates whether photogrammetry, a non-invasive and cost-effective technique for creating 3D models from 2D images , can accurately reconstruct a 3D model of a human hand for telerehabilitation purposes. We determine if these models are of sufficient quality to support remote hand therapy and improve patient outcomes. To facilitate communication between patients and therapists, we designed an interactive app, HAND SCANS.This app enables patients to self-report pain and sensation level of their injured hand, provide health information ,medical history and transfer 3D models of their injured hands digitally to a therapist. We also experiment with a novel machine learning pipeline, MediaPipe to develop a computer vision-based application called DIGITS for tracking hand landmarks in 3D space from 2D images or videos, find joint angles and evaluate finger range of motion. The idea, design and development of mobile/web applications to collect and transfer medical data, patient hand images, 3D hand models and key information during real-time hand movement tracking to the therapists form valuable aspects of hand telerehabilitation method. These cost-effective digital solutions will enable therapists to make clinical decisions on suitable therapy, medication, and orthoses for patients located remotely, improve efficiency and accessibility while bringing healthcare closer to home.

Download

Share

COinS