Engineers, researchers and clinicians have joined forces to examine peripheral nervous system-machine interfaces and brainstorm ideas to move the technology forward. Led by Claudio Castellini, PhD, researcher at the German Aerospace Center’s Institute of Robotics and Mechatronics in Oberpfaffenhofen, Germany, and Panagiotis K. Artemiadis, PhD, professor at Arizona State University in Temple, Ariz., the group first met in June 2013 and met again in August 2015. The findings from the group’s first meeting, hosted by the International Conference on Rehabilitation Robotics in Seattle, were published in August 2014 in Frontiers in Neurorobotics.
“In 2010, having explored the field [of peripheral nervous system-machine interfaces, or PMIs] for a few years, I felt that the community had produced a lot of academic research, but little of this had been brought forward to the clinics,” Castellini told O&P News. Castellini said many papers on PMIs had been published since 2000, but no commercial products had become available.
“Modern prostheses are great, but they are not as great as they could be,” Michael Wininger, PhD, said. Wininger, an assistant professor in the Prosthetics & Orthotics Program at the University of Hartford, helped organize workshop proceedings. “We feel that despite our savvy as clinicians and engineers, the patients are still getting frustrated. The devices are not doing what they want them to do.”
Castellini wanted the workshop to bridge that gap and help make PMIs work for patients.
“The goal was to try to answer four basic questions, to shed light on four broad subtopics of the field, which we deemed were still missing from the picture,” he said.
Four questions
The four questions of the workshop were:
- What is wrong with the current approach? Why do clinicians not use it?
- How can we better use surface electromyography (sEMG)?
- What alternative, radically new solutions are available, if any?
- What are the benefits of sharing control between the human subject and the prosthesis?
The workshop placed attendees into four groups to review current research and provide answers to these questions.
“We tried to pick up two or three experts of the field [who] had concentrated their attention on each of the subtopics: O&P practitioners and physicians for question one; signal processing and machine learning experts for question two; those who had experimented with unheard of [technologies] for question three; and people in control theory and human interface for the fourth question,” Castellini said.
Wininger added, “The idea was to get an inclusive, multidisciplinary, highly dynamic group of people with different expertise and different research interests thinking about this one common central problem, which is extracting volitional signals out of the peripheral nervous system.”
Problems with the current approach
The first question addressed problems with the current approach to PMIs.
“This is the most important question in upper limb prosthetics at this time,” Wininger said. “Device abandonment [is] reportedly as high as 50% in some scientific papers. So many times, people are abandoning their device because the device is not doing what they want it to do. They feel that they are just better off using their residual limb to stabilize, rather than try to get control of the prosthetic device. We can do much better.”
Artemiadis has worked in the field of human control of prosthesis using PMIs for almost 10 years.
“The main problem [with PMIs] is robustness and repeatability of the signals one can measure from the peripheral nervous system from controlling the prosthesis,” Artemiadis said. “The signals change with respect to time due to muscle fatigue, change of the characteristics of the recording site, the electrode, etc.”
As presented by the researchers in the study, several factors can influence the acceptance and the benefits of a myoelectric prosthesis for a specific patient, including weight, noise, cosmetic appearance, battery duration, purchase price and expense of prosthetic care and other associated services.
Feedback to the user is the most commonly recognized problem among clinicians working with sEMG, according to Samuel L. Phillips, PhD, CP, FAAOP, health scientist at HSR&D/RR&D Center of Innovation on Disability and Rehabilitation Research (CINDRR) in Tampa, Fla. “The problem of getting the intention from the user to the device is clearly among the most challenging problems of upper limb [prostheses],” Phillips said. Nonetheless, he said, clinicians still work regularly with sEMG because “many of the new solutions available are still early, and all the bugs and kinks to make them work on a daily basis have not been worked out completely. Many things that work [well] in the lab and have great promise are not yet reliable enough for home daily use.”
Possibly the most prominent technology making use of sEMG is targeted muscle reinnervation (TMR), which transfers the nerves of the arm to alternative muscle sites. Wininger said TMR is considered a “flagship technology,” and the topic has been covered extensively in top medical journals.
“This is absolutely one of the most promising advances I have seen in years,” Phillips said, adding he would like to see more discussion of TMR in future group meetings.
The researchers recommended a focus on the integration of additional and/or novel sensors and sEMG arrays within the prosthetic socket for future research to address the most common issues.
How sEMG can be better used
The group wanted to determine the possible ways pattern recognition and other forms of machine intelligence could improve the robustness, adaptability and situational awareness of sEMG systems and other PMIs, so the second workshop group reviewed studies on the utilization of sEMG.
There are still a number of barriers that prevent improved prosthetic hand control, according to Patrick M. Pilarski, PhD, assistant professor in the Division of Physical Medicine and Rehabilitation, Department of Medicine at the University of Alberta in Edmonton, Alberta, Canada and principal investigator for the Alberta Innovates Centre for Machine Learning and the Reinforcement Learning and Artificial Intelligence Laboratory. Pilarksi’s research focuses on improving sEMG control through computing techniques, including machine intelligence and real-time machine learning.
“A key remaining challenge is scaling up the control and feedback interactions between a user and their device such that they can utilize the full capacity of new prosthetic devices,” Pilarski said. “As amputations become more proximal, the ability to interpret intent from the human body decreases while the amount of function that needs to be restored increases. [One] of the key remaining challenges to better prosthetic hand and arm control is still the interpretation and intuitive communication of both control and feedback information flowing between someone with an amputation and their prosthetic device.”
One significant barrier the group identified was a lack of public data collections. A limited set of data for hand prostheses prevents researchers from solving common problems. Recently, however, the first version of the NinaPro database was introduced.
“Shared data allows researchers to compare their techniques on a common ground, and further allows those without direct access to myolectric control systems or participants to contribute to the development of novel control methodologies,” Pilarski said. “Continued public access to data from multiple groups will further ensure that developed solutions represent the actual populations that will use them — more data reduces the challenge of ‘over fitting’ solutions to specific tasks, databases or solutions.”
Real-time learning is an area that holds promise for the future of PMIs, Pilarski said.
“Each person is complex and unique, and techniques from real-time machine learning provide a way for devices to personalize and optimize their operation for individual users,” he said. “As real-time learning is still emerging as a key area of study, it is seeing initial translations to practical use by research participants. In terms of its use within myoelectric prostheses, [it] has not yet seen the same degree of commercial and clinical deployment as has pattern recognition.”
Pattern recognition, meanwhile, has moved to the commercial stage and is available for patients. This technology is able to learn a user’s intent through control patterns, thus enabling more intuitive prosthetic movement.
“Increasing the ability of devices to actively and consistently support prosthesis users is a huge and important area that we should focus concerted effort on developing in the near future,” Pilarski said. “Personalized, adaptive prosthetic control solutions will remove a number of remaining barriers to prosthetic acceptance by users.”
The researchers also suggest in the Frontiers paper that pattern recognition could be combined with TMR to offer control that is both intuitive and robust.
Alternative and ‘radical’ solutions
Castellini and Wininger both were part of the group that examined “alternative, radically new solutions.”
“In order to (at least partially) overcome the drawbacks of sEMG, we explored the results so far obtained using pressure sensors [through topographic force mapping] and ultrasound imaging,” Castellini said. “Although built upon totally different electrophysiological principles with reference to sEMG, both of these techniques, if correctly employed, ‘echo’ the muscular activity, and therefore can be used to control the prosthesis.”
Wininger explained the use of ultrasound thusly: “We are controlling a prosthesis by looking at the changes of the skeletal muscle of the arm [while] the user is making voluntary controlled action.
“Both have their advantages and disadvantages,” Wininger added. “The primary advantage of ultrasound is that [it] is high resolution and can be precise. The disadvantage is that it is clunky and you cannot put it on your arm.”
Meanwhile, Wininger said topographical force mapping avoids a common problem associated with electromyography: perspiration. “The musculature in the arm does not change just because you are sweating, and sweat is a major problem for prosthesis users,” he said. “The downside is, [topographical force mapping] is fairly low resolution.”
While both techniques are still in the formative stages – with about 10 years of research behind each, in comparison to the 50 years of research on sEMG — Castellini said there is potential for the use of either, or a combination of the two, as a “companion” for sEMG.
“We did some comparison and discovered that force mapping and sEMG can be proficiently used together, while ultrasound imaging needs to be somehow refined,” he said.
Castellini said an ideal technology would combine the best attributes of both force mapping and ultrasound.
“There are two main points to be explored to this aim: sensor fusion — that is, how to give the correct ‘weight’ to each kind of signal, produce a coherent flow of information and use it; and how to embed all sensors in a shaft or silicon liner to be comfortably worn by the patient. Once we figure out solutions to these two points, we can start exploring their joint usage in practice,” Castellini said.
Shared control between device, user
Artemiadis was part of the group that examined the benefits of sharing control between the prosthesis and its user. Artemiadis and colleagues advocated for the advancement of human embedded control, as opposed to a traditional system of myocontrol, described in the paper as a “classical master-slave myocontrol” interface.
“With the human embedded control, the human is responsible for learning to control the device, while in the traditional systems the decoder would try to learn to decode the human signals to control inputs,” Artemiadis said. “By adding the human in the loop, the robustness of the system increases dramatically.”
In this type of system, the artificial controller executes low level details to keep from overloading the user, then shifts control to the user for more complex movements. Human-embedded control will ideally allow users to grasp and reach with a prosthesis in a straightforward manner by triggering the system with their decisions.
“The proposed idea allows the user to learn the device and compensate in real-time for any errors in the operation, therefore increasing the reliability and robustness of the device,” Artemiadis said.
Artemiadis said adaptive, robust electrodes are needed to take human embedded control to the next level and he would like to see future research focus on improving electrodes and interfacing technology.
Impact and future plans
The group’s most recent meeting was held in Singapore, and some participants attended via Skype or teleconference. Castellini, Pilarski and Wininger said the findings of these workshops have directly contributed to their own research.
Castellini wants researchers to apply the workgroup’s findings to their methods and focus more on patients.
“Take your prototype out of the lab and bring it to the clinics where it can be tested in a quasi-real environment and on end users,” he said. “[There] is no point publishing papers in which you show that your approach goes 5% better when tested in your lab and on students/colleagues.”
He added, “We [at the DLR-German Aerospace Center] are just about to start testing our own system on an amputee [who] we recruited via a local orthopedic company.”
Pilarski said this has helped his research to better clarify the needs of prosthetic users.
“Because of the diversity of the reports and examples they shared with the workshop participants during the talks and following discussions, our group at the University of Alberta has been able to better focus our efforts on the most critical barriers to patient acceptance without getting distracted by less critical technical challenges,” he said.
Wininger said he has used the PMI workgroup as a springboard to launch a new research interest in device embodiment; in 2013, he began work on the Hartford Hand, an upper limb prosthesis design to enhance patient satisfaction. While the Hand is still in the prototype stage, students in Wininger’s lab at the University of Hartford are working on the development and pilot testing.
“Some of the things we talked about in these workshops directly led to our next innovations in the lab,” he said.
A new position document is under development, following the 2015 workshop. However, unlike the 2014 publication, which reflected contributions only from those in attendance at the 2013 meeting, the new document will invite submissions from all those with an interest in PMI technology and philosophies. The group opened to submissions in December and has a target publication of late summer, 2016, also in Frontiers in Neurorobotics. The group hopes that the open call for papers will bring even more members to the group.
“We want to get as many people involved as can be — specifically, we are looking for clinicians to join our cause because we are a bunch of engineers for the most part,” Wininger said, adding, “We are actively interested in getting the conversation to be broader and more widely disseminated.”
Phillips said he hopes to see more clinicians join him in lending a voice to the future of PMIs. Participation in workgroups allows clinicians to get a glimpse at upcoming innovations and helps them prepare to incorporate new technologies.
“Clinicians and users tend to think broadly about all the things they want a device to do, [while] researchers tend to focus more narrowly on a specific element they are advancing. The clinician helps drive home the practicality of the idea, and make researchers aware of conflicting requirements of daily use,” Phillips said. “I work both in research and clinical [practice] because I believe that the clinical requirements are often overlooked by researchers. I try to be that bridge in my work every day.”
Updates on the group’s research and upcoming workshops will be posted at http://pnsinterfaces.wordpress.com. – by Amanda Alexander
- Reference:
- Castellini C, et al. Front Neurorobot. 2014;doi:10.3389/fnbot.2014.00022.
Disclosures: Artemiadis, Castellini, Phillips, Pilarski and Wininger report no relevant financial disclosures.
