Testing Strategies for Bluetooth Medical Devices

Bernhard Kappe

Published on: December 15th, 2023

This blog contains Part 5 of the Orthogonal White Paper titled “Bluetooth for Medical Devices.” The following are links to each part of the white paper:

For more in-depth discussions on Bluetooth from medical device experts, visit our page on our Bluetooth Low Energy for Medical Devices Webinar Series, co-presented by MedSec.

– – –

Table of Contents show

Part 5: Testing

Innovation requires iteration. Developers and manufacturers can only arrive at safe and effective Bluetooth-enabled medical device solutions by employing a thoughtful and comprehensive suite of best practices for testing. In Part 5 of this white paper, we discuss the challenges of Bluetooth testing and share our recommended approaches.

Verification and Validation

Bluetooth-enabled medical device testing should be undertaken with a clear objective: to verify and validate the medical device system. Verification and Validation (V&V) are evidence that a medical device works as intended in its intended environment. Through V&V testing, manufacturers demonstrate compliance, appropriately mitigate patient risk and design a good user experience (UX) for their connected device.

Verification involves Manufacturers checking if the device or system meets its requirements, even on platforms the developer doesn’t have full control, such as a user’s personal smartphone. Orthogonal performs verification throughout our Agile development process by leveraging user stories for acceptance test criteria, employing Test-driven Development and Behavior-driven Design and implementing documentation automation. We also perform automated testing of UX edge cases.

Validation ensures the device serves its intended purpose. Orthogonal achieves validation through summative human factors testing and clinical validation.

Testing best practices

Regardless of the budget or development time, it is not feasible to test every aspect of a device in its entirety. This is particularly true for devices that run on consumer hardware, where changes to the environment (e.g., new smartphones or OS versions) are beyond the control of developers. Instead, developers need to be strategic in deciding what and how to test.

Risk analysis

Risk analysis is the process of identifying potential risks and challenges in a medical device software system, whether they affect the safe operation of the system, cybersecurity or patient health. Once identified, these risks can be further analyzed to determine if they are low, medium or high risks.

Risk-based approach to testing

A risk-based approach is when certain aspects of the device are given priority for testing based on the results of risk analysis. The higher the risk to the system, the more testing is required. This strategy is used by the FDA when conducting clinical investigations.

At the outset of development, the location of the most critical dependencies in a device may not be clear. Developers might need to conduct broader testing initially and use the data from tests to narrow their scope. A narrowed scope is also advantageous when presenting to the FDA, as it’s easier to add to testing in resubmissions than it is to pare the scope back.

Edge cases

We previously discussed testing strategies for edge cases in Part 2: Managing Edge Cases for Bluetooth Medical Devices. Likewise, we recommended taking the results of identifying, testing and mitigating edge cases and placing them in an edge case library that can be used as a baseline for future projects. Edge case libraries significantly cut down on the edge case workload for future Bluetooth devices.

BYOD testing layers

Managing a BYOD environment for a Bluetooth-enabled medical device is best done using a layered approach. Depending on the levels of risk, the following layers can be employed to maximize assurance out of your testing budget.

Platform compatibility assurance elements

Cloud-based device testing farm: Online device farms allow developers to do automated application testing at scale for a large number of device profiles, but does not support testing of Bluetooth functionality, Near Field Communication or smartphone sensors.
Private device farm: Developers can make their own testing farms by obtaining a representative set of physical phones. Private device farms are the most efficient way to test Bluetooth functionality, making them a useful investment for device manufacturers. Private farms are set up manually, but tests can be automated.
Automated testing harness/testing orchestration: A private device farm can be further automated with a testing harness and test orchestration layers. The image below is an example of a testing architecture:

bluetooth white paper part 5 testing image

Core test

A core test is a subset of the overall application verification protocol that covers the safety and essential performance of the mobile app. This kind of test is often used for platform compatibility testing instead of running the full verification protocol.

Field-level self validation

In a BYOD environment, where the pool of smartphones and OS combinations are constantly increasing, the app of a connected medical device may be loaded onto a device profile that developers have not tested before. Field-level self-validation is an additional check to ensure the app functions properly on untested device profiles.

When users open the app for the first time, the system checks if it’s on an untested environment and communicates with the device peripheral and runs a duty cycle. If the validation test shows a positive result, the users can use the app. But if the test shows a negative result, both users and the manufacturer will receive a notification, and the application can block some of its features from being used. This prevents critical functions on the app from malfunctioning and gives developers a chance to fix issues before users encounter problems. This protocol also helps developers figure out if a negative result from the test is part of a bigger pattern or just a one-time occurrence. If a fix to the software is required, field-level self-validation can be used to prevent users from accessing device features before a patch is released.

Allowlist/blocklist/greylist

What device profiles should be allowed to run an app, and which should be banned? Allowlists, blocklists and greylists (renamed from whitelists and blacklists) are maintained by manufacturers to manage functionality and feature usage for their app across different devices.

The allowlist specifies which device profiles are safe to run the app; the blocklist specifies which device profiles are not permitted to run the app; the greylist covers profiles that should be given limited functionality until they are further tested. The results of field-level self-validation tests can be used to update the manufacturer’s list. Depending on the risks, developers may opt for a longer or shorter set of devices on the allow list.

These lists can be used in conjunction with monitoring in production to inform new tests and mitigations, as well as manage new discoveries.

Monitoring

Monitoring, as discussed in Part 3: Cybersecurity for Bluetooth Medical Devices and Part 4: Patient Engagement & User Experience for Bluetooth Medical Devices, allows developers to observe and analyze real-time data on how patients use the device and how well it performs. Tools like product analytics capture data that helps identify problems and informs further testing.

Hazard architecture

A form of monitoring, hazard architecture refers to designing a medical device system to detect errors and hazards. A device can be built to alert developers when harms, hazards, hazardous situations and initiating events occur. Analyzing these signals helps developers assess the effectiveness of harm mitigations.

Conclusion

Successful Bluetooth devices must address patient engagement, Bluetooth edge cases and cybersecurity. Conscientious, targeted testing helps solidify these aspects, ensuring Bluetooth-enabled medical devices become impactful forms of treatment.

– – –

This blog contains Part 5 of the Orthogonal White Paper titled “Bluetooth for Medical Devices.” The following are links to each part of the white paper: