Testing Strategies for Bluetooth Medical Devices

Bernhard Kappe

Published on: December 15th, 2023

This blog contains Part 5 of the Orthogonal White Paper titled “Bluetooth for Medical Devices.” The following are links to each part of the white paper:

For more in-depth discussions on Bluetooth from medical device experts, visit our page on our Bluetooth Low Energy for Medical Devices Webinar Series, co-presented by MedSec.

– – –

Table of Contents show

Part 5: Testing

Innovation requires iteration. Developers and manufacturers only arrive at safe and effective Bluetooth-enabled medical device solutions through strategic testing practices. In Part 5 of this white paper, we’ll discuss the objective of testing and share our recommended approaches.

Verification and Validation

Bluetooth-enabled medical device testing should be undertaken with a clear objective: to verify and validate the medical device system. Verification and Validation (V&V) are evidence that a medical device works as intended in the environment in which it is meant to be used. Through V&V testing, manufacturers demonstrate that they have achieved compliance, appropriately mitigated patient risk and crafted good User Experience (UX) design for their connected device.

Manufacturers verify that the device or system meets its requirements, including when it is run on platforms of which the developer doesn’t have full control. Orthogonal accomplishes verification throughout our Agile development process by leveraging user stories for acceptance test criteria, employing Test-driven Development and Behavior-driven Design and implementing documentation automation. We also perform automated testing of UX edge cases.

Manufacturers validate that the device meets its intended use. Orthogonal achieves validation by engaging in summative human factors testing as well as clinical validation.

Testing best practices

No matter the size of the budget or the length of a development process, it is not possible to test every aspect of a device in its entirety. This is particularly true for devices that run on consumer hardware (a.ka., Bring Your Own Device (BYOD)), where changes to the environment (e.g., new smartphones or operating systems (OS)) are beyond the control of developers. Instead, developers need to be strategic about what they test and how they test.

Risk analysis

Risk analysis is the process of identifying risks and challenges in a medical device software system, whether they be risks to the safe operation of the system, cybersecurity risk or risks to patient health. Once identified, risks can be further analyzed to determine if they are low risk, medium risk or high risk.

Risk-based approach to testing

A risk-based approach is when aspects of the device are prioritized for testing based on the results of risk analysis. The higher the risk to the system, the more testing is required. This strategy is used by the FDA when conducting clinical investigations.

At the outset of development, the location of the most critical dependencies in a device may not be clear. Developers might need to test more widely at first and use the data from tests to narrow their scope. A narrowed scope is also advantageous when presenting to the FDA, as it’s easier to add to testing in resubmissions than it is to pare the scope back.

Edge cases

We previously discussed testing strategies for edge cases in Part 2: Managing Edge Cases for Bluetooth Medical Devices. Likewise, we recommended taking the results of identifying, testing and mitigating edge cases and placing them in an edge case library that can be used as a baseline for future projects. Edge case libraries significantly cut down on the edge case workload for future Bluetooth devices.

Bluetooth for medical devices white paper CTA

BYOD testing layers

Managing a BYOD environment for a Bluetooth-enabled medical device is best done using a layered approach. Depending on the levels of risk, the following layers can be employed to maximize assurance out of a testing budget.

Platform compatibility assurance elements

Cloud-based device testing farm: Online device farms allow developers to do automated application testing at scale for a large number of device profiles, but excludes testing of Bluetooth functionality, Near Field Communication or smartphone sensors.
Private device farm: Developers can make their own testing farms by obtaining a representative set of physical phones. Private device farms are the most efficient way to test Bluetooth functionality, making them a useful investment for device manufacturers. Private farms are set up manually, but tests can be automated.
Automated testing harness/testing orchestration: A private device farm can be further automated with a testing harness and test orchestration layers. The image below is of a sample architecture:

bluetooth white paper part 5 testing image

Core test

A core test is a subset of the overall application verification protocol that covers the safety and essential performance of the mobile app. This kind of test is often used for platform compatibility testing instead of running the full verification protocol.

Field-level self validation

With the pool of smartphones and OS combinations in a BYOD environment always expanding, a connected medical device’s app may be loaded onto a device profile that developers have not yet tested. Field-level self validation adds a check in the process to ensure validation even on untested systems.

When the app is opened for the first time, the system runs a routine to detect an untested environment and communicates with the device peripheral to run a duty cycle. If the validation test returns a positive, the user will be able to run the app. If the test is negative, the user and the manufacturer will receive a notification, and the application will not be able to be used. This prevents critical functionality from malfunctioning and can give developers the chance to act ahead of user complaints. This protocol can also help developers understand if the negative result is part of a trend or a one-off occurrence.

Allowlist/blocklist/greylist

What device profiles should be allowed to run an app, and which should be banned? Allowlists, blocklists and greylists are maintained by manufacturers to manage functionality and feature usage for their app across different devices.

The allowlist specifies which device profiles are safe to run the app; the blocklist specifies which device profiles are not permitted to run the app; the greylist covers profiles that should be given limited functionality until they are further tested. The results of field-level self validation tests can be used to update the appropriate list. Depending on the risks, developers may opt for a more restrictive allow list.

These lists can be used in conjunction with monitoring in production to inform new tests and mitigations, as well as manage new discoveries.

Monitoring

Monitoring, as discussed in Part 3: Cybersecurity for Bluetooth Medical Devices and Part 4: Patient Engagement & User Experience for Bluetooth Medical Devices, lets developers see and review analytics of real-time patient usage and device performance. Tools like product analytics capture data that identify where problems lie and inform further testing.

Hazard architecture

A form of monitoring, hazard architecture refers to designing a medical device system to detect errors and hazards. It’s possible to architect a device to signal to developers when harms, hazards, hazardous situations and initiating events occur. By analyzing those signals, developers can judge the effectiveness of harm mitigations.

Conclusion

Successful Bluetooth devices need to address patient engagement, Bluetooth edge cases and cybersecurity. Conscientious, targeted testing helps solidify those aspects and makes Bluetooth-enabled medical devices impactful forms of treatment.

– – –

This blog contains Part 5 of the Orthogonal White Paper titled “Bluetooth for Medical Devices.” The following are links to each part of the white paper: