The Product Analytics Marketplace: SaMD Best Practices

Bob Moll

Published on: February 16th, 2022

This blog contains Chapter 2 of the Orthogonal white paper titled: Accelerating the Creation of Software as a Medical Device (SaMD) with Product Analytics. The following are links to each chapter of this white paper:

Product Analytics Dashboards and Reporting

Product analytics information is consumable without the need for highly specialized skills. This allows a range of professionals to consume the data, such as product management, design, and marketing. It can be used during initial design/development, clinical trials, and in production once the product is formally launched.

The following is an example of a product analytics dashboard implemented with Mixpanel that shows the power of user data viewed at aggregate levels for individual users and/or at cross-user levels. Custom dashboards allow various groups such as R&D and marketing to see exactly how users are using the product in real-time:

pba demo product analytics orthogonal samd

The below example is of a product insights report (implemented with Mixpanel) that allows the device manufacturer to track how many people by country are using the software in a given time frame. Product analytics tools usually come with a variety of out-of-the-box reporting options based on industry best practices, as well as customizable reports. Reports can also typically be exported so that they can be used as part of a cross-system data warehouse or analytics tool. These reports can be generated and modified by analysts who do not have any programming skills.

events country code pba demo product analytics orthogonal samd

The Product Analytics Vendor Marketplace

There is a rich marketplace of product analytics tools to select from including:

At Orthogonal, our go-to tool for product analytics with connected medical devices is Mixpanel. After assessing several tools in the marketplace over 2019 and early 2020, we found Mixpanel to be a good fit for connected medical devices:

Mixpanel is built around events and user properties which are the building blocks of data that will be used for analysis and insights. It offers the ability to build custom reports and dashboards. This allows the creation of dashboards for different audiences in the organization, such as product management, UX, development, and customer service.
It is HIPAA and GDPR compliant.
It features an easy-to-use interface.
The pricing model requires a low initial investment and continues to be cost-effective as your usage grows, allowing for easy experimentation on new products and with new customers.
Finally, Mixpanel has been a collaborative partner. They support the application of their tool to the full lifecycle of connected medical devices as a product category.

Orthogonal White Paper Product Analytics CTA Banner

Alternatives to Product Analytics, and Why They Are Not As Effective

You might ask what the alternatives are to implementing off-the-shelf, person-based analytics using a tool such as Mixpanel. We’ll be blunt in our answer, so that you can spend your time making new and interesting mistakes and not old and proven ones that our team has already learned:

If you are going to implement product analytics, for the sake of your patients, their providers, your colleagues, and your family’s sanity, do not attempt to build a homegrown product analytics tool from scratch. It is possible to build your own product analytics data collection and analytics tools with custom software. However, it would be expensive to build and expensive to maintain, as you’ll be bypassing the large economies of scale that third party vendors have used to develop a rich suite of product analytics offerings.
Another product analytics alternative we are frequently asked about is the use of Google Analytics (GA). This is a very reasonable question, especially since GA is available at no licensing or usage cost. However, while we have found GA to be a very useful tool for a variety of use cases, product analytics is not one of them. The biggest problem with using GA for product analytics is that GA does not allow you to access and analyze the full suite of raw data collected about your user’s past behaviors. In other words, GA does not offer true product analytics since it does not give you access to the full person-level data. In using GA, you have already sacrificed a large part of the potential value of product analytics before you have even started your first analysis.

Product Analytics Best Practices to Adopt

We would like to point out some best practices for implementing and using product analytics:

Start by defining what you would like to learn from data. For example, you may wish to understand the percentage of drop-offs in the account creation process.
Create visual mock-ups of your proposed reports and vet them with stakeholders. This will help spur new ideas and set expectations
Make a simple but complete specification for your development team to instrument tracking of events and user properties. We use a spreadsheet template that contains everything the development team needs.
When selecting who will do the development, it is usually best to select a developer who is familiar with the codebase of the application even if the developer has no experience implementing analytics. The reason is that it is usually easier to learn to code analytics than it is to learn a codebase of a medical device application.
For the best experience using the data, create a dashboard for each user group that contains the reports most relevant to that group. On the first launch of analytics, everyone may share one common dashboard, but as time passes and more data is collected, several dashboards might be needed.

Product Analytics Worst Practices to Avoid

We would also like to point out some worst practices that should be avoided:

Do not try to instrument every click and process in the application without first defining what data is of most interest.
Naming your events using language that is not simple and intuitive should be avoided; otherwise, reports could be hard to read.
Be careful about capturing data that can be considered Protected Health Information (PHI) without first understanding the obligations that come with handling such data.

What’s Next? How Do I Start My Journey With Product Analytics?

If this white paper has piqued your interest in this topic, and you are wondering what would be involved in adding product analytics to your medical device toolkit, we recommend the following these steps:

Define your requirements for product analytics. Start by working backward: Identify the product analytics data you want to visualize and how it can help you. Figure out what you will do with the data. Identify areas to work on, and flesh out possible leads on data insights/trends with qualitative work to understand “why”. It can be helpful to prototype the analytic tools that you would want to create from your data as static, low-fidelity reports, or something more interactive like pivot tables based on mocked-up data.
Address the privacy questions: “Just because we can collect a ton of data about users, can we, and should we?” As with everything else we do as medical device professionals, privacy, ethics, rights in data, and respect for our patients need to be front and center. Choosing a privacy-minded, regulation-compliant product analytics tool is an important first step. That being said, your product analytics tool can’t prevent you from doing the wrong thing with what data you collect or how you steward that data. Active adherence to data compliance is key.
Evaluate the products against the requirements defined in the two above steps and make a selection. Consider key factors such as HIPAA and GDPR compliance, ease of use, cost at various points of scale, ease of implementation, and support of key features such as cohorts, flows, and funnels.
Get a product analytics expert on board. Even if they have limited knowledge of medical devices, they can apply their personal knowledge of healthcare and devices to help you create a tool.
Instrument your app and launch. Note where privacy and confidentiality are needed and anonymize data where necessary.
Design your dashboards to support your identified product analytics requirements (from #1). Once an app is instrumented, information is real-time and needs to be monitored. With a product analytics tool, information is easily consumed via reports and dashboards. Each audience (UX, Product, Marketing) can have its own dashboard.
Prioritize work for new features. Implement the new features and test them in the field.

Conclusion

We’ll close with two final takeaways on this topic:

Be faster. Product analytics is a powerful tool for gaining insight into how customers are actually using your product. It keeps the cycle of innovation going because the data provided can lead to rapid insights and faster release cycles that make your product less prone to user errors and more desirable to users.
Go off-the-shelf. Leveraging an off-the-shelf product analytics tool can help you implement product analytics at scale and receive feedback in real-time. This tool can be customized for the various audiences in your MedTech organization, including R&D, HF/UX, software engineering, quality and regulatory, and marketing.

If you do go down this path, we’d love to hear from you directly (or through an anonymous email account) so that we can share ideas and best practices. In our estimation, product analytics is far too important to the health of each of our family members, friends, neighbors, and co-workers to not help each other spread this value across every connected medical device. Hopefully, you agree with us that this kind of real-time data provided can enable insights that lead to more rapid conceptualization and turnaround of new features.

Who Is Orthogonal? And Why Do We Care About Digital Health and SaMD?

Orthogonal is a software developer for Software as a Medical Device (SaMD), digital therapeutics (DTx) and connected medical devices.

We work with change agents who are responsible for digital transformation at medical device and diagnostics manufacturers. These leaders need to accelerate their pipeline of product innovation to modernize patient care and gain a competitive advantage.

Orthogonal applies deep experience in SaMD and the power of fast feedback loops to rapidly develop, successfully launch, and continuously improve connected, compliant products—and we collaborate with our clients to build their own rapid SaMD development engines.

Over the last 10 years, we’ve helped a wide variety of firms develop and bring their regulated/connected devices to market.

Almost twenty years ago, we began our consulting work to create great software products in a range of industries, including financial services, education, research, and healthcare. We spent nearly a decade working with leading-edge fast feedback loop techniques that have now become recognized best practices such as Agile software development (with XP, Scrum, and Kanban), Lean User Experience (Lean UX), and Lean Startup.

Nearly a decade ago, we got a glimpse of digital healthcare and the enormous potential of the cloud, IoT, and smartphones, and the potential impact of medical device software. Realizing that’s what we wanted to focus our energies on exclusively, we narrowed our focus to the development of SaMD, DTx and connected medical devices.

We’ve never looked back. Our team gets up every day excited to help move the needle on healthcare. For almost ten years, we’ve helped a wide variety of firms develop and bring their regulated/connected devices to market. We’re even more excited about the next ten years and what we will help accomplish during The Great Acceleration of Digital Health.

If you need help building your next connected device, or to learn more,
send us a message or call us at (866) 882-7215.

Social Media

Website: https://devorthogonal.wpengine.com/

Twitter: @orthogonal_io

LinkedIn: https://www.linkedin.com/company/orthogonal

Facebook: https://www.facebook.com/OrthogonalSoftware

Join the Conversation and Improve It

Our favorite inbound messages after we publish a white paper do not start with, “Great white paper. Keep up the good work.” Our favorites are the ones that start with, “I read your white paper and I disagree when you say X because you are not taking Y and Z into consideration.”

Our ideas are ultimately strengthened by open, candid conversations. We’d love to hear from you. Do you agree with our take? Disagree? Have questions? Have you read an article that we should read? Feel free to reach out to Orthogonal’s CEO, Bernhard Kappe and Randy Horton, Orthogonal’s VP of Solutions and Partnerships.

In return, we will look for ways to share more white papers, personalized insights, content, and introductions that are highly relevant to your work.

Back to Insights