What breaks in an AI-built health app before launch?

What breaks in an AI-built health app before launch?

Health data leaks through places nobody thinks of: logs, analytics, error trackers and AI prompts. If you are a covered entity or work for one, HIPAA applies and every vendor that touches the data needs the right agreement. Work out whether the rule applies to you first, then map where patient data flows.

What is really going on

Whether HIPAA applies depends on who you are and who you serve, not on whether the topic is health. The rule applies to covered entities (such as providers and health plans) and to business associates, the companies that handle protected health information for them. A consumer wellness app that never works with a covered entity may fall outside it, though other laws and the Federal Trade Commission can still apply.

If it does apply, the technical part is mostly about knowing where the data goes. AI-built apps are assembled from many services, and each one that stores, processes or even logs patient data matters. The common failure is a vendor receiving protected data with no agreement in place.

Even where HIPAA does not apply, health data is among the most sensitive a product can hold, and the access-control and exposure mistakes in other pages are far more costly here.

Where health apps go wrong, most common first

1. Nobody has worked out whether the rules apply

   How to tell The team assumes either that HIPAA does not matter or that it covers everything. Customers who are clinics or health plans ask for a business associate agreement and the team cannot answer.

   The fix Ask a health-law specialist who you are under the rules, with your customers and data flows in hand. Decide before you build integrations that depend on the answer.

2. Vendors receive protected data without the right agreement

   How to tell Patient data reaches hosting, the database, email or SMS, analytics, error tracking or an AI model provider on a plan that does not include a business associate agreement.

   The fix List every vendor the data reaches. For each, confirm in writing whether it will sign an agreement and on which plan, or keep the data away from it.

3. Patient data in logs, analytics and error reports

   How to tell Search your logs, analytics events and error tracker for names, emails, dates of birth and free text. Request bodies and exceptions often include whatever the user typed.

   The fix Scrub identifiers before they reach any logging or analytics tool, turn off session recording on pages with patient data and set short retention.

4. Patient data sent to an AI model

   How to tell Prompts include notes, messages or records. Some AI providers retain prompts or use them for improvement unless the account is configured not to, and not all offer the agreements health data needs.

   The fix Check the provider's terms for retention, training use and an agreement, minimise and de-identify what you send and keep a record of the setting you chose.

5. Weak access control and no audit trail

   How to tell Staff share logins, every employee can open every record or there is no log of who viewed what. Row level security is off or permissive.

   The fix Give each person an individual account with two-factor authentication, limit access by role to what the job needs and keep an access log you can produce.

6. No plan for a breach, deletion or an export

   How to tell You cannot say how you would find out about unauthorised access, who must be told or how you delete one patient's data across all systems.

   The fix Write a short breach response plan and a deletion procedure that covers backups and vendors, and test both once.

How we diagnose it

1. Map the data flow. Draw every place patient data is entered, stored, processed, logged and sent, including third-party tools. This map is the basis of everything else.
2. Check each vendor's terms. For every vendor on the map, find out whether it will sign the agreement you need and whether your current plan includes it.
3. Search for data where it should not be. We look in logs, analytics, error reports and prompt history for identifiers and free text from patients.
4. Test access as the wrong person. Try to read one patient's record as another user, a different role and a stranger, through the app and through the API.
5. Read the code that handles the data. We review authorization, encryption settings, file storage and the code paths that send data out, and rank findings by harm.

What fixing it changes

• A written answer, from a qualified lawyer, on which rules apply.
• A data-flow map and a vendor list with agreements where they are required.
• Patient data removed from logs, analytics and error reports.
• AI provider settings and terms checked, with minimal data sent.
• Individual access, role limits and an access log, plus breach and deletion procedures.

When you do not need us

We can read the code and test the app for exposure, but we do not provide legal advice or certify HIPAA compliance, and no code review can. If you are building a general wellness tool with no clinical customers, the legal question may be simpler, but the access and exposure checks still apply.

Your next step

Straight answers