Key takeaways:
- Most providers are using some AI in workflows to simplify documentation, administrative workflows, scheduling and/or marketing.
- State and Federal regulations are simultaneously catching up and evolving –– introducing AI legislation, launching task forces and increasing oversight.
- Before adopting an AI solution, understand what data it collects, how it’s stored, whether it’s used to train AI models and what safeguards vendors have in place.
- Keep humans accountable as the final decision-makers.
In May, Pennsylvania’s Attorney General sued the company behind an AI chatbot, alleging it falsely presented itself as a licensed psychiatrist. According to the complaint, the chatbot, “Emilie,” claimed to hold a medical degree from the United Kingdom and even generated a fictitious Pennsylvania medical license number when questioned.
Of course, no one intended for the AI tool to impersonate a licensed provider, but the Pennsylvania case illustrates just how misleading AI can be, especially as it becomes more integrated into healthcare.
Across the United States, state lawmakers are introducing AI legislation, attorneys general are bringing enforcement actions and healthcare organizations are facing new legal questions about how AI is used — from patient-facing chatbots to ambient documentation tools.
For providers building or growing a practice, AI use is complicated by healthcare’s increased regulatory environment and HIPAA patient privacy laws. Used thoughtfully, AI can help practices improve efficiency without sacrificing quality. Misuse, or even just misalignment, with a vendor who doesn’t understand a state’s healthcare compliance laws can result in regulatory issues.
Whether you’re evaluating a ready-made platform or building a custom solution, the following are some important questions to answer before introducing AI into your practice.
How Healthcare Practices Are Using AI to Support Daily Workflows
No serious provider is looking to replace clinical expertise with artificial intelligence. Instead, providers are typically using AI to:
- Draft patient education or marketing materials
- Simplify administrative tasks and improve operations
- Document notes
- Respond to common inquiries
- Assist with scheduling or intake
Like in any business, these are practical applications designed to save time. However, even administrative AI tools can introduce new risks within the healthcare industry. Simply interacting with patients or their protected health information (PHI) requires adherence to HIPAA privacy laws.
Tip: As AI capabilities continue to evolve, it’s best to work with vetted vendors who understand healthcare, HIPAA compliance and Business Associate Agreement (BAA). Still, the one ultimately accountable for AI safety is the practice owner.
Recent AI Lawsuits Highlight Why Healthcare AI Governance Matters
Several recent legal developments demonstrate how expectations around AI are changing.
Few small practices are likely to get caught up in legislation like in Pennsylvania, where AI impersonated a provider. However, a recent lawsuit in California highlights a much more likely and equally problematic challenge. Plaintiffs alleged that an ambient AI documentation tool recorded and transmitted patient conversations without appropriate consent. It’s the kind of AI note-taking tool many practices use. The litigation is ongoing, but it illustrates how AI is a privacy issue.
At the same time, states across the country are introducing AI legislation and launching task forces to examine transparency, consumer protection, healthcare oversight and data privacy. While requirements vary by state, the overall trend points toward greater accountability for organizations deploying AI.
Tip: Err on the side of being as transparent as possible. Notify your patients in multiple ways about your use of AI, such as verbally, on your website and by posting signs.
Ask questions before adopting an AI tool
Rather than evaluating AI based solely on convenience or cost, practices should conduct the same level of due diligence they would for any technology that affects patients or business operations, such as:
Ask This First: What Problem Is the AI Tool Actually Solving?
Providers are pitched AI tools at a relentless rate. Before implementing a new tool, ask:
- Does it improve workflow?
- Does it save time?
- Does it reduce administrative burden without introducing unnecessary complexity?
- How can I determine if it’s effective?
Tip: Starting with a clearly defined use case helps you avoid adopting technology that creates more work than it eliminates.
AI Transparency: Do Patients Know When They’re Talking to a Bot?
The California case highlights how transparency is quickly becoming one of the most important principles in responsible AI use.
If a chatbot answers patient questions, if AI helps respond to patient communications or if automated tools interact directly with prospective patients on social media, practices should carefully consider how to make the AI use abundantly clear.
Tip: Patients generally appreciate, or at least tolerate, innovation when they understand how technology is used to improve their experience and when human oversight remains part of the process. Be transparent about how your practice uses AI.
How AI Vendors Handle Protected Health Information
Any AI platform that processes protected health information requires scrutiny and a BAA.
Before selecting a vendor, consider asking:
- Does the platform access patient information?
- Is data used to train future AI models?
- Will the vendor sign a BAA?
- How is information encrypted, stored and protected?
Tip: It’s no excuse to say you don’t understand technology. If you use a vendor to handle your patients’ PHI, how they use that information is your responsibility.
Why Human Oversight Is Non Negotiable With AI in Healthcare
AI makes mistakes. Whether drafting educational content, summarizing documentation, assisting with billing or generating administrative communications, AI outputs should be reviewed by qualified individuals before being relied upon.
Human oversight remains one of the strongest safeguards against inaccurate information, inappropriate recommendations or unintended bias. Oversight is also what makes patients comfortable with the use of AI.
Tip: Ensure you can integrate human oversight into your workflow before adopting any new AI technology in your practice.
How to Keep Up With Evolving State AI Regulations
Healthcare AI regulations remain fragmented, with states taking different approaches to disclosure, privacy, consumer protection and oversight.
Tip: If there’s a new bill in your state that requires a certain notification incorporated into AI, for example, you need to know who’s tracking that.
Transparency Is What Keeps Patient Trust Intact
Patients trust providers with their health and their personal information. That trust shouldn’t change simply because AI becomes part of a practice’s operations. AI is a business tool that requires thoughtful implementation, appropriate oversight and clear communication.
AI will continue to evolve. Practices that build responsible governance now are better positioned to embrace innovation while retaining patient trust.
While AI promises to save you time, ensuring and monitoring your AI use and vendors’ compliance can save you from major hassle and even a lawsuit. As technology continues to reshape healthcare, risk management remains essential.
CM&F helps allied health professionals protect their practices with insurance solutions and educational resources designed to support long-term business success. Get a quick quote on our website –– from a human, not a bot.
