Physician shortages, slashed budgets, coding inaccuracies.
AI could help solve these healthcare challenges and more — but only with a solid regulatory framework.
Healthcare organizations and agencies face critical challenges: a projected shortage of up to 86,000 doctors by 2036, family physicians spending 17 hours a week on administrative tasks, administrative backlogs due to reduced budgets, and significant claim-payment delays. AI offers practical solutions to these pain points (and more).
Smart automation with AI could reduce administrative workload, streamline prior authorization, strengthen payment integrity, and uncover insights in unstructured medical data — reallocating time to focus on outcomes and improve the experience of patients and beneficiaries.
Yet widespread AI adoption in healthcare requires clear regulatory frameworks. Legislation and regulation must protect providers, patients, and industry participants while enabling technological progress. Most AI regulation falls within these seven key areas of concern:
Beyond regulatory compliance, healthcare organizations must also consider crucial ethical implications in AI deployment. See our related article on ethical considerations in healthcare AI for a deeper exploration of these issues.
President Biden's 2023 Executive Order directed HHS to develop guidelines for safe AI deployment in healthcare and public health sectors. However, in January 2025, President Trump revoked this order and issued new AI guidance focused on reducing regulatory requirements. The new executive order emphasizes technological innovation and directs a review of previous AI policies, including those addressing potential discrimination and bias in AI systems.
The ONC HTI‑1 Final Rule, effective February 2024, introduced robust AI transparency and governance obligations for certified health IT supporting predictive Decision Support Interventions (DSIs). Developers must disclose 13 attributes for evidence-based DSIs and 31 for predictive DSIs to support user assessment of algorithm quality, fairness, validity, effectiveness, and safety (FAVES)
HHS Office of Civil Rights issued rules in April 2024 expanding anti-discrimination protections. Healthcare organizations must now demonstrate their AI systems are not discriminatory — marking the first direct federal regulation of healthcare AI fairness.
The FDA has also developed action plans to oversee AI/ML-enabled medical devices, focusing on transparency and safety.
On April 3, 2025, the Office of Management and Budget (OMB) released Memorandum M-25-21, "Accelerating Federal Use of AI through Innovation, Governance, and Public Trust." The current administration promotes a pro-innovation message that differs in tone from their predecessors. However, the memo's guidelines and policies largely maintain existing regulatory themes.
The memo introduces a "high-impact AI" classification for tracking AI-use cases requiring heightened due diligence, which specifically include systems impacting health, insurance, and government services.
The OMB also requires federal agencies to designate Chief AI Officers within 60 days and continues the requirement for agencies to inventory and document their AI-use cases annually. These continuities suggest federal oversight of healthcare AI will remain robust despite the shift in regulatory emphasis.
The Centers for Medicare and Medicaid Services (CMS) also took definitive action on AI in coverage decisions. In a policy memo, the CMS declared that Medicare Advantage plans cannot rely solely on AI to determine coverage or terminate services. A qualified healthcare professional must review these decisions.
In June 2025, NIH issued two RFIs to guide its evolving AI strategy. The first (NOT-OD-25-117) seeks public input on a strategic framework for advancing AI in biomedical research and public health, focusing on data readiness, trust, translation, and workforce development. The second (NOT-OD-25-118) addresses responsible use of generative AI with controlled-access human genomic data, aiming to identify risks, safeguards, and governance strategies. Both RFIs are open for comment through mid-July 2025.
Many states have introduced legislation to regulate AI in healthcare, including laws that require coverage determinations to be reviewed by physicians or non-AI humans. Some of these regulations also require transparency and assurances of non-discrimination. At least 60 bills introduced in states in recent years require transparency in AI, either between developers and providers or between providers and patients.
Some bills require reporting to the state. For example, California requires reporting of "Critical Harm" incidents, while Colorado requires companies using high-risk AI systems to document and disclose detailed information about the system's purpose, risks, and impacts on decisions.
Colorado is also enacting laws addressing "high-risk" AI applications — systems that influence healthcare access, treatment decisions, or insurance coverage — in healthcare and social welfare. Several states require physician involvement in AI-supported coverage decisions, including California, Georgia, Oklahoma, and Louisiana.
The Texas Responsible AI Governance Act (TRAIGA, HB 1709 / HB 149) recently passed both chambers and is expected to be signed into law. This comprehensive bill mandates transparency in AI systems used by state agencies and healthcare providers and establishes a state “AI Advisory Council” under its Department of Information Resources
Many of these state bills kick the can down the road by creating AI task forces or innovation councils instead of establishing immediate regulations. This reflects the tension between enabling AI innovation and ensuring proper oversight.
The FDA approaches AI-enabled medical devices through a risk-based framework. Their regulatory process examines an algorithm's intended use, development process, and clinical validation.
Software as a Medical Device (SaMD) faces particular scrutiny, especially when AI systems adapt or learn from new data. The FDA requires manufacturers to submit plans for monitoring real-world performance and managing algorithm updates. January 2025 draft guidance (currently under public comment) provides specific recommendations for AI lifecycle management and marketing submissions. The guidance emphasizes performance monitoring, transparency, and clear documentation of model development and validation.
State and federal regulations for healthcare AI focus on seven primary areas of concern:
Beyond regulatory compliance, healthcare organizations must also consider crucial ethical implications in AI deployment. See our related article on ethical considerations in healthcare AI for a deeper exploration of these issues.
The advancement of healthcare AI creates unique regulatory challenges, as new use cases and potential risks emerge daily. Oversight is developing through three interconnected channels:
Legal precedents:
Court decisions and settlements are setting standards for AI accountability and transparency. In 2024, the Texas Attorney General's settlement over overstated AI accuracy claims demonstrated heightened scrutiny of healthcare AI companies' marketing practices.
Federal and state regulation:
While agencies like the FDA oversee clinical decision support systems, regulatory frameworks must evolve to address emerging technologies and use cases.
Industry standards:
Organizations like the Coalition for Health AI (CHAI) are developing voluntary standards and "AI nutrition labels" to provide transparency about model capabilities and limitations. This has been adopted by the Joint Commission and accreditation processes are beginning in 2025.
Healthcare AI holds tremendous potential to address industry challenges, from provider shortages to administrative burdens. However, organizations must balance innovation with compliance as regulations develop at state and federal levels.
Early awareness of these requirements during the AI implementation process helps ensure sustainable, compliant solutions.
Keywell's healthcare AI specialists stay current with regulatory developments to guide organizations through implementation decisions. Contact us to discuss your AI strategy and learn how we help healthcare organizations navigate these requirements.
This article is intended for informational purposes only and does not constitute legal advice. Organizations should consult qualified legal counsel to address their specific regulatory obligations regarding AI in healthcare.
Healthcare AI holds tremendous potential to address industry challenges, from provider shortages to administrative burdens. However, organizations must balance innovation with compliance as regulations develop at state and federal levels.
Early awareness of these requirements during the AI implementation process helps ensure sustainable, compliant solutions.
Keywell's healthcare AI specialists stay current with regulatory developments to guide organizations through implementation decisions. Contact us to discuss your AI strategy and learn how we help healthcare organizations navigate these requirements.
Keywell provides HIPAA-compliant AI and analytics solutions for healthcare, integrating complex data to enable scalable, responsible AI deployment with expert guidance.
© 2025 Keywell. All Rights Reserved.