US states target health AI chatbots with new safeguards

State legislatures are starting to move from abstract AI principles to very specific rules about how models can touch patients. Colorado’s proposal is a good example: it doesn’t ban mental‑health chatbots, but it insists on transparency and on keeping a human in the loop for high‑stakes insurance decisions. That’s an early template for sectoral AI regulation—narrow, use‑case‑driven, and focused on disclosure and human oversight rather than broad model licensing.

For the AGI race, these kinds of rules won’t stop labs from training frontier models, but they will shape where and how those models actually get embedded into health systems. If similar bills spread, developers will need to design companion AIs that can clearly communicate limitations, hand off to humans gracefully and generate audit trails regulators can trust. That pushes the field toward more interpretable, controllable agentic systems, as opposed to opaque black boxes quietly deciding coverage or care pathways in the background.