The state also will start requiring providers to inform consumers when patient communications are generated by AI.
The laws reflect a growing trend among state lawmakers to more strictly regulate the use of AI in healthcare and other arenas in the absence of federal action.
The Physicians Make Decisions Act (SB 1120) takes effect on January 1. It was supported by dozens of physician organizations and medical groups, the California Hospital Association, and several patient advocacy groups. Insurance industry groups opposed the bill.
The new law requires the use of AI or any algorithms to be based upon a patient’s medical history and the individual’s clinical situation. A decision can’t be based solely on a group dataset, can’t supplant a clinician’s decision, and must be approved by a human physician.
In the absence of regulation, several lawsuits have been filed against insurers for their use of AI-based algorithms.
The families of two deceased Medicare Advantage recipients who lived in Minnesota sued UnitedHealth in 2023, stating that the company’s algorithm had a 90% error rate and was illegally employed, according to a CBS News report.
The US Senate Permanent Subcommittee on Investigations reported in October that its in-depth inquiry had found that insurers were using automated prior authorization algorithms to systematically deny post-acute care services for Medicare Advantage enrollees at far higher rates than denials for other types of care for other insureds.
In a 2022 report, McKinsey & Company touted AI’s potential to make prior authorization more efficient. But the authors noted that the AI would need to be monitored to ensure that it did not learn from biased datasets that “could result in unintended or inappropriate decisions,” especially for patients of lower socioeconomic status. The report concluded that “highly experienced clinicians will remain the ultimate PA decision-makers.”
While the American Medical Association (AMA) did not take a position on SB 1120, in 2023, the organization adopted a similar policy, calling for AI-based algorithms to use clinical criteria and to include reviews by physicians and other health professionals with expertise for the service under review and no incentive to deny care.
The California governor also signed AB 3030, which requires patient communications that employ AI to indicate that it was generated by AI, unless the communication was first read and reviewed by a human licensed or certified healthcare provider.
Murray said UCSF Health is already doing just that.
The health system has been testing the use of AI in helping to draft physicians’ responses to patient messages, with the goal of helping them answer more quickly. The messages have text that informs patients that AI was used to assist the doctor. It also states that the physician still reviews every communication.
RU
EN