Novel Settlement Reached in Generative AI Deceptive Trade Practices Healthcare Investigation

The state of Texas, by and through its Attorney General Ken Paxton, reached a first-of-its-kind settlement agreement with Pieces Technologies Inc. (Pieces) following an investigation into Pieces' alleged violations of the Texas Deceptive Trade Practices Act (DTPA). This agreement marks the first time artificial intelligence (AI) has been addressed in a settlement following an investigation into the accuracy and truth of the proposed and marketed metrics. As a result, technological companies advertising and marketing the statistical accuracy of generative AI should be cautious regarding claims made in advertising and marketing, as the state may allege claims for violations of the DTPA.

Implications for Generative AI-Based Companies and Their Consumers

With the rise of generative AI, dependency on reliance values, such as accuracy metrics and hallucination rates, increases as more consumers turn to technology companies for AI software and products. The Office of the Attorney General (OAG) has recently begun to scrutinize marketed hallucination, which captures incorrect or misleading generative AI output, in addition to similar metrics, especially where such values appear optimally extreme. Because these accuracy metrics are directly related to the efficient and safe use by the consumer, especially in the context of summarizing, charting and drafting clinical notes for physician and medical staff reliance in patient treatment, generative AI in the healthcare sphere has been deemed to warrant the utmost transparency in data representation.

Pieces' marketed metrics made representations that its AI products were highly accurate, boasting a hallucination error rate of less than one in every 100,000. Upon an investigation by the OAG, these metrics were determined to likely be inaccurate and deceptive despite Pieces' denial of misrepresentation. Paxton warned that "[h]ospitals and other healthcare entities must consider whether AI products are appropriate and train their employees accordingly." He further cautioned that "AI companies offering products used in high-risk settings owe it to the public and to their clients to be transparent about their risks, limitations, and appropriate use. Anything short of that is irresponsible and unnecessarily puts Texans' safety at risk[.]" Ultimately, the state of Texas is seeking more reliable assurances as to the efficacy of a company's generative AI, increasing the burden on technological companies in order to offer greater protection to consumers.

What Is Required?

Ultimately, the state of Texas and Pieces negotiated a settlement and release agreement and, in late August 2024, filed a Petition for Approval and Entry of Assurance of Voluntary Compliance in district court. In consideration of the state's release of any claims based on Pieces' metrical representations related to allegations of false, misleading and deceptive practices in violation of the DTPA, Pieces agreed to the following:

• Clear and Conspicuous Disclosures in Marketing and Advertisements. Pieces has agreed to the clear and conspicuous disclosure, or provision in a readily noticeable and comprehensible manner for the ordinary person, of its output metrics, benchmarks and other analytical measurements. This is expected to allow for greater consumer transparency.
• Clear and Conspicuous Disclosures to Consumers. Pieces has agreed to the clear and conspicuous disclosure, as noted above, of any known or reasonably knowable harms or potential misuses related to its generative AI and other products and services.
• Prohibition Against Misrepresentation. Pieces has agreed to refrain from any false, misleading or otherwise unsubstantiated representation regarding product accuracy, reliability or efficacy, as well as product metrics and methodologies for measurement.

Looking Ahead

The Petition and Entry is expected to be ordered by the court in the coming weeks, with the requirements on Pieces continuing through 2029 and with periodic monitoring by the state to ensure proper compliance. Consumers and, specifically, healthcare providers and health industry companies utilizing AI, should undertake a risk-benefit analysis as to the use of generative AI in the healthcare context and should carefully evaluate the claims made by an AI technology company. Similarly, technology companies should carefully approach their representations and assurances of generative AI output and efficacy capacity.

Stay tuned for updates, as the disclosures shed light on the potential capabilities and risks of generative AI by healthcare entities and in particular on the advertising and marketing of AI products. For more information or questions, please contact the authors.

Information contained in this alert is for the general education and knowledge of our readers. It is not designed to be, and should not be used as, the sole source of information when analyzing and resolving a legal problem, and it should not be substituted for legal advice, which relies on a specific factual analysis. Moreover, the laws of each jurisdiction are different and are constantly changing. This information is not intended to create, and receipt of it does not constitute, an attorney-client relationship. If you have specific questions regarding a particular fact situation, we urge you to consult the authors of this publication, your Holland & Knight representative or other competent legal counsel.