Teaching AI to Reason About Healthcare Data

Healthcare data is not known for its cleanliness. Yes, it is often dirty.

It arrives from many sources, with confusing definitions, implicit business rules, inconsistent formats, and value sets that keep changing. CMS itself revises report templates and claim-feed specifications over time and documents limitations when comparing Claim and Claim Line Feed data with other reports and sources (CMS Shared Savings Program guidance). The organizations sharing the data frequently inherit the same upstream problems. At every handoff, ambiguity can compound: messy source systems become messy extracts, and downstream teams receive data that is even harder to interpret. This is why healthcare data quality is multidimensional and depends on the question being asked (Weiskopf and Weng, 2013; Kahn et al., 2016).

The business of healthcare makes this more difficult. The ways providers and payers are reimbursed, the regulations that govern them, and the compliance requirements surrounding each workflow are rarely straightforward. Claims data are created primarily to support reimbursement, so information needed for payment is generally more reliable than other clinical details (ResDAC). Almost nothing is as obvious as a column name makes it appear.

No matter how advanced your training in data science or AI, this is one of the first major barriers you encounter when working with healthcare data. Even basic utilization counts can require careful methodology: CMS notes that Medicare Advantage encounter files may contain multiple records representing the same service event (CMS Medicare Advantage Encounter Data Analytical Methods). The data contains countless patterns, but many are artifacts of collection, payment, or operations rather than meaningful clinical or business signals.

So how do we teach AI to reason about healthcare data instead of merely finding patterns within it? That is the challenge Falcon has been tackling from the beginning.

Today, we are sharing an upgrade to our data catalog—one layer of Falcon's broader reasoning system—and the meaningful improvement it produced in our latest benchmark. Our previous system was already capable. Now we are measuring it against a harder version of itself. In other words, we are competing against ourselves.

Learning From Everyday Use

Falcon was already helping users answer complex healthcare questions. As more people began using our applications in their daily work, they also helped us see the places where the AI could become more precise.

Real usage reveals details that are difficult to anticipate in a laboratory. A question may be phrased differently than expected. Two sources may look compatible but describe different points in time. A familiar metric may need a different interpretation for a particular payment model or population.

We have encountered this pattern across seemingly simple questions such as “Who is my member?”, “Which MOR is this?”, and “Which Medicare coverage policy applies?”. Each question becomes difficult once timing, source systems, and healthcare-specific definitions enter the analysis.

When we encounter these moments, we do not treat them as isolated glitches. We turn the underlying reasoning challenge into a privacy-safe benchmark case. This gives us a repeatable way to improve the product and confirm that an update solves the broader problem rather than one specific example.

Making the Benchmarks Harder

Our benchmark questions are increasingly inspired by real healthcare research tasks that required extra care: understanding claims at the correct level, connecting provider and member context, interpreting changes over time, tracing where data came from, and recognizing when more evidence is needed.

We evaluate more than whether the AI produces an answer. We look at whether it:

• Uses an appropriate analytical method.
• Connects the right sources and context.
• Interprets the result in a healthcare-relevant way.
• Understands the origin and limitations of the data.
• Recognizes when the available information is not enough.

As Falcon improves, the benchmark must improve too. We add more realistic variations, harder edge cases, and questions that test whether the system learned a general principle instead of memorizing a preferred response. This is especially important as populations, workflows, coding practices, and source systems change—a challenge known as dataset shift (Subbaswamy and Saria, 2020).

The results also show where we still need to improve. Gains were much stronger on neutral questions than on prompts that suggested a flawed approach. That gap gives us a clear direction for the next generation of benchmark cases.

What We Saw

In a 34-question diagnostic run on September 2, 2026, the context-enriched version of Falcon passed 21 cases, compared with 10 for the baseline.

The largest gains came from better analytical methodology and a clearer understanding of data provenance—knowing not just which sources Falcon can support, but which information actually produced the data being analyzed.

These results are a diagnostic snapshot, not a universal score for every healthcare question. The run used one repetition, and the benchmark continues to evolve. Reviewing the results also helped us improve the evaluator itself by identifying answers that were correct but scored too rigidly. A smarter AI requires smarter tests.

More Than a Catalog Update

Improving the table catalog was an important part of this work, but the broader goal is to give Falcon a connected understanding of healthcare data.

That includes the structure and meaning of each dataset, the relationships between sources, their timing and lineage, the research methods appropriate for different questions, and the boundaries of what the evidence can establish. Together, these layers help Falcon move from locating fields to reasoning through a healthcare problem. For investigations that span claims, policies, provider information, and external evidence, we have described this connected approach as the three pillars of data for healthcare research.

The result is not simply an AI that knows more facts. It is an AI that is becoming better at applying context: choosing the right data, connecting the dots, explaining its reasoning, and avoiding conclusions the data cannot support.

The Improvement Loop

Healthcare AI will never be finished after one model release or one catalog update. The questions evolve, the data evolves, and our understanding of the difficult cases evolves with them.

Everyday use helps us discover new nuances. Those nuances become harder benchmarks. The benchmarks guide improvements to our data structure, context, tools, and evaluation. Then we test again.

That loop—real questions, careful measurement, and continuous refinement—is how we are teaching Falcon to become a more capable healthcare research partner.

References and Further Reading

Related Falcon Articles

• Who Is My Member? A Surprisingly Complex Question
• Which MOR Is This? A Surprisingly Expensive Question
• Finding the Right Medicare Coverage Information Is Surprisingly Challenging
• Three Pillars of Data for FWA Research

Research and Evaluation Guidance

1. Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association. 2013;20(1):144–151.
2. Kahn MG, Callahan TJ, Barnard J, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. eGEMs. 2016;4(1):1244.
3. Subbaswamy A, Saria S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics. 2020;21(2):345–352.
4. Research Data Assistance Center. Strengths and limitations of CMS administrative data in research. 2021.
5. Centers for Medicare & Medicaid Services. Medicare Shared Savings Program guidance and specifications. Includes report templates, CCLF file specifications, and documented uses and limitations.
6. Centers for Medicare & Medicaid Services. Medicare Advantage Encounter Data Analytical Methods. June 2026.