Linked resource: https://youtu.be/csYUDnj6qvo?a=1

Shared link

Connecting AI to Health Records (Live Workshop with Dave deBronkart, Hugo Campos, and Gilles Frydman). I recently posted about Health Skillz, a prototype I built to connect my health records to Claude. I was reacting to recent launches from OpenAI and Anthropic where connection failures and an inability to get full data into agents’ hands (including all FHIR JSON + clinical notes) frustrated my experience.

Dave saw the post and challenged me: “That’s a bunch of cool nerd stuff, Josh, but how does a normal person actually try this? What does it look like for people who aren’t programmers?”

I invited Dave & friends to a Google Meet to hand them the tools (and some explanations) and see what happened when they connected their own data live.

Video at https://lnkd.in/gpxVq2vF or read on for a quick summary.

We started by clarifying the architecture. Health Skillz uses SMART on FHIR, the same standard used by Apple Health. Importantly, the AI never sees your portal credentials. The user clicks a link generated by the AI, which opens a secure browser window for direct login to a patient portal. The browser fetches records, encrypts them, and hands the package back to the AI.

I demonstrated the “secret sauce” on my own records. My approach fetches everything—structured data (labs, meds) and unstructured clinical notes (e.g., HTML or text)—and hands it to the model inside an “agentic harness” that allows it to write code and parse files. After I connected to my provider, Claude queried structured data, but also scanned the unstructured notes to build a timeline of my post-concussive syndrome history, pulling context that a simple problem list would miss.

Hugo then connected his records. We hit friction points that illustrate the current state of these tools. First, we had to change his user settings to “Allow All Domains,” as the agent is restricted from outside network calls by default. Kaiser’s Northern California endpoint returned a “not found” error. And finding the right “Stanford” endpoint among similarly named entities was a hurdle.

Eventually data flowed. Hugo asked for a summary of a cardiology visit from March 2024. Claude correctly identified a stress echo and details about a generator change for his device, capturing nuances Hugo hadn’t immediately recalled.

We wrapped up discussing plumbing. There is industry activity around TEFCA, a national network designed for universal record access. However, TEFCA doesn’t provide consumers true visibility into their data flows, or the ability to exert controls. There’s also no “build-your-own” pathway in TEFCA for tinkerers like me.

“Using direct SMART on FHIR connections requires managing your own portal passwords, but it offers a direct pipe to the data. The session confirmed that the capability is here today. It requires patience, but we can now use off-the-shelf AI to synthesize complex medical histories in ways that existing patient portals simply don’t.