For six years I’ve had a chronic condition, and I still can’t tell you what it is.
I’ve been tested for rheumatoid arthritis. I’ve been tested for lupus. I’ve been tested, more than once, for why the exhaustion won’t lift — I’ve had enough blood drawn that the phlebotomists started looking at the order sheet and asking, “Is this right?” Somewhere in those years, childhood allergies that were nothing since kindergarten came back hard enough that the wrong food now means a serious respiratory or a skin reaction, and the wrong foods are common ones, so my diet keeps narrowing. It’s taken a real bite out of my quality of life and my ability to work, which is why I’ve spent six years trying to figure it out.
And what I’ve come to realize is that short of the U.S. healthcare system changing in a way it’s not going to change, I never will.
It’s the rubric, not the doctor
That’s not a complaint about my doctors. They’re not bad at their jobs. The trouble is the job they’re actually doing, which is running a rubric. When we were trying to work out whether I had a rheumatological problem, one of them explained the logic to me directly: they’ve got a specific set of tests for a specific set of ailments, each tied to a specific service or product the practice offers, and if nothing comes back matching that battery of options, there’s nothing they can provide.
Nobody comes up with a hypothesis, builds a test, runs it, reads the result, and cycles back through the next possibility — the plain scientific method, the sort of curiosity you’d expect from any expert in their field. Doctors end up acting like IT administrators for your body, asking if you want to get your flu shots (“did you turn it off and on?”) or if you’ve turned in your colorectal cancer screening (“did you reset your password?”).
And I feel for doctors because it’s a design problem, and the design is in the system of billing codes. If there’s no code, you can’t bill it; if you can’t bill it, you can’t recoup the cost — and the actual work of finding something new doesn’t have a code.
Now put that rubric inside fifteen minutes. In the fifteen to thirty minutes a doctor gets with you, especially in an HMO, they’ve got to get up to speed on who you are, collect the data, translate it, and come up with a next step — more analysis, or more testing. That’s not enough time to connect the dots for any one patient, and it’s nowhere near enough to reframe the whole approach around a history that doesn’t fit the template. Diagnostic error costs an estimated $100 billion a year; the federal research budget aimed at it runs between $15 million and $20 million. That’s the ratio the system picked.
You can feel it in how frustrated people are, and in how fast they turn to the internet the second they leave the office. Everybody’s looking for the same thing — something that helps them understand their own health, and gets them a better outcome than the one they’re getting through their physicians. It’s gotten bad enough that quackery is a whole business model now; there’s real money in telling people what their doctors won’t take the time to.
And the newest place people are turning is a chatbot. About a third of U.S. adults now use AI chatbots for a health reason, Pew Research Center found — to figure out what’s causing a symptom, to make sense of a lab result, to understand a diagnosis or a treatment. Given the wait, the cost, and the misdiagnosis, of course they are. I did it too.
The chatbot has the same problem
Here’s the problem, though: a chatbot with no context has the same problem the fifteen-minute doctor has. It’s like walking up to a doctor who’s a hundred percent invested in you, well educated, up to date on the latest information, and just saying — I’ve got this rash, what’s wrong with me? They still need to do tests. They still need the meta-analysis. And they still need a deeper sense of who you are — your background, your dietary and exercise habits, all of it — and you need to be honest with them about it. Without that, they can’t give you a good diagnosis, or the kind of recommendation that lives outside the prescription pad. Neither can the chatbot.
And that’s what the survey data shows. People who get health information from chatbots rate it convenient and easy to understand, but not accurate and not personalized — nearly half say what they get isn’t personalized to them at all. Of course, it depends on the question you’re asking. The flu, poison oak, the sniffles — something with an obvious set of symptoms and an obvious diagnosis — sure, it could help. Then again, maybe it’s inaccurate. The real thing is we don’t know without context. Nearly all chatbot health users say their AI conversations are helpful. But in a controlled study, the people using those same models landed on the right condition less than 35% of the time — no better than the people who just used a search engine. Helpful and accurate aren’t the same thing, and most people can’t tell them apart. I couldn’t either, until something else looked at the advice I’d been given — but I’ll come back to that.
So the question isn’t should we use AI or not. The question is how we use it better, to get better outcomes — and what our reaching for it says about what we’ve already got. People wouldn’t be doing this if their health outcomes were great. Nobody’s happy with their healthcare, whatever they tell a pollster — and I know that’s a strong claim, so here are the numbers next to it. Americans rate the system at a 24-year low, with 70% saying it has major problems or is in a state of crisis — Gallup’s most recent reading on that question, from late 2024. And yet, asked about the care they personally get, 71% call it good or excellent. Those can’t both be true in a country that finishes last of ten wealthy nations on outcomes while paying more than any of them. If I’m honest, I don’t think people are reporting their own experiences accurately. Nobody wants to admit they chose a bad doctor. I’ve chosen multiple bad doctors. I know the feeling I’m describing because I’ve talked myself out of it.
One word: context
If the answer to “how do we do this better” starts anywhere, it starts with one word. Context. It’s the thing both the doctor and the chatbot were missing. And it’s the hard part, because context is data — it’s information, and it’s the private stuff we don’t really want to give away willy-nilly.
In my case, getting there looked like this. I took all of those blood tests, every single one — actually, my wife did. She copied and pasted every result into a set of documents. She pulled the sleep tests, the dates, the times. We took my Garmin watch data, parsed it, and pushed it in. And then we had long conversations — my history, my family history, my symptoms, the timing of when things got worse and what was going on in my life when they did. Out of all of it, the AI process started to put together my total medical picture, and it was a better one than any single doctor had gotten to.
Watching that happen taught me something I hadn’t expected. The doctors were never given the systems, the time, or the freedom to do that work. A well-trained LLM could. So they were stuck selling me products I didn’t need, or telling me they couldn’t sell me products they didn’t think I needed — while the tool could look at the broader picture of all that data and help me understand where to look next.
That’s really the difference, and it’s not about the tool being smart. It’s about what it hands back to you. It’s when your healthcare becomes more than a set of products — when it gives you agency, gives you the ability to ask questions, the ability to discover things about yourself and your own experience. That’s where you start to end up with power.
I’m not saying LLMs don’t have biased data. Out of the box, an AI is trained on the whole corpus of human writing. Humans are biased, and it picks those biases up. But two things are true at once. The first is that a human carries a bias a model doesn’t. A doctor has bad days. They get angry, they have fights with their spouses, they’ve got long-standing feuds with nurses — I don’t know how many nurses I’ve talked to who have issues with the doctors — and all of that plays into the bias. One bad day for a doctor means a really, really bad outcome for a patient. And when you question the specific error, you don’t get curiosity, you get rank: I’m the doctor, I’m the one who’s right. The second thing is that a model’s bias, unlike a bad day, can be corrected. Researchers at the University of Michigan built a debiasing step that accounts for which patients had been systematically undertested, instead of treating the skewed records as ground truth — and once they applied it, a model trained on biased data performed as well as one trained on a perfectly unbiased set that doesn’t exist in the real world. You can’t run that correction on a doctor in the middle of your appointment. So I’m not saying the tool is unbiased. I’m saying its bias can be caught and fixed, and a human’s very often can’t.
None of this works, though, until we deal with privacy — because all of it runs on your data. Right now most people are on cloud tools, often the free version, where the conversation is used to train the model. And most people don’t understand what that means.
What has to change is on us
So here’s what actually has to change. We need better education, better tools, and better policy — and all three are on us, the public, to demand.
Education first. People need to understand how these tools use their data — not a CS degree, not how transformers and inference work, but enough to reason about their own privacy, and enough to know that the context you give and the way you ask changes the answer you get. That includes the difference between a model that’s just handing you answers that feel good and one that pushes you toward better ones.
Then the tools themselves, built around the patient. That starts with the system prompt. Without a well-designed, structured system prompt built around patient outcomes — and not healthcare-institution outcomes and profitability — we don’t get to a point where using an LLM is actually helpful for patients. The set of rules I built for my own case was designed to push back on me, to make me answer its questions, to challenge ideas, to log what changed as my symptoms changed over time. That’s the difference between a tool that flatters you and one that helps you. It also means access to frontier models built for this kind of work — huge context, real indexing — and this needs to be a public institution or well-governed nonprofit so the incentive isn’t profit.
Then portability. Right now I can get PDFs out of my provider, but it’s a pain — it took my wife a full day to get the information she’d need to pass my blood tests, my sleep tests, all of my analysis along to a system that could give me better outcomes. That kind of portability is essential, and it’s not an AI question. If you ask for a file, it should come out in a standardized, open format that handles different kinds of data — not a locked PDF. Honestly, if healthcare providers just exposed a secure connection with a minimum set of well-structured information, it’d be immediately usable by the frontier models that exist today. Yes, it’s a matter of managing HIPAA and privacy, but it’s not like we don’t already handle logins and two-factor authentication for a lot less. Making that information portable, clean, and open, in a way that allows for patient autonomy, is probably the most important thing we could build into our healthcare systems right now.
And the choice about your data has to be yours. I realize there are real questions about HIPAA compliance and privacy, and that letting people get at their own information feels like opening a can of worms. But in some ways that’s not the government’s choice. The choice should be mine — what I get to do with my data, whether that’s an LLM, another doctor, or another healthcare system. Until that’s available, we’re foreclosing on the autonomy and the power of patients to affect their own outcomes.
The other thing that belongs to you is the conversation itself. The litigation environment around doctors and patients is so bad that about half the time, doctors tell me I’m not allowed to record my conversations with them. Here’s why that matters. I’ve had conversations with doctors where they told me to take a literally lethal dose of an over-the-counter medicine. I asked, more than once — are you sure, because that doesn’t sound right to me. And they, in that doctor-knows-best way, said no, this is right, I know it’s right, trust me. I only found out how dangerous it was because I had the LLM look at that conversation, and it stopped and said: this is dangerous, don’t do this. Then I confirmed it myself. The prescription was random, delivered with total confidence, and wrong.
So the right to record and process your own conversations should be built into every doctor’s visit. I know there are reasons doctors avoid it, and some of it is about liability, and there may need to be some reformation of law to control for out-of-control and spurious litigation. But 81% of Americans already say they want to be told when a doctor is using AI to make a diagnosis — which is the same principle from the other direction: I’m entitled to know what happened in that room. Without access to that conversation and control over it, I can’t do the due diligence it takes to understand my own medical situation.
Through the tool I ended up putting together, I’ve been able to understand the dietary needs and the environmental changes that are, so far, the only things that have made any real difference in my health. I’m still working through the details, and I don’t have an official diagnosis — I’m not going to oversell that. But I’ve been able to improve my work life. I’ve made accommodations for myself that let me stay productive, stay engaged with my family, take care of my wife, and understand how my finances are going to change over time as my health does. I’ve made informed choices off of data I actually controlled — including catching a couple of medications that turned out to be absolutely toxic for the diagnosis I most likely have. I’ve been able to meaningfully improve my life because of my access to this tool.
None of this is about saying doctors aren’t useful. No. What it’s saying is that modern society has made it hard for them to do the job well — and we still need good research, good study, good clinical experience; we need the older doctors who can tell you what they’re seeing and the new ones who show up with fresh eyes. It’s a story about who holds the context, and so about who holds the power.
And it isn’t really an American story, even though the American system shows it worst. Until we take the profit motive out of the model it’s going to stay a problem, and we’re not going to, because that’s where we live. So we fight for something else — something that gives us more power, more agency. Everything I’m asking for can be done under any economic system. Better education about how these tools work. A private frontier model built for huge context and real indexing. Policy that says your data has to come out in an open, portable form. You could do it in England, in Germany, in Kenya, even under a government I wouldn’t otherwise trust with it — and the same moves would hand the same power to the same people.
Because in the end, alignment isn’t just an LLM thing. It’s not just what a model does in the absence of an obvious answer. It’s what we do with one another, and with the tools we already have. And the question worth asking — the one it took me six undiagnosed years to get to — is a simple one: how do I get some power over my own health, and help the people around me get some over theirs?
Further Reading
The research behind this piece, plus a few things I read that didn’t make it in but are worth your time.
On how people actually use AI for health
- From Diagnoses to Treatments: Why Americans Use AI Chatbots for Health — Pew Research Center, Aug 2026. The report on the third of Americans now using chatbots for health, and how helpful they find it.
- Where Do Americans Get Health Information, and What Do They Trust? — Pew Research Center, Apr 2026. The “convenient, not accurate” findings on chatbot and social-media health info.
- Americans Want Transparency When AI Is Used in Their Healthcare — Pew Research Center, Aug 2026. The 81% who want to be told, and the 53% who feel they have no say.
- Do Americans Think Chatbots Help or Hurt People Using Them for Loneliness, Depression or Stress? — Pew Research Center, Aug 2026. The companion piece on public skepticism I mention in passing — worth reading if you think everyone’s blindly trusting these tools.
On what the system produces
- Mirror, Mirror 2024: A Portrait of the Failing U.S. Health System — Commonwealth Fund. The ten-country comparison behind “last of ten while paying the most.”
- View of U.S. Healthcare Quality Declines to 24-Year Low — Gallup. The 71%-own-care / 44%-system split.
- How U.S. Health Spending Compares to Other Countries — Peterson-KFF Health System Tracker. The per-capita spending gap, if you want the underlying numbers.
On diagnostic and prescribing error
- Public Health Impact of Serious Harms from Diagnostic Error in the U.S. — Johns Hopkins Medicine. The 795,000 figure and the $100-billion cost.
- Diagnostic Errors Are Common in Seriously Ill Hospitalized Adults — UCSF, 2024. The study finding nearly a quarter of seriously ill hospitalized patients had a delayed or missed diagnosis.
- Medication Errors and Adverse Drug Events — AHRQ Patient Safety Network. Background on prescribing error, the category the lethal-dose story falls into.
On AI, bias, and what actually works
- Accounting for Bias in Medical Data Helps Prevent AI from Amplifying Racial Disparity — University of Michigan Engineering, 2024. The debiasing result — the “boring fix” the piece leans on.
- Reliability of LLMs as Medical Assistants for the General Public — Bean et al., Nature Medicine, 2026. The controlled study: near-perfect alone, under 35% once real people supplied the information.
- Large Language Models Propagate Race-Based Medicine — Omiye et al., npj Digital Medicine, 2023. If you want the honest counter-case: the study showing models reproducing debunked race-based claims out of the box.
- Why AI-Analyzed Medical Images Can Be Biased — MIT News, 2024. How imaging models use demographic shortcuts.
On the mechanics under the hood
- Healthcare Common Procedure Coding System (HCPCS) — CMS. The billing-code system itself, if you want to see the machinery the whole first half is about.