
Large Language Models (LLMs) like GPT-4 are changing how we think about AI's role in medicine. One key example is how well these models have performed on the United States Medical Licensing Examination (USMLE), a tough test for medical graduates. This success shows us that AI can understand and use medical knowledge in ways similar to human doctors.

In this blog post, we're going to look at what this means for the future. We'll focus on the results of GPT-4 and talk about how AI could help doctors. This includes making diagnoses more accurate, improving patient care, and making medical tasks more efficient. Our goal is to understand how AI, especially through models like GPT-4, might work alongside doctors to make healthcare better.

## **What is USMLE?**

The United States Medical Licensing Examination (USMLE) is a three-step examination for medical licensure in the United States and is one of the most critical milestones in a medical student's career. It assesses a physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that are important in health and disease and constitute the basis of safe and effective patient care.

![](</blog/uploads/DALL·E 2024-03-06 21.18.52 - Create a wide image of an emoji with a focused expression, surrounded by medical books, notes, and a laptop displaying medical charts. The emoji is we.webp>)

### Example Question and Analysis

Let's dive into an example question that illustrates the complexity and depth of knowledge required to succeed in the USMLE exams. This type of question demonstrates why AI might struggle, as it requires not only factual knowledge but also the application of that knowledge in a clinical context, understanding of patient history, and critical thinking to arrive at the correct diagnosis and treatment plan.

Question: A 25-year-old woman comes to the physician because of a two-month history of joint pain in her hands. She reports that the pain is worse in the morning. Physical examination shows swelling and tenderness in the metacarpophalangeal and proximal interphalangeal joints. Which of the following is the most likely diagnosis?

A) Osteoarthritis

B) Rheumatoid arthritis

C) Psoriatic arthritis

D) Gout

Answer: B) Rheumatoid arthritis

Explanation: The patient's symptoms of pain in the hands that is worse in the morning, along with physical findings of swelling and tenderness in the metacarpophalangeal and proximal interphalangeal joints, are characteristic of rheumatoid arthritis (RA). RA is an autoimmune disorder that typically affects the small joints in a symmetrical pattern.

Possible Mistakes: Choosing A) Osteoarthritis would be incorrect because osteoarthritis typically affects older individuals and involves the distal interphalangeal joints, not the proximal ones.

C) Psoriatic arthritis could be misleading if one focuses solely on joint pain without considering the absence of psoriasis or the typical asymmetric and distal joint involvement.

D) Gout usually presents with acute onset and affects the big toe (podagra) initially, not the hands.

This question requires an understanding of disease mechanisms, patient symptoms, and diagnostic criteria, areas where AI might not fully grasp the nuances of human medical judgment or the subtleties of patient presentations. While AI can store and retrieve vast amounts of information, the clinical reasoning and empathy involved in medicine, especially in interpreting complex cases and understanding patient experiences, remain uniquely human traits.

Well, at-least that's what we assumed until recent developments 😲

## **Capabilities of GPT-4 on USMLE**

Researchers did a comprehensive evaluation of GPT-4 on medical competency exams and benchmark datasets revealed significant advancements in the capabilities of large language models (LLMs) in the medical domain. Here's a summary of the key statistics and comparisons, including model performances and human benchmarking where available:

### **GPT-4 achieved 86.65% on USMLE**![](</blog/uploads/WhatsApp Image 2024-03-06 at 20.47.25.jpeg>)

* GPT-3.5 got 53.61% and 58.78%
* GPT-4 significantly outperforms its predecessor, GPT-3.5, and specialized models such as Flan-PaLM 540B and Med-PaLM on the United States Medical Licensing Examination (USMLE).
* The passing threshold for USMLE exams is approximately 60% correct on multiple-choice questions, indicating that GPT-4 not only surpasses this threshold but also exhibits a performance that significantly exceeds the average human performance.

The full evaluation can be found here:  [https://arxiv.org/pdf/2303.13375.pdf](https://arxiv.org/pdf/2303.13375.pdf)

### **Performance Limitations**

1\. Questions Involving Images and Media:

GPT-4's (text-only version) was evaluated on USMLE questions, some of which relied on images. While GPT-4 still performed well on these questions despite not having access to the visual media, it is reasonable to infer that its performance on these types of questions might not be as robust as on text-based questions. The paper notes that GPT-4 was able to use logical reasoning to answer questions intended to be answered with visual aids, but this indirect method could potentially lead to errors in cases where visual interpretation is critical.

2\. Calibration and Probabilities:

The paper highlights GPT-4's improved calibration over previous models, indicating it has a better understanding of when its answers are likely to be correct. However, the need to emphasize calibration suggests that there might still be issues with overconfidence or underconfidence in its answers. In high-stakes fields like medicine, any misestimation of confidence could lead to incorrect conclusions or recommendations, which can be seen as a form of failure.

3\. Memorization Concerns:

The researchers investigated whether GPT-4's performance could be attributed to memorization of exam content. They found no evidence of direct memorization, indicating its performance is likely due to understanding rather than recall. However, the paper discusses the potential for memorization or leakage from training data as a limitation of LLMs, implying that if GPT-4 had relied on memorization, it could fail in situations requiring novel application of knowledge or when faced with content it hadn't been exposed to during training.

4\. Richer Prompting Strategies:

The paper also mentions the exploration of more sophisticated prompting strategies, such as chain-of-thought prompting, which did not yield significant performance benefits for GPT-4 in this context. This might suggest that while GPT-4 is highly capable, there could be specific types of reasoning or problem-solving where its approach is not optimized, leading to lesser performance.

### **Implications for Clinical Practice**

The implications of ChatGPT's USMLE performance extend far beyond the realm of academic testing. As healthcare seeks to embrace AI, ChatGPT's capabilities hint at a future where AI can:

* Reduce Administrative Burden: Automating documentation and other time-consuming tasks, allowing clinicians to focus more on patient care.
* Enhance Diagnostic Accuracy: Offering second opinions and alerting clinicians to potential diagnostic and treatment errors.
* Personalize Patient Care: Analyzing vast datasets to tailor treatments to individual patient profiles, potentially improving outcomes.
* Facilitate Continuous Learning: Acting as an ever-ready study and reference tool for healthcare professionals at all stages of their careers.

## **Conclusion**

As we navigate the intersection of AI and healthcare, GPT-4's remarkable performance on the USMLE illuminates a path toward a future where AI not only complements but significantly enhances medical practice. Despite facing challenges with visual questions, calibration accuracy, and the intricacies of human reasoning, the potential of AI to alleviate administrative burdens, improve diagnostic precision, and personalize care is undeniable. By judiciously integrating AI tools like GPT-4 alongside human expertise, the medical community can harness these advancements to enrich patient care, optimize clinical workflows, and foster a learning environment that keeps pace with the rapid evolution of medicine. This journey requires careful consideration of the limitations and ethical implications of deploying AI in healthcare settings, ensuring that as we step into this new era, we do so with the goal of enhancing the well-being of patients and empowering healthcare professionals worldwide.



In today's rapidly advancing technological landscape, artificial intelligence (AI) has emerged as a powerful tool in various industries, including healthcare. AI holds the promise of revolutionizing medical practices, from diagnosis to treatment, with its ability to process vast amounts of data and identify patterns that human minds might miss. However, despite its potential benefits, many doctors remain skeptical and hesitant to fully trust AI. In this blog post, we'll explore the reasons behind this reluctance and the complex relationship between doctors and AI.

### Lack of Understanding

One of the primary reasons doctors may be wary of AI is a lack of understanding about how it works. AI algorithms often operate as "black boxes," meaning the reasoning behind their decisions can be opaque, even to those who develop them. Doctors, who are accustomed to making diagnoses based on tangible evidence and clinical reasoning, may struggle to trust AI systems whose inner workings they cannot fully comprehend.

### Fear of Replacement 

The fear of being replaced by machines is a common concern among professionals in various fields, and medicine is no exception. While AI is not poised to replace doctors entirely, there is a concern among some healthcare professionals that the widespread adoption of AI could diminish the role of physicians, particularly in tasks related to diagnosis and decision-making. This fear can lead to resistance and skepticism towards AI technologies.

### Reliability and Accuracy

Despite advancements in AI technology, concerns about the reliability and accuracy of AI-driven diagnoses persist. Doctors may question the validity of AI-generated recommendations, especially if they have encountered instances of incorrect or misleading outputs. Trust in AI requires a high level of confidence in its ability to deliver accurate and reliable results consistently, which may be lacking in some current implementations.

### Ethical and Legal Considerations

The integration of AI into healthcare raises complex ethical and legal questions regarding patient privacy, data security, and liability in case of errors. Doctors must navigate these challenges while adhering to strict ethical standards and regulatory requirements. The uncertainty surrounding the ethical and legal implications of AI in medicine can contribute to doctors' hesitancy to fully embrace its use.

### Impact on Patient-Doctor Relationship

Medicine is as much about empathy and human connection as it is about clinical expertise. Some doctors worry that increased reliance on AI could erode the patient-doctor relationship by depersonalizing care and reducing the role of intuition and empathy in medical decision-making. Maintaining trust and rapport with patients is essential in healthcare, and doctors may be hesitant to adopt technologies that could potentially disrupt this bond.


While the relationship between doctors and AI is complex and multifaceted, it's essential to recognize that AI has the potential to augment rather than replace human expertise in medicine. By addressing concerns related to transparency, reliability, ethics, and patient care, healthcare professionals can work towards fostering trust in AI technologies and harnessing their full potential to improve patient outcomes.

The reluctance of doctors to trust AI stems from a combination of factors, including a lack of understanding, fear of replacement, concerns about reliability and accuracy, ethical and legal considerations, and the potential impact on the patient-doctor relationship. Overcoming these barriers requires collaboration between technologists, healthcare professionals, policymakers, and patients to ensure that AI is deployed responsibly and ethically in healthcare settings. With careful consideration and collaboration, AI has the potential to revolutionize medicine and improve patient care while preserving the essential role of doctors in the healthcare ecosystem.

We'd love to hear how you feel about AI, and it's place in the future of healthcare. 





## What's Behind Doctor Burnout?

Feeling burnt out is sadly a common experience for many doctors, marked by feeling drained, disconnected, and like they're not making a difference. It's caused by lots of things, like working super long hours, feeling the pressure of the job, and dealing with a ton of paperwork thanks to clunky healthcare software.

## How Do Current Tools Contribute?

Imagine using software from the '90s to do your job today – that's the reality for many doctors. These old-school tools like Electronic Health Records (EHRs) and patient portals are more trouble than they're worth, making simple tasks way more complicated than they should be and eating up precious time that could be spent with patients.

## Can Modern UI and AI Make a Difference?

Thankfully, there's hope on the horizon. By bringing in smarter software with slick interfaces and AI smarts, we can make life a lot easier for doctors. Imagine if paperwork could just do itself, leaving doctors free to focus on what really matters – helping patients feel better.

## Combatting Burnout: Strategies for Doctors

Besides relying on fancy tech, there are other ways doctors can beat burnout. Taking care of themselves by staying active, getting enough rest, and finding ways to relax is super important. Setting boundaries, sharing the workload, and leaning on friends and colleagues for support can also make a big difference.

## The Urgency of Addressing Burnout

Burnout isn't just a problem for doctors – it's bad news for patients too. Tired doctors are more likely to make mistakes and provide less-than-stellar care, which isn't good for anyone. Plus, it's a big factor in why there aren't enough doctors to go around, making it even harder for people to get the care they need. That's why it's crucial we tackle burnout head-on and make sure doctors feel supported and valued in their work.

In the world of healthcare tech, exciting new tools like Notewand are popping up. Notewand acts like a super-smart assistant, listening in on conversations, summarizing important details, and even offering helpful insights – all through a slick, modern interface. It's a game-changer for busy doctors, helping them spend less time on paperwork and more time doing what they love – helping people feel better.


What Does GPT-4's 86% USMLE Success Mean for Healthcare?

Why Doctors Don't Trust AI

Doctor's Burnout: The Role of Software and the Promise of AI