There has been tremendous buzz about ChatGPT and how it can disrupt industries. ChatGPT is developed by OpenAI and has made it available to the public for free in some capacity and for a fee for more unrestricted uses. ChatGPT is categorized as a LLM (Large Language Model) and is pretrained on massive amounts of general data then fine tuned for a variety of tasks. Generally speaking, it’s an Artificial Intelligence model that is skilled at text generation. We will explore what ChatGPT can do and if it can help ThinkGenetic in finding patients at risk for a genetic condition.
How it works
To interact with ChatGPT one can use prompt engineering with ICL (In Context Learning) where you can quickly instruct the model how to perform a task. The tasks can vary wildly from debugging code, writing a poem, to playing tic-tac-toe. You can even get ChatGPT to write a legal brief, although it’s not recommended as one lawyer recently found out. What OpenAI has done isn’t necessarily groundbreaking in of itself, but they have managed to democratize it like no one else.
What does this mean for clinical tasks?
It’s already been well established that having a domain specific model outperforms a generalized model. However, ChatGPT is new and it’s a LLM, so does that still apply? A team of researchers pondered this very question in Do We Still Need Clinical Language Models?. They took it a bit further to also look at the amount of compute required to pretrain and perform inference.
“…. using in-context learning with extremely large language models, like GPT-3, is not a sufficient replacement for fine-tuned specialized clinical models. These findings highlight the importance of developing models for highly specialized domains such as clinical text.”
These findings confirm that it is still computationally cheaper and more accurate to use a smaller domain specific model.
What if someone created a ClinicalGPT?
A ClinicalGPT would check the box for having a domain specific model raising the accuracy. It would not only be beneficial to ThinkGenetic but others in the healthcare space as there are little to no available models that are pre trained on medical records. One reason for this is due to the risk of data leakage of PHI that it’s trained on. Model cost would have to be considered. The problem we face is finding the needle in the haystack. For every one patient we find at risk for a genetic condition, we have scoured hundreds of thousands of patients. As an example AWS Comprehend for SNOMED CT concepts for 35k charts would cost $12k. Unfortunately not everything we are looking for falls within a SNOMED code, so we would still need another method of finding what we are looking for. We would also need to consider the performance of each type of note. To name a few, we encounter discharge summaries, call logs, email logs, triage notes, and clinical notes. Each note has a different flow/ structure as well as each person has a different style of creating sentences. This all influences the performance of a model.
What about the next thing?
At ThinkGenetic, we will continue to use modern software design patterns to ensure our products are flexible to accommodate the next generation of whatever. We employ a modern, stable, and componentized Docker based technology stack. Docker allows us to quickly pivot by replacing a pipeline component, scaling up and down, or moving our technology from laptop to server or from cloud provider to cloud provider. Our clinician designed algorithms are also implemented in a technology agnostic language. This gives us the ability to translate the algorithm into any search platform now or in the future.
The one guarantee in the technology space is that its progress can evolve fast and not always at a constant pace.
