Secrets of the CMS AI Health Outcomes Challenge winner
The AI ââindustry has learned a lot from the competitions supported by the US government. DARPA (Defense Advanced Research Projects Agency) has supported a number of âGrand Challengesâ in autonomous vehicles and robotics, starting in 2004. They have had a great impact on these areas and have clearly contributed to taking them forward. There is even a “DARPA Subterranean Challenge” taking place in a cave near Louisville, KY this month. I’m sure our defense establishment has a good reason for wanting to build autonomous underground vehicles and robotic capabilities, but I’m not sure what that is.
But one smart person realized (I doubt it was Donald Trump, but it was during his administration) that the defense industry was not the only one who could benefit from competition involving AI. . In March 2019, the US Centers for Medicare and Medicaid Services (CMS) announced the âArtificial Intelligence Health Outcomes Challengeâ. It was sponsored by a public / private partnership of the CMS Innovation Center, the American Academy of Family Physicians and Arnold Ventures. The challenge was to use AI to predict an ambitious variety of health outcomes, including mortality, unplanned admissions related to heart failure, pneumonia, COPD, and various other high-risk conditions; and thirteen different adverse events such as nosocomial infections, sepsis and respiratory failure.
The challenge ended in April 2021 after several rounds of competition among more than 300 participants. There have been a number of learnings from the Global Program, which you can review here. But I was interested in speaking with the winner of the challenge, an Austin-based company called ClosedLoop. This 30-person startup has beaten some of the biggest and most sophisticated consultants in the AI ââindustry, including my former employer Accenture and my current employer type Deloitte, as well as some sophisticated vendors like Geisinger (who arrived second) and Mayo Clinic. So here’s what I learned about ClosedLoop’s secret sauce to winning the challenge, much of which will be invaluable to other organizations pursuing AI.
1. AI platforms must be vertical. ClosedLoop developed a âHealth Data Science Platformâ and âHealth Use Case Libraryâ that they used to tackle the challenge and that their clients can use to solve their own AI problems. It is geared towards health data, HIPAA health regulations, typical health issues, and more. It seems to me that for most use cases it would be much faster to start with an industry specific platform than a generic platform. There are other examples of this (eg C3’s manufacturing-oriented platform), but not enough. I predict more will come and it will be good for companies interested in AI.
2. “The first job is to sort the list better”– That’s a quote from Andrew Eye, CEO of ClosedLoop. I’ve often found this to be true with AI – that its greatest value is in prioritizing a to-do list (sales calls to do, supply chain issues to resolve). In the case of healthcare, it may be either to identify the patients most at risk and requiring a type of intervention, or for an individual patient, which medical problem creates the greatest risk and needs to be addressed first. For busy clinicians, sorting the list is a way to focus on what really matters.
3. Explainability is essentialâAs models predict the likelihood of health risk with machine learning, a common outcome is a risk score. But at ClosedLoop, they say, “Doctors tell us someone has to hand me over 20.” One of the physicians they worked with on the challenge, Dr. Jim Walton (CEO of Genesis Physicians Group, a large organization of independent physicians based in Dallas) likes to say, âThis is not about the model. , but from the patient. AI-based recommendations need to provide context, reasoning, and prioritization – as Walton also says, doctors love anecdotes and stories, and AI needs to provide them with a story.
4. User interface matters tooâOne aspect of the challenge was the ability to convey analysis and recommended actions clearly in a visual format. ClosedLoop tried 14 different versions before choosing one. Andrew Eye commented on the division of labor between machines and humans: âOur job is to predict the future, the doctor’s job is to change it. We are planning falls, but someone else is putting up guardrails. If the predictions are not presented in a way that clearly motivates action, they are not of much value.
5. “More data beats better algorithms”– The old quote from Peter Norvig (research director at Google) also applies here. Executives at ClosedLoop said their models worked well in predicting health outcomes largely thanks to the data they used. They used fourteen available data sources in their challenge models. Even with that large number, they are proponents of data parsimony, arguing that structured medical data, typically in claims databases and electronic medical records, is generally the best source of predictions. They value external source data, but not until the easier to obtain internal data is exhausted. Regarding algorithms, although they used complex deep learning models in their final submission to the challenge, they say they generally don’t see huge performance gains from them (they might make an exception. for the radiological image data, but we did not discuss it).
6. Remember the workflowâBig digital capabilities in healthcare are often not used much because they don’t match the way clinicians do their jobs. But ClosedLoop is very focused on integration into the clinical workflow. ClosedLoop CTO and Co-Founder Dave DeCaprio said in a press release (I haven’t spoken to him): âOur Patient Health Forecast (PHF) was critical in winning the challenge. We redesigned the whole concept into a comprehensive, personalized risk forecast that could be delivered directly into a clinical workflowâ¦ Each forecast highlights key variables and explains precisely how they contribute to a patient’s specific risk.
You can argue that none of these maxims are terribly secretive or scientific, and I should agree. But I think ClosedLoop won by not just focusing on the power of their AI models (although they are very predictive), but on the larger picture of how AI can help clinicians. with their patients. Hopefully this big lesson is the key the healthcare AI industry takes away from the challenge.