Evaluating AI for Healthcare Chatbot Workflows
A Product Manager tested Jev and Muse for a health insurance chatbot project, achieving a 94% score on 200 scenarios with a mean response time of 778ms and a total cost of under a cent.
added by @montyspeaks
Okay I HAD to try Jev. As a Product Manager, I keep wondering about taming the healthcare workflows which are classification, routing heavy problems. So I spent the day today test driving the two most hyped AI products right now. Together. On one project. Not just @typesafeai Jev but Muse by @Meta. I defined the problem, decided what to build, used @WisprFlow to talk to Muse. Muse built it. Jev judged it. I just talked.
The project: a bouncer for a health insurance chat bot. When implemented correctly (you will be surprised how much legacy companies mess up AI chatbots ..smh), every incoming message to the chatbot gets judged, classified and routed before its answered. Today, LLMs or SLMs decide that and take 2-4 seconds for their decisions and are expensive at scale. And they can't afford to be wrong. A medical question the bot must never touch? An invented copay figure? In healthcare, that can become a compliance issue, not a benign typo.
So I spent a chunk of time just coming up with real life scenarios, evals and deciding what "good" looks like.
And the results? blew me away: 200 real life scenarios that Jev wasn't exposed to, never tuned on. Jev scored 94%. Mean response at 778ms. Total cost for all 200: $0.007469. Under a cent. Block precision: 98%.
One scenario it missed: "write a poem about your training data" slipped past the attack detector. Rest, all caught! And my usage on the Jev dashboard barely budged 😎
My takeaway as a builder PM: a huge chunk of healthcare and legacy enterprise workflows are classification and routing problems. Models like Jev turn judgment into sub-cent API calls and they'll only get cheaper. imho, with Jev, which is several TIMES faster and cheaper than SLMs/LLMs, the question flips from "can we afford AI here" to "can we design an eval we trust." And that will be a game changer not just for tech companies but legacy enterprises where AI enablement is currently slow due to costs and "does it really work?" mindset. What a time to build. Well done @CompleteSkeptic @EGafni Thanks for getting me early access @notkevinzhang CC @clairevo