A GP App Endorsed By The NHS Has Been Criticised For Gender Bias

A new report by the Times has detailed a complaint against virtual GP app Babylon Health. According to the paper, the company's artificial-intelligence-based symptom checker gave a male and female patient different diagnoses, even though they reported experiencing the same symptoms.

According to the newspaper, a 60-year-old female smoker with sudden onset chest pain and nausea was told she was probably experiencing a panic attack or pain caused by inflammation. She was told to contact her GP within the next six hours if her symptoms lasted. A 60-year-old male smoker with the same symptoms, meanwhile, was advised to go to A&E with a suspected heart attack.

The headline-making news wouldn't be so critical if the app wasn't being endorsed by the NHS. Per The Times, Babylon Health has a partnership with the NHS in London and Birmingham, allowing patients to register for its virtual GP service: GP At Hand. According to Forbes, the NHS is also "encouraging two million citizens in North London to use NHS 111: Online," a Babylon app that provides a chatbot alternative to the NHS' advice line. (The symptom checker is not yet part of these services, but may well be in the future, Forbes reports.)

Health Secretary Matt Hancock has also publicly endorsed the app on numerous occasions, reports the Guardian, while the Prime Minister's senior advisor, Dominic Cummings, has been criticised for previously taking on a consultancy role for Babylon Health.

The company published a blog post in response to gender bias accusations. "We know that it looked bad. But we can explain it," Babylon Health wrote. It went on to explain that the symptom checker "is absolutely not a diagnostic tool" and is "not designed to highlight an unlikely and worst-case scenario." Instead, the company writes, "the purpose of our symptom checker is to provide medical information about conditions which match the entered symptoms."

Babylon Health said it can update the symptom checker's capabilities "to make quick and effective improvements should we come across any research bias." It admitted it has "plenty of work to do," but said it was "stubbornly optimistic that we can make some headway in stamping out gender bias and go on to lead the way for access to prejudice-free healthcare worldwide."

New research from the British Heart Foundation found that women who have heart attacks "receive poorer care than men" at every stage, from diagnosis to aftercare. The report states that women tend to get to hospital later than men, meaning they may be less likely to receive crucial early treatment. Women are also 50% more likely to be given the wrong diagnosis.

But heart-related problems aren't the only criticism Babylon Health has faced. A Forbes report, published in December 2018, stated that, before the symptom checker's launch, medical staff within the company found around 10-15% of the bot's 100 "most frequently suggested outcomes ... missed warning signs of a more serious condition like cancer or sepsis, or were just flat-out wrong." The launch was delayed, but, according to Forbes, doctors were still concerned.

In response, Babylon told Forbes that these assertions were "defamatory" and pointed to their whistle-blower policy and email address as an example of their "safety-first culture." They said that staff were "duty bound to refer any concerns formally to the authorities."

An NHS consultant, known only as Dr. Murphy on Twitter, uses the hashtag #DeathByChatbot to show further issues with the AI symptom checker. One worrying example was the case of a fictional female patient who reported a new breast lump to the bot. It decided that osteoporosis was a "moderately likely" cause, reports iNews, when NHS guidance states breast lumps should be checked by a GP.

A Babylon Health spokesperson told iNews: "In this instance, statistical data was used to correctly suggest that osteoporosis was the most likely medical issue to require attention. However, we have to do the best by our users who would be concerned by the breast lump, even though the risk of it being cancerous is low.

"We are constantly improving our user experience through user research and, just as with how we modified our wording, we are looking into whether we should address the presenting complaint first." (In February 2019, the Medicines and Healthcare products Regulatory Agency issued a notice stating that Babylon Health had changed a number of elements to address complaints.)

I have reached out to Babylon about the heart attack symptoms report from the Times, the statistics and concerns from staff reported by Forbes, Dr. Murphy's Twitter account, and the breast lump incident to see if they have any further comment on these matters. I did not hear anything back before the time of publishing, but will update you if I do.