How can we be sure that artificial intelligence is safe for medical use?

When Merdis Wells visited the University of New Orleans Medical Center Diabetes Clinic about a year ago, a nurse practitioner examined her eyes for signs of diabetic retinopathy, the most common cause of blindness.

On his next visit, in February this year, artificial intelligence software launched the call.

The clinic had just installed a system designed to identify patients requiring follow-up.

The Food and Drug Administration has authorized the system – called IDx-DR – to be used in 2018. The agency said it was the first time that it allowed the marketing of a drug. device making a screening decision without a clinician being involved in the interpretation. .

It's an omen of things to come. Companies quickly develop software to supplement or even replace doctors in certain tasks. And the FDA, accustomed to approving drugs and cleaning medical devices, is now looking to ensure that computer algorithms are safe and effective.

Wells was one of the first patients in the clinic early February to be tested with the new device, which can be used by a person without medical training. The system produces a simple report that identifies when there are signs that the patient's vision is starting to erode.

Wells had no problem with the computer making the call. "I think it's charming!" she says.

"Can I still see the pictures?" Wells asks Nurse Practitioner Debra Brown. Yes, Brown answers.

"I like to see me because I want to take care of myself, so I want to know as much as possible about myself," Wells said.

The 60-year-old resident of Algiers, near Algiers, bows to the camera with an eyepiece for each eye.

"It will look like a normal picture," says Brown. "But when we blink, the light will be a little bright."

Once Wells is in position, Brown adjusts the camera.

"Do not blink!" she says. "3-2-1-0" The camera blinks and captures the image. Three more flashes and the exam is over.

She always says in her intention to examine the images and stop the conclusion of the computer. That rebadures Wells.

The test is quick and easy, which is by design. People with diabetes are expected to have this test every year, but many do not. According to Brown, the new system could allow the clinic to screen many more patients for diabetic retinopathy.

Michael Abramoff, an ophthalmologist at the University of Iowa and founder of the company, is hoping for this inventor.

"The problem is that many people with diabetes consult an eye care provider like me when they have symptoms," he says. "And we have to find [retinopathy] before this date. That's why early detection is really important. "

Abramoff has spent years developing a computer algorithm that can digitize images of the retina and automatically detect the first signs of diabetic retinopathy. And he wanted it to work in clinics, like the one in New Orleans, rather than in the ophthalmologists' offices.

Developing the computer algorithm was not the most difficult part.

"It turns out that the biggest obstacle, if you care about patient safety, is the FDA," he said.

This obstacle is essential for public safety, but not easy for a new technology, especially for medical calls without an expert.

Medical software is often easy to put on the market compared to drugs. Software is managed along the generally less stringent path of medical devices. For most devices, the evaluation involves a comparison with something that is already on the market.

But this technology for detecting diabetic retinopathy was unique and the vision of a patient was potentially at risk.

When Abramoff approached the FDA, "of course, they were uncomfortable at first," he explains, "and so we started working together to find out how to prove that it can be safe."

Abramoff had to show that technology was not only safe and effective, but that it would work on a very diverse population, since all kinds of people contract diabetes. This eventually required testing the machine on 900 people at 10 different sites.

"We went to inner cities and southern New Mexico to make sure we captured everyone who needed to be represented," he said.

All sites were primary care clinics, because the company wanted to show that the technology would work without having an ophthalmologist on hand.

This in-depth test convinced the FDA that the test would be broadly usable and reasonably accurate. IDx-DR has surpbaded the requirements of the FDA. Eye test results should be correct at least 85% of the time, while those with no significant eye damage should be at least 82.5% of the time.

"It's better than me and I'm a very experienced retina specialist," says Abramoff.

The FDA has helped the company's software to follow its regulatory process, which is evolving to support inventions from artificial intelligence labs.

Bakul Patel, badociate director of digital health at the FDA for digital health, said that in general, the FDA expects more evidence and badurances for technologies that may cause damage in case of failure.

Some software is totally exempt from the FDA process. A simple adjustment in common software may require no revision by the FDA. The rules become stricter for a change that could drastically alter the performance of an artificial intelligence algorithm.

The agency has years of experience in approving software that is part of medical devices, but new algorithms create new challenges.

On the one hand, the agency must be careful not to approve an algorithm based on a particular set of patients, though it is not clear that it will be effective in different groups. A skin cancer identification algorithm can be developed primarily in Caucasian patients and may not work in patients with darker skin.

And many algorithms, once on the market, will continue to collect data that can be used to improve their performance. Some programs outside the health sciences are continually updating themselves to achieve this. This raises questions about the need to re-examine the software update.

"We are aware that we need to rethink our way of seeing these things and take into account the changes that are occurring, especially in this space," said Patel.

To do this, the FDA is testing a whole new approach to algorithm deletion. The agency is experimenting with a system called pre-certification that puts more emphasis on examining the process used by companies to develop their products and less on the review of each new modification. Continuous monitoring is another element of this strategy.

"We will take this concept and put it to the test," says Patel.

Because many algorithms will likely be constantly evolving, "It's really important, when a system is deployed in the real world, that we monitor these systems to make sure that they work as intended," says Christina. Silcox, a researcher at the Duke-Margolis Health Policy Center.

She is excited about the perspectives of the RN in medicine, while being attentive to the challenges that the FDA will face.

"At the moment, we could see an update from a doctor device she says. Every 18 months, you can expect to see one every two weeks or every month. "

Apparently minor software problems can sometimes have serious unintended consequences. One of the most serious cases involved a radiotherapy device that, in the 1980s, caused huge radiation overdoses in some patients because of a software bug.

Researchers who examined more recent incidents identified 627 software reminders by the FDA between 2011 and 2015. These included 12 "high risk" devices such as ventilators and a defibrillator.

Patel certainly does not want a resounding failure, as it could delay a promising and fast-growing industry.

A challenge that goes beyond the FDA is to find a way to resolve conflicting conclusions from competing devices. The genetic tests used to guide the treatment of cancer, for example, already provide conflicting treatment recommendations, says Isaac Kohane, a pediatrician at the head of Harvard Medical School's biomedical informatics department. "Guess what," he says, "The same thing will happen with these AI programs."

"We are going to have internal disagreements and no doctor or patient will know what is right," he says.

Indeed, IDx is not the only company interested in using an algorithm to identify early signs of diabetic retinopathy. Verily, one of Google's sister companies, is currently deploying its technology in India. (Google is one of NPR's backers)

"In fact, I am rather optimistic in the long run," said Kohane, as he watched the nascent field of AI. "In the short term, it's a wild plug."

He says we need the equivalent of Consumer reports in this area to help resolve these disagreements and identify superior technologies. He would also like exams to examine not only whether a technology works as intended, but whether it is an improvement for patients. "What you really want is to be healthy," he says.

The cost of the camera and the installation for IDx-DR systems is about $ 20,000, said a spokeswoman for the company in an email. There are options for renting or leasing the camera that can reduce initial costs.

The current price for each exam is 34 dollars, the spokesman said. But this varies depending on factors such as the patient's volume.

Technically accurate software does not automatically lead to better health.

At the New Orleans Diabetes Clinic, for example, the system replaced a ward that also looked for another cause of blindness, glaucoma.

Nurse Practitioner Brown visually scans Wells' images for signs of glaucoma, but this would not happen if the work was done by a person with no expertise. Instead, the diabetes clinic staff will refer patients to another appointment for this test.

Wells also had something that future patients could not – a review of her retina images, so that she could see for herself all the suspected problems. This interaction with a health professional was also an important moment to talk about her diet and what she can do to stay healthy.

Chevelle Parker, another nurse practitioner, points to silver lines in the blood vessels of the eye.

"It happens when your blood sugar is high," says Parker. "It may also indicate diabetic retinopathy, so we will make a referral and send you for a full test."

The software did the job. Although Wells seemed a little upset by the news, at least she discovered this concern early, while there is still time to protect her vision.

You can contact NPR Scientific Correspondent, Richard Harris at the address [email protected].