Jack Brooksbank, MJLST Staffer
Artificial Intelligence (AI) is a funny creature. When we say AI, generally we mean algorithms, such as neural networks, that are “trained” based on some initial dataset. This dataset can be essentially anything, such as a library of tagged photographs or the set of rules to a board game. The computer is given a goal, such as “identify objects in the photos” or “win a game of chess.” It then systematically iterates some process, depending on which algorithm is used, and checks the result against the known results from the initial dataset. In the end, the AI finds some pattern— essentially through brute force —and then uses that pattern to accomplish its task on new, unknown inputs (by playing a new game of chess, for example).
AI is capable of amazing feats. IBM-made Deep Blue famously defeated chess master Gary Kasparov back in 1997, and the technology has only gotten better since. Tesla, Uber, Alphabet, and other giants of the technology world rely on AI to develop self-driving cars. AI is used to pick stocks, to predict risk for investors, spot fraud, and even determine whether to approve a credit card application.
But, because AI doesn’t really know what it is looking at, it can also make some incredible errors. One neural network AI trained to detect sheep in photographs instead noticed that sheep tend to congregate in grassy fields. It then applied the “sheep” tag to any photo of such a field, fluffy quadrupeds or no. And when shown a photo of sheep painted orange, it handily labeled them “flowers.” Another cutting-edge AI platform has, thanks to a quirk of the original dataset it was trained on, a known propensity to spot giraffes where none exist. And the internet is full of humorous examples of AI-generated weirdness, like one neural net that invented color names such as “snowbonk,” “stargoon,” and “testing.”
One area of immense potential for AI applications is healthcare. AIs are being investigated for applications including diagnosing diseases and aiding in drug discovery. Yet the use of AI raises challenging legal questions. The FDA has been given a statutory mandate to ensure that many healthcare items, such as drugs or medical devices, are safe. But the review mechanisms the agency uses to ensure that drugs or devices are safe generally rely on knowing how the thing under review works. And patients who receive sub-standard care have legal recourse if they can show that they were not treated with the appropriate standard of care. But AI is helpful essentially because we don’t know how it works—because AI develops its own patterns beyond what humans can spot. The opaque nature of AI could make effective regulatory oversight very challenging. After all, a patient mis-diagnosed by a substandard AI may have no way of proving that the AI was flawed. How could they, when nobody knows how it actually works?
One possible regulatory scheme that could get around this issue is to have AI remain “supervised” by humans. In this model, AI could be used to sift through data and “flag” potential points of interest. A human reviewer would then see what drew the AI’s interest, and make the final decision independently. But while this would retain a higher degree of accountability in the process, it would not really be using the AI to its full potential. After all, part of the appeal of AI is that it could be used to spot things beyond what humans could see. And there would also be the danger that overworked healthcare workers would end up just rubber stamping the computer’s decision, defeating the purpose of having human review.
Another way forward could be foreshadowed by a program the FDA is currently testing for software update approval. Under the pre-cert program, companies could get approval for the procedures they use to make updates. Then, as long as future updates are made using that process, the updates themselves would be subject to a greatly reduced approval burden. For AI, this could mean agencies promulgating standardized methods for creating an AI system—lists of approved algorithm types, systems for choosing the dataset the AI are trained on—and then private actors having to show only that their system has been set up well.
And of course, another option would be to simply accept some added uncertainty. After all, uncertainty abounds in the current healthcare system today, despite our best efforts. For example, Lithium is prescribed to treat bipolar disorder, despite uncertainty in the medical community of how it works. Indeed, the mechanism for many drugs remains mysterious. We know that these drugs work, even if we don’t know how; perhaps using the same standard for AI in medicine wouldn’t really be so different after all.