Natural Language Search for Pharmacovigilance

Praful Krishna
3 min readAug 19, 2020

Once a pharmaceutical drug is in the market, its manufacturer is expected to review all news that relates to it and report any adverse events to regulatory bodies. A prerequisite element of this review is to distinguish reports that contain real adverse events from those that do not. Pharmacovigilance, as this activity is called, is a big headache for drug companies esp in the US.

Unfortunately, adverse event reports vary in the amount of information that can be provided or is known and there is rarely certainty that a reported event is adverse. What may at first appear to be an adverse event report can turn out to actually be something quite different.

For example, companies often receive reports of drug-drug interactions, or studies in which their drug has been tested alongside another, newer drug. A lot of the time these reports detail a successful study containing no adverse events at all. Other times, they do discuss adverse events, but the events are associated with a different drug owned by a different pharmaceutical company.

Indeed, less than 1% of reports pharmaceutical companies review contain actual adverse events related to their own drugs that need to be forwarded to the regulator.

What compounds this problem is scale — the world talks about drugs a lot. A single postmarket drug can generate hundreds of reports, from media articles and consumer feedback to studies conducted by doctors and health officials. And all of these need to be separated into two piles: adverse and non-adverse. That a task as simple as separating reports that do contain adverse events from those that don’t can divert the workflow of hundreds of scientists at a single pharmaceutical is mind-boggling.

There is currently a shortage of over seven million physicians, nurses and other health workers worldwide. Rather than filing an endless stream of paperwork, healthcare professionals would be much better put to use exercising the unique capabilities and skills that are in such short supply: finding cures, researching new drugs, improving people’s health.

While there is nothing pharmaceuticals can do about postmarket regulatory compliance, they can make the process of reporting adverse events less time-and-resource-expensive.

Artificial Intelligence technologies have been making fantastic progress in the healthcare sector in recent years, solving a variety of problems for patients, hospitals and the industry overall. Doctors can now use AI to identify skin cancer, fight blindness and diagnose sleep-loss disorders. But so far there is nothing that can help pharmaceuticals resolve the conundrum of adverse event classification.

A simple application of AI technologies like natural language search is to automatically distinguish reports that do contain adverse events from those that don’t. Reports that do contain adverse events can be flagged as requiring compliance and sent to the relevant professionals. Reports that no longer waste highly-valuable professional time.

In the realm of data, one rule is king: the more clean, reliable data available, the more accurate judgments can be reached. With such preparatory automation, pharmaceutical companies can now guarantee that they capture data on all adverse events reported, whether in the scientific literature, journals, media reports, postmarketing clinical investigations, postmarketing epidemiological studies, or even unpublished scientific papers.

We believe such an algorithm will help healthcare companies complete the regulatory filing of adverse events related to postmarketing drugs more comprehensively and faster than ever before. We also believe it will improve the quality of pharmaceutical drugs on the market, by allowing healthcare companies and regulatory bodies like the FDA quicker access to relevant information from a greater range of sources. By automating simple but time-consuming tasks like adverse event classification, pharmaceutical companies can free employees to focus on more complex and value-added work.