Building a Search Engine for Unstructured Clinical Trial Data

Our client, a large European pharmaceutical company, recognised the value of reusing data from previous clinical trials. However, they found it difficult to access due to the large amount of unstructured data and the multiple departments involved. Building an internal search engine specifically for their statistical outputs helped make this data more easily accessible and improve the efficiency of future clinical trials.

The Challenge

Our client is a rapidly expanding multinational pharmaceutical company. As a result, much of their data spans both pre- and post-digitalisation eras.

The BioStat Department within the company plays a crucial role in the drug development process by analysing clinical trials. To streamline the creation of new trials, the department leverages historical data from previous studies. Despite this, coordinating with multiple departments leads to a significant volume of unstructured data, such as texts and images.

Additionally, the sensitive nature of the data restricts employee access, hindering the sharing of knowledge across departments. This poses a challenge for statisticians who wish to reuse codes and plots, as they must go through a lengthy approval process. To address this issue, we developed an internal search engine for statistical outputs, facilitating the sharing and reusing of information.

The Approach

Our design team at ADC Denmark initiated testing and developing the tool through an iterative design thinking process, repeatedly engaging the end-users. This included several departments that use the data to inform their work, such as management, medical writing, and regulatory affairs. By employing a design thinking, user-centric approach, we ensured that the final solution met the needs of all relevant users.

Subsequently, our data science team used the prototype to guide the development of the tool. We employed an iterative approach, with regular input and feedback from end-users throughout the process. This ensured that the tool was intuitive and met the needs of the users. We continuously asked questions like: Does it feel intuitive? Is something missing? How can we create more added value? The final tool was tailored to match the design criteria and desired features specified by the end-users.

Furthermore, our client’s internal programmers were actively involved in the development process. To enhance their skills, we conducted workshops, including training sessions on working with various databases and efficient Python coding. This helped ensure that the internal team was equipped to maintain and further develop the tool in the future.

The Solution

The result of our efforts is an internal search engine, named FACT: Fast Access to Clinical Trials, hosted on Amazon Web Services (AWS). The tool has four key components:

A user-friendly front-end that was developed using Javascript (Vue.js), with a similar appearance to the Google search engine. Users can perform advanced searches to quickly locate specific treatment areas, time periods, or trial phases. Additionally, the tool includes the option to filter and view different elements, such as images, through quick filters.
The backend consists of an elastic search database, which immediately enables many NLP features, such as creating synonyms in the data that consider abbreviations or mistyped words, increasing the tool’s ability to find the correct data.
An API built in Python that helps the front-end connect to the back-end.
A data extraction engine build in AWS and Python that extracts relevant information from various formats and sources on the shared drive, such as plot images, Microsoft Word Documents, PowerPoints, and PDF files. Extracting the relevant text and saving it in the database makes it easily accessible.

Impact on the Client

Our client now has a tool that simplifies the process of finding accurate information, streamlining the workflow of its employees. The tool allows employees to search through all documents, outputs, and programs in one centralised location, eliminating the need to search through multiple datasets and formats across different departments.

Furthermore, by providing access to the codes used to generate plots for previous clinical trials, it reduces the time spent on duplicating work and improves the process of analysing clinical trials. The tool was met with high demand within the organisation, with more than one thousand searches in its first week of launch.

Learnings

The tool was designed to anonymise and aggregate personal data, providing the results presented through the search function. However, during the development process, we identified that some of the data was not completely anonymous. Subsequently, we worked closely with the pharmaceutical company’s legal team to rectify this issue.

It is essential to consult legal experts before starting any similar projects to avoid any potential setbacks, such as the need to make changes to the tool after it has been built. This not only saves time and resources but also ensures that the final product is compliant with the legal requirements.

In a data-driven organisation, the ability to share knowledge, especially when dealing with unstructured data, is crucial for improving efficiency. Our team at ADC can help your organisation gain valuable insights from your unstructured data, including data found in Microsoft Word and PowerPoint documents, which can be difficult to search through. We can assist you in creating tools like FACT to streamline your data management process.

Let’s shape the future

Are you interested in discovering how ADC can assist your life science organisation in gaining valuable insights from unstructured data? Our team has extensive experience in developing solutions that help streamline data management processes. To learn more, please contact Andreas Kjær.

Send email