NEST

Nurturing Excellence,
Strengthening Talent.

ABSTRACT SUBMISSION
GUIDELINES
Problem Statement – 2

Predicting actual enrolment

duration for clinical studies with
explainability
Approach & methodology
Overview Methodology Framework / tools used

• The task involves predicting the • The dataset, provided by Novartis, contains PyTorch: Employed for creating BioBERT
enrollment duration for clinical studies clinical trial information. Preprocessing includes embeddings, ensuring medically accurate feature
using both structured and unstructured handling null values, standardizing text representation for text data.
data, leveraging embeddings for columns using BioBERT embeddings for
medical accuracy, scaling numerical data, and Scikit-learn: Used for data preprocessing,
categorical features. The focus is on
encoding categorical variables. Exploratory encoding, scaling, and implementing traditional
improving prediction accuracy while regression models (e.g., Random Forest, Gradient
Data Analysis revealed key trends in sex
providing explainability regarding feature distribution, enrollment, and enrollment time. Boosting).
contributions to predictions. Robust regression models like Random Forest
and XGBoost were selected for their XGBoost and LightGBM: Selected for efficient
• The solution employs embeddings for explainability and predictive power. handling of large datasets and providing high
categorical features, feature engineering, accuracy with explainability.
and multiple regression models. These • Selected variables include 'Conditions,'
'Phases,' 'Enrollment,' 'Locations,' 'Time taken Pandas and NumPy: Utilized for data manipulation,
models are trained using k-fold cross-
for Enrollment,' and BioBERT-processed cleaning, and exploratory analysis.
validation to ensure robustness. Evaluation
metrics include RMSE, R², and SMAPE, with 'Study Title' and 'Brief Summary.' Derived
variables include categorical embeddings and TQDM: For tracking data processing and model
explainability and comparison of models for training progress.
interaction terms for 'Conditions' and 'Phases'
optimal performance.
to capture non-linear patterns.
Matplotlib/Seaborn: Used for visualizing data
• Metrics include RMSE for overall error, R² for insights.
variance explanation, and SMAPE for relative
These tools offer robust support for machine
error analysis. These metrics ensure both
learning workflows with scalability, accuracy, and
prediction precision and robustness across varied
explainability.
data distributions.

3
Model choice & setup
Model Selection Model Architecture

● Gradient Boosting Models (XGBoost, • Provide a technical flow of the end-to-end ML pipeline
LightGBM): These models handle large
datasets efficiently, manage missing
values, and capture complex feature
interactions, making them ideal for
structured tabular data.
● Random Forest: Chosen for its
robustness against overfitting and ease
of use in initial feature importance
analysis.
● Linear Models (Ridge, Lasso): Selected
for baseline comparison and
understanding linear relationships within
the data.
● AI-Powered Embeddings (BioBERT):
Utilized for generating domain-specific
embeddings to capture nuanced patterns
in medical text.
● Explainability: Tree-based models
provide interpretability through SHAP
values to ensure domain alignment with
predictions.
This approach balances performance,
explainability, and domain applicability.

4
DATA PRE-PROCESSING
DATA PRE-PROCESS DATA PREPROCESSING DATA

● Column Selection: Unused columns were removed,

retaining 'Study Title', 'Brief Summary', 'Conditions',
'Interventions', 'Sex', 'Age', 'Phases', 'Enrollment',
'Locations', and 'Time taken for Enrollment'.
● Text Columns:
○ Null values replaced with 'unknown'.
○ Whitespaces cleaned, and text standardized to
lowercase.
○ Special characters like ',' and '|' were formatted
correctly.
● Numerical Columns:
○ Null values replaced with median values.
○ Min-max scaling was applied to normalize the data.
● Categorical Columns:
○ Label encoding was applied.
○ Missing values were filled with the most frequent
value.
● Embeddings: BioBERT was utilized for generating
medically appropriate embeddings for text data.

5
Model Training & Evaluation
Evaluation Metrics

• Model Training Process

❖ Data: Structured features + pretrained embeddings for categorical variables.

❖ Scaling: Standardized features for consistent performance.
❖ Validation: 5-fold cross-validation.
❖ Models: Ridge, Random Forest, LightGBM, etc.
❖ Objective: Predict enrollment duration with explainability.

• Evaluation Criteria and Metrics

❖ Validation Metrics: RMSE, R², SMAPE to measure error

magnitude, variance explained, and accuracy.
❖ Test Metrics: Same metrics on unseen data for final evaluation.

• Discuss key performance metrics

❖ Best Model: LightGBM with 14.50 RMSE, 0.215 R², and 67.14 SMAPE.
❖ Interpretation: LightGBM provided the most accurate predictions, lowest error, and highest variance explanation, making it ideal for enrollment
predictions.
❖ Models like Random Forest and Ridge were close but less optimal.

6
Results and visualization
Model Outcomes Explainability

Key Findings: To interpret the model predictions, SHAP (SHapley

❖ LightGBM performed best with RMSE: 14.50, R²: 0.215, SMAPE: 67.14. Additive exPlanations) was employed, providing feature-
❖ Random Forest and Ridge were close competitors with slightly higher
RMSE and SMAPE. level insights into the impact on predicted enrollment
❖ Linear models like Ridge and ElasticNet showed moderate results, duration. SHAP values identified critical features such as
highlighting the importance of feature embeddings.
Conditions, Interventions, and Study Design, highlighting
Main Outcomes:
their positive or negative influence on predictions. This
❖ Best Model: LightGBM excels with balanced accuracy and explainability. ensured the model's behavior aligned with domain
❖ Importance of Embeddings: Categorical embeddings significantly
improved model performance, reducing RMSE and SMAPE. expectations, aiding trust and transparency. The
❖ Explainability Insights: Feature impact analysis ensures the model aligns embeddings extracted from categorical variables enhanced
with clinical trial domain understanding.
contextual understanding, while SHAP visualization
Implications pinpointed feature importance. These insights help refine
❖ Practical Usage: Optimized for predicting enrollment duration, LightGBM clinical trial designs by understanding restrictive or broad
provides actionable insights to streamline protocol design.
❖ Scalability: The model integrates embeddings, enabling scalability for criteria impacts. The use of SHAP ensures explainability
larger datasets. without compromising prediction accuracy, enhancing
❖ Efficiency Gains: Helps refine eligibility criteria, accelerating patient
recruitment. decision-making in clinical research.

7
Results and visualization
Model Outcomes

8
Results and visualization
Model Outcomes

9
Challenges & Next Steps
Limitations Next Steps

Caveats, Limitations, and Potential Biases If more time were available, the following improvements could be made:

1. Data Imbalance: The dataset has a skewed sex distribution, 1. Frontend Application with API Integration: Develop a user-
with a significantly higher count of 'ALL' category (58,771) friendly frontend application to display model predictions,
compared to females (6,828) and males (3,361), which could enabling easy access and decision-making for clinical trial teams.
affect model performance and generalizability. This could involve creating dashboards for visualizing predicted
2. Null Handling: Although missing values in text columns were enrollment times, feature importance, and model insights. API
imputed with 'unknown' and numerical columns with median integration would allow real-time predictions from the model for
values, these imputation methods may not fully capture new clinical trial data.
underlying patterns, potentially introducing bias. 2. Data Enrichment: Incorporating additional data sources, such as
3. Location & Phases: Missing values in 'Locations' (43,383) and trial results, demographic data, or geographical factors, could
'Phases' (2,141) were handled by standardizing with known further refine predictions and provide deeper insights into
methods, but this could impact the completeness and accuracy recruitment challenges. Integrating external clinical trial
of certain features. databases could enrich the model and improve accuracy.
4. Biased Embeddings: Using BioBERT for text embeddings 3. Model Performance: Testing more advanced models, including
ensures medically appropriate representation but may introduce deep learning or ensemble methods, might improve results.
inherent biases from the pretrained model, which may not fully Regular model updates based on new clinical trial data and
reflect specific clinical contexts. recalibration could help maintain accuracy over time.
5. Outliers in Enrollment Time: The wide range of 'Time taken 4. Further Research: Investigating the use of domain-specific
for Enrollment' (0 to 99 months) might distort model predictions, embeddings, like those from medical research papers, could help
especially for outliers. capture more intricate patterns in clinical trial data.

10
Thank You!