INSTITUTE OF HEALTH INFORMATICS

Graduate Programme in Health Data Science

Assessed Coursework Submission

Student candidate number: NHWJ

Module: CHME0035: Advanced Machine Learning for Healthcare

Date due: 9 July 2025, 16:00 BST

Word count:
(excluding references,
diagrams and appendices)

Disability or other medical

condition for which UCL has
granted special examination
arrangements:
Please address in formative feedback:

Formative feedback: Please ignore in formative feedback:

NLP patient Q&A system:

The aim of this task, is to build a patient question and answer (Q&A) system using natural language
processing (NLP) that is able to answer patient questions. Unstructured clinical notes contain key patient
information, however there are challenges in their interpretability. Patient Q&A systems using NLP
provide a solution to this, allowing relevant information to be extracted and aiding in clinical decision
making and enhancing patient care (KMS staff, 2025). In recent years, NLP has been used to create an
patient Q&A system called MEANS which used a number of different NLP techniques/algorithms to
extract answers from unstructured clinical data (Abacha A.B & Zweigenbaum, 2015). Medical entity
recognition of the model was found to have an F1 score of 78%. Relation extraction using patterns,
machine learning and a hybrid of both also showed a significantly high F1 score, with the best being the
hybrid method. MEANS’ ability to understand the question via a SPARQL query was also tested and
showed 98% accuracy. Additionally, the system was able to rank answers by how well they matched the
question and used query relaxation to find the most useful answers. Overall, MEANS performed well and
worked well in providing clinicians and patients with clear answers about various medical topics. (Abacha
A.B., Zweigenbaum, 2015). In this task, two different methods will be completed, TF-IDF vector similarity
will be completed to match questions to the most relevant paragraph from the document base and return
answers and prompting an LLM to classify diseases from questions using retrieval-augmented
generation (RAG), their performance compared, and their use in healthcare discussed. The dataset
used, is derived from patient information found on the [Link] website and was synthetically generated
using ChatGPT. There are two CSV files containing patient questions and answers as well as a ZIP
archive with 2,392 plain text documents, each with information relevant to answering questions about a
certain disease. From completing descriptive analysis of the data, it was found that the training set had
24,005 entries and the test set had 211. There are 2,392 unique diseases in the training dataset, and 23
in the test set. The average question word length in the training set is 8.9 and 7.3 in the test set, and the
average answer word length was 41.4 in training and 42.6 in test. Both contain four columns, question,
answer, disease and reference link to the NHS website.
For TF-IDF, pre-processing is completed using SpaCy, which includes the removal of stop words and
non-letter tokens as well as lemmatisation, which is the process used to reduce words to their base form.
For the LLM model however, this isn’t needed as the model is trained on raw text and recognises
relationships between words, lemmatisation removes some of this additional context within the
sentences. Tokenisation, the process of breaking down sequences of text into smaller units is done for
the LLM, via an autotokeniser. The training set for the LLM, won’t be used as pre-trained model is being
used and various prompt strategies will be tested, such as zero-shot prompting, where no examples are
provided and few-shot prompting, where the model is provided with a number of examples to guide the
output and improve performance.
TF-IDF, is a technique that measures how often a term appears in a document as well as how important
a term is across all documents, these are then multiplied to identify terms that are frequent in a specific
document and rare across all documents. The cosine similarity is used to determine the similarity
between the query vector and document vector, which is in this case the questions from the CSV files
and the disease .txt files. High cosine similarity scores, imply the most relevant documents. To begin
with, a term-document matrix is created. Lastly, for disease classification using an LLM, API calls is used
and as such the model used is Llama 3 8B and both zero-shot and few-shot prompting methods
completed. RAG will be achieved using Glove embeddings, which creates numerical representations of
words in a vector space in order to capture relationships between words. Initially, average word vectors
are computed to embed the text and then relevant document parts are retrieved using the Glove method.
For the LLM model, a ROUGE metric was produced for both zero shot and few shot prompting, which
measures the overlap between the generated answers and reference answers. Alongside TF-IDF,
additional vector representations were created and compared; term-document, term-term and positive
pointwise mutual information (PPMI) and results were visualised in graphs using PCA. Additionally,
performance metrics for term frequency and TF-IDF representation such as precision, recall and f1 score
were attempted to be found.
The ROUGE metrics for the zero-shot LLM, showed ROUGE-1 which measures single word overlap to
be 0.4274, ROUGE-2 measuring the overlap between two consecutive words being 0.259, ROUGE-L
which measured the longest common subsequence as 0.3430 and lastly ROUGE-Lsum which provides
a summary of ROUGE-L as 0.3604. From this it can be interpreted that about 42.7% of words generated
overlap with the reference answers, but there is a decrease seen when measuring the overlap between
two consecutive words, as there is only a 25.9% match. Overall, there is a 36% alignment between the
generated and reference answers. These results can be improved by applying few shot prompting or
altering some of the hyperparameters such as temperature, as answers may be too rigid. For few-shot
prompting, three examples were provided and the ROUGE scores generated where as follows, ROUGE-
1 was 0.4473, ROUGE-2 was 0.2796, ROUGE-L was 0.3661 and lastly ROUGE-Lsum being 0.3788. In
comparison to the method with zero-shot prompting, it can be seen that there is a slight improvement of
0.02 across all the metrics when few-shot prompting is done. Further improvement to performance could
potentially be achieved by increasing the number of examples. From the plots above, it can be seen that
TF-IDF representation has better clustering of relevant terms such as ‘diabetes’, ‘insulin’ and ‘glucose’,
compared to the term-document representation. PPMI, also shows decent clustering of terms. The
performance metrics were unfortunately unable to be generated due to issues with RAM availability in
the notebook.

References:
Ben Abacha, A., & Zweigenbaum, P. (2015). MEANS: A medical question-answering system combining
NLP techniques and semantic Web technologies. Information Processing & Management, 51(5), 570–
594. [Link]
(N.d.). [Link]. Retrieved 10 July 2025, from [Link]
language-processing-in-healthcare/#:~:text=In%20the%20healthcare%20industry%2C%20NLP,contains
%20information%20about%20multiple%20patients