Master of Science in Computer Science

September 2024

Data Analytics using Large Language

Models on ITSM Data

Ganesh Mahadev Udata

N Venkata Sai Kaparotu

Faculty of Engineering, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden

This thesis is submitted to the Faculty of Engineering at Blekinge Institute of Technology in partial
fulfilment of the requirements for the degree of Computer Science and Engineering. The thesis is
equivalent to 20 weeks of full-time studies.

The authors declare that they are the sole authors of this thesis and that they have not used any
sources other than those listed in the bibliography and identified as references. They further declare
that they have not submitted this thesis at any other institution to obtain a degree.

Contact Information:
Author(s):
Ganesh Mahadev Udata
E-mail: gaud22@[Link]

K N Venkata Sai
E-mail: nekp22@[Link]

University advisor:
Dr. Huseyin Kusetogullari
Department of Computer Science

Faculty of Engineering Internet : [Link]

Blekinge Institute of Technology Phone : +46 455 38 50 00
SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57
 Abstract

Background We explore key concepts including IT Service Management (ITSM) and

Incident Management, highlighting the importance of incident data in improving service
operations. We also cover the significance of Data Analytics in extracting insights and
its relevance to ITSM. The capabilities of Large Language Models (LLMs) in generating
code for data analysis are discussed, alongside various prompting techniques used to re-
fine LLM responses. This foundational understanding sets the stage for evaluating how
effectively LLMs can generate code for complex data analytics tasks.
Objectives The Objectives of this thesis are threefold. First, we aim to evaluate the
ability of various LLMs to generate accurate and relevant code for data analytics using
ITSM data. Second, we seek to identify and assess the effectiveness of different prompt-
ing techniques in guiding LLMs to produce structured and task-relevant outputs for
ITSM-related tasks. Finally, we explore how different LLMs can facilitate conversational
generation to enhance and streamline ITSM data analysis, ensuring more effective and
interactive data-driven decision-making.
Method We conducted a comprehensive literature review to identify key prompting
strategies—Zero-Shot Prompting, Priming, and Chain-of-Thought Prompting. These
strategies were experimentally validated using three LLMs: LLAMA3.1-8B, GEMMA2-
9B, and PHI3-14B. The models were tested on four categories of tasks: Descriptive,
Statistical, Graph Plot, and Data Cleaning. The effectiveness of each prompting strategy
and LLM was measured through correctness, relevance, and clarity across multiple task
domains.
Results The findings demonstrate that Priming emerged as the most effective prompting
strategy, significantly enhancing the models’ ability to generate task-relevant and accu-
rate code. Chain-of-Thought Prompting improved logical reasoning but still struggled
with data handling. Zero-Shot Prompting, though useful for general tasks, consistently
failed in complex multi-step tasks. Among the models, GEMMA2-9B outperformed oth-
ers, particularly in Descriptive and Data Cleaning tasks, providing accurate, relevant,
and clear outputs based on user feedback. PHI3-14B excelled in Graph Plotting, while
all models showed significant weaknesses in handling Statistical tasks such as hypothesis
testing.
Conclusions This thesis demonstrates the critical role of structured prompting strate-
gies, particularly Priming, in improving the performance of LLMs in code generation for
data analytics. The experimental results show that while GEMMA2-9B performs best
in Descriptive and Data Cleaning tasks, PHI3-14B performs best in Graphical Analysis.
However, there remains a need for further development in handling statistical reasoning
across all models.
Keywords LLM, ITSM, Data Analysis, LLAMA3.1, GEMMA2, PHI3, Prompting Strate-
gies, Zero shot prompting, Chain of Thought prompting, Priming, Prompt Engineering
 Acknowledgments

We would like to express my deepest gratitude to our thesis supervisor at Ericsson,

Tarun Prakash, for his unwavering guidance, support, and encouragement throughout
the course of this research. His invaluable insights and constant motivation have played
a pivotal role in shaping the direction of this work.

We would also like to extend our heartfelt thanks to Mikael Nilsson, our manager
at Ericsson, for his professional mentorship and for providing me with the opportunity
and resources to pursue this research. His support and belief in our abilities have been a
constant source of inspiration.

Special thanks to Dr. Huseyin Kusetogullari, our academic supervisor at the uni-
versity, whose expert advice and constructive feedback have significantly contributed to
the academic rigour of this thesis. His patience and knowledge have been greatly appre-
ciated throughout this journey.

Finally, We want to express our deepest appreciation to our friends and family for their
unwavering support, understanding, and encouragement during this challenging period.
Their belief in us has been a source of strength, and we are forever grateful for their love
and care.

iii
 Contents

Abstract i

Acknowledgments iii

1 Introduction 1
1.1 Research Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Aim and Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Background 5
2.1 ITSM and Incident Management . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 Definition and Importance . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Role of Incident Management Data in ITSM . . . . . . . . . . . . 6
2.2 Overview of Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 Data Analytics and its significance in extracting insights from data 7
2.2.2 Common Techniques and Methodologies . . . . . . . . . . . . . . 8
2.2.3 Relevance of Data Analytics in context of ITSM . . . . . . . . . . 8
2.3 Overview of LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Introduction to LLMs . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 Architecture and Functioning of LLMs . . . . . . . . . . . . . . . 10
2.4 Code Generation Capabilities of LLMs . . . . . . . . . . . . . . . . . . . 11
2.4.1 Significance of Code Generation in general . . . . . . . . . . . . . 11
2.4.2 Application of Code Generation with LLMs . . . . . . . . . . . . 12
2.4.3 Significance of Code Generation in Data Analysis Tasks . . . . . . 13
2.5 Prompting Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5.1 Various aspects of restricting an LLMs response using prompts . . 13
2.5.2 Chat and Text completion models . . . . . . . . . . . . . . . . . . 14
2.6 Evaluation of LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3 Related Work 17
3.1 LLMs in Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 LLMs in ITSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Research Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3.2 Problem Definition and Formulation . . . . . . . . . . . . . . . . 20

v
 3.3.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4 Method 21
4.1 Research Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.2 Search Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.3 Inclusion and Exclusion Criteria . . . . . . . . . . . . . . . . . . . 22
4.2.4 Quality Assessment Criteria . . . . . . . . . . . . . . . . . . . . . 23
4.2.5 Data Extraction Strategy . . . . . . . . . . . . . . . . . . . . . . 23
4.2.6 Data Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3 Experimentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3.1 Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3.2 Research Question . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3.3 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3.4 Dependent and Independent Variables . . . . . . . . . . . . . . . 24
4.3.5 Dataset Description . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.6 LLMs Selected . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.7 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3.8 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.3.9 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.3.10 Confounding Factors . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.4 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4.1 Internal Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4.2 External Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4.3 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4.4 Conclusion Validity . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.5 Validity and reliability of approach . . . . . . . . . . . . . . . . . . . . . 32

5 Results and Analysis 33

5.1 Literature Review Findings . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 Experiment Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.2.1 Results and Analysis of RQ2 . . . . . . . . . . . . . . . . . . . . . 36
5.2.2 Results and Analysis of RQ3 . . . . . . . . . . . . . . . . . . . . . 38

6 Discussion 63
6.1 Discussion for RQ1 and RQ2 . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.1 Zero-Shot Prompting . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.2 Priming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.3 Chain-of-Thought Prompting . . . . . . . . . . . . . . . . . . . . 64
6.2 Discussion for RQ3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2.1 Descriptive Questions . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2.2 Statistical Questions . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.2.3 Graph Plot Questions . . . . . . . . . . . . . . . . . . . . . . . . 67
6.2.4 Data Cleaning Questions . . . . . . . . . . . . . . . . . . . . . . . 67
6.3 Potential Drawbacks of LLMs in ITSM . . . . . . . . . . . . . . . . . . . 68

vi
7 Conclusions and Future Work 71
7.1 Summary of the findings . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
7.1.1 Prompting Strategies for Code Generation . . . . . . . . . . . . . 71
7.1.2 LLM Performance in Code Generation . . . . . . . . . . . . . . . 71
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

References 75

vii
 List of Figures

2.1 Phases of data analytics cycle [1] . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Techniques and Methods of Data Analysis . . . . . . . . . . . . . . . . . 8
2.3 Evolution of LLMs [2] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Architecture of Transformers [3] . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Application of LLMs [4] . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.1 Snippet of the dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2 Architecture of Experiment . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.1 Distribution of the articles over the years . . . . . . . . . . . . . . . . . . 33

5.2 Number of Articles Published by Source . . . . . . . . . . . . . . . . . . 34
5.3 Distribution of Prompting Strategies by Number of Studies . . . . . . . . 34
5.4 The working of prompting strategies . . . . . . . . . . . . . . . . . . . . 36
5.5 Comparison of Zero-Shot, Priming, and Chain-of-Thought outputs . . . . 37

6.1 Average scores for Descriptive Questions . . . . . . . . . . . . . . . . . . 66

6.2 Average scores for Statistical Questions . . . . . . . . . . . . . . . . . . . 66
6.3 Average scores for Graph Plot Questions . . . . . . . . . . . . . . . . . . 67
6.4 Average scores for Data Cleaning Questions . . . . . . . . . . . . . . . . 68

ix
 List of Tables

4.1 Hardware Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2 Software Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.1 Overview of Different Prompting Strategies . . . . . . . . . . . . . . . . 35

5.2 Average scores for different question types. . . . . . . . . . . . . . . . . . 39
5.3 Results of Descriptive questions with LLAMA 3.1:latest . . . . . . 40
5.4 Results of Statistical questions with LLAMA 3.1:latest . . . . . . . 42
5.5 Results of Graph plot questions with LLAMA 3.1:latest . . . . . . 44
5.6 Results of Data Cleaning Questions with LLAMA 3.1:latest . . . . 46
5.7 Results of Descriptive questions with PHI3 14b . . . . . . . . . . . 48
5.8 Results of Statistical questions with PHI3 14b . . . . . . . . . . . . 49
5.9 Results of Graph plot questions with PHI3 14b . . . . . . . . . . . 51
5.10 Results of Data Cleaning Questions with PHI3 14b . . . . . . . . . 53
5.11 Results of Descriptive questions with GEMMA2 9b . . . . . . . . . 55
5.12 Results of Statistical questions with GEMMA2 9b . . . . . . . . . 56
5.13 Results of Graph plot questions with GEMMA2 9b . . . . . . . . . 58
5.14 Results of Data Cleaning Questions with GEMMA2 9b . . . . . . . 60

7.1 Model Performance Across Various Tasks . . . . . . . . . . . . . . . . . . 72

xi
Chapter 1
Introduction

In today’s world, no business can function effectively without Information Technology

(IT), regardless of its size or global reach. Companies rely heavily on IT for various func-
tions, including production processes, decision-making, and other areas involving tech-
nologies such as computers, tablets, smartphones, and more. Managing the deployment
and utilization of these technologies efficiently has led to the development of Information
Technology Service Management (ITSM) [5].

ITSM focuses on designing, delivering, managing, and improving IT services to align

with business needs while ensuring operational efficiency. The first international standard
for ITSM, ISO/IEC 20000, defines it as an “integrated process approach that enables IT
organizations to deliver services that meet business and customer requirements” [6]. One
of the most widely adopted ITSM frameworks in the industry is the IT Infrastructure
Library (ITIL) [7].

However, despite the structure provided by ITSM frameworks like ITIL, current ITSM
processes still face significant inefficiencies, especially when it comes to incident manage-
ment and data analysis. Manual data handling is labor-intensive, and slow incident reso-
lution often occurs due to the difficulty of identifying root causes from large and unstruc-
tured incident data. Identifying trends and recurring issues is particularly challenging,
leading to reactive problem-solving instead of proactive incident prevention. Moreover,
the sheer volume of incident logs and their complexity makes it difficult for IT teams to
extract meaningful insights in a timely manner.

These challenges are exacerbated by the limitations of traditional data analytics meth-
ods, which require a substantial amount of human intervention to preprocess data, identify
patterns, and produce actionable insights. As organizations continue to handle increas-
ingly complex and voluminous ITSM data, there is a growing need for more sophisticated
tools that can streamline and automate the analysis process.

This is where Large Language Models (LLMs) come into play. LLMs have the po-
tential to revolutionize ITSM data analysis by automating tasks such as data extraction,
pattern recognition, and root cause analysis. By leveraging LLMs, organizations can
move beyond manual data processing and reactive incident handling to a more proac-
tive and automated approach . For example, LLMs can quickly analyze incident reports
to identify underlying patterns that may go unnoticed by human analysts, offering in-
sights that can help reduce downtime and improve service quality . Additionally, LLMs
can generate code for custom data analytics tasks, reducing the need for specialized data
analysts and empowering IT teams to make more informed decisions with fewer resources.

1
2 Chapter 1. Introduction

LLM’s have many applications and Data analytics also comes under one of application
where LLM generated code to analyze and explore data to give useful information.

In this research, we propose utilizing LLMs to streamline the data analytics process
in ITSM, with a specific focus on incident management data. By applying advanced
language models, we aim to address the inefficiencies currently present in ITSM processes
and enhance the decision-making capabilities of IT teams.

1.1 Research Scope

Modern LLMs offer a range of applications, including automation through tool invocation,
personal chatbots, retrieval-augmented generation (RAG) applications, code completion,
and boilerplate code generation. Generally, language models can generate or complete
code effectively. However, their performance tends to decline when dealing with large
codebases. In data analytics, though, if tasks are broken down into simpler steps, the
code becomes more manageable, and language models can better grasp the nuances in-
volved. Our thesis focuses specifically on evaluating the code generation capabilities of
these models, particularly in the context of data analysis. These LLMs are prone to
hallucinations and can produce incorrect code. Our primary focus is on multi-turn con-
versations, where the LLMs generate responses during user-assistant interactions. When
the assistant’s response contains errors or hallucinations, the user provides feedback, and
based on this input, the LLMs attempt to correct themselves and deliver accurate output.

This research will assess the ability of various language models to generate code for
data analytics, specifically using ITSM Incident Managment data. It will also explore
different prompting techniques to evaluate their effectiveness in producing accurate and
structured outputs while facilitating meaningful interactions for data analysis. By sys-
tematically comparing the performance of different models and considering ITSM-specific
constraints, this study seeks to identify the most effective approaches and models for en-
hancing data analytics workflows in the ITSM domain.

1.2 Aim and Research Questions

1.2.1 Aim
The aim of this thesis is to evaluate the code generation capabilities of modern LLMs
for data analytics, specifically using IT Service Management (ITSM) data. The study
seeks to build a system that not only generates accurate and structured code but also
engages users in meaningful conversations, providing actionable insights based on the
outputs of code execution. By exploring different prompting techniques and comparing
the performance of various language models while considering ITSM-specific constraints,
this research aims to identify the most effective approaches and models for enhancing
data analytics workflows in the ITSM domain.
1.3. Research Objectives 3

1.2.2 Research Questions

RQ1: Which prompting strategies most effectively in generating code for data analysis?
Research Methodology: Literature Review
Justification: We believe a literature review is essential for this research question to
identify existing prompting techniques and their reported effectiveness in code generation
tasks. This approach allows us to build upon prior knowledge and avoid duplicating ef-
forts. We also believe literature review will help us understand the current state-of-the-art
in prompting strategies for LLMs in code generation and data analysis tasks.

RQ2: How do these prompting strategies experimentally validated to determine the

optimal approach for refining and enhancing model outputs?
Research Methodology: Experiment
Justification: An experimental approach is then necessary to validate and compare
these prompting strategies in the specific context of ITSM data analytics. Experiments
allow for controlled testing of different prompting techniques, enabling you to measure
their impact on code quality, accuracy, and relevance to ITSM-specific requirements.

RQ3: Which Large Language Model (LLM) demonstrates superior performance in multi-
turn interactions for code generation in data analytics and builds the conversation to
analyze data based on user feedback?
Research Methodology: Experiment
Justification: We believe an experimental methodology is appropriate for assessing the
performance of various LLMs in multi-turn interactions for code generation, where it-
erative refinement based on user feedback is critical. By systematically testing different
models, the study can directly measure and compare their ability to generate, refine, and
improve code within ITSM-specific contexts.

1.3 Research Objectives

1. To evaluate the ability of various language models to generate accurate and relevant
code for data analytics on ITSM data.

2. To identify and assess the effectiveness of different prompting techniques in produc-

ing structured outputs from language models for ITSM data.

3. To explore how different language models can facilitate conversational generation

for effective ITSM data analysis.

1.4 Ethical Considerations

• Ethical Aspects: Implementing strong data governance policies, including con-
sent, anonymization, and secure data handling, is essential when using LLMs in
ITSM. LLMs may unintentionally retain sensitive information, risking data leaks.
Regular audits and compliance with data protection regulations like GDPR are
necessary to protect privacy and ensure responsible use.
4 Chapter 1. Introduction

• Bias and Fairness: LLMs can inherit biases from the data they are trained on,
leading to biased decision-making in ITSM, such as unfair prioritization of inci-
dents. Bias detection and mitigation strategies should be applied to ensure fair and
equitable outcomes across all tasks.

• Data Privacy Risks: LLMs in ITSM process large amounts of sensitive data,
raising privacy concerns. There is a risk of data exposure despite anonymization,
especially when models are connected to external systems. To mitigate this, strict
security measures like encryption, access control, and compliance with privacy laws
are crucial.

• Societal Aspects: While LLMs can improve ITSM efficiency, they may also lead to
job displacement by automating routine tasks. Ensuring that LLMs augment rather
than replace human expertise will help maintain a balance between automation and
human involvement in decision-making.

• Sustainability Aspects: LLMs require significant computational resources, con-

tributing to a larger carbon footprint. To minimize this, energy-efficient models and
green data practices should be adopted. On the positive side, automating ITSM
processes can reduce operational costs and resource use.

1.5 Outline
• Chapter 2 - Provides an in-depth review of ITSM and data analytics, including the
relevance of incident management data. This chapter also explores the foundational
concepts of LLMs, giving readers an understanding of their role in code generation
for data analytics tasks.

• Chapter 3 - Reviews the existing literature on LLMs in data analytics and ITSM.
This chapter highlights gaps in the current research and positions the contribution
of the thesis within this context.

• Chapter 4 - Describes the methodology used to conduct the research. It covers

the experimental setup, including how the LLMs were evaluated and what data was
used for experimentation. This chapter also details the research design, hypotheses,
and data extraction strategies.

• Chapter 5 - Presents the findings from the experiments, including results on the
effectiveness of different prompting techniques and the comparative performance of
LLMs in generating code for ITSM data analytics.

• Chapter 6 - Interprets the results in the context of the research questions. This
chapter discusses the implications of the findings, offering an in-depth analysis of
their significance for the ITSM and LLM communities.

• Chapter 7 - Summarizes the key conclusions drawn from the research and provides
suggestions for future studies. This chapter reflects on the contributions made by
the thesis and how it paves the way for future advancements in LLM-based data
analytics.
Chapter 2
Background

2.1 ITSM and Incident Management

2.1.1 Definition and Importance

Information Technology Service Management (ITSM) refers to a structured approach in-

volving processes, policies, and procedures aimed at managing IT services to align them
with business needs. It is often considered as bridge between that connects information
technology (IT) management with business management [8]. It includes the design, im-
plementation, management, and enhancement of IT services utilised by employees and
customers. Organisations that apply ITSM ensure that IT services are supplied efficiently
and effectively while satisfying the business’s requirements and goals. ITSM frameworks,
such as ITIL, give rules for managing IT services methodically and consistently, improv-
ing overall organisational performance [9].

The importance of ITSM lies from its ability to match IT services with business objec-
tives, hence boosting overall efficiency and effectiveness. It improves customer satisfaction
by guaranteeing structured and consistent IT services, assists with risk management by
identifying and reducing IT-related issues, and maintains company continuity by min-
imising downtime.

Incident Management, a key component of ITSM, focusses on managing and resolving

issues, which are unplanned interruptions or reductions in the quality of IT services. The
fundamental goal of incident management is to promptly restore regular service opera-
tions and minimise the impact on business activities [10]. This includes finding, docu-
menting, categorising, prioritising, investigating, diagnosing, resolving, and closing issues.

Incident management is critical for reducing downtime and sustaining productivity

by promptly addressing and resolving occurrences. It ensures the quality of IT services
by responding quickly to disturbances and resuming normal operations. Quick resolution
of incidents enhances the user experience and customer satisfaction. Structured incident
management methods improve the efficiency and efficacy of IT operations, whilst detailed
event logs give useful information for spotting patterns and optimising future service
delivery. This proactive approach prevents such issues and improves overall IT service
quality [10].

5
6 Chapter 2. Background

2.1.2 Role of Incident Management Data in ITSM

Incident Management Data plays a pivotal role in ITSM by providing essential insights
and supporting various processes that ensure the effective and efficient delivery of IT
services [10] [11]. Below is the detailed look of it’s role.

1. Monitoring and Tracking Incidents

• Incident Logging: Incident management data consists of thorough logs of

all incidents, which capture crucial information such as the time of occurrence,
issue description, affected services, and initial impact assessment.
• Status Updates: Data on the current status of incidents (e.g., open, in
progress, resolved) allows IT teams to track the progress of incident resolution
and provide timely updates to stakeholders.

2. Prioritizing and Categorising Incidents

• Severity and Priority Levels: Incident management data often includes

fields for severity and priority levels, which assist IT teams in prioritising
events based on their impact on business operations. High-priority events are
handled more quickly in order to minimise interruption.
• Categorization: Incidents are classified according to their type and how they
influence services. This categorisation helps to detect patterns and trends,
allowing for more effective problem management and root cause analysis.

3. Supporting Problem Management and Root Cause Analysis

• Historical Data: Incident management data gives a historical record of all

occurrences, which is extremely useful for identifying reoccurring issues and
performing root cause investigation. IT teams can build tactics to avoid similar
problems in the future by analysing previous instances.
• Problem Identification: Patterns discovered in incident data can reveal
underlying issues, helping IT teams to transition from reactive incident man-
agement to proactive problem management.

4. Enhancing Decision-Making and Strategic Planning

• Data-Driven Insights: Analysing incident management data provides valu-

able insights into the performance and reliability of IT services. These insights
inform strategic decisions related to resource allocation, service improvements,
and investment in new technologies.
• Trend Analysis: Incident data helps in identifying long-term trends and
emerging issues, enabling IT leaders to anticipate and address potential chal-
lenges proactively.
2.2. Overview of Data Analytics 7

2.2 Overview of Data Analytics

2.2.1 Data Analytics and its significance in extracting insights

from data

Data analytics is defined as the application of computer systems to the analysis of large
datasets to support decision-making. It is an interdisciplinary field that incorporates el-
ements from various scientific disciplines, including statistics, machine learning, pattern
recognition, system theory, operations research, and artificial intelligence [1]. The knowl-
edge discovery in databases (KDD) process involves six phases: selection, preprocessing,
transformation, data mining, interpretation, and evaluation [12]. Similarly, the Cross
Industry Standard Process for Data Mining (CRISP-DM) includes six phases: business
understanding, data understanding, data preparation, modelling, evaluation, and deploy-
ment [13]. For simplicity, these phases can be condensed into four main stages: prepara-
tion, preprocessing, analysis, and postprocessing, shown in 2.1.

The data analytics process starts with Preparation, which includes planning, data
gathering, feature creation, and data selection. Preprocessing is the process of cleaning,
filtering, completing, correcting, standardising, and transforming data to ensure quality
and consistency. During the Analysis phase, techniques including visualisation, corre-
lation, regression, forecasting, classification, and clustering are used to identify patterns.
Finally, Postprocessing entails analysing results, recording methodology and conclu-
sions, and assessing the correctness and effectiveness of the analysis. This organised
approach converts raw data into actionable insights, allowing for better decision-making
and increased operational efficiency.

Data analysis is critical for obtaining insights from data since it allows businesses to
convert raw data into usable knowledge. Data analysis, which employs techniques such as
statistical analysis, machine learning, and data visualisation, aids in the identification of
previously unknown patterns, trends, and connections. This method promotes evidence-
based decision-making, increases operational efficiency, and identifies areas for innovation
and improvement. Businesses that methodically analyse data can foresee future trends,
optimise processes, and acquire a competitive advantage in their respective industries.

Figure 2.1: Phases of data analytics cycle [1]

8 Chapter 2. Background

2.2.2 Common Techniques and Methodologies

Data analytics encompasses a variety of techniques and methodologies aimed at extract-
ing meaningful insights from data. Any method that helps to derive valuable insights
or improve the understanding of data falls into this category. The below figure 2.2 gives
a high level view of the common techniques and methods that are widely used in the
software industry.

Figure 2.2: Techniques and Methods of Data Analysis

2.2.3 Relevance of Data Analytics in context of ITSM

Data analytics significantly enhances ITSM by delivering actionable insights that lead to
better decision-making and problem solving. By analysing historical data, IT adminis-
trators can find patterns and trends that can aid in the understanding and resolution of
service issues. Predictive analytics offers preventive maintenance by projecting possible
problems, reducing downtime [14]. Moreover, data visualisation tools provide real-time
performance monitoring via dashboards that display key performance indicators (KPIs)
such as Service Level Agreement (SLA) compliance and incident response times, ensuring
that services are aligned with corporate objectives and user expectations.

Also, data analytics enhances customer satisfaction and strategic planning in ITSM.
By analysing feedback and service performance data, IT teams may better tailor ser-
vices to user needs, resulting in faster response times and higher customer satisfaction.
Data analytics insights enable businesses to stay ahead of industry trends and respond
to changing demands, making strategic planning more effective. This competitive advan-
2.3. Overview of LLMs 9

tage allows ITSM organisations to innovate and supply more services, leading to better
operational efficiency and business success.

2.3 Overview of LLMs

2.3.1 Introduction to LLMs

LLMs have evolved through four distinct generations, each with considerable advances
in task-solving capabilities. Statistical language models (SLMs) first arose in the 1990s,
using n-gram models and statistical methods to aid with specialised tasks such as speech
recognition and information retrieval. These models were basic but limiting because they
relied on set context lengths and simple probabilistic assumptions [15]. The introduction
of neural language models (NLMs) marked the next major leap, utilizing neural networks
such as multi-layer perceptrons and recurrent neural networks to learn distributed word
representations and capture more complex linguistic patterns [16] [17]]. These also im-
pacted the use of language models for representation learning and in the field of NLP.

Pre-trained language models (PLMs) represented a transformative phase in the evolu-

tion of LLMs. Early examples like ELMo and BERT [18] utilized context-aware represen-
tations and the paradigm of pre-training followed by fine-tuning on large-scale corpora,
significantly boosting performance across various natural language processing (NLP)
tasks. This phase saw the models becoming more adaptable and capable of transfer-
ring knowledge across different tasks, setting a new standard in NLP [19].

Figure 2.3: Evolution of LLMs [2]

LLMs such as GPT-3 and GPT-4 define the current era, focussing on expanding
model and dataset sizes to achieve unprecedented capabilities. These models are trained
on large datasets and given a lot of parameters to help them do real-world tasks using
prompt-based completions and emergent skills [2]. The move from task-specific helpers
to general-purpose task solvers demonstrates LLMs’ dramatically increased functional
capability, which allows them to execute complicated reasoning, write coherent language,
and even participate in meaningful dialogues with humans.
10 Chapter 2. Background

2.3.2 Architecture and Functioning of LLMs

The Transformer architecture serves as the fundamental building block for all transformer-
based language models or LLMs, making the language processing more human-like.

[Link] Transformer Architecture

The Transformer architecture is fundamentally built on the self-attention mechanism,
eliminating the need for recurrent neural networks (RNNs) or convolutional neural net-
works (CNNs). This design allows for greater parallelization and improved efficiency
during training, addressing the limitations of sequential processing inherent in RNNs and
CNNs [3]. Figure 2.4 illustrates the general structure of the Transformer model. The key
components of this architecture are outlined below.
1. Encoder-Decoder Structure
(a) Encoder: The encoder processes the input sequence and maps it to a contin-
uous representation. It is composed of a stack of N identical layers (typically
N =6). Each layer has two sub layers:
• A multi-head self attention mechanism.
• A position-wise fully connected feed-forward network.
(b) Decoder: The decoder generates the output sequence one token at a time.
It is also composed of a stack of N identical layers, with an additional third
sub-layer in each layer for multi-head attention over the encoder’s output.
The decoder uses masking to prevent positions from attending to subsequent
positions, preserving the auto-regressive property.
2. Attention Mechanisms
(a) Scaled Dot-Product Attention: The attention function transforms a query
and a set of key-value pairs into an output. The query’s dot products
√ with
all keys are computed, divided by the square root of the keys dk , and run
through a softmax function to determine the weights on the values.
(b) Multi-Head Attention: Instead of performing a single attention function
with dmodel -dimensional queries, keys, and values, the multi-head attention
mechanism projects these into h different subspaces using different learnt linear
projections, performs the attention function in parallel, and concatenates the
results.

3. Position wise Feed Forward Networks

Each layer of the encoder and decoder has a fully connected feed-forward network
that is applied to each point independently and identically. It consists of two linear
transformations with a ReLU activation in the middle.
4. Embeddings and Softmax
Learnt embeddings are used to turn input and output tokens into d-dimensional
model vectors. The decoder’s output is converted to anticipated next-token proba-
bilities using a linear transformation and the softmax function. The weight matrix
is shared by both embedding layers and the pre-softmax linear transformation.
2.4. Code Generation Capabilities of LLMs 11

Figure 2.4: Architecture of Transformers [3]

5. Positional Encoding
Since there is no recurrence or convolution in the model, positional encodings are
required to inject information about the relative or absolute position of the tokens
in succession. This is accomplished by utilising the use of sine and cosine functions
at various frequencies.

2.4 Code Generation Capabilities of LLMs

2.4.1 Significance of Code Generation in general
Code generation through LLMs has emerged as a transformative approach in software
development, fundamentally altering how code is produced and [Link] mod-
els, which translate natural language descriptions straight into executable code, cover a
wide range of software engineering difficulties, from regular coding jobs to complicated
algorithmic execution. This capability not only accelerates the development process,
12 Chapter 2. Background

but also simplifies programming by allowing non-experts to describe their needs in plain
terms [20]. Advances in machine learning approaches, such as few-shot learning, have
improved the models’ ability to interpret and write code, allowing them to perform more
accurately and efficiently on a variety of coding jobs.

The importance of code generation in artificial intelligence, particularly the usage of

LLMs, goes beyond the simple automation of coding operations. It is a significant step to-
wards the creation of completely autonomous coding agents capable of managing software
development projects from start to finish. These models are trained on large datasets of
code, allowing them to generate not only snippets but complete apps, potentially leading
to considerable reductions in development time and costs [21]. Also, the incorporation of
code generation systems such as GitHub Copilot and Amazon CodeWhisperer emphasises
the practical applicability and commercial viability of LLMs in everyday programming
tasks.

2.4.2 Application of Code Generation with LLMs

The applications of code generation with LLMs cover a wide range of software engi-
neering tasks, revolutionizing traditional development workflows and introducing novel
capabilities [4].

Figure 2.5: Application of LLMs [4]

1. Description to Code: This application converts natural language descriptions to

executable code. It enables rapid prototyping and development, which is especially
beneficial in educational settings because it can be used as both a learning tool for
students and a testing tool for educators. LLMs, such as OpenAI’s Codex, offer
high precision and diverse coding solutions based on descriptions [22].
2.5. Prompting Techniques 13

2. Code Completion: LLMs improve Integrated Development Environments (IDEs)

by generating intelligent code completion suggestions. Unlike typical code comple-
tion tools, which largely suggest based on syntax, LLMs understand context and
can recommend entire blocks of code based on their semantic comprehension of the
current activity. This dramatically increases developer productivity and reduces
cognitive burden by efficiently completing basic coding tasks.

3. Automatic Program Repair: LLMs can automatically identify bugs and suggest
fixes, transforming the debugging and maintenance phases of software development.
Understanding both the code’s goal and its actual functionality allows these models
to recommend realistic and immediately relevant improvements, minimising the
time and effort spent on debugging and maintenance.

2.4.3 Significance of Code Generation in Data Analysis Tasks

Data analysis tasks are critical in scientific research, where the capacity to quickly un-
derstand and extract relevant insights from data sets can considerably improve produc-
tivity and decision-making. LLMs have proved transformational in this field, allowing
researchers to automate difficult data processing procedures that would otherwise involve
extensive manual labour. Using LLMs, scientists can create scripts for analysing massive
amounts of data, resulting in more accurate results through advanced computational ap-
proaches [23]. This feature not only speeds up the research process, but it also allows
for more subtle data exploration, leading to findings that would otherwise be impossible
to make using traditional analytical procedures. As a result, including LLMs into data
analysis constitutes a significant step forward in scientific research procedures, suggesting
a future in which data-driven decisions are more accessible and reliable.

2.5 Prompting Techniques

Prompting techniques refer to methods used to instruct or guide large language models
(LLMs) to generate more accurate, relevant, and structured responses. These techniques
help manage the output of LLMs by making the requests clear and specific, ensuring that
the model produces results in the expected format [24]. They are crucial in improving the
performance of LLMs, especially when dealing with complex tasks like code generation,
data analysis, or multi-turn conversations (where a user interacts with the model across
multiple steps).

Several modern day prompting techniques are used to simplify and manage responses
from LLMs. These techniques focus on key aspects that help make the responses more
controllable and also in a format a prompt engineer expects answers so that he can
connect chains with other prompts together.

2.5.1 Various aspects of restricting an LLMs response using prompts

• Tone: [25] The tone of an LLM is crucial because it must adapt to different scenar-
ios. For example, when providing assistive guidance for children, the LLM should
deliver specific and simple responses. In contrast, when assisting nurses or doc-
tors, the LLM should use a more advanced tone suitable for professionals, avoiding
14 Chapter 2. Background

overly simplistic explanations. This principle extends to other domains like finance,
enterprise, schools, and hospitals, where the tone should be adjusted based on the
target audience using the application.

• Format: Format is crucial for prompt engineers. To build applications with com-
plex prompt chains, it’s essential to restrict LLM responses to a specific format,
such as YAML, JSON, bullet points, or code generation. This ensures consistency
and makes it easier to integrate responses into broader workflows.

• There are scenarios where a prompt engineer can apply different types of restrictions
to an LLMs response, such as instructing the LLM to simply say "no" when it lacks
sufficient information or limiting its responses to just "yes" or "no" etc.

2.5.2 Chat and Text completion models

• [26]Text Completion models: These models are designed to complete incom-
plete text. For instance, if you provide the sentence "This is a" to a text completion
model, it will attempt to predict the next words. A possible completion might be
"This is a toy."

• [27]Chat Completion models: Chat completion models, as the name implies,

are designed to complete conversations. These models can facilitate a dialogue
between a human and AI or involve multiple characters interacting. The primary
goal is to carry on and complete the conversation effectively. These models are fine-
tuned to create meaningful exchanges and ensure the conversation flows naturally.
For example, if the input is "Human: Hi," a chat completion model might respond
with "AI: Hi, what help do you need today?". The characters involved can vary
depending on the system’s instructions, but this is a general example.

2.6 Evaluation of LLMs

LLMs have significantly impacted the field of automated code generation, leading to the
development of various benchmarking frameworks designed to assess their effectiveness
and accuracy.
1. HumanEval: HumanEval is designed to evaluate the functional correctness of code
generated by LLMs using a series of programming tasks, each backed by a set of
unit tests. The benchmark assesses the effectiveness of these models by calculating
the proportion of problems for which the model generates code that passes all of the
specified tests. This technique quantifies a model’s capacity to understand and write
functionally correct code, which is crucial for real programming applications [28].

2. EvalPlus: EvalPlus, developed by [29], augments the HumanEval benchmark with

a larger set of test cases supplied by an automated test input generator. This
approach is intended to detect erroneous code outputs more effectively, resulting in
a more rigorous assessment of LLM capabilities in code synthesis.

3. Poly-Coder: Poly-Coder, developed by [30], is a model based on the GPT-2 archi-

tecture with 2.7 billion parameters, tested across multiple programming languages.
It is evaluated using the HumanEval benchmark and outperforms expectations,
2.6. Evaluation of LLMs 15

highlighting the importance of model architecture and parameter size on code gen-
eration effectiveness.

4. Vulnerability Detection Frameworks: These frameworks, such as those studied

by [31], use transformer-based LLMs to identify software vulnerabilities in program-
ming, notably in C/C++ source code. The performance is measured based on the
model’s ability to accurately detect known vulnerabilities from dedicated software
vulnerability datasets.
Chapter 3
Related Work

3.1 LLMs in Data Analytics

LLMs have shown significant potential in improving data analytics by providing improved
capabilities for processing and interpreting text. These models excel at tasks like senti-
ment analysis, subject identification, and keyword extraction, all of which are critical for
gaining insights from unstructured data sources like customer reviews and social media
posts. The integration of LLMs into data analytics workflows enables a more thorough
comprehension of data by bridging the gap between structured and unstructured infor-
mation, increasing corporate intelligence and improving decision-making [32].

The significance of using LLMs in data analysis lies from their transformational ca-
pacity to automate and enhance complex operations that formerly needed significant
manual effort. LLMs, such as GPT-3.5 and GPT-4, have proved their ability to handle
large volumes of textual data, allowing for the extraction of useful insights necessary for
informed decision-making. LLMs improve the efficiency and depth of data analysis by
dealing with unstructured data such as customer feedback, software documentation, and
other text-based material, allowing for the discovery of trends, patterns, and linkages that
traditional methods may miss. This not only accelerates the data analysis process but
also offers a new level of scalability and precision, which is vital in managing enormous
datasets in domains such as software engineering and qualitative research [33].

Moreover, the employment of LLMs in data analysis professions is becoming increas-

ingly crucial since they give significant benefits by automating monotonous operations,
allowing data analysts to focus on more strategic and interpretative aspects of study. The
ability of LLMs to provide context and produce insights from large and complex datasets
makes them essential tools in today’s data-driven environment. LLMs can reduce time
and effort in data analysis, but human analysts still play a crucial role [34]. They bring
domain-specific expertise and the critical thinking skills required to evaluate and alter
LLM outcomes. Combining AI and human expertise produces accurate and actionable
insights, hence increasing the effectiveness of data-driven decision-making.

An advanced system using LLMs such as GPT-3.5 and GPT-4 was introduced to auto-
mate keypoint extraction and relevance evaluation in qualitative data analysis (QDA). In
this context, keypoints relate to critical aspects or concepts within data that are required
for understanding the primary ideas or themes, and are traditionally found through a
time-consuming manual coding process. The system consists of two main components:
the Keypoints Extractor, which uses ChatGPT to generate and refine keypoints from

17
18 Chapter 3. Related Work

textual inputs while managing token limits of the models, and the Keypoints Relevance
Evaluator, which assesses how relevant these keypoints are to the input text and provides
scores to gauge their applicability [35]. This approach demonstrates LLMs’ ability to
automate the extraction and evaluation of significant thematic aspects from qualitative
data, hence streamlining difficult QDA processes.

The integration of LLMs like GPT-3.5 and GPT-4 into scientific research, particu-
larly for automating tasks such as code generation, data analysis, and visualization, has
shown potential to significantly enhance productivity and efficiency. This study rigor-
ously evaluates these models across various coding scenarios, highlighting a considerable
variability in output integrity [33]. The findings indicate that, while LLMs can greatly
reduce manual coding labour, ensuring the quality and reliability of their outputs re-
mains a significant difficulty. This involves the installation of comprehensive evaluation
processes to ensure that outputs satisfy the strict criteria necessary in scientific research,
emphasising the necessity of precision and dependability in the efficient use of LLMs
within scientific workflows.

3.2 LLMs in ITSM

In the domain of IT Service Management (ITSM), LLMs have demonstrated significant
potential for automating cognitive tasks that were previously done by human agents,
hence increasing service delivery efficiency and effectiveness. Research has shown that
LLMs can automate regular tasks like ticket categorisation, incident summarisation, and
even content generation for technical documentation, all of which are key components
of ITSM operations. For example, LLMs such as GPT-4 have been used in technical
customer service contexts to automate lower-level cognitive activities like translation,
summarisation, and question answering, decreasing the stress on human agents and im-
proving response times. For increasingly complicated reasoning tasks, LLMs frequently
need advanced approaches like RAG to ensure accuracy and dependability [36].

The integration of LLMs into ITSM represents a significant advancement in enhancing

the efficiency and effectiveness of service delivery. LLMs are particularly well-suited for
automating routine ITSM tasks such as ticket categorization, incident summarization,
and the generation of technical documentation, which are crucial for maintaining high
service quality. By automating these repetitive procedures, LLMs minimise the workload
on IT service staff, allowing them to concentrate on more complicated and essential is-
sues that require human intervention. As IT environments become more complex, LLMs’
importance in ITSM is projected to grow. They provide scalable and adaptive solutions
that match the dynamic needs of modern IT infrastructures [37].

However, integrating LLMs with existing ITSM frameworks such as ITIL offers several
hurdles that must be overcome in order to fully realise their potential. These frameworks
are complex, necessitating a thorough understanding of both the technical and organisa-
tional aspects of ITSM. To be effective, LLMs must be thoroughly trained and aligned
with these frameworks to guarantee that their outputs are in line with recognised ITSM
practices. Concerns about data privacy and security are also paramount, considering the
vast volumes of sensitive information processed by LLMs. Despite these issues, LLMs in
3.3. Research Gap 19

ITSM offer numerous benefits, including real-time incident resolution, predictive analytics
for proactive service management, and more personalised customer assistance. Effective
management of these difficulties can considerably improve an organization’s IT service
delivery capabilities [38].

Chat-GPT play an important role in IT management curriculum, with an emphasis

on their ability to automate ordinary textual operations and the need of prompt engineer-
ing. It highlights LLMs as both tools for improving operational efficiency and academic
disciplines in IT management education. The difficulties of building successful prompts
to elicit desirable replies from LLMs is underlined, implying that prompt engineering
necessitates certain abilities that students should learn [39]. The method addresses not
just the practical need to incorporate modern AI technology into managerial procedures,
but also the educational implications such as the difficulties in discriminating between
human-generated and LLM-generated information.

The "Chat2Data" system integrates LLMs with RAG and vector databases [40], im-
proving data analysis by tackling frequent difficulties such as hallucination, high opera-
tional expenses, and low accuracy for complex tasks. The system embeds domain-specific
knowledge via RAG to prevent hallucinations, leverages vector databases to limit LLM
interactions, lowering costs and improving performance, and employs a pipeline agent
to break difficult jobs into smaller subtasks for increased accuracy. "Chat2Data" has
been tested in real-world scenarios and can handle both structured and unstructured
data quickly. For structured data, it transforms natural language queries into SQL, using
vector databases to extract relevant domain knowledge and caching frequent queries to
speed up results.

GPT-4 have demonstrated significant potential in automating Technical Customer

Service (TCS) tasks, particularly in areas such as translation, summarization, and content
generation. This automation not only enhances efficiency but also reduces the workload
for human agents, allowing them to focus on more complex issues. However, more ad-
vanced tasks, such as reasoning and contextual understanding, still require sophisticated
approaches like RAG or fine-tuning to achieve effective results. The successful integra-
tion of LLMs into TCS also hinges on the development of robust data ecosystems and
addressing organizational and technical challenges, making them a promising yet complex
tool for improving IT service management processes [36].

3.3 Research Gap

3.3.1 Motivation
The rapid advancements in LLMs have demonstrated significant improvements in pro-
ductivity and efficiency across various sectors, including data analytics, research, and
code generation. However, despite extensive study undertaken in 2024, a significant gap
exists in the evaluation of LLMs in the context of data analytics, particularly in scenarios
requiring repeated, multi-turn interactions. The need to investigate how LLMs can be led
and corrected through dynamic interactions with users is critical, especially when input
is used to modify and steer the model towards more accurate and contextually relevant
20 Chapter 3. Related Work

outputs.

3.3.2 Problem Definition and Formulation

While existing research has explored LLMs’ capabilities in problem-solving, code genera-
tion, and multi-turn conversations, there is a significant lack of focus on their performance
in data analytics tasks that involve interactive feedback loops. These tasks often require
the AI assistant to engage in a dynamic exchange with the user, where the user may
correct or redirect the AI’s output when errors or hallucinations occur. The specific chal-
lenge lies in identifying how effectively LLMs can adjust their responses based on iterative
feedback to ensure the accuracy and relevance of the information generated, particularly
in the domain-specific context of ITSM data analysis.

3.3.3 Contribution
Our research aims to bridge this gap by systematically evaluating various LLMs in data
analytics tasks in a conversation exchange. We focus on identifying models that excel in
scenarios where continuous user input is necessary to refine the AI’s responses, ultimately
reducing hallucinations and enhancing the contextual relevance and accuracy of the gen-
erated outputs. Additionally, we seek to discover effective prompting techniques that can
further guide LLMs in producing desired outputs. The insights gained from this study
will contribute to a deeper understanding of LLMs’ potential and limitations in data
analytics, offering practical applications and strategies for improving their performance
in this critical field.
Chapter 4
Method

4.1 Research Method

To answer the research questions addressed in this thesis, we used a mixed-methods strat-
egy that included both literature reviews and experimental procedures. For Research
Question 1 and Research Question 2 (RQ1 & RQ2), we aim to find the most effective
prompting strategies for improving language models’ performance when generating code
for data analytics. We will identify best practices, assess the effectiveness of various dif-
ferent techniques by doing a thorough evaluation of previous studies. This fundamental
insight will be critical for investigating how language models might use feedback from
tools and users to revise and improve code outputs.

In addition to the literature research, we will conduct empirical testing and validation
of the prompting methodologies. We intend to implement the most promising method-
ologies from the literature and evaluate their impact on language models’ performance in
real-world data analytics applications. By evaluating various strategies in controlled en-
vironments, we hope to provide specific, evidence-based recommendations for improving
LLMs in code generation.

For Research Question 3 (RQ3), we will rely only on experiments to establish the ideal
LLM for multi-turn interactions when developing data analytics code. These studies will
simulate scenarios in which the model iteratively refines its code in response to user feed-
back, simulating real-world applications in ITSM. By carrying out these experiments for
RQ2 and RQ3, we ensure that the theoretical insights gained from the literature study
are carefully evaluated and validated in real-world scenarios.

When handling user feedback loops, we aim to guide the LLM when it generates
hallucinated code or output. If the code contains errors, we provide specific error mes-
sages along with any possible correction suggestions. If the LLM generates irrelevant or
incorrect information, we steer its responses in the right direction by offering targeted
feedback. These feedback loops are designed to test the model’s ability to self-correct
and improve the accuracy of its responses to user queries.

4.2 Literature Review

A Literature Review is a comprehensive summary of previous research on a particular
topic. Its purpose is to summarise and synthesize the evidence on a specific research ques-
tion [41]. There are three steps involved in the process: Plan, Conduct and Document

21
22 Chapter 4. Method

the Review. The planning phase focuses on specifying the research questions, developing
review protocol and validating the review protocol. The conduct phase includes identi-
fying relevant research. Selecting primary studies, extracting and synthesising the data.
Finally, the documentation phase is to write and validate the report [42]. The following
steps are included in our research for performing literature review.

4.2.1 Research Questions

The literature review aims to answer RQ1, where it mainly focuses on identifying the
most effective prompting strategies that improve the performance of LLMs in generating
code for data analytic tasks.

4.2.2 Search Strategy

Conducting an extensive search for relevant literature across various databases and plat-
forms using carefully selected search terms and search strings.

1. The data sources that we considered for finding the relevant literature are IEEE
Xplore, Google Scholar, Scienedirect, Semantic Scholar, Research Gate, and ACM
Digital Library.

2. The search terms that we have considered are Prompt Engineering, LLMs, Code
generation, Data Analytics and Machine Learning

3. The search strings that we have considered to find the appropriate literature are as
follows:

• "Prompt engineering AND Large Language Models"

• "Large Language Models AND Machine Learning AND Code generation"
• "Prompting Techniques AND Large Language Models"
• "Prompt Engineering AND Data Analytics"
• "(Large Language Models) OR (Code generation) AND Data Analytics"
• "Large Language Models AND Code generation"

4.2.3 Inclusion and Exclusion Criteria

[Link] Inclusion Criteria

• Focused on the use of LLMs for code generation in data analytics.

• Explored different prompting strategies and their impact on model performance.

• Included empirical evaluations of LLMs with detailed metrics on accuracy, rele-

vance,and user satisfaction.
4.3. Experimentation 23

[Link] Exclusion Criteria

• Studies that did not focus on LLMs or code generation.

• Research that did not provide empirical data or was purely theoretical without
validation.

• Articles published more than five years ago unless they were seminal works directly
influencing current research.

4.2.4 Quality Assessment Criteria

The quality of the selected studies was assessed based on

1. The robustness of the methodologies used in the studies, including the design of
experiments and the statistical analysis of results.

2. The applicability of the findings to the research question, particularly in the context
of data analytics and code generation.

3. The significance of the study’s contributions to the field, as measured by citations

and the practical implementation of the proposed strategies.

4.2.5 Data Extraction Strategy

The data extraction strategy is focused on

1. Prompting Strategies: The specific techniques used in formulating prompts,

including examples of effective prompts and the use of context and domain-specific
language.

2. Applications: Examples where these strategies were applied in real-world scenar-

ios, and the outcomes observed.

4.2.6 Data Synthesis

The data synthesis procedure in this literature review involves collecting and summaris-
ing the findings from the selected studies, with an emphasis on how different prompting
strategies affect the performance of LLMs in code generation for data analysis. The
synthesis will categorize the results based on which strategies are most effective in spe-
cific contexts, such as those that improve iterative code refinement or those that are
more suited to multi-turn interactions. This detailed analysis provides a comprehensive
overview of the current state of research and identifies areas where further exploration is
needed, serving as a foundation for the experimental phase of the research.

4.3 Experimentation
Experimental methodology, a research approach involves manipulating one variable to
determine if it can cause an effect on another variable, within a controlled environment.
The main objective of the experiment is to evaluate the hypothesis, through using various
24 Chapter 4. Method

multivariate statistical techniques. When conducting an experiment we study the out-

comes by varying the input variables in a process. Dependent and Independent variables
are those the experimental process focuses on. The experimental process includes for-
mulating an idea, planning, analysis and interpretation and presentation of experimental
results [43]. The experimental procedure to be followed is as follows.

4.3.1 Goal
The goal of the experiment is to evaluate the effectiveness of various prompting strategies
and LLMs to generate accurate and relevant code for data analytics based on ITSM
Incident data. The experiment attempts to determine the most effective strategies for
improving LLM performance, particularly in multi-turn interactions where user feedback
is used to iteratively enhance LLM answers.

4.3.2 Research Question

The experiment is designed to address RQ2 and RQ3.

4.3.3 Hypothesis
Hypothesis for RQ2

• H0: The prompting strategies identified significantly improve the performance of

LLMs in generating accurate and relevant code for data analytics.

• H1: The prompting strategies identified does not significantly improve the perfor-
mance of LLMs in generating accurate and relevant code for data analytics.

Hypothesis for RQ3

• H0: There is no significant difference in performance among different LLMs in

multi-turn interactions for code generation in data analytics tasks.

• H1: There is a significant difference in performance among different LLMs in multi-

turn interactions.

4.3.4 Dependent and Independent Variables

[Link] Dependent Variables
• Correctness: The correctness of the code generated

• Relevance: The degree to which the refined code meets the specific ITSM task
requirements.

• Clarity: The degree to which the code generated is clear i.e., well strutted amd
easy to understand.

[Link] Independent Variables

• The LLM used
4.3. Experimentation 25

4.3.5 Dataset Description

The dataset used in this research was obtained from Kaggle, specifically an open-source IT
Service Management (ITSM) incident log dataset. It comprises 141,712 incident records
and 36 columns, with each record representing a unique incident within the ITSM system.
To enhance clarity and ensure transparency, the key attributes of the dataset are detailed
below, explaining their relevance and how they were applied in this study. A snippet of
the dataset is provided in Figure 4.1.

Each row in the dataset corresponds to a single IT incident. The columns capture
various attributes of each incident, which are essential for understanding the status,
severity, and management of these incidents within an IT service environment. These
attributes include:

• Incident Number: A unique identifier for each incident.

• State: The current state of the incident (e.g., Open, Closed, In Progress).

• Priority, Impact, and Urgency: Fields that classify incidents based on their
severity and business impact, helping prioritize them for resolution.

• Timestamp Columns: These capture important lifecycle data, such as the time
the incident was opened, assigned, resolved, and closed.

• Assignment and Resolution Details: The individuals responsible for opening,

resolving, or handling the incident.

• Category and Subcategory: These fields classify incidents into broader cat-
egories (e.g., Network, Software, Hardware) and subcategories for more granular
analysis.

• SLA Indicators: Fields indicating whether incidents were resolved within the
agreed Service Level Agreement (SLA), reflecting the efficiency of incident manage-
ment.

• Reassignment Count and Reopen Count: Indicators of how many times an

incident was reassigned or reopened, which helps track the complexity and quality
of the incident resolution process.

4.3.6 LLMs Selected

[Link] LLAMA 3.1:latest
1
LLAMA 3.1, with 8.03 billion parameters, is a little smaller but more efficient variant of
the larger LLAMA models. It preserves the LLAMA series’ key characteristics, including
high performance in text production, multi-turn dialogue management, and reasoning
tasks. The lower parameter count improves computing efficiency while remaining capable
of sophisticated tasks such as code generation, text completion, and data analysis [44].
LLAMA 3.1 models are optimised for a variety of NLP tasks, including creating coherent
1
LLAMA 3.1:latest
26 Chapter 4. Method

Figure 4.1: Snippet of the dataset

text, assisting with coding, and comprehending complex user enquiries. Its training data
contains large-scale multilingual datasets, providing it an advantage in managing a wide
range of linguistic inputs and outputs.

[Link] GEMMA2 9B
2
GEMMA 2 is based on the science behind Google’s Gemini models. It has 2 billion
and 7 billion parameters, making it lighter than other LLMs, but well optimised for
tasks like instruction following, factual correctness, and coding [45]. The model was
fine-tuned using supervised fine-tuning (SFT) and reinforcement learning from human
feedback (RLHF), which improved its ability to generate accurate code and solve technical
challenges. GEMMA 2 has been evaluated in benchmarks such as HumanEval, where it
performed well in code creation.

[Link] PHI3:14B
3
PHI-3 is a very capable transformer-based model with 14 billion parameters. It has
been optimised for jobs such as multilingual code generation and advanced reasoning.
Phi-3 models are lightweight but powerful, with considerable gains in code generation
performance across a variety of programming environments. This model is suitable for
several use cases, including text creation and programming operations [46]. PHI3’s per-
formance has been examined using benchmarks such as HumanEval and PIQA, and it
has proved its capacity to generate code and solve complicated problems.

4.3.7 Experimental Setup

This section details the experimental setup required to test the proposed hypothesis. It
includes an overview of the necessary hardware and software, along with a basic expla-
nation of the experiment’s design.
2
GEMMA2
3
PHI3:14B
4.3. Experimentation 27

[Link] Hardware Specifications:

To run a LLMs with billions of parameters, a PC with strong performance is essential.
This section provides details on the hardware specifications used in the experiment, fo-
cusing on the key components: the processor, RAM, and graphics card.

Component Specification
CPU 11th Gen Intel(R) Core(TM) i9-11950H @
2.60GHz
(8 Cores, 16 Threads)
RAM 64 GB
GPU NVIDIA RTX A3000 6 GB Graphics
(3840 CUDA Cores)
Storage 1TB SSD

Table 4.1: Hardware Specifications

[Link] Software specifications

The software environment of our project has several specific requirements. These require-
ments ensure that all details are replicated exactly as they were during our experiments.

Component Specification
Operating System Windows 10
Programming Language Python 3.12.3 4
Important Libraries Langchain5
OpenAI6
Streamlit7
OLLAMA8
Pandas9
Transformers10
Docker11
Tornado12
Development Environment VS Code IDE 1.91.1
Other Tools OLLAMA
Docker

Table 4.2: Software Specifications

4
Python Documentation
5
Langchain Guide
6
OpenAI
7
Streamlit Documentation
8
OLLAMA Documentation
9
Pandas Documentation
10
Hugging Face Transformers
11
Docker Documentation
12
Tornado Documentation
28 Chapter 4. Method

4.3.8 Implementation
For this experimentation there are 3 main important [Link] UI component
is used to connect chain of prompts together and also interact with code execution
component as well as LLM component. You can refer to Figure 4.2 for high level
overview of how all of there components work together.

• LLM Component: The LLM Component serves as the backbone of our appli-
cation, enabling powerful language processing and data analysis capabilities. Cur-
rently, we’ve used OLLAMA for infering LLMs. The architecture is flexible, allowing
easy integration with other API-based LLMs such as ChatGPT, Gemini, Claude,
or any other leading LLMs (For the current experiment we are using only open
source models.) . This adaptability ensures that application can leverage the latest
advancements in language model technology to deliver optimal performance.

• UI Component: The user interface (UI) is where users engage with the appli-
cation. It offers an environment for uploading data files and interacting with the
LLM. Users can ask questions related to their datasets, and the UI facilitates easy
communication between the user and the LLM, making data analysis accessible
even to those without technical expertise. The main purpose of this UI interface is
to to create abstraction of underlyign prompts. We have designed this experiment
in such a way that no only we connect components interact but also connect chain
of prompts in such a way that we can control inputs and outputs.

• Code Execution API Component: This component manages the execution

of code generated by the LLM. Direct execution of such code can introduce bugs
and pose safety risks. To address potential issues like system failures or security
vulnerabilities, our approach prioritizes both safety and scalability. We employ
a Docker-based Jupyter kernel execution environment, which securely isolates the
execution process within a container, reducing risks. This setup is scalable through
Docker, enabling the application to support multiple users concurrently. The Code
Execution API is built with scalability in mind, adhering to industry best practices
and integrating components from the specified Git repository.

4.3.9 Evaluation
We have curated a dataset of questions that will be presented to each LLM sequen-
tially. If an LLM generates irrelevant or incorrect answers, we provide guiding feedback
to assist in correcting its output. These questions are designed to mimic real-world sce-
narios, comparing how a human might approach a task versus how an LLM would solve
the same problem. We have established specific guidelines to evaluate the LLMs’ outputs.

RQ2: To determine the most effective prompting technique, we can evaluate them
using a set of curated questions which were mentioned before. The assessment focuses
on a single LLM and begins by importing the dataset on which we plan to perform data
[Link] we pose a question to an LLM, the first crucial step is importing the
necessary data into the environment. If this step fails, the entire data analytics task
cannot proceed, making it a key criterion for evaluating the effectiveness of prompting
techniques. Successful data import is essential, as it allows the LLM to perform further
4.3. Experimentation 29

Figure 4.2: Architecture of Experiment

analysis and answer questions based on the data. We assess the prompting technique
based on this criteria.

RQ3 : Given that we are assessing the LLMs’ data analytics capabilities in multi-
turn conversations with a feedback loop, traditional automated assessment tools are in-
adequate. Therefore, this evaluation is conducted manually, as no existing frameworks
can handle this task. The LLMs’ outputs are assessed based on the following criteria:
• Correctness: The correctness score is assigned to the overall output of a conver-
sation according to these rules:
– 5 - Completely Correct: The response is entirely accurate, aligns perfectly
with known facts or data, and has no errors.
– 4 - Mostly Correct: The response is largely accurate, with minor errors or
omissions that do not significantly impact the overall correctness.
– 3 - Partially Correct: The response contains a mix of accurate and inac-
curate information, with the correct information being more prominent, but
some important details are missing or incorrect.
– 2 - Mostly Incorrect: The response is largely inaccurate, with only a few
correct details that do not substantially affect the overall wrongness.
– 1 - Completely Incorrect: The response is entirely wrong, with no correct
information or a complete misinterpretation of the question.
• Relevance: The relevance score is assigned to the overall output of a conversation
according to these rules:
– 5 - Highly Relevant: The response is fully aligned with the question’s intent,
directly addressing the main point without any deviation.
– 4 - Mostly Relevant: The response is generally aligned with the question’s
intent but includes some minor irrelevant details or misses some nuances.
30 Chapter 4. Method

– 3 - Moderately Relevant: The response addresses the question but with

some significant irrelevant content or omits essential aspects of the query.
– 2 - Slightly Relevant: The response only tangentially addresses the question,
with most of the content being irrelevant or off-topic.
– 1 - Completely Irrelevant: The response does not address the question at
all and focuses entirely on unrelated information.

• Clarity: The clarity score is assigned to the overall output of a conversation

according to these rules:

– 5 - Very Clear: The response is easy to understand, well-structured, and

free of ambiguity, making the information accessible and straightforward.
– 4 - Mostly Clear: The response is generally clear but may have minor issues
with wording, structure, or slight ambiguities that do not significantly affect
understanding.
– 3 - Moderately Clear: The response is understandable, but there are no-
ticeable issues with structure, wording, or clarity that make it harder to follow.
– 2 - Slightly Clear: The response is somewhat difficult to understand, with
poor structure, unclear wording, or significant ambiguity that obscures the
meaning.
– 1 - Very Unclear: The response is highly confusing, poorly structured, or
completely ambiguous, making it difficult or impossible to understand.

The combined score of clarity, relevance, and correctness indicates which model per-
forms best.

4.3.10 Confounding Factors

In this experiment involving data analytics on ITSM data using LLMs, several confound-
ing factors were identified and they are:

• Variability in LLM performance: Different LLMs may have differences in per-

formance due to their training data, architectures, or fine-tuning processes. This
variability could lead to inconsistent results when comparing models.

• Contextual drift: In multi-turn conversations, LLMs may experience contextual

drift, where they lose track of the initial context or focus on irrelevant details. This
can affect the clarity and relevance of their responses and may not be uniformly
managed across different models.

• Manual Evaluation Subjectivity: The manual scoring of correctness, relevance,

and clarity is inherently subjective. Different evaluators might assign different
scores for the same output, leading to inconsistencies in the results.

• Overfitting to feedback: Repeatedly providing feedback to correct LLM re-

sponses could lead to overfitting, where the model begins to generate responses that
overly conform to the feedback, rather than genuinely understanding the problem.
4.4. Threats to Validity 31

4.4 Threats to Validity

4.4.1 Internal Validity
1. Bias in Manual Evaluation: Manual evaluation of LLM outputs presents a
risk of evaluator bias, potentially affecting the consistency and fairness in scoring
correctness, relevance, and clarity.
Mitigation: To minimize this risk, we have established clear, objective scoring
criteria and provide evaluators with specific examples for each level of scoring. This
will help maintain consistency and reduce the impact of individual biases during
evaluation.

2. Context Loss in Multi-turn Conversations: In multi-turn conversations, LLMs

may lose the context of previous interactions, leading to incorrect or irrelevant re-
sponses.
Mitigation: We have carefully structured the conversation flow by designing
prompts that reinforce previous context at critical points. Additionally, we will
limit the conversation length to prevent drift and ensure each conversation is fo-
cused on a specific task or question

4.4.2 External Validity

1. Generalizability of Dataset: The dataset used in this experiment, focused on
ITSM incident data, may not be representative of other domains, which could limit
the applicability of the results to different datasets or scenarios.
Mitigation: We acknowledge this limitation by clearly stating that our results are
specific to ITSM data. Future research will explore different datasets to test the
generalizability of the findings across multiple domains.

2. Limited Model Selection: The experiment focuses on a specific set of LLMs,

which may not reflect the performance of other models, especially those that are
not open-source or that have different architectures.
Mitigation: We will document this limitation and clarify that our results are
specific to the LLMs used in the experiment.

4.4.3 Construct Validity

1. Subjective Definitions of Key Metrics: The constructs of correctness, rele-
vance, and clarity are subjective, and different evaluators may interpret these terms
differently, leading to variability in scoring.
Mitigation: We will clearly define each metric and ensure evaluators have access
to a detailed rubric that describes the scoring levels. We will also conduct a pilot
phase where evaluators score a set of outputs to align their understanding of the
evaluation criteria.

2. Prompt Construction Variability:Small differences in how prompts are con-

structed might influence LLM performance, which could introduce variability into
the results.
Mitigation: We will use consistent prompts across all models and tasks to reduce
32 Chapter 4. Method

variability. All LLMs will receive the same prompt structures, ensuring that the
performance differences are due to the models themselves rather than variations in
prompting.

4.4.4 Conclusion Validity

1. Small Sample of Questions: A small set of curated questions might not be suf-
ficient to capture the full range of LLM capabilities, which could limit the validity
of the conclusions drawn from the experiment.
Mitigation: We will diversify the types of questions used in the experiment to in-
clude both simple and complex tasks. By testing the models on a range of scenarios,
we ensure that the results better reflect the LLMs overall performance.

2. Lack of Automated Evaluation Tools: Relying solely on manual evaluation

could introduce noise or inconsistencies, as manual evaluations can be prone to hu-
man error.
Mitigation: Although we are not using automated evaluation tools, we will mit-
igate this by implementing careful review processes. For example, evaluators will
review their scores after completing each session to catch any inconsistencies, and
we will cross-check results periodically to ensure accuracy.

4.5 Validity and reliability of approach

The design of this experiment ensures a robust approach to evaluating the effectiveness of
LLM-based data analytics systems. Validity is addressed by creating a multi-component
system that mirrors how LLMs are applied in real-world scenarios, incorporating UI
interaction, prompt management, and secure code execution. The experiment focuses on
important performance factors like how correct, relevant, and clear the LLM outputs are.
This makes sure that the results show real abilities and not just test conditions.
We carefully structure the interactions between the UI, LLM, and code execution
components, ensuring that the experiment isolates the LLM’s performance without intro-
ducing external biases. By using an open-source models and Docker-based Jupyter kernel
environments for code execution, we reduce system-specific dependencies that could skew
results. Furthermore, the use of curated datasets and consistent prompting techniques
ensures that the LLM’s performance is evaluated.
Reliability is maintained through several measures, including using standardised eval-
uation metrics—correctness, relevance, and clarity—across all models tested. Each con-
versation with the LLM is scored based on fixed criteria, limiting the variability in as-
sessment.
Chapter 5
Results and Analysis

The results and analysis discussed in this section are intended to address the research
issues that underlie this study. The first two research questions (RQ1 and RQ2) seeks
to determine which prompting tactics optimise LLM performance when producing code
for data analytics. It also examines how these ideas are used in real-world data analytics
systems. The research question (RQ3) looks at which LLM does better in multi-turn
interactions for code generation, especially when user feedback is continuously added to
create conversations and guide the model towards making more accurate and relevant
results.

5.1 Literature Review Findings

Using a literature review to answer the RQ1, is a thorough and evidence-based ap-
proach for determining the most effective methodologies. By systematically reviewing
and analysing relevant papers, the Literature Review ensures that the findings are based
on a thorough understanding of current research.

Figure 5.1: Distribution of the articles over the years

Figure 5.1 shows the trend of published articles from 2019 to 2024. Starting with
one article in 2019, publications will gradually rise over the next few years, with two
articles in 2020 and three in 2021. Between 2021 and 2022, the number of papers grows
dramatically, reaching seven, indicating greater research interest. This trend continues
into 2023, with 13 papers produced, indicating a significant increase. By 2024, the number

33
34 Chapter 5. Results and Analysis

of publications had reached a peak of 16, suggesting a continuous and sustained upward
trend in research output over the period.

Figure 5.2: Number of Articles Published by Source

From Figure 5.2. With 16 published articles, Google Scholar is the leading source of
research on this topic. Semantic Scholar and Research Gate are close behind, with nine
and ten articles, respectively. ACM Publication and Hugging Face have a small presence,
with six and one articles, respectively.

Figure 5.3: Distribution of Prompting Strategies by Number of Studies

Figure 5.3 depicts the distribution of prompting strategies based on the number of ar-
ticles studied in the literature. It shows that zero-shot prompting is the most researched
strategy, with 15 papers, highlighting its importance for tasks that require models to
generalise without examples. 13 studies have closely followed Chain of Thought (CoT)
prompting, highlighting its effectiveness in improving logical reasoning and complex prob-
lem solving. Priming/Seed prompting, which is explored in nine studies, demonstrates
5.1. Literature Review Findings 35

a great interest in guiding models with specific inputs to provide contextually relevant
results.

Table 5.1 summarises the most successful prompting strategies for enhancing LLM
performance in data analysis code generation. Based on a literature review , the table
lists different strategies, such as zero-shot and self-consistency prompting, that have been
shown to make generated code more correct, logical, and useful. The papers included in
this review were chosen based on how they used these strategies in LLM applications.

Prompting Strat-
Articles Definition
egy
This technique asks the model to do
a task without providing any examples
Zero shot [47–55] , [56–59] or prior context. The model is meant
to understand the task just from the
prompt and produce an answer.
In this strategy, the model is given
a few instances (usually two or five)
within the prompt to help it respond.
Few Shot [26, 60–64] , [58] The examples help the model under-
stand the necessary format and con-
text, which improves its ability to com-
plete the task.
This technique involves prompting the
model to generate intermediate reason-
ing steps before arriving at the final an-
Chain of Thought [55,65] , [66–72] , [58,
swer. By breaking down the thought
(CoT) 59, 73]
process, the model performs better on
tasks that require logical reasoning or
complex problem-solving.
In this strategy, the model is given a
seed or "priming" input that contains
Priming or Seed information that directs it to a specific
[74–82]
Prompting task or output. The prompt serves as a
cue for the model to provide more rele-
vant responses.
This prompting strategy is designed to
improve the reasoning performance of
Self Consistency
[83–87] LLMs while performing complex tasks
Prompting
that demand logical thinking or various
reasoning paths.

Table 5.1: Overview of Different Prompting Strategies

Based on the literature review, we selected the Zero-shot, Chain of Thought (CoT),
and Priming/Seed prompting strategies for their distinct advantages in increasing the
performance of LLMs for code generation in data analysis. Zero-shot prompting allows
the model to generalise completely from the prompt, eliminating the need for previous
examples and making it suitable for tasks with no training data available. Chain of
Thought prompting improves the model’s capacity to handle complex problem-solving
36 Chapter 5. Results and Analysis

and logical thinking by breaking down tasks into intermediate steps, which is essential
for producing coherent and accurate code in multi-step data analysis operations. Prim-
ing/Seed prodding provides the model with an initial input or context, ensuring that
the output is appropriate and aligned with the specified task or domain, hence improv-
ing code generation precision. Together, these techniques offer a balanced approach for
assessing generalisation, increasing reasoning, and maintaining contextual accuracy.

5.2 Experiment Findings

5.2.1 Results and Analysis of RQ2

Figure 5.4: The working of prompting strategies

To address RQ2, we conducted a series of experiments using the LLAMA 3.1 model,
where, as illustrated in Figure 5.4, we provided the model with a task using three distinct
strategies—zero-shot prompting, priming, and chain-of-thought prompting—based on the
user prompt, "I want to see monthly trends of incidents," tasking the assistant with
generating the necessary Python code to import the dataset, process the time information,
and create a plot of monthly trends. We experimented with different questions from the
curated questions dataset and the following are our observations from the generated
responses from LLM.
• Zero-Shot Prompting: In this instance, the task was presented to the LLM
without any prior or guidance. The model was intended to interpret the task and
generate the appropriate Python code using just its own knowledge and compre-
hension of the request. This strategy relies solely on the LLM’s prior knowledge to
do the work.
• Priming: In contrast to zero-shot prompting, the priming strategy involved pro-
viding additional context to guide the model. Before the model was prompted, a
5.2. Experiment Findings 37

user-forged prelude was included, such as: "Let’s import the data before we start the
analysis and generate a plot to check the monthly trend." This crafted context was
designed to influence the LLM’s subsequent responses, ensuring that the generated
Python code aligned with the conversation’s tone and direction.

• Chain-of-Thought Prompting: This strategy encouraged the model to break

down the problem into logical steps by guiding it with partial reasoning, such as:
"Let’s think this through step by step." The model was guided through the rea-
soning process, allowing it to approach the task in a systematic manner by first
importing the data, then converting the time format, and finally producing a plot.
This method intended to improve the model’s ability to process tasks in a more
systematic manner.

Across these three scenarios, the nature of the output generated by the LLM varies
significantly. In zero-shot prompting, the LLM relies entirely on its learnt information,
frequently generating wide or generalised responses. Priming, on the other hand, directs
the model towards a more contextually appropriate and cohesive continuation of the con-
versation. Meanwhile, chain-of-thought prompting encourages the model to break down
the problem and address each step independently, resulting in a more logical and reasoned
output. These variations highlight the impact of different prompting strategies on the
quality and nature of the LLM’s completions.

Figure 5.5: Comparison of Zero-Shot, Priming, and Chain-of-Thought outputs

Figure 5.5 shows that the outcomes from different strategies. differ dramatically. Dur-
ing zero-shot prompting, the model failed to load the dataset and repeatedly provided
38 Chapter 5. Results and Analysis

messages such as "Dataset not found." While the generated code was generally relevant
and syntactically acceptable, the lack of dataset handling prevented task completion. This
approach also resulted in broad, generalised results that lacked the specificity needed to
solve the problem completely.

In priming, the results were more consistent and relevant. The inclusion of a user-
crafted prelude helped guide the model towards generating contextually appropriate
Python code. While some dataset import issues persisted, the responses were notably
more aligned with the task. The model was able to produce correct and coherent outputs
for data processing and visualisation when given this initial direction.

In chain-of-thought prompting, the model was encouraged to reason through each step
methodically. Despite following a logical structure in the generated code, the dataset im-
port issue still posed a challenge. However, this strategy often led to more structured
and step wise outputs, as the LLM attempted to solve the problem one component at a
time.

Considering all of this information, we can say that priming technique works better
in this scenario as this drives LLM to generate context relevant text and it also helps it
in initiating the conversation.

5.2.2 Results and Analysis of RQ3

To address RQ3, we developed a curated set of questions designed to be applied to ITSM
incident data. These questions were created to reflect scenarios where leadership teams
are likely to consult a data analyst for insights. The set also includes questions that
cover exploratory data analysis, data cleaning, creating visuals and hypothesis testing
such that these mirror the type of inquiries a data analyst would typically perform. For
this purpose we have taken mainly 3 models and they are LLAMA3.1-8b, GEMMA2-9b,
PHI3-14b here "b" stands for the number of billion parameters a model is trained upon.
All of these models are "4_k" quantized models.

Quantization of models, especially LLMs, is a technique used to reduce the compu-

tational complexity and memory footprint by representing model weights and activation
in lower bit formats (e.g., 4-bit or 8-bit) instead of the usual 32-bit or 16-bit floats. This
compression is useful for faster inference, lower energy consumption, and easier deploy-
ment on edge devices without sacrificing much accuracy. [88]. For the evaluation in RQ2,
we utilized priming as our primary prompting method. This decision was based on the
findings from RQ1 and RQ2, where priming proved to be the most effective technique for
initially loading data into the environment.

The results for LLAMA-3.1 can be found in Tables 5.3, 5.4, 5.5, and 5.6. For the
results of PHI3 14B, refer to Tables 5.7, 5.8, 5.9, and 5.10. The results for GEMMA2 are
presented in Tables 5.11, 5.12, 5.13, and 5.14.

During the evaluation, it was found that both the LLAMA3 and GEMMA2 models
performed well, despite using 4k quantization, with no issues in text generation or irreg-
ularities. In contrast, the PHI3 model displayed significant irregularities, including the
5.2. Experiment Findings 39

addition of gibberish text and random letters in the generated output, which were not
expected. This hindered PHI3’s performance, even though the PHI3-14B model is larger
compared to the LLAMA3.1-8B and GEMMA2-9B models. Despite its potential, PHI3’s
output was comparable to LLAMA and GEMMA, and could have yielded better results
under different conditions. Due to constraints, only 4k quantized models were used for
experiment.

Model-> LLAMA3.1-8b GEMMA2-9b PHI3-14b

Q Type Corr Rel Clr Corr Rel Clr Corr Rel Clr
Descriptive 3.2 4 4.2 4.4 5 4.4 3.6 4.4 4.4
Stat Test 1 1.75 2 1 1.5 1.5 1 2.25 2.25
Graph Plot 2.6 3.2 2.8 1.8 2 1.8 3 3.2 3.2
Data Clean 3.4 3.8 3.6 3.6 4.2 4 3.2 3.8 3.6
Avg 2.96 2.93 3.15

Table 5.2: Average scores for different question types.

Based on the scores presented in Table 5.2, it is evident that PHI3-14b outperforms
both the GEMMA2-9B and LLAMA3.1-8B models. PHI3-14b has the highest overall
average score (3.15) across all question types. This is followed closely by LLAMA3.1-8b
(2.96) and GEMMA2-9b (2.93), with GEMMA2-9b performing slightly lower overall.

However, it’s important to note that none of these models performed particularly well
on Statistical Test questions. All models struggled with these questions, with scores not
exceeding 3. In addition, they also encountered challenges when it came to generating
plots and handling data cleaning tasks, achieving only moderate success in these areas.

Focusing specifically on correctness, GEMMA2-9b consistently achieves high scores,

particularly excelling in Descriptive and Data Cleaning tasks. In contrast, both LLAMA3.1-
8b and PHI3-14b perform similarly, with PHI3-14b showing higher correctness for Sta-
tistical questions. LLAMA3.1-8b, meanwhile, performs well in Graph Plot and Data
Cleaning tasks, though its overall correctness is slightly lower.

When analysing relevance, GEMMA2-9b leads the way across most question types,
especially excelling in Descriptive and Data Cleaning questions. PHI3-14b performs par-
ticularly well in Graph Plot and Statistical questions, while LLAMA3.1-8b remains com-
petitive, though not as dominant in any specific category.

In terms of clarity, both PHI3-14b and GEMMA2-9b demonstrate strong perfor-

mances, particularly in Descriptive and Data Cleaning questions, where their scores are
nearly identical. LLAMA3.1-8b lags slightly behind in terms of clarity but still performs
reasonably well, especially in Graph Plot and Descriptive tasks.
 Table 5.3: Results of Descriptive questions with LLAMA 3.1:latest

40
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant analysed the trend of IT inci-
Whether the number of dents by converting the ’opened_at’ column
incidents has increased, to datetime format and grouping the inci-
What is the trend in the
decreased, or remained dents by month. It provided a line plot show-
number of IT incidents 2 3 5
the same over the past ing a general downward trend in the number
over the past 6 months?
6 months, with the pos- of incidents over the past 6 months. After
sibility of visualisation. correcting the date time conversion error, the
final output suggested a decline in incidents.
Identification of specific
times (hours, days, or The assistant successfully identified and plot-
What are the peak months) when the most ted the distribution of incidents over time, fo-
5 5 4
times for IT incidents? incidents occur, likely cusing on peak periods like days and hours,
requiring a graphical by grouping data appropriately.
analysis.
Determine the typi-
cal time it takes to The assistant attempted to calculate the av-
resolve an IT inci- erage resolution time but encountered multi-
What is the average res-

Chapter 5. Results and Analysis

dent, which might ple errors due to missing columns. The final
olution time for IT inci- 1 3 3
involve computing the response indicated that the ’created_at’ col-
dents?
difference between umn did not exist, so the average resolution
the opened_at and time could not be computed.
resolved_at columns.
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant initially used the
Determine the fre- ’u_priority_confirmation’ column but
quency of incidents switched to the ’priority’ column after user
What is the distribution across different severity clarification. It successfully plotted the dis-
of incident severity lev- levels, potentially re- tribution of incident severity levels, showing 5 4 5
els? quiring a bar chart or the frequency of incidents based on priority.
countplot for visualiza- The final output was a correct plot of the
tion. severity distribution using the appropriate
column.
The assistant filtered incidents based on
whether they met the SLA requirement
This outcome will indi-
What percentage of and calculated the percentage of resolved
cate the proportion of
incidents are resolved incidents. Initially, it incorrectly used
incidents that meet SLA 3 5 4
within SLA time sys_mod_count, but after feedback from the
(Service Level Agree-
frames? user, it revised the calculation. The final out-
ment) requirements.
put indicated that 19.43% of incidents were
resolved within SLA time frames.

41
 Table 5.4: Results of Statistical questions with LLAMA 3.1:latest

42
Expected Out- Correctness
Question LLM Generated Summary Relevance Clarity
come
The assistant attempted to test
the hypothesis by calculating the
"Mean Time to Resolution
MTTR for each team. Ini-
(MTTR) Comparison:
tially, there was an error in
Question: Is there a significant differ-
parsing the datetime columns,
ence in the MTTR between incidents
A valid statistical but it was resolved by apply-
handled by different teams (e.g., Team
test either prov- ing the errors=’coerce’ parame-
A vs. Team B)?
ing or contradict- ter. The assistant also faced an 1 3 2
Hypothesis:
ing the null hy- issue with missing or incorrect
Null Hypothesis (H0): The mean
pothesis. column names (team). After a
MTTR is the same for all teams.
correction, it provided a struc-
Alternative Hypothesis (H1): The
ture for performing an ANOVA
mean MTTR is different between the
analysis, although it did not ul-
all teams."
timately execute the test due to
further column-related issues.
"Incident Resolution Time:
Weekdays vs. Weekends:"

Chapter 5. Results and Analysis

Question: Is the average resolution
time for incidents reported on week- The assistant successfully identi-
A valid statistical
days significantly different from those fied and plotted the distribution
test either prov-
reported on weekends? of incidents over time, focusing
ing or contradict- 1 2 2
Hypothesis: Null Hypothesis on peak periods like days and
ing the null hy-
(H0): The average resolution time is hours, by grouping data appro-
pothesis.
the same on weekdays and weekends. priately.
Alternative Hypothesis (H1): The
average resolution time is different
between weekdays and weekends."
 5.2. Experiment Findings
Expected Out- Correctness
Question LLM Generated Summary Relevance Clarity
come
The assistant started by im-
"Impact of Incident Volume on
porting the dataset and prepar-
Resolution Time:"
ing the relevant columns (such
Question: Does a higher volume of in-
as opened_at and resolved_at).
cidents significantly increase the aver-
A valid statistical It then attempted to categorize
age resolution time?
test either prov- incidents by whether they oc-
Hypothesis:
ing or contradict- curred on a weekday or week- 1 1 2
Null Hypothesis (H0): There is no
ing the null hy- end using the datetime module.
relationship between incident volume
pothesis. The assistant calculated the res-
and resolution time. Alternative Hy-
olution time by subtracting the
pothesis (H1): Higher incident vol-
opened_at timestamp from the
ume leads to a significant increase in
resolved_at timestamp, con-
resolution time.
verting the difference into hours.
"Correlation Between Incident
Severity and Resolution Time:"
Question: Is there a significant corre- The assistant started by import-
lation between the severity of incidents ing the dataset and preparing
A valid statistical
and the time taken to resolve them? the relevant columns (such as
test either prov-
Hypothesis: opened_at and resolved_at. It
ing or contradict- 1 1 2
Null Hypothesis (H0): There is then attempted to categorize in-
ing the null hy-
no correlation between incident sever- cidents by whether they occurred
pothesis.
ity and resolution time. Alternative on a weekday or weekend using
Hypothesis (H1): There is a signifi- the datetime module.
cant correlation between incident sever-
ity and resolution time."

43
 Table 5.5: Results of Graph plot questions with LLAMA 3.1:latest

44
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant grouped in-
What are different categories cidents by categories (e.g.,
in the dataset and how does Identifying different categories incident_state) and pro-
the distribution of incident in the dataset and plot the dis- duced a bar plot of incident
5 5 5
count in each categories look tribution of each category by counts. It also added more
in the dataset? Give your an- incident count details by analyzing active,
swes as plot. reassignment_count, and
reopen_count.
A plot is likely to show that
some categories have a longer
The assistant attempted to
median resolution time with a
What is the average time to calculate and plot the aver-
wider interquartile range, in-
resolution for incidents across age resolution time but en-
dicating more variability. On 1 3 1
different categories? Give countered errors due to non-
the other hand, less com-
your answer as a plot numeric and incorrectly for-
plex issues should have shorter
matted date time values.
and more consistent resolution
times.

Chapter 5. Results and Analysis

Time series line graph should
reveal trends such as an in- The assistant grouped inci-
How does the volume of inci- crease in incident volume dur- dents by month, created a
dents vary over time (e.g., by ing certain periods (e.g., end monthly incident count using
5 5 4
month or quarter)? Give your of fiscal quarters or during resample, and plotted a line
answer as a plot. major software updates) and graph showing incident vol-
a possible decrease during oth- ume over time.
ers (e.g., holiday seasons).
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant attempted to
Scatter plot might show a pos- calculate the correlation be-
Is there a correlation between
itive correlation where higher tween priority and resolution
the priority of incidents and
priority incidents generally time but faced issues due
the time taken to resolve 1 1 1
have shorter resolution times, to non-numeric values in the
them? Give your answer as a
indicating that critical issues priority column. As a result,
plot.
are resolved faster. the final plot could not be gen-
erated.
Stacked bar chart should dis-
play that certain departments,
like IT or Security, deal with
The assistant tried
How does the severity of inci- more severe incidents (e.g.,
grouping by the non-
dents vary across different de- "High" or "Critical" severity),
existent department col- 1 2 3
partments? Give your answer while others, like Human Re-
umn, then switched to
as a plot. sources, might predominantly
assignment_group.
handle less severe incidents
(e.g., "Low" or "Medium"
severity).

45
 Table 5.6: Results of Data Cleaning Questions with LLAMA 3.1:latest

46
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant successfully
The LLM should identify
identified different formats
all variations in the inci-
Are there any inconsistencies in the priority column and
dent priority labels and rec-
or variations in how incident standardised the labels (re-
ommend standardising them
priorities are labelled (e.g., moving prefixes and ensuring
(e.g., converting all to a com-
"High" vs. "Hgh"), and consistent labels like ’Criti- 3 4 3
mon format such as "High,"
how can these be standardized cal’, ’High’, ’Moderate’, and
"Medium," "Low"). This en-
to ensure uniformity in data ’Low’). However, the output
sures that priority levels are
analysis? had minor issues in execution,
consistent across the dataset
impacting correctness and
for accurate analysis.
clarity.
The LLM should detect un- The assistant attempted to
resolved incidents that have check the opened_at and
Are there any unresolved in- been open for an unusually closed_at columns for unre-
cidents in the dataset that long time, indicating potential solved incidents exceeding a 7-
have abnormally long dura- errors in data entry or sys- day threshold. Due to errors 1 3 3
tions, and how can these be tem tracking. It should sug- in datetime conversion and fil-

Chapter 5. Results and Analysis

identified and corrected? gest setting these to a default tering, it provided incomplete
value or flagging them for re- results, affecting correctness
view. and clarity.
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant accurately
converted the opened_at,
The LLM should identify du-
resolved_at, and closed_at
Are there any duplicate inci- plicates with varying resolu-
columns to the correct
dent records where the inci- tion times and provide strate-
datetime format, resolving
dent details match but the res- gies for resolving these dis- 5 4 4
inconsistencies that would
olution times differ, and how crepancies, such as averaging
otherwise impact time-based
should these be handled? the resolution times or select-
calculations. The explanation
ing the most frequent value.
was clear and the corrections
were well-executed.
The LLM should detect in- The assistant correctly iden-
consistencies in date and time tified incidents with high
Are there any inconsistencies formats that could affect the reassignment_count and
in the date and time formats calculation of time duration reopen_count, suggesting to
within the dataset that could (e.g., the difference between review the history of these in-
3 3 3
lead to incorrect time duration incident creation and resolu- cidents and look for patterns.
calculations for incidents, and tion times). It should convert Although the approach was
how can these be corrected? all date and time fields to a appropriate, minor clarity
standard format to ensure ac- issues were observed in the
curate calculations. explanation.
The assistant successfully
Identifies incidents where
identified no unresolved inci-
the reassignment_count or
Are there any incidents dents exceeding the duration
reopen_count is far above the
with unusually high threshold, and the suggestions
normal range. Suggest setting
reassignment_count or to correct future issues were 5 5 5
a reasonable threshold for
reopen_count, and how clear and comprehensive. The
review and flagging incidents
should these be addressed? explanation was well-received,
with values exceeding this
leading to full marks in all
threshold.
categories.

47
 Table 5.7: Results of Descriptive questions with PHI3 14b

48
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant correctly ana-
Whether the number of in- lyzed the trend of incidents,
What is the trend in the num- cidents has increased, de- noting an increase in March
ber of IT incidents over the creased, or remained the same followed by a gradual de- 4 5 5
past 6 months? over the past 6 months, with crease. The analysis was
the possibility of visualization. mostly accurate and clearly
presented.
The assistant provided the
Identification of specific times correct peak times, particu-
(hours, days, or months) larly for hours, but missed
What are the peak times for
when the most incidents oc- some depth in the analysis of 3 4 3
IT incidents?
cur, likely requiring a graph- peak days and months. The
ical analysis. output lacked some clarity in
explaining the results.
Determine the typical time it The assistant attempted to
takes to resolve an IT in- calculate the average resolu-
What is the average resolution cident, which might involve tion time but failed due to for-
1 3 4

Chapter 5. Results and Analysis

time for IT incidents? computing the difference be- matting issues. Despite clear
tween the opened_at and re- explanation, the output was
solved_at columns. incomplete and incorrect.
The assistant successfully cal-
Determine the frequency of in-
culated that 93.5% of inci-
cidents across different sever-
What is the distribution of in- dents were resolved within
ity levels, potentially requir- 5 5 5
cident severity levels? SLA timeframes. The re-
ing a bar chart or countplot
sult was correct, relevant, and
for visualization.
clearly presented.
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant accurately pro-
duced a bar plot showing
This outcome will indicate
What percentage of incidents the distribution of incidents
the proportion of incidents
are resolved within SLA time- by severity, with "Medium" 5 5 5
that meet SLA (Service Level
frames? severity being the most com-
Agreement) requirements.
mon. The output was correct,
relevant, and clear.

Table 5.8: Results of Statistical questions with PHI3 14b

Expected Out- Correctness

Question LLM Generated Summary Relevance Clarity
come
"Mean Time to Resolution
(MTTR) Comparison:
Question: Is there a significant differ-
ence in the MTTR between incidents
The assistant suggested using
handled by different teams (e.g., Team A valid statistical
ANOVA to compare MTTR be-
A vs. Team B)? test either proving
tween teams but encountered 1 4 3
Hypothesis: or contradicting the
date formatting errors that pre-
Null Hypothesis (H0): The mean null hypothesis.
vented the statistical test.
MTTR is the same for all teams.
Alternative Hypothesis (H1): The
mean MTTR is different between the
all teams."

49
 50
Expected Out- Correctness
Question LLM Generated Summary Relevance Clarity
come
"Incident Resolution Time:
Weekdays vs. Weekends:"
Question: Is the average resolution
time for incidents reported on week-
The assistant proposed using a t-
days significantly different from those A valid statistical
test but encountered datetime is-
reported on weekends? test either proving
sues that prevented the necessary 1 2 2
Hypothesis: Null Hypothesis or contradicting the
calculation. The statistical anal-
(H0): The average resolution time is null hypothesis.
ysis was incomplete
the same on weekdays and weekends.
Alternative Hypothesis (H1): The
average resolution time is different
between weekdays and weekends."
"Impact of Incident Volume on
Resolution Time:"
Question: Does a higher volume of in-
cidents significantly increase the aver-
The assistant suggested using
age resolution time? A valid statistical
correlation analysis but failed to
Hypothesis: test either proving
handle the necessary datetime 1 2 2
Null Hypothesis (H0): There is no or contradicting the

Chapter 5. Results and Analysis

conversion and data aggregation,
relationship between incident volume null hypothesis.
leading to an incomplete result.
and resolution time. Alternative Hy-
pothesis (H1): Higher incident vol-
ume leads to a significant increase in
resolution time.
 5.2. Experiment Findings
Expected Out- Correctness
Question LLM Generated Summary Relevance Clarity
come
"Correlation Between Incident
Severity and Resolution Time:"
Question: Is there a significant corre-
The assistant attempted to map
lation between the severity of incidents
severity to numeric values and
and the time taken to resolve them? A valid statistical
perform a correlation analysis
Hypothesis: test either proving
but encountered data formatting 1 1 2
Null Hypothesis (H0): There is or contradicting the
errors that prevented the execu-
no correlation between incident sever- null hypothesis.
tion of the statistical test. The
ity and resolution time. Alternative
result was incomplete.
Hypothesis (H1): There is a signifi-
cant correlation between incident sever-
ity and resolution time."

Table 5.9: Results of Graph plot questions with PHI3 14b

Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
What are different cate-
The assistant successfully gen-
gories in the dataset and
Identifying different categories erated a bar plot showing
how does the distribu-
in the dataset and plot the dis- the distribution of incidents
tion of incident count in 5 5 5
tribution of each category by across categories such as inci-
each categories look in
incident count dent state. The output was
the dataset? Give your
correct, relevant, and clear.
answes as plot.

51
 52
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
A plot is likely to show that
some categories have a longer The assistant encountered er-
What is the average median resolution time with a rors with datetime formats
time to resolution for in- wider interquartile range, in- and failed to generate the cor-
cidents across different dicating more variability. On rect plot. Despite multiple at- 1 2 2
categories? Give your the other hand, less com- tempts, the analysis remained
answer as a plot plex issues should have shorter incomplete, and the explana-
and more consistent resolution tion was unclear.
times.
Time series line graph should
The assistant proposed group-
reveal trends such as an in-
How does the volume of ing incidents by month to
crease in incident volume dur-
incidents vary over time create a time-series plot but
ing certain periods (e.g., end
(e.g., by month or quar- failed to handle datetime for- 1 1 2
of fiscal quarters or during
ter)? Give your answer mats correctly. The final plot
major software updates) and
as a plot. was not generated, and the ex-
a possible decrease during oth-
planation lacked clarity.
ers (e.g., holiday seasons).
After resolving initial errors,
Is there a correlation Scatter plot might show a pos- the assistant successfully cre-

Chapter 5. Results and Analysis

between the priority of itive correlation where higher ated a scatterplot showing the
incidents and the time priority incidents generally correlation between priority
5 4 3
taken to resolve them? have shorter resolution times, and resolution time. The final
Give your answer as a indicating that critical issues output was correct, though
plot. are resolved faster. the explanation was initially
unclear.
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
Stacked bar chart should dis-
play that certain departments,
like IT or Security, deal with The assistant successfully gen-
How does the severity
more severe incidents (e.g., erated a plot showing how
of incidents vary across
"High" or "Critical" severity), severity varies by department.
different departments? 3 4 4
while others, like Human Re- The output was correct, but
Give your answer as a
sources, might predominantly minor clarity issues affected
plot.
handle less severe incidents the overall explanation
(e.g., "Low" or "Medium"
severity).

Table 5.10: Results of Data Cleaning Questions with PHI3 14b

Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The LLM should identify
Are there any inconsis- all variations in the inci-
tencies or variations in dent priority labels and rec- The assistant attempted to
how incident priorities ommend standardizing them standardize labels but encoun-
are labeled (e.g., "High" (e.g., converting all to a com- tered execution errors due to
1 2 2
vs. "Hgh"), and how mon format such as "High," incorrect column names. The
can these be standard- "Medium," "Low"). This en- explanation lacked clarity, and
ized to ensure unifor- sures that priority levels are the output was incomplete.
mity in data analysis? consistent across the dataset
for accurate analysis.

53
 54
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The LLM should detect un-
Are there any unre- resolved incidents that have
The assistant successfully
solved incidents in been open for an unusually
identified unresolved inci-
the dataset that have long time, indicating potential
dents with durations over
abnormally long du- errors in data entry or sys- 5 5 5
7 days. The output was
rations, and how can tem tracking. It should sug-
accurate, relevant, and clearly
these be identified and gest setting these to a default
explained.
corrected? value or flagging them for re-
view.
Are there any dupli- The LLM should identify du-
cate incident records plicates with varying resolu- The assistant correctly iden-
where the incident de- tion times and provide strate- tified duplicate incidents but
tails match but the res- gies for resolving these dis- did not explain clearly how to 3 4 3
olution times differ, and crepancies, such as averaging handle the differences in reso-
how should these be the resolution times or select- lution times.
handled? ing the most frequent value.
The LLM should detect in-
Are there any inconsis- consistencies in date and time
tencies in the date and formats that could affect the

Chapter 5. Results and Analysis

The assistant successfully
time formats within the calculation of time duration
identified and corrected date
dataset that could lead (e.g., the difference between
inconsistencies, providing a 4 5 5
to incorrect time dura- incident creation and resolu-
clear explanation of the issue
tion calculations for in- tion times). It should convert
and how it was resolved.
cidents, and how can all date and time fields to a
these be corrected? standard format to ensure ac-
curate calculations.
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
Identifies incidents where
The assistant correctly identi-
Are there any incidents the reassignment_count or
fied incidents with unusually
with unusually high reopen_count is far above the
high counts but encountered
reassignment_count normal range. Suggest setting
a typo when printing the re- 3 3 3
or reopen_count, and a reasonable threshold for
sults, leading to a minor error.
how should these be review and flagging incidents
The explanation was clear but
addressed? with values exceeding this
affected by this error.
threshold.

Table 5.11: Results of Descriptive questions with GEMMA2 9b

Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
Whether the number of in- The assistant generated a line
What is the trend in the num- cidents has increased, de- plot showing the trend over
ber of IT incidents over the creased, or remained the same the past six months, revealing 5 5 4
past 6 months? over the past 6 months, with the pattern of incident volume
the possibility of visualization. over time.
Identification of specific times The assistant successfully
(hours, days, or months) identified peak times for
What are the peak times for
when the most incidents oc- incidents using time-related 5 5 5
IT incidents?
cur, likely requiring a graph- features and provided the
ical analysis. analysis clearly and concisely.
Determine the typical time it The assistant eventually cal-
takes to resolve an IT in- culated the average resolution
What is the average resolution cident, which might involve time but faced several issues
3 5 3
time for IT incidents? computing the difference be- with date formatting and er-
tween the opened_at and re- ror correction, which affected
solved_at columns. the clarity of the explanation.

55
 56
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
The assistant correctly calcu-
Determine the frequency of in-
lated the distribution of inci-
cidents across different sever-
What is the distribution of in- dents by severity levels using
ity levels, potentially requir- 4 5 5
cident severity levels? the impact column and pro-
ing a bar chart or countplot
vided a clear explanation and
for visualization.
visual representation.
The assistant correctly calcu-
This outcome will indicate lated the percentage of in-
What percentage of incidents
the proportion of incidents cidents resolved within SLA
are resolved within SLA time 5 5 5
that meet SLA (Service Level time frames, providing an ac-
frames?
Agreement) requirements. curate, relevant, and clearly
presented result.

Table 5.12: Results of Statistical questions with GEMMA2 9b

Expected Out- Correctness

Question LLM Generated Summary Relevance Clarity
come
"Mean Time to Resolution

Chapter 5. Results and Analysis

(MTTR) Comparison:
Question: Is there a significant differ-
ence in the MTTR between incidents
The assistant attempted to calcu-
handled by different teams (e.g., Team A valid statistical
late MTTR but faced date con-
A vs. Team B)? test either proving
version issues. An ANOVA test 1 3 3
Hypothesis: or contradicting the
was not completed due to incor-
Null Hypothesis (H0): The mean null hypothesis.
rect team identification.
MTTR is the same for all teams.
Alternative Hypothesis (H1): The
mean MTTR is different between the
all teams."
 5.2. Experiment Findings
Expected Out- Correctness
Question LLM Generated Summary Relevance Clarity
come
"Incident Resolution Time:
Weekdays vs. Weekends:"
Question: Is the average resolution
time for incidents reported on week-
Errors occurred during datetime
days significantly different from those A valid statistical
conversion, and no conclusive re-
reported on weekends? test either proving
sult was generated. The attempt 1 1 1
Hypothesis: Null Hypothesis or contradicting the
to categorize incidents into week-
(H0): The average resolution time is null hypothesis.
days and weekends failed.
the same on weekdays and weekends.
Alternative Hypothesis (H1): The
average resolution time is different
between weekdays and weekends."
"Impact of Incident Volume on
Resolution Time:"
Question: Does a higher volume of in-
cidents significantly increase the aver-
The assistant grouped incidents
age resolution time? A valid statistical
by time periods but failed to cal-
Hypothesis: test either proving
culate resolution time correctly, 1 1 1
Null Hypothesis (H0): There is no or contradicting the
leading to no valid analysis or
relationship between incident volume null hypothesis.
conclusion.
and resolution time. Alternative Hy-
pothesis (H1): Higher incident vol-
ume leads to a significant increase in
resolution time.

57
 58
Expected Out- Correctness
Question LLM Generated Summary Relevance Clarity
come
"Correlation Between Incident
Severity and Resolution Time:"
Question: Is there a significant corre-
lation between the severity of incidents
The assistant attempted the
and the time taken to resolve them? A valid statistical
analysis but encountered errors
Hypothesis: test either proving
with date parsing and could not 1 1 1
Null Hypothesis (H0): There is or contradicting the
calculate the correlation between
no correlation between incident sever- null hypothesis.
severity and resolution time.
ity and resolution time. Alternative
Hypothesis (H1): There is a signifi-
cant correlation between incident sever-
ity and resolution time."

Table 5.13: Results of Graph plot questions with GEMMA2 9b

Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
What are different cate-

Chapter 5. Results and Analysis

gories in the dataset and The assistant successfully gen-
Identifying different categories
how does the distribu- erated a bar plot and provided
in the dataset and plot the dis-
tion of incident count in a clear explanation of how in- 5 5 5
tribution of each category by
each categories look in cidents are distributed across
incident count
the dataset? Give your categories.
answers as plot.
 5.2. Experiment Findings
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
A plot is likely to show that
some categories have a longer
What is the average median resolution time with a
The assistant encountered er-
time to resolution for in- wider interquartile range, in-
rors with date parsing and
cidents across different dicating more variability. On 1 2 2
failed to generate the correct
categories? Give your the other hand, less com-
plot.
answer as a plot plex issues should have shorter
and more consistent resolution
times.
Time series line graph should
reveal trends such as an in-
How does the volume of
crease in incident volume dur- The assistant encountered is-
incidents vary over time
ing certain periods (e.g., end sues processing time data,
(e.g., by month or quar- 1 1 1
of fiscal quarters or during leading to a failure in produc-
ter)? Give your answer
major software updates) and ing the correct plot.
as a plot.
a possible decrease during oth-
ers (e.g., holiday seasons).
Is there a correlation Scatter plot might show a pos-
between the priority of itive correlation where higher The assistant encountered er-
incidents and the time priority incidents generally rors due to missing or incor-
1 1 1
taken to resolve them? have shorter resolution times, rect data, leading to no plot
Give your answer as a indicating that critical issues being generated.
plot. are resolved faster.

59
 60
Correctness
Question Expected Outcome LLM Generated Summary Relevance Clarity
Stacked bar chart should dis-
play that certain departments,
like IT or Security, deal with
How does the severity
more severe incidents (e.g., The assistant failed to gener-
of incidents vary across
"High" or "Critical" severity), ate a plot due to a KeyError,
different departments? 1 1 1
while others, like Human Re- caused by an incorrect column
Give your answer as a
sources, might predominantly reference.
plot.
handle less severe incidents
(e.g., "Low" or "Medium"
severity).

Table 5.14: Results of Data Cleaning Questions with GEMMA2 9b

LLM Generated Correctness

Question Expected Outcome Relevance Clarity
Summary
The LLM should identify
all variations in the inci-
Are there any inconsisten-
dent priority labels and rec-
cies or variations in how in- The assistant identified

Chapter 5. Results and Analysis

ommend standardizing them
cident priorities are labeled inconsistencies and
(e.g., converting all to a com-
(e.g., "High" vs. "Hgh"), and standardized the prior- 5 5 5
mon format such as "High,"
how can these be standardized ity labels to a consistent
"Medium," "Low"). This en-
to ensure uniformity in data format.
sures that priority levels are
analysis?
consistent across the dataset
for accurate analysis.
 5.2. Experiment Findings
LLM Generated Correctness
Question Expected Outcome Relevance Clarity
Summary
The LLM should detect un-
resolved incidents that have
Are there any unresolved in- been open for an unusually The assistant calculated
cidents in the dataset that long time, indicating potential the average duration
have abnormally long dura- errors in data entry or sys- and flagged incidents 4 5 5
tions, and how can these be tem tracking. It should sug- with abnormally long
identified and corrected? gest setting these to a default duration.
value or flagging them for re-
view.
The LLM should identify du-
Are there any duplicate inci- plicates with varying resolu- The assistant struggled
dent records where the inci- tion times and provide strate- with the execution but
dent details match but the res- gies for resolving these dis- eventually confirmed 3 4 2
olution times differ, and how crepancies, such as averaging that no duplicate
should these be handled? the resolution times or select- records were found.
ing the most frequent value.
The LLM should detect in-
consistencies in date and time
The assistant iden-
Are there any inconsistencies formats that could affect the
tified inconsistencies
in the date and time formats calculation of time duration
in date formats but
within the dataset that could (e.g., the difference between
encountered difficulties 2 2 3
lead to incorrect time duration incident creation and resolu-
in converting mixed
calculations for incidents, and tion times). It should convert
formats, affecting the
how can these be corrected? all date and time fields to a
outcome.
standard format to ensure ac-
curate calculations.

61
 62
LLM Generated Correctness
Question Expected Outcome Relevance Clarity
Summary
Identifies incidents where
the reassignment_count or The assistant success-
Are there any incidents
reopen_count is far above the fully identified incidents
with unusually high
normal range. Suggest setting with unusually high
reassignment_count or 4 5 5
a reasonable threshold for counts and provided a
reopen_count, and how
review and flagging incidents clear explanation and
should these be addressed?
with values exceeding this recommendations.
threshold.

Chapter 5. Results and Analysis

Chapter 6
Discussion

6.1 Discussion for RQ1 and RQ2

RQ1: "Which prompting strategies most effectively in generating code for data analysis?"
RQ2: "How do these prompting strategies experimentally validated to determine the op-
timal approach for refining and enhancing model outputs?"

In our process of determining which prompting strategies most effectively enhance the
performance of LLMs in generating code for data analysis, we drew insights from both
a thorough literature review and experimental validation. Through our experiments, we
observed significant differences in output quality and relevance when employing zero-shot
prompting, priming, and chain-of-thought prompting with the LLAMA 3.1 model.

6.1.1 Zero-Shot Prompting

When we tested zero-shot prompting, both our literature review and the results of our
experiments showed that the model could generate syntactically correct code, but it had
trouble with more difficult tasks, especially those that needed external datasets or multi-
ple steps. The literature supports our findings, as it highlights how zero-shot prompting
often leads to broad, generalised outputs. The model relies solely on its pre-trained
knowledge, without additional contextual guidance. In our experiments, we repeatedly
encountered errors such as “Dataset not found,” which prevented the model from complet-
ing tasks successfully. We found that this strategy is less effective for tasks that require
domain-specific context or multi-step reasoning.

Our literature review (Figure 5.3) revealed that zero-shot prompting is one of the most
explored tactics, with 15 studies on the topic. While zero-shot prompting is effective for
tasks that require LLMs to generalise without examples, both the literature and our
experiments demonstrated that it is ineffective for tasks involving multi-step processes,
such as data analysis, because it does not provide the necessary guidance for LLMs to
generate precise or contextually relevant outputs.

6.1.2 Priming
In contrast, we discovered that priming was the most successful method for this task.
Our literature study and experimental results both show that priming is successful at
leading the model to generate contextually relevant and coherent outputs. We were able

63
64 Chapter 6. Discussion

to direct the model towards creating accurate and task-relevant Python code by pro-
viding an initial user-crafted context—for example, instructions to import the dataset
before proceeding with the analysis. Priming produced more consistent outcomes than
zero-shot prompting, as illustrated in Figure 5.5. The model handled dataset imports
more efficiently and created consistent code for time conversion and data visualisation.

Our literature review revealed that priming or seed prompting is less explored than
zero-shot or chain-of-thought prompting (covered in 9 papers as shown in Figure 5.3), but
we found that priming gives considerable advantages when precision and task alignment
are crucial. In data analysis tasks where context is important, the prelude provided in
priming plays a pivotal role in enhancing the relevance and specificity of the model’s
output. This strategy is especially effective in ensuring that the generated code adheres
to the specific domain requirements, addressing the limitations encountered in zero-shot
prompting.

6.1.3 Chain-of-Thought Prompting

We also explored chain-of-thought prompting, which encourages the model to break down
a problem into logical steps. Our experiments showed that this strategy improved struc-
tured reasoning during code generation. The model produced more methodical, stepwise
outputs when employing chain-of-thought prompting. However, like zero-shot prompt-
ing, it struggled with dataset imports. Our findings suggest that while chain-of-thought
prompting enhances reasoning and stepwise problem-solving, it may not fully address
technical challenges related to data handling in complex, multi-step tasks.

Our literature review, which included 13 studies as shown in 5.3 on chain-of-thought

prompting, supports the notion that this strategy is effective in improving logical rea-
soning and handling complex problem-solving tasks. Our experimental results confirmed
these findings, showing that chain-of-thought prompting is useful for generating coher-
ent reasoning steps. However, its application to code generation, particularly for tasks
requiring external data management, remains limited. We believe that combining chain-
of-thought prompting with other strategies, such as priming, may optimize performance
in data analysis scenarios.

The experiment results support H0, which says that prompting strategies make LLMs
much better at generating correct and useful code for data analysis. Both priming and
chain-of-thought prompting showed clear improvements in leading the model to outputs
that were relevant to the data analytic task. Specifically, priming led to the most suc-
cessful outputs in generating code for dataset imports and visualisations, aligning the
model’s responses closely with the task’s specific requirements.

Zero-shot prompting, while effective for general tasks, consistently failed in the context
of data analytics, as the model was unable to handle the more complicated, multi-step
processes required for data analysis. Chain-of-thought prompting increased structured
thinking, but it fell short of meeting external data requirements. These findings support
H0 by indicating that prompting strategies, particularly priming, are effective in refining
and improving the model’s outputs for data analysis tasks.
6.2. Discussion for RQ3 65

On the contrary, the findings disprove H1 (the hypothesis that prompting strategies
do not significantly improve performance). The distinct differences between zero-shot
prompting and more structured prompting strategies like as priming and chain-of-thought
demonstrate the importance of these techniques in improving the model’s ability to gen-
erate accurate and contextually relevant code.

In general, these results strongly support priming as the best prompting approach
in this scenario. Priming not only supplies the necessary contextual information, but
it also assists the model in initiating conversation and aligning its outputs with the
task’s domain-specific requirements. While chain-of-thought prompting helps with logical
thinking, it appears that combining it with priming may increase performance in multi-
step data processing tasks.

6.2 Discussion for RQ3

RQ3: "Which LLM demonstrates superior performance in multi-turn interactions for
code generation in data analytics and builds the conversation to analyze data based on
user feedback?"

In this section, we break down the results based on the question types, following the
provided format. The discussion analyzes the performance of the models LLAMA3.1-8B,
GEMMA2-9B, and PHI3-14B across four categories of questions: Descriptive, Statistical,
Graph Plot, and Data Cleaning. We also include visual representations from the graphs
for each question type to support the analysis.

6.2.1 Descriptive Questions

In the Descriptive Questions, which focus on identifying trends, peak times, and average
resolution times, we observe that GEMMA2-9B consistently outperforms the other mod-
els in correctness, relevance, and clarity, as shown in Figure 6.1. The nature of these tasks
typically involves recognising patterns over time, where GEMMA2-9B demonstrates clear
strengths.

GEMMA2-9B scores the highest in correctness (4.4), excelling in tasks such as trend
analysis and peak time identification. For instance, in response to the question "What
is the trend in the number of IT incidents over the past 6 months?", GEMMA2-9B pro-
vides a detailed analysis of an upward trend in March followed by a gradual decline. The
model’s ability to parse and interpret time-series data is a key factor in its success here.

PHI3-14B, although strong in clarity, struggles with precision. We notice significant

difficulties with tasks like calculating the average resolution time, where the model fails
to handle the timestamp columns (opened_at and resolved_at) correctly, leading to in-
complete results.

LLAMA3.1-8B performs moderately, scoring 3.2 in correctness. It handles simpler

descriptive tasks adequately, such as identifying peak times for incidents, but struggles
with more complex operations like date arithmetic.
66 Chapter 6. Discussion

Figure 6.1: Average scores for Descriptive Questions

6.2.2 Statistical Questions

In the Statistical Questions category, which involves hypothesis testing and statistical
computations like ANOVA, we observe significant challenges across all models. None of
the models are able to perform complex statistical operations effectively.

PHI3-14B slightly outperforms the others with a correctness score of 1.5 as shown in
Figure 6.2, but it still struggles to handle complex statistical functions. In the question
"Is there a significant difference in Mean Time to Resolution (MTTR) between teams?",
PHI3-14B attempts to perform an ANOVA test but fails due to data formatting errors.

Both GEMMA2-9B and LLAMA3.1-8B score 1 in correctness, showing limited ability

to handle tasks such as correlation analysis or hypothesis testing. For example, in "Is
there a correlation between incident severity and resolution time?", both models fail to
calculate the correlation correctly due to issues with parsing numeric values.

We find that statistical questions requiring multi-step calculations are beyond the
current capabilities of these models. Issues such as improper parsing of numeric val-
ues and errors in applying statistical functions suggest that the models need significant
improvements in handling structured mathematical operations.

Figure 6.2: Average scores for Statistical Questions

6.2. Discussion for RQ3 67

6.2.3 Graph Plot Questions

In Graph Plot Questions, LLAMA3.1-8B delivers the best performance, with scores of 2.6
for correctness, 3.2 for relevance, and 2.8 for clarity (Figure 6.3). GEMMA2-9B, however,
struggles significantly in this area, particularly in correctness, where it scores just 1.8.

For the question "What are the different categories in the dataset, and how does their
distribution look?", LLAMA3.1-8B and PHI3-14B successfully generates a bar plot that
accurately represents the distribution of incidents across categories. This demonstrates
the models ability to handle categorical data and produce simple visualisations effectively.

GEMMA2-9B’s difficulties likely arise from challenges in handling numerical data

and format parsing, which lead to errors when plotting incident counts and trends.
LLAMA3.1-8B provides more balanced results but still fails in more complex graph plot-
ting tasks, such as correlating priority and resolution times.

Overall, PHI3-14B’s strength lies in its ability to interpret categorical data and gener-
ate basic visualisations. However, all models show limitations when it comes to handling
more complex plots that involve multiple variables or statistical overlays, such as those
involving correlations between factors like priority and resolution times.

Figure 6.3: Average scores for Graph Plot Questions

6.2.4 Data Cleaning Questions

The results for Data Cleaning Questions (Figure 6.4 show GEMMA2-9B leading with
high scores for correctness (3.6), relevance (4), and clarity (3.6). The model effectively
identifies and resolves data inconsistencies, such as mismatched labels or date formats,
and detects unresolved incidents or duplicates.

LLAMA3.1-8B and PHI3-14B also perform well, but they face challenges in handling
complex data cleaning tasks like managing duplicate records or converting inconsistent
date formats. LLAMA3.1-8B scores higher in clarity compared to PHI3-14B, suggesting
better communication of results, even when technical execution is not as accurate.

Overall, GEMMA2-9B excels in dataset cleaning and ensuring data uniformity, while
the other models show room for improvement in handling these tasks.
68 Chapter 6. Discussion

Figure 6.4: Average scores for Data Cleaning Questions

Based on the analysis across all question types, the results support the hypothesis that
there is a significant difference in the performance of LLMs in multi-turn interactions for
code generation. GEMMA2-9B consistently outperforms the other models, particularly in
Descriptive and Data Cleaning tasks, while LLAMA3.1-8B excels in Graph Plotting tasks.

For Descriptive and Data Cleaning Questions, GEMMA2-9B proves to be the most
effective. It provides accurate, relevant, and clear outputs in response to multi-turn inter-
actions. Its ability to refine its output based on user feedback and handle complex data
operations such as trend analysis and data cleaning confirms its superiority in these areas.

In terms of Graph Plot Questions, PHI3-14B demonstrates strength in generating

visual outputs and interpreting simple datasets. However, both LLAMA3.1-8B and
GEMMA2-9B encounter more difficulties in numerical and graphical tasks, with LLAMA3.1-
8B clearly leading in this domain.

For Statistical Questions, none of the models performed well, indicating a significant
gap in their ability to handle statistical analysis tasks, such as hypothesis testing or
ANOVA. This highlights an area where all models need further development.

The fact that GEMMA2-9B exhibits superior performance in multi-turn interactions

for the majority of tasks, particularly descriptive and data cleaning, with LLAMA3.1-8B
excelling in graphical analysis, supports the hypothesis (H1). The Null Hypothesis (H0)
is rejected, as there are clear differences in the LLMs’ performance across different tasks.

6.3 Potential Drawbacks of LLMs in ITSM

While LLMs like PHI3-14B, GEMMA2-9B, and LLAMA3.1-8B show potential for en-
hancing ITSM workflows, they face several limitations in real-world applications:
• Scalability Issues: As tasks grow in complexity, such as statistical computations
or graph plotting, these models struggle to scale effectively. LLMs are primarily
optimized for textual analysis rather than precise numeric computation or complex
data visualizations, which limits their use in large-scale ITSM operations.
• Model Bias: Like all machine learning models, LLMs are susceptible to inherent
biases based on the data they are trained on. In ITSM contexts, if models are
6.3. Potential Drawbacks of LLMs in ITSM 69

trained on biased datasets, they could unintentionally favor certain types of inci-
dents, resolutions, or even categories of tickets, which might affect decision-making
in unpredictable ways.

• Data Security and Privacy Concerns: ITSM systems handle vast amounts of
sensitive data, and employing LLMs raises concerns about data security. The use
of LLMs could lead to potential data leakage, where sensitive information may be
exposed through model outputs, particularly if the models are trained or deployed
in environments where data privacy protocols are insufficient.

• Overfitting and Generalization: LLMs, when fine-tuned for specific tasks like
ITSM data analysis, may risk overfitting to certain patterns within the incident
data, making them less effective at handling novel or unique incidents that do not
follow these patterns. This limitation could lead to inaccurate predictions or faulty
analysis in situations involving unusual or highly dynamic ITSM environments.
Chapter 7
Conclusions and Future Work

7.1 Summary of the findings

In this thesis, we investigated the effectiveness of different LLMs in generating code for
data analysis tasks through multi-turn interactions. We also explored various prompting
strategies to enhance model performance. Based on the findings from Chapter 6 (Results
and Analysis) and Chapter 7 (Discussions), we have drawn the following conclusions.

7.1.1 Prompting Strategies for Code Generation

We identified three key prompting strategies—Zero-Shot, Priming, and Chain-of-Thought
Prompting—and evaluated their effectiveness through both literature review and exper-
imental validation.
• Zero-Shot Prompting: This strategy often resulted in generalised and broad
outcomes. While the model generated syntactically correct code, it struggled with
more complex tasks involving external data and multi-step reasoning. Due to a lack
of additional context, the model struggled to deliver particular and relevant results.
• Priming: This strategy emerged as the most effective for guiding the model in gen-
erating task-relevant code. By providing initial context, priming significantly im-
proved the relevance and accuracy of the generated Python code. It was particularly
useful for tasks that required dataset handling, data imports, and domain-specific
operations.
• Chain-of-Thought Prompting: This strategy improved structured reasoning
and logical step-by-step processes in code generation. However, it faced challenges
with handling complicated data-related tasks such as importing datasets and per-
forming multi-step operations. We believe that integrating Chain-of-Thought with
Priming can increase performance on certain activities.
Our results effectively support the conclusion that structured prompting strategies
improve LLM performance in generating code for data analytics. Priming was notably
effective in providing contextual guidance, allowing the models to better align their out-
puts with task-specific requirements. The findings revealed that prompting strategies
play an important role in refining and improving model output.

7.1.2 LLM Performance in Code Generation

We compared the performance of LLAMA3.1-8B, GEMMA2-9B, and PHI3-14B across
different data analytics tasks and found significant differences between these models. We

71
72 Chapter 7. Conclusions and Future Work

evaluated their performance across four question types: Descriptive, Statistical, Graph
Plot, and Data Cleaning.
• Descriptive and Data Cleaning Tasks: GEMMA2-9B consistently outper-
formed the other models in terms of correctness, relevance, and clarity. It was
particularly effective in handling trend analysis, peak time identification, and re-
solving data inconsistencies. The model’s ability to refine its outputs based on user
feedback made it the most suitable for multi-turn interactions.
• Graph Plot Tasks: PHI3-14B excelled in tasks related to graphical output gen-
eration and interpreting simpler datasets. It demonstrated strong performance in
generating visualizations and handling basic data plotting tasks. However, it strug-
gled with more complex plotting tasks, such as correlating multiple variables.
• Statistical Tasks: None of the models performed well in handling statistical anal-
ysis, such as hypothesis testing, ANOVA, or correlation calculations. This indicates
a significant gap in their ability to manage advanced statistical reasoning. Our re-
sults suggest that LLMs need further development to handle statistical tasks more
effectively.

Task Model Performance

Descriptive Tasks GEMMA2-9B > PHI3-14B > LLAMA3.1-8B
Statistical Analysis PHI3-14B> LLAMA3.1-8B> GEMMA2-9B
Graph Plotting PHI3-14B > LLAMA3.1-8B > GEMMA2-9B
Data Cleaning GEMMA2-9B > LLAMA3.1-8B > PHI3-14B

Table 7.1: Model Performance Across Various Tasks

In general, according to Figure 7.1, PHI3-14B emerged as the best performer across
most tasks, especially in data statistical analysis and graph plotting. GEMMA2-9b
showed strong results in descriptive analysis and data cleaning while LLAMA3.1-8B of-
fered a more balanced, though slightly lower, performance across all question types. From
Table 5.2 overall score for PHI3-14B is higher compared to other 2 models.

7.2 Future Work

While our research provides valuable insights, it also highlights areas for further explo-
ration and development
• Improving Statistical Analysis Capabilities: Given the poor performance of
all models in handling statistical tasks, we recommend future work focus on enhanc-
ing LLMs’ capabilities in statistical reasoning and hypothesis testing. Addressing
this gap would significantly improve the models’ utility in data analytics.
• Optimizing Models for Specific Tasks: While GEMMA2-9B performed best
in most tasks, there may be opportunities to optimize LLAMA3.1-8B and PHI3-
14B for specific data analytics tasks. Future work could focus on improving these
models’ performance in areas where they currently lag behind, such as data cleaning
and multi-step reasoning.
7.2. Future Work 73

• Exploring Transformer-Based Model Architectures for Efficiency: LLMs,

especially in larger configurations, are resource-intensive. Research into optimizing
transformer-based architectures to reduce resource consumption without compro-
mising performance in ITSM tasks could make these models more accessible for
industry applications, particularly for small to medium-sized enterprises.

• Integration of Hybrid AI Techniques: Investigating the integration of LLMs

with other AI techniques, such as reinforcement learning or symbolic reasoning,
could enhance their problem-solving capabilities. This hybrid approach could im-
prove the models’ ability to handle multi-step reasoning processes and decision-
making, particularly in scenarios that involve complex IT workflows like change
management and service request handling.

Further research could also investigate the benefits of combining Priming with Chain-
of-Thought prompting. This combination may improve both the logical structuring of
tasks and the alignment of model outputs with specific requirements.

We suggest future experiments explore LLM performance in more complex data an-
alytics scenarios, such as machine learning model building, data preprocessing pipelines,
or advanced data transformations. These tasks would provide a more comprehensive
evaluation of the models’ overall capabilities.

We have chosen to work with open-source models due to the sensitive nature of the
data involved in our experiments. Many organizations and enterprises are reluctant to
share internal operational data with external entities, as this information is often critical
to their business. By using open-source models, we ensure that the data remains secure
and is not stored in the cloud or other environments where it could be compromised.
While our current approach utilizes models with fewer parameters, future work could
explore state-of-the-art models like ChatGPT or Gemini, which have the potential to
deliver improved results.

We have focused on ITSM incident data for evaluating the LLMs, but there is potential
for improvement by incorporating a wider variety of datasets. Our current evaluation is
limited to a few scenarios that we found helpful; however, this approach can be extended
to include numerous other scenarios and datasets. Broadening the evaluation with more
questions and diverse datasets could enhance the credibility of our approach and provide
deeper insights into the weaknesses of the models’ training data.

In conclusion, our thesis has provided a detailed evaluation of LLMs performance in

generating code for data analytics, highlighting the importance of effective prompting
strategies and offering insights into how future models can be improved to better handle
a wide range of tasks.
 References

[1] Thomas A Runkler. Data analytics. Springer, 2020.

[2] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou,
Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large
language models. arXiv preprint arXiv:2303.18223, 2023.
[3] A Vaswani. Attention is all you need. Advances in Neural Information Processing
Systems, 2017.
[4] Jianxun Wang and Yixiang Chen. A review on code generation with llms: Appli-
cation and evaluation. In 2023 IEEE International Conference on Medical Artificial
Intelligence (MedAI), pages 284–289. IEEE, 2023.
[5] Tarcisio Talles Dias Pereira. Tailoring a data analysis methodology to the Information
Technology Service Management. PhD thesis, Politecnico di Torino, 2020.
[6] Donna Knapp. The ITSM process design guide: developing, reengineering, and im-
proving IT service management. J. Ross Publishing, 2010.
[7] Eghbal Ghazizadeh, Moonjee Jeong, Juan Pablo Angel, and Yehan Que. It service
management and incident management: Literature review and a case study. 2019.
[8] Freddy Aquino, Diego Pacheco, Paula Angeleri, Rosanna Janampa, Karin Melendez,
and Abraham Dávila. Information technology service management processes for
very small organization: a proposed model. In Trends and Applications in Software
Engineering: Proceedings of the 7th International Conference on Software Process
Improvement (CIMPS 2018) 7, pages 55–68. Springer, 2019.
[9] Don MacLean and Ryad Titah. Implementation and impacts of it service man-
agement in the it function. International Journal of Information Management,
70:102628, 2023.
[10] Rajeev Gupta, K Hima Prasad, and Mukesh Mohania. Automating itsm incident
management process. In 2008 International Conference on Autonomic Computing,
pages 141–150. IEEE, 2008.
[11] Ajaya K Swain and Valeria R Garza. Key factors in achieving service level agree-
ments (sla) for information technology (it) incident resolution. Information Systems
Frontiers, 25(2):819–834, 2023.
[12] Usama M Fayyad, Gregory Piatetsky-Shapiro, Padhraic Smyth, and Ramasamy
Uthurusamy. Advances in knowledge discovery and data mining. American As-
sociation for Artificial Intelligence, 1996.
[13] Colin Shearer. The crisp-dm model: the new blueprint for data mining. Journal of
data warehousing, 5(4):13–22, 2000.

75
76 References

[14] Donia Y Badawood. Data visualization and visual data analytics in itsm. Interna-
tional Journal of Computer Science & Network Security, 23(6):68–76, 2023.
[15] Ronald Rosenfeld. Two decades of statistical language modeling: Where do we go
from here? Proceedings of the IEEE, 88(8):1270–1278, 2000.
[16] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu,
and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of
machine learning research, 12:2493–2537, 2011.
[17] T Mikolov. Distributed representations of words and phrases and their composition-
ality. arXiv preprint arXiv:1310.4546, 2013.
[18] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-
training of deep bidirectional transformers for language understanding. In Proceed-
ings of naacL-HLT, volume 1, page 2, 2019.
[19] M Lewis. Bart: Denoising sequence-to-sequence pre-training for natural language
generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
[20] Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shu-
vendu K Lahiri. Llm-based test-driven interactive code generation: User study and
empirical evaluation. arXiv preprint arXiv:2404.10100, 2024.
[21] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on
large language models for code generation. arXiv preprint arXiv:2406.00515, 2024.
[22] James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James
Prather. The robots are coming: Exploring the implications of openai codex on intro-
ductory programming. In Proceedings of the 24th Australasian Computing Education
Conference, pages 10–19, 2022.
[23] Mohamed Nejjar, Luca Zacharias, Fabian Stiehle, and Ingo Weber. Llms for science:
Usage for code generation and data analysis. arXiv preprint arXiv:2311.16733, 2023.
[24] Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Tong Xiao, and
Jingbo Zhu. Efficient prompting methods for large language models: A survey.
arXiv preprint arXiv:2404.01077, 2024.
[25] Sabit Ekin. Prompt engineering for chatgpt: a quick guide to techniques, tips, and
best practices. Authorea Preprints, 2023.
[26] Tom B Brown. Language models are few-shot learners. arXiv preprint
arXiv:2005.14165, 2020.
[27] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao,
Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-
training for conversational response generation. arXiv preprint arXiv:1911.00536,
2019.
[28] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira
Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brock-
man, et al. Evaluating large language models trained on code. arXiv preprint
arXiv:2107.03374, 2021.
[29] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code
generated by chatgpt really correct? rigorous evaluation of large language models
for code generation. Advances in Neural Information Processing Systems, 36, 2024.
References 77

[30] Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A system-
atic evaluation of large language models of code. In Proceedings of the 6th ACM
SIGPLAN International Symposium on Machine Programming, pages 1–10, 2022.
[31] Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef
Pieprzyk, and Surya Nepal. Transformer-based language models for software vulner-
ability detection. In Proceedings of the 38th Annual Computer Security Applications
Conference, pages 481–496, 2022.
[32] Jorge Valverde-Rebaza, Aram González, Octavio Navarro-Hinojosa, and Julieta
Noguez. Advanced large language models and visualization tools for data analytics
learning. In Frontiers in Education, volume 9, page 1418006. Frontiers Media SA,
2024.
[33] Mohamed Nejjar, Luca Zacharias, Fabian Stiehle, and Ingo Weber. Llms for science:
Usage for code generation and data analysis. arXiv preprint arXiv:2311.16733, 2023.
[34] Zeeshan Rasheed, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Wang
Xiaofeng, Anh Nguyen Duc, and Pekka Abrahamsson. Can large language models
serve as data analysts? a multi-agent assisted approach for qualitative data analysis.
arXiv preprint arXiv:2402.01386, 2024.
[35] Fengxiang Zhao, Fan Yu, Timothy Trull, and Yi Shang. A new method using llms
for keypoints generation in qualitative data analysis. In 2023 IEEE Conference on
Artificial Intelligence (CAI), pages 333–334. IEEE, 2023.
[36] Jochen Wulf and Juerg Meierhofer. Exploring the potential of large language models
for automation in technical customer service. arXiv preprint arXiv:2405.09161, 2024.
[37] Stephany Bellomo, Shen Zhang, James Ivers, Julie B. Cohen, and Ipek Ozkaya.
Assessing opportunities for llms in software engineering and acquisition, Nov 2023.
[38] Marko Jäntti and Virpi Hotti. Defining the relationships between it service manage-
ment and it service governance. Information Technology and Management, 17:141–
150, 2016.
[39] O. Resch. Prompt engineering in it management education. In INTED2024 Proceed-
ings, 18th International Technology, Education and Development Conference, pages
2590–2597. IATED, 4-6 March, 2024 2024.
[40] Xinyang Zhao, Xuanhe Zhou, and Guoliang Li. Chat2data: An interactive data
analysis system with rag, vector databases and llms. Proc. VLDB Endow, 2024.
[41] Andrew S Denney and Richard Tewksbury. How to write a literature review. Journal
of criminal justice education, 24(2):218–234, 2013.
[42] Staffs Keele et al. Guidelines for performing systematic literature reviews in software
engineering. Technical report, Technical report, ver. 2.3 ebse technical report. ebse,
2007.
[43] Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders
Wesslén, et al. Experimentation in software engineering, volume 236. Springer, 2012.
[44] Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey [Link]. The llama 3 herd
of models, 2024.
78 References

[45] Gemma Team, Thomas Mesnard, and Cassidy Hardin [Link]. Gemma: Open models
based on gemini research and technology, 2024.
[46] Marah Abdin, Jyoti Aneja, Hany Awadalla, and Ahmed Awadallah [Link]. Phi-3
technical report: A highly capable language model locally on your phone, 2024.
[47] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa-
sawa. Large language models are zero-shot reasoners. Advances in neural information
processing systems, 35:22199–22213, 2022.
[48] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika,
Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al.
Multitask prompted training enables zero-shot task generalization. arXiv preprint
arXiv:2110.08207, 2021.
[49] Shuoqi Sun, Shengyao Zhuang, Shuai Wang, and Guido Zuccon. An investigation of
prompt variations for zero-shot llm-based rankers. arXiv preprint arXiv:2406.14117,
2024.
[50] Lukas Vöge, Vincent Gurgul, and Stefan Lessmann. Leveraging zero-shot prompting
for efficient language model distillation. arXiv preprint arXiv:2403.15886, 2024.
[51] Kaustubh D Dhole and Eugene Agichtein. Genqrensemble: Zero-shot llm ensemble
prompting for generative query reformulation. In European Conference on Informa-
tion Retrieval, pages 326–335. Springer, 2024.
[52] Ahmed Elshabrawy, Yongix Huang, Iryna Gurevych, and Alham Fikri Aji. Enabling
natural zero-shot prompting on encoder models via statement-tuning. arXiv preprint
arXiv:2404.12897, 2024.
[53] Feihu Jin, Yifan Liu, and Ying Tan. Zero-shot chain-of-thought reasoning guided by
evolutionary algorithms in large language models. arXiv preprint arXiv:2402.05376,
2024.
[54] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever,
et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9,
2019.
[55] Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng
Yan. Guiding large language models via directional stimulus prompting. Advances
in Neural Information Processing Systems, 36, 2024.
[56] Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram
Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu,
Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint
arXiv:2302.07842, 2023.
[57] Imran Chamieh, Torsten Zesch, and Klaus Giebermann. Llms in short answer scor-
ing: Limitations and promise of zero-shot and few-shot approaches. In Proceedings of
the 19th Workshop on Innovative Use of NLP for Building Educational Applications
(BEA 2024), pages 309–315, 2024.
[58] Catherine Tony, Nicolás E Díaz Ferreyra, Markus Mutas, Salem Dhiff, and Ric-
cardo Scandariato. Prompting techniques for secure code generation: A systematic
investigation. arXiv preprint arXiv:2407.07064, 2024.
References 79

[59] Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal,
and Aman Chadha. A systematic survey of prompt engineering in large language
models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024.
[60] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models
better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
[61] Timo Schick and Hinrich Schütze. Few-shot text generation with natural language
instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural
Language Processing, pages 390–402, 2021.
[62] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.
Fantastically ordered prompts and where to find them: Overcoming few-shot prompt
order sensitivity. arXiv preprint arXiv:2104.08786, 2021.
[63] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text clas-
sification and natural language inference. arXiv preprint arXiv:2001.07676, 2020.
[64] Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. Llms are few-shot in-context
low-resource language learners. arXiv preprint arXiv:2403.16512, 2024.
[65] Xufeng Zhao, Mengdi Li, Wenhao Lu, Cornelius Weber, Jae Hee Lee, Kun Chu, and
Stefan Wermter. Enhancing zero-shot chain-of-thought reasoning in large language
models through logic. arXiv preprint arXiv:2309.13339, 2023.
[66] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi,
Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in
large language models. Advances in neural information processing systems, 35:24824–
24837, 2022.
[67] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting
for code generation. ACM Transactions on Software Engineering and Methodology,
2023.
[68] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting
for code generation. ACM Transactions on Software Engineering and Methodology,
2023.
[69] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought
prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
[70] Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. Towards better chain-
of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959, 2023.
[71] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang,
Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most
prompting enables complex reasoning in large language models. arXiv preprint
arXiv:2205.10625, 2022.
[72] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang,
Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought
reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi,
Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in
large language models. Advances in neural information processing systems, 35:24824–
24837, 2022.
80 References

[74] Michael D Murray. Prompt engineering and priming in law. Available at SSRN,
2024.
[75] Suet-Ying Lam, Qingcheng Zeng, Kexun Zhang, Chenyu You, and Rob Voigt. Large
language models are partially primed in pronoun interpretation. arXiv preprint
arXiv:2305.16917, 2023.
[76] Bushi Xiao, Chao Gao, and Demi Zhang. Modeling bilingual sentence processing:
Evaluating rnn and transformer architectures for cross-language structural priming.
arXiv preprint arXiv:2405.09508, 2024.
[77] Jaap Jumelet, Willem Zuidema, and Arabella Sinclair. Do language models exhibit
human-like structural priming effects? arXiv preprint arXiv:2406.04847, 2024.
[78] James A Michaelov, Catherine Arnett, Tyler A Chang, and Benjamin K Bergen.
Structural priming demonstrates abstract grammatical representations in multilin-
gual language models. arXiv preprint arXiv:2311.09194, 2023.
[79] Sawan Kumar and Partha Talukdar. Reordering examples helps during priming-
based few-shot learning. arXiv preprint arXiv:2106.01751, 2021.
[80] Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. Structural
persistence in language models: Priming as a window into abstract language repre-
sentations. Transactions of the Association for Computational Linguistics, 10:1031–
1050, 2022.
[81] Jo Inge Arnes and Alexander Horsch. Schema-based priming of large language model
for data object validation compliance. Available at SSRN 4453361, 2023.
[82] Eli Fennell. Action identification characteristics and priming effects in chatgpt. 2023.
[83] Alexandre Piché, Aristides Milios, Dzmitry Bahdanau, and Christopher Pal. Self-
evaluation and self-prompting to improve the reliability of llms. In ICLR 2024
Workshop on Secure and Trustworthy Large Language Models.
[84] Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and
Jian Li. Prompt engineering in consistency and reliability with the evidence-based
guideline for llms. npj Digital Medicine, 7(1):1–9, Feb 2024.
[85] Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and
Jing Gao. Sayself: Teaching llms to express confidence with self-reflective rationales,
2024.
[86] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang,
Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought
reasoning in language models, 2023.
[87] Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiahui Gao, Sichun Luo, Hanxu
Hou, Xiaojin Fu, and Linqi Song. Learning from correctness without prompting
makes llm efficient reasoner, 2024.
[88] Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting
Cheng. Llm-fp4: 4-bit floating-point quantized transformers. arXiv preprint
arXiv:2310.16836, 2023.
Faculty of Engineering, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden