Received 28 December 2024, accepted 19 January 2025, date of publication 23 January 2025, date of current version 29 January 2025.

Digital Object Identifier 10.1109/ACCESS.2025.3533217

Exploring LLMs Applications in Law: A Literature

Review on Current Legal NLP Approaches
MARCO SIINO 1,2,3 , MARIANA FALCO 2, DANIELE CROCE 2,3 , AND PAOLO ROSSO4,5
1 Department of Electrical, Electronics and Informatics Engineering, University of Catania, 95131 Catania, Italy
2 Department of Engineering, University of Palermo, 90133 Palermo, Italy
3 Palermo Research Unit, National Inter-University Consortium for Telecommunications (CNIT), 90128 Palermo, Italy
4 PRHLT Research Center, Universitat Politècnica de València, 46022 Valencia, Spain
5 ValgrAI–Valencian Graduate School and Research Network of Artificial Intelligence, 46022 Valencia, Spain

Corresponding author: Marco Siino ([Link]@[Link])

This work was supported in part by the European Union through the Italian Ministero dell’Università e della Ricerca (MUR) National
Recovery and Resilience Plan (NRRP) of NextGenerationEU, partnership on ‘‘Telecommunications of the Future’’ (program ‘‘Research
and Innovation on Future Telecommunications Systems and Networks, to make Italy more Smart (RESTART)’’), within the project
Net4Future under Grant PE0000001; in part by the research project Malicious Actors Profiling and Detection in Online Social Networks
through Artificial Intelligence (MARTINI) funded by Ministerio de Ciencia e Innovación (MCIN)/Agencia Estatal de Investigación
(AEI)/10.13039/501100011033 under Grant PCI2022-135008-2; in part by the European Union NextGenerationEU/Plan de Recuperación,
Transformación y Resiliencia (PRTR) and FairTransNLP-Stereotypes—Fairness and transparency for equitable NLP applications in social
media: Identifying stereotypes and prejudices and developing equitable systems funded by MCIN/AEI/10.13039/501100011033 under
Grant PID2021-124361OB-C31; and in part by the European Regional Development Fund (ERDF), EU, A way of making Europe.

ABSTRACT Artificial Intelligence (AI) is reshaping the legal landscape, with software tools now impacting
various aspects of legal work. The intersection of Natural Language Processing (NLP) and law holds potential
to transform how legal professionals, including lawyers and judges, operate, resolve disputes, and retrieve
case information to formulate their decisions. To identify the current state of the applications of Transformers
(also known as Large Language Models or LLMs) in the legal domain, we analysed the existing literature
from 2017 to 2023 through a database search and snowballing method. From 61 selected publications,
we identified key application categories such as legal document analysis, case prediction, and contract
review, along with their main characteristics. We observed a discernible upsurge in the volume of scholarly
publications, a diversification of tasks undertaken (e.g., legal research, contract analysis, and regulatory
compliance), and an increased range of languages considered. There has been a notable enhancement in
the methodological sophistication employed by researchers in practical applications. The performance of
models grounded in the Generative Pre-trained Transformer (GPT) architecture has consistently improved
across various legal domains, including contract review, legal document summarization, and case outcome
prediction. This paper makes several significant contributions to the field. Firstly, it identifies emerging
trends in the application of LLMs within the legal domain, highlighting the growing interest and investment
in this area. Secondly, it pinpoints methodological gaps in current research, suggesting areas where further
development and refinement are needed. Lastly, it discusses the broader implications of these advancements
for real-world legal tasks, offering insights into how LLM-based AI can enhance legal practice while
addressing the associated challenges.

INDEX TERMS Natural language processing, law, AI for law, legal NLP, legal tech, GPT, transformers,
literature review.
I. INTRODUCTION vast amount of information generated in the legal domain has
Law, as a discipline, continuously evolves in response to soci- prompted the exploration of machine assistance, leveraging
etal, political, economic, and technological changes [1]. The computers’ ability to analyse large textual datasets rapidly.
Artificial Intelligence (AI), which refers to the simulation
The associate editor coordinating the review of this manuscript and of human intelligence processes by machines, particularly
approving it for publication was Turgay Celik . computer systems, has become instrumental in this context.

2025 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License.
VOLUME 13, 2025 For more information, see [Link] 18253
 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

AI encompasses capabilities such as learning, reasoning, a database search and snowballing method, resulting in
and self-correction, enabling machines to perform tasks that an initial pool of publications. After a rigorous quality
typically require human intelligence. Today, AI is crucial assessment and filtering process, we selected 61 publications
for legal tech firms, enhancing operations to serve clients that met our criteria for relevance and methodological
more affordably and accurately. Key AI applications include rigour. By identifying and analysing specific instances where
contract review, legal research, and predictive analytics for LLM-based methodologies are applied, we highlight their
case outcomes and court rulings. Additionally, chatbots are effectiveness and limitations in addressing various legal chal-
becoming increasingly popular for providing self-service lenges. Our goal is to offer a thorough understanding of how
legal information. Political institutions and governments these technologies can aid domain experts, such as law firms,
are actively supporting AI development. For instance, a judges, and lawyers, and facilitate automated resolution and
2020 report [2] highlighted AI’s role in improving agency document generation processes within the legal domain.
operations like regulatory enforcement and data analysis in This endeavour involves defining and extracting legal tasks
the U.S. The European Union also emphasizes excellence and and identifying trends and uses of LLMs in the legal
trust in AI, aiming to enhance research and industrial capacity domain.
while protecting fundamental rights [3]. The rest of the article is structured as follows: Section II
The integration of Natural Language Processing (NLP) and presents the background and related work, and Section III
AI into legal tasks is a natural progression, given the linguistic describes the research method. Section IV reports a bib-
nature of law. This combination allows for more efficient and liometric analysis of the literature reviewed, Section V
accurate analysis of legal texts, enhancing various aspects of provides the overview of the models and approaches, whereas
legal practice. The use of NLP and AI in legal tech has a long Section VI the trends within legal tasks and real-life
history, dating back to the 1960s with the development of implementations. Section VII states the conclusions, and
online legal content search systems [4], [5], [6]. NLP, particu- Section VIII discusses the implications of our findings for
larly with the advent of Large Language Models (LLMs), has future research.
made significant strides in legal applications, aiding in tasks
requiring language processing and understanding. LLMs II. BACKGROUND AND RELATED WORK
represent cutting-edge technology, advancing AI approaches In this section, the landscape of legal AI is thoroughly
in various domains involving textual contents such as examined in various studies, highlighting its diverse appli-
medicine [7], [8], [9], engineering [10], [11] and law [12], cations and advancements. The application of AI in the
[13], [14], [15]. Overall, the integration of AI and NLP in the legal sector has long garnered significant interest, focusing
legal domain holds great promise for improving efficiency on its implications for legal practice, administration, and
and decision-making processes across various legal tasks. the ethical considerations surrounding its use with legal
However, addressing challenges related to context, data data. Early explorations into online legal content search
availability, and interpretability remains essential for the systems date back to the 1960s [5], [27], highlighting the
reliable application of these technologies in the legal domain longstanding intersection of AI and law. Zhong et al. [28]
[16], [17]. Within this context, applications refer to the classify legal AI tasks into three categories: judgement
broader use cases where AI technologies address specific prediction, similar case matching, and legal question answer-
legal challenges or fulfil specific needs, such as contract ing, with early Transformer-based language models being
review, document automation, and compliance monitoring. notable contributors. Sansone and Sperlí [29] categorize
These applications are representations of particular instances legal information retrieval approaches into natural language-
of how models, algorithms, or tools are applied to legal tasks. based, ontology-based, and deep learning-based systems.
Conversely, tasks within legal AI are the specific activities Additionally, Katz et al. [30] provide an extensive review
that AI systems perform within these applications, such of NLP in the legal domain, documenting the growth in
as identifying key clauses in contracts, extracting relevant research publications and tasks over the past decade. Dale
information, or generating summaries. et al. [31] focus on NLP applications in contract review and
This article aims to comprehensively examine the appli- document automation, emphasizing their importance for legal
cations of LLMs in the legal domain, focusing on how practitioners.
various models and techniques are used in legal tasks. Based The advent of Transformer models (from now on also
on recent literature [3], [17], [18], [19], [20], [21], [22], LLMs) [26] marked a defining moment as well the LLMs,
[23], [24], [25], we have grouped the tasks into three main revolutionizing the landscape of deep-learning architectures
areas: Legal Search, Legal Document Review and Legal and setting new benchmarks for performance across a
Prediction. To identify the current state of the applications of spectrum of intricate NLP tasks. Central to the Transformer’s
Transformers in the legal domain, we analysed the existing innovation is its integration of attention mechanisms [32],
literature from 2017 to 2023. The year 2017 was chosen as which diverge from conventional Recurrent Neural Net-
the starting point because it marks the introduction of the works (RNNs) and Convolutional Neural Networks (CNNs).
Transformer architecture [26]. A timeline of the evolution By harnessing attention mechanisms, the Transformer
during these years is depicted in the Figure 1. We conducted architecture orchestrates a holistic consideration of an

18254 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

FIGURE 1. Timeline of key developments in Transformer models impacting the legal domain.

input context adeptly discerning the salient features while To systematically categorize the tasks in the legal domain
disregarding noise, thus enabling nuanced assessments of where LLMs are applied, we adapted the structure proposed
word dependencies regardless of their sequential proximity. by Greco et al. [3], which divides legal challenges into three
The pervasive adoption of attention mechanisms underscores main areas: Legal Search [18], Legal Document Review [43]
their pivotal role in NLP advancements, propelling significant and Legal Prediction [17], [19], [20], [21], [22], [23],
improvements in task performance and scalability. [24], [25].
It represents a pioneering departure from traditional Legal Search encompasses the following tasks:
architectures by leveraging attention mechanisms, relying • Document Retrieval (T1): The process of finding
also on word embedding and neural networks. Attention is relevant legal documents, such as case law, statutes,
manifested in two distinct modalities within the Transformer or legal opinions, from a large corpus of texts [17],
framework. Initially, attention gauges the relative signifi- [18], [44].
cance of words within the same sequence, treating input • Case Entailment (T2): Retrieving documents that log-
words as both origins and destinations. This self-attention ically follows from or are supported by another case
mechanism endeavours to compute optimal representations [18], [44].
of the input sequence, encapsulating syntactic and semantic • Question Answering (T3): Answering specific legal
relationships. Subsequently, input representations undergo questions by retrieving and synthesizing information
weighted attention to predict target tokens in an autoregres- from legal texts [18], [45].
sive fashion. Legal Document Review includes:
LLMs such as ChatGPT demonstrate significant potential
• Named Entity Recognition (T4): Identifying and clas-
in various legal tasks, particularly in legal judgement
sifying entities (e.g., names of people, organizations,
prediction and statutory reasoning [33]. Trautmann et al.
locations) within legal documents [43].
[34] introduce legal prompt engineering to enhance LLM
• Similarity Estimation (T5): Measuring how similar two
performance in judgement prediction tasks, demonstrat-
legal documents or cases are, which is essential for tasks
ing effectiveness across multilingual datasets. Blair-Stanek
like case law comparison [43], [46].
et al. [35] explore GPT-3’s aptitude for statutory rea-
• Classification (T6): Categorizing legal documents into
soning, achieving high accuracy with dynamic few-shot
predefined categories based on their content [24],
prompting. Advancements in prompting techniques, like
[46], [47].
Chain-of-Thought (CoT) prompts presented by Yu et al.
• Document Summarization (T7): Producing concise sum-
[36], further improve LLM performance in legal reasoning
maries of lengthy legal documents [48], [49].
tasks. LLMs are also explored for their potential in legal
• Datasets and Benchmarking (T8): Creating and using
education and supporting legal professionals [37]. Research
datasets to evaluate and compare the performance of
by Iu Wong et al. [38] and Hargreaves [39] discusses the
various AI models in legal tasks [50].
ethical utilization of AI in law school assessments, proposing
• Document Automation (T9): Automating the review
methods to educate students on appropriate AI usage [40].
and synthesis process of legal documents to enhance
Pettinato [41] suggests that LLMs could assist law professors
efficiency and accuracy [51].
in administrative duties and streamline scholarly work.
Macey-Dare [42] investigates LLMs as quasi-expert legal Legal Prediction consists of:
advisors, showcasing their feasibility in providing affordable • Judgement Prediction (T10): Predicting the outcomes
legal counsel. In summary, LLMs have exhibited promising of legal cases based on previous rulings and case
outcomes across diverse legal tasks, with advancements characteristics [19], [20], [21], [22], [52], [53].
in prompting techniques playing a pivotal role in their • Next Sentence Prediction (T11): Predicting the subse-
efficacy. Nevertheless, challenges persist in ensuring the quent sentence in a legal document to assist in drafting
ethical utilization of LLMs and addressing their potential and understanding legal texts [16], [17].
impact on the legal profession. Continued exploration of the These tasks represent the specific activities that LLMs
capabilities and constraints of LLMs in the legal arena is and other AI technologies perform within broader legal
essential, while ensuring their alignment with human values applications, facilitating advancements in legal research,
and societal requirements. document automation, and outcome prediction.

VOLUME 13, 2025 18255

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

III. RESEARCH METHOD examining case studies, pilot projects, and industry reports
In conducting our systematic review, we adhered to the to identify instances where LLMs are successfully being
Preferred Reporting Items for Systematic Reviews and Meta- used by legal professionals or institutions. Finally, RQ3 and
Analyses (PRISMA) guidelines [54]. PRISMA is a widely RQ4 aim to explore advantages and limitations of LLMs
accepted standard designed to help researchers conduct and in the legal domain, and legal tasks that have not yet been
report systematic reviews and meta-analyses in a transparent extensively researched or implemented. They aim to identify
and comprehensive manner. It provides a checklist of items gaps in the current literature, brainstorming innovative uses
that should be included in the report of a systematic review of LLMs, and proposing new areas where these models
or meta-analysis, ensuring that the review is conducted could significantly impact legal practices, such as novel ways
systematically and that all relevant information is clearly of automating legal processes, enhancing legal decision-
presented. By following the PRISMA guidelines, we aimed making, or improving access to legal resources.
to enhance the quality and reliability of our review. Our Overall, these research questions follow a logical progres-
methodology included a thorough search strategy, rigorous sion, starting with broad inquiries about the usage of LLMs
study selection criteria, systematic data collection, and in the legal domain and gradually narrowing down to specific
comprehensive analysis. This approach ensured that our aspects and areas for improvement.
findings are robust and reproducible, providing a clear and
transparent overview of the current state of LLMs in the B. INCLUSION AND EXCLUSION CRITERIA
legal domain. To identify relevant papers, we employed
This subsection outlines the criteria used for assessing
an adaptation of PRISMA guidelines, including a database
the relevance of the studies. We excluded all publications
search and complemented by the backward snowballing
meeting any of the following criteria: (E1) studies unrelated
procedure [55], in order to reduce the risk of overlooking
to applications or tools of LLMs in the legal domain,
pertinent documents within the time frame selected. In the
(E2) non-English publications, (E3) non-research article
following subsections, we define the scope of the literature
types (e.g., type magazine, secondary study, course material,
review through the research questions, the inclusion and
doctoral symposium paper, thesis, position paper, keynote
exclusion criteria, the search and selection procedure, and the
presentation, abstract, or book chapter), (E4) inaccessible
quality assessment criteria.
publications, and (E5) gray literature sources (e.g. blogs,
manuals, tutorials, newsletter or project reports). Conversely,
A. RESEARCH QUESTIONS
we included publications meeting at least one of the following
The scope of our literature review is related to the application
criteria: (I1) studies addressing applications of LLMs in law,
of LLMs to the legal domain tailored to address both general
and (I2) relevant gray literature sources such as arXiv studies
and specialized tasks within the realm of law, and it is defined
that fulfilled I1.
by the following Research Questions (RQs):
We evaluated the relevance of the studies by exam-
1) RQ1: What are the NLP applications in the legal
ining their titles, abstracts, and keywords. If the initial
domain that have already made use of LLMs?
assessment was inconclusive, we delved deeper into the
2) RQ2: Is the pre-training or the fine-tuning of LLMs
article’s structure, introduction, methodology, results, and
specifically for the legal domain always beneficial?
conclusions. The initial screening process was carried out
And in what cases and for what tasks?
by the first two authors. Any discrepancies or uncertainties
3) RQ3: What are the main advantages and limitations of
were resolved through collaborative discussions among all
LLMs in the legal field?
four authors to achieve a consensus. It’s important to note
4) RQ4: What are the possible applications of LLMs to
that while conducting the search, surveys, or systematic
the legal domain not yet fully explored?
reviews emerged as results. Although these were assessed
RQ1 serves as the overarching inquiry, aiming to identify
for relevance, they were included in the related section
the specific NLP tasks within the legal domain that have
only if they aligned with our objectives, but they were not
employed Transformer models. Building upon the initial
incorporated into the comprehensive list of analysed articles.
question, RQ2 seeks to determine whether the pre-training
of LLMs on general data, followed by fine-tuning them
specifically for the legal domain, consistently yields ben- C. SEARCH AND SELECTION PROCEDURE
eficial outcomes in legal domains. Pre-training involves This subsection outlines the selection process. Regarding the
training the model on a large corpus of diverse texts to database search, we selected a set of known repositories:
develop a broad understanding of language, while fine- IEEE Xplore, ACM Digital Library, SpringerLink, and
tuning adjusts the model’s parameters using a smaller, Elsevier ScienceDirect, and we also complemented these
specialized dataset to improve its performance on specific with Google Scholar. Both searches were carried out during
tasks within the legal field. RQ2 also seeks to identify the last quarter of 2023. In order to refine the search,
which of the NLP applications are currently viable for we formulated a combination of the following strings: Legal
use in real-world legal settings. It involves assessing the AI, Legal NLP, NLP and Law, Case retrieval NLP and
maturity and performance of LLMs in these applications, Contract review, between 2017 to 2023.

18256 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

It is pertinent to mention that the chosen timeframe The [ref number] citations refer to the standard bibliographic
aligns with the emergence of pivotal advancements in NLP, entries listed at the end of the paper. Additionally, we use
particularly marked by the introduction of the Transformer Pxx references to directly link to the publication details
architecture [26] and the subsequent development of the as collected during our research. These Pxx references
BERT model [56], which occurred between the years correspond to the attached file containing all the selected
2017 and 2019. papers, allowing readers to easily find and verify the specific
We screened document titles and metadata to also apply the publications discussed in our review. For example, P01 refers
snowballing approach, and we removed duplicated records, to the first publication in the attached file, while [1] refers
records that were not accessible, that weren’t applications to the first entry in the bibliography section. This dual
or were written in a non-English language. As a result, referencing system aims to enhance the transparency and
we screened 307 studies and following the inclusion and reproducibility of our work, providing readers with easy
exclusion criteria, we excluded 151 studies. 156 studies were access to the relevant publication information. We believe
assessed for eligibility and after the quality analysis we obtain this approach strikes a balance between streamlining the
the final set of 61 articles. reading experience and maintaining the transparency and
reproducibility of our research.
D. QUALITY ASSESSMENT CRITERIA In this section we analyse the 61 selected publications
This section outlines the criteria utilized for assessing in terms of venue type, geographic distribution, and author
the quality of the included publications. We employed a affiliation. These publications span from 2020 to 2023,
checklist, as depicted in Table 1, constructed based on criteria showing a consistent upward trend in the amount of articles
outlined in the Critical Appraisal Skills Program (CASP)1 published each year regarding AI in the legal domain (10
and the methodology proposed in [57]. The questions in studies in 2020, 15 in 2021, 17 in 2022, and 19 in 2023).
Table 1 are sequentially organized from Q1 to Q5. For
each question, we devised sub-questions to facilitate the A. DISTRIBUTION BY TYPE OF VENUE
evaluation process for each publication. These sub-questions The majority of the selected publications were distributed
guided the assessment of publication quality, with responses across conferences (20 studies), workshops (13), and journals
categorized as either ‘yes’ or ‘no’, corresponding to values (7). There were also two articles presented in a single
of 1 and 0, respectively. If any sub-question under a main symposium (JSAI International Symposium on Artificial
question received a ‘yes’, the corresponding main question Intelligence) and one article in a forum (Forum for Infor-
was also marked as ‘yes’. Additionally, we deemed Q1 as the mation Retrieval Evaluation). Additionally, we identified
threshold criterion for further quality assessment, requiring a 18 pre-prints available on arXiv. Among the conferences,
positive response to proceed. the most prominent were the International Conference on
Artificial Intelligence and Law (3 publications) and the
TABLE 1. Questions of the quality assessment checklist. Conference on Neural Information Processing Systems (3),
followed by 14 different conferences. In terms of workshops,
the Natural Legal Language Processing Workshop led with
four publications, followed by the Workshop on AI for Public
Administration with two, alongside seven other workshops.
Notably, the dominant journal is Artificial Intelligence and
Law, with four articles.

B. DISTRIBUTION BY AFFILIATION
We identified 37 studies within a purely academic context,
10 associated with various industries (encompassing compa-
nies, institutes, centers, and government entities), and 14 with
shared affiliations. Among the 61 selected publications,
IV. BIBLIOMETRIC ANALYSIS authors are affiliated with 56 different universities, primarily
After the quality assessment, we carefully selected a set of located in Italy, Denmark, USA, India, Germany, and Greece.
publications from which we were able to extract relevant data Additionally, affiliations include two government agencies
to answer the research questions. The resulting list of studies (Senato della Repubblica and Istituto Poligrafico e Zecca
is composed of 61 publications, labelled as P1 to P61.2 We dello Stato, Italy), 14 institutes, six centres (Center for AI and
use two types of references to ensure clarity and transparency. Cognitive Computing at Thomson Reuters, Canada; Centre
for AI, University College London, UK; German Research
1 CASP Checklists, available at: [Link]
Center for Artificial Intelligence; Research Center for
(Accessed: Dec. 28, 2024)
2 The complete list of publications can be found at the following link: Digital Sustainability, University of Bern; CodeX, Stanford
[Link] (Accessed: Law School; and iCourts, University of Copenhagen),
Dec. 28, 2024) and one foundation (Fondazione Bruno Kessler, Trento,

VOLUME 13, 2025 18257

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

Italy). Moreover, one project (The Atticus Project3 ) and (2), Università per Stranieri di Siena (2), University of Trieste
17 companies are represented. Notably, 14 studies had (1), Politecnico di Torino (1), Roma Tre University (1),
shared affiliations, which may involve partnerships between Sapienza Università di Roma (1), University of Bari Aldo
universities and companies, centres, institutes, projects, Moro (1), IUSS Pavia (1), and University of Bologna (1).
or foundations (P10, P11, P13-P15, P17, P19-P21, P23, P28,
P29, P31, P32). D. DISTRIBUTION BY LEGAL DOMAINS
Additionally, we found two articles authored by multiple In our analysis, we discerned distinct legal domains addressed
universities: the first one is authored by University of by each study, namely: case law, legislation, legal documents,
Bern, Bern University of Applied Sciences, University of legal research, and miscellaneous topics. These categories
Zurich, University of Bologna, University of Copenhagen, encompass various sub-areas grouped based on thematic
and Stanford University (P24), and the second one by ETH similarities. Table 2 summarizes the areas, sub-areas and
Zurich, Harvard University, The Atticus Project, The Nueva studies. It’s feasible to mention that the Case Law area
School, University of Wisconsin, Madison, Yale University, is mostly focused on court cases, including supreme court
Stanford University, and UC Berkeley (P37). Furthermore, cases, European Court of Human Rights (ECHR) cases,
out of 56 universities, 27 appeared only once in the set of civil cases, European Court of Justice cases, and supporting
articles. cases; showing interest in court decisions, opinions and
We also noticed a predominance of conferences published precedents. In the Legislation area there is attention on
in a purely academic context, with 13 out of 20 articles. national legislation, statutes, and civil law codes, with a
Likewise, workshops aligned with this trend, with seven out significant focus on EU law, US law, and UK law, although
of 13 situated within academic settings. Regarding journals, also appeared the exploration of Indian Law. Then, in Legal
four were exclusively affiliated with academia, while three Documents, there is an emphasis on contracts and case
had shared affiliations. judgements, while there is also a notable interest in deal
points, maxims and measures (e.g. COVID-19), and fewer
C. DISTRIBUTION BY COUNTRY studies showed interest on legal questions, legal news, and
In terms of the geographic distribution of authors, Italy online legal resources.
stands out with 15 publications, with three more featuring
shared affiliations with Germany (P54), the UK (P61), TABLE 2. Distribution of legal areas, sub-areas and studies.

as well as Switzerland, Denmark, and the USA (P24).

Following Italy, the USA has seven publications, while India
and the Netherlands each have three. Additionally, Canada,
China, Denmark, Germany, Greece-UK, and Romania each
have two publications to their credit. Moreover, five countries
have only one article each: Brazil (P33), France (P56), Japan
(P7), Korea (P3), and Singapore (P19).
Out of a total of 61 publications, 17 have shared affili-
ations. For instance, India collaborated with the USA (P5),
the United Kingdom (P55), and Australia (P11). Switzerland
also demonstrated shared affiliations with Germany (P14),
Denmark (P2), the USA (P37), and the UK (P50). Further-
more, the United Kingdom has collaborated with Denmark
(P42), Germany, the USA, Denmark, and Greece (P21), while
the USA collaborated with Canada (P49). There are also
instances of co-authorship between countries: Netherlands
and Austria co-author one publication (P13), as do China and
Japan (P28), and Greece and Denmark (P10).
It’s worth noting the distribution of studies among universi-
ties and countries. For instance, in Denmark, we found only
the University of Copenhagen with seven studies, followed
by both University of Pisa, Italy and Stanford University,
USA with five studies each. The distribution across Italy
is also noteworthy, showcasing the broad range of regions
engaged in this topic: University of Calabria (2), Scuola
Superiore Sant’Anna, Pisa, Università degli Studi di Milano

3 The Atticus Project, Open-Source Dataset of Legal Contracts for AI.

Available at: [Link]

18258 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

V. MODELS AND APPROACHES Various tools, methods, frameworks, and language models
The LLMs application in law encompass a wide range of have been proposed to enhance document retrieval. For
functionalities and objectives, such as contract review, legal instance, P05 [58] presents a Virtual Legal Assistant (VLA)
research, document automation, case prediction, and com- that enables legal professionals to consult on legal situations
pliance monitoring, among others. Furthermore, we define with an AI-based assistant. The study emphasizes the
application as particular cases of how a model, an algorithm, Information Retrieval phase, refining search scope with
a tool, or others is applied to a legal task. increasing query intricacies and filters.
To address the research questions, we approach legal The COLIEE competition4 stands out in this research
AI tasks in two ways. First, we examine each legal task field. In P06 [59], a tool with specialized multilingual search
by reviewing the relevant studies, highlighting their key functionalities for document retrieval and entailment tasks
contributions, and identifying trends. Second, we summarize was introduced. CatBoost was used for case law retrieval,
these tasks to offer valuable insights for the research and pre-trained embeddings combined with TF-IDF (i.e.,
community. Term Frequency–Inverse Document Frequency) [60] were
The rest of this section presents the main contributions to employed for statutory information retrieval. The preprocess-
the identified legal tasks, with a summary of the tasks and ing included lowercase conversion, punctuation removal, and
corresponding studies shown in Table 3. numeric digit-to-text conversion, with FastText for embed-
ding training. Embedding-based methods proved to be more
TABLE 3. Distribution of legal tasks and studies. effective than those based on TF-IDF. A knowledge-based
approach for legal document retrieval - relying on document
embedding also in this case - and based on the organization
of a textual data repository is introduced in [61]. Documents
that have been pre-processed and embedded undergo iterative
sentence-level classification through a cycle of terminology
extraction and concept formation. This method leverages
the ASKE (Automated System for Knowledge Extraction)
engine, which tackles a multilabel classification problem
without requiring any initial annotations of the documents.
The authors present an application of ASKE in a practical
case study focused on retrieving legal documents from a
repository of Italian court decisions. This work is part of the
Next Generation UPP (NGUPP) project, funded by the Italian
Ministry of Justice, and aims to integrate AI and advanced
information management techniques to facilitate the digital
transformation of Italian legal processes and digital justice.
Specifically, the case study addresses a common challenge
faced by legal practitioners and administrators: the retrieval
of past court decisions, known as ‘‘precedents,’’ based on one
or more input text fragments, such as sentences, definitions,
or excerpts from articles. The goal is to identify and retrieve
the most relevant documents, such as court decisions or
A. LEGAL SEARCH specific sentences within them, that match the input query.
1) DOCUMENT RETRIEVAL (T1) Similarly, P07 [62] addressed long documents and ambiguity
In the legal domain, practitioners frequently require access in the legal domain, proposing a document-level attention
to specific documents for tasks such as research, case mechanism and passage mining. They used abstract meaning
preparation, drafting, and client advising. Document retrieval representation to reduce noise and identified use-cases. The
is pivotal in accessing relevant legal texts, including court datasets comprised Canadian Federal Court cases and statute
cases, statutes, regulations, and legal opinions, to support law. LEGAL-BERT was utilized to extract semantic relation-
these activities effectively [17], [18], [44]. ships, and Spacy for text segmentation. TF-IDF outperformed
Among the articles reviewed, 15 focused on Document BM25 in obtaining superior performance on long-document-
Retrieval in the legal domain (P04-P10, P12-P16, P20, P28, related tasks. Regarding long-document tasks, in P10 [63],
P37), with a primary focus on Case Law and Legislation two methods for long-document processing were explored:
(see Table 2). In particular, 11 articles focus on Legal Case modifying Longformer warm-started from LEGAL-BERT
Retrieval, one on Statute Law Retrieval (P09), and two for longer texts (up to 8,192 tokens), and modifying
on both Legal Case Retrieval and Statute Law Retrieval
(P06, P07). Notably, five articles specifically address long- 4 Competition on Legal Information Extraction/Entailment (COLIEE),
document retrieval (P06, P07, P10, P13, P28). available at [Link] rabelo/COLIEE2024/

VOLUME 13, 2025 18259

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

Legal-BERT to use TF-IDF representations. Experiments on standard benchmarks derived from Open Case Book and
on LexGLUE showed these variants outperformed LEGAL- Wiki source. The objective of the task is to retrieve relevant
BERT, establishing new state-of-the-art solutions. Opposite literature given a specific case. The methods assessed in the
than TF-IDF, using embedding representation P08 [64] study can be broadly categorized into three groups: word-
introduced a knowledge-based approach for retrieving prece- vector based, Transformer based and citations based.
dent sentences using document embedding models, with What emerged from the review of the literature is that
a zero-shot approach for classification and knowledge performing fine-tuning or pre-training on a Transformer-
extraction from text fragments. Furthermore, in P20 [65] the based architecture as BERT, or on a word embedding
authors introduce a novel method named PRILJ, designed representations, is often beneficial and common in the
to identify paragraph regularities in legal case judgments field of article and case retrieval. Some exceptions are
and assist legal experts in drafting legal documents. PRILJ traditional approaches, as TF-IDF. However, we barely found
employs a two-step approach: first, it groups documents foundational models able to outperform domain-specific
into clusters based on their semantic content, and then it models and approaches for retrieval tasks.
identifies regularities within the paragraphs of each cluster.
The method utilizes embedding techniques to represent 2) CASE ENTAILMENT (T2)
documents and paragraphs in a semantic numerical feature Case Entailment is the task of determining whether the
space. Additionally, an approximated nearest neighbor search facts, legal principles and arguments introduced in a legal
method is used to efficiently retrieve the most similar case logically support or imply the outcome of another
paragraphs relative to those in a document being prepared. case, requiring to analyse the content of legal documents
Another successful application of embedding representation to identify relationships and inferring conclusions based on
is reported in P13 [66]. It combined lexical and dense precedents [18], [44]. We have found 4 studies that focus on
retrieval methods for paragraph-level case retrieval. The this task (P06, P07, P13, P28), and their main legal area of
best results were achieved with BM25 and dense passage the application is Case Law.
retrieval using domain-specific embeddings. BM25 was used In P06 [59], out of 4 tasks, Task 2 (a case law
for initial retrieval, followed by BERT for aggregation and entailment task) aims to distinguish which paragraph in
reordering. A BERT variation is discussed in P28 [67] a supporting case implies the provided text fragment and
where the authors proposed BERT-PLI to model paragraph- Task 4 (statutory entailment task) focuses on determining
level interactions and infer relevance between cases, using the potential implication of a bar exam question by a set of
a cascade framework to reduce computational costs. The relevant articles. For Task 2, the jurisprudence implication
model was fine-tuned with a small-scale case law entailment task requires finding an implying paragraph from a case,
dataset, demonstrating effectiveness in legal scenarios. Also given a base case with a specified fragment f. Three groups
based upon BERT, P09 [68] focused on information retrieval of features (traditional, embedding similarity, and Natural
systems to identify relevant precedents and statutes using text Language Inference (NLI)) were used also in this case.
similarity approaches. An ensemble classifier (i.e., XGBoost) was employed, and for
Domain-specific adaptation of BERT are often accom- each related case R, results were ranked based on the number
plished pre-training or fine-tuning the original BERT model of paragraphs implying f. The best embedding results were
on specific domains. For instance, P12 [69] introduced obtained with BERT. With this model and on this task, the
LamBERTa, a BERT-derived model pretrained on Italian authors achieved satisfactory results in the competition. The
Civil Code laws, highlighting predictive justice. LamBERTa Task 4, the statutory law implication task, on the other hand,
predicts relevant articles from the Italian Civil Code in involves determining whether a legal bar exam question Q
few-shot scenarios, emphasizing domain-specific adaptation. is included in the text of a series of articles S1, S2, . . . SN
P16 [70] revised LamBERTa, incorporating legal-specific relevant to Q. Implication means that Q is true or false based
pre-training and out-of-vocabulary legal terms. Extensive on the content of S1, S2, . . . SN. This objective was pursued
evaluation revealed significant improvements in law article in two ways: using a BERT-XGBoost combination and
retrieval due to domain- and task-adaptation. employing legal embeddings with a Bi-GRU. An interesting
Some variations of the original Transformer architecture consideration is the results achieved by operating directly
have also been found in the literature. For example, P15 on the Japanese language, which typically contains more
[71] introduced SAILER, a structure-aware pre-trained informative tokens than their English counterparts [73], [74].
language model for legal case retrieval, integrating structural Their findings illustrate that using legal embeddings and
information and an asymmetric encoder-decoder architec- auxiliary linguistic features, such as NLI, showed the most
ture. SAILER outperformed previous methods in legal promise for future improvements.
case retrieval tasks, demonstrating strong discriminative Also in P13 [66] the authors took advantage of a domain-
capabilities. focused embedding. The authors detail their methodologies
Finally, in P14 [72], the authors evaluate 27 methods for two specific tasks in the COLIEE 2021 competition: legal
(based on both Transformers and traditional architectures) case retrieval and legal case entailment. The objective of the

18260 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

legal case entailment task is to develop a system capable of questions. The proposed model is designed to address legal
identifying paragraphs within a relevant case that support questions across a wide range of legal domains. The system
the decision of a new, given case. In this task, a query selects and ranks answers using a combination of sparse
paragraph is provided along with candidate paragraphs from vector techniques like BM25 and dense vectors (semantic
a legal case, and the system must pinpoint which candidate embeddings). The answer selection process is conducted
paragraphs logically support the decision outlined in the on a collection comprising over 100 million passages. The
query paragraph. The datasets used for training and testing system employs Query By Document (QBD) implemented
include cases from the Federal Court of Canada. To identify with BM25 for sparse vector representation, and Legal GloVe
the entailing paragraphs p for a given query paragraph and Legal Siamese BERT for semantic embeddings. The
q, the authors employ both lexical and semantic ranking values on the answer evaluation scale serve as a threshold
techniques to rank the candidate paragraphs. They evaluate to determine which answers to accept and which to reject.
the performance of two approaches: BM25 and lawDPR. Many of the solutions implemented in this study hold notable
The most effective results were achieved by combining relevance and are of interest for future developments.
BM25 with dense passage retrieval, utilizing domain-specific In P53 [77], the authors introduce FRAQUE, a system
embeddings to enhance accuracy. designed to answer factual questions within the Public
It is also worth reporting that for this task (i.e., entailment) Administration sector. This system utilizes semantic frames,
the most recurrent approaches are based on domain-specific which are structured collections of slots with defined possible
embedding or BERT adaptation to the legal domain. Even values. FRAQUE operates by querying unstructured data
if the tasks related to entailment are not yet fully solved from various sources such as documents, websites, and
in the literature, LLM-based approaches have consistently social media. Leveraging statistical components like word
outperformed traditional and statistical method. This is due embeddings, the system offers flexibility in adapting to
to their ability to identify contextual and semantic relations different domains and languages. The primary objective of
between words. FRAQUE is to match questions with relevant frames and
corresponding document passages stored in a knowledge
3) QUESTION ANSWERING (T3) graph, which are then presented as answers. To ensure
This task is focused on the application of NLP and machine user-friendliness, FRAQUE’s development follows a user-
learning techniques to automatically provide answers to centered design approach, allowing for the monitoring of
legal queries, which involves understanding and interpreting linguistic patterns used by users and identifying the most
complex legal questions and retrieving or generating accurate frequently occurring structures in their queries.
and relevant answers based on legal texts such as statutes, Both P49 [76] and P53 [77] highlight the importance of
case law, regulations, and legal opinions [18], [45]. We have developing question-answering systems for legal and public
found two studies for this task (P49 and P53) which are administration domains. The use of sparse vector techniques
embedded within the Legal Research and Miscellaneous legal like BM25 and dense vectors (semantic embeddings) in P49
areas. and pattern-based systems in P53, have shown promising
Understanding legal texts presents a considerable chal- results in addressing legal and factoid questions, respectively.
lenge due to their extensive and intricate clauses, com- The user-centred design process employed in P53 ensures the
pounded by a scarcity of datasets annotated by experts. system’s usability and adaptability to different domains and
To tackle this issue, P37 [75] introduced the Merger languages. Overall, the development of such systems has the
Agreement Understanding Dataset (MAUD), a reading potential to significantly improve the efficiency and accuracy
comprehension dataset meticulously annotated by experts of legal and public administration processes.
based on the American Bar Association’s 2021 Public Target
Deal Points Study.5 With over 39,000 examples and more
than 47,000 annotations, MAUD serves as a significant B. LEGAL DOCUMENT REVIEW
resource. The authors assert that MAUD stands as the 1) NAMED ENTITY RECOGNITION (T4)
sole expert-annotated merger agreement dataset, making Named Entity Recognition (NER) is a crucial stage in many
it an invaluable benchmark for both the legal profession legal AI tasks, facilitating the organization of unstructured
and the NLP community. Their fine-tuned LLMs displayed text data into a more structured and searchable format [43].
promising performance, consistently outperforming random We reviewed 16 studies that contribute to the advancement
chance on most questions. of NER in the legal domain. These studies encompass the
In P49 [76], a question-answering system is introduced introduction of new datasets, development of NER systems,
to address legal queries. Sparse vectors and embeddings are creation of language models, evaluation of models, and
employed as input for a BERT-based answer ranking model. proposition of legal annotation procedures. The primary
The distinction is made between factoid and non-factoid focus of these studies is Case Law, although other legal areas
are also covered.
5 [Link] Several studies have introduced new datasets for NER.
mergers-and-acquisitions/deal-points/ (Accessed: Dec. 28, 2024) These include a dataset comprised of German federal court

VOLUME 13, 2025 18261

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

decisions (P34 [78]), which were evaluated using Conditional Document Classification. The experimental results revealed
Random Fields (CRF) and Bidirectional Long Short-Term that UmBERTo outperformed other Transformers for both
Memory (Bi-LSTM). Another dataset, CUAD, was curated tasks. The Transformers considered were BERT-BASE-
for legal contract review (P35 [79]), with experiments con- ITA, UmBERTo, Multilingual BERT, XLM-RoBERTa, and
ducted using Transformer-based architectures such as BERT, GePpeTto, the latter being the first autoregressive Italian
RoBERTa, AlBERT, and DeBERTa. Additionally, MAUD, language model built upon GPT-2. Furthermore, the authors
a dataset based on the 2021 American bar examination, was introduced a novel dataset specifically for this task, derived
presented in (P37) [75]. P42 utilized a publicly available from a corpus they constructed, named the ‘‘ATTO’’ Corpus.
legal NER dataset (ENER), with a subset of filings extracted The results consistently demonstrated UmBERTo’s superior
for training different NER algorithms on the general English performance across various tasks and dataset variants.
CoNLL-2003 corpus. In conclusion, the recent studies reviewed in this section
In terms of NER system development, P36 [80] presented have provided significant contributions to the advancement
a system within the Romanian legal domain. The authors of NER in the legal domain. The introduction of new
curated a manually annotated corpus, MARCELL-RO, which datasets, such as those comprised of German federal court
encompasses legal documents extracted from the extensive decisions, CUAD, MAUD, and E-NER, has expanded the
MARCELL project. resources available for training and evaluating NER models.
Regarding language models, P38 [81] introduced LEGAL- The development of NER systems, like the one presented in
BERT, a family of language models adapted to the specific P36 [80], has improved the accuracy and efficiency of entity
legal domain through domain-specific pre-training or trained recognition within legal documents. The creation of language
from scratch. The performances achieved were superior models, such as LEGAL-BERT and BureauBERTo, has
to using BERT-base. The same study introduced LEGAL- further enhanced the performance of NER tasks by adapting
BERT-SMALL, with experiments conducted on several to the specific legal domain and extending base vocabularies.
datasets, including one for NER focused on USA contracts. The evaluation of these models has demonstrated their supe-
P39 [82] extended the base vocabulary of the Italian riority over general-purpose language models in handling
Transformer model UmBERTo, resulting in BureauBERTo, legal text. Moreover, the proposition of novel architectural
through further pre-training on a corpus of documents approaches, like the hierarchical Bi-LSTM in P40 [84],
related to the Italian Public Administration, banking, and has opened new avenues for improving the structural
insurance sectors. Two evaluation strategies were employed: understanding of legal documents. The comparison of various
an intrinsic strategy involving predicting masked words Transformer models in P44 [86] has also highlighted the
within the sequence, and an extrinsic approach involving effectiveness of domain-specific models, such as UmBERTo,
a specific NER task tailored to the Public Administration in handling legal NER tasks.
domain. Also related to the Italian domain, in P41 [83], the However, despite these advancements, challenges remain.
authors evaluated NER tasks using ITALIAN BERT + Spacy The complexity and variability of legal language, as well
NER and ITALIAN-LEGAL-BERT + Spacy NER. For the as the need for large annotated corpora, continue to pose
semantic similarity task, a subset of sentences from the Italian obstacles. Future research should focus on addressing
Civil Law DB was utilized. these challenges, further improving the performance and
A novel architectural approach based on a hierarchical applicability of NER in the legal domain.
Bi-LSTM was proposed in P40 [84], with the SemEval
task encompassing three subtasks: RR Prediction, L-NER, 2) SIMILARITY ESTIMATION (T5)
and CJPE. The proposed model, HLBERT-CRF, incorporates In the context of legal AI, this tasks focus on measuring how
both word-level and sentence-level encoders. For tokeniza- closely related or alike two pieces of legal text are, which
tion, a LEGAL-BERT-compatible tokenizer was employed, can be comprised of comparisons between various types of
and F1-score and weighted F1 were used as evaluation legal documents such as court cases, statutes, contracts and
metrics. legal opinions [43], [46]. We have identified 8 studies that
P42 [85] tested various NER algorithms on the CONLL- approach this task (P04, P05, P09, P11, P12, P19, P20, P33),
2001 corpus and evaluated them using the provided dataset and most of the studies are embedded within the Case Law
collection. The paper also describes the E-NER corpus, and Legislation areas.
an annotated collection of legal documents. It contains One common theme among these studies is the impor-
detailed annotations for entities like case numbers, court tance of domain-specific pretraining and the use of text
names, statutes, and legal parties, making it a valuable similarity techniques. In P04 [23], the authors introduced
resource for training models that can accurately extract criti- a dataset called CaseHOLD and evaluated the performance
cal information from complex legal texts, streamlining tasks of different models on it. The results showed that domain
such as case law research and contract analysis. In P44 [86], pretraining with a custom legal vocabulary exhibited the
a comparison was made among five Transformers pretrained most substantial performance gains, with a 7.2% gain on F1,
on general-purpose text for two tasks within the domain representing a 12% improvement over BERT. This approach
of Public Administration in Italy: NER and Multi-Label also demonstrated consistent performance gains across other

18262 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

legal tasks. Similarly, in P09 [68], the authors proposed a text based on their content, using machine learning and NLP
similarity approach for retrieving previous cases and statutes. techniques to automate the organization and analysis of
They applied three variations of word representation based large volumes of legal information [24], [46], [47]. We have
on Glove, Doc2Vec, and TF-IDF methods. The experiments found 22 studies (P03, P10, P12, P21, P24, P26-P31,
demonstrated that the TFI-DF method achieved reasonable P38, P41, P44, P48, P50, P51, P52, P54, P55-P57) that
results compared to Doc2Vec and Glove methods, which refer to this task, where Case law is the most frequently
usually require large training datasets. In P11 [87], the addressed area, appearing in 16 articles, indicating its critical
authors tackled the challenge of measuring the similarity importance in legal research. Legislation is also a significant
between two legal cases. They introduced Hier-SPCNet, focus, present in 13 articles, reflecting the ongoing need to
an enhancement of PCNet that incorporates a heterogeneous understand and interpret statutory laws. Legal documents
network of statutes or written laws relevant to the jurisdiction. are prominently featured in 10 articles, emphasizing the
The experiments were conducted using data from the Indian necessity of understanding various legal texts. Additionally,
judiciary, where the benchmark similarity between document miscellaneous topics and legal research are covered in a few
pairs was assessed by legal experts from two esteemed law articles, highlighting emerging areas and interdisciplinary
institutes in India. The findings demonstrated that Hier- approaches within legal studies.
SPCNet achieved state-of-the-art performance in network- Several studies have proposed different approaches to
based legal document similarity. tackle various legal text classification tasks. Since it is not
Another common point among these studies is the use feasible to discuss all the existent literature, in this paragraph
of document clustering and topic modelling techniques. we provide an overview of four representative studies, namely
In P12 [69], the authors created LamBERTa, a specialized P03, P12, P24, P26 and P27 and highlights their findings.
deep learning framework for civil-law codes, specifically P03 [89] introduced LBOX OPEN, a dataset consisting
trained using the Italian civil code. This framework involves of a corpus and two classification tasks, namely case
refining an Italian pre-trained BERT model on either the name and statute prediction from the factual description of
entire Italian civil code or parts of it, treating the retrieval individual cases. The authors also released realistic variants
of law articles as a classification task. The authors utilized a of the datasets by extending the domain to infrequent case
centroid-based partition clustering algorithm on a document- categories in case name and statute classification tasks.
term matrix. They constructed a vectorial Bag-of-Words P12 [69] presented LamBERTa, a model derived from
(BoW) model within the term feature space, employing TF- BERT and pre-trained on Italian Civil Code laws. Through
IDF for term relevance weighting and cosine similarity to a classification stage, the model predicts the most relevant
compare documents. In P33 [88], the authors evaluated the articles from the Italian Civil Code. The study evaluated
use of BERTopic for topic modelling in legal documents. two learning approaches, a global one based on the entire
The researchers concentrated on a selection of landmark Italian Civil Code, and a local approach based on individual
cases from the US Caselaw dataset to assess the impact books. The results showed that the local learning-based
of topic modelling, utilizing domain-specific embeddings method outperformed the others, and LamBERTa exhibited
pre-trained with LEGAL-BERT. Their findings indicate that generally superior performance compared to the other models
incorporating references to statutory law, such as the US considered.
Code, during the text embedding process enhances the quality P24 [90] introduced LEXTREME, a new benchmark for
of topic modelling. evaluating NLP models on legal tasks. The benchmark
In conclusion, these studies highlight the potential of comprises three selections of 11 datasets spanning a total of
text similarity, document clustering, and topic modelling 24 languages. The three classification tasks are Single Label
techniques for legal document analysis. The proposed Text Classification (SLTC), Multi Label Text Classification
methods have shown promising results in improving the (MLTC), and Named Entity Recognition (NER). The best-
performance of legal document analysis tasks, such as law performing baseline model (XLM-R) achieved an aggregated
article retrieval, case retrieval, and topic modelling. However, final score of 61.3.
further research is needed to address the challenges posed P26 [47] applied the sliding window concept based on
by the complexity and variability of legal language and the attention, previously proposed with Longformer, to Dis-
need for large annotated corpora. The development of more tilBERT. The proposed model consists of 6 Longformer
sophisticated techniques for legal document analysis could attention heads and a total of 69 million parameters. The
have significant implications for the legal industry, enabling maximum context window size is increased to 4096, allowing
more efficient and accurate analysis and retrieval of legal the model to process texts up to 8 times longer than
documents. standard BERT [91], [92]. The authors used a pre-training
dataset of 8GB, consisting of five types of documents, and
3) CLASSIFICATION (T6) applied two pre-training strategies. The results showed that
Through the classification task, it is possible to categorize the proposed model outperformed BERT, RoBERTa, and
legal texts or documents into predefined classes or categories DistilBERT.

VOLUME 13, 2025 18263

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

Finally, P27 [93] addressed the classification of legal cases expert-annotated summaries. The authors leveraged Legal-
by employing BERT, RoBERTa, Legal-BERT, Longformer, BERT, pre-trained on Harvard’s legal case corpus, and pro-
and LegalFormer. The authors divided the documents into posed a multitask model to distinguish Reasoning/Evidence
segments with a maximum size of 512 tokens and selected the from other rhetorical roles. Comparative baselines included
text fragment that yielded the best result given the considered MMR and TextRank, and evaluation metrics encompassed
metric. The results showed that Legal-BERT and Legal- ROUGE-1 and ROUGE-2.
Longformer achieved the best performance. In conclusion, the studies discussed in this paragraph
In conclusion, the studies discussed in this paragraph highlight the potential of NLP techniques for summarizing
propose different approaches to tackle various legal text legal documents. However, the results also indicate that there
classification tasks. The findings suggest that pre-training is still room for improvement in terms of summary quality
models on domain-specific data and employing attention- and accuracy, particularly in low-resource environments and
based mechanisms can significantly improve the performance for complex legal documents. The use of domain-specific pre-
of legal text classification. However, it is worth noting that trained models, such as LEGAL-BERT, and the creation of
some studies lack comparisons with more contemporary new datasets, such as ITA-CaseHold and Multi-LexSum, are
iterations of GPT models, and further research is needed to important steps towards improving the performance of NLP
explore their potential in legal tasks. techniques in summarizing legal documents.

4) DOCUMENT SUMMARIZATION (T7) 5) DATASETS AND BENCHMARKING (T8)

This task allows generating concise and coherent summaries Datasets and Benchmarking help to measure and compare
of lengthy legal documents, extracting the most relevant the performance of AI models in the legal domain, and also
information from legal texts, such as court cases, statutes foster the creation of new datasets that can become available
and contracts, making it easier to understand for legal for the community; these are crucial elements for advancing
professionals [48], [49]. We identified 6 studies (P03, P13, research, development, and application of AI technologies in
P27, P58, P59, P60) that have considered this task, and all of legal contexts [50]. We have found 15 studies that focused
them are focused on Case Law as legal area. on this field, most of them introducing new datasets (P02,
A recent study - P03 [89] - presented LBOX OPEN, a legal P03, P04, P17, P21, P22, P23, P32, P34, P35), proposing new
corpus with Korean precedents, along with a summarization benchmarks (P24, P56,), new benchmark datasets (P14, P18,
task consisting of Supreme Court precedents and their P58) or new corpus (P61). Table 4) summarized the datasets
corresponding summaries. Another study - P58 [83] - introduced that can be categorized as court cases (P02, P14,
introduced a system for extracting legal holdings from Italian P32), court precedents (P03), and various legal documents
judgments. The problem was modelled as a summarization such as holdings (P04), contracts (P17, P22, P35), and facts
task, with a focus on identifying salient passages. The authors from ECHR rulings (P21, P56).
curated a new dataset, ITA-CaseHold, containing over 1,100 Additionally, there is a strong emphasis on developing
pairs of judgments and their corresponding holdings. The benchmarks for court precedents, case names, statutes, and
authors fine-tuned Italian-LEGAL-BERT-SC to predict the various types of legal decisions (P03, P24, P58). Contract
most relevant sentences within a document, and the proposed review is another area of interest, with studies focusing
model outperformed other baseline models in terms of Rouge on analysing contract terms and conditions (P18, P35).
R-1 and R-2 metrics. Furthermore, NER datasets are being created to improve
P59 [94] addressed the Civil Rights Litigation Clearing- the extraction of information from court decisions (P34
house (CRLC) and introduced Multi-LexSum, a collection [78]). The trend also includes benchmarking legal texts
of 9,280 summaries authored by domain experts and derived using support vector machine classifiers on the LexGLUE
from CRLC documents. The authors demonstrated that state- dataset, which encompasses facts from ECtHR rulings, court
of-the-art models performed inadequately on this dataset. The opinions, and online terms of service (P56 [96]). Overall,
authors employed the best-performing Transformers for sum- these trends underscore the critical role of creating and
mary generation, including BART, PEGASUS, PRIMERA, evaluating new datasets and benchmarks to advance the
and LED. However, human evaluation of the generated capabilities of legal AI in handling diverse legal documents.
summaries revealed an average modification of 87 tokens per The performance and effectiveness of LLMs in legal
paragraph, 76% paragraph length modification, and 65% total domain are evaluated using a combination of standardized
length modification, highlighting the unsatisfactory perfor- benchmarks and custom criteria tailored for the legal domain.
mance of all the Transformers considered in comparison to Standardized benchmarks such as LexGLUE and CUAD
expert-authored summaries. provide a robust framework for comparing the performance
To address the challenge of limited data annotated by of different LLMs across various legal NLP tasks. LexGLUE
experts, P60 [95] introduced extractive summarization tech- includes tasks like legal judgment prediction, statute law
niques for legal decisions in a low-resource environment. The retrieval, and contract review, while CUAD is specifically
dataset consisted of 112 decisions and their corresponding designed for contract review and understanding. In addition

18264 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

to these benchmarks, custom criteria that are tailored to the TABLE 4. Datasets and benchmarking: Legal areas and contributions.
specific needs of the legal domain are present in the literature.
These include domain-specific metrics such as precision,
recall, and F1-score for tasks like named entity recognition
(NER) in legal documents, legal text classification, and case
outcome prediction. Also, evaluations by legal experts who
assess the practical relevance and accuracy of the outputs
generated by LLMs are reported in the literature. This
expert evaluation helps ensure that the models meet the high
standards required in legal applications. Furthermore, given
the importance of context in legal texts, the models’ ability
to understand and generate contextually accurate responses is
also evaluated, which is particularly crucial for tasks like legal
document summarization and contract analysis. By using a
combination of standardized benchmarks and custom criteria,
the aim is to provide a comprehensive assessment of the
performance and effectiveness of LLMs in legal applications.
One of the pivotal datasets within the legal domain is
introduced in P21 [50] which represents a compilation of
distinct sub-datasets aimed at assessing the performance
across various Natural Language Understanding (NLU) tasks
within the legal domain. It also offers an evaluation of
several generic and legal-oriented models. The LexGLUE
dataset comprises seven NLP datasets that are domain-
specific, thoughtfully selected using SuperGLUE criteria.
The most prevalent task pertains to the prediction of legal
case outcomes. Subsequent tasks include Topic Classifi-
cation, Information Extraction, Legal Question/Answering,
and others. In the initial release of this dataset, the sole
language considered was English. Beyond the utilization of
Transformers, this study also encompasses the evaluation
of Support Vector Machines (SVM). The SVM model’s
performance was fine-tuned using TF-IDF representation and
hyperparameter optimization through grid search. Notably,
the SVM exhibited particularly promising results, although
LEGAL-BERT generally outperformed other models across
all datasets. The LexGLUE dataset and benchmark are readily
accessible via Hugging Face, along with the associated In P23 [98], a corpus of 65,000 laws pertaining to the
experimental code.6 European Union is employed. Taxonomic labels are derived
In P22 [97], the authors present an expanding dataset, from Eurovoc.7 These laws, issued by the single European
currently encompassing approximately 256 gigabytes of countries, undergo translation from the 23 official languages.
data. This dataset comprises a wide array of legal and A significant portion of the dataset pertains to the issue
administrative documents, including judgements, contracts, of contractual advantage imbalance between contracting
and legal materials released by the European Parliament. parties. The dataset is meticulously stratified, comprising
Notably, they initiated their work from a BERT-based model. over 50,000 samples in the training set spanning the period
A primary objective of this endeavour is the creation of a from 1958 to 2019. Additionally, approximately 5,000
dataset devoid of privacy violations and toxic content. This samples are allocated both to the development set, covering
ensures that any model, be it a LLM or a traditional one, the years 2010 to 2012, and the test set, from 2012 to 2020.
can be trained free from biases. The presented results have Notably, this dataset is readily available on Hugging Face for
shown that smaller models pretrained on domain-specific seamless integration with Transformer models.8
data exhibit superior performance compared to their more In P14 [72], the authors evaluate 27 methods, including
broadly pretrained, larger counterparts. both Transformer-based and traditional architectures, using
7 Eurovoc, available at: [Link]
(Accessed: Dec. 28, 2024)
6 LexGLUE dataset on Hugging Face, available at: [Link] 8 Eurovoc on Hugging Face, available at: [Link]
datasets/lex_glue (Accessed: Dec. 28, 2024) multi_eurlex (Accessed: Dec. 28, 2024)

VOLUME 13, 2025 18265

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

standard benchmarks derived from Open Case Book and dataset comprises 47,457 annotations derived from legal texts
Wiki source. The task’s objective is to retrieve relevant extracted from 152 English-language merger agreements.
literature for a specific case. The evaluated methods fall into MAUD serves as a multiple-choice reading comprehension
three categories: word-vector based, Transformer based, and task, with performance evaluation conducted using the Area
citation based. They have created the benchmark datasets Under the Precision and Recall Curve (AUPR) metric.
specifically for this study. Literature recommendations are Notably, this study demonstrates that larger and more recent
considered correct if they cover the same topic or provide architectural models yield improved results on the proposed
essential background information for the case at hand. task.
Specifically, a recommendation is deemed accurate if the We can conclude that datasets and benchmarking are
suggested case is found in the same case-book or within the crucial for advancing AI technologies in the legal domain.
same category as the case being examined. The authors used They enable the improvement of the performance of AI
Mean Average Precision (MAP) and Mean Reciprocal Rank models on various legal tasks. Studies have introduced
(MRR) as evaluation metrics. Notably, Poincaré embeddings important datasets like LexGLUE for NLP tasks, CUAD
were generated in hyperbolic space rather than Euclidean for contract review, and MAUD for merger agreements.
space. The study confirms that due to the token length These efforts underscore the importance of curated datasets
limitations of Transformers, they may not always be the and benchmarking in enhancing the capabilities of legal AI
most effective choice. The best results in both datasets were across different types of legal documents and tasks. However,
achieved through a hybrid combination of FastTextLegal and it is important to note the limitations and diversity issues
Poincaré. within these datasets. While datasets like CUAD, MAUD,
In P34 [78], the authors introduced a dataset for NER and E-NER have been introduced, their generalizability
specifically tailored to decisions from the German Federal across different legal subdomains remains a challenge.
Court. The dataset comprises approximately 67,000 sen- These datasets often reflect the specific legal contexts and
tences with over 2 million tokens. It includes 54,000 manually jurisdictions in which they were created, which may limit
annotated entities, categorized into 19 detailed semantic their applicability to other legal systems. For instance,
classes. These classes are: person, judge, lawyer, landscape, datasets focused on common law jurisdictions may not
street, city, country, organization, company, institution, generalize well to civil law systems due to differences in
court, brand, law, ordinance, European legal norm, contract, legal frameworks and terminology. Additionally, the diversity
regulation, court decision, and legal literature. Remarkably, within these datasets can be limited, as they may not fully
this comprehensive dataset draws upon the content of represent the wide range of legal scenarios and demographics
seven distinct datasets derived from seven different courts. encountered in real-world applications. Addressing these
Evaluation of the dataset employs two prominent models: limitations requires ongoing efforts to create more diverse
Conditional Random Fields (CRF) and Bidirectional Long and representative datasets that can better generalize across
Short-Term Memory (Bi-LSTM). different legal subdomains.
In P35 [79], a novel dataset, known as the Contract
Understanding Atticus dataset (CUAD), is introduced.
CUAD is meticulously curated for legal contract review and 6) DOCUMENT AUTOMATION (T9)
was collaboratively developed by dozens of legal domain The integration of AI in document review and automation
experts associated with the Atticus Project. The core task represents a significant advancement in the legal industry,
involves annotating salient sections of legal contracts, with improving efficiency, accuracy, and scalability. By leveraging
the ultimate objective of reducing the time and costs AI technologies, legal professionals can handle complex
typically associated with performing this task by one or document review tasks more effectively, allowing them to
more domain experts. This dataset encompasses more than focus on higher-value work and deliver better outcomes for
500 contracts and boasts over 13,000 annotations spanning their clients [51]. We have found 5 studies that focused on
across 41 labels or categories. Notably, CUAD stands out this task (P17-P19, P25, P35).
as one of the relatively few machine learning datasets The contract review process is a highly time-consuming
that have been diligently curated by domain experts. The and costly task. Small companies often sign contracts without
experiments were carried out exclusively using Transformer- an additional review stage. It is within this context that
based architectures, among which include BERT, RoBERTa, P17 [99] addresses the topic of contract review, with a
AlBERT, and DeBERTa. DeBERTa and RoBERTa emerged specific focus on evaluating the fairness of a contract. They
as the top-performing models, yet with significant room have also released a corpus of 607 annotated contracts.
for future enhancements, as for the case of performance In their work, the authors demonstrate that certain linguistic
imbalance for each category. features, such as negations, significantly complicate the
In P37 [75], the authors present MAUD, a dataset task. Also, they intend to determine, framing it as an
based on the 2021 American bar examination. The paper NLI task, whether a given hypothesis implies, contradicts,
introduces the concept of Merger Agreements, which are or is not mentioned in relation to the entire text of the
legal documents governing corporate acquisitions. The contract. In this context, the task can be broken down as

18266 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

follows: a) NLI that involves sentence classification for the in lease agreements. P19 [101] and P20 [65] illustrate the
hypothesis concerning the three mentioned classes: entailed, effectiveness of generative models and document embedding
contradiction, not mentioned; and b) Evidence Identification techniques in contract drafting and the identification of reg-
that involves identifying the text span associated with the ularities within legal documents. These findings underscore
hypothesis as mentioned above (in the case of entailed the potential for AI-driven solutions to streamline the contract
or contradiction). The proposed final model (SPAN NLI review process, making it more efficient and accessible,
BERT) achieves significantly better performance than all the particularly for small companies. However, certain linguistic
considered baselines. features, such as negations, can complicate the task, and
In P18 [100], the authors focus on detecting two crucial further research is needed to address these challenges.
elements in contract reviews: entities and red flags, which Additionally, while AI models such as Chat-GPT and Vicuna
are terms or sentences that indicate that there is some danger show promising results, human intervention is still necessary
for one or more of the signing parties. Their focus is solely to control hallucination phenomena and ensure the fairness
on leasing contracts. The released benchmark comprises and legal-compliances of contracts.
179 documents. Additionally, a new Language Model tailored
for this task, called ALeaseBERT, is introduced. C. LEGAL PREDICTION
In P19 [101] and in P20 [65], AI, including generative 1) JUDGEMENT PREDICTION (T10)
models, is employed to assist users in contract draft- Judgement prediction refers to forecasting the outcomes of
ing. In P19, the public dataset used for this purpose is legal cases which involves analysing vast amounts of legal
LEDGAR [101]. LEDGAR is a multilabel corpus containing data, including past court decisions, case facts, and applicable
legal provisions from contracts. It was created by collecting laws, to predict the verdicts of ongoing or future cases.
and extracting data from publicly available SEC filings, and, The primary purpose of judgement prediction is to provide
to the best of the authors’ knowledge, is the first open- insights that can assist legal professionals in case strategy,
access corpus of this nature. As the dataset was developed risk assessment, and decision-making [19], [20], [21], [22],
using a semi-automated approach, several noise reduction [52], [53]. We identified six studies focused on judgment
methods were employed and are thoroughly analysed. With prediction (P02, P03, P32, P50, P52), and only one for the
more than 12,000 distinct labels applied to nearly 100,000 Court Judgement Prediction and Explanation task (P01).
provisions across over 60,000 contracts, LEDGAR offers P01 [22] introduced the Indian Legal Documents Corpus
significant potential for advancing research in Legal Natural (ILDC), a repository of Indian legal documents featuring
Language Processing (NLP), particularly in large-scale or 35,000 annotated Supreme Court cases. The ILDC serves
extreme text classification, and for supporting legal studies. as the foundation for the Court Judgement Prediction and
The proposed approach consists of three steps. Working Explanation (CJPE) task. The CJPE task involves predicting
at the keyphrase level, rather than individual words, the the final judgement based on case facts and arguments.
authors aim to capture the semantics of broader concepts, The cases span from 1947 to April 2020 and have been
not just isolated words. They also discuss Vicuna (based on examined using classical machine learning models, such
LLAMA), which achieves 90% of ChatGPT’s performance as word and sentence-level embeddings, logistic regressors,
with only 7% of the total parameters. To control hallucination SVMs, and Random Forest. Additionally, sequential models
phenomena, the authors explore two possibilities involving like Transformers and hierarchical Transformers have been
human intervention: a) direct clause revision, and b) refor- employed, with XLNet and BiGru emerging as the two most
mulating the input provided to steer a new text generation. effective models. The most proficient model achieved an
The framework proceeds through the following three steps: 1) accuracy rate of 78% when compared to a domain expert, who
Sentence Transformer, representing sentences with vectors; attained an accuracy of 94% on the same corpus.
2) Pattern Rank, finding keyphrases most similar to the Similarly, P02 [90] introduced a multilingual dataset
input; and 3) UMAP used for vector dimension reduction. centred around cases from the Swiss Federal Supreme Court,
The dataset comprises 60,540 contracts with a total of comprising over 85,000 cases. The authors employed state-
846,274 clauses. Chat-GPT emerges as the top-performing of-the-art BERT-based models, including those surpassing
model, although Vicuna, with far fewer parameters, still the 512-token limitation. The original available case judge-
delivers notable results. In P20 [65] is proposed a model to ments, considered as labels, encompass: Approval, Partial
assist in drafting legal documents by identifying regularities Approval, Dismissal, Partial Dismissal, Inadmissible and
within paragraphs, using document embeddings and nearest Write Off. The initial four labels pertain to merit, while the
neighbour search, evaluated on the EUR-Lex dataset. last two are based on formal reasons. The models utilized
In conclusion, these studies demonstrate the potential of encompass Standard BERT, LongBERT, and Hierarchical
AI-based solutions to improve the efficiency and accuracy BERT. Results have underscored that, given the dataset’s
of contract review and drafting processes. P17 [99] demon- inherent imbalance, the Majority system tends to perform
strates the potential of NLI-based approaches to enhance favourably concerning micro-F1. However, when considering
contract fairness evaluation, while P18 [100] showcases a Macro-F1, Hierarchical BERT emerged as the superior
specialized language model for identifying critical elements choice.

VOLUME 13, 2025 18267

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

Another language-specific model is presented in P32 [52], of state-of-the-art models applied to the legal domain. The
which introduced a Romanian BERT model pre-trained on a results showed that the Facts section plays a key role in Court
large specialized corpus. The authors claimed that their model Case Predictions. Also, a generalist model trained on all
outperforms several strong baselines for legal judgement articles performs better than a specialized ensemble model.
prediction on two different corpora, consisting of cases Additionally, predicting future events based on past cases is
from trials involving banks in Romania. Also in this study, more difficult than using a mix of past and future cases for
the beneficial effect of the domain-specific pre-training is training.
supported by the results. In conclusion, with regard to this task, there are still
In another domain, P03 [89] presented LBOX OPEN, challenges to be addressed, such as the inherent imbalance
which is a large-scale benchmark of Korean legal AI in legal datasets and the difficulty in predicting negative
datasets. The legal judgment prediction tasks include 10,500 outcomes. Especially in this area, the language-specific
criminal cases, where the model predicts fine amounts and trained models have proved to be the elective choice for state-
imprisonment types based on the facts, and 4,700 civil cases, of-the-art results in judgement prediction.
where the model predicts the degree of claim acceptance
from the facts and claims for relief. The authors introduced 2) NEXT SENTENCE PREDICTION (T11)
LCUBE, an LLM pretrained on the newly created corpus, Next Sentence Prediction using Transformer models like
based on the GPT-2 architecture. The authors highlight their BERT in legal AI offers promising advancements in automat-
result, showing that for more difficult tasks, pre-training ing and enhancing the understanding of legal texts. By lever-
from scratch is more helpful than domain adaptation using aging the capabilities of these models, legal professionals
fine-tuning. can improve the efficiency and accuracy of legal document
Moreover, P50 [102] concentrated on predicting cases with drafting, research, and analysis. With regard to next sentence
negative outcomes instead of predicting only the case of prediction, we have identified 4 studies that contribute to this
positive outcomes. The authors made their entire codebase task (P39, P41, P42, P48), addressing Miscellaneous, Legal
available on GitHub and introduced two probabilistic models Research, Legal Documents and Case Law, respectively (see
to address this challenge. The set of random variables used Table 2).
relates to the article associated with a positive, negative, In P39 [82], the authors introduce BureauBERTo,
or null outcome. Another random variable encodes whether a an enhanced version of the Italian Transformer model
particular article has been cited or not, and a random variable UmBERTo. By performing additional pre-training using
is employed to represent the textual description of the facts. a corpus of documents related to public administration,
The first model they propose (Joint Model) assumes that the banking, and insurance, they expanded UmBERTo’s base
two former random variables are conditionally independent vocabulary. This training methodology underscores the
regarding the i-th article. Furthermore, both variables solely unique characteristics of Italian administrative jargon, which
depend on the facts. The second model presented (Claim- extensively uses domain-specific terms (e.g., ‘‘ravvedimento
Outcome Model) is based on the probability that, given the operoso’’ and ‘‘imponibile’’). Leveraging databases from
facts, articles are chosen first, followed by the prediction of the SEMPLICE9 and ABI2LE10 projects, the additional
a specific outcome. The ECHR was used as the validation pre-training improved performance on proposed tasks,
corpus, utilizing the golden labels provided in [50]. The demonstrating BureauBERTo’s efficacy for legal applications
outcome corpus was generated from this source. The results in the public administration sector.
show that while a basic BERT-based classification model Also in the Italian legal domain, P41 [104] addresses
can predict positive outcomes with an F1 score of 75.06, the limitations of using Transformers pre-trained on general
it only achieves an F1 score of 10.09 for negative outcomes, corpora like Wikipedia. Given the cryptic nature of legal
performing below a random baseline, which reaches an F1 language and the prevalence of Latin-based terms and archaic
score of 11.12. terminology, the authors performed additional pre-training on
Finally, P52 [103] implemented four ML models for the Italian Civil Code. Starting with ITALIAN XXL BERT,
sentence prediction on the dataset of the ECHR. The which had been pre-trained on a large corpus of 81 GB, they
authors investigated the impacts on performance concerning further trained the model using the National Jurisprudential
a) metrics, b) the inclusion of various combinations of Archive, containing millions of legal documents. This pre-
parts of the considered case, c) the effect of more or training resulted in the proposed ITALIAN-LEGAL-BERT
less domain-specialized architectures, and d) the temporal outperforming ITALIAN-BERT by 18.2% in civil cases and
effect of past decisions available. The dataset has been 15.4% in criminal cases, highlighting the importance of
occasionally adapted for the Article Classification and Binary domain-specific pre-training for legal tasks.
Classification tasks. For the former, there are 9 DSs for 9 SEMantic instruments for PubLIc administrators and CitizEns:
each article. The evaluation metrics used are Accuracy and [Link] (Accessed: Dec. 28, 2024)
MCC. The authors noted that the use of MCC as a metric 10 [Link]
significantly reframes the expectations and actual results (Accessed: Dec. 28, 2024)

18268 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

P48 [105] presents ConfliBERT, where the authors demon- techniques and the adaptation of LLMs for specific legal tasks
strate the superiority of both pre-training from scratch and are shaping the future of legal NLP technologies.
continual pre-training over standard BERT. ConfliBERT was In particular, in Document Retrieval, authors have been
implemented using two methods, Cont and SCR, each with combining traditional retrieval methods with Transformers
cased and uncased versions. The model architecture mirrors and integrating similarity detection techniques. Key tech-
BERT-Base, with 12 layers and 110 million parameters. nologies employed include BERT, Sentence-BERT, LEGAL-
By employing domain-specific vocabulary (ConfliVocab) for BERT, BM25, GloVe, Doc2Vec, CatBoost, TF-IDF, and
SCR models and optimizing the learning process without Longformer (e.g., P04, P06, P10). This hybrid approach
the Next Sentence Prediction (NSP) task, the authors leverages the strengths of both traditional and modern
achieved enhanced performance. Pre-training on a corpus of methods to enhance the accuracy and relevance of retrieved
7 billion words using four V-100 GPUs, they found that both documents [106].
approaches significantly improved masked language model For Question Answering, BM25, Legal GloVe, and Legal
loss optimization, underscoring the effectiveness of tailored Siamese BERT are commonly used to develop systems tai-
pre-training strategies for specialized domains. lored for legal queries, leveraging embeddings and traditional
In conclusion, these studies highlight that the extension retrieval methods to improve performance (e.g., P49, P53).
of the vocabulary through additional pre-training on specific These techniques aim to accurately understand and respond
administrative domains, leads to significant performance to complex legal questions.
improvements over general-purpose models, emphasizing the Regarding Named Entity Recognition, combining sequence
necessity of domain-specific pre-training for legal language models with Transformers and performing domain adaptation
processing. Finally, both pre-training from scratch and for specific legal systems has been proved to be effec-
continual pre-training methodologies successfully enhanced tive. Technologies like BERT, LEGAL-BERT, RoBERTa,
performance in the legal domain, demonstrating the critical DeBERTa, Bi-LSTM, CRF, BureauBERTo, and UmBERTo
role of using a domain-specific vocabulary and optimized are employed to enhance entity recognition in legal texts (e.g.,
training strategies in achieving state-of-the-art results. P34, P35, P36).
Moreover, Similarity Detection has seen the use of
VI. TECHNOLOGIES: TRENDS WITHIN LEGAL TASKS AND enhanced embeddings for legal text and the integration
REAL-LIFE IMPLEMENTATIONS of text and network-based similarity measures. Frequently
A. TRENDS used technologies include BERT, LEGAL-BERT, RoBERTa,
As part of the analysis and across various legal NLP Sentence-BERT, GloVe, Doc2Vec, and TF-IDF (e.g., P09,
tasks, we identified that Transformer models such as BERT, P11, P33). These methods improve the identification of
LEGAL-BERT, RoBERTa, and their variants are the predom- similar legal documents and cases.
inant technologies driving advancements. These models are Also, in Classification extensive use of Transformer
extensively used for tasks including judgement prediction, models, along with traditional classifiers for baseline compar-
document retrieval, classification, and NER. A table to isons, is evident. Popular technologies are BERT, LEGAL-
summarize the content of this section is shown in the BERT, RoBERTa, DeBERTa, Longformer, SVM, CNN,
Table 5. LSTM, and Bi-LSTM (e.g., P21, P26, P55). This combination
There is a noticeable trend towards domain-specific ensures robust classification performance across various legal
adaptations of these Transformers, like LEGAL-BERT and texts.
LamBERTa, which are fine-tuned to handle the unique For Document Summarization BERT, PEGASUS, BART,
complexities of legal language and documents. Additionally, LED, and PRIMERA are utilized for both extractive and
the integration of traditional models and embeddings, such abstractive summarization techniques, along with multitask
as SVM, BM25, GloVe, and Doc2Vec, with Transformers learning approaches (e.g., P03, P60, P59). These technologies
indicates a hybrid approach where conventional methods help generate concise and informative summaries of lengthy
complement the capabilities of newer, more sophisticated legal documents.
models. Another significant trend is the increasing use of Datasets and Benchmarking efforts have focused on
LLMs like ChatGPT and Vicuna for tasks such as contract creating multilingual and multi-label datasets, evaluating pre-
review and automation. This demonstrates the evolving trained models on legal data. Technologies such as BERT,
landscape where these powerful models are being leveraged RoBERTa, DeBERTa, Longformer, MiniLM, DistilBERT,
to handle more complex and nuanced legal tasks, providing mDeBERTa-V3, and XML-R are frequently used (e.g., P21,
more comprehensive and contextually accurate outputs. P22, P23). These datasets and benchmarks are crucial for
The development and benchmarking of multilingual and advancing legal NLP research.
multi-label datasets also highlight the growing emphasis on Likewise, in Document Review and Automation leveraging
creating more inclusive and versatile models that can perform LLMs models for contract drafting and automation, along
across different legal systems and languages. Overall, the with creating domain-specific BERT extensions, is common.
integration of advanced Transformer models with traditional Technologies like Vicuna, ChatGPT, Span NLI BERT, and

VOLUME 13, 2025 18269

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

TABLE 5. Technologies and trends in legal NLP tasks.

ALeaseBERT are utilized to streamline and enhance the • Kira Systems.12 Kira Systems offers an AI solution
automation of legal document review (e.g., P17, P19). for contract analysis that uses machine learning to
In Judgement Prediction the most used technologies extract and analyze key information from contracts.
are BERT, LEGAL-BERT, BiGRU, RoBERTa, Logistic The tool has been implemented by numerous law firms
Regression, SVM, Random Forest, XL-Net, and Longformer. and corporate legal departments to streamline contract
Authors employ hierarchical models, domain-specific BERT review processes and improve accuracy.
models, and multilingual capabilities to predict legal judg-
ments effectively (e.g., P01, P02, P03). These technologies 2) LEGAL DOCUMENT ANALYSIS
enable more accurate and nuanced predictions in judicial • LexisNexis.13 LexisNexis has integrated AI and NLP
contexts. technologies into its legal research platform to enhance
We can conclude that Transformer models such as BERT, document analysis and retrieval. The platform uses
LegalBERT, RoBERTa, and their variants dominate the LLMs to understand and categorize legal documents,
field, while traditional models like SVM, BM25, GloVe, making it easier for legal professionals to find relevant
and Doc2Vec are still used alongside newer technologies. information and precedents.
Domain-specific Transformer adaptations (e.g., LegalBERT, • Westlaw Edge.14 Westlaw Edge, a product of Thomson
LamBERTa) highlight the importance of fine-tuning for Reuters, employs AI to provide advanced legal research
legal tasks. The integration of large language models (e.g., capabilities. The platform uses NLP to analyze legal
ChatGPT, Vicuna) for contract review and automation is an documents, identify key issues, and predict case out-
emerging trend, reflecting the evolving landscape of legal comes, helping lawyers make more informed decisions.
NLP research.
3) CASE PREDICTION AND OUTCOME ANALYSIS
B. REAL-LIFE IMPLEMENTATIONS • CaseCrunch15 (now Luminance). CaseCrunch, now
To provide a practical context for our survey, we discuss known as Luminance, developed an AI system for
several real-life implementations of LLMs in the legal predicting the outcomes of legal cases. The platform
domain. These examples demonstrate the feasibility and uses machine learning algorithms to analyze case data
impact of LLMs in enhancing legal practice and decision- and provide predictions, assisting lawyers in assessing
making. the strength of their cases and developing strategies.
• Premonition.16 Premonition is an AI-powered litigation
1) CONTRACT REVIEW AND ANALYSIS analytics tool that predicts case outcomes based on
• ROSS Intelligence.11 ROSS Intelligence is an AI- historical data. The platform has been used by law firms
powered legal research tool that uses NLP and machine and corporations to assess litigation risks and make data-
learning to assist lawyers in legal research and contract driven decisions.
review. The platform helps legal professionals find 12 [Link] (Accessed: Dec. 28, 2024)
relevant case law, analyze contracts, and identify key 13 [Link] (Accessed: Dec. 28, 2024)
clauses, significantly reducing the time and effort 14 [Link] (Accessed: Dec.
required for these tasks. 28, 2024)
15 [Link] (Accessed: Dec. 28, 2024)
11 [Link] (Accessed: Dec. 28, 2024) 16 [Link] (Accessed: Dec. 28, 2024)

18270 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

4) REGULATORY COMPLIANCE to improve the performance of AI tools on various legal tasks.

• Ascent.17 Ascent uses AI to automate regulatory com- Studies have introduced important datasets like LexGLUE
pliance for financial institutions. The platform employs for NLP tasks, CUAD for contract review, and MAUD for
NLP to ananalyseegulatory texts and identify relevant merger agreements. These efforts underscore the importance
obligations, helping organizations stay compliant with of curated datasets and benchmarking in enhancing the
complex and ever-changing regulations. capabilities of legal AI across different types of legal
• [Link].18 [Link] offers an AI-driven documents and tasks.
regulatory compliance solution that uses NLP to monitor
and analyse regulatory changes. The platform helps
B. (RQ2) IS THE PRE-TRAINING OR THE FINE-TUNING OF
organizations track and comply with regulatory require-
LLMS SPECIFICALLY FOR THE LEGAL DOMAIN ALWAYS
ments, reducing the risk of non-compliance.
BENEFICIAL? AND IN WHAT CASES AND FOR WHAT
These real-life implementations represent practical appli-
TASKS?
cations and benefits of LLMs in the legal domain. By high-
1) DOCUMENT EMBEDDING MODELS ARE OFTEN
lighting these examples, we aim to provide a comprehensive
EMPLOYED FOR CAPTURING SEMANTIC RELATIONSHIPS IN
understanding of the current state and potential of LLMs in
LEGAL DOCUMENTS
enhancing legal practice and decision-making.
The studies highlighted the importance of document embed-
ding models in capturing semantic relationships between
VII. CONCLUSION
documents and retrieving relevant information. These models
In this work, we have explored recent applications of LLMs
can learn to represent legal documents as dense vectors,
models to the legal domain. After performing a detailed
enabling efficient comparison and retrieval in terms of
review of the recent literature, we formulated four research
distance in vector space. According to several studies, word
questions. According to all the works discussed in the
embeddings trained from scratch are able to outperform
previous sections, here we conclude our work listing our main
pre-trained embedding and traditional approaches as TF-
findings and mentioning the specific RQs formulated in the
IDF. Thanks to the word embedding, the studies highlight
Section I and answered here.
the potential of text similarity, document clustering, and
topic modelling techniques for legal document analysis.
A. (RQ1) WHAT ARE THE NLP APPLICATIONS IN THE
The proposed methods have shown promising results in
LEGAL DOMAIN THAT HAVE ALREADY MADE USE OF
improving the performance of legal document analysis
LLMS?
tasks, such as law article retrieval, case retrieval, and topic
1) LEGAL SEARCH IS A CRITICAL COMPONENT OF LEGAL
modelling.
RESEARCH AND PRACTICE
The studies highlight the importance of document retrieval,
case entailment and question answering in facilitating legal 2) TASK-SPECIFIC MODELS USUALLY OUTPERFORM
research, case preparation, document drafting, and client GENERAL-DOMAIN PRE-TRAINED MODELS ON LEGAL
advice. The legal domain is characterized by a vast amount DOCUMENT RETRIEVAL TASKS
of unstructured and semi-structured information, making Legal texts often contain synonyms and specialized jargon,
efficient document retrieval crucial for legal professionals which can affect similarity results. The current approaches
to make informed decisions. The majority of the studies include several mechanisms to handle these aspects effec-
are mainly focused on case law retrieval, demonstrating its tively. Pre-trained language models like BERT, Sentence-
significance in the legal domain. Case law retrieval involves BERT, and LEGAL-BERT capture semantic relationships
identifying relevant precedents and previous judgments to and understand context, helping to recognize synonyms and
inform current legal decisions. The complexity of this task interpret jargon accurately. Domain-specific fine-tuning fur-
lies in capturing the nuances of legal language and the context ther enhances these models’ ability to understand legal terms.
in which legal cases are presented. Embedding techniques such as GloVe and Doc2Vec generate
word and document embeddings that capture semantic
2) REAL-WORLD DATASETS ARE REQUIRED FOR TRAINING similarities. Additionally, some approaches incorporate legal
AND EVALUATING LEGAL DOCUMENT RETRIEVAL MODELS synonym dictionaries and ontologies to map different terms
The studies relied on real-world datasets, such as LexGLUE, to their standardized forms. Contextual analysis, combined
CaseHOLD, and EUR-Lex, to train and evaluate their models. with models like BM25 and TF-IDF, helps in handling the
Real-world datasets provide a realistic representation of variability in legal language by considering the context in
the complexities and nuances of legal language, enabling which terms appear. The studies emphasized the importance
models to learn from practical applications. Datasets and of domain adaptation for legal tasks, including fine-tuning
benchmarking enable the assessment of model performance pre-trained models on legal datasets or using domain-
specific knowledge graphs. This approach enables models
17 [Link] (Accessed: Dec. 28, 2024) to learn from the specific characteristics of legal language
18 [Link] (Accessed: Dec. 28, 2024) and adapt to the unique challenges of the legal domain.

VOLUME 13, 2025 18271

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

Several studies demonstrated that task-specific models, such refers to the model’s ability to generate answers or perform
as SAILER, LEGAL-BERT and LamBERTa, can outperform tasks based on examples provided within the input prompt,
general-domain pre-trained models like BERT on specific without updating its internal parameters, by leveraging the
tasks like legal case retrieval. This suggests that domain contextual information to generate appropriate and desired
adaptation is crucial, as task-specific models can learn to outputs. Crucially, there’s no need to adjust the model
capture the nuances of legal language and context. Even parameters for the new assignment. The work on pre-trained
if some tasks (e.g. entailment) are not yet fully solved big language models, which are capable of performing ICL
in the literature. LLM-based approaches have consistently without requiring retraining, has popularized ICL. It may
outperformed traditional and statistical method, due to their be expected that similar approaches will also be further
ability to comprehend contextual and semantic relations employed in the legal domain to interact in a Q&A manner
between words. with the LLMs to accomplish legal tasks without further
pre-training.
C. (RQ3) WHAT ARE THE MAIN ADVANTAGES AND
VIII. FUTURE DIRECTIONS
LIMITATIONS OF LLMS IN THE LEGAL FIELD?
Based on the findings and analysis presented in our survey
Long-Document Retrieval Presents a Unique Challenge in
paper on the application of LLMs in the legal domain,
Legal Document Retrieval. Despite progress made in legal
in this section we propose the following future directions and
document retrieval, there are still several open challenges
suggestions to advance the field.
that require further research attention. These include handling
ambiguity in legal language, dealing with irrelevant or A. ENHANCED DATASET DIVERSITY AND QUALITY
redundant information, and improving model interpretability.
Development of Multilingual Datasets. Future research
The increasing complexity of legal documents and the need
should focus on creating and curating multilingual datasets to
for efficient retrieval methods highlight the importance of
support the development of LLMs that can handle legal texts
developing novel approaches to long-document retrieval.
in various languages. This will enhance the applicability of
This may involve leveraging advancements in NLP, informa-
these models in global legal contexts. Inclusion of Diverse
tion retrieval, or machine learning techniques to overcome
Legal Subdomains. Efforts should be made to include
the limitations. Five studies specifically addressed long-
datasets from diverse legal subdomains, such as intellectual
document retrieval, emphasizing the need for efficient and
property, environmental law, and international law, to ensure
effective methods to retrieve relevant information from
that LLMs can generalize well across different areas of law.
lengthy documents. Long documents often require novel
approaches to capture the semantic relationships between B. ADVANCED MODEL ARCHITECTURES
documents and retrieve relevant information. Unfortunately,
Hybrid Models. Explore the integration of LLMs with other
it is evident from the literature [107] that performance might
AI techniques, such as rule-based systems and knowledge
suffer greatly when essential information is moved within
graphs, to create hybrid models that leverage the strengths
long context, suggesting that language models as they exist
of different approaches. Domain-Specific Fine-Tuning. Fur-
today are not able to effectively utilize information in lengthy
ther investigate advanced fine-tuning techniques tailored to
input scenarios. The results indicate that there is still room
specific legal tasks, such as contract analysis, legal research,
for improvement in terms of summary quality and accuracy,
and case outcome prediction, to improve the performance and
particularly in low-resource environments and for long legal
accuracy of LLMs in these areas.
documents. The use of domain-specific pre-trained models,
such as LEGAL-BERT, and the creation of new datasets, C. ETHICAL AND LEGAL CONSIDERATIONS
such as ITA-CaseHold and Multi-LexSum, are important
Bias and Fairness. Conduct studies to identify and mitigate
steps towards improving the performance of NLP techniques
biases in LLMs used for legal tasks. This includes developing
in summarizing legal documents. However, they do not yet
methods to ensure fairness and transparency in model
fully solve the issues related to the understanding of long
predictions and decisions. Data Privacy and Security.
documents (RQ3).
Address data privacy and security concerns associated with
the use of LLMs in legal applications. This may involve
D. (RQ4) WHAT ARE THE POSSIBLE APPLICATIONS OF developing secure data handling protocols and ensuring
LLMS TO THE LEGAL DOMAIN NOT YET FULLY EXPLORED? compliance with relevant data protection regulations.
Finally, there is a growing interest in the field of prompt
engineering for interacting with LLMs [7], [108], [109] D. INTERDISCIPLINARY COLLABORATION
(RQ4). We argue that future applications will be mainly Legal Expertise Integration. Foster collaboration between
based on human-centered techniques [110], [111]. All of AI researchers and legal experts to ensure that the devel-
these techniques are usually based on zero-shot and few- opment and deployment of LLMs in the legal domain are
shot prompting, and more generically on the concept of informed by practical legal knowledge and insights. Cross-
In-Context Learning (ICL) [112]. In the case of LLM, ICL Disciplinary Research. Encourage research that bridges the

18272 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

gap between AI and other disciplines, such as ethics, sociol- [8] K. K. Bressem, J.-M. Papaioannou, P. Grundmann, F. Borchert,
ogy, and psychology, to address the broader implications of L. C. Adams, L. Liu, F. Busch, L. Xu, J. P. Loyen, S. M. Niehues,
M. Augustin, L. Grosser, M. R. Makowski, H. J. W. L. Aerts, and A. Löser,
using LLMs in legal contexts. ‘‘[Link]: A comprehensive German BERT model for the medical
domain,’’ Expert Syst. Appl., vol. 237, Mar. 2024, Art. no. 121598.
E. STANDARDIZED BENCHMARKS AND METRICS [9] S. Wada, T. Takeda, K. Okada, S. Manabe, S. Konishi, J. Kamohara,
and Y. Matsumura, ‘‘Oversampling effect in pretraining for bidirectional
Develop and adopt standardized benchmarks and evaluation encoder representations from transformers (BERT) to localize medical
metrics specifically tailored for legal tasks. This will facilitate BERT and enhance biomedical BERT,’’ Artif. Intell. Med., vol. 153,
Jul. 2024, Art. no. 102889.
fair and consistent comparisons of different LLMs and their
[10] M. Siino and I. Tinnirello, ‘‘GPT prompt engineering for scheduling
performance in legal applications. appliances usage for energy cost optimization,’’ in Proc. IEEE Int.
Symp. Meas. Netw., Rome, Italy, Jul. 2024, pp. 1–6.
F. CONTINUOUS LEARNING AND UPDATING [11] M. Siino, F. Giuliano, and I. Tinnirello, ‘‘LLM application for knowledge
extraction from networking log files,’’ in Proc. 4th Int. Conf. Electr.,
Implement continuous learning and updating mechanisms Comput., Commun. Mechatronics Eng. (ICECCME), Nov. 2024, pp. 1–6.
for LLMs to keep them up-to-date with the latest legal [12] T. Bench-Capon et al., ‘‘A history of AI and law in 50 papers: 25 years
developments and changes in legislation. This will ensure that of the international conference on AI and law,’’ Artif. Intell. Law, vol. 20,
no. 3, pp. 215–319, Sep. 2012.
the models remain relevant and accurate over time. [13] I. Chalkidis and D. Kampas, ‘‘Deep learning in law: Early adaptation and
legal word embeddings trained on large corpora,’’ Artif. Intell. Law, vol. 27,
no. 2, pp. 171–198, Jun. 2019.
G. EDUCATION AND TRAINING
[14] M. Siino, ‘‘Mistral at SemEval-2024 task 5: Mistral 7B for argument
Provide education and training programs for legal profes- reasoning in civil procedure,’’ in Proc. 18th Int. Workshop Semantic
sionals to enhance their understanding and effective use of Eval. (SemEval), Mexico City, Mexico, A. K. Ojha, A. S. Dogruöz,
LLMs in their practice. This includes workshops, courses, H. T. Madabushi, G. Da San Martino, S. Rosenthal, and A. Rosá, Eds.,
2024, pp. 155–162.
and resources that explain the capabilities and limitations of [15] C. M. Greco, A. Tagarelli, and E. Zumpano, ‘‘A comparison of transformer-
these models. based language models on NLP benchmarks,’’ in Proc. 27th Int. Conf. Appl.
Natural Lang. Inf. Syst., vol. 13286, Valencia, Spain, P. Rosso, V. Basile,
R. Martínez, E. Métais, and F. Meziane, Eds., Cham, Switzerland: Springer,
H. REGULATORY FRAMEWORKS Jan. 2022, pp. 490–501.
Work towards the development of regulatory frameworks that [16] D. Aumiller, S. Almasian, S. Lackner, and M. Gertz, ‘‘Structural text
govern the use of LLMs in legal applications. This will help segmentation of legal documents,’’ in Proc. 18th Int. Conf. Artif. Intell.
Law, São Paulo, Brazil, J. Maranhão and A. Z. Wyner, Eds., Jun. 2021,
ensure that these models are used responsibly and ethically, pp. 2–11.
while also promoting innovation and progress in the field. [17] R. Z. Mahari, ‘‘AutoLAW: Augmented legal reasoning through legal
precedent prediction,’’ 2021, arXiv:2106.16034.
[18] J. Rabelo, R. Goebel, M.-Y. Kim, Y. Kano, M. Yoshioka, and K. Satoh,
AUTHOR CONTRIBUTIONS ‘‘Overview and discussion of the competition on legal information extrac-
Marco Siino and Mariana Falco were involved in inves- tion/entailment (COLIEE) 2021,’’ Rev. Socionetw. Strategies, vol. 16, no. 1,
tigation, conceptualization, formal analysis, methodology, pp. 111–133, Apr. 2022.
[19] N. Aletras, D. Tsarapatsanis, D. Preoţiuc-Pietro, and V. Lampos,
validation, visualization, writing original draft, review, and ‘‘Predicting judicial decisions of the European court of human rights:
editing. Daniele Croce and Paolo Rosso were involved in A natural language processing perspective,’’ PeerJ Comput. Sci., vol. 2,
methodology, supervision, writing review, and editing. All p. e93, Oct. 2016.
authors have read and agreed to the published version of the [20] H. Zhong, Z. Guo, C. Tu, C. Xiao, Z. Liu, and M. Sun, ‘‘Legal
judgment prediction via topological learning,’’ in Proc. Conf. Empirical
manuscript. Methods Natural Lang. Process., Brussels, Belgium, E. Riloff, D. Chiang,
J. Hockenmaier, and J. Tsuji, Eds., 2018, pp. 3540–3549.
REFERENCES [21] I. Chalkidis, I. Androutsopoulos, and N. Aletras, ‘‘Neural legal judgment
prediction in English,’’ in Proc. 57th Annu. Meeting Assoc. Comput.
[1] E. Mumcuolu, C. E. Öztürk, H. M. Ozaktas, and A. Ko, ‘‘Natural language Linguistics, Florence, Italy, A. Korhonen, D. R. Traum, and L. Màrquez,
processing in law: Prediction of outcomes in the higher courts of Turkey,’’ Eds., 2019, pp. 4317–4323.
Inf. Process. Manage., vol. 580, no. 5, 2021, Art. no. 0102684. [22] V. Malik, R. Sanjay, S. K. Nigam, K. Ghosh, S. K. Guha, A. Bhattacharya,
[2] D. F. Engstrom, D. E. Ho, C. M. Sharkey, and M.-F. Cuéllar, ‘‘Government and A. Modi, ‘‘ILDC for CJPE: Indian legal documents corpus for court
by algorithm: Artificial intelligence in federal administrative agencies,’’ judgment prediction and explanation,’’ in Proc. 59th Annu. Meeting Assoc.
NYU School Law, Public Law Res. Paper, vol. 12, pp. 20–54, Mar. 2020. Comput. Linguistics 11th Int. Joint Conf. Natural Lang. Process., C. Zong,
[3] C. M. Greco and A. Tagarelli, ‘‘Bringing order into the realm of F. Xia, W. Li, and R. Navigli, Eds., 2021, pp. 4046–4062.
transformer-based language models for artificial intelligence and law,’’ [23] L. Zheng, N. Guha, B. R. Anderson, P. Henderson, and D. E. Ho, ‘‘When
Artif. Intell. Law, vol. 310, no. 2, p. 148, 2023. does pretraining help?: Assessing self-supervised learning for law and the
[4] F. B. Wiener, ‘‘Decision prediction by computers: Nonsense cubed—And CaseHOLD dataset of 53,000+ legal holdings,’’ in Proc. 18th Int. Conf.
worse,’’ Amer. Bar Assoc. J., vol. 10, pp. 1023–1028, Apr. 1962. Artif. Intell. Law, São Paulo, Brazil, J. Maranhão and A. Z. Wyner, Eds.,
[5] F. R. Dickerson, ‘‘The electronic searching of law,’’ Amer. Bar Assoc. Jun. 2021, pp. 159–168.
J., vol. 470, no. 9, pp. 902–908, 1961. [24] N. Limsopatham, ‘‘Effectively leveraging BERT for legal document
[6] R. C. Lawlor, ‘‘What computers can do: Analysis and prediction of judicial classification,’’ in Proc. Natural Legal Lang. Process. Workshop, Punta
decisions,’’ Amer. Bar Assoc. J., vol. 490, no. 4, pp. 337–344, 1963. Cana, Dominican Republic, N. Aletras, I. Androutsopoulos, L. Barrett,
[7] M. Siino, ‘‘T5-medical at SemEval-2024 task 2: Using T5 medical C. Goanta, and D. Preotiuc-Pietro, Eds., 2021, pp. 210–216.
embedding for natural language inference on clinical trial data,’’ in Proc. [25] J. Lam, D. Liang, S. Dahan, and F. Zulkernine, ‘‘The gap between deep
18th Int. Workshop Semantic Eval. (SemEval), Mexico City, Mexico, learning and law: Predicting employment notice,’’ in Proc. Natural Legal
A. K. Ojha, A. S. Doğruöz, H. T. Madabushi, G. Da San Martino, Lang. Process. Workshop, vol. 2645, N. Aletras, I. Androutsopoulos,
S. Rosenthal, and A. Rosá, Eds., 2024, pp. 40–46. L. Barrett, A. Meyers, and D. Preotiuc-Pietro, Eds., 2020, pp. 52–56.

VOLUME 13, 2025 18273

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

[26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, [50] I. Chalkidis, A. Jana, D. Hartung, M. Bommarito, I. Androutsopoulos,
Ł. Kaiser, and I. Polosukhin, ‘‘Attention is all you need,’’ in Proc. Adv. D. Katz, and N. Aletras, ‘‘LexGLUE: A benchmark dataset for legal
Neural Inf. Process. Syst., vol. 30, Long Beach, CA, USA, I. Guyon, language understanding in English,’’ in Proc. 60th Annu. Meeting
U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, Assoc. Comput. Linguistics, Dublin, Ireland, S. Muresan, P. Nakov, and
S. V. N. Vishwanathan, and R. Garnett, Eds., Jun. 2017, pp. 5998–6008. A. Villavicencio, Eds., 2022, pp. 4310–4330.
[27] J. S. Melton and R. C. Bensing, ‘‘Searching legal literature electronically: [51] S. G. Graham, H. Soltani, and O. Isiaq, ‘‘Natural language processing
Results of a test program,’’ Minnesota Law Rev., vol. 450, no. 2, for legal document review: Categorising deontic modalities in contracts,’’
pp. 229–248, Jan. 1960. Artif. Intell. Law, Nov. 2023.
[28] H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, ‘‘How does [52] M. Masala, R. C. A. Iacob, A. S. Uban, M. Cidota, H. Velicu, T. Rebedea,
NLP benefit legal system: A summary of legal artificial intelligence,’’ in and M. Popescu, ‘‘JurBERT: A Romanian BERT model for legal
Proc. 58th Annu. Meeting Assoc. Comput. Linguistics, D. Jurafsky, J. Chai, judgement prediction,’’ in Proc. Natural Legal Lang. Process. Workshop,
N. Schluter, and J. R. Tetreault, Eds., 2020, pp. 5218–5230. Punta Cana, Dominican Republic, N. Aletras, I. Androutsopoulos,
[29] C. Sansone and G. Sperlí, ‘‘Legal information retrieval systems: State-of- L. Barrett, C. Goanta, and D. Preotiuc-Pietro, Eds., 2021,
the-art and open issues,’’ Inf. Syst., vol. 106, May 2022, Art. no. 101967. pp. 86–94.
[30] D. M. Katz, D. Hartung, L. Gerlach, A. Jana, and M. J. Bommarito, ‘‘Nat- [53] Y. Huang, X. Shen, C. Li, J. Ge, and B. Luo, ‘‘Dependency learning for
ural language processing in the legal domain,’’ 2023, arXiv:2302.12039. legal judgment prediction with a unified text-to-text transformer,’’ 2021,
[31] R. Dale, ‘‘Law and word order: NLP in legal tech,’’ Natural Lang. Eng., arXiv:2112.06370.
vol. 25, no. 1, pp. 211–217, Jan. 2019.
[54] M. J. Page et al., ‘‘The PRISMA 2020 statement: An updated guideline for
[32] D. Bahdanau, K. Cho, and Y. Bengio, ‘‘Neural machine translation by
reporting systematic reviews,’’ BMJ, vol. 372, p. 71, Mar. 2021.
jointly learning to align and translate,’’ in Proc. 3rd Int. Conf. Learn.
Represent., San Diego, CA, USA, Y. Bengio and Y. LeCun, Eds., Jan. 2015, [55] C. Wohlin, ‘‘Guidelines for snowballing in systematic literature studies
pp. 1–15. and a replication in software engineering,’’ in Proc. 18th Int. Conf. Eval.
[33] Z. Sun, ‘‘A short survey of viewing large language models in legal aspect,’’ Assessment Softw. Eng., London, U.K., M. J. Shepperd, T. Hall, and
2023, arXiv:2303.09136. I. Myrtveit, Eds., May 2014, pp. 1–10.
[34] D. Trautmann, A. Petrova, and F. Schilder, ‘‘Legal prompt engineering for [56] J. Devlin, M. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-training
multilingual legal judgement prediction,’’ 2022, arXiv:2212.02199. of deep bidirectional transformers for language understanding,’’ in Proc.
[35] A. Blair-Stanek, N. Holzenberger, and B. V. Durme, ‘‘Can GPT-3 perform Conf. North Amer. Chapter Assoc. Comput. Linguistics, Human Lang.
statutory reasoning?’’ Vanderbilt Law Rev., vol. 76, pp. 59–90, Jun. 2023. Technol., Minneapolis, MN, USA, J. Burstein, C. Doran, and T. Solorio,
[36] F. Yu, L. Quartey, and F. Schilder, ‘‘Legal prompting: Teaching a language Eds., Jan. 2018, pp. 4171–4186.
model to think like a lawyer,’’ 2022, arXiv:2212.01326. [57] T. Dybå and T. Dingsøyr, ‘‘Empirical studies of agile software devel-
[37] J. J. Nay, ‘‘Large language models as fiduciaries: A case study opment: A systematic review,’’ Inf. Softw. Technol., vol. 50, no. 9,
toward robustly communicating with artificial intelligence through legal pp. 833–859, Aug. 2008.
standards,’’ 2023, arXiv:2301.10095. [58] N. Jain and G. Goel, ‘‘An approach to get legal assistance using artificial
[38] K. Y. Iu and V. M.-Y. Wong, ‘‘Chatgpt by OpenAI: The end of litigation intelligence,’’ in Proc. 8th Int. Conf. Rel., INFOCOM Technol. Optim.
lawyers?’’ SSRN, 2023. (ICRITO), Jun. 2020, pp. 768–771.
[39] S. Hargreaves, ‘‘‘Words are flowing out like endless rain into a paper cup’: [59] H. Alberts, A. Ipek, R. Lucas, and P. Wozny, ‘‘COLIEE 2020: Legal
ChatGPT & Law School Assessments,’’ Chin. Univ. Hong Kong Fac. Law information retrieval and entailment with legal embeddings and boosting,’’
Res. Paper, vol. 33, p. 69, Apr. 2023. in Proc. JSAI Int. Symp. Artif. Intell. Cham, Switzerland: Springer,
[40] J. H. Choi, K. E. Hickman, A. Monahan, and D. B. Schwarcz, ‘‘ChatGPT Jan. 2021, pp. 211–225.
goes to law school,’’ J. Legal Educ., vol. 71, p. 387, May 2023. [60] P. Bafna, D. Pramod, and A. Vaidya, ‘‘Document clustering: TF-IDF
[41] T. P. Oltz, ‘‘ChatGPT, professor of law,’’ SSRN4347630, 2023. approach,’’ in Proc. Int. Conf. Electr., Electron., Optim. Techn. (ICEEOT),
[42] R. Macey-Dare, ‘‘ChatGPT & generative AI systems as quasi-expert legal Mar. 2016, pp. 61–66.
advice lawyers–case study considering potential appeal against conviction
[61] F. Sovrano, M. Palmirani, and F. Vitali, ‘‘Legal knowledge extraction for
of tom Hayes,’’ SSRN 4342686, 2023.
knowledge graph based question-answering,’’ in Proc. Legal Knowl. Inf.
[43] S. Shaghaghian, L. Y. Feng, B. Jafarpour, and N. Pogrebnyakov, ‘‘Cus- Syst., in Frontiers in Artificial Intelligence and Applications, vol. 334,
tomizing contextualized language models for legal document reviews,’’ Brno, Czech Republic, Dec. 2020, pp. 143–153.
in Proc. IEEE Int. Conf. Big Data (Big Data), Atlanta, GA, USA,
[62] Q. M. Bui, C. Nguyen, D.-T. Do, N.-K. Le, D.-H. Nguyen,
X. Wu, C. Jermaine, L. Xiong, X. Hu, O. Kotevska, S. Lu, W. Xu,
T.-T.-T. Nguyen, M.-P. Nguyen, and M. L. Nguyen, ‘‘JNLP team: Deep
S. Aluru, C. Zhai, E. Al-Masri, Z. Chen, and J. Saltz, Eds., Dec. 2020,
learning approaches for tackling long and ambiguous legal documents in
pp. 2139–2148.
COLIEE 2022,’’ in New Frontiers in Artificial Intelligence, Y. Takama,
[44] M.-Y. Kim, J. Rabelo, H. K. B. Babiker, M. A. Rahman, and R. Goebel,
K. Yada, K. Satoh, and S. Arai, Eds., Cham, Switzerland: Springer, 2023,
‘‘Legal information retrieval and entailment using transformer-based
pp. 68–83.
approaches,’’ Rev. Socionetwork Strategies, vol. 18, no. 1, pp. 101–121,
Apr. 2024. [63] D. Mamakas, P. Tsotsi, I. Androutsopoulos, and I. Chalkidis, ‘‘Pro-
[45] J. Martínez-Gil, ‘‘A survey on legal question–answering systems,’’ cessing long legal documents with pre-trained transformers: Mod-
Comput. Sci. Rev., vol. 48, Mar. 2023, Art. no. 100552. ding LegalBERT and longformer,’’ in Proc. Natural Legal Lang.
[46] M. P. Prajwal and M. Anand Kumar, ‘‘Legal text analysis using pre-trained Process. Workshop, Abu Dhabi, United Arab Emirates, N. Aletras,
transformers,’’ in Advanced Machine Intelligence and Signal Processing, I. Chalkidis, L. Barrett, C. Goanta, and D. Preotiuc-Pietro, Eds., 2022,
D. Gupta, K. Sambyo, M. Prasad, and S. Agarwal, Eds., Singapore: pp. 130–142.
Springer, 2022, pp. 493–504. [64] V. Bellandi, S. Castano, P. Ceravolo, E. Damiani, A. Ferrara, S. Montanelli,
[47] P. Bambroo and A. Awasthi, ‘‘LegalDB: Long DistilBERT for legal S. Picascia, A. Polimeno, and D. Riva, ‘‘Knowledge-based legal document
document classification,’’ in Proc. Int. Conf. Adv. Electr., Comput., retrieval: A case study on Italian civil court decisions,’’ in Proc. 23rd
Commun. Sustain. Technol. (ICAECT), Feb. 2021, pp. 1–4. Int. Conf. Knowl. Eng. Knowl. Manage., vol. 3256, Bozen-Bolzano, Italy,
[48] S. Klaus, R. Van Hecke, K. D. Naini, I. S. Altingovde, J. Bernabé-Moreno, D. Symeonidou, R. Yu, D. Ceolin, M. Poveda-Villalón, D. Audrito, L. Di
and E. Herrera-Viedma, ‘‘Summarizing legal regulatory documents using Caro, F. Grasso, R. Nai, E. Sulis, F. J. Ekaputra, O. Kutz, and N. Troquard,
transformers,’’ in Proc. 45th Int. ACM SIGIR Conf. Res. Develop. Inf. Eds., 2022, pp. 1–16.
Retr., Madrid, Spain, E. Amigó, P. Castells, J. Gonzalo, B. Carterette, [65] G. De Martino, G. Pio, and M. Ceci, ‘‘PRILJ: An efficient two-step method
J. S. Culpepper, and G. Kazai, Eds., Jul. 2022, pp. 2426–2430. based on embedding and clustering for the identification of regularities
[49] D. Núñez-Robinson, J. Talavera-Montalto, and W. Ugarte, ‘‘A comparative in legal case judgments,’’ Artif. Intell. Law, vol. 30, no. 3, pp. 359–390,
analysis on the summarization of legal texts using transformer models,’’ in Sep. 2022.
Proc. Adv. Res. Technol., Inf., Innov. Sustainability, vol. 1675, Santiago de [66] S. Althammer, A. Askari, S. Verberne, and A. Hanbury, ‘‘DoSSIER@
Compostela, Spain, T. Guarda, F. Portela, and M. F. Augusto, Eds., Cham, COLIEE 2021: Leveraging dense retrieval and summarization-based re-
Switzerland: Springer, Jan. 2022, pp. 372–386. ranking for case law retrieval,’’ 2021, arXiv:2108.03937.

18274 VOLUME 13, 2025

M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

[67] Y. Shao, J. Mao, Y. Liu, W. Ma, K. Satoh, M. Zhang, and S. Ma, ‘‘BERT- [84] Y. Chen, Y. Zhang, J. Wang, and X. Zhang, ‘‘YNU-HPCC at SemEval-
PLI: Modeling paragraph-level interactions for legal case retrieval,’’ in 2023 task 6: LEGAL-BERT based hierarchical BiLSTM with CRF for
Proc. 29th Int. Joint Conf. Artif. Intell., C. Bessiere, Ed., Jul. 2020, rhetorical roles prediction,’’ in Proc. The 17th Int. Workshop Semantic
pp. 3501–3507. Eval. (SemEval), Toronto, ON, Canada, A. K. Ojha, A. S. Doğruöz,
[68] I. Almuslim and D. Inkpen, ‘‘Document level embeddings for identifying G. Da San Martino, H. T. Madabushi, R. Kumar, and E. Sartori, Eds., 2023,
similar legal cases and laws,’’ in Proc. Work. Notes-Forum Inf. Retr. Eval. pp. 2075–2081.
(FIRE), Hyderabad, India, P. Mehta, T. Mandl, P. Majumder, and M. Mitra, [85] T. W. T. Au, V. Lampos, and I. Cox, ‘‘E-NER—An annotated named entity
Eds., Jan. 2020, pp. 42–48. recognition corpus of legal text,’’ in Proc. Natural Legal Lang. Process.
[69] A. Tagarelli and A. Simeri, ‘‘Unsupervised law article mining based on Workshop, Abu Dhabi, United Arab Emirates, N. Aletras, I. Chalkidis, L.
deep pre-trained language representation models with application to the Barrett, C. Goanta, and D. Preotiuc-Pietro, Eds., 2022, pp. 246–255.
Italian civil code,’’ Artif. Intell. Law, vol. 30, no. 3, pp. 417–473, Sep. 2022. [86] S. Auriemma, M. Miliani, A. Bondielli, L. C. Passaro, and A. Lenci,
[70] A. Simeri and A. Tagarelli, ‘‘Exploring domain and task adaptation of ‘‘Evaluating pre-trained transformers on Italian administrative texts,’’ in
Lamberta models for article retrieval on the Italian civil code,’’ in Proc. Proc. of 1st Workshop AI Public Admin., vol. 3285, Udine, Italy, P. Lops,
19th Conf. Inf. Res. Sci. Connecting Digit. Library Sci., vol. 3365, Bari, P. Basile, L. Siciliani, V. Taccardi, M. Di Ciano, and N. Lopane, Eds., 2022,
Italy, A. Falcon, S. Ferilli, A. Bardi, S. Marchesin, and D. Redavid, Eds., pp. 54–70.
2023, pp. 130–143. [87] P. Bhattacharya, K. Ghosh, A. Pal, and S. Ghosh, ‘‘Legal case document
[71] H. Li, Q. Ai, J. Chen, Q. Dong, Y. Wu, Y. Liu, C. Chen, and Q. Tian, similarity: You need both network and text,’’ Inf. Process. Manage., vol. 59,
‘‘SAILER: Structure-aware pre-trained language model for legal case no. 6, Nov. 2022, Art. no. 103069.
retrieval,’’ in Proc. 46th Int. ACM SIGIR Conf. Res. Develop. Inf. Retr., [88] R. Silveira, C. G. Fernandes, J. A. M. Neto, V. Furtado, and J. E. P. Filho,
Taipei, Taiwan, H.-H. Chen, W.-J. E. Duh, H.-H. Huang, M. P. Kato, ‘‘Topic modelling of legal documents via LEGAL-BERT,’’ in Proc. 1st
J. Mothe, and B. Poblete, Eds., Jul. 2023, pp. 1035–1044. Int. Workshop Rel.-Relations Legal Domain, So Paulo, Brazil, Jan. 2023,
[72] M. Ostendorff, E. Ash, T. Ruas, B. Gipp, J. Moreno-Schneider, and pp. 1–18.
G. Rehm, ‘‘Evaluating document representations for content-based legal [89] W. Hwang, D. Lee, K. Cho, H. Lee, and M. Seo, ‘‘A multi-task benchmark
literature recommendations,’’ in Proc. 18th Int. Conf. Artif. Intell. Law, São for Korean legal language understanding and judgement prediction,’’ in
Paulo, Brazil, J. Maranhão and A. Z. Wyne, Eds., Jun. 2021, pp. 109–118. Proc. Adv. Neural Inf. Process. Syst., New Orleans, LA, USA, S. Koyejo,
[73] M. Siino, F. Lomonaco, and P. Rosso, ‘‘Backtranslate what you are saying S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., Jan. 2022,
and I will tell who you are,’’ Expert Syst., vol. 41, no. 8, Aug. 2024, pp. 1–15.
Art. no. e13568. [90] J. Niklaus, V. Matoshi, P. Rani, A. Galassi, M. Stürmer, and I. Chalkidis,
[74] F. Lomonaco, M. Siino, and M. Tesconi, ‘‘Text enrichment with Japanese ‘‘LEXTREME: A multi-lingual and multi-task benchmark for the
language to profile cryptocurrency influencers,’’ in Proc. Work. Notes legal domain,’’ in Proc. Findings Assoc. Comput. Linguistics, EMNLP,
Conf. Labs Eval. Forum (CLEF), vol. 3497, Thessaloniki, Greece, Singapore, H. Bouamor, J. Pino, and K. Bali, Eds., 2023, pp. 3016–3054.
M. Aliannejadi, G. Faggioli, N. Ferro, and M. Vlachos, Eds., 2023, [91] S. Bano and S. Khalid, ‘‘BERT-based extractive text summarization of
pp. 2708–2716. scholarly articles: A novel architecture,’’ in Proc. Int. Conf. Artif. Intell.
[75] S. Wang, A. Scardigli, L. Tang, W. Chen, D. Levkin, A. Chen, S. Ball, Things (ICAIoT), Dec. 2022, pp. 1–5.
T. Woodside, O. Zhang, and D. Hendrycks, ‘‘MAUD: An expert-annotated [92] S. Bano, S. Khalid, N. M. Tairan, H. Shah, and H. A. Khattak,
legal NLP dataset for merger agreement understanding,’’ in Proc. Conf. ‘‘Summarization of scholarly articles using BERT and BiGRU: Deep
Empirical Methods Natural Lang. Process., Singapore, H. Bouamor, J. learning-based extractive approach,’’ J. King Saud Univ.-Comput. Inf. Sci.,
Pino, and K. Bali, Eds., 2023, pp. 16369–16382. vol. 35, no. 9, Oct. 2023, Art. no. 101739.
[76] S. Khazaeli, J. Punuru, C. Morris, S. Sharma, B. Staub, M. Cole, [93] S. Vatsal, A. Meyers, and J. E. Ortega, ‘‘Classification of U.S. supreme
S. Chiu-Webster, and D. Sakalley, ‘‘A free format legal question answering court cases using BERT-based techniques,’’ in Proc. Conf. Recent Adv.
system,’’ in Proc. Natural Legal Lang. Process. Workshop, Punta Cana, Natural Lang. Process.-Large Lang. Models Natural Lang. Process.,
Dominican Republic, N. Aletras, I. Androutsopoulos, L. Barrett, C. Varna, Bulgaria, R. Mitkov and G. Angelova, Eds., 2023, pp. 1207–1215.
Goanta, and D. Preotiuc-Pietro, Eds., pp. 107–113. [94] Z. Shen, K. Lo, L. Yu, N. Dahlberg, M. Schlanger, and D. Downey,
[77] M. Miliani, L. C. Passaro, and A. Lenci, ‘‘FRAQUE: A frame-based ‘‘Multi-LexSum: Real-world summaries of civil rights lawsuits at multiple
question-answering system for the public administration domain,’’ in granularities,’’ in Proc. Adv. Neural Inf. Process. Syst., Jan. 2022,
Proc. 1st Workshop Lang. Technol. Government Public Admin., Marseille, pp. 13158–13173.
France, D. Samy, D. Pérez-Fernández, and J. Arenas-García, Eds., [95] A. Agarwal, S. Xu, and M. Grabmair, ‘‘Extractive summarization of legal
May 2020, pp. 7–14. decisions using multi-task learning and maximal marginal relevance,’’ in
[78] E. Leitner, G. Rehm, and J. M. Schneider, ‘‘A dataset of German legal Proc. Findings Assoc. Comput. Linguistics, EMNLP, Abu Dhabi, United
documents for named entity recognition,’’ in Proc. 12th Lang. Resour. Eval. Arab Emirates, Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds., 2022,
Conf., Marseille, France, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. pp. 1857–1872.
Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, [96] B. Clavié and M. Alphonsus, ‘‘The unreasonable effectiveness of
A. Moreno, J. Odijk, and S. Piperidis, Eds., Jan. 2020, pp. 4478–4485. the baseline: Discussing SVMs in legal text classification,’’ in Proc.
[79] D. Hendrycks, C. Burns, A. Chen, and S. Ball, ‘‘CUAD: An expert- Legal Knowl. Inf. Syst.-34th Annu. Conf., in Frontiers in Artificial
annotated NLP dataset for legal contract review,’’ in Proc. Neural Inf. Intelligence and Applications, vol. 346, Vilnius, Lithuania, Dec. 2021,
Process. Syst., J. Vanschoren and S.-K. Yeung, Eds., Jan. 2021, pp. 1–13. pp. 58–61.
[80] V. Pais, M. Mitrofan, C. L. Gasan, V. Coneschi, and A. Ianov, ‘‘Named [97] P. Henderson, M. Krass, L. Zheng, N. Guha, C. D. Manning, D. Jurafsky,
entity recognition in the Romanian legal domain,’’ in Proc. Natural Legal and D. E. Ho, ‘‘Pile of law: Learning responsible data filtering from the law
Lang. Process. Workshop, Punta Cana, Dominican Republic, N. Aletras, and a 256GB open-source legal dataset,’’ in Proc. Adv. Neural Inf. Process.
I. Androutsopoulos, L. Barrett, C. Goanta, and D. Preotiuc-Pietro, Eds., Syst., New Orleans, LA, USA, S. Koyejo, S. Mohamed, A. Agarwal, D.
2021, pp. 9–18. Belgrave, K. Cho, and A. Oh, Eds., Jan. 2022, pp. 1–17.
[81] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and [98] I. Chalkidis, M. Fergadiotis, and I. Androutsopoulos, ‘‘MultiEURLEX–
I. Androutsopoulos, ‘‘LEGAL-BERT: The Muppets straight out of A multi-lingual and multi-label legal document classification dataset
law school,’’ 2020, arXiv:2010.02559. for zero-shot cross-lingual transfer,’’ in Proc. Conf. Empirical Methods
[82] S. Auriemma, M. Madeddu, M. Miliani, A. Bondielli, L. C. Passaro, and Natural Lang. Process., Punta Cana, Dominican Republic, M.-F. Moens,
A. Lenci, ‘‘BureauBERTo: Adapting UmBERTo to the Italian bureaucratic X. Huang, L. Specia, and S. W.-T. Yih, Eds., 2021, pp. 6974–6996.
language,’’ in Proc. Italia Intelligenza Artificiale, vol. 3486, Pisa, Italy, [99] Y. Koreeda and C. Manning, ‘‘ContractNLI: A dataset for document-
F. Falchi, F. Giannotti, A. Monreale, C. Boldrini, S. Rinzivillo, and S. level natural language inference for contracts,’’ in Proc. Findings Assoc.
Colantonio, Eds., 2023, pp. 240–248. Comput. Linguistics, EMNLP, Punta Cana, Dominican Republic, M.-F.
[83] D. Licari and G. Comandè, ‘‘ITALIAN-LEGAL-BERT models for Moens, X. Huang, L. Specia, and S. W.-T. Yih, Eds., 2021, pp. 1907–1919.
improving natural language processing tasks in the Italian legal domain,’’ [100] S. Leivaditi, J. Rossi, and E. Kanoulas, ‘‘A benchmark for lease contract
Comput. Law Secur. Rev., vol. 52, Apr. 2024, Art. no. 105908. review,’’ 2020, arXiv:2010.10386.

VOLUME 13, 2025 18275

 M. Siino et al.: Exploring LLMs Applications in Law: A Literature Review

[101] D. Tuggener, P. von Däniken, T. Peetz, and M. Cieliebak, ‘‘LEDGAR: MARIANA FALCO received the Ph.D. degree in
A large-scale multi-label corpus for text classification of legal provisions engineering from Universidad Austral, in 2022,
in contracts,’’ in Proc. 12th Lang. Resour. Eval. Conf., Marseille, France, N. with a doctoral fellowship. She has been actively
Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, involved in research projects since 2013. She has
H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. over nine years of experience as a Lecturer in
Piperidis, Eds., pp. 1235–1241. software engineering and project management.
[102] J. Valvoda, R. Cotterell, and S. Teufel, ‘‘On the role of negative precedent She has also managed software development
in legal outcome prediction,’’ in Proc. Trans. Assoc. Comput. Linguistics, projects as a Project Manager, specializing in agile
vol. 11, Jan. 2023, pp. 34–48. methodologies for more than six years. In the past
[103] C. Steging, S. Renooij, and B. Verheij, ‘‘Taking the law more seriously years, she has conducted audits on processes and
by investigating design choices in machine learning prediction research,’’ the application of agile methodologies at scale in large companies. She has
in Proc. 6th Workshop Automated Semantic Anal. Inf. Legal Text,
served as a reviewer for academic journals for the past three years, regarding
vol. 3441, Braga, Portugal, F. Lagioia, J. Mumford, D. Odekerken, and
software engineering and software quality.
H. Westermann, Eds., 2023, pp. 49–59.
[104] D. Licari and G. Comandè, ‘‘ITALIAN-LEGAL-BERT: A pre-trained
transformer language model for Italian law,’’ in Proc. 23rd Int.
Conf. Knowl. Eng. Knowl. Manage., vol. 3256, Bozen-Bolzano, Italy,
D. Symeonidou, R. Yu, D. Ceolin, M. Poveda-Villalón, D. Audrito, L. Di
Caro, F. Grasso, R. Nai, E. Sulis, F. J. Ekaputra, O. Kutz, and N. Troquard, DANIELE CROCE received the double [Link].
Eds., 2022, pp. 1–16. degree in networking engineering from the
[105] Y. Hu, M. Hosseini, E. S. Parolin, J. Osorio, L. Khan, P. Brandt, and Politecnico di Torino and EURECOM Institute,
V. D’Orazio, ‘‘ConfliBERT: A pre-trained language model for political Sophia Antipolis, France, in 2006, the Research
conflict and violence,’’ in Proc. Conf. North Amer. Chapter Assoc. Comput. Master Diploma degree (ex DEA) in networking
Linguistics, Hum. Lang. Technol., 2022, pp. 5469–5482. and distributed systems from the Université de
[106] S. Khalid, S. Wu, A. Alam, and I. Ullah, ‘‘Real-time feedback query Nice-Sophia Antipolis (UNSA), Nice, France,
expansion technique for supporting scholarly search using citation network in 2006, and the joint Ph.D. degree from the
analysis,’’ J. Inf. Sci., vol. 47, no. 1, pp. 3–15, Feb. 2021. Politecnico di Torino, Turin, Italy, and UNSA,
[107] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and in 2010. He is currently an Assistant Professor
P. Liang, ‘‘Lost in the middle: How language models use long contexts,’’ in with the University of Palermo, Palermo, Italy. He has long experience of
Proc. Trans. Assoc. Comput. Linguistics, vol. 12, Jan. 2024, pp. 157–173. research collaborations, in several European and national research projects,
[108] M. Siino, ‘‘BadRock at SemEval-2024 task 8: DistilBERT to detect on wireless networks, the Internet of Things, high-quality TV streaming,
multigenerator, multidomain and multilingual black-box machine- smart grid communications, and smart cities. He also worked on assistive
generated text,’’ in Proc. 18th Int. Workshop Semantic Eval. (SemEval), technologies for visually impaired people and with the Arianna Project.
Mexico City, Mexico, A. K. Ojha, A. S. Doğruöz, H. T. Madabushi, He was the co-founder of the start-up company [Link] s.r.l., spin-off of
G. Da San Martino, S. Rosenthal, and A. Rosá, Eds., 2024, pp. 239–245. Palermo University.
[109] M. Siino, ‘‘McRock at SemEval-2024 task 4: Mistral 7B for multilingual
detection of persuasion techniques in memes,’’ in Proc. 18th Int. Workshop
Semantic Eval. (SemEval), Mexico City, Mexico, A. K. Ojha, A. S.
Doğruöz, H. T. Madabushi, G. Da San Martino, S. Rosenthal, and A. Rosá,
Eds., 2024, pp. 53–59.
PAOLO ROSSO is a Full Professor of computer
[110] Y. Shu, H. Gu, P. Zhang, H. Zhang, T. Lu, D. Li, and N. Gu, ‘‘RAH!
science with the Universitat Politècnica de Valèn-
RecSys–assistant–human: A human-centered recommendation framework
cia (UPV), Spain. He is a member of the Pattern
with LLM agents,’’ IEEE Trans. Computat. Social Syst., vol. 110, no. 5,
pp. 6759–770, Jan. 2023.
Recognition and Human Language Technology
(PRHLT) Research Center and the Valencian
[111] D. Chin, Y. Wang, and G. Xia, ‘‘Human-centered LLM-agent user
interface: A position paper,’’ 2024, arXiv:2405.13050.
Graduate School and Research Network of Artifi-
cial Intelligence (ValgrAI). He has published more
[112] M. Luo, X. Xu, Y. Liu, P. Pasupat, and M. Kazemi, ‘‘In-context learning
with retrieved demonstrations for language models: A survey,’’ 2024, than 400 peer-reviewed papers in conferences and
arXiv:2401.11624. journals. He is among the best computer science
scientists in Spain ([Link]
rankings/computer-science/es). He is the PI of several related research
MARCO SIINO received the bachelor’s and projects, such as FairTransNLP-Stereotypes: Fairness and Transparency for
master’s degrees (cum laude) in computer engi- equitable NLP applications in social media–Identifying stereotypes and
neering from the University of Palermo, and the prejudices and developing equitable systems (Grant PID2021-124361OB-
Ph.D. degree in information and communication C31), FAKEnHATE-PdC: FAKE news and HATE speech (Grant PDC2022-
technologies from the University of Palermo, 133118-I00), and XAI-DisInfodemics: eXplainable AI for disinformation
in 2023. He is a freelance full stack Developer. and conspiracy detection during infodemics (Grant PLEC2021-007681),
He is currently an Assistant Professor with the funded by MCIN/AEI/10.13039/501100011033 and by European Union
University of Catania, where he teaches courses NextGenerationEU/PRTRPI. His current research interests include detection
on network intelligence and information theory. of harmful information in social media, both fake news and hate speech.
His main interests include machine learning, deep In 2022, he received the UPV Research Award for Excellent Publication in
learning, natural language processing, and recommender systems. His work Engineering and Technology on misogyny identification.
involves social-networks-related tasks (e.g., sentiment analysis, hate speech
detection, and fake news detection). He is an IEEE IMS Member.

18276 VOLUME 13, 2025