Using Generative AI: issues and considerations
Adrian Byrne, Donghyeok Lee, and Quan Le, CeADAR, University College Dublin
Keywords: Generative AI, Large Language Models, education, responsible design, acceptable
use, synthetic data
The emerging Generative AI landscape
Artificial Intelligence (AI) has been a hot technology topic for the past decade, but generative
AI, and specifically the arrival of ChatGPT in 2022, has thrust AI into the spotlight globally
and has launched an unprecedented surge of AI innovation and adoption. Generative AI,
sometimes referred to GenAI, is AI that can create new content such as text, images, video,
audio, or software code in response to a user’s prompt/request. Generative AI systems are built
upon a limited number of foundation models, i.e., very large deep learning models trained on
vast amounts of training datasets to perform multiple tasks (IBM, What is generative AI?).
Generative AI is being applied widely across different industries, including education,
entertainment, healthcare, scientific research, and cybersecurity. This paper outlines some of
the key issues and considerations relating to generative AI.
Generative AI systems have been developed across multiple modalities, with the earliest and
most notable progress achieved in text generation. Well-known GenAI models for text include
GPT-4 and LLAMA 3. Capable generative AI systems for images have emerged, including
DALLE-3 and Stable Diffusion. The latest generative AI systems, GPT-4 or 4M-21i, are trained
on multimodal input, i.e., text and image, and can generate output in different modalities.
According to the Artificial Intelligence Index Report 2024, AI continues to make advancements
that see it outperform humans on certain tasks such as image classification as well as help
certain workers be more productive and accelerate scientific research. This AI charge is being
led by industry rather than academia and the money to invest in these ever more expensive
machine learning models is flowing into the former much more so than the latter.
According to AI Index estimates, the training costs of state-of-the-art AI models have reached
unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of
computing power to train, while Google’s Gemini Ultra cost $191 million for computing power.
These costs are prohibitively expensive meaning very few entities can operate at that level of
expense leading to a concentration at the top of the AI provider pyramid. On a regional level,
the United States is the leading source of top AI models. In 2023, 61 notable AI models
originated from U.S.-based institutions, far outpacing the European Union’s 21 and China’s 15.
1
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
Generative AI is gaining popularity as a means of creating autonomous AI agentsii,iii that can
process multiple input sources, interact with tools or other generative AI systems, and make
automated decisions. These agents rely on generative AI systems to serve as their brains to
coordinate tasks and make inferences. Adopting generative AI in AI agents increases its impact
enormously while simultaneously amplifying its serious vulnerabilities. One major issue is that
current generative AI systems are not built to be secure-by-design, making them vulnerable to
exploitation by malicious actors who can easily circumvent their safeguards (e.g. by prompt
injection attacks).
Complementing this AI charge are the increasing number of regulations being published to
address citizens’ growing concerns over the role of AI in society. However, rigorous,
standardised evaluations of these ever-prevalent AI models are seriously lacking. The leading
developers are free to conduct responsible AI benchmarking as they see fit which complicates
efforts to systematically compare the risks and limitations of top AI models. According to the
UN High Commissioner for Human Rights, AI has the potential to benefit humanity, but
regulation is essential to ensure these benefits outweigh the risks. Regulations should be based
on human rights principles, requiring transparency, independent oversight, and effective
remedies (Artificial intelligence must be grounded in human rights, says High Commissioner)
Impact on education
Generative AI has huge potential to affect our daily lives and society in general, including the
education sector. Generative AI can make teachers more effective when preparing lessons and
students can receive more personalised learning experiences. More administrative tasks can be
automated, which enables teachers to spend more time interacting with students. Moreover,
generative AI can make individualised learning possible by adapting to the student’s style and
pace of study. Instant feedback increases learning efficiency, and accessibility is enhanced
through translation or in other formats for learners with very different learning abilities. These
advantages provide a quality improvement in the delivery of education.
According to Shaping the Future of Learning: The Role of AI in Education 4.0 from the World
Economic Forum, the West African ed-tech start-up, Kabakoo Academies, utilises a Large
Language Model (LLM) for the creation of GenAI-powered virtual mentors to guide young
people in developing the mindset and skills needed for self-employment. The above
programme yields several positive effects:
1. Personalised Learning Support: LLM-based AI mentor is available 24/7 for advice,
access to information, and customised feedback for better learner support.
2. Accessibility of Language: Development of an AI model for education in Bambara,
the most widely spoken language in Mali, improves educational accessibility for those
with diverse linguistic backgrounds.
3. Tangible Outcomes: This programme led to a 23% shift in learners' growth mindset
and incomes increased by 44% six months after completion.
2
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
At the same time, generative AI poses several challenges. One major concern is the reliability
of the content generated by AI, which can sometimes be inaccurate or misleading, leading to
incorrect decisions/learnings. Another significant issue is data privacy, as generative AI often
requires and ingests large amounts of personal data, raising concerns about the security and
privacy of student information and the serious consequences of improper handling or data
breaches.
Furthermore, unequal access to AI technology can exacerbate educational disparities. Students
and institutions with limited resources may not be able to fully utilise the advanced features of
generative AI, thereby widening the gap between those who can afford such technologies and
those who cannot, leading to significant inequalities in the quality of education received by
students from different socioeconomic backgrounds (sciencedirect, Unveiling the shadows:
Beyond the hype of AI in education).
In March 2023, more than 1,000 industry experts signed an open letter urging industry leaders
to pause GenAI development so that the capabilities and dangers of systems such as GPT-4 can
be assessed and mitigated if needs be. A New York attorney, who used ChatGPT to write a legal
brief, had to provide an apology in court, as six of the cases cited in their submitted brief did
not exist (Bridging the Gap Between Artificial Intelligence Implementation, Governance and
Democracy: An Operational and Regulatory Perspective).
These disruptive examples underscore the need for careful implementation, regulation and
guidance with respect to generative AI in the education sector to enhance the learning
experience while minimising the potential risks. However, a recent ISACA (Information
Systems Audit and Control Association) study on generative AI found that only 10% of
organisations have formal, comprehensive policies in place for generative AI (Key
Considerations for Developing Organizational Generative AI Policies).
Particularly in education, a field where research integrity is paramount, there must be precise
regulation that guides generative AI and particularly with respect to the three cardinal
principles: plagiarism, fabrication, and falsification. GenAI can replicate existing sources so
accurately that sometimes it is difficult to distinguish from the original source (plagiarism). It
can also generate fabricated data that can result in threats to research credibility (fabrication
and falsification). Therefore, it is important for educational institutions to address these issues
to maintain and improve high levels of research integrity.
Additionally, the increased uptake of AI will significantly affect the academic and career plans
of students. In a new Inside Higher Ed-College Pulse's Student Voice series survey, with more
than 1,250 students responding from 49 four-year and two-year colleges, 14% said AI has
influenced their academic plans a lot, and another 34% said somewhat. Also, regarding career
plans, 11% said that AI influenced their plans a lot and 31% were somewhat influenced. More
than two-thirds of students, 72%, believe that their colleges should be preparing them for
working with AI. As for how institutions should be preparing them for AI and work, the top
priority, according to 72% of surveyed students, is being taught the ethics of using AI.
3
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
Therefore, colleges should reflect this request by updating curricula, offering interdisciplinary
programmes, and establishing continuous learning pathways (Survey: How AI Is Impacting
Students’ Career Choices).
According to OECD DIGITAL EDUCATION OUTLOOK 2023 Towards an Effective Digital
Education Ecosystem, the status of regulation, guidance, and school responsibility, as of
December 2023, is displayed in Table 1 below.
School
Regulation Guidance (non-binding)
responsibility
Drafted and
Passed Pending approval Published
pending approval
Austria ✓ ✓
Canada ✓
Czechia ✓ ✓ ✓
Finland ✓ ✓
France ✓ ✓
Hungary
Iceland ✓
Japan ✓
Korea ✓ ✓ ✓
Latvia ✓
Luxembourg ✓ ✓
New Zealand ✓ ✓
Slovakia ✓
Spain
Sweden ✓
Türkiye
England
(United ✓ ✓ ✓
Kingdom)
Flemish comm.
✓ ✓
(Belgium)
Total (18) 0 2 9 7 9
Table 1: Use of generative AI in education: guidance and practices, 2024
4
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
We can delineate AI’s impact on education on both a global and local level. Starting with the
global level, it is largely recognised that generative AI is one of the transformative tools that
will be useful in education. This is reflected in the set frameworks and guidelines drafted by
organisations at a global level on its ethical and effective use:
1) Guidance for generative AI in education and research (UNESCO) - In September 2023,
UNESCO proposed guidelines concerning the ethical use of generative AI in education
and research, which includes protection against violation of data privacy, age restriction
setting, and mitigation policy frameworks. It also advises that teachers and students
must be educated on how to use generative AI tools safely. It urges governments to
quickly regulate the use of generative AI in schools so that it is integrated effectively
into the education system.
2) OECD DIGITAL EDUCATION OUTLOOK 2023 Towards an Effective Digital
Education Ecosystem - In December 2023, the OECD explained how generative AI
could be used within the digital education ecosystem. OECD Digital Education Outlook
2023 discussed how generative AI contributes as a push factor to innovation in
education and sets out recommendations on how policymakers can grasp its benefits.
However, to capture these benefits brought about by generative AI in an educational
setting, comprehensive strategies are needed.
At the local level, various countries mentioned in Table1 have developed specific frameworks
and guidelines to regulate and promote the use of generative AI in education.
1) Australian Framework for Generative Artificial Intelligence (AI) in Schools - In
November 2023, the Australian Framework for Generative AI in Schools highlighted
the responsible and ethical use of generative AI tools for students, schools, and society.
It provided guidelines for policymakers, school leaders, teachers, support staff, parents,
and students on why they need to understand, use, and respond to generative AI in
education. The framework aimed at improving educational outcomes with appropriate
utilisation of AI tools by introducing safe and ethical practices, advancing equity and
inclusion with 6 principles (see Figure 1) and 25 guiding statements.
5
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
Figure 1: Visualisation of Australian Framework for Generative AI in Schools
2) Generative artificial intelligence (AI) in education - In October 2023, the UK's
Department for Education (DfE) published a policy paper guided by the government’s
white paper on a pro-innovation approach to AI regulation. The paper stated that
generative AI allows for a potential decrease in workload thereby allowing more time
for teachers. At the same time, challenges arise from the content generated by
generative AI. Since AI requires training based on input data, the students' work is not
to be used in the training of the model without their consent. Also, there must not be
any malpractice with respect to these generative AI tools in any formal assessment.
Students need to be educated on how to use technology appropriately and recognise its
potential social impact. Furthermore, the DfE said it will provide support to primary
and secondary schools in the delivery of a knowledge-rich computing curriculum.
These global/local guidelines and regulations are crucial for ensuring that generative AI is used
safely and effectively in education. They aim to support teachers and students in leveraging
innovative tools while maintaining ethical standards and protecting personal data.
The Irish context
Turning to the Irish context in relation to AI, recent studies conducted on behalf of the
government suggest that around two-thirds of employment in Ireland will be affected by AI
with up to one half of those jobs being at risk and the other half being enhanced. The Irish
government welcomes these studies as it is calling for a national conversation on AI and the
changes it will bring. Another study shows that multinational companies claim to use 30% more
generative AI than indigenous, Irish organisations. The study surveyed 400 senior managers
across both the public and private sectors in Ireland. The findings revealed an emergence of a
6
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
“shadow” AI workplace culture, where staff are using publicly available AI tools regardless of
any organisational acceptable usage policy. Such policies were only in operation in 2% of the
firms surveyed despite 49% reporting some form of generative AI use already.
Call for responsible design
The introduction of generative AI has intensified concerns around misinformation and biased
training data. It also presents new challenges in relation to authenticity, transparency, and
intellectual property/data privacy. Getting to grips with these concerns and challenges is not
only important and urgent but may also affect providers’ bottom line. A survey by The
Economist Intelligence Unit found that 75% of executives oppose working with AI service
providers whose products lack responsible design. Therefore, “FMOps” (Foundation Model
Operations) / “LLMOps” (Large Language Model Operations) is an emerging area of research
which includes bringing generative AI applications into production and monitoring them
afterwards. Tasks include evaluating data and models, taking corrective actions, conducting
audits and risk assessments, and establishing processes for continued model improvement.
There will also be a need to expand the data infrastructure to govern and manage new, diverse,
unstructured datasets to ensure alignment with privacy, security, and quality control as well as
minimise the risk of sharing inaccurate or incorrect information that can lead to harmful or
libelous outcomes. Such endeavours are critical if we want to ensure that the benefits of
generative AI outweigh the risks.
Acceptable usage policy
Another dimension of generative AI regarding its use within a professional setting is
establishing an acceptable usage policy for any organisation wanting to harness its capabilities
in a responsible manner. Any such policy needs to address the concerns of different
organisational stakeholders such as customers, regulators, and staff starting with why, i.e.
motivate the business case for its use. It must also outline how any AI system adopted will be
used safely, securely, and responsibly as well as how it will not be used. The policy should
indicate how it chimes with existing regulation, e.g. data handling in conjunction with the
General Data Protection Regulation within the EU, as well as existing policies within the
organisation. Training for appropriate use and establishing a violation reporting mechanism is
another important element of such a policy as is reinforcing the “human-in-the-loop” oversight
aspect by identifying who will manage, audit, review, and update the policy and when.
Moreover, it is important to provide a human point of contact in relation to this policy to address
the concerns of any/all organisational stakeholders.
As developing generative AI solutions in-house can be extremely costly, most businesses and
organisations are opting to adopt third-party generative AI tools or fine-tuning such models
using their own data. This economic decision means that any business/organisation choosing
to use a third-party generative AI tool must review the terms and conditions of use in advance
and satisfy all business/organisational requirements through dialogue with any potential
provider before greenlighting its use in a professional capacity. For example, ask a potential
7
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]
generative AI solution provider about any appropriate protections and assurances in place to
address confidential information generated by your prompts. Also, ask any potential provider
if they are willing to offer indemnities against IP/copyright/open-source (if commercially
constraint) infringements and if these indemnities are subject to any contractual agreements.
Furthermore, double-check the terms and conditions of these generative AI tools to understand
who owns the IP (if any) in relation to the outputs generated.
Synthetic data
A final area of generative AI worth exploring in this paper is its use in relation to synthetic data
generation. Its capability in this regard is threatening to revolutionise our approach to data
creation and the training of AI models akin to how AI mastered the boardgames Chess and Go
in recent decades. A key driver for synthetic data generation is due to a specific lack of readily
available data to help us get better at specific tasks. If viable and done appropriately, synthetic
data generation can help researchers overcome the biases and ethical dilemmas often inherent
in real-world datasets as many domains suffer from a lack of quality data, especially when it
comes to rare events or minority classes. LLMs can augment existing datasets, creating
balanced and comprehensive datasets that improve the training and performance of machine
learning models. That said, we should not assume the data generated is automatically accurate
and representative given LLMs may be trained on biased data that is also lacking in both intra-
class and inter-class diversity. These LLMs may also produce completely fictitious data that is
totally disconnected from the desired model of representation/reality, i.e. the LLMs may
“hallucinate”. Therefore, there is a pressing need to develop effective strategies to detect and
address hallucinations in the context of synthetic data generation, ensuring the reliability and
authenticity of the output. At the other end of the spectrum, it has been demonstrated that it is
possible to extract specific information from the datasets used in training LLMs. Consequently,
there exists a risk that synthetic data generation might inadvertently reveal elements of the
underlying training data, which may raise some data privacy and ethical concerns thereby
highlighting the need for careful consideration in the use and dissemination of synthetic data
generated by LLMs.
Conclusion
Addressing the concerns associated with generative AI requires not just technological
solutions, but also robust policy frameworks, ethical guidelines, and harmonised standards to
ensure responsible use of data (synthetic or otherwise) and AI models.
i
[Link]
ii
[Link]
iii
[Link]
8
__________________________________________________________________________
UCD Geary Institute for Public Policy, University College Dublin, Belfield, Dublin 4, Ireland. Phone +353 1 716 4615
Website: [Link] Email: geary@[Link]