Project Objective
ur project’s goal is to create challenging prompts and corresponding evaluation rubrics to
O
effectively test the limits of AI models. Specifically, we aim to probe their capacity for nuanced
reasoning, consistent performance, and comprehensive understanding. This document outlines
the approach for constructing sophisticated tasks and rubrics to ensure a fair and objective
evaluation of AI responses.
Key Components
1. Prompt Creation
urpose: The prompts you create should encourage advancedreasoning and synthesis of
P
information. Each one should be designed to reveal potential weaknesses in the model, such as
its handling of ambiguity or complex scenarios.
● Structure of Prompts:
○ Level of Difficulty: Prompts should be detailed enoughto challenge even
subject matter experts, ensuring that they go beyond basic or easily searchable
information.
○ Clear and Precise Wording: Provide enough contextfor clarity without
overwhelming the reader. The prompt’s goal should be easily identifiable while
allowing flexibility in how the AI might approach it.
○ Focus on Reasoning: Instead of asking straightforwardfactual questions,
structure prompts to elicit analytical or comparative answers. For example,
instead of asking for facts, encourage the model to analyze, evaluate, or draw
conclusions based on general knowledge.
○ Expected Response Format: Specify the format you expect(such as a
structured report, bullet points, or essay format). This ensures consistency in
responses and simplifies the evaluation process.
● Examples of Well-Constructed Prompts:
○ “Examine the economic and social impact of remote work trends as of 2024,
discussing both benefits and challenges.”
○ “Assess the role of renewable energy policies in Southeast Asia, considering
regulatory, economic, and environmental factors as of 2024.”
○ “Compare two national policy approaches to addressing climate change,
discussing the strengths, limitations, and expected outcomes of each.”
2. Developing the Rubric
● P urpose: A rubric provides a clear, objective frameworkfor assessing AI responses.
Each rubric item should ensure consistency in scoring across evaluators and
transparently show where responses meet or fall short of the standards.
● Components:
○ B inary, Measurable Criteria: Every rubric item must be answerable with a
straightforward "Yes" or "No." Avoid subjective language like "good" or
"excellent," and instead, frame each item as a binary question to simplify scoring
and reduce bias.
○ Equal Weighting: Each criterion in the rubric shouldcarry equal weight. This
approach standardizes scoring across different tasks, with each item making an
equal contribution to the final score.
○ Focus on Key Qualities: Limit criteria to essentialqualities such as accuracy,
relevance, clarity, and depth of reasoning. Avoid overlap or redundancy between
items to maintain a clear focus.
○ Timelessness: Ensure rubric items remain applicableover time. If the prompt
includes time-sensitive data (e.g., "as of 2024"), reflect this in the criteria to
guarantee consistent evaluations in future assessments.
○ Safety and Ethics: Include criteria that confirm contentappropriateness,
ensuring responses are free from harmful, biased, or inappropriate language.
Examples of Binary Rubric Criteria:
●
○ Is all information provided factually accurate?—Yes / No
○ Does the response address all main points required by the prompt?— Yes /
No
○ Does the response provide sufficient analytical detail?— Yes / No
○ Is the writing clear and free of major grammatical errors?— Yes / No
○ Is the content relevant and on-topic?— Yes / No
3. Steps for Executing the Task
● Selecting a Task:
○ Begin by choosing a task that aligns with general knowledge but is complex
enough to challenge the AI’s reasoning ability. Avoid questions that are purely
factual, opting instead for prompts that require analysis and synthesis.
● Crafting the Prompt:
○ Draft prompts that push the model beyond basic recall, encouraging it to engage
with real-world scenarios that demand a combination of information synthesis
and critical thinking.
○ Use clear, straightforward language, but allow room for the model to interpret and
approach the answer in a meaningful way.
● Testing the Prompt:
○ Test the prompt in the model to ensure that it struggles with the answer. If the
model completes the task too easily, increase the complexity or adjust the
context.
○ Run the prompt multiple times to verify that the model consistently encounters
difficulties in the same areas, which confirms the prompt’s effectiveness at
exposing model limitations.
● Example Task Execution:
○ P rompt Example: "Analyze how urbanization impacts biodiversity in developing
countries. Discuss at least three consequences, citing general principles and
trends where applicable."
○ Binary Rubric Example:
■ Does the response identify three key consequences?— Yes / No
■ Does the response support analysis with general principles?— Yes /
No
■ Are relevant examples provided?— Yes / No
■ Is the structure clear and organized?— Yes / No
■ Is the writing free of major grammatical errors?—Yes / No
4. Final Evaluation and Feedback Process
● S coring the Response: Using the binary rubric, assesseach response against specific
criteria. Each item should be answered with "Yes" or "No" to ensure objectivity and
simplicity in scoring.
● Providing Constructive Feedback: When scoring, notestrengths and areas for
improvement. Detailed, constructive feedback is essential, especially if responses lack
key elements or contain inaccuracies.
● Reviewing and Iterating: Adjust prompts or rubricitems as necessary based on
response patterns. If a prompt is frequently misinterpreted or fails to expose model
weaknesses, consider rewording it for clarity and effectiveness.
5. Best Practices
● E xample Prompts and Ideal Responses: Maintain a listof prompts that effectively
expose model limitations, along with ideal answers that meet all rubric criteria. This list
serves as a reference for creating future prompts.
● Quality Assurance Checklist:
○ Prompt Checklist:
■ Clear, focused, and sufficiently challenging.
■ Includes necessary context without excessive detail.
■ Specifies the expected response format.
■ Avoids simple factual recall.
○ Rubric Checklist:
■ Each criterion is clear, measurable, and binary.
■ Criteria are relevant and applicable over time.
■ Binary (Yes/No) answers are applicable to each item.
■ Covers all essential aspects, from accuracy to ethical considerations.
xamples of Abstractive Summarization
E
Tasks at Three Levels of Complexity
his document provides examples of three summarization tasks for varying levels of complexity:
T
low, medium, and high. Each example includes a prompt, source text, and rubric for a
high-quality response. Following each example, a brief explanation clarifies the complexity level
and requirements.
Example 1: Low Complexity
Comment:
his is a simple example because the source text is short, the rubric has minimal criteria, and
T
the required data is straightforward to extract. The task only requires organizing numerical data
into a table, focusing on a few key points.
Prompt:
I am preparing a report on this topic. I need a table that covers all numerical data mentioned in
the text, including figures on the number of buses, their features, infrastructure details, and so
on.
Source Text:
In the Dominican Republic (DR), the adoption of electric vehicles (EVs) is growing.
In mid-October, the government delivered 150 electric buses for school
transportation. Onesimo González, director of the DR's Student Transportation
System, says this first fleet of eco-friendly, electric school buses – which are
equipped with GPS, 360-degree cameras, seatbelts, foldable STOP signs, and a
range of 200 kilometers per 100 percent charge – would impact over 160,000 public
school students. The vehicles also feature regenerative brakes for consistent and
safe operation.
he DR also recently installed the first fast-charging station for electric vehicles in
T
the Caribbean and Latin America. Located in Punta Cana, the station has
photovoltaic panels, state-of-the-art energy storage, and CCS1 chargers — the
fastest available in the region. With a capacity of 225 kilowatts (kW), it can
simultaneously charge 29 electric cars with 100 percent renewable energy. The
infrastructure required an investment of over USD 10 million and was developed
entirely with Dominican talent.
Rubrics:
A high-quality response should include:
● umber of electric buses: 150
N
● Range of buses on 100% charge: 200 kilometers
● Potential impact of 160,000 public school students
● Number of simultaneously charging cars in the Punta Cana station: 29
● The value of capacity of the Punta Cana station: 225 kilowatts
● The level of investment cost for infrastructure: USD 10 million
isclaimer: A high-quality response will fully adhereto the prompt and rubric criteria, using only
D
information provided in the text and avoiding external sources.
Example 2: Medium Complexity
Comment:
his medium-complexity task requires a structured summary suitable for a high school ESL
T
class. The summary must include specific key terms, avoid complex language, and keep within
a 300-word limit. These added requirements, including vocabulary restrictions and a concise
word count, make it more challenging than a basic summarization.
Prompt:
lease summarize this text in 300 words. I need this summary to be appropriate for a high
P
school essay for an English as a Second Language class, so don't use complicated words.
Include the following key terms in the summary: electric vehicles, charging stations, green
energy, maintenance cost, fuel cost, electricity cost, CO₂ emissions.
Source Text:
In theDominican Republic(DR), the adoption of electricvehicles (EVs) is growing. In
mid-October, the government delivered 150 electric buses for school transportation. Onesimo
González, director of the DR'sStudent TransportationSystem, says this first fleet of
co-friendly, electric school buses – which are equipped with GPS, 360-degree cameras,
e
seatbelts, foldable STOP signs, and a range of 200 kilometers per 100 percent charge – would
impact over 160,000 public school students. The vehicles also feature regenerative brakes for
consistent and safe operation.
he DR also recentlyinstalledthe firstfast-chargingstation for electric vehicles in the Caribbean
T
and Latin America. Located in Punta Cana in eastern Dominican Republic, the station has
photovoltaic panels, state-of-the-art energy storage, and CCS1 chargers — the fastest available
in the region. With its capacity of 225 kilowatts (kW), it can simultaneously charge 29 electric
cars with 100 percent renewable energy. The infrastructure required an investment of over USD
10 million and was developed entirely with Dominican talent.
he Evergo Connect charging station has only been operating for seven months, in which time it
T
has logged an average of 400 monthly users, with a steady growth of five percent month over
month. Each user visits the station to charge their vehicle up to four times a month, amounting
to around 1,600 charging transactions monthly. Joan Félix Benitez, Evergo’s senior
superintendent of innovation, estimates that the Punta Cana station will serve over 500 regular
users.
he company leading the project launched the first charging station in 2019. It now has over
T
560 stations, 487 of which are publicly accessible, while the rest are in private locations,
including businesses and residences. The company soon plans to open its first branch in the
heart of the capital,Santo Domingo, which Benitezsays “will be a pioneer”: “The charging
station we are setting up in the National District […] will have chargers with the highest available
power in the region, reaching up to 300 kW per unit. This will allow users to recharge large
vehicles in minimal time, offering unprecedented capacity in regional charging infrastructure.”
The station is set to open in the first quarter of 2025.
How the EV industry is being embraced in the DR
25 kW ultra-fast chargers, the most powerful of its class in Latin America. Photo by Evergo,
2
courtesyClimate Tracker, used with permission.
etween 2020 and 2023, there was a 60 percent increase in electric vehicle registrations — and
B
according to theHuman Settlements of SustainableMobility reportfrom the National Statistics
Office of the Dominican Republic, 3,043 electric vehicles have entered the country over the past
four years. CEO of Electric Vehicles, RD Rafael Flores, predicts that as time goes on, the
percentage will continue to grow. He said they anticipate an electric revolution soon: “The
Dominican Republic has made strides in creating infrastructure for electric mobility. Installing
charging stations, including fast chargers, has facilitated the use of electric vehicles in the
country.”
ccording to Flores, motorcycles are a primary means of transportation in the Dominican
A
Republic due to the ease of movement they offer amid frequent traffic jams. In 2023, there were
as many as 14,000 electric motorcycles on the country’s roads, some also used for deliveries.
lores says it is the category of electric vehicles that has grown the most: “Consumer
F
acceptance — primarily among merchants — has been exceptional. You can already see many
small grocery stores in the city with electric bikes, supported by large companies providing
options for these micro-entrepreneurs to acquire their units.”
Obstacles to mass adoption
he electric vehicle market in the Dominican Republic still faces challenges, however, including
T
the need for optimised incentives for EV purchases and improved charging infrastructure in rural
areas.
lores notes that while trends show sustained growth and a gradual shift toward more
F
sustainable mobility, the lack of accurate information and the spread of false news are among
the main obstacles to adopting sustainable mobility in the EV realm. Such myths and
misinformation can raise doubts among consumers, he says, delaying the transition to cleaner
and more efficient mobility.
ut what about cost? How do the initial and ongoing maintenance costs of EVs compare with
B
those of internal combustion vehicles in the country? Currently, you can purchase an electric
vehicle at roughly the same price as a fuel vehicle of similar quality or size, and several reliable
brands are already well established. In terms of maintenance, since EVs have fewer parts that
create friction, they require less preventive maintenance.
lores explains that the first major preventive maintenance for an electric vehicle could be after
F
15,000 to 20,000 kilometres, compared to about 5,000 kilometres for a combustion vehicle.
Preventive maintenance inspections are straightforward, including tyre rotation and checking
vehicle fluids (brake fluid, coolant, etc.), resulting in over 70 per cent annual savings on
maintenance costs.
In terms of electricity versus fuel consumption, savings are estimated at around 80 percent and,
in some cases, as much as 90 percent. The average person spending 12,000 Dominican pesos
(about USD 200) on fuel would only spend about 2,000–2,500 pesos (USD 33-42) on energy.
What do Dominicans think about EVs?
hile opinions have changed significantly in recent years thanks to available online information,
W
Flores says that some people still feel uncertain about whether it’s the right time to make the
switch.
ear of change is one of the main reasons most Dominicans don’t purchase electric vehicles, so
F
the Dominican Association of Electric Mobility (ASOMOEDO)hostsevents, talks, and forums to
inform the public about the benefits of sustainable mobility, encouraging people to consider a
hybrid or electric vehicle for their next purchase.
till, Flores insists, “the perception of electric vehicles in the Dominican Republic is generally
S
positive, especially among those looking for savings and sustainability,” but he admits “there are
s till barriers related to infrastructure, initial cost, and misinformation that hinder widespread
adoption.”
o, what does the future hold for electric vehicles in the Dominican Republic over the next five
S
to 10 years? Charging infrastructure is key to the success of electric vehicles, so both the
private and public sectors are expected to expand it further. In the medium term, the used car
market is expected to grow. As more people purchase new electric vehicles, there will be a
larger supply of used electric vehicles at more affordable prices.
Environmental benefits
ominican environmental activist Yenaro Ali calls electric vehicles allies of the environment,
D
explaining that their use reduces fuel use and excessive CO₂ emissions from fuel-powered
vehicles.
Vsdon’t producethe same negative emissions as fuel-poweredvehicles — and the fact that
E
even the charging stations have the ability to operate on renewable energy makes their use
even more viable in reducing environmental impact.
“ With the current adoption projections for electric vehicles in the Dominican Republic,” Ali
explains, “national electricity consumption could increase by three to six percent by 2030 due to
electro-mobility — a manageable level from an energy standpoint.”
In the case of Evergo and its partners, more than 1.9 GWh of energy used annually in their
stations in the Dominican Republic is from renewable sources. This saves the emission of over
1.7 million kg of CO₂ being released into the environment.
User experiences
or many, the decision may come down to the driving. Electric vehicle user Helena Saenz
F
notices no difference between her car and combustion vehicles. In fact, among the advantages
she’s observed are privileged parking spaces where charging stations are located, lower energy
costs, time savings, and less maintenance.
he cost is about DOP 1,000 (about USD 17) each time she charges it. For a combustion
T
vehicle, it was 4,000 pesos (USD 66). “Charging the vehicle is now part of my daily routine,”
says Saenz. “I don’t have to add an extra task to my daily to-do list to charge my vehicle.”
nother user, Robert Burgos, drives a Skywell and has had an excellent experience as he no
A
longer worries about oil or coolant changes or the vehicle overheating. His vehicle provides him
with a range of 400 to 500 kilometres per charge.
Rubrics:
To qualify as a high-quality response, the summary should include:
● A brief introduction to the growth of EV adoption in the Dominican Republic, highlighting
both government and private sector efforts.
● Mention of the government’s introduction of 150 electric school buses and their features.
● A description of the new fast-charging station in Punta Cana and its renewable energy
use.
● Information on the expansion of the Evergo Connect charging network, emphasizing
both public and private access points.
● A summary of environmental benefits, including reduced CO₂ emissions and the use of
renewable energy in charging.
● Details on cost savings (maintenance, fuel, and electricity).
● An overview of obstacles, such as infrastructure needs, misinformation, and high initial
costs.
● Expectations for infrastructure expansion and the potential rise in the used EV market.
isclaimer: A high-quality response will strictlyadhere to the prompt and rubric criteria, using
D
only the provided information and avoiding external sources.
Example 3: High Complexity
Comment:
his is a high-complexity example due to numerous constraints. The task requires splitting
T
content into slide-specific notes with designated headers, focusing on factual details, limiting
bullet points per slide, and providing a final comparison table. These additional structuring and
formatting requirements significantly increase the complexity.
Prompt:
lease rewrite this as key notes for my presentation. Split slides by headers in the text, take the
P
first slide from the introductory part. Each slide should have the same title as in the text except
for the first one. The key notes should focus on facts. Use bullet-point lists with up to 4 bullet
points per slide. Also, please provide a final slide with a table comparing electric vehicles and
combustion vehicles based on the text given.