Gen AI Task: Build an SHL Assessment Recommendation System

Problem Overview
Hiring managers often struggle to find the right assessments for the roles that they are hiring for. The current
system relies on keyword searches and filters, making the process time-consuming and inefficient. Your task
is to build an intelligent recommendation system that simplifies this process. Given a natural language
query or a job description text or URL, your application should return a list of relevant SHL assessments.
You can take a look at the data sources that you are going to work with here,
[Link]

Your Task
Design and develop a web application that:

1. Takes a given natural language query or job description text URL

2. Recommends at most 10 (min 1) most relevant Individual test solutions from here in the tabular
format
3. Each recommendation needs to have at least the following attributes
– Assessment name and URL (linked to SHL’s catalog)
– Remote Testing Support (Yes/No) and Adaptive/IRT Support (Yes/No)
– Duration and Test type

Submission Materials
You need to submit the following items using this form.

- Three URLs, 1) First URL where you can host the working demo (we type in queries and see the
result) 2) API end point which can be queried using a query or piece of text and returns result in
JSON (API Configurations mentioned below in Appendix 2) 3) URL of the code on GitHub which we
can see (URL of the complete code - Including your experiments and evaluation)

- 1-page document outlining your approach on how you solved this problem. Write this document as
concisely as possible with appropriate information. Highlight the tools, libraries that you have used.

These points are vital for us to evaluate the technical completeness and robustness of your solution:

• Accessibility: Ensure that the API URL providing access to your recommendation engine is
functional, your code is accessible via a public or private (shared with us) GitHub repository, and
the UI (if you build one) is reachable and working.

• Evaluation: You must detail the evaluation metrics you used, the achieved evaluation score, and
the strategies you tried to measure the system's performance. Implementation and clear
documentation of your evaluation pipeline and results are mandatory for this option. (This is
critical to consider your submission, and you can find the test dataset to evaluate and report
metric at the bottom of this document)

(Optional) Optimization: You are welcome to briefly document any efforts you put into optimizing the
overall performance score, including initial results and how you improved them.

Evaluation Criteria
We will be using the following criteria to evaluate your solution.

• Approach: The approach that you took to solve this problem. How have you crawled, represented,
stored, and searched the data? How much of emerging LLM stack that you leveraged? We love
evals and tracing -- would like to see how you have leveraged those.
 • Accuracy: Accuracy on a benchmark set. Measured using Mean Recall@3 and MAP@3.

• Demo quality: The quality of the end-to-end demo and attention to the details. It is OK if you do not
have front-end skills. You can use low-code frameworks like Streamlit, Gradio.

Resources
You are not restricted to use these – feel free to use from anywhere else. Number of cloud platforms allow you
to host applications and APIs for free for some time. Feel free to leverage those.

1. LLMs/Gemini Free APIs: [Link]

Index: Metrics to compute accuracy

Your solution will be assessed using the following ranking evaluation metrics:

1. Mean Recall@K

This metric measures how many of the relevant assessments were retrieved in the top K
recommendations, averaged across all test queries.
Number of relevant assessments in top K
𝑅𝑒𝑐𝑎𝑙𝑙@𝐾 =
Total relevant assessments for the query
𝑁
1
𝑀𝑒𝑎𝑛𝑅𝑒𝑐𝑎𝑙𝑙@𝐾 = ∑ 𝑅𝑒𝑐𝑎𝑙𝑙@𝐾𝑖
𝑁
𝑖=1

where N is the total number of test queries.

2. Mean Average Precision @K (MAP@K)

MAP@K evaluates both the relevance and ranking order of retrieved assessments by calculating
Precision@k at each relevant result and averaging it over all queries.
𝐾
1
𝐴𝑃@𝐾 = ∑ 𝑃 (𝑘) ⋅ 𝑟𝑒𝑙(𝑘)
min(𝐾, 𝑅)
𝑘=1
𝑁
1
𝑀𝐴𝑃@𝐾 = ∑ 𝐴𝑃@𝐾𝑖
𝑁
𝑖=1

where:

• R = total relevant assessments for the query

• P(k) = precision at position k
• rel(k) = 1 if the result at position k is relevant, otherwise 0
• N = total number of test queries

A higher Mean Recall@K and MAP@K indicate a better-performing recommendation system.

Appendix 1: Metrics to compute accuracy

Here are some of the queries that you can use to test your application.

• I am hiring for Java developers who can also collaborate effectively with my business teams. Looking
for an assessment(s) that can be completed in 40 minutes.

• Looking to hire mid-level professionals who are proficient in Python, SQL and Java Script. Need an
 assessment package that can test all skills with max duration of 60 minutes.

• Here is a JD text, can you recommend some assessment that can help me screen applications. Time
limit is less than 30 minutes.

• I am hiring for an analyst and wants applications to screen using Cognitive and personality tests,
what options are available within 45 mins.

Appendix 2: API Structure & Endpoints

This section outlines the required API configuration for the take-home assignment. Your API must implement the
endpoints described below exactly as specified to ensure proper evaluation of your submission.

Base requirements:

- Your API should be accessible via HTTP/HTTPS

- All responses should use proper HTTP status codes

- All data exchanges must be in JSON format

Required Endpoints:

1. Health Check Endpoint: This endpoint provides a simple status check to verify the API is running.

2. Assessment Recommendation Endpoint: This endpoint accepts a job description or Natural language
query and returns recommended relevant assessments (At most 10, minimum 1) based on the input.

Response Fields Explanation: For the /recommend endpoint response ->

 Example Response:

Testing your API: Before submitting your API link, please verify that ->

1. Both endpoints are functioning correctly

2. The response formats match exactly what is given

To assist you with implementing the evaluation component of your chosen assignment, we have
attached a test set that you can use in your evaluation pipeline.

SHL Assessment Recommendations

Query Assessments URL

[Link]
Automata - Fix (New) | SHL catalog/view/automata-fix-new/
Core Java (Entry Level) (New) [Link]
I am hiring for Java developers who | SHL catalog/view/core-java-entry-level-new/
can also collaborate effectively with
[Link]
my business teams. Looking for an
Java 8 (New) | SHL catalog/view/java-8-new/
assessment(s) that can be
completed in 40 minutes. Core Java (Advanced Level) [Link]
(New) | SHL catalog/view/core-java-advanced-level-new/
Agile Software Development [Link]
| SHL catalog/view/agile-software-development/
 Technology Professional 8.0 [Link]
Job Focused Assessment | catalog/view/technology-professional-8-0-job-
SHL focused-assessment/
Computer Science (New) | [Link]
SHL catalog/view/computer-science-new/
Entry level Sales 7.1 [Link]
(International) | SHL catalog/view/entry-level-sales-7-1/
Entry Level Sales Sift Out 7.1 [Link]
| SHL catalog/view/entry-level-sales-sift-out-7-1/
Entry Level Sales Solution | [Link]
SHL catalog/view/entry-level-sales-solution/
Sales Representative Solution [Link]
| SHL catalog/view/sales-representative-solution/
Sales Support Specialist [Link]
I want to hire new graduates for a Solution | SHL catalog/view/sales-support-specialist-solution/
sales role in my company, the Technical Sales Associate [Link]
budget is for about an hour for each Solution | SHL catalog/view/technical-sales-associate-solution/
test. Give me some options [Link]
SVAR - Spoken English (Indian catalog/view/svar-spoken-english-indian-accent-
Accent) (New) | SHL new/
Sales & Service Phone [Link]
Solution | SHL catalog/view/sales-and-service-phone-solution/

Sales & Service Phone [Link]

Simulation | SHL catalog/view/sales-and-service-phone-simulation/
English Comprehension [Link]
(New) | SHL catalog/view/english-comprehension-new/
Motivation Questionnaire [Link]
I am looking for a COO for my MQM5 | SHL catalog/view/motivation-questionnaire-mqm5/
company in China and I want to see
Global Skills Assessment | [Link]
if they are culturally a right fit for
SHL catalog/view/global-skills-assessment/
our company. Suggest me an
assessment that they can complete [Link]
in about an hour Graduate 8.0 Job Focused catalog/view/graduate-8-0-job-focused-
Assessment | SHL assessment-4228/
[Link]
Drupal (New) | SHL catalog/view/drupal-new/
Search Engine Optimization [Link]
(New) | SHL catalog/view/search-engine-optimization-new/
[Link]
Content Writer required, expert in Administrative Professional - catalog/view/administrative-professional-short-
English and SEO. Short Form | SHL form/
Entry Level Sales Sift Out 7.1 [Link]
| SHL catalog/view/entry-level-sales-sift-out-7-1/
[Link]
General Entry Level – Data catalog/view/general-entry-level-data-entry-7-0-
Entry 7.0 Solution | SHL solution/
Find me 1 hour long assesment for
the below job at SHL
Job Description

Join a community that is shaping

the future of work! SHL, People
Science. People Answers.

Are you a seasoned QA Engineer

with a flair for innovation? Are you
ready to shape the future of talent
assessment and empower
organizations to unlock their full
potential? If so, we want you to be a
part of the SHL Team! As a QA
Engineer, you will be involved in
creating and implementing software
solutions that contribute to the
development of our ground-
breaking products.

An excellent benefit package is

offered in a culture where career
development, with ongoing
manager guidance, collaboration,
flexibility, diversity, and inclusivity
are all intrinsic to our culture. There
is a huge investment in SHL
currently so there’s no better time [Link]
to become a part of something Automata Selenium | SHL catalog/view/automata-selenium/
transformational.
[Link]
Automata - Fix (New) | SHL catalog/view/automata-fix-new/
What You Will Be Doing
[Link]
Getting involved in engineering Automata Front End | SHL catalog/view/automata-front-end/
quality assurance and providing [Link]
inputs when required. JavaScript (New) | SHL catalog/view/javascript-new/
Create and develop test plans for [Link]
various forms of testing. HTML/CSS (New) | SHL catalog/view/htmlcss-new/
Conducts and/or participates in [Link]
formal and informal test case HTML5 (New) | SHL catalog/view/html5-new/
reviews.
[Link]
Develop and initiate functional
CSS3 (New) | SHL catalog/view/css3-new/
tests and regression tests.
Rolling out improvements for [Link]
testing and quality processes. Selenium (New) | SHL catalog/view/selenium-new/
[Link]
Essential SQL Server (New) | SHL catalog/view/sql-server-new/
[Link]
What we are looking for from you: Automata - SQL (New) | SHL catalog/view/automata-sql-new/
 Development experience – Java or
JavaScript, CSS, HTML (Automation)
Selenium WebDriver and page
object design pattern (Automation)
SQL server knowledge
Test case management experience.
Manual Testing

Desirable

Knowledge the basic concepts of

testing
Strong solution-finding experience
Strong verbal and written
communicator.

Get In Touch

Find out how this one-off

opportunity can help you achieve
your career goals by making an
application to our knowledgeable
and friendly Talent Acquisition
team. Choose a new path with SHL.

#CareersAtSHL #SHLHiringTalent

#TechnologyJobs
#QualityAssuranceJobs

#CareerOpportunities
#JobOpportunities

About Us

We unlock the possibilities of

businesses through the power of
people, science and technology.
We started this industry of people
insight more than 40 years ago and
continue to lead the market with
powerhouse product launches,
ground-breaking science and
business transformation.
When you inspire and transform
people’s lives, you will experience
the greatest business outcomes
possible. SHL’s products insights,
experiences, and services can help [Link]
achieve growth at scale. Manual Testing (New) | SHL catalog/view/manual-testing-new/
 What SHL Can Offer You

Diversity, equity, inclusion and

accessibility are key threads in the
fabric of SHL’s business and culture
(find out more about DEI and
accessibility at SHL )
Employee benefits package that
takes care of you and your family.
Support, coaching, and on-the-job
development to achieve career
success
A fun and flexible workplace where
you’ll be inspired to do your best
work (find out more LifeAtSHL )
The ability to transform workplaces
around the world for others.

SHL is an equal opportunity

employer. We support and
encourage applications from a
diverse range of candidates. We
can, and do make adjustments to
make sure our recruitment process
is as inclusive as possible.

SHL is an equal opportunity

employer.
[Link]
Administrative Professional - catalog/view/administrative-professional-short-
Short Form | SHL form/
Verify - Numerical Ability | [Link]
SHL catalog/view/verify-numerical-ability/
Financial Professional - Short [Link]
Form | SHL catalog/view/financial-professional-short-form/
ICICI Bank Assistant Admin,
Experience required 0-2 years, test [Link]
should be 30-40 mins long Bank Administrative Assistant catalog/view/bank-administrative-assistant-short-
- Short Form | SHL form/
[Link]
General Entry Level – Data catalog/view/general-entry-level-data-entry-7-0-
Entry 7.0 Solution | SHL solution/
[Link]
Basic Computer Literacy catalog/view/basic-computer-literacy-windows-
(Windows 10) (New) | SHL 10-new/
KEY RESPONSIBITILES: [Link]
Verify - Verbal Ability - Next catalog/view/verify-verbal-ability-next-
Manage the sound-scape of the Generation | SHL generation/
 station through appropriate [Link]
creative and marketing SHL Verify Interactive - catalog/view/shl-verify-interactive-inductive-
interventions to Increase or Inductive Reasoning | SHL reasoning/
Maintain the listenership
Acts as an interface between
Programming & sales team, thereby
supporting the sales team by
providing creative inputs in order to
increase the overall ad spends by
clients
Build brand Mirchi by ideating fresh
programming initiatives on air
campaigns, programming led on-
ground events & new properties to
ensure brand differentiation & thus
increase brand recall at station level
Invest time in local RJs to grow &
develop them as local celebrities
Through strong networking, must
focus on identifying the best of local
talent and ensure to bring the
creative minds from the market on
board with Mirchi
Build radio as a category for both
listeners & advertisers
People Management
Identifying the right talent and
investing time in developing them
by frequent feedback on their
performance
Monitor, Coach and mentor team
members on a regular basis
Development of Jocks as per
guidelines
Must have an eye to spot the local
talent to fill up vacancies locally

TECHNICAL SKILLS &

QUALIFICATION REQUIRED:

Graduation / Post Graduation (Any

specialisation) with 8 -12 years of
relevant experience
Experience in digital content
conceptualisation
Strong branding focus [Link]
Must be well-read in variety of Occupational Personality catalog/view/occupational-personality-
areas and must keep up with the Questionnaire OPQ32r | SHL questionnaire-opq32r/
latest events in the city / cluster /
country
Must know to read, write & speak
English

PERSONAL ATTRIBUTES:

Excellent communication skills

Good interpersonal skills
People management

Suggest me some tests for the

above jd. The duration should be at
most 90 mins