Code replacing code

Using LLMs to judge query intent,

provide more relevant responses, and
stand up smarter shopping experiences

Introduction

The rapid advancement of artificial intelligence (AI) has not only drastically altered
what and why technologists build, it's also started to change how technologists build
those things. Computer code writing more computer code is no longer a futuristic
concept synonymous with Skynet going live, and it doesn’t require taking a red pill to see
its effects. AI is now a tangible, subtle, and nearly omnipresent force shaping our daily
choices and interactions.

From the subtle curation of our music playlists, to the split-second selection of the next
video on our feed, or from the pleasantly robotic voices in our kitchens to the dark window
that’s doing our children’s homework, AI's influence permeates our lives. For those at the
helm of technology – CTOs and senior developers – this transformation presents both an
opportunity and a challenge. The ability to leverage large language models (LLMs) both as
a means to rapidly write or rewrite code and as a means to replace the need to write those
lines of code in the first place has, in the span of three years, become table-stakes for
companies on the cutting edge of software development.

Based on the presently available domain knowledge, we’ll consider this trend through the
scope of search, and how it's already manifested in the ability of large language models to
judge the relevance and intent of an admittedly rushed, sometimes brief, almost always
context-less search query. While search experiences have been a core component of
dynamic and user-centric ecommerce experiences almost as long as ecommerce has
been a word, never before have the engines behind those simple search bars been so
good at taking only a few words, and easing consumers or potential customers to their
most likely purchases with the minimum amount of friction or stress. Knowing a searcher's
intent based on such little information borders on telepathy, but there is a subtle art to it,
and no little amount of science.

This article aims to move beyond the theoretical "when" and "where" can AI (and more
specifically LLMs) play a role in the consumer search experience (those answers are “now”
and “everywhere,” respectively). What follows focuses instead on the practical "how." It
will dissect the latest trends in AI-human entanglement, provide a strategic framework for
employing LLMs to significantly enhance development efforts, and offer concrete code
examples, particularly within the retail and ecommerce sectors that provide better results
because they have LLMs under the hood, and even more specifically as these topics relate
to standing up next-gen search experiences quickly and effectively.

Code replacing code 2

What we’ll be building
If our problem statement is to provide more
Sport relevant responses to succinct queries, let’s
start with something at the core of that: the
BASKETBALL FAN difference between recommendation and
personalization. In laymen’s terms, a
Sport Match
recommendation is a type of response you
Semi-final - $110 ticket
could get from a thing like a human or an
LLM or a search experience (though by the
SNEAKERS ADDICT end of this article there should be little
distinction between the last two), and
Sport Shoes
personalization is a way that
Nike Air - $140 recommendation could be more relevant to
you, specifically.

After all, you probably get recommendations all the time: parents telling you to save your
money, in-laws telling you how to mow the lawn, your children telling you not to use that
word. If you already get the picture then feel free to skip ahead, but if you want a more
formal definition, Emile Contal, PHD in Statistical Learning defines the practice of
personalization as one in which:

“...content is modified based on some personal [Link] can be your past

purchases on an online store, or your age, or where you live or simply what device you are
using… What a user gets differs from what another user gets.”

Recommendation on the other hand requires:

“filter[ing] a collection of things based on historical behavior of a user (typically likes/

dislikes or behavioral history). Framing it this way recommendation is a form of
personalization.”

Some might argue, pedantically, that a recommendation must necessarily build upon
personalization, because what are your previous shopping or clicking or viewing habits if
not personal? Perhaps this data is even more personal than demographic information.
Others, like Contal, would argue that the knowledge of previous activity and the history
necessary for a recommendation are all ways to personalize. In any case, it's clear that the
two are inextricably linked, because a good recommendation is not general. Your parents
tell everyone to save their money, regardless of circumstance, your in-laws give everyone
the same lawn advice and your children, hopefully, have learned that it's never okay to
swear. Good recommendations are personalized. This Algolia blog post defines a
personalized recommendation, in the scope of site search, as:

“...a relevant suggestion generated by a recommendation engine (a.k.a. a recommender

system) using an algorithm and filtering options based on what’s known about the
customer’s on-site meanderings.”

Code replacing code 3

A quick note on why we do this, before we get to the “how.” McKinsey research found that
shoppers are 56% more likely to return to a site that recommends products, 35% of
[Link]’s revenue comes from product recommendations, and 75% of what people
watch on Netflix comes from recommendations. Product and engineering people will see
this as a great win for the technology they built; the result of a thoughtful and successful
development process. Business people will see this as an opportunity to increase retention
and/or return users, increase customer lifetime value, decreased time customers spend
looking for what they want, increased basket sizes, and customers more efficiently getting
to what they may not have even known they wanted. More on that from Lucia Kubinska at
Algolia DevBit here.

She goes on to outline three ways to build recommender systems, either hard-coding
rules subjectively chosen by developers or stakeholders, using machine learning to predict
what they might want, or using personalized recommendations to dynamically rerank
items based on additional signals about the user. If you want more information about the
pros and cons of each, check out her video, but for the purposes of this article, we’ll make
an assumption and say that the personalized recommendations perform significantly
better, and we want to focus on how to build them, specifically using the most cutting
edge artificial intelligence to do so, for optimal results. So without further preamble, let's
get into how we could start planning our LLM search experience.

Choosing between recommendation engines

Before we get to the design of the recommendation model itself, let's consider how much
horse-power we need to create good recommendations, defined as: the customer or user
actually wants what your engine recommended. Suffice it to say that there’s a continuum
of very simple recommender engines and extremely complex, you could even call them
black-box, recommendation systems.

Categorical and parametric

The two most simple recommendation systems basically take a single data-point and
provide extremely general recommendations. The first has existed for about as long as the
bookstore has. You walk in, you tell the clerk you like to read fantasy and they point you to
a section, possibly with a sign on it. The limitations in this methodology are immediately
apparent for anyone who reads fantasy because that clerk probably pointed you to the
science fiction section. To the recommender, the distinction between dragons and space
ships is arbitrary but not to you, the potential reader. Neither is the distinction between
paranormal romance (AKA “romantasy”) and George R.R. Martin, a grim-dark fantasy
writer who may incorporate some romance in his otherwise gritty tales. More on this later,
but for now lets call this simple, historic recommendation engine “categorization,” where
only the overall category is used to recommend very broad swathes of products to a
customer and the rules are hard-coded for everyone in every bookstore by someone in a
New York publisher’s office.

Code replacing code 4

The next level up from that, we’ve already hinted at, which is parametric recommendation,
or the ability to pull in specific components of an object and present it to the user, most
commonly if they’ve previously expressed interest in and engaged content with similar
parameters. That’s like going up to a bookstore clerk and saying “do you have anything
about teen love-triangles and vampires,” so that they can point you to Twilight and the
subgenre it spawned (the parameters being “teen,” “love-triangle,” and “vampires”). 

But there’s a more famous example in technology and it also comes from 2005: Pandora.
Pandora was a game-changing use of recommendation engines for music and it was
certainly one of the most comprehensive systems at the time. Pandora allowed users to
like and dislike songs they hear on a digital radio, and with only those two possible options
(plus the categories the user inputs originally) the product could create a radio station just
for them. How? The Music Genome Project. Every single song that Pandora had the rights
to play was tagged with a growing number of parameters (now hundreds) and the more
songs a user liked with certain parameters the more Pandora would recommend songs
with those same parameters. E.g., Evanescence would have something like: symphonic, alt
rock, female lead vocals, gothic vibe, and many more tags that Pandora’s interface can’t
possibly show on the “why am I hearing this” section. And if on subsequent listens the
user “liked” other songs that had the symphonic and female lead vocals tags, but not the
gothic vibe, it would adjust to that. 

An image of a pandora interface identifying the many, specific parameters used to generate a
recommendation 

Source

Code replacing code 5

Sounds good, and it was, until you consider the amount of effort it takes to tag up all
those songs. This Medium article says that each song has approximately 450 “genes” that
(at least originally) got manually assigned by musicians. Multiply 450 by how many songs
exist in Pandora’s database. Then add that many songs are cross-referenced with ratings
from others within the Music Genome Project to eliminate the inherent bias of humans
tagging these things up — you get the picture. It's effortful.

So parametric recommendations could be comprehensive and they

can work really well, but they’re effortful, not dynamic, and often
subjective. Therefore, products that use them are losing.

See this eMarketer graphic if you want a scoreboard for parametric recommenders vs.
more modern means, in the music space.

Predictive

The natural progression of all this parametric data is to use statistical models, or machine
learning algorithms, to predict future behavior. While Spotify has a vastly different
recommendation system from Pandora, there’s another element to Spotify that Pandora
doesn’t take into account, which is user behavior. Spotify layers in, not only preferences in
types of music, but it also layers in the context of when the user might be in the mood for
something else.

A simple, general example of how this might be put to use is the advent of the “New Music
Friday” playlist. Someone within Spotify – presumably – did an analysis and found that
people are most open to discovering and exploring new releases over the weekend,
perhaps because many users are more likely to actively listen on days where they’re not
working. Meanwhile, “Discover Weekly” playlists update with recommendations
personalized to the user on Mondays. Again, something in user behavioral trends led
product developers to think that “more of what you like but new to you” was best placed
at the beginning of many users’ work weeks.

Neither of these are obvious uses of machine learning (ML), yet, but let's start to layer in
the two concepts of personalization and predicting behavior via machine learning applied
to trends. Given the (probably manual) analysis on listening patterns over the course of
the week, it's not hard to imagine Spotify applying a statistical learning model to each and
every user’s listening habits, and then semi-dynamically recommending different moods or
genres at different times of the day. To build the concepts atop one another, Pandora told
users “make radio stations with what you want to listen to and then choose between your
curated stations.” Spotify responded, “Listen to what you want, we’ll make you radio
stations, and bump them to the front of the list based on what you normally go for around
this time.” You want something upbeat during your commute to work? Great. Spotify has
built you a station for that and it's the first thing you see when you open the app at 8 AM.
You link to wind down after work? Here’s a “Mellow Mood” playlist at 8 PM.

Code replacing code 6

In retail and ecommerce, applying some machine learning models to buyer behaviors
would allow for more holiday-themed items — you guessed it – around the holidays. Yes, a
product manager could do that, but the important thing to focus on here is that the
machine learning model has no idea what a holiday is or how to celebrate one, it just
knows that there are certain types of products more likely to be purchased at certain
times of the year. A laser-focused goal of increasing a metric, like number of sales or
revenue, plus the generic, almost blithe understanding of the interaction between two
variables (perhaps agnostic to or outside of the “human” logic that causes it) allows a
more dynamic approach to other seasonal trends, like Gillette sales going up in spring. A
product manager doesn’t need to know why the sales go up, in fact, if they can grow
beards they might not have thought of it, but the machine learning model picked up on a
statistical trend, and without caring what the causal pathway is, can do something like
reranking razors above lip balm for certain stores in certain recommendation formats.

Now this leads to the limitations of such

a system. Firstly, machine learning-
based recommender systems can only
function on certain types of data. They
tend to be very easy to implement – and
perform well on – things like numerical
and/or temporal variables, more
specifically on cyclical or seasonal
trends, or they can be used to classify
Ai datasets into “likely to buy now” or “not
likely to buy now.” But they have their
limitations. Language, or other types of
unstructured data, is a big limitation for
ML. Also, machine learning models need
lots of data to train, and while you might
have lots of data on users in general,
what do you recommend to a first time
site-visitor? Personalization can be
difficult with machine learning.

We’ve alluded to another type limitation that has nothing to do with the type of
recommendation engine and everything to do with the type of recommendation model
the recommendation engine is powering. It's all well and good to put razors ahead of lip
balm in a generic “trending items” carousel, but where else would you see both of those
things together? Certainly not in a “complete the outfit,” carousel of products. So while
what’s under the hood of the recommendation can be extremely important, it might only
be as useful as the model’s design. Let's discuss a few different recommendation models
and how to best use (or avoid) them when designing media and ecommerce user
experiences. 

Code replacing code 7

Choosing between recommendation models
If you’ve been in the chip aisle of a grocery store, you already know of one type of
recommendation model. You probably walked past the tortilla chips and saw, on a little
shelf right in the middle of them, different types of dips and salsas. This is exactly what
we mean by a recommendation model as opposed to an engine. While it's essentially the
simplest possible engine in a shopping experience not at all enabled by technology, the
recommendation to buy salsa with your chips works. Shoppers are happier because they
don’t have to double back to “condiments” if they did want salsa, and the store is happier
because the figurative and literal cart size has increased for shoppers who might have
been teetering on the edge of chips with dip or just the chips.

Complete the x, customers also bought, and related items

This type of recommendation model is called a complementary product model, or more
colloquially, “complete the x.” Very similar to the complementary product model is the
“customers also bought” model, with the subtle difference that “complete the x” could be
based on an authority choosing which products are complementary. This authority could
be the brand, it could be the retailer or the website’s product team if they have specific
enough domain knowledge. It could even be a designer or influencer if that “x” that's being
completed is an outfit or look. Something like this could be accomplished by setting up
recommend rules in your Algolia dashboard, to set up a recommendation when a specific
item is viewed (e.g. boosting that top when viewing those jeans), versus generally
boosting a specific product.

Whereas, in the “customers also bought” model, specifically the buying patterns of other
customers are the ones who are choosing complementary products. In this model, the
products don’t even need to be logically complementary; they could simply be indicative
of a very frequent basket of goods, like eggs and milk. Documentation for the Algolia
recommend product calls this the “strict” variant of “frequently bought together,” in that
Items that have been purchased together in the past.

Code replacing code 8

To use a different example, with more clear pros and cons, if you are shopping around for a
new gaming computer, you could use a “customers also bought” recommendation model if
you have no vested interest in which other products a customer adds to their shopping
cart, so long as basket size increases. Or, if, as the retailer, you have a vested interest in a
brand, or perhaps you’ve done analysis that shows customers are more likely to add those
other items to their cart (and then checkout) if those other items match the gaming
computer in terms of brand and color, then both you and your customers would be better
served by a “related items” model, where content with related attributes (like brand) are
shown, rather than related items (like computer).

Buy again and continue shopping

There are a few model selections here that would actively disrupt your customer’s
buying journey, perhaps causing them stress or fatigue, and causing the likelihood of a
completed sale to decrease. First, benign but still rather annoying, is the scenario in
which you serve one of two other recommendation models to a returning customer.
Either a “buy again” or a “continue shopping” recommendation would waste an
opportunity to better serve that returning customer on a second visit. Why? A
“continue shopping recommendation” might be the right choice, if your customer never
checked out, was only browsing gaming computers, but in the end decided they
couldn’t afford it right now. But if they did complete a purchase, what is the likelihood
that they want to buy a $3,000 alienware computer right after buying a $2,000 Legion
computer? Pretty slim, right? A “buy again” recommendation model makes no sense
here for the same reasons, the purchase is simply too large and redundant for most
shoppers. You’re better off putting just about any other model in place after checkout.

Trending and visually similar

It would, however, make perfect sense for a skincare brand’s site, where products have
limited use and customers frequently restock a brand they’ve grown loyal to. The
"trending items" recommendation model is also very popular in beauty and fashion,
which recommends products with sharp upticks in recent conversion, either within a
category or across the catalog, depending upon configuration. What wouldn’t likely
work for that skincare brand is a “visually similar” model, which provides other
recommendations based on things that look the same. There are any number of
products that have similar packaging or come in similarly shaped containers, and there
might be little correlation between a buyer's intent to purchase and the shape of that
container. 

But “visually similar” recommendation models are very popular for home decor and for
clothing for the exactly converse reason. If a shopper likes the look of one type of
garment, there is a chance, depending upon what garments you're selling, that they’ll
like the look of others. In this case, the purchase might not be redundant, or overly
financially burdensome, but might be smartly building a repertoire or wardrobe, or in
the case of home decor, tying a room together.

Code replacing code 9

For more information specifically on image-based recommendations, check out this
documentation.

So, the product, first of all, is extremely important context for the recommendation model.
And second of all, the customer’s checkout history, not just their clicking and viewing
history, is also necessary context for top notch recommendation systems and therefore a
large component in frictionless, pleasurable shopping experiences.

Comparison
Speaking of friction, there are also a few instances in which there's nothing wrong with the
selected model. Indeed, it might be the right model, but the way it's been implemented is
actively unhelpful. For instance, many shoppers find it useful to compare the current item
they are viewing to other recommendations (which could be accomplished with a variety
of Algolia models or rules). Go back to the gaming computer example and imagine a
customer considering graphics card and cooling system specifications, side-by-side.
Online shopping experience developers might want to make this “comparison”
recommendation available for customers, but not too available. Putting such a
recommendation “below the fold” or low enough on the page that customers have to scroll
for it might be the right balance of easy to find, but not forcing a customer to second
guess their initial product selection if they’ve already made up their mind before they got
to this page. Causing this type of decision paralysis so late in the buying journey with
thoughtless design could cost you sales and stress out your customers, fatiguing them to
the point where they say, “You know what? I’m just going to look into this more later,” and
X out. They may or may not “look into this more later.”

Similarly, the position of the recommendation system within the checkout flow is critical.
Your product might have a clearly distinct “exploration” phase and then a separate phase
where a customer has probably already made up their mind and just wants checkout to be
easy. In this case – and this case is probably true for many products – you wouldn’t want
to provide the types of recommendations that make your customer rethink the entire
purchase, just as they’re going to click “buy.”

Code replacing code 10

It might be fine to ask a customer if they want desert along with their pizza – data analysis
could probably tell you if its helpful or making people hangry – but you certainly don’t want
to send your customers back to “go” by showing them a different Alienware computer
when they’ve already braced themselves for their biggest purchase of year.

Recommendation models in the checkout flow need to be very carefully selected so that
they can increase average order value (AOV) without overwhelming your customer to the
point of cart abandonment. This could come down to having a top tier product team or
partner, or there are some other cutting edge breakthroughs that could supercharge this.
We’ll discuss those next. More on leveraging recommendations at different touch points,
here.

Using LLMs

Large Language Models (LLMs) are one of many types of artificial intelligence that have
advanced rapidly in the last few years, with huge impacts to both business applications
and everyday lives. If you’ve got a handle on what LLMs really are, feel free to skip ahead.
But before we dive into how LLMs could be used to shorten development cycles and make
more dynamic products, let's cover some LLM basics.

LLMs are foundational models, or pre-trained. Here’s an example of what “foundational”

means in this context: humans are foundational models in the sense that we get a lot of
very general training before we’re ready to complete any specific higher-level tasks. You
can think of your childhood and teenage years as foundational in that you’re learning to
walk, talk, interface with others in a (hopefully) healthy manner, discovering a work ethic
and ambitions, all before you get your first real job. That’s you getting pre-trained for
whatever life throws at you next.

LLMs differ from more purpose built AI

10101010

01010101

10100101

10101100
in this important respect. So, instead of
differentiating between curb and crack
in the road, plastic bag or ill-advised cat
crossing, like the AI in a Tesla does
everyday, or looking at something else
like sales datapoints to forecast a future
pattern like statistical models do, LLMs’
T T
LLM
T purpose during their training is just to
learn all of language. The purpose of
learning language for no reason in
particular is so that they can use it as a
foundation to complete more specific
tasks, later, that are given them by
1
users, via prompts.
2

Code replacing code 11

 T he training set for something like ChatGPT (an LLM) is necessarily very general,
because its job isn’t to drive or predict things based on one dataset. Its job could be
anything from writing a school paper to reading a legal contract. LLMs might not be the
right choice to drive your car, but when they start out with enough language to figure
out what’s being asked, why, and what makes a good answer, they’re really good at
solving lots of different tasks in a way we might call “creative.”



Where they really shine is when given more specific guidance and training, then
properly applied to specific problems. When this happens, they can be very powerful
given how much they already “know.” What an LLM knows is an important fundamental
concept for our more specific use-case of using an LLM to provide more relevant
results in search, so we’ll come back to this concept. But for the moment let's keep
building.
Data
Text Images Speech Structured Data 3D S ig n a ls

Training

Foundation

Model

Adaptation

Object
Question
Sentiment
Information
Image
Instruction

Recognition Answering Analysis Extraction Captioning Following

Tasks
A visualization of how foundational, pre-trained models receive broad training to learn general concepts like
language in order to adapt to future tasks- Source 

We now have an understanding of how LLMs are so powerful for basic tasks that just rely
on a language and generally available knowledge. What about complex tasks, that require
more specific knowledge, that we might not even consider language?

Use of LLMs in the software development process

One way LLMs are being used now that might
perhaps surprise some people – though maybe
not to those in technical roles reading this – is Write code

based on

that LLMs are increasingly being used to write what they

know (now)
code. Machines building machines? Sounds
apocalyptic to most, until they consider how
and why this is happening. 

Find

something
Tap the more

A common loop for more junior programmers doesn’t work

quite as

senior

programmer

(and even very experienced ones) at the turn of expected behind them

the millenia might be to: 

Code replacing code 12

In essence they would then pose a query, or provide a prompt to their fellow human
roughly in the format of “why does this do that?” 

For the last two decades or so, rather than asking a more senior programmer this question,
they would essentially ask the question of Stack Overflow (or a search engine that points
them to some coding forum similar to Stack Overflow). That’s been going on for so long
that there are already a lot of questions asked and answered in every half-decently popular
coding language. So much of the basics and even not-so basics have been covered by
these open-source forums that our junior programmer can almost certainly find the
guidance they need to then apply a previous solution to their own specific problem. Of
course, in the 2010s that required a programmer to search their problem as best they
could, in general terms, find something similar to their problem, or someone who
experienced the same error with a slightly different set-up, then try to use a solution to
another person’s problem on their own. It’s still a bit like asking another programmer “hey,
have you seen this error” and getting a response like, “yes, but mine was when I….” 

A stack overflow question-answer pair. Note the search for a general topic, the question that is hopefully
what the user is looking for, and the answer that will not be specific to the searcher’s code

The difference between asking a question of your manager and the person on Stack
Overflow is that (a) you get to ask follow ups to your manager in real time, and (b) your
manager probably has more specific domain knowledge on what you’re trying to do, either
specific to the business case or specific to the code base, or both!

Code replacing code 13

So, in short, when people ask a site like Stack Overflow about coding, they’re getting the
pieces of information they need to fix their problem (probably), but they need to do all the
mental leg work to assemble those pieces into a context-specific, workable solution. 

Write code

based on what

they know

(now)

Find something
Search stack

doesn’t work
overflow for

quite as
something

expected similar

That’s been the case in the last ten years. How is that changing now? Most programmers
don’t need to find a problem like theirs and map it to their specific code anymore. Large
language models are trained on datasets like Stack Overflow (where a programmer asks a
question and another programmer answers it). In essence, now that LLMs “know” Python,
or almost any other coding language with a lot of users, about as well as they “know” any
other languages that are part of their very general training set, junior programmers – or
time-and-resource-strapped senior devs – get the best of both worlds. They get huge
amounts of information being processed to form the foundation of the LLM’s knowledge,
and they get a specific response to their question, already mapped to the problem they are
having. Plus, they get to ask follow up questions.

Of course there are limitations to this process where senior devs, creative coders, and
managers with deep programming and domain-specific knowledge are still essential to the
development of both a product and the development of their junior devs. To understand
where the limitations might be, let's back up a bit to understand how LLMs are capable of
providing such responses in the first place. Again, if you’re an LLM guru feel free to skip the
next few paragraphs.

Here’s how LLMs went from pretty basic prompts to the crazy powerful tools they are
today. Many large language models, specifically OpenAI’s ChatGPT, started out, essentially,
as question-answer machines. They still basically are, but they’re really good at answering
questions. The “chat” part of ChatGPT, to be extremely reductive, is designed to provide
an answer to a user’s question. 

To do this, it's trained on lots of question-answer pairs. “What color is the sky? Blue,” is one
single string that’s used to train something like ChatGPT. Train it on enough of those
questions and you can split your datasets on question marks and start asking it to predict
the next word when only given the question.

Code replacing code 14

Early versions of this were basically predictive text machines, in the same way that when a
user is typing on their iphone, predictive text can guess that the next letter in the string
“Good Mornin_” is going to be “g,” or the next few letters in the text “How are…” is going to
be “y,” “o,” and finally, “u.”

The large language model is trained on enough question-answer pairs and even just
general text without a question mark in the middle, that a user no longer has to say “why is
the sky blue?” and expect an answer about nitrogen. They can ask the model to write a
paper about the electromagnetic spectrum with specific mentions of gases in the
atmosphere. This is possible by predicting the next few thousand words after the prompt.
It doesn’t matter that the simple question-answer pair is overbalanced toward a long
answer from a relatively short question. The difference between “What color is the sky?
Blue,” and “Write me a paper about the electromagnetic spectrum (a few thousand word
paper ensues),” is just a matter of scale. Keep a pin in this idea of lots of information
coming from a relatively short query, because it gets important when we get to our code.

If you’re a developer, what is the point of the LLM having all that fundamental knowledge?
It means what used to be difficult or time consuming is now a little easier. So, back to our
junior programmer using an LLM to write code. Well, the LLM is trained in the same format
as: here is a series of question-answer pairs that just happen to be code, now start
predicting what the next few hundred characters will be given a newer question on coding.
Voilà, junior coders can now use LLMs to generate code beyond their years, and several
major players in the space have made it their niche use-case to do exactly that (Microsoft
Copilot touts this ability and China’s DeepSeek has gained traction in the market thanks to
its more purpose-built coding LLM). 

A (fairly simple) request to an LLM for python code solving a specific problem incorporating the
information specific to the user’s problem (filename, column name, string to search for)

Code replacing code 15

Of course, you can imagine that there are some very experienced and/or very elegant
programmers who are doing things in ways that users on Stack Overflow (and thus the
models trained on Stack Overflow) have never even considered. But you can also imagine,
that now, junior programmers who are still learning some of the concepts – or busy senior
programmers who want to save themselves some keystrokes on something relatively
simple for them – can work with an LLM to save themselves time and effort on a first pass
at generating any code short of genius, inspired, or cutting-edge.

The job – and the value of a computer science degree, in this case – becomes knowing
when the human didn’t ask their question properly, the LLM misunderstood, or the code is
just plain wrong, for a variety of other reasons. In short, the onus shifts from generating the
code and using some other tool to troubleshoot it, to letting some other tool generate it
and then have the human troubleshoot it. Maybe the model just made up some stuff that
looks like it could be code, but it isn’t workable. A human is still in the loop, just doing less
of the legwork, because it starts with the external source!

Prompt an

LLM to

write code

for you

Evaluate
Tweak the

the code for

code or tweak

correctness the prompt

One other possibility for the reason why the LLM could provide an incorrect response – for
a coding prompt or otherwise – is that the LLM doesn’t have enough information, or
context, to provide a complete answer.

That’s where this concept really starts to take off, and that’s also where we begin to merge
back into the primary purpose of this article, which is to get LLMs to judge query intent,
provide more relevant responses, and stand up smarter search experiences.

Consider this use case for building tax compliance software. Let's say you’re building a
TurboTax competitor. Long-form English legalese that outlines tax codes will take a long
time to read, and you’d probably need a product manager who knows both the tax code
and computer code very well to translate them from tax laws into programming
specifications. Then your programmers would need to take those concepts in the spec and
actually code them. If an LLM can both read long prompts now (with context windows
growing multiple times over in the last few years) and the LLM can write code, this
development process simply becomes a well crafted prompt and some quality assurance
work. 

Code replacing code 16

You still need those product managers and those engineers to make sure the LLM didn’t
miss anything vital. You still need the “human in the loop” because your product probably
won’t survive if your computer code has missed certain tax codes that were “lost in the
middle” or has hallucinated others, but a lot of work has just been removed from the
development process. And less work means faster, possibly also cheaper products, and a
quicker road to monetization.

The big takeaway here is that you can currently give an LLM a set of
human-readable rules, and let it write the code for you, because it
knows human languages and it knows coding languages.

Two big concepts arise from this (outside of development being cheaper, faster, and more
profitable with conceivably no drop-off in quality, provided qualified humans are involved in
the QA process). First, the speed at which new rules can be written means that programs
can be much more dynamic, flexible, and specific. If you’re standing up search experiences
for a plethora of industries, this means you can write code for more personalized rules by
industry. Softwaredoug’s blog says that customers are holding their software partners to a
higher standard of personalization and performance as a result. It reads:

“If you sell customers a full solution, they will expect their subjective, unconscious, and
domain specific definition of success. Customers assume you read their mind, projecting
their definition of “good search” onto your product. Multiply that by every stakeholder in
that company. Multiply that by every domain—medical search is completely dissimilar from
ecommerce. Even in ecommerce, book search differs from fashion search. Passive
discovery for consumers differs from B2B. Book ecommerce for librarians differs from
search for sci-fi nerds. And on and on.”

According to softwaredoug, search is hard and there are lots of industries that need lots of
very specific experiences.

The second big concept to arise from using LLMs as first-draft software engineers, is the
shift from a focus on the rules themselves, the lines of code, to a focus on the outcome
those rules and that code were meant to generate. If you're familiar with machine learning
concepts, you could say that this shift means that programming is becoming less of an
exercise in syntax and more of an exercise in training, specifically re-training – or training
further – the right large language models with the right data (possibly your entire existing
codebase) and then letting it come up with the rules and the code for those rules that
provide the most successful recommendations.

Let’s take a pause and try to understand how fundamental a shift this is in the way of
producing programs. While before LLMs, devs were essentially writing rules that product
managers thought would result in the right outcome (hopefully with lots of domain
knowledge, stakeholder meetings, etc.), and now devs are guiding AI to write those rules
with a focus on how well the program performs its core function.

Code replacing code 17

Every program basically becomes a performance-based machine learning model, with
engineers not spending their time writing programs but fine tuning them based on their key
performance indicators and domain knowledge.

To make this a bit more real, pretend you’re some little company that starts with an A,
named after a South American river, that has the bright idea of selling books on a website,
with hopes of growing into not just an online marketplace but the online marketplace. You
could build your recommendation engine statically. Which is to say you could hard-code in
holiday book recommendations every December. If you’re starting this company up during
the dot com boom, that’s probably pretty impressive. Then some future product manager
could come up with the concept to promote new releases for up-and-coming authors, then
kick off a project where the engineers write new code to account for the release date of an
author’s book. Slowly you add “buy again” options, “continue shopping for…”
recommendation models, “customers also bought…” and so on until an in-house
recommendation engine forms.

Or, if you're building your company today, your website could focus on the outcome, which
is selling books, and dynamically writing code to figure out what rules work best to achieve
that end. Wouldn’t it be better if your website could adapt to say, the unexpected explosion
of something called “romantasy,” which leads to the recurring but acyclical resurgence of
infatuation with dragons, and then recommend other books that perhaps speak to different
audiences who might be interested in some of that but not all components? This is to say
that dynamic rules for recommendation could figure out that some readers aren’t
interested in the romance part of romantasy but the dragons, and might be interested in
going back and reading George R.R. Martin. All of this could happen without a product
manager having to come up with that chain of logic, and quite possibly, no new code
having to be written. All it requires is the right context, and the right technology.

How do we make the most of this?

To answer the question, “how do we capitalize on this trend,” let’s start by looking at the
converse case. Let’s look at a modern failure of a recommendation engine. Well, a true-life
horror-story, really. Consider a user who likes to read sci-fi and fantasy, has hundreds of
books completed on Goodreads, providing numerical/star ratings for many of them, if not
text-based reviews. None of the highly rated books are romantasy. This user has read
romantasy, including that one about dragons and did not rate it highly. That user is looking
for their next read, and rather than shooting a text to their friends for a recommendation,
decides to allow Goodreads to influence a potential future purchase. Highly motivated
reader and purchaser, with the likelihood of buying hundreds if not thousands of books over
their lifetime, giving a recommendation a chance to influence that few-thousand dollars of
customer lifetime value.

Goodreads has a category that says “picked for you,” and guess what the number one
recommendation is for this user? Yup. That romantasy about dragons. The user has already
read this (purchasing history ignored). The user has already rated this (rating ignored).

Code replacing code 18

The user has linked their facebook account to their goodreads account, and, if you want to
generalize, this person is not in that book’s core demographic (personal information
ignored). Even if the recommendation engine were among the simplest discussed in this
paper, many other instances of this user not being interested in romantasy exist
(categories and parameters ignored). 

“Ok, bad personalization,” says the user, then scrolls down to see what other carousels on
Goodreads might have for them. Do you want to guess what the very next carousel is on
the page? “Top reads on Goodreads.” It's the same list as “Picked for You!” User scrolls up
scrolls back scrolls up scrolls back. Ninety-five percent of these identical lists are
populated by romantasy or feminist mythological retellings. That user is never again going
to trust Goodreads to influence their purchasing behavior again, because not only did it fail
to make the path-to-purchase shorter and smoother, but it was actively deceptive about
its ability to provide a personalized recommendation. 

We’re using this example not to pick on Goodreads, but to illustrate how best to
incorporate recommendation models by converse example. So how can we make it better?
The most obvious improvement is in the design. If the recommendation models are the
same with different titles, don’t put them next to one another. If they have habitually very
similar results, space them out. Next, it comes down to model selection. Having a “Picked
for You” section should never have a “Top Products” model underneath the hood. 

Beyond those two glaring issues, what could be done? As suggested, the user has
provided a treasure trove of data about their personal preferences and demographic
information. Goodreads has been actively providing good data. Put all that incredible
context to use in a recommendation engine! The pitfall to avoid here is that the data must
be good. For instance, that user may purchase a lot of other books on their Amazon
account – books they tried and put down, books as gifts for others, purchases made by a
spouse. Purchase history for this user may not be reliable, and if the data is flawed so is
the recommendation. 

So to summarize how we make the most of recommendations, that’s: good data, with the
right models, in the right places. Sounds easy, right? Now lets kick it into overdrive by
applying LLMs to an already successful recommendation framework.

Code replacing code 19

Providing the right context to LLMs for more relevant

responses

So far in this article, we’ve discussed the broad strategies used for recommender systems,
more specific models commonly employed in the ecommerce space, with some examples
of do’s and don’t’s. We’ve also talked about how developers are starting to use LLMs
creatively in the product development lifecycle. Now let's bring those concepts together in
a very specific example so we can look at some code doing next-level things with LLMs:
discerning query intent to provide better responses.

One strategy that’s been gaining a lot of popularity, and for good reason is RAG. This
Algolia article has this to say about RAG:

“Retrieval-augmented generation is a clever way to avoid the hallucinations and

inaccuracies of general-use LLMs by restricting content generation to your specific corpus
of data. But even then, there are still challenges around testing those responses.”

RAG basically provides context to a query so that the querier doesn’t have to.

If you sell consumer electronics products, you want your product search bar or FAQs to
understand that “apple” isn’t a fruit, but a brand. How does RAG do this? It makes the
query from your user a super-query, containing all the context your RAG pipeline is made
to collect and include.

Before we jump into any code, let's make another real world example of how important
context is to a query. Let’s take the scenario of asking someone how they feel. Now, if you
were to ask someone how they feel at the finish line of a marathon, you could say, “Given
that you’ve just run a marathon, your time was four hours and fifty minutes, this is your
first marathon, and I am your friend who has run zero marathons, thus thinks its very
impressive that you’ve merely survived irrespective of your completed time… how do you
feel?” That certainly provides some context to the situation and provides some guardrails
to the conversation. But humans are naturally retrieving all of that context on their own,
without having to be provided with it. So you can simply ask, “how do you feel?” and
receive the expected response of “Ugh, glad that’s over.”

Who is
Who is being
Relationship

asking asked between asker

and asked
Question: How are you?
Runner’s
Past marathons
Past marathons

marathon time run by runner run by asker

Context

Generic Bad Answer Good Context-ful

Answer

A visualization of a basic “context-ful” human response

Code replacing code 20

Now, that same response to that same question in a therapist's office would not go over
well. Given the context of the therapist's office, your response to “how do you feel?” is
expected to be both more verbose and highly specific to the question’s intent.

Let's put that in the context of a site search scenario and then into code. A user makes a
simple search, like “gaming computers” on your website, but to provide a better response
to the user, you want that query augmented by retrieved information, perhaps product
description and ratings on your site, perhaps consumer reports, blogs, or written reviews
ranking the best computers for gaming, perhaps specs that are associated with high-
performance graphics cards or perhaps features typically associated with gamers but not
actually relevant to gaming, like clear side-panels and colorful lights. These types of
guardrails to the query prevent your search response from wasting space in your response
or recommendation page and wasting your potential buyer’s time by putting a Macbook Air
– which technically could be used for “gaming,” but fails in fulfilling the underlying
expectations of the query – in front of the user. So the context augmenting the query
could improve the results of just that simple two word query on its own.

Audrey Lorberfeld in the presentation cited below sums it up perfectly: 

“[RAG] bridges the gap between what users explicitly ask and what they intend to ask.”

Here’s a basic look at the fundamentals of building out a RAG pipeline. The general goal is
to turn a regular old query (what a user explicitly asks) into a Retrieval Augmented
Generated query (what a user intended to ask): a RAG query. After importing necessary
Python libraries and setting up an OpenAI API key, used later in this code, start by loading
a document in and chunking it for easier processing.

loader = TextLoader(document_path)

documents = [Link]() 

text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) 

texts = text_splitter.split_documents(documents)

Use something like OpenAI’s Embeddings to convert text to model-readable vectors. These
vectors represent the semantic meaning of the text. And use something like Facebook AI
Similarity Search to store and retrieve those vectors for use. Read more on these concepts
here. 

embeddings = OpenAIEmbeddings()

db = FAISS.from_documents(texts, embeddings)

Code replacing code 21

Then create a prompt template that has placeholders for the query that was actually input
and all the context that query might need to become supercharged. In some of our above
examples the “context” was common sense (like asking someone how they feel after a
marathon vs. in a therapist’s office), but when implementing these for search experiences,
domain knowledge becomes critical, and you need real expertise to decide what context is
necessary to pass for optimal results (e.g., product metadata, user behavior data, etc.).

template = """

Use the following pieces of context to answer the question at the end. If you don't know the
answer, just say you don't know, don't try to make up an answer. 

{context} 

Question: {question} 

Answer: """ 

prompt = PromptTemplate(template=template, input_variables=["context", "question"])

Choose your LLM and create an LLMChain, or the pipeline for the query.

llm = OpenAI() 

chain = LLMChain(llm=llm, prompt=prompt)

Now combine the LLM, prompt template, and vector store in one definition that queries
your RAG pipeline.

def rag_query(query): 

docs = db.similarity_search(query)

context = "\n".join([doc.page_content for doc in docs]) 

return [Link](context=context, question=query)

All that’s left is to take all the above code, define it as create_rag_pipeline where it
receives a document_path and returns a raq_query, and then pass it your collated
document for context.

Code replacing code 22

 document_path = "context_document.txt" 

rag_query_function = create_rag_pipeline(document_path)

Then you can start passing that raq_query_function some example queries and see the
answers the rag query comes up with.

query1 = "gaming computer" 

query2 = "fantasy books" 

query3 = "what other clothes might go well with this outfit?" 

print(f"Query 1: {query1}")

print(f"Answer 1: {rag_query_function(query1)}")

print(f"\nQuery 2: {query2}")

print(f"Answer 2: {rag_query_function(query2)}")

print(f"\nQuery 3: {query3}")

print(f"Answer 3: {rag_query_function(query3)}")

So RAGs can be relatively simple to stand up, but really powerful if done properly. In case
you missed it in the example queries, they can be used to take those super simple queries
for gaming computers or fantasy books and turn them into much more context-full queries
with any information you can jam into a document, like reading history or graphics card
specs.

If you’re trying to imagine how those print statements will play out for another example,
think back to the original scenario, where you ask somehow “how are you?” after they ran a
marathon and then visualize a context document that has all the extra information about:
who’s asking, who is being asked, what their relationship is, what the runner’s marathon
time was and any past marathons run by either of the two conversants. The query would
be relatively simple, but by virtue of the RAG query, the answer would be relevant with all
of that context taken into account.

Code replacing code 23

 Who is
Who is being
Relationship

asking asked between asker

and asked
Question: How are you?
Runner’s
Past marathons
Past marathons

marathon time run by runner run by asker

Context Document

LLM

RAG Query

( Question+Context)

Generic Bad Answer Good Context-ful

Answer

A conceptual visualization of a RAG query

Here’s where it all comes together conceptually. That question “How are you?” alone lacks
intent. But RAG takes the context into account and uses an LLM to provide true intent to
the query, thus the response, or answer to the question, may be far superior to one that
lacks the context. RAG or other information retrieval methodologies can turn simple site
searches into really powerful results that understand the intent beneath the search and
provide better results. But they also could not.

Of course, you could hard-code all of the pathways for providing context to the query, but
that would make you more like Pandora and less like the music platform of the future.

If you want to go deeper down the information-retrieval rabbit-hole and understand some
more ways LLMs can be super-charged for various tasks in various ways, Audrey
Lorberfeld’s Women of Search presentation at Haystack US 2024 is a great place to start.

A breakdown of one specific implementation of RAG, from Haystack 2024.

Code replacing code 24

What this means for shoppers
We started at a high, conceptual level and have brought it all the way down into the
proverbial weeds. Let’s take one more step back and ask what this means for the end user.
After all, a focus on the technology is awesome, an understanding of your customer’s use
case to make them happy is great, too, but ecommerce often comes down to what this
technology is doing to your customer’s experience.

If recommender systems did great things for average order value but
customers were annoyed by “pushiness” in the checkout process,
they might not come back.

Optimizing for one short-term metric might mean pushing a customer away, and you could
sacrifice another, perhaps much more important long-term, metric like customer lifetime
value on the altar of AOV.

This is the fear, but it's also a pitfall easily avoided by experienced implementation
professionals, who have the right technology at their disposal and have mastered the art
of fine-tuning experiences, whether they be checkout, search, or overall shopping
journeys. As in our Goodreads example, at least one user was very annoyed by the
recommender system and likely won’t be allowing that app to weigh in on future
purchases. They may even stop using the app altogether. But there were some obvious
and some not so obvious ways to turn this horrendous user experience into a good one,
where the app plays an integral part in customers’ book-shopping journey for years to
come.

Returning to our more positive example of purchasing a gaming computer, the right RAG-
superpowered search results based on a limited-context query saves a customer
keystrokes and an annoying re-querying loop. Then, the right recommendation models in
the right place helps them compare models without either needing to cycle between a
dozen tabs or giving them decision paralysis when they’ve made up their minds. Another
well placed recommendation model can help them complete their new gaming set-up,
then speed them along through checkout without any blockers. What used to be a painful
few hours of research becomes all about how excited that customer is to get their hand on
their shiny new rig.

Less effort. Less anxiety. More speed and relevant results. No annoyance with the brand
or the shopping experience. Boom. Happy shopper and quite possibly a customer for life.
That’s good for everyone involved because, at the rate gaming PCs are advancing, those
specs might be obsolete in a few years. You can bet that the next few thousand dollars
that customer drops is not going to be in a place with a painful experience.

Code replacing code 25

Conclusion
So, to sum it all up, customers can have both better shopping experiences when
developers are using some of these cutting edge technologies in tandem with extensive
experience in recommender systems. That means devs can spend more time thinking
about how to make their products better and less time just making their products. But,
just like the pros and cons of certain types of recommender systems or the pitfalls of
using recommendation models in the wrong place at the wrong time (as in our TurboTax
competitor example where we kept the human in the loop so our LLM doesn’t miss or
make up tax codes) optimism toward any new technology must be tempered by a deep
understanding of the technologies being used, incorporating extensive knowledge of
how to identify and mitigate its most common drawbacks. Engaging the right partners
that employ people who are passionate about these technologies is a great way to
prepare your product for all the ecommerce advancements to come.

For context, while the above hopefully demonstrates how LLMs are changing the
development cycles plus what RAG is, why it's important, and the basic components in
its code, Algolia products are incorporating RAG in a much more robust and serious way.
Some concerns not addressed in this whitepaper are generating and managing
responses, scaling efficiently, writing efficient prompts, resiliency and error handling,
using APIs to serve up all that important context to the RAG query, and, of course,
adversarial testing. Part of RAG implementation on an enterprise level should include
asking questions like: are we set up to protect against prompt extraction attacks? Data
extraction attacks? Cache injections? Denial-of-service of the underlying search?

At Algolia we think a lot about all of these things so that you don’t have to, and if you are
looking to benefit from a modern RAG API on top of your search experience, sign up to
be in Algolia’s GenAI toolkit beta group today.

Code replacing code 26