Prompt Engineering Implementation Partner
Using LLM Systems in Practice: Models,
Platforms and Settings
Dr. Satya Jayadev P
Principal Data Scientist at Gyan Data
Guest Faculty at IIT Madras
Recap and Roadmap
Quick Recap: What you’ll Learn now:
● Context helps the model interpret the ● Ways to access and use LLMs in
task and generate more relevant practice
responses ● Different types of LLMs
● Attention helps the model focus on the ● Model settings that affect output
most relevant parts of the input behaviour
● Transformers help language models
process relationships across text
● Context window limits how much total
text the model can handle at once
From understanding LLMs to using LLM systems
● In practice, users interact not only with a model, but with an LLM system
● An LLM system usually includes:
○ a model that generates responses
○ an interface through which the user or software accesses the model
○ settings that control response behavior
○ Supporting platform-level features such as file upload, web search, memory,
tools, or integrations
● Key idea: Prompt engineering is not only about writing prompts; it is about
using the right model, interface, settings, and platform features for the task
Ways to Access LLMs in Practice
Chat interface: Direct interaction with LLMs through an conversational UI
o Best for direct human interaction
o Useful for learning, drafting, summarizing, brainstorming, reviewing documents, and
exploratory work
o Usually gives less direct control over low-level settings
API: Interacting with LLMs by calling a hosted API
○ Best for integrating LLMs into software workflows
○ Useful for automation, repeated tasks, products, dashboards, agents, and backend
systems
○ Usually gives more explicit control over model choice, temperature, max tokens, output
format, and tool calls
Ways to Access LLMs in Practice
IDE assistants: Interacting with LLMs within a coding environment
o Best for coding support inside development environments
o Useful for code completion, debugging, refactoring, documentation, and test
generation
Same Model but Different Usage Contexts
● The same underlying LLM capability can appear differently depending on the access
mode.
● For example:
○ In a chat interface, the user may simply ask for a summary.
○ Through an API, the same task can be automated for thousands of documents.
○ In an IDE assistant, the same model capability may be used to explain or modify
code.
○ In agentic systems, the model may call tools, search information, or trigger
workflows.
● Key takeaway: Before choosing model or its settings, first understand the usage
context
Why Model Choice Matter
● Different models can vary in:
○ response quality – how accurate, coherent, or helpful the output tends to be
○ speed – how quickly the model produces a response
○ cost – how expensive it is to use the model, especially at scale
○ context window – how much text the model can handle at once
○ capabilities – such as coding support, reasoning strength, or multimodal
support
● Key Takeaway: Choosing a model is not only about getting the “best” one, but
about choosing one that fits the task and practical constraints.
● Note: In many modern chat interfaces, the model is often selected automatically
in the background unless the user specifically chooses a model
Broad types of Models
● General-purpose chat models – suitable for a wide range of everyday text
tasks
● Lightweight or fast models – useful when low cost or quick response matters
● Code-capable models – useful for programming-related support tasks
● Multimodal models – can handle not only text, but also inputs such as images
and/or audio
● Open-weight models – give more flexibility for self-hosting and customization
● Hosted proprietary models – accessed through a provider's platform or API
and often easier to start with
What Happens during Inference?
● Inference: Inference is the stage where the model generates a response from
the prompt.
● At inference time:
1. the model reads the input
2. predicts likely next tokens step by step
3. uses settings to control how deterministic, creative, or long the output
should be
4. returns the generated response through the chat interface, API, or IDE
assistant
Model Settings for Inference
● Inference Settings:
○ Once a model is chosen, we can control how it behaves through these
settings
○ These settings do not change what the model has learned
○ Instead, they influence how the model generates its response at runtime
● Some important settings are:
○ temperature
○ top-p
○ max tokens
● Notes:
○ In most chat interfaces, these settings are internally controlled by the
system and not exposed to the users
○ They are commonly available while using APIs
Temperature Setting
● Temperature controls how conservative or creative the text generation is
● Lower temperature usually gives:
○ more focused output
○ more repeatable output
● Higher temperature usually gives:
○ more variety
○ more creative or diverse responses
● Potential Temperature Settings:
○ Use low temperature for extraction, classification, or short factual
summaries
○ Use medium temperature for explanations and drafting
○ Use higher temperature for brainstorming and ideation
Top - p and Max Tokens
Top-p
● Top-p controls diversity by limiting selection to a probable set of next tokens
o Lower top-p gives narrower choices
o Higher top-p allows more variety
● It can be thought of as controlling how wide the model looks among plausible
next-word options
Max tokens
● Max tokens controls the maximum length of the output
o Too low may cut off useful responses
o Too high may increase cost and verbosity
● Note: This can be controlled through the prompt as well by mentioning the limit of
number of words in the output
Key Takeaways & What’s Next
Key Takeaways
● Using LLMs in practice involves more than just writing a prompt
● Different models vary in quality, speed, cost, context length, and capability
● Inference settings such as temperature, top-p, and max tokens affect output
behaviour
● LLMs can be accessed through chat interfaces, APIs, and IDE assistants
Up-Next in Module 2
● What makes a prompt work in practice
● Different prompting techniques
● Role, audience, and tone control
● Structuring prompts with RICE
● Getting usable output in bullets, tables, JSON, and other formats
Thank you