The transition of Generative AI powered products from proof-of-concept to
production has proven to be a significant challenge for software engineers
everywhere. We believe that a lot of these difficulties come from folks thinking
that these products are merely extensions to traditional transactional or
analytical systems. In our engagements with this technology we’ve found that
they introduce a whole new range of problems, including hallucination,
unbounded data access and non-determinism.
We’ve observed our teams follow some regular patterns to deal with these
problems. This article is our effort to capture these. This is early days
for these systems, we are learning new things with every phase of the moon,
and new tools flood our radar. As with any
pattern, none of these are gold standards that should be used in all
circumstances. The notes on when to use it are often more important than the
description of how it works.
In this article we describe the patterns briefly, interspersed with
narrative text to better explain context and interconnections. We’ve
identified the pattern sections with the “✣” dingbat. Any section that
describes a pattern has the title surrounded by a single ✣. The pattern
description ends with “✣ ✣ ✣”
These patterns are our attempt to understand what we have seen in our
engagements. There’s a lot of research and tutorial writing on these systems
out there, and some decent books are beginning to appear to act as general
education on these systems and how to use them. This article is not an
attempt to be such a general education, rather it’s trying to organize the
experience that our colleagues have had using these systems in the field. As
such there will be gaps where we haven’t tried some things, or we’ve tried
them, but not enough to discern any useful pattern. As we work further we
intend to revise and expand this material, as we extend this article we’ll
send updates to our usual feeds.
| Direct Prompting | Send prompts directly from the user to a Foundation LLM |
| Evals | Evaluate the responses of an LLM in the context of a specific task |
Direct Prompting
Send prompts directly from the user to a Foundation LLM
The most basic approach to using an LLM is to connect an off-the-shelf
LLM directly to a user, allowing the user to type prompts to the LLM and
receive responses without any intermediate steps. This is the kind of
experience that LLM vendors may offer directly.
When to use it
While this is useful in many contexts, and its usage triggered the wide
excitement about using LLMs, it has some significant shortcomings.
The first problem is that the LLM is constrained by the data it
was trained on. This means that the LLM will not know anything that has
happened since it was trained. It also means that the LLM will be unaware
of specific information that’s outside of its training set. Indeed even if
it’s within the training set, it’s still unaware of the context that’s
operating in, which should make it prioritize some parts of its knowledge
base that’s more relevant to this context.
As well as knowledge base limitations, there are also concerns about
how the LLM will behave, particularly when faced with malicious prompts.
Can it be tricked to divulging confidential information, or to giving
misleading replies that can cause problems for the organization hosting
the LLM. LLMs have a habit of showing confidence even when their
knowledge is weak, and freely making up plausible but nonsensical
answers. While this can be amusing, it becomes a serious liability if the
LLM is acting as a spoke-bot for an organization.
Direct Prompting is a powerful tool, but one that often
cannot be used alone. We’ve found that for our clients to use LLMs in
practice, they need additional measures to deal with the limitations and
problems that Direct Prompting alone brings with it.
The first step we need to take is to figure out how good the results of
an LLM really are. In our regular software development work we’ve learned
the value of putting a strong emphasis on testing, checking that our systems
reliably behave the way we intend them to. When evolving our practices to
work with Gen AI, we’ve found it’s crucial to establish a systematic
approach for evaluating the effectiveness of a model’s responses. This
ensures that any enhancements—whether structural or contextual—are truly
improving the model’s performance and aligning with the intended goals. In
the world of gen-ai, this leads to…
Evals
Evaluate the responses of an LLM in the context of a specific
task
Whenever we build a software system, we need to ensure that it behaves
in a way that matches our intentions. With traditional systems, we do this primarily
through testing. We provided a thoughtfully selected sample of input, and
verified that the system responds in the way we expect.
With LLM-based systems, we encounter a system that no longer behaves
deterministically. Such a system will provide different outputs to the same
inputs on repeated requests. This doesn’t mean we cannot examine its
behavior to ensure it matches our intentions, but it does mean we have to
think about it differently.
The Gen-AI examines behavior through “evaluations”, usually shortened
to “evals”. Although it is possible to evaluate the model on individual output,
it is more common to assess its behavior across a range of scenarios.
This approach ensures that all anticipated situations are addressed and the
model’s outputs meet the desired standards.
Scoring and Judging
Necessary arguments are fed through a scorer, which is a component or
function that assigns numerical scores to generated outputs, reflecting
evaluation metrics like relevance, coherence, factuality, or semantic
similarity between the model’s output and the expected answer.
Model Input
Model Output
Expected Output
Retrieval context from RAG
Metrics to evaluate
(accuracy, relevance…)
Performance Score
Ranking of Results
Additional Feedback
Different evaluation techniques exist based on who computes the score,
raising the question: who, ultimately, will act as the judge?
- Self evaluation: Self-evaluation lets LLMs self-assess and enhance
their own responses. Although some LLMs can do this better than others, there
is a critical risk with this approach. If the model’s internal self-assessment
process is flawed, it may produce outputs that appear more confident or refined
than they truly are, leading to reinforcement of errors or biases in subsequent
evaluations. While self-evaluation exists as a technique, we strongly recommend
exploring other strategies. - LLM as a judge: The output of the LLM is evaluated by scoring it with
another model, which can either be a more capable LLM or a specialized
Small Language Model (SLM). While this approach involves evaluating with
an LLM, using a different LLM helps address some of the issues of self-evaluation.
Since the likelihood of both models sharing the same errors or biases is low,
this technique has become a popular choice for automating the evaluation process. - Human evaluation: Vibe checking is a technique to evaluate if
the LLM responses match the desired tone, style, and intent. It is an
informal way to assess if the model “gets it” and responds in a way that
feels right for the situation. In this technique, humans manually write
prompts and evaluate the responses. While challenging to scale, it’s the
most effective method for checking qualitative elements that automated
methods typically miss.
In our experience,
combining LLM as a judge with human evaluation works better for
gaining an overall sense of how LLM is performing on key aspects of your
Gen AI product. This combination enhances the evaluation process by leveraging
both automated judgment and human insight, ensuring a more comprehensive
understanding of LLM performance.
Example
Here is how we can use DeepEval to test the
relevancy of LLM responses from our nutrition app
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
def test_answer_relevancy():
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)
test_case = LLMTestCase(
input="What is the recommended daily protein intake for adults?",
actual_output="The recommended daily protein intake for adults is 0.8 grams per kilogram of body weight.",
retrieval_context=["""Protein is an essential macronutrient that plays crucial roles in building and
repairing tissues.Good sources include lean meats, fish, eggs, and legumes. The recommended
daily allowance (RDA) for protein is 0.8 grams per kilogram of body weight for adults.
Athletes and active individuals may need more, ranging from 1.2 to 2.0
grams per kilogram of body weight."""]
)
assert_test(test_case, [answer_relevancy_metric])
In this test, we evaluate the LLM response by embedding it directly and
measuring its relevance score. We can also consider adding integration tests
that generate live LLM outputs and measure it across a number of pre-defined metrics.
Running the Evals
As with testing, we run evals as part of the build pipeline for a
Gen-AI system. Unlike tests, they aren’t simple binary pass/fail results,
instead we have to set thresholds, together with checks to ensure
performance doesn’t decline. In many ways we treat evals similarly to how
we work with performance testing.
Our use of evals isn’t confined to pre-deployment. A live gen-AI system
may change its performance while in production. So we need to carry out
regular evaluations of the deployed production system, again looking for
any decline in our scores.
Evaluations can be used against the whole system, and against any
components that have an LLM. Guardrails and Query Rewriting contain logically distinct LLMs, and can be evaluated
individually, as well as part of the total request flow.
Evals and Benchmarking
Benchmarking is the process of establishing a baseline for comparing the
output of LLMs for a well defined set of tasks. In benchmarking, the goal is
to minimize variability as much as possible. This is achieved by using
standardized datasets, clearly defined tasks, and established metrics to
consistently track model performance over time. So when a new version of the
model is released you can compare different metrics and take an informed
decision to upgrade or stay with the current version.
LLM creators typically handle benchmarking to assess overall model quality.
As a Gen AI product owner, we can use these benchmarks to gauge how
well the model performs in general. However, to determine if it’s suitable
for our specific problem, we need to perform targeted evaluations.
Unlike generic benchmarking, evals are used to measure the output of LLM
for our specific task. There is no industry established dataset for evals,
we have to create one that best suits our use case.
When to use it
Assessing the accuracy and value of any software system is important,
we don’t want users to make bad decisions based on our software’s
behavior. The difficult part of using evals lies in fact that it is still
early days in our understanding of what mechanisms are best for scoring
and judging. Despite this, we see evals as crucial to using LLM-based
systems outside of situations where we can be comfortable that users treat
the LLM-system with a healthy amount of skepticism.
Evals provide a vital mechanism to consider the broad behavior
of a generative AI powered system. We now need to turn to looking at how to
structure that behavior. Before we can go there, however, we need to
understand an important foundation for generative, and other AI based,
systems: how they work with the vast amounts of data that they are trained
on, and manipulate to determine their output.