commerceclarity.com/blog/eval-framework-retail.mdFor humansFor AI agents
COMMERCECLARITY
Platform:
[Platform overview](/platform)
[Context](/platform/context)
[Agents](/platform/agents)
[Governance](/platform/governance)
[Grid View](/platform/gridview)
Solutions:
[Catalog Foundation](/solutions/catalog-foundation)
[Channels & Markets](/solutions/channels-markets)
[Search & Discovery](/solutions/search-discovery)
[SEO & GEO](/solutions/seo-geo)
[Compliance & Quality](/solutions/compliance-quality)
[Merchandising](/solutions/merchandising)
[Physical Retail](/solutions/physical-retail)
[Agentic Commerce](/solutions/agentic-commerce)
Company:
[About](/about)
[Careers](/careers)
[Blog](/blog)
[Events](/events)
[Press](/press)
[Trust Center](/trust-center)
Talk to us:
[Book a demo](/book-a-demo)
---
# How we built an eval framework for retail
> Writing product content is now cheap; knowing whether it is right is the hard part. Our eval framework turns each customer rule into a pass-or-fail check, verified by code, an LLM as a judge or a decision model, and runs it on any spreadsheet: the customer's own catalog or our output. A second check compares values with ground truth data. On STIGA's Leroy Merlin listings, the same checks scored 91.1% on what existed and 99.8% on what our workflow wrote.
When writing product content is a commodity, evaluating it is the bottleneck. This is the eval framework we built for retail, and what it showed on STIGA.
**Published at:** 2026-11-20T09:00:00Z
Now that writing effective code is a commodity, the true differentiator of a company becomes its ability to write good and comprehensive [evals](https://commerceclarity.com/blog/user-facing-ai-evals). Why? Because with evals you can measure how good your solution is, and therefore point your inference compute towards identifying and solving the right problems.
In retail that is harder than it sounds: there is no single rule for a good product page, only hundreds of small ones, about [marketplace requirements](https://commerceclarity.com/solutions/channels-markets), [legal compliance](https://commerceclarity.com/solutions/compliance-quality), brand voice, [SEO and GEO](https://commerceclarity.com/solutions/seo-geo), that change with every customer, every channel and every language.
This article is about how we built ours.
## Our mission
At CommerceClarity we meet customers where they are at. Every retailer and every brand describes its products in its own way, so our data structure has to behave like a chameleon. It is made of small primitives called [attributes](https://commerceclarity.com/platform/context) (i.e. text, rich-text, boolean, number, single select and multi-select), and we map them to whatever the client needs.
Our goal is to write those attributes as well as we can and as cheaply as we can. It does not matter what technology is behind them: agents, deterministic workflows, a single prompt. What's important is that "well" can be measured consistently many times.
## Why judging in retail is hard
Take one product title. To be good, it has to be true first: every fact in it has to come from a source, the manufacturer's sheet or the customer's own data. Then it has to pass the marketplace it is going to, follow the brand's way of writing, and be found by whoever is looking for the product.
![A product title, "Sécateur à enclume 23 mm, lame acier SK5, STIGA", in the center. Four labels point at it: true to a source (23 mm and SK5 are in the datasheet), passes the marketplace (10 to 150 characters, one category), in the brand's voice (no product type said twice), found by search and AI (the words a shopper types).](https://cdn.sanity.io/images/xd4hrbt2/production/b259dfa6e736258333bb4d5f0a2a5d87b5c6403a-2400x1350.png?auto=format)
*One title has to do four jobs at once, and each job has its own rules.*
I kept it simple with the title. The description of a pharmacy product has 35 checks on it, and which ones apply depends on what the product is. A medicine must carry fixed sentences. A supplement may never say "prevents", [because EU food law forbids claims about preventing a disease](https://eur-lex.europa.eu/eli/reg/2011/1169/oj). A cosmetic may never make [a medicine's claim](https://eur-lex.europa.eu/eli/reg/2009/1223/oj).
Nobody has these rules written in one place. They arrive spread over calls, documents and people's heads.
At CommerceClarity one person's job is to find them: the [Retail Engineer](https://commerceclarity.com/blog/meet-the-retail-engineer). They sit with the customer, discover what the customer actually requires, and pull it all together. The eval framework is built for them to transform the customer's rules into verifiable pieces of software.
Therefore a good eval framework for retail must have these three properties.
**Business friendly.** Say the customer wants titles under 150 characters. The Retail Engineer should not need a developer for that. They pick the length check, write 150 next to it, and the rule is live.
**Runs on what already exists.** A customer's current catalog or an export, and not only our own output. In general, and with rare exceptions, customers give us a long list of rules, and when we check those rules against their own catalog, its current state does not match their expectations. Without that measure there is no baseline, and no way to show a customer what good is and how far they are from it. That is why the eval has the shape of a spreadsheet: a product per row, an attribute per column, and it reads the final value in each cell, whatever wrote it.
**Flexible.** Any rule that can be stated can become a check, and a check can be plain code, an [LLM as a judge](https://arxiv.org/abs/2306.05685) or a decision model. The work is not only product pages: marketplace listings, translations, photos, the audit of a whole catalog.
## What we built
What we built at CommerceClarity is an eval framework that evaluates these attributes [end to end](https://vercel.com/blog/agent-evals). It does two things.
First, it tells us whether a value follows the customer's rules. Each rule is a check: one rule on one attribute, verified by code, by an LLM as a judge or by a decision model, with pass or fail as the answer. We run the same checks on what the customer already has and on what we write, so we can see where each one stands.
Second, it tells us whether a value is correct, because following the rules is not the same as being true. When ground truth data exists, the customer's own verified data or values we collected from the manufacturers' documents, every value we write is compared with it, one by one, using a correctness check.
## STIGA
Take [STIGA](https://commerceclarity.com/customers/stiga). They make garden machinery, and asked us to put their catalog on Leroy Merlin. Leroy Merlin has its own rules for what a listing may say: a title between 10 and 150 characters, one of 62 categories for every product, and for each attribute only the values its category accepts, 5,555 of them. STIGA adds its own, such as no promotional claims and titles that read as natural French.
Each of those rules became a check, and every listing goes through all of them. Take one title. It can be the right length, name the brand and promise nothing, and still fail, because it names the product twice in a way no French writer would.
![Two versions of one French product title go through the same four title checks: title length, brand in the title, no promotional claims, natural French. The first, "Cisaille à élaguer à enclume, 23 mm, sécateur à enclume STIGA", names the product type twice and fails the natural French check. The second, "Sécateur à enclume 23 mm, lame acier SK5, STIGA", passes all four.](https://cdn.sanity.io/images/xd4hrbt2/production/d30332e82c7814054139ae1fdba9883b7639f551-2400x1350.png?auto=format)
*Two versions of one title on the same checks: the first names the product twice, and the check says so.*
A check like this one has no formula. It can be answered by a person labelling titles, or by a simple prompted decision model. We do both: we label by hand first to calibrate the model, then the decision model runs the next iterations.
## Feedback loop: agent evals on STIGA's listings
We encoded Leroy Merlin's rules as checks and let an agent build the workflow against them. That is what agent evals are for. A failed check tells the agent three things: the attribute, the rule, and the value that broke it. That is enough to know what to change. One of its first changes was to stop letting the model choose list values freely, and to narrow each list in code to the values of the product's category. We kept most of the products aside, so we could tell whether the agent had learned the task or only the products it was working on.
Then we ran STIGA's own listings through the same checks. The eval reads a spreadsheet, and their marketplace export is one. The listings STIGA already had passed 91.1% of the checks. The ones our workflow wrote pass 99.8%. Our workflow was built against those checks and their listings never were, and that is exactly the difference the checks exist to make. For the listings that are not ready yet, the same checks say what is missing.
![Two large figures. Before: 91.1% of checks passed on the listings STIGA already had. After: 99.8% on the listings our workflow wrote. Same 97 products on Leroy Merlin, same checks.](https://cdn.sanity.io/images/xd4hrbt2/production/286bce9cedeefdaa10bea6504b2595fc4aa6ee4c-2400x1350.png?auto=format)
*The same checks, run on what STIGA had and on what our workflow wrote: the share that passes.*
## Conclusion
Building evals for retail came down to three things. Let the person who knows the customer write the rules. Run them on whatever exists, starting from the customer's own catalog. And make every rule, from a character limit to a judgment call, the same kind of check.
The models will keep changing, and so will the way our workflows are built. The evals are what stays: the customer's requirements, written down once, in a form a machine can check and an agent can aim at.
![Two large figures. Before: 91.1% of checks passed on the listings STIGA already had. After: 99.8% on the listings our workflow wrote. Same 97 products on Leroy Merlin, same checks.](https://cdn.sanity.io/images/xd4hrbt2/production/286bce9cedeefdaa10bea6504b2595fc4aa6ee4c-2400x1350.png?auto=format)
## Author
[Vittorio Rossi](/author/vittorio-rossi) · AI Engineer, CommerceClarity
## Questions
### What is an eval framework?
An eval framework is the set of checks that says whether an AI output is good enough, and the machinery that runs them. In ours, each check is one rule on one attribute, answered with pass or fail, and verified by code, by an LLM as a judge or by a decision model.
### Can an eval framework run on a catalog that AI did not write?
Yes, and that is where we start. The eval reads a spreadsheet, one product per row and one attribute per column, so it scores the customer's current catalog or a marketplace export the same way it scores our output. That first run is the baseline.
### When is a check code, an LLM as a judge or a decision model?
Code for anything with a formula: a length limit, a required field, a value from a closed list. An LLM as a judge or a decision model for a judgment call, such as whether a title reads as natural French. We label examples by hand first and use them to calibrate the model before it runs on its own.
### Does passing every check mean a product page is correct?
No. Following the rules is not the same as being true. A title can respect every marketplace rule and still state the wrong blade length. That is why a second check compares each value with ground truth data, the customer's verified data or the manufacturer's documents, whenever it exists.
### What are agent evals?
Agent evals are checks an agent works against while it builds or runs a workflow. A failed check names the attribute, the rule and the value that broke it, which tells the agent what to change. On STIGA's Leroy Merlin listings, the agent built the workflow against the same checks we later ran on STIGA's own listings.
### Who writes the rules?
The person who knows the customer: at CommerceClarity, the Retail Engineer. They collect the rules from calls, documents and the customer's teams, and turn each one into a check without needing a developer.
## Entities
- Eval framework · Thing, about
- Retail Engineer · Thing, mentions
- [STIGA](https://www.stiga.com/) · Organization, mentions
- [Leroy Merlin](https://en.wikipedia.org/wiki/Leroy_Merlin) · Organization, mentions
- [Vercel](https://en.wikipedia.org/wiki/Vercel) · Organization, mentions
## Sources
- [Agents are users, so your software needs evals](https://vercel.com/blog/agent-evals) · Vercel
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) · arXiv
- [Regulation (EU) No 1169/2011 on the provision of food information to consumers](https://eur-lex.europa.eu/eli/reg/2011/1169/oj) · EUR-Lex
- [Regulation (EC) No 1223/2009 on cosmetic products](https://eur-lex.europa.eu/eli/reg/2009/1223/oj) · EUR-Lex
---
Canonical: https://commerceclarity.com/blog/eval-framework-retail
Every page of this site is available as markdown: append `.md` to its path. Index: /llms.txt
Legal:
[Privacy](/legal/privacy-policy)
[Legal](/legal)
[Trust Center](/trust-center)
Elsewhere:
[LinkedIn](https://www.linkedin.com/company/commerce-clarity)
[X](https://x.com/CommerceClarity)
[GitHub](https://github.com/CommerceClarity)
---