Underwing · Data · November 3, 2026

What does an AI agent hear when it asks your Shopify store about returns? I asked 200 stores

Every Shopify store now has a public address where any AI agent can ask about its policies, and Shopify writes the answer. I asked 200 randomly chosen Shopify stores three questions each through that address on 3 November. 199 answered. The answers stuck closely to what each store had written: where an answer and a refund policy both gave a number of days, they matched in 100 of 104 stores, and not one store was handed a price-match policy it doesn’t have. The risk is in your own pages. 9 of the 199 answers told the shopper, in plain words, that the store contradicts itself. Here’s how the tool works, what it said, and how to hear your own store’s answer in one command.

Watch this explainer (3 min).

What Shopify built

On 26 October Shopify moved what it calls the knowledge base into every admin. The merchant changelog says it’s “a single page where you review, edit and publish the facts AI agents get about your store”, found at Sales channels > Agentic > Knowledge base. It replaces the Knowledge Base app, “which will no longer be available after January 15, 2027.”

The other half is on the developer side. Shopify’s Ask MCP documentation describes a tool called ask_business at yourstore.com/api/ucp/mcp: it “returns an answer from the store’s knowledge base and policies.” It’s part of the draft version of UCP, the shopping protocol Shopify and Google back, and Shopify publishes an example agent profile “for testing”. That profile was all I needed. No account, no key, no approval.

Who uses it? Shopify’s Help Center page says your published knowledge base “is publicly available through the Storefront MCP and can be used by AI channels including Shop and third-party platforms.” It doesn’t name ChatGPT, Gemini or anyone else, and I won’t either.

Where an agent's answer about your store comes fromShopify's Ask tool, live on every store I tested (Nov 2026)Any AI agent asks a question"How many days do I have to return this?"yourstore.com/api/ucp/mcp, tool ask_businessno sign-up, no key: Shopify's example agent profile worksShopify writes the answer from your knowledge base and policiesFrom your adminPolicies, contact details,shipping settingsLive as soon as Shopifycreates themDrafted from your pagesFacts Shopify scansfrom your storefrontUnpublished untilyou publish themAdded by youYour own facts,up to 1,000 in totalLive whenyou publish themYou manage all three in one place:Sales channels > Agentic > Knowledge baseSources: shopify.dev Ask MCP docs; Shopify Help Center, Knowledge base;Shopify changelog 26 Oct 2026; my test, 3 Nov 2026

What’s live without you doing anything

The changelog’s headline promise is “Nothing Shopify drafted reaches AI agents until you publish it.” That’s true for the facts Shopify drafts by scanning your storefront. It isn’t the whole picture. The Help Center adds: “The facts that are linked to your Shopify admin, such as your store policies and your contact details, are published when Shopify creates them.” The same page lists your shipping and delivery settings as an admin source.

So on a store whose owner has never opened the knowledge base page, an agent still gets answers built from the policies you wrote and the countries you ship to. That matches what I saw: almost every store answered, and the answers quoted their policies.

What I asked, and what came back

I took a random 200 active stores from the Shopify crawl corpus I used for my crawler test last week: mostly small and mid-size stores, in many countries. Each got three questions, as a US shopper in English: what’s your return policy and how many days do I have; do you offer a price match guarantee; do you ship internationally. That’s 597 questions. 25 came back with a server error the first time, and all 25 answered on one retry.

  • 199 of 200 answered. The exception was a closed store.
  • 177 of 199 gave a specific return window. Most of the other 22 have none to give: 6 have no refund policy page at all, and the rest say things like all sales are final, or returns for defects only. The agent said so rather than inventing a number.
  • The numbers held up. In 104 stores both the answer and the refund policy page state a number of days. In 100, every number in the answer is on the policy page. Of the other four, two are stores the agent itself flagged as contradictory, and in one I found the number on a separate returns page on the same store. I also read eight random answers against their policies by hand. All eight were exact.
  • It translated. Stores with policies in German, Japanese, Croatian and Spanish got English answers that carried the right windows.
  • The price-match question was my trap, because few small stores have one. 198 answers said the store’s information doesn’t say. The one “yes” is backed by that store’s own refund policy page, which says “Price matching and adjustments are subject to verification and approval.”
What 199 Shopify stores told an AI agentRandom sample of 200 active stores, US shopper, English. My test, 3 Nov 2026Answered the agent at all199 of 200the one that didn't is a closed storeGave a return window177 of 199most of the rest have none to give (final sale, defects only)Matched the policy page100 of 104stores where answer and refund page both state daysSaid the store contradicts itself9 of 1995 on returns, 4 on where the store shipsInvented a price-match policy0 of 199198 said the store doesn't say; the 1 yes is on its own pageThe answers stuck to what each store had written.The trouble came from stores whose own pages disagreed.597 questions; 25 returned a server error and all 25 answered on one retry

The finding that matters: it repeats your contradictions

The agent doesn’t pick a winner when your pages disagree. It tells the shopper. Nine answers did this, five about returns and four about shipping. A few, quoted from the answers:

  • One store: “The store’s information gives conflicting return windows: one policy says returns are allowed within 7 days of delivery, while another says non-personalized items may be returned within 14 days of delivery.” Its refund page also mentions 41 days.
  • An artist merch store: “one policy says to request a return or exchange within 30 days of delivery, while another says within 45 days of delivery.”
  • A UK store asked about shipping: “its delivery countries list includes the US and many other countries, but its terms say it ships to UK addresses only. It doesn’t specify which statement governs.”
  • A jewellery store: “The policy is inconsistent: it says Hira does not accept returns or issue refunds for any reason, but also says items must be returned within 7 days.”

All four shipping cases have the same shape: a delivery-country list with more countries than the store’s written terms allow. My read, and it is inference, is that the list is the store’s shipping settings in the admin, which the Help Center names as one of the knowledge base’s sources. A human visitor reads one page and moves on. An agent reads both and reports the clash to someone deciding whether to buy.

The reverse also happens. One store’s banner promises “3-DAY EASY RETURNS”, but its refund policy names no window, so the agent said the policy “does not specify a general number of days to return an item.” A promise that only lives in a banner doesn’t reach the agent.

A caution on what I can’t see. Two stores with no refund policy page still got specific windows, so the tool reads more than the policy pages. Whether that came from facts those owners published, or from somewhere else, the answers don’t say: Shopify’s docs warn not to depend on “source links” in the response.

What I’d do this week

  1. Ask your own store the return question with the command below, and read the answer as a shopper would.
  2. Make your refund policy, terms of service, FAQ and any returns page say the same number of days. Then check your shipping zones against your written shipping policy.
  3. Open Sales channels > Agentic > Knowledge base. Review the drafted facts, publish the ones that are right, and add the answers you want agents to give. If you change a page, the Help Center says a requested rescan “can take up to 48 hours”.
  4. If you used the old Knowledge Base app with more than 1,000 facts, re-add the ones you need before 15 January 2027. They didn’t move over.

Hear your own store’s answer

Paste this into a terminal and replace YOURSTORE.com with your domain. It uses Shopify’s own example agent profile, the same way I did:

curl -s https://YOURSTORE.com/api/ucp/mcp -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"ask_business","arguments":{"meta":{"ucp-agent":{"profile":"https://shopify.dev/ucp/agent-profiles/examples/draft/ask.json"}},"ask":{"query":"What is your return policy?","context":{"address_country":"US","language":"en"}}}}}' | grep -o '"plain":"[^"]*"' | tail -1

If it prints nothing, run it again: 25 of my 597 questions hit a server error the first time and answered on the second. Change the question to anything a shopper might ask. Ask is a draft capability, so Shopify may change it, and one sample of 200 stores on one day is a snapshot, not a rate you should quote for your own store. Your own answer is the one that counts.

Written by Aman, founder of Elytron Labs: Pollen (agent commerce readiness) and DripSync (AI product image sync).

Leave a comment

Your email stays private. First-time comments are held for review.