What does an AI agent hear when it asks your Shopify store about returns? I asked 200 stores

Every Shopify store now has a public address where any AI agent can ask about its policies, and Shopify writes the answer. I asked 200 randomly chosen Shopify stores three questions each through that address on 3 November. 199 answered. The answers stuck closely to what each store had written: where an answer and a refund policy both gave a number of days, they matched in 100 of 104 stores, and not one store was handed a price-match policy it doesn’t have. The risk is in your own pages. 9 of the 199 answers told the shopper, in plain words, that the store contradicts itself. Here’s how the tool works, what it said, and how to hear your own store’s answer in one command.
What Shopify built
On 26 October Shopify moved what it calls the knowledge base into every admin. The merchant changelog says it’s “a single page where you review, edit and publish the facts AI agents get about your store”, found at Sales channels > Agentic > Knowledge base. It replaces the Knowledge Base app, “which will no longer be available after January 15, 2027.”
The other half is on the developer side. Shopify’s Ask MCP documentation describes a tool called ask_business at yourstore.com/api/ucp/mcp: it “returns an answer from the store’s knowledge base and policies.” It’s part of the draft version of UCP, the shopping protocol Shopify and Google back, and Shopify publishes an example agent profile “for testing”. That profile was all I needed. No account, no key, no approval.
Who uses it? Shopify’s Help Center page says your published knowledge base “is publicly available through the Storefront MCP and can be used by AI channels including Shop and third-party platforms.” It doesn’t name ChatGPT, Gemini or anyone else, and I won’t either.
What’s live without you doing anything
The changelog’s headline promise is “Nothing Shopify drafted reaches AI agents until you publish it.” That’s true for the facts Shopify drafts by scanning your storefront. It isn’t the whole picture. The Help Center adds: “The facts that are linked to your Shopify admin, such as your store policies and your contact details, are published when Shopify creates them.” The same page lists your shipping and delivery settings as an admin source.
So on a store whose owner has never opened the knowledge base page, an agent still gets answers built from the policies you wrote and the countries you ship to. That matches what I saw: almost every store answered, and the answers quoted their policies.
What I asked, and what came back
I took a random 200 active stores from the Shopify crawl corpus I used for my crawler test last week: mostly small and mid-size stores, in many countries. Each got three questions, as a US shopper in English: what’s your return policy and how many days do I have; do you offer a price match guarantee; do you ship internationally. That’s 597 questions. 25 came back with a server error the first time, and all 25 answered on one retry.
- 199 of 200 answered. The exception was a closed store.
- 177 of 199 gave a specific return window. Most of the other 22 have none to give: 6 have no refund policy page at all, and the rest say things like all sales are final, or returns for defects only. The agent said so rather than inventing a number.
- The numbers held up. In 104 stores both the answer and the refund policy page state a number of days. In 100, every number in the answer is on the policy page. Of the other four, two are stores the agent itself flagged as contradictory, and in one I found the number on a separate returns page on the same store. I also read eight random answers against their policies by hand. All eight were exact.
- It translated. Stores with policies in German, Japanese, Croatian and Spanish got English answers that carried the right windows.
- The price-match question was my trap, because few small stores have one. 198 answers said the store’s information doesn’t say. The one “yes” is backed by that store’s own refund policy page, which says “Price matching and adjustments are subject to verification and approval.”
The finding that matters: it repeats your contradictions
The agent doesn’t pick a winner when your pages disagree. It tells the shopper. Nine answers did this, five about returns and four about shipping. A few, quoted from the answers:
- One store: “The store’s information gives conflicting return windows: one policy says returns are allowed within 7 days of delivery, while another says non-personalized items may be returned within 14 days of delivery.” Its refund page also mentions 41 days.
- An artist merch store: “one policy says to request a return or exchange within 30 days of delivery, while another says within 45 days of delivery.”
- A UK store asked about shipping: “its delivery countries list includes the US and many other countries, but its terms say it ships to UK addresses only. It doesn’t specify which statement governs.”
- A jewellery store: “The policy is inconsistent: it says Hira does not accept returns or issue refunds for any reason, but also says items must be returned within 7 days.”
All four shipping cases have the same shape: a delivery-country list with more countries than the store’s written terms allow. My read, and it is inference, is that the list is the store’s shipping settings in the admin, which the Help Center names as one of the knowledge base’s sources. A human visitor reads one page and moves on. An agent reads both and reports the clash to someone deciding whether to buy.
The reverse also happens. One store’s banner promises “3-DAY EASY RETURNS”, but its refund policy names no window, so the agent said the policy “does not specify a general number of days to return an item.” A promise that only lives in a banner doesn’t reach the agent.
A caution on what I can’t see. Two stores with no refund policy page still got specific windows, so the tool reads more than the policy pages. Whether that came from facts those owners published, or from somewhere else, the answers don’t say: Shopify’s docs warn not to depend on “source links” in the response.
What I’d do this week
- Ask your own store the return question with the command below, and read the answer as a shopper would.
- Make your refund policy, terms of service, FAQ and any returns page say the same number of days. Then check your shipping zones against your written shipping policy.
- Open Sales channels > Agentic > Knowledge base. Review the drafted facts, publish the ones that are right, and add the answers you want agents to give. If you change a page, the Help Center says a requested rescan “can take up to 48 hours”.
- If you used the old Knowledge Base app with more than 1,000 facts, re-add the ones you need before 15 January 2027. They didn’t move over.
Hear your own store’s answer
Paste this into a terminal and replace YOURSTORE.com with your domain. It uses Shopify’s own example agent profile, the same way I did:
curl -s https://YOURSTORE.com/api/ucp/mcp -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"ask_business","arguments":{"meta":{"ucp-agent":{"profile":"https://shopify.dev/ucp/agent-profiles/examples/draft/ask.json"}},"ask":{"query":"What is your return policy?","context":{"address_country":"US","language":"en"}}}}}' | grep -o '"plain":"[^"]*"' | tail -1
If it prints nothing, run it again: 25 of my 597 questions hit a server error the first time and answered on the second. Change the question to anything a shopper might ask. Ask is a draft capability, so Shopify may change it, and one sample of 200 stores on one day is a snapshot, not a rate you should quote for your own store. Your own answer is the one that counts.