I spent some time testing the AI Agent in Shopify Inbox and wanted to compare notes.
A few things I ran into:
Shipping questions: it couldn’t answer them, even though everything was written in the Shipping policy. The info was right there and it still missed it.
No pre-defined Q&A = weak results: I didn’t set up the Shopify Knowledge Base app, just installed it. Without pre-defined answers it doesn’t seem to work well at all.
Product search: it does return products, but sometimes it can’t find items that are actually in the store.
Barely any customization: I couldn’t upload documents for it to use as a knowledge base, and I couldn’t even change the text on the button. The UI is locked down.
I was expecting more, honestly. It was presented really well in the '26 Spring Edition, so I went in with higher hopes.
It just goes to show how hard it is to build a great AI assistant, even for Shopify, with all the detail and the exclusive access to internal data they have.
Anyone getting solid results out of it, or is this just where things are right now?
I think the Knowledge Base is one of the biggest factors here. Without well defined Q&A, it’s understandable that the AI struggles with questions like shipping, even when the information exists on the store.
I also agree that more customization options would make the AI much more practical for different business needs. Hopefully Shopify continues to improve it as the feature matures.
Has anyone been able to get consistently accurate responses after fully setting up the Knowledge Base?
We’re really impressed with the new Shopify Inbox AI, but we’d love to see a little more control over the chat window’s appearance.
Currently, the chat window uses a frosted glass/translucent background. While it suits some stores, it doesn’t work well for every brand and can reduce readability depending on what’s behind the chat.
It would be great to have a simple setting to choose between:
Solid background colour
Glass/translucent background (current behaviour)
Optional blur or transparency level
The appearance settings already allow us to customise colours, fonts and border radius, so adding this option would make it much easier to match the chat to the rest of a store’s branding.
Hopefully this is something that could be added in a future update. It would give merchants a lot more flexibility without changing the existing default design.
Hi there @david.kis
I’ve seen similar results. The largest difference is made by fully setting up the Knowledge Base with good FAQs rather than using only the existing content of the store. You may also want to consider looking at the product, policy and layout organization, as better data = better answers. For shiping, adding popular inquiries straight to the Knowledge Base can have a visible impact. It’s getting better, but it still needs the good infrastructure of content rather than expecting to get good answers straight out of the box.
Ran into the same wall. The agent is only as good as the structured Q&A behind it, which is exactly why it whiffs on shipping questions that are sitting right there in your policy. Policy pages are prose; the retrieval layer wants clean question-answer pairs.
Two things that actually helped us:
Feed it the real questions, not hypothetical ones. Pull your last few weeks of Inbox and email threads, find the 15-20 that keep repeating, and write a short answer for each. “Do you ship to Canada,” “how long does it take,” “what does it cost” as three separate entries beats one long shipping policy every time.
Publish those answers on the product page too, not just inside the chat KB. A visible product Q&A block deflects a chunk of the questions before they ever reach the agent, and the same curated pairs become structured content that AI search and Google’s AI Overviews can cite. Reactive chat and a persistent on-page Q&A library cover two different moments in the buying journey.
On the locked-down UI and no document upload, yep, that’s where it is right now. Not much you can change there yet.
Full disclosure: we build one of the on-page product Q&A apps (AC - Questions & Answers), so I lean toward that second point. But the lesson holds with any tool: the agent is retrieval, and it starves without curated Q&A feeding it.
I’ve been reading the Shopify Inbox App reviews since the AI assistant release. They’re pretty brutal, a lot of new 1 star reviews.
Some of them simply want Shopify to revert that last release.
Here’s a quick summary of the biggest complaints:
Forced customer sign-in killed lead capture. The old quick guest form is gone, and people won’t create an account just to ask a question, so merchants can’t follow up. Some report inquiries down ~80%.
Turning sign-in off doesn’t help - you lose all customer details anyway. Lose-lose.
No more order history or customer context in chats. Returning buyers show up as “new visitor.”
The AI is unreliable and you can’t jump into a chat or correct it when it answers wrong.
New UI is slower and clunkier. More clicks, harder to scan conversations.
Bugs everywhere: covering the whole mobile screen, settings not sticking, display breaking on storefronts.
Basically they added friction and gutted the merchant workflow for AI features that aren’t ready yet.
If you want to test something cool, take a look at Talkom AI in the App Store. Happy to help with the setup!
