This post may contain affiliate links. If you buy through one, we may earn a commission at no extra cost to you. See our Affiliate Disclosure for details.
Somewhere in the last couple of years, a share of the traffic that used to arrive through a blue link started arriving through a paragraph instead. Someone types a question, a system writes four sentences back, and two or three small source links sit underneath. Those links are the click now.
Very little of the familiar optimisation playbook maps cleanly onto them. A ranking is a position you can watch. A citation is the outcome of a query you never saw, run against an index you can't inspect, resolving to a passage you didn't choose.
That sounds hopeless. It isn't — but the levers are further down the stack than most advice suggests.
AI search visibility is decided by retrievability far more than by anything resembling a ranking trick. Three conditions do most of the work: the right crawlers can actually fetch the page, the substance sits in server-rendered HTML rather than behind a script or a login, and the page contains a short self-contained passage that answers a specific question without needing the surrounding article for context. Retrieval happens at passage level, not page level, so a strong article with the answer buried in paragraph nine is structurally harder to cite than a modest page that states its answer in forty words near the top. The single most common own-goal is a robots.txt rule that blocks a search-oriented AI crawler while the owner believes they only blocked a training crawler — those are different user agents with different jobs. Meanwhile llms.txt, the file most often recommended for this, has no confirmed consumer among the major providers. Check the crawler rules first; that one is binary, free, and takes about a minute.
A citation is not a ranking position
Answer engines don't hand you a list and let you pick. They take a question, break it into several narrower queries, pull candidate passages from an index, and assemble a response from whatever came back. The sources shown are the passages the system leaned on while writing.
Two consequences follow, and both are awkward for anyone used to thinking in page rankings.
The first is that the retrieval unit is a chunk of text, not a URL. A 3,000-word guide isn't retrieved as a guide. It's cut into pieces, and one piece competes on its own merits against pieces from other sites. Everything you wrote above and below that piece is invisible at the moment of selection.
The second is that there's no queue. Position one for a keyword doesn't secure a mention, and a page that ranks somewhere on the second page can be quoted if it happens to hold the cleanest answer to the sub-question that got generated. This is genuinely different from search, and it is the part that makes AI search visibility feel arbitrary from the outside.
It isn't arbitrary. It's just judged at a smaller resolution than you're writing at.
Three things a retrieval system needs from a page
Before any of the interesting optimisation questions, a page has to clear three mechanical gates. Nothing further matters if it fails one.
| Gate | What it means | Typical failure |
|---|---|---|
| Fetchable | The relevant bot is permitted and gets a 200 | A robots.txt rule blocking the wrong user agent |
| Parseable | The text is in the HTML the crawler receives | Content injected client-side, or gated behind a signup |
| Quotable | One passage answers one question, standing alone | The answer is spread over four paragraphs and a table |
The third gate is the one worth sitting with, because it's a writing problem rather than a technical one.
A quotable passage states its claim, qualifies it, and stops — no "as we saw above," no pronouns pointing at something three paragraphs back, no answer that only makes sense after the setup. If a passage would confuse someone who was dropped into it cold, a retrieval system dropping into it cold has the same problem.
This isn't a call for stiff, robotic writing. It's an argument for putting the direct answer near the top of each section and then elaborating, rather than building to a reveal. Journalists have done it for a century; the inverted pyramid turns out to be machine-friendly for reasons nobody planned.
Retrieval happens one passage at a time. The article around it never gets a vote. Photo via Pexels.
The robots.txt line that quietly costs you every citation
Here is the mistake that shows up most often on small sites, and it's almost always well-intentioned.
At some point the owner read that AI companies were training on scraped content, decided they'd rather not participate, and pasted a block into robots.txt. Fair enough. The problem is that a single vendor runs several crawlers with unrelated purposes, and blocking them as a group also blocks the one responsible for showing your site as a source.
OpenAI publishes the distinction in its own documentation for OpenAI's crawlers: separate user agents cover model training, the search index that powers citations, and live fetches triggered by a user's request. They are configured independently in robots.txt. Blocking all of them is a legitimate choice — but it's a choice to be absent from the answers, not just from the training data, and plenty of site owners have made it without realising which one they made.
Google splits the same way. Its overview of Google crawlers documents Googlebot alongside Google-Extended, the control for Gemini-related uses, and the two are deliberately decoupled from ordinary Search crawling and indexing. Microsoft's side is tied to Bingbot, which you can inspect through Bing Webmaster Tools.
So the audit is short: open yoursite.com/robots.txt, read every Disallow under every User-agent, and confirm each block is one you'd defend on purpose. A CDN or security plugin may have added rules you never wrote. Broad AI-bot blocking is also increasingly offered as a one-click toggle in hosting dashboards, which means it can be switched on by someone who thought they were enabling a privacy feature.
Two related checks belong in the same pass. Make sure the page renders its content server-side — view source and search for a sentence from the middle of your article; if it isn't there, a crawler may not see it either. And confirm the pages you care about aren't orphaned, because a page nothing links to gets crawled late and refreshed rarely. The one-pass fix for internal linking on small blogs covers that side properly. If you'd rather have something else find these problems, the ground covered by AI SEO audit tools for solo business owners overlaps with about half this list.
llms.txt: an honest status report
The recommendation you'll meet everywhere is to add an llms.txt file — a markdown index of your site, placed at the root, meant to give language models a clean map of your content. The proposal is public and readable at llmstxt.org, and as an idea it's sensible.
The status, as far as anything can be publicly confirmed: no major AI provider has committed to reading it as part of retrieval. The vendors document their crawlers and their robots.txt handling in detail. None of them documents llms.txt as an input. That gap has persisted since the proposal appeared, and it's the thing most articles recommending the file leave out.
Which doesn't make it harmful. It's a static text file; it costs nothing to serve and breaks nothing. If you want to publish one as a bet on future adoption, that's a defensible bet.
Just don't let it displace the work that has a documented mechanism behind it. Structured data does — Google's introduction to structured data explains what search systems actually do with schema.org markup, and while none of that is a documented lever for AI citations either, it at least sits inside a pipeline the vendors describe. The question of which types are worth the effort on a small site is its own argument, made at length in schema markup for small sites.
The general shape of this problem: a lot of AI-visibility advice is confident about mechanisms nobody has published. Treat a claim about how retrieval works with the same scepticism you'd apply to a claim about the ranking algorithm.
A file at the root of your site is a card catalogue nobody has agreed to read yet. Photo via Pexels.
What you can measure, checked September 4, 2026
Measurement here is genuinely worse than in search, and pretending otherwise wastes money. Three things are available to a one-person site, in ascending order of cost.
Referrer traffic, free and immediate. When someone follows a citation, your analytics records a referral from the assistant's domain — chatgpt.com, perplexity.ai, copilot.microsoft.com and so on. It undercounts, because plenty of readers get their answer and never click. But it's real data about real citations, it's already sitting in your reports, and almost nobody looks at it.
Search Console, free and partial. Google Search Console folds AI-surface activity into its performance reporting without giving those surfaces their own filter, so you can see totals shifting without being able to attribute the shift cleanly. Useful for trend, useless for attribution.
Paid monitoring tools. A category has grown up around prompting assistants on a schedule and recording which brands and domains get mentioned. The established SEO suites have added modules; several standalone products do only this. They're worth evaluating if a meaningful share of your revenue depends on being named in answers.
No prices appear here deliberately. This corner of the market is young and repricing frequently, and a figure written on September 4, 2026 would mislead somebody reading in spring. Check the vendor's own pricing page, note the date you checked, and treat any number quoted second-hand — including in comparison posts — as unverified. The same discipline applies to rank tracking tools, where the underlying data is at least stable enough to compare.
One caution before spending anything: these tools sample. They ask a set of prompts and report what came back, and the same prompt asked twice can return different sources. Treat the output as a weather report, not a scoreboard.
Open your robots.txt before you change anything else
One thing, and it takes about a minute.
Type your domain followed by /robots.txt into a browser and read the whole file. For every User-agent line naming an AI crawler, decide deliberately whether that block is what you want — remembering that a vendor's search crawler and its training crawler are separate entries with separate consequences. If you find a broad block you don't remember writing, that is very likely the reason nothing cites you, and removing it is a two-minute edit rather than a content strategy.
Then, if the file is clean, do the follow-up on a single page. Take the post you'd most like an assistant to quote, look at the first paragraph under each heading, and rewrite one of them so it answers that heading's implied question in about forty words, standing entirely alone. Not a summary of the section — the answer itself, stated flatly, before the elaboration begins.
Come back to it in a month and look at your referral report. One page won't move a total, but it will tell you whether the passage-level rewrite is worth doing to the other twenty, which is a question you can only answer with your own pages.
저는 한국에서 직원 없이 혼자서 두 개의 소규모 사업체를 운영하고 있습니다. 견적, 회계, 마케팅, 웹사이트 관리까지 모두 제가 직접 하고 있습니다. 개발자는 아니지만, 이 사이트의 자동화 시스템은 AI 코딩 도우미를 활용하여 밤늦게까지 오랜 시간 공들여 구축했고, 모든 콘텐츠 관리 및 게시도 제가 직접 담당하고 있습니다.