Skip to content

Journal

The llms.txt and Schema Checklist: What Your Site Actually Needs

What to check in robots.txt, structured data, and your sitemap, plus what llms.txt can and cannot do for search.

By , . 4 min. Updated

If we’re working on your site’s SEO, I want to know that people and search tools can reach the useful pages. That’s where I start.

A special file isn’t going to make up for a site that barely explains what your business does.

Check that search crawlers can get in

Open your robots.txt and check the rules. Also check that your hosting or firewall isn’t blocking the crawlers you want to allow.

One important distinction: search crawling and AI training are different things. OpenAI uses OAI-SearchBot for search and GPTBot for training. You can make separate choices about them. OpenAI explains those controls here.

For Google Search, check Googlebot access, whether the page is indexable, and your Search Console settings. Don’t assume that changing Google-Extended turns Google search visibility on or off.

I’d also check:

  • Important pages return a normal success response.
  • There isn’t an accidental noindex tag.
  • Each page has a sensible canonical URL.
  • Navigation and other useful pages link to it.
  • The XML sitemap lists the pages you actually want indexed.
  • The important words are in the page, not just inside an image.

Is llms.txt worth adding?

I keep a short one on this site. It’s an optional summary for systems that choose to use it.

But let’s be clear about its limits. Google says llms.txt does not improve or hurt Google Search visibility. It does not replace your pages or your sitemap. That’s in Google’s own guidance.

If you have one, keep it accurate. Outdated prices, missing services, and old project links don’t become useful just because they’re in a text file.

Use structured data that fits the page

Structured data describes information in a consistent format. I use it to make the site’s business, services, articles, and navigation explicit.

The details still need to be true and visible on the page.

Business and service information

Use the type that fits the business. For example, Organization can describe the company, while Service describes a particular service. Don’t invent a public street address or reviews just to fill a field.

Keep names, service areas, contact details, and offers consistent with what customers can actually read.

Articles and authors

For a guide, I want the title, original publication date, any genuine update date, and the author to match the visible page. The author should link to a useful profile. Google’s article documentation explains the supported properties.

Changing a date on every rebuild doesn’t make an old article new.

What about FAQs?

Useful answers still belong on a site. But Google discontinued FAQ rich results in May 2026. Don’t buy a service on the promise that FAQ markup will put extra answers under your search listing. See Google’s documentation updates.

I keep FAQs because customers ask the questions, not because they are a shortcut.

Check the result after a rebuild

I’d validate the markup, compare it with the visible page, and check the URLs again after a redesign or migration. The Rich Results Test checks supported Google features; it isn’t a complete SEO audit.

My free AI search grader can flag some basic setup issues. Its score is a checklist score, not a prediction of whether an AI will recommend you.

If the technical side is a mess, send me the site. We can work through it. The full AI search guide covers the content side too.

Written by Ian

Founder of Northern Web Dev. Twenty years designing, building and ranking sites, by hand. More about me.

Keep reading

Want this handled for you?

Send me your site and tell me what you want to improve. I'll take a look and we can talk about where to start.

Book a free call