Adding llms.txt to Your Astro Blog: A Guide

IRC-Mania Die Seite rund um Sicherheit und Programmierung

Internet, Sicherheit , Internet Relay Chat

Gesponsert

Background

As a software developer, I run IRC-Coding.de, which hosts over 600 articles on software development and programming. AI-powered search engines like ChatGPT, Perplexity, and Claude crawl websites and attempt to understand their content. Yet they struggle with HTML markup, JavaScript rendering, and unnecessary code clutter.

The llms.txt specification addresses exactly this: a simple text file in your site’s root directory that serves content in a machine-readable format.

When implementing it, I faced several questions:

• How do I generate llms.txt automatically during the build?
• Which files do I actually need (llms.txt, llms-full.txt, llms-small.txt)?
• How can I protect my content from straightforward copying by bad actors?

I examined two established approaches: a custom API route (as Alex OP describes) and the @4hse/astro-llms-txt plugin (see GitHub). In this article, I’ll walk you through how I compared both and which solution I chose for IRC-Coding.de.

By the way, Alex OP’s site is genuinely worth checking out. I found it through a Google search for “llms.txt Astro.”

Important: Astro v7 Incompatibility

Solution

Step 1: Install the Plugin (Astro v6 and earlier)

npm install @4hse/astro-llms-txt

The plugin uses Astro’s astro:build:done hook to read the generated HTML files from the dist/ directory and convert them to Markdown using rehype-remark. This means you don’t need to implement custom parsing or content extraction logic yourself.

Step 2: Update Your Astro Configuration

In my astro.config.mjs, I added the plugin to the integrations array:

import astroLlmsTxt from '@4hse/astro-llms-txt';

export default defineConfig({
  site: 'https://www.deine-seite.de',

  integrations: [
    // ... other integrations
    astroLlmsTxt({
      title: 'IRC-Coding',
      description: 'Software Development and Programming, Tutorials, Artikel und Ressourcen.',
      notes: '- Content is auto-generated from the official source at https://www.irc-coding.de',
      docSet: [
        {
          title: 'Complete site',
          description: 'Excerpts of all blog articles with links to full content',
          url: '/llms-full.txt',
          include: ['**'],
          promote: ['index', 'blog/**'],
          excerptLength: 600,
          visitLinkText: 'Visit full article',
        },
        {
          title: 'Compact overview',
          description: 'Structure-only index of all pages',
          url: '/llms-small.txt',
          include: ['**'],
          onlyStructure: true,
          promote: ['index', 'blog/**'],
        },
      ],
      pageSeparator: '\n\n---\n\n',
    }),
  ],
});

Step 3: Run the Build

npm run build

After the build completes, you’ll find three files in dist/:

• llms.txt: An index with titles, descriptions, and links to the doc sets
• llms-full.txt: All page content converted to Markdown
• llms-small.txt: Structure only (headings and lists)

Step 4: Excerpt Limits for Content Protection

By default, the plugin generates the full content in llms-full.txt. With 600 articles on my site, this resulted in a file exceeding 140,000 lines—a goldmine for content scrapers. Anyone could download https://www.irc-coding.de/llms-full.txt and copy all article text wholesale.

I extended the plugin to address this. The excerptLength option caps each article to the first 600 characters, followed by a link back to the original:

# Algorithmus einfach erklärt

> Algorithmus verständlich erklärt: Eigenschaften, Entwurfsparadigmen...

## Definition

Ein Algorithmus ist eine endliche Folge von wohldefinierten Anweisungen...

[Visit full article](https://www.irc-coding.de/algorithmus-begriffserklaerung-komplexitaet-korrektheit)

AI search engines get enough context to understand the topic, but the full article only lives on your website. To implement this, I extended the DocSet interface in the plugin and modified the buildEntryFromHtml function to truncate content at excerptLength characters and append the visit link.

Why 600 characters? I tested a few articles and found that 600 characters is enough for an AI to understand the context, but not enough to simply copy the article verbatim. Try it yourself—400 or 800 characters might work better for your use case.

Alternative Approaches

Custom API Route (Alex OP)

Alex OP outlines a straightforward approach using an Astro API route:

// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { getCollection } from "astro:content";

export const GET: APIRoute = async () => {
  const posts = await getCollection("blog", ({ data }) => !data.draft);
  const sortedPosts = posts.sort(
    (a, b) => new Date(b.data.pubDatetime).valueOf() - new Date(a.data.pubDatetime).valueOf()
  );

  let llmsContent = "";
  for (const post of sortedPosts) {
    llmsContent += `---\ntitle: ${post.data.title}\ndescription: ${post.data.description}\n---\n\n`;
    // ... content extraction
  }

  return new Response(llmsContent, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

Advantages:

• No extra dependencies
• Full control over formatting
• Works in the dev server too

Disadvantages:

• You must implement content extraction yourself (removing MDX components, filtering Shiki Twoslash directives, parsing frontmatter)
• No llms-small.txt or structured doc sets
• No onlyStructure mode

It’s generally a solid approach, though I’d recommend reading Alex’s article directly to be sure you understand all the implications.

@4hse/astro-llms-txt Plugin

The 4HSE plugin takes a different approach: it reads the finished HTML files after the build and converts them back to Markdown using rehype-remark.

Advantages:

• Complete pipeline (rehype-parse → rehype-remark → remark-gfm → remark-stringify)
• MDX components, Expressive Code, and tabs are handled automatically
• onlyStructure mode for compact summaries
• DocSets with include, promote, demote pattern matching

Disadvantages:

• Only works during build, not on the dev server
• No built-in excerpt limit (I added it myself)

Comparison

I chose the @4hse/astro-llms-txt plugin because it automatically handles MDX components and Expressive Code blocks. With over 600 articles, custom content extraction would have been too error-prone. Alex OP’s custom API route approach works well for smaller blogs, but at my content scale, the maintenance overhead for extraction would have been disproportionate.

Solution for Astro v7: Custom Script

Since @4hse/astro-llms-txt wasn’t compatible with Astro v7 initially, I now generate the files with a custom Node script. It runs after the Astro build and reads HTML files from dist/, similar to how the plugin worked.

The integration happens directly in your package.json build script:

{
  "scripts": {
    "build": "astro build && node scripts/generate-llms-txt.mjs"
  }
}

The script itself is compact and follows the llms.txt specification:

// scripts/generate-llms-txt.mjs
import { readdir, readFile, writeFile, stat } from 'node:fs/promises';
import { join } from 'node:path';

const SITE = 'https://www.irc-coding.de';
const DIST = './dist';
const EXCERPT_LENGTH = 600;

async function findHtmlFiles(dir, base = '') {
  const entries = await readdir(dir, { withFileTypes: true });
  const files = [];
  for (const entry of entries) {
    const fullPath = join(dir, entry.name);
    const relPath = base ? `${base}/${entry.name}` : entry.name;
    if (entry.isDirectory()) {
      files.push(...await findHtmlFiles(fullPath, relPath));
    } else if (entry.name === 'index.html') {
      files.push({ fullPath, relPath: base || '' });
    }
  }
  return files;
}

function extractText(html) {
  // Remove script/style tags
  let text = html.replace(/<(script|style)[^>]*>[\s\S]*?<\/\1>/gi, '');
  // Remove HTML tags
  text = text.replace(/<[^>]+>/g, ' ');
  // Clean up whitespace
  text = text.replace(/\s+/g, ' ').trim();
  return text;
}

function extractTitle(html) {
  const match = html.match(/<title[^>]*>([^<]+)<\/title>/i);
  return match ? match[1].trim() : 'Untitled';
}

async function main() {
  const files = await findHtmlFiles(DIST);
  const pages = [];

  for (const file of files) {
    if (file.relPath.startsWith('404') || file.relPath.startsWith('admin')) continue;
    const html = await readFile(file.fullPath, 'utf-8');
    const title = extractTitle(html);
    const text = extractText(html);
    const excerpt = text.slice(0, EXCERPT_LENGTH);
    const url = file.relPath ? `${SITE}/${file.relPath}/` : `${SITE}/`;
    pages.push({ title, excerpt, url, relPath: file.relPath });
  }

  // llms.txt (index)
  let llmsTxt = `# ${SITE}\n\n> Software Development and Programming\n\n`;
  llmsTxt += `## Docs\n\n- [Complete site](${SITE}/llms-full.txt): Excerpts of all articles\n- [Compact overview](${SITE}/llms-small.txt): Structure-only index\n`;
  await writeFile(join(DIST, 'llms.txt'), llmsTxt, 'utf-8');

  // llms-full.txt
  let llmsFull = `# ${SITE}\n\n> Software Development and Programming\n\n`;
  for (const page of pages) {
    llmsFull += `\n---\n\n## ${page.title}\n\n${page.excerpt}\n\n[Visit full article](${page.url})\n`;
  }
  await writeFile(join(DIST, 'llms-full.txt'), llmsFull, 'utf-8');

  // llms-small.txt
  let llmsSmall = `# ${SITE}\n\n> Software Development and Programming\n\n`;
  for (const page of pages) {
    llmsSmall += `- [${page.title}](${page.url})\n`;
  }
  await writeFile(join(DIST, 'llms-small.txt'), llmsSmall, 'utf-8');

  console.log(`Generated llms.txt, llms-full.txt, llms-small.txt with ${pages.length} pages`);
}

main().catch(console.error);

Advantages of the custom script:

• No peer dependency conflicts during Astro upgrades
• Full control over excerpt length and format
• Runs on any deployment platform (Railway, Vercel, Netlify)
• Easy to adapt when requirements change

Disadvantages:

• No automatic MDX component handling (text extraction only)
• No onlyStructure mode with nested headings
• You’re responsible for maintenance

For IRC-Coding.de with over 300 articles, this approach works well. AI systems get the context they need, and I maintain content protection through excerptLength.

Common Issues

• Error: Expected pattern to be a non-empty string Solution: The plugin uses picomatch internally. Empty strings in promote or include arrays cause this error. Use ['index', 'blog/**'] instead.

• Error: File not found: dist/de/index.html Solution: The include: ['**'] pattern matches all pages. If some pages aren’t generated (e.g., via draft: true), this warning appears. It’s harmless—the file is simply skipped.

• Error: llms-full.txt is too large Solution: I set excerptLength to 600 characters. This protects my content and keeps the file compact. This required a small modification to the plugin in node_modules/@4hse/astro-llms-txt/src/index.ts, where I extended the DocSet interface with excerptLength and visitLinkText.

• Error: AI systems can’t find the file Solution: Make sure llms.txt is in your domain’s root (e.g., https://www.irc-coding.de/llms.txt). On Vercel or Netlify, the file is automatically served correctly from dist/.

Does ChatGPT or Claude.ai actually use your llms.txt? Is it worth the effort?

Probably not—it mostly helps content scrapers.

I was shocked at how quickly someone could copy my WordPress site to Markdown, since WordPress already has all the tools built in.

It’s more of a community-minded approach, but AI systems are typically trained differently.

When I asked ChatGPT about this, here’s what it said:

For me as ChatGPT in this conversation, llms.txt doesn't provide a direct advantage because I don't simply crawl arbitrary websites live when answering and look for an llms.txt file. If you don't bring your website to me via web search or a link, I won't see the file at all.

However, if I do access information from your website via web search, a well-structured llms.txt can definitely be useful—especially when it:

lists important documentation clearly,
has good titles and descriptions,
points to relevant pages,
and ideally provides llms-full.txt or similar structured content.

For other AI systems

The potential value is currently higher here. Systems and tools that explicitly support llms.txt can:

find documentation faster,
prioritize content better,
search fewer irrelevant pages,
and access API or project documentation more efficiently.

For SEO?

In the traditional sense (Google rankings), there are currently no proven benefits.

I'd think of llms.txt more as:

robots.txt → for crawlers,
sitemap.xml → for search engines,
llms.txt → for AI-driven systems

They serve different purposes.

I wouldn't treat it as an SEO measure, but as an investment in the discoverability and usability of your content for AI-powered tools. If llms.txt becomes more established, you're already well positioned. If it doesn't, the effort for automated generation is usually manageable anyway.

Why did I implement it anyway?

The site helped me learn, and every additional visitor makes me happy. Hope dies last.

FAQ llms.txt for Astro

Article sources and references

• llms.txt specification
• 4hse/astro-llms-txt on GitHub
• Alex OP: How I Added llms.txt to My Astro Blog
• Astro documentation

Premium VPS Hosting

Schnelle und sichere VPS Server

Gesponsert