Agent Access

This site is intentionally agent-friendly. The published content is public reference material and one of the project goals is to make it easily consumable by AI assistants, MCP clients, search bots, and similar tooling.

This page documents the agent-facing surfaces, what they contain, what is excluded, and how they are maintained.

Agent surfaces

Surface	URL pattern	Purpose
Curated index	/llms.txt	Short, hand-picked index of canonical entry points + pointers to bundles
Compact bundle	/llms-small.txt	Hierarchy + summaries — for token-tight agents
Full bundle	/llms-full.txt	Entire site as a single Markdown blob — for one-shot RAG ingestion
Raw page source	/<slug>.md	Any rendered page also served as raw Markdown
Crawler policy	/robots.txt	Explicit allow rules for known AI crawlers

All four are emitted as part of the standard build and deployed alongside the rendered HTML. Their absolute URLs at production are:

• https://arda-cards.github.io/llms.txt
• https://arda-cards.github.io/llms-small.txt
• https://arda-cards.github.io/llms-full.txt
• https://arda-cards.github.io/robots.txt

For preview builds the prefix is https://arda-cards.github.io/documentation/....

What’s in /llms.txt

The curated index follows the llms.txt proposal. It contains:

• A title and one-paragraph description of the site.
• Pointers to the small and full bundles.
• An Optional section listing about 20 hand-picked entry points — the top-level section indexes plus a few high-leverage deep references (architecture patterns, the information model, the authoring guide, the four core technology references).

The index is deliberately short. Agents that need more should follow it into the small or full bundles.

What’s excluded

The agent surfaces deliberately exclude roadmap/**. Roadmap content is project-internal — designs, plans, analyses, and session logs that decay in relevance after the project ships. Agents would otherwise pick up superseded designs and cite them as authoritative.

Excluded:

• roadmap/in-progress/<project>/...
• roadmap/completed/<project>/...
• roadmap/backlog/...
• roadmap/ideas/...

The exclusion only affects the agent surfaces (/llms.txt, /llms-small.txt, /llms-full.txt, and the /<slug>.md raw endpoints don’t render for excluded paths). The human site (/roadmap/... HTML pages) remains fully browsable.

Crawler policy

The robots.txt explicitly allows:

Vendor	Training crawler	Live fetch	Search index
Anthropic	ClaudeBot	Claude-User	Claude-SearchBot
OpenAI	GPTBot	—	OAI-SearchBot
Google	Google-Extended	—	Googlebot (default)

All other user-agents are also allowed (User-agent: * Allow: /). The site is reference material; we choose breadth over scarcity.

How the surfaces are generated

Two Starlight plugins handle generation; both are configured in astro.config.mjs.

• starlight-llms-txt emits /llms.txt, /llms-small.txt, and /llms-full.txt. Configuration: project name, description, exclude patterns (roadmap/**), promote patterns (durable section roots), and an optionalLinks array driving the curated index.
• starlight-dot-md emits a /<slug>.md route alongside every rendered page, returning the raw source.

robots.txt is a static file at public/robots.txt — Astro copies it through to dist/ unchanged.

Maintaining the curated index

When a section grows or shrinks, update the optionalLinks array in astro.config.mjs. Keep it short — fewer than 30 entries is the target. The full bundle covers the rest.

When a project completes and its content is durable enough to live outside roadmap/, promote the relevant pages into a permanent section (e.g. current-system/, domain/) before retiring the project’s roadmap directory. The agent surfaces will pick up the promoted content automatically.

Discoverability tips for agent authors

If you are building an agent or MCP client that consumes this site:

• Start at /llms.txt. It is short and tells you which bundles to fetch next.
• For per-page reads, append .md to any URL. The raw markdown contains frontmatter (title, description, tags, domain, maturity, author) plus the unrendered body.
• For bulk ingestion, /llms-full.txt is one ~10 MB markdown file with all included pages concatenated. Suitable for a single embedding pass.
• For sparse retrieval, /llms-small.txt contains hierarchy and per-page descriptions only — useful when a small index needs to fit a tight context.

Related

• knowledge-base/agent-access-surfaces.md — repo-side notes on plugin configuration and update procedures.
• process/craft/documentation/ — broader documentation conventions.