Description
Summary
- Add sitemap.xml to the documentation website
Improve discoverability and ranking - Investigate llm.txt
Token limitations makes it hard to use LLMs to parse and provide usable information for large documentation sites. Though technically possible to process documentation directly from HTML, the presence of non-essential content (navigation elements, CSS, JavaScript, metadata and tracking scripts) add to the context window in lieu of actual content. Filtering out this ‘noise’ is resource-intensive and slows down the LLM’s performance.
Why is this needed?
- sitemap.xml
Traditional sitemaps (sitemap.xml) help search engines find website pages and optimizes SEO to rank higher in search results. - LLM.txt link and link
llms.txt is designed specifically for LLMs, providing a structured, semantic summary of website content. While sitemaps aid discovery and SEO improves ranking, llms.txt enables AI to understand content. In an AI-driven future where LLMs power search and other applications, this semantic understanding becomes essential for accurate, relevant AI responses. Sitemaps and SEO still remain relevant for traditional search, but llms.txt is better positioned for an AI-centric future by prioritizing semantic understanding for LLMs.
Which area does this relate to?
No response
Solution
No response
Acknowledgment
- This request meets Powertools for AWS Lambda (.NET) Tenets
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
✅ Done