Website Markdown Checker for AI Agents
Test any webpage for Markdown, inspect what AI agents receive, compare it with the original HTML and generate a clean fallback when it's missing.
Free · No signup · Nothing stored
Check if a webpage supports Markdown for AI agents
AI agents do not always need the full HTML version of a webpage. Some websites can return a cleaner Markdown representation when a client requests text/markdown, while others expose a .md version or advertise an alternate Markdown URL. Tuurbo checks these mechanisms automatically and shows exactly what the server returns.
What does the Markdown checker test?
Accept: text/markdown
We request the original URL using the Accept: text/markdown HTTP header and verify whether the server returns a genuine Markdown representation.
Markdown alternate URLs
We inspect .md variants, HTML rel="alternate" links and HTTP Link headers that may point to another Markdown representation.
llms.txt
We check whether the website publishes an llms.txt file and whether it references the page or an equivalent Markdown resource. llms.txt and Markdown delivery are related but separate mechanisms.
Scraping and crawler blocks
If the website returns a CAPTCHA, WAF challenge, 403 response, rate limit or login wall, the checker reports the access restriction instead of incorrectly saying that Markdown is missing.
Does the Markdown actually contain the page content?
Finding a Markdown response is not enough. A server can return text/markdown while still providing an incomplete, noisy or outdated representation of the webpage. Tuurbo compares meaningful HTML content with the Markdown representation and checks what was preserved, lost or unnecessarily included.
Markdown Content Fidelity
Markdown Content Fidelity measures how much meaningful information from the original HTML webpage is preserved in its Markdown representation: main text, headings, contextual links, images, lists, tables and metadata.
Missing content
The checker identifies important sections that exist in the webpage but disappear from the Markdown version, such as pricing tables, FAQs, product details or article sections.
Unnecessary noise
A useful Markdown representation should prioritize page content rather than menus, footers, cookie notices, repeated navigation or interface boilerplate.
Compare HTML and Markdown token usage
HTML often contains layout markup, scripts, styles, navigation and interface elements that are irrelevant to an AI agent. The checker estimates the tokens required for the original HTML and its Markdown representation so you can compare the difference directly.
HTML
18,420 estimated tokens
Markdown
4,210 estimated tokens
77% fewer estimated tokens
Example values
No Markdown version? Convert the webpage to Markdown
If the publisher does not provide a Markdown version, Tuurbo can create a clean fallback directly from the public webpage. The deterministic conversion removes scripts, styles and unnecessary interface elements while preserving useful structure, links, lists, tables and images.
Generate MarkdownDeterministic conversion · No AI rewriting
Why serve Markdown to AI agents?
Cleaner content
Markdown can remove browser-oriented markup that automated systems do not need.
Lower processing overhead
A smaller representation may reduce the amount of text and markup an agent needs to retrieve and parse.
Explicit structure
Headings, lists, links, code blocks and tables remain easy to identify.
What is Accept: text/markdown?
Accept: text/markdown is an HTTP content-negotiation request. The client asks whether the same resource can be returned as Markdown instead of HTML.
curl -H "Accept: text/markdown" https://example.com/articleExpected response
HTTP/2 200
Content-Type: text/markdown; charset=utf-8
Vary: AcceptMarkdown for Agents vs llms.txt
| Topic | Markdown for Agents | llms.txt |
|---|---|---|
| Main purpose | Deliver page content | Advertise/select site resources |
| Scope | Individual resource | Site/resource list |
| Typical format | Markdown | Markdown/plain text |
| Common location | Page URL or alternate | /llms.txt |
| Changes representation | Yes | No |
| Replaces robots.txt | No | No |
llms.txt can help advertise important content, while Markdown delivery determines what representation an agent receives when accessing a resource.
What happens if the website blocks the checker?
Some websites restrict automated requests using a WAF, CAPTCHA, login wall, JavaScript bot challenge or rate limit.
In that case, Tuurbo does not assume that Markdown is unavailable. The result is marked as blocked, rate-limited or authentication-required and shows the available technical evidence.
- HTTP 401
- HTTP 403
- HTTP 429
- Retry-After
- CAPTCHA
- Cloudflare challenge
- WAF challenge
- JavaScript bot check
- robots.txt restriction
- login wall
How to make a website serve Markdown
HTTP content negotiation
Return HTML to normal browser requests and Markdown when a client requests text/markdown, with Vary: Accept.
Publish a .md alternate
Make a Markdown version available next to the page, for example /article and /article.md.
Advertise the Markdown alternate
Add an alternate link in the page HTML:
<link
rel="alternate"
type="text/markdown"
href="/article.md"
/>Generate Markdown at the edge
A CDN or edge layer can create the Markdown representation without changing the underlying CMS.
Make your entire website readable by AI agents
Checking one URL tells you whether a problem exists on that page. Tuurbo can work across the whole website, helping manage crawler accessibility, machine-readable content, structured data, llms.txt and other signals used by AI-oriented retrieval systems.
Frequently Asked Questions
What is a Markdown for Agents checker?
A Markdown for Agents checker tests whether a webpage provides a Markdown representation for automated clients, how that representation is discovered and whether it contains the same meaningful information as the original webpage.
How can I check if a webpage supports text/markdown?
Enter the URL in the checker above, or run curl -H "Accept: text/markdown" https://example.com/page. If the server responds with Content-Type: text/markdown and a Markdown body, the page supports Markdown content negotiation.
What is Accept: text/markdown?
Accept: text/markdown is an HTTP content-negotiation request. The client asks whether the same resource can be returned as Markdown instead of HTML. Servers that support it should also send Vary: Accept so caches serve the right version.
Is Markdown for Agents the same as llms.txt?
No. llms.txt is a site-level file that advertises important resources. Markdown for Agents is about the representation of each page itself. They are complementary: llms.txt can point to Markdown versions of your pages.
Can Tuurbo convert a webpage to Markdown?
Yes. When no publisher-provided Markdown is found, Tuurbo generates a clean fallback from the public webpage with a deterministic conversion — no AI rewriting. You can copy or download it.
Does Markdown improve Google rankings?
Markdown availability should not be treated as a direct Google ranking factor. Its main value is providing a cleaner representation for clients that support it.
Why does the checker say the website blocked access?
The website returned a WAF or CAPTCHA challenge, a 401/403 response, a 429 rate limit or a login wall. In that case the page could not be read, so the checker reports the restriction instead of claiming that Markdown is missing.
What is Markdown Content Fidelity?
Markdown Content Fidelity measures how much meaningful information from the original HTML webpage is preserved in its Markdown representation. A page can serve technically valid Markdown that still omits most of the real content.
Limitations: public webpages only · JavaScript-rendered pages may be partially analyzed · fidelity is a heuristic · Markdown support is not a ranking guarantee.