User-Agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /admin-login
Disallow: /dashboard
Disallow: /settings
Disallow: /rfp/
Disallow: /login

# 2026-10-02: the 09-14 group above is working. Over 14.3 hours of today's log, Amazonbot,
# ClaudeBot, Applebot, meta-externalagent, PetalBot, SemrushBot, DotBot and Bytespider made
# zero requests to the noindex detail pages. Four more agents were still spending their crawl
# there, and each one fetches this file, so a rule reaches them: MJ12bot 1,109 requests with
# 1,028 on noindex pages, LinkupBot 578 with 317, AhrefsBot 59 with 46, and gtm-ai-index, quiet
# in this window but material earlier and kept because crawler volume is bursty. Majestic and
# Ahrefs lose nothing they index: the detail pages carry noindex and are in no sitemap, and every
# real page, the vehicle hubs included, stays open to them so our backlink profile stays visible.
# 2026-09-14: the vehicle and vendor detail pages are noindex, not in any sitemap, and rendered on
# demand from the database. A handful of crawlers fetched them about 1,700 times an hour around the
# clock (36 hours: Amazonbot 38,810, ClaudeBot 25,956, Applebot 11,522, meta-externalagent 2,802,
# SemrushBot 1,570, PetalBot 1,282, Bytespider 616), about a third of all requests to the site, and
# real visitors queued behind those renders. These agents keep every indexable page, including the
# vehicle hubs; they lose only the noindex detail pages and the filtered vehicle lists. Google and
# Bing are not listed here on purpose: they must keep crawling the detail pages to see the noindex tag.
User-agent: Amazonbot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Applebot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: PetalBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: PerplexityBot
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: CCBot
User-agent: MJ12bot
User-agent: LinkupBot
User-agent: AhrefsBot
User-agent: gtm-ai-index
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /admin-login
Disallow: /dashboard
Disallow: /settings
Disallow: /rfp/
Disallow: /login
Disallow: /vehicles/CONT_
Disallow: /vehicles?
Disallow: /vendor/

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Host: https://bidsparq.com
Sitemap: https://bidsparq.com/sitemap.xml
Sitemap: https://bidsparq.com/sitemap-pseo.xml