# VeritaHire robots.txt — ALLOW-BY-DEFAULT (Rob 2026-07-30: "just switch robots to allow all for now").
#
# WHY THIS SHAPE: this file used to be an ALLOWLIST — "Disallow: /" plus a hand-maintained Allow line
# per path. That failed twice in three days, silently and expensively:
#   * /style.css was never listed, so Googlebot was refused the stylesheet on EVERY page and judged
#     the whole site unstyled (07-29);
#   * /verify.php and /reports/ shipped in the sitemap before their Allow lines existed — the exact
#     self-contradiction Search Console flags (07-29/30).
# In an allowlist, the default outcome of forgetting a line is INVISIBLE TO SEARCH. That default is
# backwards for a site whose whole promise is getting jobs indexed. Now the default is ALLOW, and only
# genuinely private or functional surfaces are named below. No "Allow:" lines are needed or present —
# in robots.txt allow IS the default, and Allow only exists to carve exceptions out of a Disallow.
#
# ⚠️ FIXED 2026-07-30 (second pass): the first inverted version put a BLANK LINE between "User-agent: *"
# and the Disallow list. A blank line TERMINATES a record, so every Disallow below it belonged to no
# user-agent group and was discarded — urllib.robotparser read the file and extracted ZERO rules, and
# /me.php, /admin.php, /manage.php and /api/ were all crawlable. Keep this record CONTIGUOUS: comments
# are fine inside it, blank lines are not. Verify after any edit by parsing the file, never by reading it.
#
# RULE GOING FORWARD: robots.txt controls CRAWLING, not indexing. If a page should exist but not rank,
# let it be crawled and give it <meta name="robots" content="noindex"> — a page blocked here can never
# be crawled, so Google can never SEE the noindex and may index it anyway from external links.
User-agent: *
# Private candidate surfaces — someone's own account, résumé and matches. Nothing here is public.
# 2026-09-08: the PARAMETERISED search form only. .htaccess answers cold /search.php?q=
# with 429 (anti-scraper gate), and robots.txt being allow-by-default was inviting
# Google to request exactly those URLs - a 429 to Googlebot is a site-wide back-off
# signal. Deliberately NOT `Disallow: /search.php`: the bare page IS linked from real
# pages and already serves `noindex,nofollow`, so blocking it would stop Google reading
# that noindex and leave it free to index the bare URL from anchor text alone.
# search.php is ALLOWED. Rob 2026-09-21: "why would we disallow search.php? That is ridiculous - I told
# you specifically to allow it?" The 09-08 line (Disallow: /search.php?) added by laptop-1 without asking blocked every parameterised
# search page Google had already indexed and produced the GSC "indexed, though blocked by robots.txt"
# warning. Removed 2026-09-21 (laptop-1). Backup: robots.txt.pre-searchallow-20260921.
Disallow: /me.php
Disallow: /jobs.php
Disallow: /saved.php
Disallow: /settings.php
Disallow: /profile.php
Disallow: /resumes.php
Disallow: /resume_view.php
Disallow: /resume_dl.php
Disallow: /resume_build.php
Disallow: /packet.php
Disallow: /package_build.php
Disallow: /messages.php
Disallow: /notifications.php
Disallow: /develop.php
Disallow: /upload.php
Disallow: /onboard.php
Disallow: /matrix.php
# Employer console — reached by a private manage token; crawling it is meaningless and leaks tokens.
Disallow: /manage.php
Disallow: /manage-help.php
Disallow: /manage-resume.php
Disallow: /managearea.php
Disallow: /managefilters.php
Disallow: /manageskillpref.php
Disallow: /addjob.php
# Admin, auth, machine endpoints and internals.
Disallow: /admin.php
Disallow: /ops.php
Disallow: /auth_google.php
Disallow: /auth_link.php
Disallow: /login.php
Disallow: /logout.php
# /join.php?job= and ?q= were blocked here from 08-27 (laptop-3: an infinite url space, one per job and per search).
# The block hid this page's noindex from Google, which then indexed the bare urls from anchor text alone
# ("Indexed, though blocked by robots.txt", GSC 09-21). Rob 2026-09-21: the guest doors are POST forms now
# (_uihelpers.php vh_door_form), so no crawler ever sees a parameterised join url, and the block is REMOVED so
# Google can re-read the known ones, see noindex,nofollow, and drop them. Backup: robots.txt.pre-joinallow-20260921.
Disallow: /unsub.php
Disallow: /csp-report.php
# ⚠️ EXCEPTION — DO NOT REMOVE (2026-08-19). /api/jobs-jsonld.php is the data endpoint the PostOnce
# EMBED calls: an employer puts <div id="veritahire-jobs" data-employer="slug"> + /embed/jobs-loader.js
# on their own careers page, the loader fetches this endpoint and injects the JobPosting JSON-LD.
# Blanket "Disallow: /api/" broke that silently and completely. MEASURED: Googlebot crawled
# phantomales.com/careers.php 11x in August and DID render it (fetched /embed/jobs-loader.js at
# 11/Aug 19:01:09, referer that page) -- but never once fetched this endpoint, because this file
# forbade it. Result: the employer's careers page renders jobs for humans and NOTHING for Google, so
# no PostOnce job can ever reach Google Jobs through the employer's own domain.
# This is the THIRD time a blanket rule here has starved the crawler of a resource the page needs
# (style.css 07-29, /verify.php 07-30). The data is already public on the job page -- nothing is
# exposed by allowing it. Allow comes FIRST: longest-match wins, so it carves this one path out.
Allow: /api/jobs-jsonld.php
# Benchmark API read endpoints (laptop-3 2026-10-01): answer-time assistants fetch these for users; /api/ stays closed otherwise.
Allow: /api/benchmark.php
Allow: /api/job-status.php
Allow: /api/jobs-search.php
Allow: /api/job.php
Disallow: /api/
Disallow: /webhook/
# 2026-09-08 (laptop-2): /data/ became the Verified Jobs Feed PRODUCT page (job-data project) with a
# search-written title/description, and this blanket block made it uncrawlable - the same orphan
# shape /verify.php and /reports/ had. Longest-match wins, so these three carve the product surface
# out: the page itself, the API doc, and the public sample files (Google Dataset Search reads them).
# Everything else under /data/ (vocab JSON, the Apify source zip) stays blocked.
# The blanket Disallow: /data/ is GONE (laptop-2 06:58Z): GSC live test still said "blocked" from Google's
# cached copy, and a rule that only works because Allow /data/$ out-lengths Disallow /data/ is a rule
# that fails on any parser that scores it differently. Name the files to hide instead; the product
# page, doc and samples then need no exception at all.
Disallow: /data/us_cities.json
Disallow: /data/job_titles.json
Disallow: /data/onboard_vocab.json
Disallow: /data/apify-verified-jobs-feed.zip
Disallow: /data/index.php
Disallow: /preview/
Disallow: /_review_bcc9d6af/
# /search.php is CRAWLABLE ON PURPOSE (laptop-5 2026-08-01). It serves
# <meta name="robots" content="noindex,nofollow">, and this file's own rule above says it: robots.txt
# controls CRAWLING, not indexing -- a page blocked here can never be crawled, so a crawler can never
# SEE the noindex and keeps whatever stale record it already holds. Bing was still listing it for
# exactly that reason. Disallowing it was self-defeating: it PRESERVED the listing it meant to remove.
# The original worry here was facet-crawl load on the infinite query space. That is handled by the
# NOFOLLOW half of the tag -- crawlers do not follow links out of the page into the query space -- and
# noindex additionally makes Google crawl it less over time. Do NOT re-add a Disallow: it would freeze
# whatever is in the index at that moment, permanently.

# ── AI TRAINING CRAWLERS: BLOCKED 2026-08-10 ───────────────────────────────────────────────────────
# MEASURED from /usr/local/apache/domlogs over an 11.5-hour window (12:03-23:39, 10 Aug):
#     470,450 requests total
#     ClaudeBot   285,718  (61%)
#     GPTBot       56,136  (12%)   -> 341,854 combined = ~8.3 requests/SECOND, sustained
#   of which:  /j.php 151,894 · /join.php 93,820 · /search.php 44,950
#
# THIS WAS THE LOAD. The box ran fine on 8 GB a week ago; RAM was doubled, the database was halved,
# harvest was cut to 2x/day and job count was flat — yet it was swamped. None of that reconciled
# until the access log was read: the pressure was never the pipeline. /j.php fires two related-jobs
# queries per view and /search.php runs fulltext, so 8 req/s on those pages IS the mariadbd CPU.
#
# WHY BLOCK THESE AND NOTHING ELSE: they scrape for model training. They send no seekers, no
# candidates, no indexed listings. Googlebot and bingbot — the crawlers that actually deliver jobs to
# people — are NOT in the top agents and are untouched by this. 93,820 crawler hits on /join.php is a
# bot repeatedly loading a SIGNUP FORM, which is also the surface the email-abuse defence exists for.
#
# They honour robots.txt (10 fetches of this file in the window), so this is a config fix.
#
# SEPARATE RECORDS, each contiguous. The blank line BETWEEN records is required — it is what
# separates them. The 07-30 incident was a blank line INSIDE a record, which terminated it early and
# discarded every rule below. Not the same thing. Verify by PARSING, never by reading.
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# AhrefsBot BLOCKED 2026-09-16 (Rob: "disallow AhrefsBot"). Measured that day on
# domlogs/veritahire.com-ssl_log: 14,444 requests in one day, the 3rd-largest agent on the site,
# against 1 (one) Googlebot request. Ahrefs is a paid SEO-tool crawler: it indexes our corpus to
# sell backlink reports to other people. It sends no seekers and no candidates, and we are not its
# customer, so every one of those renders was cost with no return. It DOES honour robots.txt and it
# fetched this file the same day, so a Disallow is sufficient - no Apache rule needed.
# NOTE: SemrushBot, DataForSeoBot, DotBot and MJ12bot are the same category and are NOT yet blocked
# here; they are already in the .htaccess 429 list but /j/ is exempted from it.
User-agent: AhrefsBot
Disallow: /

Sitemap: https://veritahire.com/sitemaps/sitemap-index.xml
Sitemap: https://veritahire.com/sitemap.xml