← Issues index

4. AccessType token endpoint → 401 for Googlebot — High (PV)

What: https://www.prajavani.net/api/auth/v1/access-token/integrations/vMyvRVh97dcV4nDspcBBXyKq returns 401 Unauthorized to Googlebot ~14,000–27,000 times per day since 10 Aug 2026 (6.3% of all PV crawl requests; DH: 0%).

Cause: the site header wraps every page in Quintype <AccessType enableAccesstype={true}> (layouts/header/index.js ~line 838). On mount it runs initAccessType → runSequentialCalls → fetch(/api/auth/v1/access-token/integrations/<id>) for every visitor, logged in or not. Anonymous visitors and Googlebot get 401. Header change reached master in cf151636 (31 Aug 2026); 401s start 10 Aug (likely deployed before merge — confirm with release log).

Impact: wastes ~6–8% of Googlebot's PV crawl; loads AccessType script + failing request for every anonymous reader. Does not appear to block body rendering (live test rendered).

DH has the same issue: tpml-dh main header also renders <AccessType enableAccesstype={true}> (layouts/header/index.js:935). DH's endpoint /api/auth/v1/access-token/integrations/WVXWA5UYqHx2oGx3CZDBVeHa answers anonymous requests with 400 Bad Request (not 401), so it appears in DH Crawl stats as "Other client error (4XX)" = 4.1% of DH crawl requests (~23,000/day). Likely, not yet confirmed with a DH "By response → Other client error" export.

Fix: 🛠️ (both sites) render/initialise AccessType in the header only for logged-in users (as GlobalSubscriptionSyncer already does); keep it on subscribe/paywall flows. Stopgap: Disallow: /api/auth/ in robots.txt.

Verified 12 Nov 2026 (PV): Crawl stats → By response → Unauthorized (401/407), export 12 Nov (crawl_stats/2026-11-12/pv-401/): average 18,010/day (14 Aug – 24 Oct) → about 1/day (27 Oct – 9 Nov). The drop is on 23 Oct (22 Oct: 21,863; 23 Oct: 2), so Google picked up the block on 23 Oct, earlier than the 25 Oct date recorded below. Last sample 401 URL: 23 Oct 2026, 5:33 PM; all samples are the token URL. About 7–8% of PV's crawl is freed. Verified 12 Nov 2026 (DH): Crawl stats → By response → Other client error (4XX) (crawl_stats/2026-11-12/dh-4xx/): average 30,023/day (14 Aug – 22 Oct; higher than the ~23,000 estimated earlier) → about 15/day (23 Oct – 9 Nov). Same drop date as PV (22 Oct: 38,230; 23 Oct: 163). All 85 sample URLs (25 Aug – 23 Oct) are the token URL /api/auth/v1/access-token/integrations/WVXWA5UYqHx2oGx3CZDBVeHa, so DH's 4XX were entirely AccessType.

Status (11 Nov 2026): ✅ fixed on both sites by the robots.txt block (confirmed by the team; live since 25 Oct 2026, details below). The code fix is now optional: it only saves the failing call for anonymous readers.

Status (25 Oct 2026): ✅ robots.txt stopgap live on both sites and verified.

16 Nov 2026: the Crawl stats JSON sample (12 Nov 2026, crawl_stats/2026-11-16/pv-json/Table.csv) has 2 Googlebot fetches of /auth/accesstype-pv-token with 200 OK. 🔍 Check that the robots.txt rule covers this path (low volume).

Cause traced in code (16 Nov 2026; PV tpml upstream/master 70d7bebf and PR #1616 branch, fetched read-only as upstream/pr-1616):

SEO status (16 Nov 2026): resolved. Since the robots.txt block, Googlebot no longer requests the token endpoint (401s ~18,000/day → ~1/day). When Google renders a page the blocked call simply fails; story content does not depend on it (live render test fine). Remaining cost is not SEO: every anonymous reader still loads the AccessType script and makes one failing call. The code change is optional clean-up; severity for SEO: Low.