4. AccessType token endpoint → 401 for Googlebot — High (PV)
What: https://www.prajavani.net/api/auth/v1/access-token/integrations/vMyvRVh97dcV4nDspcBBXyKq returns 401 Unauthorized to Googlebot ~14,000–27,000 times per day since 10 Aug 2026 (6.3% of all PV crawl requests; DH: 0%).
Cause: the site header wraps every page in Quintype <AccessType enableAccesstype={true}> (layouts/header/index.js ~line 838). On mount it runs initAccessType → runSequentialCalls → fetch(/api/auth/v1/access-token/integrations/<id>) for every visitor, logged in or not. Anonymous visitors and Googlebot get 401. Header change reached master in cf151636 (31 Aug 2026); 401s start 10 Aug (likely deployed before merge — confirm with release log).
Impact: wastes ~6–8% of Googlebot's PV crawl; loads AccessType script + failing request for every anonymous reader. Does not appear to block body rendering (live test rendered).
DH has the same issue: tpml-dh main header also renders <AccessType enableAccesstype={true}> (layouts/header/index.js:935). DH's endpoint /api/auth/v1/access-token/integrations/WVXWA5UYqHx2oGx3CZDBVeHa answers anonymous requests with 400 Bad Request (not 401), so it appears in DH Crawl stats as "Other client error (4XX)" = 4.1% of DH crawl requests (~23,000/day). Likely, not yet confirmed with a DH "By response → Other client error" export.
Fix: 🛠️ (both sites) render/initialise AccessType in the header only for logged-in users (as GlobalSubscriptionSyncer already does); keep it on subscribe/paywall flows. Stopgap: Disallow: /api/auth/ in robots.txt.
Verified 12 Nov 2026 (PV): Crawl stats → By response → Unauthorized (401/407), export 12 Nov (crawl_stats/2026-11-12/pv-401/): average 18,010/day (14 Aug – 24 Oct) → about 1/day (27 Oct – 9 Nov). The drop is on 23 Oct (22 Oct: 21,863; 23 Oct: 2), so Google picked up the block on 23 Oct, earlier than the 25 Oct date recorded below. Last sample 401 URL: 23 Oct 2026, 5:33 PM; all samples are the token URL. About 7–8% of PV's crawl is freed. Verified 12 Nov 2026 (DH): Crawl stats → By response → Other client error (4XX) (crawl_stats/2026-11-12/dh-4xx/): average 30,023/day (14 Aug – 22 Oct; higher than the ~23,000 estimated earlier) → about 15/day (23 Oct – 9 Nov). Same drop date as PV (22 Oct: 38,230; 23 Oct: 163). All 85 sample URLs (25 Aug – 23 Oct) are the token URL /api/auth/v1/access-token/integrations/WVXWA5UYqHx2oGx3CZDBVeHa, so DH's 4XX were entirely AccessType.
Status (11 Nov 2026): ✅ fixed on both sites by the robots.txt block (confirmed by the team; live since 25 Oct 2026, details below). The code fix is now optional: it only saves the failing call for anonymous readers.
Status (25 Oct 2026): ✅ robots.txt stopgap live on both sites and verified.
- Rule
Disallow: /api/auth/v1/access-token/integrations/is in theUser-agent: *group on www.prajavani.net (line 112) and www.deccanherald.com (line 107); no Googlebot-specific group overrides it. - Evaluated with Google's longest-match rule (the files also contain
Allow: /): token URLs on both sites → BLOCKED; article URLs and AMP URLs → still ALLOWED. - Not covered: DH
/api/access/v1/set-reader-id?key=…(404s, see #14) is still crawlable — considerDisallow: /api/access/as well. - Code fix (AccessType only for logged-in users) still with tech — the block stops Googlebot's requests, not the failing call for every anonymous reader.
- Verify over the next 1–2 weeks: Google caches robots.txt for up to ~24 h. In Search Console → Crawl stats → By response, PV "Unauthorized (401/407)" should fall from ~18,000/day to ~0, and DH "Other client error (4XX)" should drop by roughly 4% of requests. Search Console → Settings → robots.txt shows when Google last fetched the new file.
Verify: Crawl stats → By response → Unauthorized (401/407) drops to ~0. File:
crawl_stats/2026-10-23/pv-401/.
16 Nov 2026: the Crawl stats JSON sample (12 Nov 2026, crawl_stats/2026-11-16/pv-json/Table.csv) has 2 Googlebot fetches of /auth/accesstype-pv-token with 200 OK. 🔍 Check that the robots.txt rule covers this path (low volume).
Cause traced in code (16 Nov 2026; PV tpml upstream/master 70d7bebf and PR #1616 branch, fetched read-only as upstream/pr-1616):
- Mechanism: the site header wraps every page in Quintype
<AccessType enableAccesstype={true} accessTypeBkIntegrationId={ck.accesstype_subauth}>(app/isomorphic/components/layouts/header/index.js:843-850). The library starts on mount, for every visitor:componentDidMount → initAccessType → loadScript → runSequentialCalls → fetch("/api/auth/v1/access-token/integrations/<id>")(node_modules/@quintype/components/dist/es/components/access-type.js:1041-1043, 434-456, v3.8.8 as locked). Nothing checks for a logged-in user first, so anonymous readers and Googlebot get 401. The header also mountsGlobalSubscriptionSyncer(header/index.js:790), which wrapsAccessTypeagain (helper/GlobalSubscriptionSyncer.js:324-327). - Why 401s start 10 Aug although master changed on 31 Aug: master had no commits between 7 Jul (
b099bf5a;AccessTypeonly on the subscription page) and 31 Aug, when PR #1616 "Subscribe landing text" (branchsubscribe-landing-text, 485 commits from 10 Jul) was squash-merged ascf151636. On that branch the header gotGlobalSubscriptionSynceron 22 Jul 2026 (4529f009, "Changes for datalayer1") and the<AccessType>wrapper on 26 Jul 2026 (27f57833, "verification banner fixes"). The branch was evidently running in production by ~10 Aug, three weeks before the merge. Git cannot show the deploy date: 🔍 confirm in Quintype's deploy history which branch was live from ~10 Aug. - Second token call:
/auth/accesstype-pv-tokenis PV's own POST route (app/server/app.js:201, forwards to the SSOgetAccessTypePvToken), called byrequestAccessTypeToken(pages/subscription-page/auth-client.js:49) from the header/syncer subscription sync. It is not covered by the robots.txt rule; only 2 Googlebot fetches in the 12 Nov sample, so low priority. Optional:Disallow: /auth/accesstype-pv-token. - Code fix (unchanged): pass
enableAccesstype={isLoggedIn}(or mountAccessType/GlobalSubscriptionSyncerin the header only for logged-in users), keeping the subscribe/paywall pages as they are. The robots.txt block already stops Googlebot's 401s; the code fix removes the failing call for every anonymous reader.
SEO status (16 Nov 2026): resolved. Since the robots.txt block, Googlebot no longer requests the token endpoint (401s ~18,000/day → ~1/day). When Google renders a page the blocked call simply fails; story content does not depend on it (live render test fine). Remaining cost is not SEO: every anonymous reader still loads the AccessType script and makes one failing call. The code change is optional clean-up; severity for SEO: Low.