14. DH crawl waste: redirects, 404s and errors — High (DH)
Crawl stats (same period): 22.9% of Googlebot requests return 301, 4.1% other 4XX, 2.5% 404, 0.85% 5XX; host status "Problems in the past".
Status (11 Nov 2026): ✅ AMP 301 fixed. Live check as Googlebot on 3 DH articles: <link rel="amphtml"> now uses plain slashes (e.g. /amp/story/india/bihar/…-4205712, no %2F) and returns 200 with no redirect. Still to confirm: the 301 count in Crawl stats (was ~130K/day) should fall over the next 1–2 weeks. The 404 and 5XX items below remain open.
Cause of the 301s (found, now fixed): every DH article's <link rel="amphtml"> points to a URL-encoded AMP path, e.g.
/amp/story/india%2Fkarnataka%2Freporting-crime-…-4070521. DH answers that encoded URL with 301 → /amp/story/india/karnataka/reporting-crime-…-4070521 (decoded slashes), which returns 200. So Googlebot fetches two URLs for every AMP version.
- Search Console 301 examples: 960 of 994 (97%) are
/amp/story/…URLs (shown decoded in the report). - Volume: avg ~130,000 301s/day (≈175K/day in late July → ≈90K/day by late October).
- PV still emits the encoded form (re-checked 11 Nov 2026 on 3 articles, e.g.
/amp/story/news%2Findia-news%2Fair-india-flight-hijack-threat-note-kolkata-mumbai-4387728). PV has no redirect, so there is no crawl waste like DH's, but both the encoded and the plain-slash AMP URL return 200 (both canonical to the HTML article) → two AMP URLs per article. 🛠️ Fix (PV): output theamphtmlhref with real slashes, as DH now does; optionally 301 the%2Fform to it. Status (11 Nov 2026): ⏸️ on hold — the change was pushed for DH only; PV to follow later.
Code (found in tpml-dh main, app/server/app.js ~lines 396–425, DH only — PV has no equivalent): an AMP middleware looks up every AMP request's story via the API and compares the raw request path to /amp/story/${storySlug}. The raw path contains %2F, so it never matches → 301 on every AMP hit. The same block calls next() after res.redirect(...), so the AMP handler keeps running on an already-sent response — a possible source of DH's 5XX errors (unverified).
🛠️ Fix (DH): compare the decoded path (decodeURIComponent(storyPathName)) and return res.redirect(...) without calling next(); or output the amphtml href with real slashes. Removes ~130K wasted Googlebot requests/day (~23% of DH crawl) plus one story-API call per AMP request. Also review whether AMP is still wanted at all.
404s (Crawl stats → By response → Not found, avg ~14,000/day, 2.5% of DH crawl) — 282 example URLs:
| Pattern | Share of examples | Example | Likely source / action |
|---|---|---|---|
| Image file name at site root | 151 (54%) | /file7zyi3v570kxscmiw57l.jpg, /pti07172022000107b-sixteennine.webp |
Relative image references in old/imported content (not found in 25 recent articles). 🔍 Find referring stories (search story text for the file names); fix or redirect to the CDN image. |
Legacy /content/<id> URLs |
50 (18%) | /content/337144/mysore-boy-tops-iit-jam.html, /amp/content/448850/… |
Old CMS article URLs with no redirect. 🛠️ Map /content/<id> → new story URL (301) where a match exists — recovers old backlinks. |
| AccessType API | 37 (13%) | /api/access/v1/set-reader-id?key=WVXWA5UYqHx2oGx3CZDBVeHa |
Another AccessType call made on every page (part of #4). |
| Other | 32 (11%) | /static/uploads/v/….mp4?t={seek_to_start_number}, story URLs with trailing junk (…-4173788ev) |
Unfilled video template placeholder; truncated/garbled links. |
| Legacy Drupal files | 6 | /https://www.deccanherald.com/sites/dh/files/… (absolute URL appended to path) |
Old image references with doubled host. |
| CDN path appended to page path | 4 | /india/media.assettype.com/deccanherald/…jpg |
Image URL missing https:// → resolved relative to the page. |
| Malformed Taboola URL | 2 | ///cdn.taboola.com/libtrc/…/loader.js |
Script URL with an extra slash somewhere. |
✅ 5XX explained (12 Nov 2026): an outage on AMP story URLs, 24 Aug – 1 Sep 2026 (~40,000 5XX/day; 980/984 samples /amp/story/…), matching Host status → Server connectivity (~25% failed, "high in the past"); ~0 since 4 Sep. It cut DH's crawl, which has not recovered (#3). 301 trend: 173,864/day (14–24 Aug) → 87,023/day (23 Oct – 9 Nov); 949 of 995 samples still /amp/story/…; the fix went live ~11 Nov, after this export's last day (9 Nov), so the drop isn't in this data yet. 404 trend: spike 25 Aug – 30 Sep (22,000–39,000/day), now ~4,000/day; samples: /api/access/… (37), legacy image paths, /amp/content/….
Files: crawl_stats/2026-10-23/dh-301/ (301), crawl_stats/2026-10-23/dh-404/ (404); 12 Nov: crawl_stats/2026-11-12/dh-5xx/, dh-301/, dh-404/.
16 Nov 2026 (Crawl stats "By file type", crawl_stats/2026-11-16/dh-*): DH "Other file type" requests fell from 161,526 (10 Nov) to 18,693 (11 Nov) and stay at 17–31K/day: consistent with the AMP 301s stopping on 11 Nov (redirects carry no file type). ⚠️ Same day, DH HTML response time to Googlebot rose from ~140 ms (1–10 Nov) to 245–271 ms (11–14 Nov) and has not come back: check what else shipped with the AMP fix (and whether AMP pages, now served directly, are slower or uncached). Follow up with the next Crawl stats export.
DH response-time jump on 11 Nov 2026: investigation (16 Nov 2026).
- Only code change that day:
3b847c28"changes for amp (#2097)", 11 Nov 13:20 IST, one line inapp/server/app.jsgenerateSeo:decodeAmpUrl: true(the #14 fix:amphtmlwith plain slashes). Next deploys:47a7e85e"Cwv issue" (13 Nov, 21 files incl.story-page-data.js,render-layout.js,layout.ejs) and22f4c0f0"Schema changes" (16 Nov,seo-hard-gate.js), both after the jump. No slower code path was added to page rendering on 11 Nov. - Crawl stats (DH HTML, daily): 10 Nov 577,786 requests, 164.7 KB/req, 155 ms → 11 Nov 337,578, 187.5 KB/req, 263 ms; 12–14 Nov 245–271 ms, 187–193 KB/req. Response time and bytes per request stepped up together on the day Googlebot started fetching AMP pages directly instead of
%2Fredirects (which were counted as "other"). - Live timing as Googlebot (16 Nov, 6 stories): first request median 666 ms main page (mostly Cloudflare MISS;
s-maxage=900) and 752 ms AMP; cached repeat ~310–330 ms. Forced origin (cache-busting query): ~1.0 s for both main and AMP pages, so AMP is not slower to build. - Reading: most likely a change in what Googlebot fetches (more full AMP pages, fewer cheap redirects, so fewer cache hits and more bytes per fetch), not a code regression. Not proven from our side. 🔍 Ask Quintype for Cloudflare cache-hit ratio and origin response time for Googlebot before/after 11 Nov; re-check with the next Crawl stats export.
- Found on the way: AMP responses carry
cache-control: public, max-age=60(nos-maxage; Cloudflare still holds them for hours via an edge rule); main pagess-maxage=900, so first fetches after 15 min hit a ~1 s origin. Heavy HTML: see #39.
📣 16 Nov 2026: emailed Quintype asking which data they can share for www.deccanherald.com, 1–16 Nov 2026: daily Cloudflare cache-hit ratio; split by page type (article / AMP / other); Googlebot-only filter; origin response times (median, p95); changes around 11 Nov (deploys, cache rules, purges); current cache settings; and to save the data before Cloudflare retention (~30 days) expires. PV asked as a baseline. Next: re-export DH Crawl stats ~19–20 Nov; chase reply.
Page indexing export (18 Nov 2026; checks/index_coverage/summary_dh_2026-11-18.md):
- Server error (5xx): 617 pages, left over from the August AMP outage. 481 were last crawled in Aug 2026 and 127 in Sep; 95% are AMP URLs (24 Aug – 1 Sep outage). The count jumped +3,115 on 22 Aug and has been falling since; nothing new. It clears as Google recrawls them.
- Page with redirect: 1,424,424 (+42,870 since 21 Aug, falling since 24 Oct: −2,647, −4,505, −1,134). Sample: 66% AMP URLs, consistent with the AMP 301s fixed on 11 Nov.
- Not found (404): 68,473, down from 123,214 (−54,741). Sample: old
/content/<id>URLs and/public/video/*.mp4files; no action. - Crawled – currently not indexed: 293,725, down from 406,103. Sample: mostly old stories (549 of 685 with a story number are from before Sep 2026); 3 Nov 2026 stories in it were indexed by 18 Nov (export lag).
- Duplicate, Google chose different canonical: 80,379 → new issue #43 (old URLs served with 200).
PV re-checked 19 Nov 2026: NOT live. Fresh pages straight from the server (Cloudflare MISS) still link AMP as /amp/story/<section>%2F<slug>, and that URL returns 200 (4 of 4 articles from the news sitemap). The fix (decodeAmpUrl: true, same one-line change as DH) exists only on branch amp-issue1 (5610ce4f, 26 Oct 2026), which is not merged into master (0b3f7227, 16 Nov).