← Issues index

14. DH crawl waste: redirects, 404s and errors — High (DH)

Crawl stats (same period): 22.9% of Googlebot requests return 301, 4.1% other 4XX, 2.5% 404, 0.85% 5XX; host status "Problems in the past".

Status (11 Nov 2026): ✅ AMP 301 fixed. Live check as Googlebot on 3 DH articles: <link rel="amphtml"> now uses plain slashes (e.g. /amp/story/india/bihar/…-4205712, no %2F) and returns 200 with no redirect. Still to confirm: the 301 count in Crawl stats (was ~130K/day) should fall over the next 1–2 weeks. The 404 and 5XX items below remain open.

Cause of the 301s (found, now fixed): every DH article's <link rel="amphtml"> points to a URL-encoded AMP path, e.g. /amp/story/india%2Fkarnataka%2Freporting-crime-…-4070521. DH answers that encoded URL with 301 → /amp/story/india/karnataka/reporting-crime-…-4070521 (decoded slashes), which returns 200. So Googlebot fetches two URLs for every AMP version.

Code (found in tpml-dh main, app/server/app.js ~lines 396–425, DH only — PV has no equivalent): an AMP middleware looks up every AMP request's story via the API and compares the raw request path to /amp/story/${storySlug}. The raw path contains %2F, so it never matches → 301 on every AMP hit. The same block calls next() after res.redirect(...), so the AMP handler keeps running on an already-sent response — a possible source of DH's 5XX errors (unverified).

🛠️ Fix (DH): compare the decoded path (decodeURIComponent(storyPathName)) and return res.redirect(...) without calling next(); or output the amphtml href with real slashes. Removes ~130K wasted Googlebot requests/day (~23% of DH crawl) plus one story-API call per AMP request. Also review whether AMP is still wanted at all. 404s (Crawl stats → By response → Not found, avg ~14,000/day, 2.5% of DH crawl) — 282 example URLs:

Pattern Share of examples Example Likely source / action
Image file name at site root 151 (54%) /file7zyi3v570kxscmiw57l.jpg, /pti07172022000107b-sixteennine.webp Relative image references in old/imported content (not found in 25 recent articles). 🔍 Find referring stories (search story text for the file names); fix or redirect to the CDN image.
Legacy /content/<id> URLs 50 (18%) /content/337144/mysore-boy-tops-iit-jam.html, /amp/content/448850/… Old CMS article URLs with no redirect. 🛠️ Map /content/<id> → new story URL (301) where a match exists — recovers old backlinks.
AccessType API 37 (13%) /api/access/v1/set-reader-id?key=WVXWA5UYqHx2oGx3CZDBVeHa Another AccessType call made on every page (part of #4).
Other 32 (11%) /static/uploads/v/….mp4?t={seek_to_start_number}, story URLs with trailing junk (…-4173788ev) Unfilled video template placeholder; truncated/garbled links.
Legacy Drupal files 6 /https://www.deccanherald.com/sites/dh/files/… (absolute URL appended to path) Old image references with doubled host.
CDN path appended to page path 4 /india/media.assettype.com/deccanherald/…jpg Image URL missing https:// → resolved relative to the page.
Malformed Taboola URL 2 ///cdn.taboola.com/libtrc/…/loader.js Script URL with an extra slash somewhere.

✅ 5XX explained (12 Nov 2026): an outage on AMP story URLs, 24 Aug – 1 Sep 2026 (~40,000 5XX/day; 980/984 samples /amp/story/…), matching Host status → Server connectivity (~25% failed, "high in the past"); ~0 since 4 Sep. It cut DH's crawl, which has not recovered (#3). 301 trend: 173,864/day (14–24 Aug) → 87,023/day (23 Oct – 9 Nov); 949 of 995 samples still /amp/story/…; the fix went live ~11 Nov, after this export's last day (9 Nov), so the drop isn't in this data yet. 404 trend: spike 25 Aug – 30 Sep (22,000–39,000/day), now ~4,000/day; samples: /api/access/… (37), legacy image paths, /amp/content/…. Files: crawl_stats/2026-10-23/dh-301/ (301), crawl_stats/2026-10-23/dh-404/ (404); 12 Nov: crawl_stats/2026-11-12/dh-5xx/, dh-301/, dh-404/.

16 Nov 2026 (Crawl stats "By file type", crawl_stats/2026-11-16/dh-*): DH "Other file type" requests fell from 161,526 (10 Nov) to 18,693 (11 Nov) and stay at 17–31K/day: consistent with the AMP 301s stopping on 11 Nov (redirects carry no file type). ⚠️ Same day, DH HTML response time to Googlebot rose from ~140 ms (1–10 Nov) to 245–271 ms (11–14 Nov) and has not come back: check what else shipped with the AMP fix (and whether AMP pages, now served directly, are slower or uncached). Follow up with the next Crawl stats export.

DH response-time jump on 11 Nov 2026: investigation (16 Nov 2026).

📣 16 Nov 2026: emailed Quintype asking which data they can share for www.deccanherald.com, 1–16 Nov 2026: daily Cloudflare cache-hit ratio; split by page type (article / AMP / other); Googlebot-only filter; origin response times (median, p95); changes around 11 Nov (deploys, cache rules, purges); current cache settings; and to save the data before Cloudflare retention (~30 days) expires. PV asked as a baseline. Next: re-export DH Crawl stats ~19–20 Nov; chase reply.

Page indexing export (18 Nov 2026; checks/index_coverage/summary_dh_2026-11-18.md):

PV re-checked 19 Nov 2026: NOT live. Fresh pages straight from the server (Cloudflare MISS) still link AMP as /amp/story/<section>%2F<slug>, and that URL returns 200 (4 of 4 articles from the news sitemap). The fix (decodeAmpUrl: true, same one-line change as DH) exists only on branch amp-issue1 (5610ce4f, 26 Oct 2026), which is not merged into master (0b3f7227, 16 Nov).