Technical SEO Checklist: Crawlability, Indexing and Page Experience

Most business websites do not lose search visibility because of a weak keyword strategy. They lose it because a crawler cannot reach a page, a canonical tag points somewhere unexpected, or a template ships a hero image that takes four seconds to load on a mid-range Android phone. Technical SEO is unglamorous plumbing, and it is the part of search work that breaks quietly after a redesign, a domain migration or a plugin update.

This checklist is deliberately narrow. It covers the three areas that decide whether everything else has a chance to work: crawlability, indexing and page experience. Every item can be verified with free tooling — Google Search Console, Bing Webmaster Tools, the URL Inspection tool, Lighthouse and an ordinary browser.

1. Crawlability: can a bot reach the page at all?

Crawlability is about access, not ranking. If a bot cannot fetch a URL, nothing later in this article matters.

robots.txt

• Keep the file minimal and block only what genuinely must stay out: admin paths, internal search results, and faceted parameter patterns that inflate crawl space.
• Never block the CSS or JavaScript files a rendering engine needs to understand the page.
• Test every change before deploying. A stray Disallow: / has taken more sites out of search than any algorithm update.

XML sitemaps

• Use a sitemap index that references section-level sitemaps. Each individual sitemap is capped at 50,000 URLs and 50 MB uncompressed under the sitemap protocol.
• Include only canonical, indexable URLs returning a 200 status. A sitemap full of redirects and noindexed pages trains everyone to ignore the report.
• Reference the sitemap in robots.txt, submit it in Search Console, and review the discovered-but-not-indexed bucket on a schedule.

Status codes and redirects

• Every URL should return exactly one of 200, 301, 302 (sparingly), 404 or 410. Soft 404s — templates that return 200 while displaying an empty or no-results state — are among the most common hidden problems on e-commerce and listing pages.
• Flatten redirect chains. A to B to C to D wastes crawl budget and dilutes signals; point everything at the final destination in a single hop.
• Audit internal links after any URL change. Stale links to redirected URLs linger in navigation, footers and old blog posts for years.

Internal linking and crawl depth

• Commercial and informational pages that matter should be reachable within a few clicks of the homepage. Orphan pages that exist only inside the sitemap get crawled late, if at all.
• Use descriptive anchor text and real a href links. Buttons wired to JavaScript click handlers are not links, and crawlers will not follow them.

2. Indexing: is the right version of the page eligible to appear?

Canonicalisation

• Each page should carry a self-referencing canonical unless it is a deliberate duplicate. Use absolute URLs, not relative paths that resolve unpredictably across templates.
• Check for conflicting signals. A page with a noindex tag and a canonical pointing elsewhere is giving two instructions at once.

Noindex versus robots.txt

A robots.txt disallow prevents crawling, which means the noindex tag is never read. If the goal is removal from the index, allow crawling and use noindex, or delete the page and return a 404 or 410.

Duplicate and near-duplicate URLs

• Parameter URLs for sorting and filtering, tracking parameters, www versus non-www, http versus https and trailing-slash variants all create duplicates. Pick one form and redirect the others permanently.
• For paginated category pages, keep each page indexable and self-canonical. Do not point page two at page one.

JavaScript rendering

If your catalogue, pricing or article body only appears after client-side JavaScript runs, verify what actually renders rather than assuming. Server-side rendering or static generation removes a whole class of risk. Use the rendered HTML view in the URL Inspection tool as the reference, and check that key content is not gated behind an interaction.

Structured data and language signals

• Add schema.org markup for Organisation, LocalBusiness with accurate address and hours, Product and BreadcrumbList where relevant, then validate with the Rich Results Test. Markup must match what users see on the page.
• For multilingual or multi-region sites, implement hreflang with reciprocal return links and a sensible x-default.

Search Console hygiene

• Review the Page Indexing report and decide, page by page, whether each exclusion is intentional.
• Verify a domain property and keep ownership documentation with the business, not only with an agency. Access that lapses when a contract ends is a recurring and avoidable problem.

3. Page experience: what happens once someone lands

Page experience is not a single score. It is a set of measurable signals: Core Web Vitals, mobile usability, HTTPS and freedom from intrusive interstitials. Google publishes thresholds that make a reasonable target: Largest Contentful Paint within 2.5 seconds, Interaction to Next Paint under 200 milliseconds and Cumulative Layout Shift below 0.1, assessed at the 75th percentile of real user experiences.

Largest Contentful Paint

• Identify the LCP element for each template — usually the hero image or the main heading — and never lazy-load it.
• Serve modern formats such as WebP or AVIF at the correct dimensions, preload the hero asset, and cut render-blocking CSS and font requests.
• Time to first byte is part of LCP. Slow shared hosting or an uncached content management system caps your ceiling; page caching and a CDN are usually the cheapest wins available.

Interaction to Next Paint

• INP replaced First Input Delay as a Core Web Vital, and it is unforgiving of heavy JavaScript. Audit third-party tags: chat widgets, heatmaps, A/B testing tools, ad pixels and click-to-chat scripts.
• Break up long tasks, defer non-critical scripts, and remove tags nobody has reviewed in a year.

Cumulative Layout Shift

• Reserve space for images, ads, embeds and cookie banners using explicit dimensions or aspect ratios.
• Avoid injecting promotional banners above existing content after load, which is a common cause of a poor shift score on news and offer pages.

Mobile and trust basics

• Test on a mid-range Android phone over a throttled connection, not just a desktop browser on office Wi-Fi.
• Serve HTTPS site-wide with no mixed content, and enable HSTS once you are confident the certificate setup is stable.
• Keep interstitials small and dismissible. Full-screen popups on entry are a page-experience problem, particularly on mobile.

4. A sane order of operations

Run your own crawl with a tool such as Screaming Frog, then compare the result against Search Console index coverage. Fix status codes, redirect chains and robots.txt first — these are cheap and they unblock everything downstream. Move to canonicalisation and duplicate URL patterns next. Then tackle template-level performance, starting with the templates that carry the most traffic and revenue. Re-check after every release, because technical SEO is a regression test rather than a one-time project.

None of this requires a large budget. It requires someone with access to the server, the templates and Search Console sitting down with a checklist before each deployment, rather than after traffic has already slipped.