Skip to main content

HTTP Headers Inspector

What it does

The HTTP Headers Inspector fetches a URL and returns every response header organised into categories — SEO, Security, Caching, CORS, Server, Compression, and Content. The grouping makes it easy to scan for missing or misconfigured headers without wading through an unsorted list, and to spot the headers that matter for SEO, security posture, and edge caching at a glance.

Common situations

You are auditing a site’s security posture and need to verify the standard security headers (HSTS, CSP, X-Frame-Options) are in place. The inspector groups them under “Security” so missing ones are visible without comparing against a checklist.

A page is being rewritten by Google or behaving oddly in search and you suspect an X-Robots-Tag header is overriding the page’s meta robots. The inspector surfaces it in the SEO group; if X-Robots-Tag is set to noindex, you have your answer regardless of what the page’s HTML says.

You are debugging a CORS issue between your front-end and your API. The inspector shows the full Access-Control-* header set returned by the API, which is faster than digging through DevTools network panel.

A static asset (image, CSS, JS) is being requested on every page load instead of cached. Check the Cache-Control and ETag headers — usually the cache directive is no-cache, private, or missing entirely. The inspector groups them under “Caching” so the relevant headers are right there.

You are inspecting a competitor’s stack to understand what they are running. The Server header (or Cloudflare’s CF-Ray, AWS’s X-Cache, etc.) reveals their hosting and CDN layer. Useful for understanding why their performance differs from yours.

What you need to know

HTTP response headers carry metadata about the request and the response. They are invisible to most users but critical to how browsers, bots, and intermediaries (CDNs, proxies) handle the page. The inspector covers the categories that actually matter:

SEO headers:

  • X-Robots-Tag — header-level robots directives. Can noindex, nofollow, etc. for non-HTML resources or as an alternative to meta robots.
  • Link — when used with rel="canonical" or rel="alternate", can declare canonical or hreflang at the header level.
  • Content-Language — declares the page’s primary language.
  • Vary — tells caches that responses can vary based on the listed request headers.

Security headers (the modern security baseline):

  • Strict-Transport-Security (HSTS) — forces HTTPS for the configured duration.
  • Content-Security-Policy (CSP) — controls which scripts, styles, and resources can load.
  • X-Frame-Options — prevents the page from being framed (clickjacking protection).
  • X-Content-Type-Options: nosniff — prevents MIME type confusion attacks.
  • Referrer-Policy — controls what referrer information is sent on outbound links.
  • Permissions-Policy — controls browser feature access (camera, microphone, etc.).
  • Cross-Origin-* headers — newer headers controlling cross-origin behaviour.

Caching headers:

  • Cache-Control — the modern caching directive. public, max-age=31536000 means cache aggressively; no-cache, no-store, must-revalidate means don’t cache.
  • Expires — legacy absolute expiry time. Cache-Control supersedes it but both can coexist.
  • ETag — an opaque token for conditional requests; lets browsers ask “has this changed?” without re-downloading.
  • Last-Modified — when the resource was last modified. Used in conditional requests.

CORS headers for cross-origin API requests:

  • Access-Control-Allow-Origin — which origins can read the response.
  • Access-Control-Allow-Methods, Access-Control-Allow-Headers, etc.

Server / CDN identification:

  • Server — the web server software (nginx, Apache, etc.).
  • X-Powered-By — application stack hint.
  • CDN-specific: CF-Ray (Cloudflare), X-Cache (Varnish), Via, X-Served-By.

Compression:

  • Content-Encoding: gzip or br — the response body is compressed.
  • Transfer-Encoding: chunked — streamed response.

What the inspector doesn’t cover: response body analysis (use the Page SEO Audit for that), inspecting request-side headers (the inspector shows only what the server returns), and tracking header changes over time.

Frequently asked questions

Which security headers are required?

There is no formal “required” set, but the modern baseline is HSTS, CSP, X-Frame-Options, X-Content-Type-Options, and Referrer-Policy. Sites without these are flagged by automated security scanners and progressively penalised by browsers (some browser features are gated on certain headers being present).

What does X-Robots-Tag do?

Same directives as the meta robots tag, but at the HTTP header level. Useful for non-HTML resources (PDFs can’t have meta tags) and for setting robots rules at the server or CDN level instead of in the page template.

How is Cache-Control different from Expires?

Cache-Control is the modern directive; Expires is legacy. Cache-Control overrides Expires when both are present. Use Cache-Control for new configurations; you can leave Expires in place for backward compatibility but it adds noise.

What’s a good Cache-Control for HTML pages?

For pages that change occasionally: public, max-age=300 (5-minute cache) or longer. For frequently-updated pages: no-cache (always revalidate but allow caching). For sensitive pages: no-store (don’t cache at all). Static assets (images, CSS, JS) should be public, max-age=31536000, immutable with cache-busting filenames.

What does the Vary header do?

It tells caches that the response depends on listed request headers. Vary: Accept-Language means caches must serve different versions for different languages; Vary: User-Agent means desktop and mobile responses are cached separately. Use sparingly — every Vary value multiplies the cache size.

Why does my CDN not be respecting my Cache-Control?

CDNs often have their own cache rules that override origin headers. Cloudflare’s “Cache Everything” rule, for example, ignores origin Cache-Control. The fix is in the CDN configuration, not in the application headers.

Is X-Powered-By a security risk?

Mild — it discloses the application stack, which can help attackers fingerprint vulnerabilities. Removing it is a small security improvement; not removing it is not a major risk. Most modern frameworks remove it by default.

What’s the difference between X-Frame-Options and CSP frame-ancestors?

CSP frame-ancestors is the modern replacement. Both prevent clickjacking by controlling whether the page can be framed. Use both for compatibility; CSP frame-ancestors is more flexible but X-Frame-Options is universally supported.

Common problems

Problem: HSTS is set but the site is still loading over HTTP for some users.

HSTS is browser-cached after the first HTTPS visit. New users hitting the URL via HTTP have to be redirected to HTTPS first; only after the HSTS header is received do their browsers refuse plain HTTP. Combine HSTS with a server-side HTTP-to-HTTPS 301.

Problem: CSP is configured but breaks something on every page load.

CSP is strict by default — any script, style, or resource not explicitly allowed is blocked. The fix is to use Report-Only mode (Content-Security-Policy-Report-Only) first, log violations, build the allowlist incrementally, then switch to enforcing mode.

Problem: X-Robots-Tag set to noindex but the page is still in Google.

Index lag — Google has to recrawl the page, see the X-Robots-Tag, and update the index. Days to weeks for individual URLs. Use Search Console URL Inspection to expedite.

Problem: Cache-Control says max-age=86400 but pages are loading slowly on every request.

The CDN or browser isn’t honouring the directive. Common causes: Vary headers fragmenting the cache, query strings making each URL unique, or a CDN misconfiguration. Check the X-Cache header (HIT vs MISS) to confirm whether the CDN is caching at all.

Problem: Multiple security headers are set but a security scanner still rates the site low.

The scanner is probably checking for more recent additions (Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy, Permissions-Policy). The baseline has expanded over time. Check what the scanner is missing and add those headers.

Tips

  • HSTS, CSP, X-Frame-Options, X-Content-Type-Options, and Referrer-Policy are the modern security baseline. Sites missing them get flagged by automated audits.
  • Use Cache-Control over Expires for new work. Both can coexist; Cache-Control wins where there’s conflict.
  • The Server header reveals more than necessary. Most production deployments hide or generic-name it.
  • If you’re using a CDN, audit headers at the edge as well as at the origin — they often differ. The headers users see are the edge’s, not yours.
  • For SEO debugging, X-Robots-Tag is the header most often overlooked. If a page is being unexpectedly noindexed, check this header before assuming the meta tag is the issue.

Related tools in this suite

The Page SEO Audit reports the most-relevant headers as part of its overall report. The Redirect Chain Tracer is useful when redirect-related headers (Location, Status) need investigating.

What this looks like at scale

For a single URL, the inspector is fine. For a site, security and caching headers should be configured at the edge or web server layer (one place, applied site-wide) rather than per-page. The WordPress development service is where header configuration lands during a security or performance audit.

Take it further

If a site’s security headers are missing or misconfigured across many resources, the right scope is a security-headers configuration pass — usually at the CDN or web server config level. Talk through the audit and we can scope what configuration looks like.