Technical SEO Audit Part 2: Crawling — How to Make Sure Google Can Find Every Page on Your Website
📘 Part 2 of 7 — Technical SEO Audit Masterclass
Series Progress
1 · Introduction 2 · Crawling 3 · Rendering 4 · Indexing 5 · Core Web Vitals 6 · Schema 7 · Site Structure
Crawling — Making Sure Google Can Find Your Pages
Before Google can rank your website, it must first find your pages. This process is called crawling. Google sends out small programs called bots (also called spiders or crawlers) that visit your website, read the content, and follow all the links.
If Google cannot crawl your pages, nothing else matters — your content will never appear in search results no matter how good it is.
📦 ANALOGY
Think of Google's crawler like a delivery person. Your website is a building. If the doors are locked (robots.txt blocking), the address is wrong (bad URLs), or there is no map (no sitemap) — the delivery person gives up and leaves.
1. How to Audit Your robots.txt File
The robots.txt file sits at the root of your website and tells search bots which pages to visit and which to skip.
Where to find it: Open your browser and go to yourwebsite.com/robots.txt — example: a2itsoft.com/robots.txt
What a good robots.txt looks like
# Allow all bots to crawl everythingUser-agent: *Allow: /# Block private pagesDisallow: /admin/Disallow: /login/Disallow: /public/items/Disallow: /api/# Allow CSS and JS for GooglebotUser-agent: GooglebotAllow: /*.cssAllow: /*.js# Sitemap locationSitemap: https://www.a2itsoft.com/sitemap.xml
⚠ WRONG
Disallow: / — This single line blocks Google from your ENTIRE website. It is the most dangerous mistake in SEO.
Common robots.txt mistakes
| Mistake | What happens | Fix |
|---|---|---|
| Disallow: / | Site disappears from Google search | Change to Allow: / |
| No Sitemap line | Google has to guess where your pages are | Add Sitemap: https://yoursite.com/sitemap.xml |
| Blocking CSS/JS files | Google cannot render your pages properly | Add Allow: /*.css and Allow: /*.js |
| Blocking /public/items/* | Private URLs stay out of index (CORRECT) | Keep this — it is good for privacy |
💡 TASK
Open a2itsoft.com/robots.txt right now. Check: (1) Is homepage allowed? (2) Is /public/items/ blocked? (3) Is there a Sitemap line? This is your first real audit task.
2. XML Sitemap Best Practices — 50,000 URL & 50MB Limits
A sitemap lists all important pages on your website. It guides Google to every page you want indexed — especially pages not easily found through links.
| Limit | Value | If exceeded |
|---|---|---|
| URLs per sitemap file | 50,000 URLs | Google ignores the file — split into multiple sitemaps |
| Max file size | 50 MB uncompressed | Compress with .gz or split the file |
| Sitemap index files | 500 sitemaps | Use a sitemap index to link all together |
Good XML sitemap example
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://www.a2itsoft.com/</loc> <lastmod>2026-05-01</lastmod> <changefreq>weekly</changefreq> <priority>1.0</priority> </url> <url> <loc>https://www.a2itsoft.com/internship</loc> <lastmod>2026-04-20</lastmod> <priority>0.8</priority> </url> </urlset>
- Only include pages you want Google to index — never include login, admin, or /public/items/ pages.
- Submit in Google Search Console → Sitemaps → Add a new sitemap.
- Use Yoast SEO or Rank Math on WordPress — they auto-generate sitemaps.
- Update lastmod only when content actually changes — faking dates reduces Google's trust.
⭐ PRO TIP
Submit your sitemap in robots.txt AND Google Search Console. Two ways for Google to find it means faster crawling of your pages.
3. Indexable vs Non-Indexable URLs
Just because a page exists does not mean Google will index it. Some are intentionally blocked. Others are accidentally blocked — and that is a serious SEO problem.
| Reason | How it happens | Fix |
|---|---|---|
| noindex tag in HTML | Developer added meta noindex tag | Remove the noindex tag |
| Blocked by robots.txt | Disallow rule in robots.txt | Add Allow rule or remove Disallow |
| Canonical pointing elsewhere | Page points to different URL as original | Check and correct the canonical tag |
| Soft 404 error | Page shows no results but returns 200 status | Return proper 404 or fix the page content |
| Orphan page | No internal links pointing to this page | Add internal links from other pages |
📌 REAL
A student found their website had 200 pages but only 12 were indexed. The developer had added noindex to the entire site during development and forgot to remove it before launch. One tag — 188 invisible pages.
How to find them in Google Search Console
- Go to Google Search Console → Pages (Coverage) report.
- Click the Excluded tab — these are pages Google chose NOT to index.
- Look for: "Excluded by noindex tag", "Crawled but not indexed", "Discovered but not crawled".
- Fix each issue based on the reason shown.
4. Canonical Tags and Duplicate Content
Duplicate content happens when the same content appears on more than one URL. Google gets confused about which version to rank — and often ranks none of them.
How duplicate content happens
- HTTP vs HTTPS: http://a2itsoft.com and https://a2itsoft.com — same content, two URLs.
- WWW vs non-WWW: www.a2itsoft.com and a2itsoft.com — two URLs, same page.
- Trailing slash: /internship and /internship/ — Google treats these differently.
- URL parameters: /courses?sort=price and /courses — same page, different URLs.
How to add a canonical tag
<!-- Add inside <head> section of your page --><link rel="canonical" href="https://www.a2itsoft.com/internship" /><!-- For a filtered/parameterised page --><!-- Add this to: /courses?sort=price --><link rel="canonical" href="https://www.a2itsoft.com/courses" />
💡 TIP
In WordPress, Yoast SEO and Rank Math add canonical tags automatically. In custom PHP or Node.js sites, you must add them manually to every page template.
5. Crawl Budget for Large Sites (50,000+ URLs)
Every website gets a crawl budget — the number of pages Google will crawl per day. For small sites this is not a problem. For large sites with 50,000+ URLs, it becomes critical.
📖 DEF
Crawl budget = the number of URLs Google crawls on your website per day. Waste it on unimportant pages and Google may never reach your most important content.
What wastes your crawl budget?
- Faceted navigation URLs like /courses?sort=price&category=python — creates thousands of duplicate pages.
- Session IDs in URLs — /page?sessionid=abc123 — creates a new URL for every visitor.
- /public/items/ URLs — private order pages that serve no SEO purpose.
- Duplicate HTTP/HTTPS versions not redirected to one version.
- Soft 404 pages that return 200 status code.
⭐ PRO TIP
Google gives more crawl budget to fast-loading websites. Improving Core Web Vitals helps Google crawl more pages per day — two benefits from one fix.
6. Log File Analysis — Detect Spam Bots and AI Crawlers
Server log files record every visit to your website — including every bot. Log file analysis is one of the most powerful and most underused SEO techniques.
What a log file entry looks like
66.249.66.1 [05/May/2026:14:32:01]"GET /internship HTTP/1.1" 200User-Agent: Googlebot/2.1192.168.1.100 [05/May/2026:14:33:45]"GET /public/items/J973789556 HTTP/1.1" 200User-Agent: SemrushBot/7
| Bot in logs | Who it is | Block it? |
|---|---|---|
| Googlebot/2.1 | Google's main crawler | Never block |
| Bingbot/2.0 | Microsoft Bing crawler | Keep allowed |
| AhrefsBot | Ahrefs SEO tool | Optional |
| SemrushBot | Semrush SEO tool | Optional — saves bandwidth |
| GPTBot | OpenAI AI training crawler | Block if unwanted |
| ClaudeBot | Anthropic AI training crawler | Block if unwanted |
| MJ12bot | Aggressive unknown bot | Block — wastes budget |
| Blank user agent | Often spam or scrapers | Block always |
Block unwanted bots in robots.txt
# Block AI training crawlersUser-agent: GPTBotDisallow: /User-agent: ClaudeBotDisallow: /# Block aggressive SEO botsUser-agent: MJ12botDisallow: /
7. Query Parameters — When to Block and When Not To
Query parameters are parts of a URL after a ? mark. They create hundreds of duplicate URLs — wasting crawl budget and confusing Google.
Example: a2itsoft.com/courses?category=python&sort=popular&page=2
| Parameter type | Example | Block? |
|---|---|---|
| Sorting | ?sort=price | YES |
| Filters | ?category=python | YES |
| Session IDs | ?sessionid=abc123 | YES |
| UTM tracking | ?utm_source=google | YES |
| Language/currency | ?lang=hi | NO — different content |
| Search queries | ?q=python+course | YES |
# Block common parameter types in robots.txtDisallow: /*?sort=Disallow: /*?filter=Disallow: /*?sessionid=Disallow: /*?utm_
8. Redirect Chains — Find and Fix 301/302 Issues
A redirect chain happens when URL A redirects to URL B, which redirects to URL C. Google loses link authority at every hop — and your rankings suffer.
| Type | SEO impact | When to use |
|---|---|---|
| 301 Permanent | Passes ~99% link equity | Permanent URL changes |
| 302 Temporary | Does NOT pass link equity | Temporary moves only |
| Meta refresh | Weak — avoid for SEO | Never use for SEO |
⚠ IMPORTANT
Always use 301 redirects for permanent URL changes. Using 302 by mistake means Google treats the OLD URL as the main page — your rankings stay on a URL you no longer use.
Good vs bad redirect example
# BAD — 3 redirect hops (loses authority)http://a2itsoft.com → https://a2itsoft.com → https://www.a2itsoft.com → https://www.a2itsoft.com/# GOOD — 1 direct hophttp://a2itsoft.com → https://www.a2itsoft.com/
Fix in .htaccess (Apache)
RewriteEngine On# HTTP to HTTPS + non-www to www in ONE stepRewriteCond %{HTTPS} off [OR] RewriteCond %{HTTP_HOST} !^www\. [NC] RewriteRule ^ https://www.a2itsoft.com%{REQUEST_URI} [L,R=301]
⭐ PRO TIP
After fixing redirect chains, use the URL Inspection tool in Google Search Console to request re-crawling of your most important pages. This speeds up ranking recovery.
Crawling Checklist — Section Summary
- robots.txt — allows important pages, blocks /admin/ /login/ /public/items/
- XML sitemap — submitted in GSC, max 50,000 URLs, only indexable pages
- Non-indexable URLs — find and fix accidental noindex tags and orphan pages
- Canonical tags — every page has a self-referencing canonical set correctly
- Crawl budget — block parameterised, session, and private URLs on large sites
- Log files — check which bots crawl your site, block spam and AI crawlers
- Query parameters — block sort, filter, session, UTM via robots.txt or canonical
- Redirect chains — all redirects are single-hop 301s to the final URL
Frequently Asked Questions — Crawling in Technical SEO
Crawling is the process where Google sends automated bots (called Googlebot or spiders) to visit your website, read the content on each page, and follow all the links. This is how Google discovers your pages before it can index and rank them.
robots.txt is a plain text file at the root of your website (yoursite.com/robots.txt) that tells search bots which pages to crawl and which to skip. If set incorrectly — for example Disallow: / — it can block Google from your entire website and remove all your pages from search results.
Crawl budget is the number of pages Google will crawl on your website per day. For small websites this is rarely a problem. For large websites with 50,000+ URLs, wasting budget on session IDs, parameters, or private pages means Google may never crawl your most important content — which directly hurts your rankings.
A 301 redirect is permanent and passes nearly all link equity (ranking power) to the new URL. A 302 redirect is temporary and does not reliably pass link equity — Google keeps treating the old URL as the main page. Always use 301 for permanent URL changes to protect your SEO rankings.
A canonical tag is an HTML tag added to the head section of a page. It tells Google which version of a page is the original when duplicate content exists — for example when the same page is accessible via HTTP and HTTPS, or with and without www. Every page should have a canonical tag, even if pointing to itself.
That depends on your preference. GPTBot (OpenAI) and ClaudeBot (Anthropic) are AI training crawlers. Blocking them prevents your content from being used to train AI models. This has no effect on your Google rankings — these bots are completely separate from Googlebot. To block them, add Disallow: / under each bot's User-agent in your robots.txt file.
Go to Google Search Console → Settings → Crawl Stats. This shows Google's crawling activity over the last 90 days including total crawl requests, average response time, and which pages are being crawled most. You can also use the URL Inspection tool to check if any specific page has been crawled and indexed.
A2IT InternEdge in Phase 8, Mohali offers a Digital Marketing and SEO Internship Course where you learn technical SEO, Google Search Console, robots.txt auditing, sitemap optimisation, and more — with live projects and placement support. Visit a2itsoft.com/digital-marketing-internship-course to book a free demo class.
▶ Coming Next — Part 3
Rendering: What Google Sees vs What Users See
Now that Google can find your pages, the next step is making sure it can actually read them. In Part 3 we will cover:
- What is rendering and why it matters
- JavaScript SEO problems and how to fix them
- URL Inspection tool in Google Search Console
- Using AI tools to assist rendering audits
- Screaming Frog vs GSC — which to use when
