CrawlIQ Technical Capabilities
A complete reference for evaluating, assessing, and improving any website's technical SEO — powered by Python, FastAPI, and multiple AI models.
Technical Capabilities
Async Technical SEO Evaluation Engine
Evaluates up to 500 pages using a breadth-first async engine built with Python aiohttp and BeautifulSoup. The evaluation engine processes pages concurrently — a 100-page site typically completes in under 60 seconds. Each page is scored on over 50 on-page and technical signals including title tags, meta descriptions, heading structure, canonical correctness, canonical loops, internal link counts, image alt coverage, Open Graph completeness, and page-level indexability. Results stream back as a structured JSON reference report ready for AI remediation or Excel export.
Multi-Model AI Remediation Engine
Route your evaluation report to any of four AI providers: Groq (Llama 3.3 — fastest), Google Gemini 1.5, Anthropic Claude 3 Haiku, or OpenAI GPT-4o. Each model receives structured issue data for every evaluated page and returns paste-ready remediation content for titles, meta descriptions, H1 tags, and content gap guidance. If you have no API key, CrawlIQ falls back to a deterministic rules-based engine that still produces 80% of actionable remediation guidance without any external call.
AI PoweredCompetitor Technical Comparison
Enter up to three competitor domains alongside your own. CrawlIQ evaluates all four sites in parallel, then produces a side-by-side reference covering average page score, evaluation depth, unique internal link count, schema markup coverage, and Core Web Vitals indicators. The comparison immediately surfaces which areas your site underperforms — whether that's canonical hygiene, heading structure consistency, or average title length relative to current SERP leaders.
FreeSERP Signal Intelligence
Examines the live search results page for any query. The evaluation engine retrieves the top-10 results and extracts title tag patterns, meta description length distribution, structured data types present (FAQ, HowTo, Review, Product), and estimated word counts. Use this reference data to benchmark your own page before publishing — so you enter a SERP already matching the content format Google has validated for that keyword.
FreeKeyword Gap Methodology
Identifies the full keyword set your competitors rank for that your site does not appear in. The evaluation engine cross-references anchor text, heading keywords, and structured data content across your site and competitor sites to surface missing topic clusters. Filter the reference output by search intent (informational, commercial, navigational), estimated volume tier, and keyword difficulty score. The result is a prioritised content calendar you can act on immediately.
AI EnhancedPosition Tracking Methodology
Track daily keyword position changes across your target query set. CrawlIQ records a rolling history of SERP positions so you can observe rank velocity — not just current position. Configure threshold alerts: if a keyword drops more than five positions in 24 hours, you receive an email notification before the traffic loss becomes visible in Google Analytics. Early identification is the difference between a recoverable algorithmic dip and a permanent ranking loss.
Free AccountSitemap Structure Examiner
Provide any sitemap.xml URL and CrawlIQ evaluates every listed URL in seconds. The reference report flags HTTP 4xx and 5xx errors, redirect chains (any URL returning 3xx), missing lastmod values that reduce indexation frequency signals, duplicate priorities, and URLs blocked by robots.txt. The full results export to Excel with one click — useful for monthly evaluation-health reviews or pre-migration URL inventories.
AI Content Guidance System
Produce SEO-optimised content outlines, meta descriptions, and H1 tags at scale across all evaluated pages. Select your preferred AI model, specify the target keyword, tone, and approximate word count, and CrawlIQ returns a structured content brief for each page — section headers, recommended word count, semantic keyword suggestions, and FAQ questions to address People Also Ask positions. Export the complete brief set to Excel for your content team.
AI PoweredEvaluation Scope
What the Evaluation Engine Identifies
CrawlIQ's evaluation engine examines every page for over 50 SEO signals including:
- Missing or duplicate title tags; titles outside 30–60 character range
- Missing or duplicate meta descriptions
- Missing H1, multiple H1s, or empty heading structure
- Canonical tag absent, self-referencing, or pointing to a different URL
- Pages marked
noindexornofollowunintentionally — accidental blocking - Broken internal links (4xx and 5xx HTTP status codes)
- Redirect chains (3xx → 3xx) wasting indexation budget
- Images missing alt text attributes
- Thin content pages with under 300 words
- Open Graph and Twitter Card tag coverage
- Core Web Vitals proxies (LCP, CLS, FID indicators)
- HTTPS enforcement and mixed-content warnings
Excel Export Formats
Every evaluation can be exported to a structured Excel reference workbook with multiple sheets:
- Full SEO Report — all evaluated pages with scores, issues, and keywords
- Live Optimisation Table — paste-ready optimised titles, descriptions, and H1s
- Technical SEO Assessment — per-page scores (0–100) with A–F grades
- Per-Field Issues — field-level breakdown for bulk fixing workflows
- AI Generated Content — full AI output per page for content teams
AI Provider Comparison
CrawlIQ routes AI remediation through the provider you choose. All providers are free-tier compatible:
- Groq + Llama 3 — fastest response time; free API at console.groq.com
- Google Gemini — strong multilingual and long-context performance
- Anthropic Claude — best for nuanced content quality assessment
- OpenAI GPT-4o — industry-standard benchmark model
- Rules-based fallback — works with zero API key; deterministic issue detection
CrawlIQ vs Competing SEO Evaluation Systems
| Feature | CrawlIQ | Screaming Frog | SEOptimer | Seobility |
|---|---|---|---|---|
| Price (basic tier) | Free | Free up to 500 URLs / £259/yr unlimited | Free limited / $19/mo | Free limited / €50/mo |
| Pages per evaluation (free) | 50 (no login) / 500 (account) | 500 | 1 page per report | 1,000 |
| On-page issue detection | ✓ 50+ signals | ✓ 200+ signals | ✓ ~30 signals | ✓ ~40 signals |
| AI-generated fixes | ✓ 4 model choices | ✗ | ~ limited suggestions | ✗ |
| Multi-model AI (Groq / Gemini / Claude / GPT-4o) | ✓ | ✗ | ✗ | ✗ |
| Competitor side-by-side analysis | ✓ | ~ via separate crawls | ✓ | ✓ |
| SERP result analysis | ✓ | ✗ | ✗ | ~ basic |
| Keyword gap analysis | ✓ | ✗ | ✗ | ✓ paid |
| Sitemap URL validator | ✓ | ✓ | ✗ | ✓ |
| AI content brief generator | ✓ | ✗ | ✗ | ✗ |
| Excel export | ✓ multi-sheet | ✓ | ✓ paid | ✓ paid |
| Rank monitoring | ✓ free | ✗ | ✓ paid | ✓ paid |
| Installation required | No — runs in browser | Desktop app (Windows/Mac) | No — web app | No — web app |
✓ available ✗ not available ~ partial / limited. Pricing as of 2026.
Who Uses CrawlIQ Evaluation Methodology
Freelance SEO Consultants
Perform fast technical evaluations for client pitches without paying for a monthly subscription. Export multi-sheet Excel reference reports to hand directly to clients.
Small Business Owners
No SEO background required. Begin a free evaluation with your URL, read the AI-produced plain-English remediation guidance, and implement changes yourself.
In-House Marketing Teams
Track weekly rank positions, observe competitor changes, and use the keyword gap reference to brief content writers on exactly what to create next.
Web Developers
Identify technical SEO regressions before deployment. Perform a 50-page evaluation during QA to verify canonical tags, hreflang correctness, and internal link structure.