By using seoforum’s services you agree to our Cookies Use and Data Transfer outside the EU.
We and our partners operate globally and use cookies, including for analytics, personalisation, ads and Newsletters.

  • Join the best UK dedicated SEO Forum

    Provide or get advice on everything SEO, ask questions, gain confirmation or just become apart of a friendly, like minded community who love SEO and Online Marketing.

    Join 50,000 members!

Screaming Frog crawl - questions



New Member

I'm working on a website that has approx 70 pages indexed. When I crawl this website using Screaming Frog, I'm getting literally tens of thousands of pages coming up in the HTML section of the crawl. A lot of these pages are redirects from URLs containing .html to the non.html version of the URL. There's also a lot of URLs containing /CSS/asests/.... etc but the 'Content Type' of these URLs is text/html; charset= UTF 8.

I'm unable to share screenshots (apologies!) but does anyone know if I should be concerned about crawl budget, etc with this amount of URLs being crawled? Any advice appreciated on how best to handle.

Thank you!

Latest Products

  • [Rivmedia] Lazy Loader XF2
    [Rivmedia] Lazy Loader XF2
    Load images asynchronously on your forum, allowing images to be loaded only when they are in view
    • Rivmedia
    • Updated:
  • [Rivmedia] Guest Redirect & Profile unlink
    [Rivmedia] Guest Redirect & Profile unlink
    Forums which prevent member profile access for guests, redirction and unlinking for profiles
    • Rivmedia
    • Updated:
  • [Rivmedia] Simple Redirects
    [Rivmedia] Simple Redirects
    Simple redirects allows forum admins to make simple 301 or 302 redirects via their admin panel
    • Rivmedia
    • Updated:
  • [Rivmedia] Minimum Message Length
    [Rivmedia] Minimum Message Length
    Eradicate short, pointless posts with a minimum message length and improving content worth on a thre
    • Rivmedia
    • Updated: