How To Increase Your Website Crawl Rate

Wifi

website crawl rate

One of the major restrictions on the speed at which an SEO campaign can move forward, is how quickly changes to the site are indexed. This is especially true when working with sites that already have some very established rankings and you want to make small changes at a time. If the site is only getting crawled once every couple of weeks this can really slow down the process.

Luckily there are things you can do to increase you websites crawl rate and in turn also have a positive effect on depth of crawl and also rankings, regularly crawled websites are also likely to be crawled a lot deeper meaning lots more pages in the search engine index

So here are five of the most important things you can do to help increase how often your site is visited:

1 ) Increase your pagerank

Not only are links vital for increased rankings, but they also increase the pagerank of your website. A lot of the search engines crawl patterns are dictated by pagerank, with higher PR pages generally being crawled more regularly. But as usual this isn’t the only factor, while you build links to your site there are other things you should be doing.

2 ) Faster and smaller pages

The search engines have to crawl an immensely large amount of data and crawling takes up lots of valuable resource. If you can increase the speed at which your page loads and decrease the size of it you are likely to see benefits from a deeper crawl and an increased in the rate.

3 ) Servers

Make sure you have a reliable server. The last thing you want when the search engines come to visit is for your site to be unavailable.  This will seriously hamper how often the search engines come to visit and can also damage your rankings as it leads to a bad user experience.

4 ) Fresh content

One of reasons blogs get indexed so regularly is because they provide a large amount of regularly updated content. This gives the search engines something new to be indexing each time they visit. If you have a static page which you need to remain the same, as is the case with most business sites, then you need to consider ways in which you can integrate regularly updated content into the site. Think of things along the lines of news and articles, latest posts, a twitter feed or some other form of update.

5 ) Duplication

One of the major issues that a lot of sites come across is the damaging effect of duplication on the site. This can lead to a lot of search engine resources being taken up on unnecessary pages and can also lead to indexation and ranking issues. If there are any duplication issues on site you need to sort them out as soon as possible.

Implementing these changes should help increase how often your site is crawled, and if you have a webmaster tools account (you should :)) you will be able to monitor the results.

Image Credit: Kristy

Article by Ben Hook

Ben has written 1 articles.

You can follow webtrafficroi on Google Plus, Facebook and Twitter here. Subscribe to WebTrafficROI feed via RSS or EMAIL to receive instant updates.


Comments on this entry are closed.

  • Harshad January 14, 2011, 8:58 am

    Frequent updates on the site can result in faster crawl rates. Also, there is an option in Google Webmaster Tools to increase the crawl rates.

  • Alex Dumitru January 14, 2011, 10:08 am

    Pagerank has become irrelevant and it hasn’t been updated for a year now and probably it won’t be updated again. Though you are right with the other points.

    • Aluminum Cases January 14, 2011, 6:28 pm

      I disagree. I don’t expect Google to drop PageRank or stop updating it. They may have just changed the format where they used to do mass updates across all sites. Perhaps they are now updating PR on a more regular basis.

    • ZK January 16, 2011, 6:40 am

      Google will not mess with Pagerank it’s still a factor for authority although Google has toned down it’s importance in the last 12 – 18 months.

      • Alex@Jocuri January 16, 2011, 12:54 pm

        I think Google it’s only updating their internal pagerank and may even stop updating the PR of website publicly. This may be a measure to ditch PR, because of their exclusive license with Standford will stop in 2011.

        On topic:
        I think one of the easiest way to make your crawl rate increase, is to go to Google webmaster tools and increase your crawl rate.
        Also, writing unique and new content often can definitely help!

    • used tires January 18, 2011, 6:19 pm

      Alex, In my opinion it is still being factored behind the scenes where we can’t see the raw numbers.

      -Jean

    • Free Voip Calls January 19, 2011, 10:07 am

      Pagerank is losing its importance. You will find very high crawling rate/good serps for site with low PR, and there are plenty of them.

  • Dean Saliba January 14, 2011, 11:16 am

    Does running the XML Sitmap plguin help? I’ve been told this tells the search engines to come look at your site.

    • Edinburgh Driving School January 14, 2011, 3:57 pm

      I’ve not heard that it brings in the search engine spiders but it definitely helps to get all your pages indexed once the spiders.

    • ZK January 16, 2011, 6:44 am

      XML Sitemap plugin is a must have plugin for any blog, it helps you monitor and regulate the crawl rate of your blog.

      Other must have plugins are mentioned on the below post

      http://www.webtrafficroi.com/11-wordpress-plugins-youve-gotta-have/

  • Edinburgh Driving School January 14, 2011, 3:49 pm

    These are some great tips. I heard recently that the speed of your website, how fast it loads, is becoming a more important SEO factor.

    • Aluminum Cases January 14, 2011, 6:30 pm

      Yes Google openly admitted to this ranking factor. It makes sense as they do not want to direct people to sites that load too slowly. Quality sites are more likely to be on a fast, reliable server. Meanwhile, low quality sites are usually on slower servers.

    • Komodo Dragon January 18, 2011, 3:19 am

      Physical location of the server is also a very important factor. If you’re catering to an audience mostly in the US, it might not be good idea to have the server farm located in Australia as the response time will be around two hundred milliseconds more than a server in the Americas.

  • Shailender @ Valentine Vacation January 15, 2011, 7:29 am

    I think Harshad is right. One can update our page regularly so that it can be crawled frequently.

  • Seo company Kanpur January 15, 2011, 8:21 am

    That’s really a great colllection of nice advice on Increase Your Website Crawl Rate.I will keep this suggestion in mind.
    Thanks
    Raj………….!

  • Pacquiao Mosley January 15, 2011, 1:58 pm

    I agree that providing fresh content over 300 words can make your site index by search engines fast. If you provide quality content evreyday, search engine will notice that and will crawl your new content everyday. -Mark

  • Car Rental January 16, 2011, 2:15 am

    I think fresh content should come at number two. Also i think it depends on the number of backlinks you have. If there are few backlinks, Google will not index too often.

    • Dodge Neon SRT4 February 2, 2011, 12:04 pm

      Yes, Getting Back links is the main way to get index fast. Another way is crawling rate.

  • Melvin January 16, 2011, 11:10 am

    True. Funny enough a lot of bloggers either don’t have a webmaster central account or they don’t use it that much.

    I mean just by looking at the data, you can see a lot of metrics that can help you improve on things on your blog.

  • Jasmine January 16, 2011, 11:43 am

    Good article here. Ironically, it is very difficult to increase the pagerank of a website’s pages, it requires a lot of hard work and patience.

  • Navya January 17, 2011, 1:24 am

    Hi Ben Hook,
    I liked your tips and I have really come to know a new thing from this faster and smaller pages, hope implementing this will increase my traffic….

  • Application Developer January 17, 2011, 5:32 am

    Out of all an above five tips for increasing crawl rate for a website, i mostly believe in blog. Blog can ensure you the indexing so fast than any other method as it should be containing a high quality and interesting content to post regularly.

  • Software Development Company January 17, 2011, 7:43 am

    Update the content of webpage that helps to increase crawl rate for a site.

  • Mahendra January 17, 2011, 7:50 am

    Fast and small page is not always possible for blogs

  • Rakesh Kumar January 17, 2011, 8:00 am

    I have 15 duplicate titles in my web master tools. I think that’s why Google doesn’t crawl my site regularly. Am i right?

  • Car Rental January 17, 2011, 9:03 am

    Most of the people use Wordpress for their blogs and i think it is pretty fast as compared to ASP blogs. Unless your theme is full of images and you are on shared hosting.

  • claire bright January 17, 2011, 1:21 pm

    By experience it crawls regularly when you post regularly. So daily fresh content matters a lot to search engines.

  • used tires January 17, 2011, 5:24 pm

    I’ve seen cases where too much crawl is bad but having a server that can handle it makes it all worth it! I noticed that my average load speed was going up on the Google Webmaster’s tool, which was due to increased crawl rate speed, but that was because the server couldn’t handle it, once that was fixed everything was golden!

    -Jean

  • p90x January 18, 2011, 5:35 pm

    Some great tips Ben. I think it is important to have a host that can handle increased crawl rates as well.

    – Robert

  • Free Voip Calls January 19, 2011, 4:06 am

    Well, I guess these points are more in general for better seo, better rankings.
    Choosing the right hosting is also very imp…thanks for these tips!

  • seo social media January 20, 2011, 1:37 am

    this article about increasing website website crawl rate is great. huge thanks for sharing. learned a lot reading this. 🙂

  • Garen @ Get Backlinks January 22, 2011, 8:09 pm

    Go through your plugins and only use 5-7 of them and can the rest of them. I recently did and it helped my blog load quicker.

    Also, don’t ignore 404 errors as they can really kill your rankings in Google. Made that mistake the beginning of Jan 2011 🙁

    Garen

  • FirstHosting January 26, 2011, 4:05 pm

    fresh content and regularity in it can really get you crawled fast.

  • network backup January 26, 2011, 11:36 pm

    This is really a great collection of advice on Increase the Website Crawl Rate. In my opinion, if one can update the page regularly, it can be crawled frequently. Anyway, thank you for the useful tips.

  • Jasmine February 1, 2011, 9:59 am

    I wonder if you submit sitemap to Google, will it increase the crawl rate?

  • search engine optimization usa February 2, 2011, 1:00 am

    That’s really a great to Increase Website Crawl Rate.
    Thanks for information.

  • Casey February 2, 2011, 6:29 am

    Hi, I am not agree with you because Google to drop Page Rank or stop updating it. They may have just changed the format where they used to do mass updates across all sites.Perhaps they are now updating PR on a more regular basis.

  • Dodge Neon SRT4 February 2, 2011, 12:02 pm

    Another Way, to increase crawl rate is just to add in google webmaster, and increase crawling rate in the settings.

  • Durable Goods February 3, 2011, 3:19 am

    Adding Fresh Content and Avoiding Duplication Of Content is the Key to Faster Crawl rate.

  • Cassy February 5, 2011, 12:31 am

    Hi,
    This is really a great collection of advice on Increase the Website Crawl Rate. In my opinion, if one can update the page regularly, it can be crawled frequently. Anyway, thank you for the useful tips.That’s really a great to Increase Website Crawl Rate.

    Thanks for information

  • eCommerce Features February 5, 2011, 12:57 am

    I know one way to increase the crawl rate of any website using google webmaster tool but do not know that is it proper to change the rate from their. Also, do not have idea for what reason it needs to be changed.

  • windows 8 February 7, 2011, 3:40 pm

    This was helpful.. Thanx , but i guess one needs to work lot on content to get all of these.

  • aftab February 16, 2011, 5:51 pm

    i am facing low crawl issue from start of this year. crawling period is about 5 to 10 days and in previous year it was twice a day.

    in google support forum, some one reply my question that this is for “duplicate content”.

    how can we stop sites that fetch my content before google crawl my site like technorati and sonorika etc. and my content became duplicate after crawling..

    i hate google’s new algorithm.

  • rijans February 22, 2011, 5:23 am

    Some times I see that my new posts are quickly indexing and sometimes not. Now I realized the issues.

  • Snoring Solutions March 14, 2011, 9:35 am

    I know one way to increase the crawl rate of any website using google webmaster tool but do not know that is it proper to change the rate from their. Also, do not have idea for what reason it needs to be changed.

  • hcg reviews August 23, 2011, 11:28 am

    blogs tend to get indexed more often anyway because bloggers keep adding content to them. If you have a blog and want to speed this process up even more then pinging it at a site like Pingomatic definitely helps.

  • Hiren Modi October 11, 2011, 1:18 am

    I am working on eCommerce website and having 5K+ products. Here, you have suggested such a great point to improve crawling rate. But, I have confusion with it because, I have implemented all points which you have mentioned in this blog post. My website is faster, server is very good, content is unique, remove duplication and waiting for few more page rank. But, I am still surviving from crawling. Google is not crawling my website properly compare to other website. Can you give me few more idea about it? Why should it happen with me? BTW: Thanks for sharing such a great blog post on very useful topic. I hope too many SEO guys have enjoyed this blog post. Thanks again!!

  • Ben Hook October 11, 2011, 3:31 am

    Hi Hiren,

    Thanks for your comments. Have you considered creating an XML sitemap and submitting it to Google via webmaster tools? This can really help the search engines when crawling and may be something you would benefit from.

    Hope this helps,

    Ben

    • Hiren Modi October 11, 2011, 4:09 am

      Hi Ben,

      Yes, I have submitted multiple XML sitemaps to my Google webmaster tools. You can get more idea by visiting following XML sitemaps.

      http://www.lampslightingandmore.com/pendant_lighting_sitemap.xml
      http://www.lampslightingandmore.com/table_lamps_sitemap.xml

      I have submitted 24 XML sitemaps with above structure. I have separated XML sitemaps associated to categories. I have big confusion with data which is available on Google webmaster tools. It indicates that, Submitted URLs: 5162 and 1178 URLs in web index. But, I am not able to see that much URLs in Google index.

      Can you help me to understand more about it?

  • Ben Hook October 11, 2011, 8:48 am

    If you put this into Google: site:http://www.lampslightingandmore.com, you can see that there are 2000+ results. These are the ones in the index. The others are not included.

    With a site this size, the main thing I would recommend doing is building links to the subcategories and deeper pages on the site to spread around link weight on the site. This will encourage the search engines to enter at different points in your site and hopefully crawl the pages that it isn’t currently getting. It’s going to be a lot of work but it will be worth it.

    Focus on building links to some of your most popular product pages and go from there. This will help to increase these rankings and strengthen the domain overall.

    • Hiren Modi October 12, 2011, 1:08 am

      Oh.. That’s great and happy to know that Google have crawled 2190 pages for my website. I was checking with number which is available in bottom and go to the last page of search result. 🙂 So, Why Google have included my maximum pages in omitted result? I think it will come in visible portion after getting few quality external links. Right? Thanks for your recommendation regarding that post and going to read it.

      • Ben Hook October 12, 2011, 6:56 am

        Google may have omitted some results as they seem very similar. They do it to vary up the search results. On a site this size, it’s hard to avoid this as most likely there will be some overlap.

        I wouldn’t be to concerned and instead I would focus on building those links.

        Best of luck!

        Ben

  • free call November 25, 2011, 4:35 pm

    You have missed services like pingshot that can be activated from feedburner.com and importance of publishing the new articles or updates to twitter or facebook that can bring the spiders/bots quickly to your website.