One of the major restrictions on the speed at which an SEO campaign can move forward, is how quickly changes to the site are indexed. This is especially true when working with sites that already have some very established rankings and you want to make small changes at a time. If the site is only getting crawled once every couple of weeks this can really slow down the process.
Luckily there are things you can do to increase you websites crawl rate and in turn also have a positive effect on depth of crawl and also rankings, regularly crawled websites are also likely to be crawled a lot deeper meaning lots more pages in the search engine index
So here are five of the most important things you can do to help increase how often your site is visited:
1 ) Increase your pagerank
Not only are links vital for increased rankings, but they also increase the pagerank of your website. A lot of the search engines crawl patterns are dictated by pagerank, with higher PR pages generally being crawled more regularly. But as usual this isn’t the only factor, while you build links to your site there are other things you should be doing.
2 ) Faster and smaller pages
The search engines have to crawl an immensely large amount of data and crawling takes up lots of valuable resource. If you can increase the speed at which your page loads and decrease the size of it you are likely to see benefits from a deeper crawl and an increased in the rate.
3 ) Servers
Make sure you have a reliable server. The last thing you want when the search engines come to visit is for your site to be unavailable. This will seriously hamper how often the search engines come to visit and can also damage your rankings as it leads to a bad user experience.
4 ) Fresh content
One of reasons blogs get indexed so regularly is because they provide a large amount of regularly updated content. This gives the search engines something new to be indexing each time they visit. If you have a static page which you need to remain the same, as is the case with most business sites, then you need to consider ways in which you can integrate regularly updated content into the site. Think of things along the lines of news and articles, latest posts, a twitter feed or some other form of update.
5 ) Duplication
One of the major issues that a lot of sites come across is the damaging effect of duplication on the site. This can lead to a lot of search engine resources being taken up on unnecessary pages and can also lead to indexation and ranking issues. If there are any duplication issues on site you need to sort them out as soon as possible.
Implementing these changes should help increase how often your site is crawled, and if you have a webmaster tools account (you should :)) you will be able to monitor the results.
Image Credit: Kristy
53 comments
Frequent updates on the site can result in faster crawl rates. Also, there is an option in Google Webmaster Tools to increase the crawl rates.
Pagerank has become irrelevant and it hasn’t been updated for a year now and probably it won’t be updated again. Though you are right with the other points.
I disagree. I don’t expect Google to drop PageRank or stop updating it. They may have just changed the format where they used to do mass updates across all sites. Perhaps they are now updating PR on a more regular basis.
Google will not mess with Pagerank it’s still a factor for authority although Google has toned down it’s importance in the last 12 – 18 months.
I think Google it’s only updating their internal pagerank and may even stop updating the PR of website publicly. This may be a measure to ditch PR, because of their exclusive license with Standford will stop in 2011.
On topic:
I think one of the easiest way to make your crawl rate increase, is to go to Google webmaster tools and increase your crawl rate.
Also, writing unique and new content often can definitely help!
Alex, In my opinion it is still being factored behind the scenes where we can’t see the raw numbers.
-Jean
Pagerank is losing its importance. You will find very high crawling rate/good serps for site with low PR, and there are plenty of them.
Does running the XML Sitmap plguin help? I’ve been told this tells the search engines to come look at your site.
I’ve not heard that it brings in the search engine spiders but it definitely helps to get all your pages indexed once the spiders.
XML Sitemap plugin is a must have plugin for any blog, it helps you monitor and regulate the crawl rate of your blog.
Other must have plugins are mentioned on the below post
http://www.webtrafficroi.com/11-wordpress-plugins-youve-gotta-have/
These are some great tips. I heard recently that the speed of your website, how fast it loads, is becoming a more important SEO factor.
Yes Google openly admitted to this ranking factor. It makes sense as they do not want to direct people to sites that load too slowly. Quality sites are more likely to be on a fast, reliable server. Meanwhile, low quality sites are usually on slower servers.
Physical location of the server is also a very important factor. If you’re catering to an audience mostly in the US, it might not be good idea to have the server farm located in Australia as the response time will be around two hundred milliseconds more than a server in the Americas.
I think Harshad is right. One can update our page regularly so that it can be crawled frequently.
That’s really a great colllection of nice advice on Increase Your Website Crawl Rate.I will keep this suggestion in mind.
Thanks
Raj………….!
I agree that providing fresh content over 300 words can make your site index by search engines fast. If you provide quality content evreyday, search engine will notice that and will crawl your new content everyday. -Mark
I think fresh content should come at number two. Also i think it depends on the number of backlinks you have. If there are few backlinks, Google will not index too often.
Yes, Getting Back links is the main way to get index fast. Another way is crawling rate.
True. Funny enough a lot of bloggers either don’t have a webmaster central account or they don’t use it that much.
I mean just by looking at the data, you can see a lot of metrics that can help you improve on things on your blog.
Good article here. Ironically, it is very difficult to increase the pagerank of a website’s pages, it requires a lot of hard work and patience.
Hi Ben Hook,
I liked your tips and I have really come to know a new thing from this faster and smaller pages, hope implementing this will increase my traffic….
Out of all an above five tips for increasing crawl rate for a website, i mostly believe in blog. Blog can ensure you the indexing so fast than any other method as it should be containing a high quality and interesting content to post regularly.
Update the content of webpage that helps to increase crawl rate for a site.
Fast and small page is not always possible for blogs
I have 15 duplicate titles in my web master tools. I think that’s why Google doesn’t crawl my site regularly. Am i right?
Most of the people use Wordpress for their blogs and i think it is pretty fast as compared to ASP blogs. Unless your theme is full of images and you are on shared hosting.
By experience it crawls regularly when you post regularly. So daily fresh content matters a lot to search engines.
I’ve seen cases where too much crawl is bad but having a server that can handle it makes it all worth it! I noticed that my average load speed was going up on the Google Webmaster’s tool, which was due to increased crawl rate speed, but that was because the server couldn’t handle it, once that was fixed everything was golden!
-Jean
Some great tips Ben. I think it is important to have a host that can handle increased crawl rates as well.
– Robert
Well, I guess these points are more in general for better seo, better rankings.
Choosing the right hosting is also very imp…thanks for these tips!
this article about increasing website website crawl rate is great. huge thanks for sharing. learned a lot reading this. 🙂
Go through your plugins and only use 5-7 of them and can the rest of them. I recently did and it helped my blog load quicker.
Also, don’t ignore 404 errors as they can really kill your rankings in Google. Made that mistake the beginning of Jan 2011 🙁
Garen
fresh content and regularity in it can really get you crawled fast.
This is really a great collection of advice on Increase the Website Crawl Rate. In my opinion, if one can update the page regularly, it can be crawled frequently. Anyway, thank you for the useful tips.
I wonder if you submit sitemap to Google, will it increase the crawl rate?
That’s really a great to Increase Website Crawl Rate.
Thanks for information.
Hi, I am not agree with you because Google to drop Page Rank or stop updating it. They may have just changed the format where they used to do mass updates across all sites.Perhaps they are now updating PR on a more regular basis.
Another Way, to increase crawl rate is just to add in google webmaster, and increase crawling rate in the settings.
Adding Fresh Content and Avoiding Duplication Of Content is the Key to Faster Crawl rate.
Hi,
This is really a great collection of advice on Increase the Website Crawl Rate. In my opinion, if one can update the page regularly, it can be crawled frequently. Anyway, thank you for the useful tips.That’s really a great to Increase Website Crawl Rate.
Thanks for information
I know one way to increase the crawl rate of any website using google webmaster tool but do not know that is it proper to change the rate from their. Also, do not have idea for what reason it needs to be changed.
This was helpful.. Thanx , but i guess one needs to work lot on content to get all of these.
i am facing low crawl issue from start of this year. crawling period is about 5 to 10 days and in previous year it was twice a day.
in google support forum, some one reply my question that this is for “duplicate content”.
how can we stop sites that fetch my content before google crawl my site like technorati and sonorika etc. and my content became duplicate after crawling..
i hate google’s new algorithm.
Some times I see that my new posts are quickly indexing and sometimes not. Now I realized the issues.
I know one way to increase the crawl rate of any website using google webmaster tool but do not know that is it proper to change the rate from their. Also, do not have idea for what reason it needs to be changed.
blogs tend to get indexed more often anyway because bloggers keep adding content to them. If you have a blog and want to speed this process up even more then pinging it at a site like Pingomatic definitely helps.
I am working on eCommerce website and having 5K+ products. Here, you have suggested such a great point to improve crawling rate. But, I have confusion with it because, I have implemented all points which you have mentioned in this blog post. My website is faster, server is very good, content is unique, remove duplication and waiting for few more page rank. But, I am still surviving from crawling. Google is not crawling my website properly compare to other website. Can you give me few more idea about it? Why should it happen with me? BTW: Thanks for sharing such a great blog post on very useful topic. I hope too many SEO guys have enjoyed this blog post. Thanks again!!
Hi Hiren,
Thanks for your comments. Have you considered creating an XML sitemap and submitting it to Google via webmaster tools? This can really help the search engines when crawling and may be something you would benefit from.
Hope this helps,
Ben
Hi Ben,
Yes, I have submitted multiple XML sitemaps to my Google webmaster tools. You can get more idea by visiting following XML sitemaps.
http://www.lampslightingandmore.com/pendant_lighting_sitemap.xml
http://www.lampslightingandmore.com/table_lamps_sitemap.xml
I have submitted 24 XML sitemaps with above structure. I have separated XML sitemaps associated to categories. I have big confusion with data which is available on Google webmaster tools. It indicates that, Submitted URLs: 5162 and 1178 URLs in web index. But, I am not able to see that much URLs in Google index.
Can you help me to understand more about it?
If you put this into Google: site:http://www.lampslightingandmore.com, you can see that there are 2000+ results. These are the ones in the index. The others are not included.
With a site this size, the main thing I would recommend doing is building links to the subcategories and deeper pages on the site to spread around link weight on the site. This will encourage the search engines to enter at different points in your site and hopefully crawl the pages that it isn’t currently getting. It’s going to be a lot of work but it will be worth it.
Focus on building links to some of your most popular product pages and go from there. This will help to increase these rankings and strengthen the domain overall.
Oh.. That’s great and happy to know that Google have crawled 2190 pages for my website. I was checking with number which is available in bottom and go to the last page of search result. 🙂 So, Why Google have included my maximum pages in omitted result? I think it will come in visible portion after getting few quality external links. Right? Thanks for your recommendation regarding that post and going to read it.
Google may have omitted some results as they seem very similar. They do it to vary up the search results. On a site this size, it’s hard to avoid this as most likely there will be some overlap.
I wouldn’t be to concerned and instead I would focus on building those links.
Best of luck!
Ben
You have missed services like pingshot that can be activated from feedburner.com and importance of publishing the new articles or updates to twitter or facebook that can bring the spiders/bots quickly to your website.