SEO Basics – Attracting Google Bot And Other Spiders


The first part of gaining good rankings with search engines is ensuring that the search engines can find you. There are a few methods to ensure your site gets indexed and therefore listed but they can differ greatly in effectiveness, timeliness, and cost. One of the most common ways that Webmasters attempt to grab the attention of the search engines is through search engine submission, but does this really provide the results that you want? In this article we will really only be looking at the three major search engines – Google, Yahoo, and MSN.

Search Engine Submission

Search engine submission is often hocked as being the most effective way of getting your site indexed. While it is possible to submit your site to the major search engines the largest flaw in this process is that you will have to wait for weeks, or months, before you can even begin to hope to see results.

Search engines prefer to work in a more natural and fluid way so this staged method of submitting your site is not the preferred solution for them. All of the big three search engines include a "submit your site" link but do have prepared to wait some reasonable time for inclusion. In the majority of cases, your site will be indexed by one of the other methods before the search engines respond to your inclusion request.

Directory Submission

Search engines tend to use a number of directories in order to bulk up their results. They find one or more directories that they trust and regularly update their own index with that of the directories. For years, gaining inclusion into the Open Directory Project, also known as the ODP or DMOZ, meant a listing in Google was soon to follow.

However, over the last year or so DMOZ has come under fire from many avenues. Some Webmasters claimed that editors were not including their sites because they were in direct competition to the editors' own sites. Exactly how true this is, I would not like to speculate but there are some factors that are certainly true.

It takes a long time to be accepted in the DMOZ directory, if your site is ever accepted at all. For many months the site submission pages and process were down, and the same was also true of the editing directory. This meant that no sites whatever would have been accepted. While this has now been fixed you should still expect to wait many months before getting DMOZ inclusion.

There are a great many directories included on the Internet, some are niche directories based on specific topics while others are general topic directories. While the search engines may not consider all of these to be authoritative links and may not use them to update their index with, you still gain the advantage of having an inbound link to your website.

Inbound Links

Inbound links are the preferred method to getting your site indexed. Spiders are deployed by search engines in order to crawl the Internet. These spiders follow HTML links from one page to another and from one site to another. When the spider reaches a page that has not already indexed already, they add it to the index. This results in your site and your pages (if your navigation is done properly) being indexed.

Links from websites can provide quick results in terms of being indexed but it depends on the site that provides the link. The more often the site is indexed, the quicker your site is likely to find its way into the search engine indexes. Try to find a popular site that updates with new content on a regular basis and gain a link back to your site.

In most cases you can gain a free link from another website and be indexed within a week or two. This is one of, if not the quickest method to get your site found by the spiders and subsequently indexed by the search engines.

Sitemap Submission

Another comparatively new technique of getting indexed is through the use of compliant sitemaps. Google, Yahoo, and ever MSN Live! offer Webmasters a suite of tools to help produce and publish their websites, as well as tools to monitor performance and help with indexing. As such, they now all provide a method of submitting a sitemap and claim that doing so means quick indexing.

The preferred format for sitemaps is XML but opinions seem to be divided on the effects of trying to get indexed in this way. The fact that the three major search engines have agreed to offer this feature and have even agreed to standard protocol for the use of sitemaps suggests that it is beneficial.

Using The Preferred Methods

All of the above methods will work, ever. Search engine submission takes the longest and is essentially redundant because the remaining methods offer quicker inclusion and are much simpler. Gaining inbound links is a vital aspect of search engine optimization anyway so gaining a small number of quality links to start with is certainly not a bad thing, and can get your site indexed in a matter of a few days.

Sitemaps are a fairly new innovation and while the jury is still out until proven results are published, they are easy and free to establish, maintain, publish, and submit. Even if you are already indexed they can help in ensuring that all the pages of your site are included in the search indexes.


Leave a Reply