Webmasters and internet providers began optimising sites for search engines in the mid-1990s, as the pioneering search engines were cataloging the early Web. At first all they needed to do was submit the address of a page to the various engines which would send a spider to “crawl” that page, extract links to other pages from it, and return information found on the page to be indexed. The search engine spider would download a page and store it on the server of that particular search engine. Then a second program, called an indexer, would extract relevant information from the page, for example words it contained and where they are located and any links on the page, and then placed on a schedule to crawl at a later date.

The phrase Search Engine Optimisation came into use in 1997 according to the industry expert Danny Sullivan. Opportunities arose for both black and white hat SEO practitioners, when web-sites started to realise the value of having highly-ranked and visible search engine results.

Early search engines suffered from abuse due to their over-reliance on factors such as keyword density, which is completely in control of the individual web-master. Search engines then responded by developing a more complex algorithmic procedure, which took into account additional factors such as page rank and keywords within the site’s actual content, not just the Meta tags.

A huge opportunity arose in 1998. This was when the world renowned search engine Google was founded. Google attracted an expansive, loyal following, who loved the sites simplistic design. This led to more emphasis on off-page factors such as link and site structure, as well hyperlink analysis. Page rank is more difficult to gain than these and do web-masters developed link building tools and sharing schemes to boost their page rank. Recent years have seen search engines relying more heavily on factors such as the age, sex and location of the people conducting searches.

Google have recently (in 2007) said that their search engine takes over 200 signals into account, not all of which are published.