According to Matt Cutts at Google, there are some 200 variables in the Google Algorithm.  Each factor unto itself is less than the sum of the parts. The key idea is to efficiently compute for each sensor an estimate of the probability that it matches the query and process sensors in the order of decreasing probability, such that effort is first spent on sensors that are very likely to actually match the query.

Another class of techniques and sensors, known as black hat SEO or spamdexing, use methods such as link farms, keyword stuffing and article spinning that degrade both the relevance of search results and the user-experience of search engines. Search engines look for sites that employ these techniques in order to remove them from their indices. What should be noted, techniques considered acceptable today may be viewed upon with disdain going forward.

Google makes note in its Quality Guidelines, using off-the-shelf software can lead to detrimental results in your rankings.

Increasing prominence
A variety of other methods are employed to get a web page shown at the top of search results. These include:
Cross linking between pages of the same website. Giving more links to main pages of the website, to increase PageRank used by search engines.  Linking from other websites, including link farming and comment spam. Writing content that includes frequently searched keyword phrase, so as to be relevant to a wide variety of search queries. Adding relevant keywords to a web page meta tags, including keyword stuffing. URL normalization of web pages accessible via multiple urls, using the “canonical” meta tags.

Domain

- Age of Domain
- History of domain
- KWs in domain name
- Sub domain or root domain?
- TLD of Domain
- IP address of domain
- Location of IP address / Server

Architecture

- HTML structure
- Use of Headers tags
- URL path
- Use of external CSS / JS files

Content

- Keyword density of page
- Keyword in Title Tag
- Keyword in Meta Description (Not Meta Keywords)
- Keyword in KW in header tags (H1, H2 etc)
- Keyword in body text
- Freshness of Content

Per Inbound Link

- Quality of website linking in
- Quality of web page linking in
- Age of website
- Age of web page
- Relevancy of page’s content
- Location of link (Footer, Navigation, Body text)
- Anchor text if link
- Title attribute of link
- Alt tag of images linking
- Country specific TLD domain
- Authority TLD (.edu, .gov)
- Location of server
- Authority Link (CNN, BBC, etc)

Cluster of Links

- Uniqueness of Class C address.

Internal Cross Linking

- Number of internal links to page
- Location of link on page
- Anchor text of FIRST text link

Penalties

- Over Optimization
- Purchasing Links
- Selling Links
-Links from Web Spam sites
-frequent server down time
- Comment Spamming
- Cloaking
Cloaking by cookie detection
Cloaking by JavaScript
Cloaking by rich media support
Cloaking by IP address
Cloaking by User Agent

- Hidden Text
- Duplicate Content
- Keyword stuffing
- Manual penalties
- Sandbox effect

Miscellaneous

- JavaScript Links
- No Follow Links

Pending

- Performance / Load of a website
- Speed of JavaScript

Misconceptions

- XML Sitemap (Aids the crawler but doesn’t help rankings)
- PageRank (General Indicator of page’s performance)

Photo by pyratorKeyword stuffing is a technique used – and overused – by SEOs to recharge keywords on a site so that search engines find it more relevant to those particular keywords. This technique is based on one factor that search engines weighed to determine the relevance of a document X for a keyword.

This practice is strongly discouraged by the rules of search engines as it is considered a hoax, destroys the relevance of the web, the end user experience and the readability of the article gets polluted and diluted.

For example, say this post is relevant to the search engines keyword ‘overload’.

“I say the keyword overload is a technique referred to SEO keywords and overload overload which consists of keywords, keywords and keywords, and more stress from them, and that the results of this technique for keywords on search engines based on your keywords overload”

Clearly I’m doing keyword stuffing (and it reads pretty stupid too).

The reality is the world of search engines today are less naive and sorting algorithms are just no longer sensitive to keyword stuffing, with much of the burden of relevance moved to off-page factors and calculations of proximity. Keyword stuffing is a primitive practice that demonstrates little knowledge and a certain contempt for the reader.

Indeed, this technique can still get some results on MSN and to a lesser extent Yahoo, but Google definitely does little or nothing for indexing in the SERP’s.

There are several ways to do keyword stuffing. A slightly more elaborate method is repeatedly putting in a keyword input type = “field” or in the alt text of an image. The user does not see the form field but the search engines do. Others love stuffing hidden text pages full of keywords. The concealment is the color of the font or CSS.

Keyword stuffing is also called loading or spamdexing.

According to various mathematical formulas, keyword stuffing (the high density of keywords) has no effect on the body of the text of a website for the relevance of the site in search engines. The reason is that in the process of indexing the keywords weight of each is based on its position to the other keywords. This is what determines the prominence of each keyword. In other words, if you fill your text with keywords, each keywords prominence is reduced significantly, thereby reducing its impact in relation to other keywords.

Controlling what content is blocked from being found in search engines is crucial for many websites. Fortunately, the major search engines and other well-behaved robots observe the Robots Exclusion Protocol (REP), which has evolved organically since the early 1990′s to provide a set of controls over what parts of a web site search engines robots can crawl and index.

Article Sections:

* Capabilities of REP
* Deciding what should be Public vs. Private
* Implementing the REP
* Common implementation mistakes
* Testing your implementation
* Verifying robot identity
* Removing content from search engine indices
* Additional resources

Capabilities of the REP

The Robots Exclusion Protocol provides controls that can be applied at the site level (robots.txt), at the page level (META tag, or X-Robots-Tag), or at the HTML element level to control both the crawl of your site and the way it’s listed in the search engine results pages (SERPs). Below is a table listing the common scenarios, directives, and which search engines support them.
Use Case  Robots.txt  META/ X-Robots-Tag  Other  Supported By
Allow access to your content  Allow  FOLLOW
INDEX    Google
Yahoo
Microsoft
Disallow access to your content  Disallow
NOINDEX
NOFOLLOW    Google
Yahoo
Microsoft
Disallow access to index images on the page    NOIMAGEINDEX    Google
Disallow the display of a cached version of your content in the SERP    NOARCHIVE    Google
Yahoo
Microsoft
Disallow the creation of a description for this content in the SERP    NOSNIPPET    Google
Yahoo
Microsoft
Disallow the translation of your content into other languages    NOTRANSLATE    Google
Do not follow or give weight to links within this content    NOFOLLOW  a href attribute:
rel=NOFOLLOW  Google
Yahoo
Microsoft
Do not use the Open Directory Project (ODP) to create descriptions for your content in the SERP    NOODP    Google
Yahoo
Microsoft
Do not use the Yahoo Directory to create descriptions for your content in the SERP    NOYDIR    Yahoo
Do not index this specific element within an HTML page      class=robots-nocontent  Yahoo
Stop indexing this content after a specific date    UNAVAILABLE_AFTER    Google
Specify a sitemap file or a sitemap index file  Sitemap      Google
Yahoo
Microsoft
Specify how frequently a crawler may access your website  Crawl-Delay
Google WMT  Yahoo
Microsoft
Authenticate the identity of the crawler      Reverse DNS Lookup  Google
Yahoo
Microsoft
Request removal of your content from the engine’s index      Google WMT
Yahoo SE
Microsoft WMT  Google
Yahoo
Microsoft
Deciding what should be Public vs. Private

One of the first steps in managing the robots is knowing what type of content should be public vs. private. Start with the assumption that by default, everything is public, then explicitly identify the items that are private.

If you want search engines to access all the content on your site, you don’t need a robots.txt file at all. When a search engine tries to access the robots.txt file on your site and the server can’t return one (ideally by returning a 404 HTTP status code), the search engine treats this the same as a robots.txt file that allows access to everything.

Every website and every business has a different set of needs, so there’s no blanket rule for what to make private, but some common elements may apply.

* Private data – You may have content on your site that you don’t want to be searchable in search engines. For instance, you may have private user information (such as addresses) that you don’t want surfaced. For this type of content, you may want to use a more secure approach that keeps all visitors from the pages (such as password protection). However, some types of content are fine for visitor access, but not search engine access. For instance, you may run a discussion forum that is open for public viewing, but you may not want individual posts to appear in search results for forum member names.
* Non-content content – Some content, like images used for navigation, provides little value to searchers. It’s not harmful to include these items in search engine indices, but since search engines allocate limited bandwidth to crawl each site and limited space to store content from each site, it may make sense to block these items to help direct the bots to the content on your site that you do want indexed.
* Printer-friendly pages – if you have specific pages (URLs) that are formatted for printing you may want to block them out to avoid duplicate content issues. The drawback to allowing the printer-friendly page to be indexed is that it could potentially be listed in the search results instead of the default version of the page, which wouldn’t provide an ideal user experience for a visitor coming to the site through search.
* Affiliate links and advertising – If you include advertising on your site, you can keep search engine robots from following the links by redirecting them to a blocked page, then on to the destination page. (There are other methods for implementing advertising-based links as well.)
* Landing pages – Your site may include multiple variations of entry pages used for advertising purposes. For instance, you may run AdWords campaigns that link to a particular version of a page based on the ad, or you may print different URLs for different print ad campaigns (either for tracking purposes or to provide a custom experience related to the ad). Since these pages are meant to be an extension of the ad, and are generally near duplicates of the default version of the page, you may want to block these landing pages from being indexed.
* Experimental pages – As you try new ideas on your site (for instance, using A/B testing), you likely want to block all but the original page from being indexed during the experiment.

Implementing the REP

REP is flexible and can be implemented a number of ways. This flexibility lets you easily specify some policies for your entire site (or subdomain) and then enhance them more granularly at the page or link level as needed.
Site Level Implementation (Robots.txt)

Site wide directives are stored in a robots.txt file, which must be located in the root directory of each domain or sub-domain. (e.g. http://janeandrobot.com/robots.txt). Robots.txt files located in subfolders are ignored.

A robots.txt file is a UTF-8 encoded file that contains entries that consist of a user-agent line (that tells the search engine robot if the entry is directed at it) and one or more directives that specify content that the search engine robot is blocked from crawling or indexing. A simple robots.txt file is shown below.

User-agent: *

Disallow: /private

user-agent: – Specifies which robots the entry applies to.

* Set this to * to specify that this entry applies to all search engine robots.
* Set this to a specific robot name to provide instructions for just that robot. You can find a complete list of robot names at robotstxt.org.
* If you direct an entry at a particular robot, then it obeys that entry instead of any entries defined for user-agent: * (rather than in addition to those entries).

The major search engines have multiple robots that crawl the web for different types of content (such as images or mobile). They generally begin all robots with the same name so that if you block the major robot, all robots for that search engine are blocked as well. However, if you want to block only the more specific robot, you can block it directly and still allow web crawl access.

* Google – The primary search engine robot is Googlebot.
* Yahoo! – The primary search engine robot is Slurp.
* Live Search – The primary search engine robots is MSNbot.

Disallow: – Specifies what content is blocked

* Must begin with a slash (/).
* Blocks access to any URLs that begin with the characters after the /. For instance, Disallow: /images blocks access to /images/, /images/image1.jpg, and /images10.

You can specify other rules for search engine robots in addition to the standard instructions that block access to content as noted in other robot instructions.

Some things to note about robots.txt implementation:

* The major search engines support pattern matching using the asterisk character (*) for wildcard match and the dollar sign ($) for end of sequence matching as described below in using pattern matching.
* The robots.txt file is case sensitive, so Disallow: /images would block http://www.example.com/images but not http://www.example.com/Images.
* If conflicts exist in the file, the robot obeys the longest (and therefore generally more specific) line.

Basic Samples

Block all robots – Useful when your site is in pre-launch development and isn’t ready for search traffic.

# This keeps out all well-behaved robots.

# Disallow: * is not valid.

User-agent: *

Disallow: /

Keep out all bots by default – Blocks all pages except those specified. Not recommended as is difficult to maintain and diagnose.

# Stay out unless otherwise stated

User-agent: *

Disallow: /

Allow: /Public/

Allow: /articles/

Allow: /images/

Block specific content – The most common usage of robots.txt.

# Block access to the images folder

User-agent: *Disallow: /images/

Allow specific content – Block a folder, but allow access to selected pages in that folder.

# Block everything in the images folder

# Except allow images/image1.jpg

User-agent: *

Disallow: /images/Allow: /images/image1.jpg

Allow specific robot – Block a class of robots (for instance, Googlebot), but allow a specific bot in that class (for instance, Googlebot-Mobile).

# Block Googlebot access

# Allow Googlebot-Mobile access

User-agent: Googlebot

Disallow: /

User-agent: Googlebot-Mobile

Allow: /

Pattern Matching Examples

The major engines support two types of pattern matching.

* * matches any sequence of characters
* $ matches the end of URL.

Block access to URLs that contain a set of characters – Use the asterisk (*) to specify a wildcard.

# Block access to all URLs that include an ampersand

User-agent: *

Disallow: /*&

This directive would block search engines from crawling http://www.example.com/page1.asp?id=5&sessionid=xyz.

Block access to URLs that end with a set of characters – Use the dollar sign ($) to specify end of line.

# Block access to all URLs that end in .cgi

User-agent: *

Disallow: /*.cgi$

This directive would block search engines from crawling http://www.example.com/script1.cgi but not from crawling http://www.example.com/script1.cgi?value=1.

Selectively allow access to a URL that matches a blocked pattern – Use the Allow directive in conjunction with pattern matching for more complex implementations.

# Block access to URLs that contain ?

# Allow access to URLs that end in ?

User-agent: *Disallow: /*?Allow: /*?$

That directive blocks all URLs that contain ? except those that end in ?. In this example, the default version of the page will be indexable:

* http://www.example.com/productlisting.aspx?

Variations of the page will be blocked:

* http://www.example.com/productlisting.aspx?nav=price
* http://www.example.com/productlisting.aspx?sort=alpha

Other robot instructions

Specify a Sitemap or Sitemap index file – If you’d like to provide search engines with a comprehensive list of your best URLs, you can provide one or more Sitemap autodiscovery directives. Note, user-agent does not apply to this directive so you cannot use this to specify a Sitemap to some but not all search engines.

# Please take my sitemap and index everything!

Sitemap: http://janeandrobot.com/sitemap.axd

Reduce the crawling load – This only works with Microsoft and Yahoo. For Google you’ll need to specify a slower crawling speed through their Webmaster Tools. Be careful when implementing this because if you slow down the crawl too much, robots won’t be able to get to all of your site and you may lose pages from the index.

# MSNBot, please wait 5 seconds in between visits

User-agent: msnbot

Crawl-delay: 5

# Yahoo’s Slurp, please wait 12 seconds in between visits

User-agent: slurp

Crawl-delay: 12

Page Level Implementation (META Tags)

The REP page-level directives allow you to refine the site wide policies on a page-by-page basis

Placing a meta tag on the page – Place the meta tag in the head tag. Each directive should be comma delimited inside the tag. E.g. <meta name=”ROBOTS” content=”Directive1, Directive 2>.

<html>

<head>

<title>Your title here</title>

<meta name=”ROBOTS” content=”NOINDEX”>

</head>

<body>Your page here</body>

</html>

Targeting a specific search engine – Within the meta tag you can specify which search engine you would like to target, or you can target them all.

<!– Applies to All Robots –>

<meta name=”ROBOTS” content=”NOINDEX”>

<!– ONLY GoogleBot –>

<meta name=”Googlebot” content=”NOINDEX”>

<!– ONLY Slurp (Yahoo) –>

<meta name=”Slurp” content=”NOINDEX”>

<!– ONLY MSNBot (Microsoft) –>

<meta name=”MSNBot” content=”NOINDEX”>

Control how your listings – there are a set of options you can use to determine how your site will show up on the SERP. You can exert some control over how the description is created, and remove the “Cached page” link.

Example search engine results page (SERP)

<!– Do not show a description for this page –>

<meta name=”ROBOTS” content=”NOSNIPPET”>

<!– Do not use http://dmoz.org to create a description –>

<meta name=”ROBOTS” content=”NOODP”>

<!– Do not present a cached version of the document in a search result –>

<meta name=”ROBOTS” content=”NOARCHIVE”>

Using other directives – Other meta robots directives are shown below.

<!– Do not trust links on this page, could be user generated content (UCG) –>

<meta name=”ROBOTS” content=”NOFOLLOW”>

<!– Do not index this page –>

<meta name=”ROBOTS” content=”NOINDEX”>

<!– Do not index any images on this page (will still index the if they are linked

elsewhere) Better to use Robots.txt if you really want them safe.

This is a Google Only tag. –>

<meta name=”GOOGLEBOT” content=”NOIMAGEINDEX”>

<!– Do not translate this page into other languages–>

<meta name=”ROBOTS” content=”NOTRANSLATE”>

<!– NOT RECOMMENDED, there really isn’t much point in using these –>

<meta name=”ROBOTS” content=”FOLLOW”>

<meta name=”ROBOTS” content=”UNAVAILABLE_AFTER”>

HTTP Header Implementation (X-ROBOTS-Tag)

Allows developers to specify page-level REP directives for non text/html content types like PDF, DOC, PPT, or dynamically generated images.

Using the X-Robots-Tag – to use the X-Robots-Tag, simply add it to your header as shown below. To specify multiple directives you can either comma delimit them, or add them as separate header items.

HTTP/1.x 200 OK

Cache-Control: private

Content-Length: 2199552

Content-Type: application/octet-stream

Server: Microsoft-IIS/7.0

content-disposition: inline; filename=01 – The truth about SEO.ppt

X-Robots-Tag: noindex, nosnippet

X-Powered-By: ASP.NET

Date: Sun, 01 Jun 2008 19:25:47 GMT

The X-Robots-Tag directive supports most of the same directives as the meta tag. The only limitation with this method over the meta tag implementation is that there is no way to target a specific robot – though that probably isn’t a big deal for most use cases.

* X-Robots-Tag: noindex
* X-Robots-Tag: nosnippet
* X-Robots-Tag: notranslate
* X-Robots-Tag: noarchive
* X-Robots-Tag: unavailable_after: 7 Jul 2007 16:30:00 GMT

Content Level Implementation

You can further refine your site level and page level directives within several content tags.

Each anchor tag (link) can be modified to tell search engines that you do not trust where this URL is pointing to. This is typically used for links within user generated content (UCG) like wikis, blog comments, reviews and other community sites.

<a href=”#” rel=”NOFOLLOW”>My Hyperlink</a>

Also, in Yahoo Search you can specify which <div> elements on a page you would not like indexed using the class=robots-nocontent attribute. However, we don’t highly recommend using this tag because it is not supported in any other engine, making it not super-useful.

<div class=”robots-nocontent”>

No content for you! (or at least Yahoo!)

</div>

Common implementation mistakes

While implementing the REP is generally straight-forward, there are a few common mistakes.

*

GoogleBot follows the most specific directive, ignoring all others. In the robots.txt file, if you specify a section for all user-agents (user-agent: *) and also declare a section for Googlebot (user-agent: Googlebot), Google will disregard all sections in the robots.txt file except the Googlebot section. This could potentially leave you exposing much more content to Google that you might have thought.

# This keeps out all well-behaved robots

User-agent: *

Disallow: /

# This looks like it is giving Google access to only this directory, but since it is a

# GoogleBot specific section, Google will disregard the previous section

# and access the whole site.

User-agent: Googlebot

Allow: /Content_For_Google/

*

NOFOLLOW will most likely not prevent indexing – if you use NOFOLLOW at either the page or the link level, it is still possible for the links from the page to be indexed because the search engine may have found a reference to them from another source. Another note, using rel=”NOFOLLOW” within your anchor text is still perceived as a recommendation by the search engines, not a command.

To ensure that content is not indexed, either use the Disallow directive at the site level, or use NOINDEX at the page level.
*

Directives that are not recommended – the directives in the REP are all about exceptions, by default the robots assume they can crawl your whole site. Therefore, you do not need to explicitly use the FOLLOW and INDEX directives as they will not be taken into account by the search engines. It sounds silly but I’ve seen a few sites that have implemented these on every page and every link.

Another directive that is not recommended is the NOCACHE directive. This was created by Microsoft, and is synonymous with NOARCHIVE. While they will most likely always continue to support the directive, it is better to use NOARCHIVE so it will work on all the search engines.

Testing your implementation

As you’re implementing your REP design, you may want to test it to see how it is working out. The easiest way to test this is to use the robots validator in either Google or Microsoft’s Webmaster Tools. These tools are generally good enough test beds for most folks, however advanced developers (or paranoid ones with critical business requirements) will want to definitively know what the robots are doing, not simply rely on what the robots say they are doing. These folks will want to look at their tools as well look at their server logs and even possibly setup some specific test.
The Easy Way

Both Google and Microsoft provide some tools as part of their Webmaster Centers to help you verify if you’ve configured your REP the way you expect. Let’s start with Google’s tools:

The first thing you should check are the list of URLs that Google has seen from your website and not indexed due to the REP. Note you can also download the list and filter, sort, and have-your-way-with-it in Excel.

Google Webmaster Tools: Blocked URLs

The next step is to use their interactive robots.txt tool to analyze your rules and test specific URLs for blockage. When you pull up the tool they already should have it pre-populated with the robots.txt file they have on file from the last time they crawled. You can input a list of URLs you’d like to check below, select the user-agent you’d like to check against and the tool will tell you if they are blocked or not. You can also use the tool to test changes to your robots.txt file to see how Google would interpret things.

Google Webmaster Tools: robots.txt analysis

Microsoft has a similar tool in their Webmaster Center that will validate a robots.txt file against the standard that MSNBot supports. To use the tool, simply log in copy & paste your robots.txt file into the top field and select Validate. A list of all detectable issues are displayed in the bottom box.

Microsoft Live Search Webmaster Tools: robots.txt validator
The Hard Way (More Accurate)

If you have a specific business need to ensure that the robots are following your rules, (or you’re just paranoid) then you should not simply rely on the tools they provide to test compliance. You’re going to need to go straight to the horse’s mouth and analyze your web server logs to see exactly what they are doing. There is no one easy tool for doing this, you’ll likely have to use an existing tool like one of these (Microsoft HTTP Log Parser) or write your own. It isn’t difficult, it will simply take some time to implement. A useful reference for this is a list of all the robot user agents, and more complete list of bots from Google, and Microsoft.
When Blocked Content Appears to be Indexed

If search engines are blocked from crawling pages, they may still index the URL if the robot finds a link to that URL on a page that isn’t blocked. The listing may display the URL only, such as shown below.

Google partially indexed results

Or, it may include a title and in some instances, a description. This makes it appear as though the search engine robot is disregarding the directive that blocks access to the page, but the search engine is in fact obeying the directive not to crawl the page and is using anchor text from the link to that page and descriptive details from either the page that contains the link or a source such as the Open Directory Project.

For more details, see:

* Google: partially indexed page
* Yahoo!: thin documents

Verifying Robot Identity

Another thing you’ll likely want to consider in this endeavor is to validate that the robots are who they actually say they are. Google, Yahoo and Microsoft all support Reverse DNS authentication of their robots. The process is pretty simple and described here by Google, Yahoo and Microsoft, essentially you simply find out what range their robot’s DNS is hosted in, and use that in your tool. This way, if the address changes (which it will), you don’t need to update your code.

Should you find any issues, where one of the robots are not minding the REP, or are misbehaving in some other way, you can always communicate directly with each engine through one of their forums:

* Google Crawling, Indexing and Ranking Forum
* Yahoo Crawler Feedback Form
* Microsoft Crawler Error and Feedback Forum

Removing Content From Search Engine Indices

If you find that you haven’t implemented the techniques described here correctly and private content from your site is indexed, each of the major search engines has methods available for requesting that it be removed. For more information, see:

* Google: Requesting removal of content from our index
* Yahoo!: Deleting URLs

Additional Resources:

* Google
o How to create a robots.txt file
o Descriptions of each user-agent that Google uses
o How to use pattern matching
o How often we recrawl your robots.txt file
o All about Googlebot
* Yahoo!
o Wild card support
o X-Robots tag directive support
* Microsoft Live Search
o Search robots in disguise
* Other resources
o Search Engine Land: Meta Robots Tag 101
o Search Engine Land: Yahoo!, Microsoft, Google Clarify Robots.txt Support
o Search Engine Land: URL Removal Options
o robotstxt.org
o Wikipedia: Robots Exclusion Standard