Types of search engines
There are mainly two types of search engines:
Search crawler often provide up-to-date results, while Directories provide much more
Index created by search crawler or spider: the search spider visits a webpage, reads it,
and then follows links to other pages within the site. Everything the spider finds goes in
Directories created by person, listing your page to an appropriate category.
How to indicate search engine spiders which part of your website should not be accessed
You can use robot.txt to control search spiders, the content of the file should be like:
For example, if you don't want "google" to access "./images/flower.gif" and all files under
"./tmp" directory, you may use following:
For more information, see The Robot Exclusion Standard
How to indicate search engine spiders not keep a cached version of your page section of your documents:
You can use META tag to control search spiders,
Place this in the
<META NAME="ROBOTS" CONTENT="NOARCHIVE">
How to use META tag to promote your site
Most search engines use META tags as one component of their ranking formulas.
Following are some META tag you may need to add to your page:
- Description: META Tags Describe Your Content.
- Keywords: META Tags include your key words.
- Copyright: The copyright date of the page.
- Rating: Use this META tag to rate your page.
- Robots: tell the robot the path of crawl
- Revisit-after: tell the robot after how many days to revisit your page.