Spider

A program that automatically fetches Web pages. Spiders are used to feed pages to search engines. It’s called a spider because it crawls over the Web. Another term for these programs is webcrawler.

Because most Web pages contain links to other pages, a spider can start almost anywhere. As soon as it sees a link to another page, it goes off and fetches it. Large search engines, like Alta Vista, have many spiders working in parallel.

Also see How Web Search Engines Work in the Did You Know . . . ? section of Webopedia.

Vangie Beal
Vangie Beal
Vangie Beal is a freelance business and technology writer covering Internet technologies and online business since the late '90s.

Related Articles

@ Sign

Pronounced at sign or simply as at, this symbol is used in e-mail addressing to separate the user' name from the user's domain name,...

Munging

(MUHN-jing) Munging (address munging), is the act of altering an email address posted on a Web page to make it unreadable to bots and...

How to Create an RSS Feed

In the second installment of RSS how-to, we look at some of the nonrequired (optional) channel and item tags, discuss RSS specifications in-depth and...

Dictionary Attack

(n.) (1) A method used to break security systems, specifically password-based security systems, in which the attacker systematically tests all possible passwords beginning with...

ScalaHosting

ScalaHosting is a leading managed hosting provider that offers secure, scalable, and affordable...

HRIS

Human resources information system (HRIS) solutions help businesses manage multiple facets of their...

Best Managed Service Providers...

In today's business world, managed services are more critical than ever. They can...