Click here

robots.txt

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.



Top Terms
  • 1

    enterprise application

    An enterprise application is the term used to describe applications -- or software -- that a business would use to assist the organization in...

    Read more »

  • 2

    open source

    Generically, open source refers to a program in which the source code is available to the general public for use and/or modification from its...

    Read more »

  • Click Here!

Connect with Webopedia

Did You Know? Archive »

  • Quick Reference Archive »