Webopedia on Google+Webopedia on TwitterWebopedia on FacebookTech Bytes Blog
Main » TERM » R »

robots.txt

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.







TECH RESOURCES FROM OUR PARTNERS
DID YOU KNOW?
Hype Versus Action in the Developer's World

Often times technologies start as hype but with time become adopted. As a developer or technologist, it is worth reading the hype and knowing the... Read More »

Microsoft Hyper-V Network Virtualization Q&A

The top 5 Hyper-V questions with answers provided by Nirmal Sharma, a MCSEx3, MCITP and Microsoft MVP in Directory Services. Read More »

Storage Trends: Solid State and Software Defined

Solid state drives and software defined storage are two leading trends in the rapidly growing storage market.  Read More »

QUICK REFERENCE
How to Create a Desktop Shortcut to a Website

Creating desktop shortcuts to a websites is useful. When you double-click the icon from your desktop it automatically launches the browser and... Read More »

Flash Data Storage Vendor Trends

Although it is almost impossible to keep up with the pace of ongoing product releases, here are three recent highlights in the flash data storage... Read More »

15 Important Big Data Facts for IT Professionals

Keeping track of big data trends, research and statistics gives IT professionals  a solid foundation to plan big data projects. Here are 15... Read More »