Webopedia on Google+Webopedia on TwitterWebopedia on FacebookTech Bytes Blog
Main » TERM » R »

robots.txt


Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.







TECH RESOURCES FROM OUR PARTNERS
DID YOU KNOW?
Keeping Data Secure Is Tougher than Ever

If hackers get their hands on your company's data, they can wreak havoc on customer relationships and cause tremendous damage to your brand and... Read More »

Windows XP: Move Along, There's Nothing to See Here

After more than 12 years of holding the title of most popular operating system in the world, Windows XP is taking center stage for its final... Read More »

Report: The Role of Big Data in the Marketing Industry

According to a new study from Infogroup Targeting Solutions, we can expect to see companies spend heavily on big data marketing initiatives in... Read More »

QUICK REFERENCE
How to Create a Desktop Shortcut to a Website

Creating desktop shortcuts to a websites is useful. When you double-click the icon from your desktop it automatically launches the browser and... Read More »

Flash Data Storage Vendor Trends

Although it is almost impossible to keep up with the pace of ongoing product releases, here are three recent highlights in the flash data storage... Read More »

15 Important Big Data Facts for IT Professionals

Keeping track of big data trends, research and statistics gives IT professionals  a solid foundation to plan big data projects. Here are 15... Read More »