Webopedia on Google+Webopedia on TwitterWebopedia on FacebookTech Bytes Blog
Main » TERM » R »

robots.txt


Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.







TECH RESOURCES FROM OUR PARTNERS
DID YOU KNOW?
Small Business Technology Then and Now

Technology for small business has changed a lot in the last decade. But as technology evolves, so do the challenges. Read More »

Windows XP: Move Along, There's Nothing to See Here

After more than 12 years of holding the title of most popular operating system in the world, Windows XP is taking center stage for its final... Read More »

Report: The Role of Big Data in the Marketing Industry

According to a new study from Infogroup Targeting Solutions, we can expect to see companies spend heavily on big data marketing initiatives in... Read More »

QUICK REFERENCE
Flash Data Storage Vendor Trends

Although it is almost impossible to keep up with the pace of ongoing product releases, here are three recent highlights in the flash data storage... Read More »

15 Important Big Data Facts for IT Professionals

Keeping track of big data trends, research and statistics gives IT professionals  a solid foundation to plan big data projects. Here are 15... Read More »

Enterprise Storage Vendors

There's a number of vendors that sell enterprise storage hardware or offer cloud-based enterprise storage. View Webopedia's Enterprise storage... Read More »