Scrape

Scraping is the process of using automated tools to collect large amounts of data output from an application, website or application programming interface (API). The most common tool used for scraping is bots. These bots extract HTML code and other data stored inside databases.

Data scraping is used for both legitimate purposes, as well as with illegal, nefarious intentions. Digital businesses may use scraping to harvest data for legal purposes, such as auto-fetching prices for price comparison sites or to perform market research from forums and social media platforms. On the other hand, malicious actors may use scraping to steal copyrighted content or undercut unlisted competitor prices.

How scraping works

Legitimate scraping typically uses pre-built bots, scripts, or scraping-as-a-service providers. 

Malicious parties often create their own scripts for data scraping that don’t abide by certain restrictions, such as disguising themselves as real users. 

The following are the typical steps in the malicious scraping process:

1. Identify the target website or application.
2. Malicious actors will then limit the possibility of detection by creating fake user accounts and obfuscate source IP addresses.
3. Bots are deployed across the resource. In addition to scraping, these illegal bots use can sometimes overload servers, leading to slow website performance and possibly crashing it entirely.
4. Finally, content and database information is extracted and stored in the actor’s own database.

Types of data scraping

There are three primary types of data scraping.

Content scraping

Content scraping refers to when bots scrape the content present on a website. This information can then be replicated to mirror the unique advantages of products and services that rely on site content.

Price scraping

Price scraping using bots to pull data on prices. This can be used for legitimate purposes for comparison sites but can also be used to undercut competitor prices or create unique advantages over pricing plans.

Contact scraping

Contact scraping pulls user data, such as email addresses and phone numbers. Spammers and scammers use this information for bulk email lists, robocalls and social engineering attacks.

Data scraping vs. data crawling

Crawling

Crawling is used to index content. The most common example of this is Google using Googlebots to crawl website content to inform search engine results. Crawler bots make no attempt to hide their identity when crawling sites.

Scraping

Scraping specifically pulls data and stores it in other databases. Scraper bots typically hide their identity by pretending to be web browsers or users. They take more advanced actions than crawler bots, such as filling out form fields.

How to protect against data scraping

The following steps can be used to protect against data scraping:

1. Monitor new and existing user accounts with high levels of activity but who haven’t made any purchases.
2. Look for unusually high traffic to particular assets.
3. Look at competitors for signs of price and catalog matching.
4. Use software tools that use behavioral analysis to identify malicious activity in identifying bad bots.