Scrape Defender

The Problem: Many software developers rely on screen scraping to access free content that is published on web sites. These developers access a web site programmatically and process the large numbers of pages to steal any information that they want.

The Solution: So, how do they tell good bots from bad? ScrapeDefender generates a diverse collection of anti-scraping metrics about the offending user to determine what their goals are. For example, the GoogleBot announces who it is clearly when it attempts to crawl your site. Most poorly behaved bots try to hide their identity by stating that they are well known browsers like Internet Explorer or Firefox. By combining the behavior of the user with the declared identity of the browser, ScrapeDefender is able to clearly determine if this is a person or a computer. With this information, you have to decide if this scraper is a net negative or a net positive for your company and your bottom line. The ScrapeDefender team can help you figure that out, but for each suspicious scraper identified, there is a business decision to make. For example, you might have a scraper grabbing one page of data once a day. That is probably not costing you enough to spend the effort to block them. Or you could have a scraper pulling down your entire site every day, which is worth blocking.