3 Steps to Successful Bot Management

Bots are a part of life for any organization doing business online, with many companies estimating that 40 percent of their online traffic is generated by bots. This profusion of bot activity is even more relevant for government agencies as more services move online and constituents expect equally exceptional service from public sector websites as they do from private. With more than 2.54 billion visits from September 2017 through November 2017, U.S. federal websites must be prepared to effectively and securely serve a growing number of visitors—both bot and human. In fact, some government agencies see 60 percent of their traffic attributed to bots.

Traditional IP blocking provides only temporary pain relief until, inevitably, a bot operator detects the block, changes the IP, and starts a new bot invasion. Instead of the traditional approach of simply blocking as many bots as possible, a new paradigm of bot management helps agencies make more real-time and informed decisions on how to handle bot activity. This new model of bot management empowers agency IT to take a proactive approach to detecting, analyzing and making informed decisions on handling all types of bot activity on their websites.

Through this approach, IT groups will be able to:

• Identify known and unknown bots.
• Categorize bots based on business impact and detection method.
• Assign management policies to each bot category.
• Use a range of sophisticated methods to manage unique bot traffic.
• Minimize burden on the origin server and business and IT impact.

With proactive in-line bot management, multiple bots that behave similarly can be managed as a group rather than individually. Instead of blocking all bots, agency IT teams can allow all human traffic through and monitor bots that have use of the website, like search engines, business partners and vendors. IT teams can also serve alternative content to harmful bots and redirect other bot traffic to an alternate origin. All of this ensures seamless operations and minimal disruption for constituents enjoying government services online.

Instead of blocking a bad bot and alerting the operator, a proactive bot management solution offers several ways to either slow down the rate that a bad bot can retrieve information or deceive the bot by feeding it different information than what it’s seeking. Rate-based actions can also be implemented to minimize the impact of heavy traffic from good bots. In any case, agencies can better manage the impact on web infrastructure and maintain greater control over the information they choose to give to other parties on the internet, a key concern for government agencies facing cyber threats and hacking attempts. Agencies have a number of strategies to choose from to manage bot activity.

Rate-Based Actions

Websites can manage the impact on the origin of all types of bots by controlling how fast they allow them to scrape the site. This approach is useful for managing partner bots and other good bots that might otherwise cause performance degradation. Even for a bad bot, sometimes the best course of action may be to give it the information it wants, but delay or slow the rate at which it can extract that information and minimize impact on site performance.

Serve Alternative Origin or Cached Content

This strategy allows organizations to better manage the load on their origin from bot traffic. For example, agencies can set aside a number of high-performance servers specifically for legitimate users and other servers for bot categories determined by the business, as well as bad bots detected in real-time. Serving cached content minimizes the load on the origin as well, and returns information already cached without affecting site performance.

Signal the Origin

Agencies may simply prefer to be alerted to bot traffic in order to take action at the origin themselves. In this case, a bot management solution should be able to insert identifying information into any web request labeled as bot-generated.

Overall, the best approach to successful bot management no longer includes simply blocking bots. Instead, agencies should implement a proactive bot management strategy that leverages real-time, in-line big data analytics to enable more options for managing different types of bot traffic. Agencies should be able to evaluate bot traffic for analytics and sample log traffic that indicates the bot’s response to the action taken. Without this visibility, it would be impossible to know for sure if the bot management solution deflected the bot’s impact.

Tom Ruff is the vice president of Public Sector—Americas for Akamai Technologies.