3 Steps to Successful Bot Management

Toria/Shutterstock.com

Bots drive plenty of online traffic and agencies need to figure out how to deal with them.

Bots are a part of life for any organization doing business online, with many companies estimating that 40 percent of their online traffic is generated by bots. This profusion of bot activity is even more relevant for government agencies as more services move online and constituents expect equally exceptional service from public sector websites as they do from private. With more than 2.54 billion visits from September 2017 through November 2017, U.S. federal websites must be prepared to effectively and securely serve a growing number of visitors—both bot and human. In fact, some government agencies see 60 percent of their traffic attributed to bots.

Traditional IP blocking provides only temporary pain relief until, inevitably, a bot operator detects the block, changes the IP, and starts a new bot invasion. Instead of the traditional approach of simply blocking as many bots as possible, a new paradigm of bot management helps agencies make more real-time and informed decisions on how to handle bot activity. This new model of bot management empowers agency IT to take a proactive approach to detecting, analyzing and making informed decisions on handling all types of bot activity on their websites.

Through this approach, IT groups will be able to:

  • Identify known and unknown bots.
  • Categorize bots based on business impact and detection method.
  • Assign management policies to each bot category.
  • Use a range of sophisticated methods to manage unique bot traffic.
  • Minimize burden on the origin server and business and IT impact.

With proactive in-line bot management, multiple bots that behave similarly can be managed as a group rather than individually. Instead of blocking all bots, agency IT teams can allow all human traffic through and monitor bots that have use of the website, like search engines, business partners and vendors. IT teams can also serve alternative content to harmful bots and redirect other bot traffic to an alternate origin. All of this ensures seamless operations and minimal disruption for constituents enjoying government services online.

Instead of blocking a bad bot and alerting the operator, a proactive bot management solution offers several ways to either slow down the rate that a bad bot can retrieve information or deceive the bot by feeding it different information than what it’s seeking. Rate-based actions can also be implemented to minimize the impact of heavy traffic from good bots. In any case, agencies can better manage the impact on web infrastructure and maintain greater control over the information they choose to give to other parties on the internet, a key concern for government agencies facing cyber threats and hacking attempts. Agencies have a number of strategies to choose from to manage bot activity.

Rate-Based Actions

Websites can manage the impact on the origin of all types of bots by controlling how fast they allow them to scrape the site. This approach is useful for managing partner bots and other good bots that might otherwise cause performance degradation. Even for a bad bot, sometimes the best course of action may be to give it the information it wants, but delay or slow the rate at which it can extract that information and minimize impact on site performance.

Serve Alternative Origin or Cached Content

This strategy allows organizations to better manage the load on their origin from bot traffic. For example, agencies can set aside a number of high-performance servers specifically for legitimate users and other servers for bot categories determined by the business, as well as bad bots detected in real-time. Serving cached content minimizes the load on the origin as well, and returns information already cached without affecting site performance.

Signal the Origin

Agencies may simply prefer to be alerted to bot traffic in order to take action at the origin themselves. In this case, a bot management solution should be able to insert identifying information into any web request labeled as bot-generated.

Overall, the best approach to successful bot management no longer includes simply blocking bots. Instead, agencies should implement a proactive bot management strategy that leverages real-time, in-line big data analytics to enable more options for managing different types of bot traffic. Agencies should be able to evaluate bot traffic for analytics and sample log traffic that indicates the bot’s response to the action taken. Without this visibility, it would be impossible to know for sure if the bot management solution deflected the bot’s impact.

Tom Ruff is the vice president of Public Sector—Americas for Akamai Technologies.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.