A bot is an automated application used to perform simple and repetitive tasks that would be time consuming, mundane or impossible for a human to perform. Bots are frequently used for malicious purposes which include web scraping, competitive data mining, personal and financial data harvesting, brute-force login and digital ad fraud, spam, transaction fraud, and more. These are just few of them to name and are termed as bad bots. These bad bots interacts with applications in the same way a legitimate user would, making them harder to prevent. They enable high-speed abuse, misuse, and attacks on websites and APIs.
The frequency has increased over the years as we see more sophisticated bad bots coming insurface. The 2019 Distil Network report on Bad Bots reveals that attackers are using more sophisticated techniques and their ability to go undetected grow at a rapid speed. The study says that in the year 2018, 37.9% of all internet traffic wasn’t human, and there were year-over-year decreases in both bad bot (-6.4%) and good bot (-14.4%) traffic. Human traffic increased by 7.5% to 62.1%.
Every industry has its own bad bot problem and ecosystem of bot operators. The threat is more from Advanced persistent bots (APBs) as they as affecting industries. The goal of each bot operator might be different depending on their industry. There is an ecosystem within many industries that rely on bots for survival.
The report investigated hundreds of billions of bad bot requests from 2018 over thousands of domains to provide deeper insight into the daily automated attacks wreaking havoc on websites, mobile apps and APIs. Increasingly, the major industry analyst firms are realizing that bot management was a blind spot in the cybersecurity landscape. They’re recommending that for comprehensive web application security, addressing bad bots is a key component.
Sophisticated Bad Bots Problem Affecting Various Industry
Sophisticated Bad Bots problem are affecting every industry and few are industry specific. The Distil study further takes us to examine the traffic from various industries and dig deeper insight into the bot problem.
For the 2018 Bad Bot Report, data was collected from 11 industries. For this report the number of industries expanded to 20. Comparing bad bot sophistication levels by industry reveals a very different picture. Ticketing, healthcare, directories & classifieds, and ecommerce see the highest proportion of sophisticated bots.
Financial Services companies have the highest percentage of bad bots with 42.2%. Such companies typically suffer from bad bots attempting to access user accounts. Ticketing industry one of the first industries targeted by bad bots, has the second highest percentage with 39.3%. Scalping bots, seat inventory checkers, and credential stuffing bots that access user accounts are most prevalent on these sites. Education A new industry sector included in this study, had 37.9% bad bot traffic. Bots are deployed by malicious operators looking for research papers, class availability, and to access user accounts.
Government with 29.9% of bad bots, is interested in protecting business registration listings from scraping bots, and in stopping election bots from interfering with voter registration accounts. Gambling and Gaming companies, with 25.9% of bad bot traffic, suffer from aggregators relentlessly scraping for ever-changing betting lines. Account takeovers are also a major problem because each account contains money or loyalty points that, once compromised, can easily be transferred to another user and emptied. Airlines have a challenging problem with 25.9% of their traffic comprising bad bots. Prices are scraped not only by direct competitors, but also by third-party players in the expansive travel ecosystem. Unauthorized online travel agencies (OTAs), competitors, price aggregators, and metasearch sites use sophisticated scraping bots to abuse the business logic of booking engines.
Ecommerce companies see a wide range of bad bot attacks. These include price scraping, content scraping, account takeovers, credit card fraud, and gift card abuse. Having one of the largest datasets, ecommerce has 18.0% of the bad bot traffic.
“Bot operators and bot defenders are playing an incessant game of cat and mouse, and techniques used today, such as mimicking mouse movements, are more human-like than ever before,” said Tiffany Olson Kleemann, CEO of Distil Networks.
Key Findings of the study:
- There has been increase in bad bots among all website traffic in 2018 which is 20.4%.Good Bots traffic amounted to 17.5% in 2018. Bad bots accounted for 1 in 5 website requests in 2018.
- 6% of bad bots spread from data centres, which are persistent, change their identities etc. They are also known as advanced persistent bots (APB). They can cycle through IP addresses and switch user agents, simple IP blacklisting is wholly ineffective.
- Nearly half (49.9%) of bad bots in browser which are popular in usage,Google Chrome tops the list. Mobile browsers, such as Safari Mobile, Android, and Opera increased to 13.9% from 10.4% last year. The remaining 8% reported themselves as other user agents, such as Googlebot and Bingbot.
- Amazon is the leading ISP for originating bad bot traffic. In 2018, 18.0% of bad bot traffic originated from it compared to 10.6% the previous year.
- United States remains the most bad bot originating country. Russia and Ukraine combined accounted for 48.2% of country-specific block requests. India is now the third most-blocked country at 15.2%.
- The human users is up in comparison with bots for the first time since 2016, to 62% of all internet traffic. The goal is to attract real humans to website, these numbers show that the bot problem is still significant.
Recommendation for Detecting Bad Bot Activity
- Block or CAPTCHA Outdated User Agents/Browsers The default configurations for many tools and scripts contain user-agent string lists that are largely outdated. The risk in blocking outdated user agents/browsers is very low; most modern browsers force auto-updates on users, making it more difficult to surf the web using an outdated version.
- Block Known Hosting Providers and Proxy Services Even if the most advanced attackers move to other, more-difficult-to-block networks, many less sophisticated perpetrators use easily accessible hosting and proxy services. Disallowing access from these sources might discourage attackers from coming after your site, API, and mobile apps.
- Block All Access Points Be sure to protect exposed APIs and mobile apps – not just your website – and share blocking information between systems wherever possible.
- Carefully Evaluate Traffic Sources Monitor traffic sources carefully. Do any have high bounce rates? Do you see lower conversion rates from certain traffic sources? They can be signs of bot traffic.
- Investigate Traffic Spikes Traffic spikes appear to be a great win for your business. But can you find a clear, specific source for the spike? One that is unexplained can be a sign of bad bot activity.
- Monitor for Failed Login Attempts Define your failed login attempt baseline, then monitor for anomalies or spikes. Set up alerts so you’re automatically notified if any occur. Advanced “low and slow” attacks don’t trigger user or session-level alerts, so be sure to set global thresholds.
- Monitor Increases in Failed Validation of Gift Card Numbers An increase in failures, or even traffic, to gift card validation pages can be a signal that bots such as GiftGhostBot are attempting to steal gift card balances.
- Pay Close Attention to Public Data Breaches Newly stolen credentials are more likely to still be active. When large breaches occur anywhere, expect bad bots to run those credentials against your site with increased frequency.
- The bot problem is an arms race. Bad actors are working hard every day to attack websites across the globe. The tools used constantly evolve, traffic patterns and sources shift, and advanced bots can even mimic human behavior. Hackers using bots to target website are distributed around the world, and their incentives are quite high.
(Image Courtesy: WWW.threatpost.com)