Do IPs have personality?

 

There's been quite a lot on the news recently about AI agents (or robots) attacking websites. Some years ago our office received about 1.5 million attacks over a weekend. For context, we were a very small NGO and we repelled the attackers. A banking services company across the road wasn't so lucky and it cost them tens of thousands of Euros. Another occasion some years back one of our servers was attacked, enough that it started to overheat!

We don't enjoy attacks or indeed over zealous crawlers, so we developed a system called Pynblok. Pynblok is a South African work mean 'pain block'. The aim is to block the pain of attacks and defend our servers. And, no, we're not some big corporation but a very small non-profit. Nevertheless we still get frequent attacks and scans, sometimes hundreds, sometimes thousands per day. 

One thing to remember though is no system is impenetrable. We know from recent history even the Department of Defence in the USA was able to be hacked. A really aggressive hacker will get in. It's where backups are important!

We talk about IP addresses like the name of someone or something that attacks you or crawls your website for information. It's less of a name and more like your passport number or your ID card number. But behind that is something or someone with personality. We do have to be careful about not anthropolising a computer. However having run the system now for about two months we are beginning to recognise certain behaviour characteristics. Hostile actors like 'the hammer', 'the scanner', 'the sniper' or 'the ghost'. Crawler personalities like 'the harvester', 'the scraper', 'the librarian' or 'the explorer'.

As we are collecting data and at the same time blocking attackers, we are noticing dimensions of repeated behaviour:

Persistence

Keeps returning over days/weeks

Curiosity

Breadth of paths and technologies probed

Focus

Repeatedly attacks one particular target

Aggression

Volume and frequency of requests

Opportunism

Rapidly changes targets when something fails

Stealth

Slow, dispersed probing vs huge bursts

Deception

Frequent changes in User-Agent/claimed identity

Methodical

Systematic enumeration of related paths

Adaptability

Changes behaviour after responses

Specialisation

Concentrates on one vulnerability/technology

Breadth

Attempts many unrelated attack classes

Recurrence

Comes back after periods of inactivity

The first thing we realised was that the characteristics were multidimensional, which also meant it was all too easy to get caught in the trap of complexity.  Let's give you an example. Over a 10 second burst one attacker identified by it's IP made 221 attacks; 22 attacks per second. Here's a few of them:

Now this was the same IP, so it's the same attacker most likely, and in this case all of the attacks were looking for password credentials which many people store in what are called 'environmental variables'. One would be very unwise to do so but many people apparently do! The interesting thing about the list to is the 'User-Agent'. Notice how each different attack is using a different apparent 'User-Agent'. Of all the 221 attacks in that burst almost all of them are different. Many pretending to be legitimate bots like Google or Bytespider or Perplexity. 

So for that IP we might say

Persistence:     very high
Breadth:            very high
Deception:         very high
Curiosity:           very high
Specialisation:   low
Aggression:       high
Adaptability:      unknown

So we might have called that the The Opportunistic Reconnaissance Bot. Another IP might emerge as The Persistent Specialist or the The Noisy Brute. But what we wanted with Pynblok was not twenty five attributes which would have been difficult to remember or understand but to collapse the presentation onto two axes: Horizontally we made that narrow to broad and vertically low activity to high activity. 

Looking at the horizontal axis we go from focus on the left to opportunism on the right. Focused IPs repeatedly attack using the same technology, consistent target using specialised exploits or probes. Opportunistic IPs try everything, look for passwords, look for Wordpress exploits, PHP exploits etc etc then follows whatever looks potentially exploitable.

Looking at the vertical axis we go from noisy to stealthy. Sometimes the stealthy is not a single IP but a cluster of IPs working in concert to try to get through what otherwise might be blocked. Noisy IPs run huge bursts with lots of requests, are obviously scanning and have short attack campaigns. When they try against version 2 of Pynblok we try to cut them off within 3 seconds.

Stealthy attacks work on low frequency often with long intervals between attacks, which can be minutes or hours between and have carefully distributed requests which are harder to notice.

Hence the example that I cited earlier would be categorised as a Scanner: High-volume, semi broad-spectrum reconnaissance. Like human personality tests the four quadrants aren't a measure of badness. Unlike human tests, in our measure they are all bad. They are all attackers! What they help us do is describe how the IP behaves. 

It also helps us evaluate how we respond to that IP personality. That part I won't describe as it would facilitate attackers to optimise their attacks against a Pynblok protected server. However, the Pynblok system doesn't use external AI nor is it externally accessible from the web. 

Crawlers (bots from AI engines or search engines) are different. One wants search engines to crawl one's site so that people can find it. AI is potentially different, some people want to block them, some people encourage them. However, very aggressive crawlers can put unreasonable load on your server. There are also loads of fake crawlers pretending to be Google or Bing or whatever. In Pynblok we added processing to identify real vs fake crawlers.

So in the two axes for crawlers we had similarities with attackers: horizontally we had focused vs exploratory. The focused crawler concentrates on a small set of known URL types or pages, whereas the exploratory, discovers broadly and follows lots of different paths. 

The vertical axis is gentle vs aggressive. Gentle crawlers have a low request rate and respect normal site behaviour. Aggressive crawlers have a high request rate with lots of parallel requests and little regard for load. 

This table gives you a clue as to how variable the bots are at crawling sites. 

Obviously OpenAI's GPTBot is by a long way the most aggressive followed by Meta's Facebook AI crawler and Anthropic's ClaudeBot. Identifying and verifying bots is almost more art form than science!

Different sites will have very different patterns, this is purely an example from one server, not something to use as common to all sites on the Internet. Those figures are just for one server over a couple of weeks. 

So, in consequence we called those four personalities The Harvester, The Scraper, The Librarian and The Explorer.

The aim was to answer two questions: What type of attacker are you? What type of crawler are you?

Every IP has a personality. It's not just an address, it's behaviour. Finding out their behaviour helps us to know how to react to them. And that is what Pynblok is about... blocking the pain of attackers and unfriendly crawlers!