A Review of Popular Bot Checkers

Share:

We are often asked about bot checkers. Which ones are better? Are they worth using at all? On one hand, we always recommend that people learn how to identify bots themselves (and here’s a handy bot tutorial for just that purpose). On the other hand, bot checkers can be a helpful back-up tool if they work. And that is the the vital question: do they work? So, we reviewed several bot checkers commonly used by Twitter users–by testing them.

Please note that this is an authentic review. We have no financial or personal ties to any bot checker.

Which Bot Checkers?

For the test, we selected bot checkers that seemed to be most often recommended and used by Twitter users. The list is not meant to be exhaustive.

1 > Botcheck.me

2 > Bot Sentinel

3 > Botometer

How We Tested the Bot Checkers

We randomly selected 10 fake accounts from three bot groups, for a total of 30 test accounts. Then we ran all 30 fake accounts through each of the bot checkers. Since our goal was to test the bot checkers (not trick them), we selected low-level bot groups that are some of the easiest to identify visually: model bots, Russian bots with English names, and lottery bots.

The Results

> Please see below for full test documentation.

> We counted any rating outside of the “safe” or “normal” zone as a detection.

Bot Checker Test: Model Bots

These are the 10 model bots we tested on the bot checkers.

The Model Bot Test Results

Bot Checker Test: Russian Bots with English Names

These are the Russian accounts with English names that we tested on the bot checkers.

The Russian Bot Test Results

Bot-Checker Test: Lottery Bots

These are the lottery bots we used to test the bot checkers.

The Lottery Bot Test Results

One More Bot Checker Test – A Troll Test

Given how poorly two of the bot checkers performed with low-level bots, we were curious to see if their algorithms focused more specifically on trolls rather than all fake accounts. So we ran one more test using the Elsa Darner troll-bots as the test accounts.

These are the Elsa Darner troll-bots we used to test the bot checkers.

The Elsa Darner Troll-Bot Test Results

*Error is a non-result. Thus, a bot isn’t detected.

As you can see, Botcheck.me didn’t improve with the troll-bot test. On the other hand, Bot Sentinel performed equally as well as Botometer in the troll-bot test. This result implies that Bot Sentinel’s algorithms are geared toward assessing language and tweet technique. Furthering the implication is the fact that Bot Sentinel’s three “no” conclusions were Elsa Darner troll-bots that haven’t tweeted yet. Interestingly, the “errors” from Botcheck.me and Botometer were on the same “no-tweet” accounts.

Bot Checkers – Additional Features

This wouldn’t be a proper review of bot checkers without looking at some of the other features they offer.

Botcheck.me offers a Chrome plugin. Probably the best item it offers, however, is an instructive description of propaganda bots. The description also details some of the difficulties inherent in algorithm-based bot checking.

Bot Sentinel publishes a variety of statistics and has a browser extension. One nifty feature is its “Trending Topics” where it tracks trending hashtags (so often generated and amplified by fake accounts).

Botometer has an enormously useful feature–it allows users to bot-check their followers in batches of 20. It also has impressive credentials: “Botometer is a collaboration between Indiana University Network Science Institute (IUNI) and the Center for Complex Networks and Systems Research (CNetS).”

A Note About Bot-Checker Algorithms

Existing bot checkers’ algorithms rely on Twitter’s API data (i.e., raw data) to determine the veracity of an account. But the programmers’ algorithm inputs are unknowns to us. The fact that the algorithm is computerized doesn’t change the fact that a human has programmed it. And, as you can see from the results in our review of bot checkers, some algorithms are better able to assess data than others. This is why we recommend that social media users have at least a working knowledge of what fake accounts look like.

Finally, a word of caution

If you choose to use one of the bot checkers, keep in mind that our review of the bot checkers consisted of testing low-level bots and obvious troll-bots. They are easy to identify visually and they often have identifiable tweeting patterns in content and technique. The more advanced fakes are much more sophisticated. And the more sophisticated fakes are far more difficult to detect (especially for algorithms).