Spam Blocking Quality: Lightspeed Systems vs. Barracuda Networks

Rigorous testing against competitive products is the only way to know for certain if our product has superior quality. We choose to first test our spam blocking against the Barracuda Spam Firewall spam blocker because Barracuda is most frequently used or tested by customers evaluating our software.

The quality metrics for spam blocking are very clear. The product should block as many spam emails and virus infected emails as possible without blocking good emails. This quality metric ends up with three categories of errors:

  1. Missed Spam: spam emails that got through the spam blocker product
  2. Blocked Ham: good emails that were mistakenly blocked
  3. Missed Virus: virus-infected emails that should have been blocked, but were not

There is actually a fourth type of error, over-blocked virus, which is good email that is mistakenly identified as virus infected. Fortunately the over blocked virus error happens so rarely for either product that we decided not to track it at all.

Overview of Barracuda Spam Blocking

Barracuda Networks sells its solution as an appliance with an annual maintenance contract. It is very affordable, and in general it does a good job. The Barracuda software is a modified version of the open-source product "Spam Assassin," which has an excellent reputation.

Spam Assassin is a rule-based spam blocker with a Bayesian learning algorithm. Each of the rules has a "score." The email is judged to be spam or not based on the total score of all of the rules. The Barracuda version allows the administrator to change some of these scores.

In addition to rules-based scoring, there are also RBL blacklists and user-generated blacklists of IP addresses, domains, subjects, body text, and headers. This is very similar to Lightspeed's use of RBL blacklists and spam patterns.

Problematically, because Spam Assassin is a commonly used open-source product, it is often used by spammers themselves to see if their spam mail will get through spam blockers.

Differences between Lightspeed and Barracuda

The single largest difference between the two spam blockers is Lightspeed's extensive use of SQL databases. To identify spam, Lightspeed uses and frequently updates large, shared SQL tables of Internet domains, IP addresses, URLs, and spam patterns. In addition, Lightspeed stores into SQL tables email addresses and IP addresses of spammers who failed different spam tests so that future spam can be more effectively blocked.

Notably, some of Lightspeed's most effective methods of spam blocking have no equivalent in Barracuda. These methods include:

  • blocking mail that has embedded URLs of porn and gambling websites,
  • blocking mail from IP addresses of well known spammers,
  • blocking mail from proxy server IP addresses, and
  • sending challenge emails to unknown email addresses.

All of these spam-blocking methods depend on using large and accurate SQL database tables.

Testing Methodology

We decided the best real-world test of the spam blocking of both Lightspeed and Barracuda was to use the actual email that comes into Lightspeed every day. This email is similar to email of many other organizations with one exception – at Lightspeed we get many more viruses emailed to us than a typical company would because of our ongoing antivirus testing.

The testing methodology is as follows:

  • as the daily email flows in, the Lightspeed spam blocker records all the mail on disk—as it blocks spam;
  • recorded email is relayed back later through the Barracuda Spam Firewall;
  • blocked good mail (ham), missed spam, and missed emails with a virus are discovered as below;
  • the results from both Lightspeed and Barracuda are stored into a SQL database; and
  • so that any part of the testing can be recreated, each email is also stored on disk to make an archive.

The only difference between the emails run through Lightspeed and run through Barracuda is the IP address of the mail server. Emails run through Lightspeed have the source IP address of the originating mail server. The same emails run later through Barracuda have the source IP address of the relay server. This difference should not have affected the results very much.

After the emails have been run through each spam blocker, we then find any mistakes in categorizing the email as ham, spam, or virus. The easiest way to find mistakes is to start with Lightspeed's results and track any changes. We set up an automated process to track Lightspeed's "blocked ham" and "missed virus" emails that is very accurate.

Any blocked email that a Lightspeed employee accesses as good email is automatically tracked as a "Blocked Ham" for Lightspeed. This statistic is accurate because people are interested in not missing any important email.

"Missed Viruses" are found by scanning every email by up to 10 different virus scanners to detect any mistakes.

Lightspeed's "Missed Spam "emails require each Lightspeed employee to copy any missed spam they receive to a global email folder. Some employees are better at doing this than others, so the "Missed Spam" statistic for Lightspeed is not as accurate as the "Blocked Ham" and "Missed Virus" statistics. Combining the Lightspeed spam blocker original results with the three error statistics, the actual classification of each email was calculated and stored into a SQL database. Barracuda's three quality metrics were then calculated using the actual email classification.

Test Results

Lightspeed is consistently better at all three quality metrics of spam blocking.

  • The "Blocked Ham" statistics show that Barracuda routinely over blocks good emails—10 times as often as Lightspeed.
  • The "Missed Virus" statistic shows that Lightspeed rarely misses a virus-infected email and that Barracuda usually misses a few viruses every day.
  • And finally, the "Missed Spam" statistic shows that Barracuda misses quite a lot of spam email, although this statistic is not as accurate as the others because of the limitations of our testing methodology.