We’ve been receiving inquiries about a recent anti-virus comparative test that was performed at LinuxWorld by a start-up network gateway vendor named Untangle. Their goal was to prove that open-source anti-virus solutions (in this case ClamAV) were just as effective, if not better than commercial anti-virus products. It seems that they were highly motivated to prove this because evidently they use ClamAV in their gateway product (more on this conflict of interest later).

As with any comparative test involving McAfee, we analyzed the results and testing methodology used. Here is a basic breakdown of the testing that was performed by this vendor (their presentation can be found here):

  • 10 anti-virus vendors were tested (ClamAV, FProt, Fortinet, Global Hauri, Kaspersky, McAfee, SonicWall, Sophos, Symantec, Watchgaurd)
  • 35 samples were used (6 EICAR samples, 12 from Untangle, and 17 user-submitted samples)
  • It appears they performed an on-demand scan of the sample set.

Before delving into the many problematic facets of this test, if you would like a quick primer on comparative tests, please read “Comparing the Comparatives”. We recently discussed “(Mis)interpreting Reviews” here. For an introduction into false statistical interpretation go here.

Now let’s take a look at the flaws in the methodology used in this test:

  1. Blatantly False Results - We ran our own scan on the exact same files and our results showed we detected everything that was not a password-protected zip or 0-byte file. How many other AV vendors’ results were wrong? I’m sure I’ll soon find out from my counterparts.
  2. Small Sample Size – One of the first rules of Statistical Method is using a large enough sample size for your test in order to accurately represent the entire population. In this case, they used only 35 samples of the hundreds of thousands malware samples in the wild today.
  3. Biased Samples - The fact that 12 of the 35 samples came from the CTO of Untangle’s mailbox negates the fact that they used a random sampling that accurately represents the true population of malware samples.
  4. Comparing the Wrong Products – The test compares 5 Linux, 2 Windows, and 3 Gateway products. This is like comparing apples with oranges with kumquats.
  5. Misconfigurations of Vendor’s Products – The CTO admits in his blog that “In fact, one audience participant significantly improved one vendor’s performance, Sophos, by pointing out that I needed to add a command-line option. Others pointed out mistakes I made recording results.” What other misuses occurred during this test?
  6. Conflict of Interest – One of the first rules of comparative tests is that it needs to be performed by an arbitrary third party with no vested interests in the outcome of the results. The fact that this test was performed by Untangle who develops, markets, and sells an anti-virus solution with their gateway product is a blatant example of a conflict of interest.
  7. Improper Handling & Distribution of Viruses - By offering a link to these live viruses on their company’s public website, they are in violation of the Computer Fraud & Abuse Act which prohibits the distribution of computer viruses because it is endangering public safety. There is a reason why only trained security professionals should handle computer viruses. I have already contacted them to remove the links. Note: If the link is not removed, SiteAdvisor’s webcrawlers will automatically identify the company’s site as red due to the malicious link.

Needless to say, the methodology used in this test was riddled with flaws. So if anyone thinks this test proves anything, we have $14 million in a Nigerian bank account waiting to be transferred to a US account. ;-)

* Rather than adding to the confusion of proper anti-virus comparative testing, those in the industry are working towards a better solution. McAfee Avert Labs recently participated in an Anti-Virus Testing Workshop in order to define more robust testing methodologies.