It was very encouraging to see that more than 40 people came to Budapest, Hungary, to discuss and agree on new industry standards as part of the effort undertaken by the Anti-Malware Standards Organization (www.amtso.org.) The awesome historic surroundings set the mood for our discussions.

Seeing such a great turnout in the current economic climate shows how much AMTSO members care about raising the standards of testing anti-malware products. Especially considering the recent rise in the number of rogue security products (such as the now infamous “Anti-virus XP 2009″), it is clear that we need transparent and fair testing more than ever.

AMTSO members finalized and adopted several new documents to the current portfolio. (Have a look at the collection of documents here: www.amtso.org/documents.html.)

But I would like to draw your attention to two papers that, in my opinion, represent very significant steps for the security industry as a whole.

• The first one is “AMTSO Analysis of Reviews Process,” and it presents the process of analyzing reviews. The creation of such a process paves the way to highlight great reviews and/or to expose substandard tests in public. (AMTSO promises to publish all the analyses they undertake.) I really hope that this process, designed to be transparent and fair, will improve the quality of testing and benefit both the developers and consumers of anti-malware technology. If you have doubts that this process is going to be unbiased I will remind you that AMTSO members work for competing security companies, and there would not be a snowball’s chance in hell to agree on the process if it were not designed to be fair. The next step is to put the “AMTSO Analysis of Reviews Process” into practice. I cannot wait to see how it will go.
• “AMTSO Best Practices for Testing In-the-Cloud Security Products” is the second very important milestone. Some anti-virus products started using “cloud” technologies (such as McAfee’s Artemis, which was launched in the beginning of 2008) and the number of cloud-based products is growing; so there is a need to address the fundamental problems associated with testing solutions that are not under the control of the tester. (That is, part of the product is not “in the hands” of the tester; moreover, it can change at any moment in time.) I think it is amazing that representatives of so many competing security companies agreed on fair and scientific principles of how to test cloud-based products. To be honest, when we started this effort we were rather sceptical about finding a sensible way to address all the problems that testers face when evaluating such technologies. The adoption of AMTSO best practices for testing in-the-cloud products means that our brainstorming was successful. I am very pleased to see the agreed results adopted and published. Thanks for that effort go to all the security researchers who contributed to the document and all AMTSO members who voted for it.

Guys from AV-Comparatives have just posted a new scanners’ review on their Web site – http://www.av-comparatives.org/comparativesreviews/main-tests:

AV-Comparatives is a non-profit independent test organization based in Austria and they have been running comparative tests for many years but this last one in February 2009 was different for two reasons:

• Firstly, the criteria for getting awards were more stringent than ever. For example, malware older than 9 months did not count towards the awards at all – so only products detecting contemporary trojans and viruses would have done well. Perhaps due to these strict criteria the previous comparative report found 7 products worthy of  “Advanced+” rating  while now it is only 4 products.
• Secondly, for the first time a product which runs with “in-the-cloud” technology enabled by default (McAfee VirusScan 2009 consumer product with Artemis Technology) was tested side-by-side with traditional scanners.

Everybody at AvertLabs are excited about the great prospects that Artemis cloud technology is opening for security and very happy to see that it did well in the test.  We are working hard on enhancing protection based on the cloud-based approach and the only cloud on the horizon (pardon the pun!) is that the adoption of Artemis is going really well for home users (we have more than 30 million customers already!) but in businesses and corporate sector the growth was less pronounced. We urge administrators to look at adopting Artemis quicker and hope that mentioned comparative test would eliminate their concerns – the technology is now mature and stable. But most importantly – it adds to the protection.

P.S. Note that www.AV-Comparatives.org Web site got revamped and has a new look (I personally like it).

– Update March 24 –
For corporate administrators looking for more information on enabling Artemis, see:
How to enable McAfee Artemis Technology in VirusScan Enterprise (KB53732)

Igor Muttik just had a parcel arrive whilst I was nearby. McAfee has just won the Lowest False Alarm Rate award from AV Comparatives for VirusScan.

….I didn’t stay for the speech

For those who do not know, false alarms are caused where an anti-virus product detects a clean file as infected and is something all AV companies try hard to avoid. Recognition that we’ve got the lowest false alarm rate on test is awesome.

Today marks another day of momentous change for McAfee’s research teams.

I just spent two days with my new colleagues from Secure Computing and some of my team members from McAfee Avert Labs. It was two grueling days of discussion and education as we both came up to speed on our research methodologies and technologies. Let me say that I am truly excited to be working with Dmitri Alperovitch, Sven Krasser, and Paula Greve, who head up the research group there. These are sharp and capable research leaders who have done amazing things. TrustedSource is a great technology and has so many applications that McAfee can leverage. Once our new Artemis technology begins to leverage TrustedSource capabilities McAfee will become the undisputed leader in security intelligence in the Internet “cloud.” Together we will see millions of spam messages, evaluate thousands of web sites, and see thousands of new pieces of malware–all in the span of 24 hours. We now have the ability to see and react to the threat landscape better than ever before. This is something that every McAfee product, technology, appliance, and SAAS (software as a service) solution will come to leverage, differentiating themselves from the competition even more.

At first we thought we would have overlapping technologies, but this is definitely not the case. In combating spam, web, and malware we have approached these threats from very different directions; thus we find our technologies very complementary. In the case of anti-spam protection, for example, we have two technologies that provide better than 99% detection using very different methodologies and approaches. Once combined, we will have the most robust solution on the market. The same holds true for the SmartFilter and SiteAdvisor technologies, as well as our malware solutions.

Today we have very good security intelligence. Tomorrow, with a bit of nurturing, we will have great security intelligence.

We welcome Secure Computing to the McAfee research family.

Jeff Green
Senior Vice President
McAfee Avert labs

Artemis was a Greek Goddess of hunt, forests and hills (http://en.wikipedia.org/wiki/Artemis). It is also a name for McAfee’s new “always-on,” real-time protection technology (http://www.mcafee.com/artemis) which is now available, without charge, in many of the latest McAfee products.

The legendary home of Greek Gods is the mount Olympus – the highest mountain in Greece.

Well, today Artemis reached another new level – I am very glad to let you know that VirusTotal (a free service run by Spanish company Hispasec through http://www.virustotal.com) have just added Artemis scanning to their portal. So, as of today, instead of just one command-line scanner, the basic detection technology from McAfee Avert Labs, we will be represented by two scanners. They are labeled “McAfee” and “McAfee+Artemis”. Here is how it looks in the VirusTotal portal:

Let us have a close look at this malware sample. We first saw it this morning at 06:35 UTC. Artemis recorded 32 instances of this file before it was analyzed and detection was added to Artemis. Since that moment and until now (~8 hours after first sighting) we saw 586 more samples. These samples, of course, were all successfully detected and blocked. The map shows geographical distribution of the Artemis clients that sent a fingerprint of this malware to the Avert servers.

White dots represent initial submissions (32 of them). Red dots – the blocked ones (586 of them).

Thanks to our colleagues at Hispasec for adding our Artemis technology to their site. This provides a great service to the public and to our Avert Labs researchers!

It is very exciting to see that finally AMTSO published two documents on its Website (http://www.amtso.org/documents/cat_view/13-amtso-principles-and-guidelines.html):

• AMTSO Fundamental Principles of Testing
• AMTSO Best Practices for Dynamic Testing

These documents were posted by AMTSO for public comments as RFC versions back in August 2008. Most of the comments from http://blog.amtso.org actually got reflected in the final text so AMTSO did incorporate many different opinions in its standards, which is a good thing!

The most important thing about these standards is that there is now hope that the quality of anti-malware reviews will improve over time because vendors and testers can work more closely together for the benefit of all computer users.

Here is what Jeff Green, Senior Vice President of McAfee Avert Labs said about this event: “While there have been many great security software reviews in the past, many poor reviews reviews have confused or misled people. We are glad to see that Anti-Malware Testing Standards Organization has taken this problem by the horns and formalized the principles of fair testing. This is a significant milestone that should skew the balance towards fair and scientific testing, providing users with a true viewpoint on the security protection vendors provide.”

Let’s hope that there will be more standards from AMTSO and they would look as good as those just published.

A recent ZDnet blog discusses a large number of vulnerabilities German research team N.Runs says it found in antimalware products from nearly every vendor. The ZDNet posting includes scary graphs to frighten users of security products. We researched the N.Runs claims by analyzing the raw data and found their claims to be somewhat exaggerated. We will discuss our findings (and make available our source data) in the attached document. We have also provided our source data for anyone who wishes to examine it.

First, N.Runs has indeed found many vulnerabilities and they deserve credit for that. We have worked with the N.Runs team in the past and have found them to be very responsible and intelligent researchers.  We don’t want to attack the legitimacy of the vulnerabilities they found, but do call into question the conclusions drawn on what this means to the state of security.

Due to the amount of information required to examine the ZDNet and N.Runs claims in depth, we have felt it better to provide the entire blog entry in a PDF format. Please see the attached document for much more detail on the subject.

Full Article (in PDF Format)

Source Data (in excel format)

Following the news from my colleague Dr. Igor Muttik about his recent trip to Bilbao, Spain, to participate in the Anti-Virus Testing Workshop, AV-Test.org just released the results of their latest comparative test. It was picked up by many media outlets:

1. PC Magazine (USA)
2. Dark Reading (USA)
3. PC Welt (Germany)
4. CHIP (Germany)
5. Security.nl (The Netherlands)

Unlike in many previous reviews, AV-Test.org ran various types of tests, and McAfee scored well in most of them:

• We are pleased that we made the most progress of any vendor from AV-Test.org’s last test, which was published by c’t magazine (Germany) earlier this month. Our detection-rate improvement was +7.3%.
• We are proud that we did not detect any false positives. (We are one of only three vendors that can make that claim.)
• We received the second-best rating in the Rootkits test.

Signature-Based Tests are usually an on-demand scan (ODS) by anti-malware products on a computer system against a set of known malware. We have discussed the challenges in making this test fair in the past.

Proactive Tests are similar to signature-based tests, except that they attempt to measure how well an anti-malware product can detect samples that it has never seen before–by taking an old DAT version and scanning with malware that was discovered after the DAT release date. This test often gives a sense of how well an anti-malware vendor does in writing generic, heuristic, or behavioral signatures. The caveat with this is that if a product ventures too far into this realm, the likelihood of false-positives increases.

False-Positive Tests are also an ODS test, except with a sample set of clean files instead of malicious files. False positives are the bane of the anti-malware industry as they could have far worst collateral damage than a false-negative (missed detection) depending on the severity. Because of our large customer base, we take this metric very seriously and have an internal zero-tolerance policy.

Rootkits Tests are one of the most complex and time-consuming tests that a tester can run, and are similar to the behavioral tests described above. However, these require even more intimate knowledge of both the target operating system and known rootkit techniques to accurately judge whether an anti-malware product was able to properly remediate the rootkit infection.

Response Times tests attempt to determine how quickly an anti-malware vendor responds to a new threat with their definition updates and heuristic detections.

Individually, each of these tests gives us a way to gauge one of the many facets of measuring the value of an anti-malware product. However, when grouped together, they can give a holistic picture of how well we balance the many criteria by which we are judged.