A colleague of mine from McAfee Avert QA and I have just returned from a summit in Bilbao, Spain where more then 40 experts gathered together for almost two full days. Security researchers, QA people from many AV companies, independent AV testing bodies and magazine reviewers were present. The purpose of the meeting was to form a non-profit organization that would work towards improving testing standards for anti-malware products. We want to give help to everybody who is eager to be involved in the area of testing anti-malware security solutions. Helping computer users is what we do on a 24×7x365 basis and we very much want to promote quality independent testing because bad tests mislead, confuse, and frustrate everybody.

The necessity to create such an organization rose due to occasionally seeing AV reviews that compared apples and oranges or, sometimes, not even saying what kind of fruit were compared! It would be beneficial for everybody if there were some minimal requirements for the tests. For example, an ability to contact the reviewer is very important and so is publishing the testing methodology. If there were many competent independent testing bodies it would be ideal but, unfortunately, contemporary anti-malware tests require big well-equipped research labs and it takes literally years to set them up. Neither the governments nor academic institutions have had much success on this front so far.

The delegates agreed on the name of this organization. Many suggestions were considered – such as “The Bilbao group”, “CATS” (Computer Anti-Malware Testing Standards) and “iTOSS” (International Testing of Security Software). In the end we all settled on a rather simple and straightforward name. The formal press release will follow soon where the name and the participants will be announced.

We all also worked together on a charter of this newly formed body and it was mutually agreed upon. I would expect a Web site to be set up soon and the charter published there.

We set up temporary committees - they will be in charge of organizing future meetings, establishing new contacts, and drafting the standards. The plan of action is to meet several times a year and seek agreement on the ways to objectively compare anti-malware security solutions, taking the output of the working committees as working drafts.

The idea is to have the organization open for the anti-malware companies, the academic institutions, testing bodies, magazine reviewers and everybody, who would wish to participate in improving the standards of testing for security software.

I could hear objections along the line - “How can we trust AV industry to set standards of how they themselves should be tested? Can say, car manufacturers be allowed to set their own testing standards?” I would reply that, firstly, the security software makers are vitally interested in the existence of comparative tests - they show us our strengths and weaknesses and only then can we better decide where to invest our resources in order to improve protection and increase the number of users. Secondly, the spectrum of security software is getting wider all the time (heuristics, generics, sandboxing, behavioral protection, herd-intelligence, HIPS, NIPS, etc. - new technologies are being constantly added) and traditional focus of testing on scanning piles of files (sometimes very old piles) is stifling innovation. Improvements in AV testing are urgently needed and involving the developers of the new technologies in the discussions sooner rather then later should benefit everybody. And finally, independent testers are already represented in the new organization and it will be them actually running the tests – nobody can force them to do what they disagree with.

It was a pleasure to work together with so many talented people and I really do hope that our joint efforts would not only improve testing standards but would also stimulate innovation in testing methodologies and in the security technology.

Avert Labs will keep you updated on the developments.