Computer users worldwide seek to identify the best, the most comprehensive and thorough anti-spyware, anti-trojan, anti-keyloggers and anti-virus scanners (collectively anti-malware) to defeat the abundance of the constantly increasing “malware” infecting their computers and affecting their use of the internet and their computer in general.
The various computer user group’s websites and online forums are filled with users’ requests seeking assistance and asking which product or products are best and also “why isn’t there adequate testing of “anti-malware” software to assist users in their selection of adequate solutions”. This is particularly directed at the Anti-spyware market.
As we have stated previously, malware testing is a “daunting task” that must be performed under conditions that are above reproach and that are completely transparent and readily reproducible by others to ensure the validity of the results.
The methodology of testing and some details of requirements of a protocol have appeared and have been discussed elsewhere. The question that is not addressed or considered by users is what are the requirements of the testing facility and competence of their staff, and very importantly “what are the incentives for malware testing?”
The criteria for the testing facility are readily identifiable and all would agree that they include an unbiased procedure and testing facility. Obviously vendors of anti/malware cannot serve in this capacity as they and the results would be subject to claims of bias. Thus, the first requisite is that the facility and those doing the testing cannot be associated with any vendor. It must be totally independent.
Secondly, the competency of the staff carrying out the testing becomes an issue. Although not insurmountable, it is important that consideration must be given to ensure the “intellectual honesty” of those implementing the testing procedure. Redundancy and proper supervision can be used to address these issues. In addition to competency of the staff, the adequacy of the physical facility itself requires consideration.
These issues are dwarfed by the second question which is “What Are The Incentives For Malware Testing?”
Exclusion of monetary return for testing seems to be self evident to ensure independence and to eliminate bias. A fee structure for testing of a vendor’s software and/or participation in a vendor association to underwrite funding likewise raises issues. Will only those participating in funding be part of the testing? How do we ensure complete independence?
The necessary fee structure could be prohibitive for smaller companies as maintenance of full time personnel and full time facilities would require substantive commitments that would not necessarily fulfill the requisite conditions of independence.
On line promotion of vendor products on the testing facilities web site raises issues of bias towards those vendors advertising on theses web sites. Recognizing that some testing facilities already utilize such a model does not satisfy the need for complete transparency and absence of any association between the facility and the vendors whose products are being tested. Even the appearance of impropriety suffices to create suspicion and doubt in vendors’ and users’ minds.
Thus, the issue of Testing Facilities to identify reliable and effective products is an extremely difficult problem with no readily identifiable solution. There must be more than an incentive, but a monetary structure to maintain and support such a facility. What is the incentive?
Currently, users are served by a faithful group of fellow users who give counsel and advice based upon their own experiences via forums. They are to be applauded for their dedication and willingness to assist unknown fellow computer users. Admittedly, such a system is less than perfect. Thus the various threads will continue to routinely ask the question: What is the best anti-malware software?
Even in the reported malware tests, we see that there is a continual reshuffling of the top products which confirms that “best is transient.” Best today does not ensure best tomorrow. Does this mean that the tests are meaningless? We invite your comments and opinions.