What Are The Incentives For Malware/Spyware Testing?

Computer users worldwide seek to identify the best, the most comprehensive and thorough anti-spyware, anti-trojan, anti-keyloggers and anti-virus scanners (collectively anti-malware) to defeat the abundance of the constantly increasing “malware” infecting their computers and affecting their use of the internet and their computer in general.

The various computer user group’s websites and online forums are filled with users’ requests seeking assistance and asking which product or products are best and also “why isn’t there adequate testing of “anti-malware” software to assist users in their selection of adequate solutions”. This is particularly directed at the Anti-spyware market.

As we have stated previously, malware testing is a “daunting task” that must be performed under conditions that are above reproach and that are completely transparent and readily reproducible by others to ensure the validity of the results.

The methodology of testing and some details of requirements of a protocol have appeared and have been discussed elsewhere. The question that is not addressed or considered by users is what are the requirements of the testing facility and competence of their staff, and very importantly “what are the incentives for malware testing?”

The criteria for the testing facility are readily identifiable and all would agree that they include an unbiased procedure and testing facility. Obviously vendors of anti/malware cannot serve in this capacity as they and the results would be subject to claims of bias. Thus, the first requisite is that the facility and those doing the testing cannot be associated with any vendor. It must be totally independent.

Secondly, the competency of the staff carrying out the testing becomes an issue. Although not insurmountable, it is important that consideration must be given to ensure the “intellectual honesty” of those implementing the testing procedure. Redundancy and proper supervision can be used to address these issues. In addition to competency of the staff, the adequacy of the physical facility itself requires consideration.

These issues are dwarfed by the second question which is “What Are The Incentives For Malware Testing?”

Exclusion of monetary return for testing seems to be self evident to ensure independence and to eliminate bias. A fee structure for testing of a vendor’s software and/or participation in a vendor association to underwrite funding likewise raises issues. Will only those participating in funding be part of the testing? How do we ensure complete independence?

The necessary fee structure could be prohibitive for smaller companies as maintenance of full time personnel and full time facilities would require substantive commitments that would not necessarily fulfill the requisite conditions of independence.

On line promotion of vendor products on the testing facilities web site raises issues of bias towards those vendors advertising on theses web sites. Recognizing that some testing facilities already utilize such a model does not satisfy the need for complete transparency and absence of any association between the facility and the vendors whose products are being tested. Even the appearance of impropriety suffices to create suspicion and doubt in vendors’ and users’ minds.

Thus, the issue of Testing Facilities to identify reliable and effective products is an extremely difficult problem with no readily identifiable solution. There must be more than an incentive, but a monetary structure to maintain and support such a facility. What is the incentive?

Currently, users are served by a faithful group of fellow users who give counsel and advice based upon their own experiences via forums. They are to be applauded for their dedication and willingness to assist unknown fellow computer users. Admittedly, such a system is less than perfect. Thus the various threads will continue to routinely ask the question: What is the best anti-malware software?

Even in the reported malware tests, we see that there is a continual reshuffling of the top products which confirms that “best is transient.” Best today does not ensure best tomorrow. Does this mean that the tests are meaningless? We invite your comments and opinions.

Are Suites really “Sweet”?

The proliferation of “Software Suites” to deal with the ever increasing “malware” problems raises serious issues for users of these products.

As the market for anti-virus software has stabilized, the major anti-virus software companies are entering other sections of the malware market in order to expand their user base and also to secure their current customers.

Depending upon the vendor, these suites now encompass not only the traditional anti-virus area, but have also expanded into the Firewalls, Anti-Spyware, Anti-Trojan, Anti-Keylogger, Anti-Phishing And Anti-Spam areas. This shift in emphasis and expansion is one result of the stabilization if not the decline of the anti-virus market and the movement of those writing malicious software into the lucrative adware/spyware area. In addition, as the distinction between anti-virus and anti-spyware products has blurred, considerable overlap of software capabilities has developed.

At first appraisal by computer users, the evolution of suites appears as a desirable solution to a continually evolving problem. The question arises whether these suites are really serving the end user’s needs?

In order to ensure adequate protection and to protect their customer base, the software suites appear to have adopted a philosophy that there is no need to co-exist with other competing programs. Thus users attempting to maintain dual means of protection are thwarted both by slowdowns of their computer system as well as by direct conflicts with users’ competing products. The use of resources by the suites is heavy even without competing components. The adage “jack of all trades, but master of none” seems appropriate here because the results of numerous tests clearly demonstrate that no single product can be relied upon to provide the required protection because rapid changes characteristic of the spyware area are in sharp contrast to the previous viral area. Thus users must seek and use multiple products to provide them the required margin of safety.

Co-existing with suites is a challenge because of the aforementioned problems, not the least of which is the heavy use of system resources by the “suite” products. The result of this dilemma is that users must rely upon products which are “light” on system resources. We have realized this requirement, as no single product can catch every infection on a given day, and have specifically designed our product, SUPERAntiSpyware, to be light on system resources and co-exist with existing security solutions. I am certain other companies will follow suit. Users will ultimately dictate the verdict on whether “suites are sweet.”.

The Importance of Testing Methodology

In today’s oversaturated market of anti-spyware/malware/adware applications, it is becoming increasingly difficult for users to determine which applications will perform best for their specific needs. Thus, they look for standardized and legitimate “comparative tests” of these applications.

Testing anti-spyware applications is not an easy task. It is imperative that those who are going to undertake the task of testing need to have the skills to perform the tests competently and to test the products in real-world situations. Otherwise they are not performing a service to users. Users also need to examine the credibility of the party testing the applications and not simply “look at the numbers.” Currently, most tests are not comparing “apples to apples” because every anti-spyware application uses different methods of reporting the “numbers” of infections detected and removed.

There are standardized and widely accepted elements of any investigative report. These include an Introduction, Materials and Methods or Procedures, Results, Discussion and usually, but not always, Conclusions.

The most critical elements of an adequate report or investigation are to provide the reader with the Materials and Methods used which would allow others to duplicate the experiment or investigation to determine the validity of the results; that is, are the results reproducible in the hands of others using the same procedures (Materials and Methods). Thus, the methodology used in any investigation must be of sufficient detail to allow any interested parties the opportunity to independently validate the results.

In using non-standardized methods, it is critical to provide detailed procedure in order to ensure validation by allowing reproduction of the results by others.

When or if it is determined that the methodology is itself flawed or contains documented errors, this invalidates the results and casts serious concerns on other components or elements of the entire methodology and on the results.

The level of detail cannot be assumed or taken on faith. Therefore, it is of the utmost importance to provide a level of detail which removes any ambiguity as to how something was done and to provide and detail the safeguards used to ensure that the procedures were indeed followed.

In examining the testing methodology used by recent tests by Malware-Test.com, it is unclear whether their own procedures were followed when flaws or errors are discovered as detailed elsewhere. This casts doubt on how other elements were carried out. Furthermore, it is one thing to say how you are carrying out the testing and another to actually follow the protocol. Thus, alleged transparency by providing the purported methodology cannot in and of itself be accepted on faith and can be extremely misleading particularly in view of any demonstrated inadequacies.

Malware testing is certainly a daunting task and adequate documentation of methodology is the single most important element in validation of the results. When testing is performed by individuals one can accept and or excuse minor inadequacies. However, when the results are performed by alleged experts, in testing facilities which exist for testing purposes, they must be held to the highest standards.

Are cookies really spyware?

This subject has been the debate of many newsgroups and online forums over the years. Cookies are simply text files stored on your hard drive and cannot themselves harm your computer in any way.

Typically cookies are used to remember logins and keep track of user settings on web-sites. Cookies can be used to track your movement on the Internet ONLY if a site is aware of the cookies and is designed to use the specific cookies.

Because of their use in tracking, many feel that this constitutes spyware. Most anti-spyware applicatoins, including SUPERAntiSpyware, detect tracking cookies in one form or another. If an application does not detect cookies, users often feel the application is “missing” critical spyware items because another scanner will detect them and label them as “spyware”.