Dr Vesselin Bontchev, anti–virus researcher
FRISK Software International
Postholf 7180, 127 Reykjavik, ICELAND
E–mail: bontchev@complex.is
Abstract: During the past six years of its existence, the WildList—a list of viruses confirmed to be in–the–wild—has changed from a curiosity to an indispensable information source on which many authoritative tests of anti–virus software are relying heavily. However, it also has its problems. Since its authority has grown significantly, so have its responsibilities—and it is therefore important to analyze its problems and to consider whether they still allow it to fulfill these responsibilities. This paper analyses in details the role and the problems of the WildList and suggests ways in which the List can be improved, and most of the problems—resolved. It is the result of the author’s several years of experience as both anti–virus researcher and WildList reporter.
| Download this paper as a Word document | Download this paper as a ZIPped Word document |
2.1. Cross–Reporting
2.1.1. Inter–Producer Dependency
2.1.2. Inter–Product Dependency
2.2. Only Problems Are Reported
2.3. Mostly New Viruses Are Reported
2.4. Samples Are Not Sent
2.5. Methodological Errors
2.5.1. Methodological Errors of the Data Gathering Process
2.5.2. Lack of Clear Definitions
2.6. Sluggishness
3.1. CARO Virus Names Not Used
3.2. Viruses Not Identified Exactly
3.3. Reliance on the Competence of the Reporters
4.1. Second Part Not Used in Tests
4.2. Misuse of Test Results
4.3. Quality Equalizer
4.4. Different Viruses Used in Tests
The WildList, a list of computer viruses confirmed to be in–the–wild, was started by the anti–virus researcher Joe Wells about six years ago. He correctly observed that of the thousands of known viruses, only a small percentage seemed to be causing real–life infections and wanted to establish which particular viruses those were.
Initially started as a curiosity–driven experiment and maintained almost single–handedly by Mr. Wells with some help from CARO (the Computer Anti–virus Researchers’ Organization), the WildList project has nowadays grown into the WildList Organization, with a board of five directors, its own Web site (http://www.wildlist.org), and an extensive network of people reporting computer virus incidents all over the world.
Concurrently to this growth, the importance of the WildList project has grown accordingly. It has been correctly observed that a greater emphasis should be put on testing whether the existing anti–virus products handle successfully the computer viruses which are really out there, causing infections (as opposed to those which seem to exist only in the virus collections). Therefore, nowadays most well–known anti–virus testing bodies (e.g., the ICSA, Virus Bulletin, VTC–Hamburg, University of Tampere, Secure Computing, etc.) put a special emphasis on using a collection of the viruses listed in a relatively current issue of the WildList when testing anti–virus software. Some of them go as far as using only such viruses in their tests and certification schemes.
All this means that the authority of the WildList has grown tremendously. With this growth of authority come also greater responsibilities. Yet the WildList is far from problem–free. This paper aims at examining the problems which currently exist in the WildList and at suggesting solutions which would help solving these problems and increasing the quality of the WildList and of the benefits it eventually brings to the end user.
The main flaws in any information which relies extensively on multiple reports are always caused by problems in the quality of the reporting. When it has to rely on human beings (as opposed to on some kind of automatic information–gathering devices), the quality of the reporting often leaves a lot to be desired. People simply aren’t very good at reporting in a standard, uniform way plain, boring facts—especially when the methodology of the reporting is not designed to facilitate this task. In this section we shall examine some of the problems of the WildList caused by problems in the quality of the reporting and what can be done to solve at least some of them to at least some degree.
Contrary to what might appear at a first glance, the reports of the WildList reporters are far from independent. Two kinds of dependency exist among them— inter–producer dependency and inter–product dependency.
Many of the WildList reporters are affiliated, in one way or another, with some anti–virus producer. Sometimes the affiliation is not obvious, or it is not obvious that two reporters are affiliated with practically one and the same producer.
For instance, the author of this paper is a WildList reporter for FRISK Software International (FSI). However, FSI has partners (Data Fellows, Command Software Systems, etc.) who are large anti–virus vendors themselves. The company (and its partner companies) have distributors all over the world, some of which are WildList reporters themselves. Yet the main anti–virus research and development is done at FSI. What this means in practice is that viruses reported to one of our partner companies, or to one of our or their distributors, are likely to be sent (by those recipients) to FSI—since it is usually us at FSI who develop the updates to our scanning engines and virus definition databases, necessary to detect these viruses. As a result, everybody who is affiliated directly or indirectly with FSI and is a WildList reporter, is likely to report the virus. Therefore, multiple reports will be sent for what is essentially a single incident. Without doubt, similar redundant reporting is received from the WildList reporters who are affiliated (directly or indirectly) to the other large anti–virus producers with a lot of partners, agents, distributors and so on. All this leads to a skew in the accuracy of the WildList data toward over–reporting.
What can be done about this problem? One solution is for the WildList Organization (WLO) to keep track of who of the reporters is affiliated with which anti–virus producer and, if two possibly duplicated reports are received, to double–check with the other reporters and determine whether it is indeed a duplicate report or two independent reports. However, this would require too much additional work and would put too much additional strain on the WLO.
The WLO currently implements this approach only to a very minor extent. In the WildList, the affiliations of the reporters are listed both as companies and as products. This way it can be seen which reporters are affiliated with which product—even if they are affiliated with different companies. However, even this is not always sufficient. For instance, our partners, Data Fellows, have a product named F-SECURE—while our product is named F-PROT. Thus, at a first glance it seems that the people reporting from Data Fellows and the ones reporting from FSI are completely independent (different companies, different products)—while in fact they are not. Furthermore, no efforts are made to cross-check the reports for the kind of dependency described in this section; the user of the WildList is simply left wondering whether the listed reports are indeed independent or not.
Another solution (voluntarily adopted by the author of this paper in his WildList reports) is for each developer to indicate unambigously in (his)her report whether the report is of a virus is received directly from a customer or from some of the entities associated with the respective anti–virus producer—developer, partner, agent, vendor, etc. If this is done in a standard enough format, the cross–correlation of received data can easily be automated at the receiving point (the WLO) and the redundant reports can be correspondingly eliminated.
It is well–known that many users use more than one anti–virus product, in the hope of increasing their security by the means of redundancy. Discussion of whether such an approach indeed succeeds in achieving its desired goal is beyond the scope of this paper—we are just observing the fact that often one and the same user uses more than one anti–virus product. If that user is also inclined to report virus incidents to the WildList reporters (something which is by far not always the case; see sections 2.2 and 2.3), then (s)he is also likely to send such reports to the producers of all the anti–virus products (s)he is using. As a result, there will be again a skew in the accuracy of the WildList reports towards over–reporting—this time originating at the place of the virus incident instead of in the reporting chain.
Solving this problem is significantly more difficult than the previous one. The users are significantly more numerous, significantly less qualified and significantly more difficult to control than the WildList reporters. One approach to improve the situation to at least some degree is to have each reporter obtain from the user information whether that user is using any other anti–virus products and which ones, and to report this information to the WildList, together with the remaining usual data (virus sample, virus name, date and country of the report). The WLO could then eliminate the reports of the same virus from the same date and coming from the same country, if the "anti–virus products used" part of these reports suggest that they come from one and the same user.
However, this approach is imprecise and can lead to a level of under–reporting by eliminating seemingly duplicated reports which are in fact independent. One improvement is, in order to achieve better accuracy of these reports, to report not only the country but also the town from which the report originates. Unfortunately, this information is often unavailable (because many reports are sent by e–mail with sufficient indication of the country of origin but no information about the town of origin) and gathering it would pose additional strain to the WildList reporting system.
During our twelve years of experience as an anti–virus researcher, we have noticed that the users tend to report only viruses with which the anti–virus product they are using has a problem. Problems, caused either because the anti–virus product doesn’t know these viruses and cannot, therefore, detect, recognize, identify and disinfect them, or because it has some other problem finding and removing the virus(es).
A few years ago Dr. Alan Solomon did the following experiment. By that time virtually nobody was getting reports of the Cascade.1701.A virus, so it was suggested that this virus had already become extinct. Dr. Solomon released a version of his anti–virus product in which the disinfection routines for this virus were intentionally disabled. And suddenly his company began receiving a lot of reports about this virus—because the users had problems removing it and were calling the technical support department. It turned out that the virus was not extinct at all—simply all the popular anti–virus products were handling it without problems, so the users were not reporting anything.
Another example. The current versions of the scanner VirusScan from Network Associates Inc. have generic drivers for automatic handling of whole families of "popular" macro viruses—like W97M/Class, WM/Wazzu, and many others. This approach permits new variants of these families to be automatically recognized and disinfected without being reported as new variants and without the user having to send a sample to the producer of the anti–virus product. (Of course, the drawback is that the exactness of identification of the viruses of these families is greatly reduced.) Once these generic drivers have been implemented and shipped to the customers, NAI suddenly observed a major drop of the user reports of the viruses of these families. In the same time, other companies whose scanners kept reporting every new variant in these families as, indeed, a new variant and requesting a sample from the customer, kept receiving many user reports about new viruses from these virus families.
All this results in a tremendous level of under–reporting. The WildList reporters receive reports mostly of viruses which some anti–virus products have problems with. In addition to the lack of reports, this also introduces a bias suggesting that some viruses (the problematic ones) are more prevalent than others (which, in reality, are just as prevalent, but which most anti–virus products have no problems dealing with). The end result is that the WildList does not correctly reflect the reality.
Solving this problem, as most problems caused by the properties of the human nature, is extremely difficult. The only solution we can see is to eliminate the human factor from the equation as much as possible. Humans are inherently unreliable. Therefore, they must be replaced by automatic information–gathering devices whenever possible.
One way to achieve this is to implement the idea proposed by Roger Thompson at the ICSA’97 conference. The idea consists of having the scanner producers introduce some kind of automatic reporting in their products. Nowadays, most computers are connected to the Internet in one way or another. Many anti–virus products use this connection to automatically download updates of themselves. In theory, nothing prevents them from using the connection to also automatically report every virus they have found to some kind of a centralized report gathering agency—or at least to their producer.
Of course, in order for this idea to be useful to the WildList, several problems have to be solved first.
First of all many users would regard it as a serious intrusion of their privacy, if the fact that their computers have been hit by a computer virus is reported to third parties. Therefore, this automatic reporting has to be turned off by default and should be explicitly enabled by the owner of the system—if this owner agrees to be part of this reporting scheme.
Second, the WildList requires that a sample of the virus is always sent to accompany the report of it. Especially with macro viruses, this can post serious confidentiality problems—because the infected sample is usually a company document whose contents is often confidential and should not be disclosed to third parties. In order to solve this problem, the samples have to be somehow "purified" first—so that any information in them not related to the virus itself is removed. Some anti–virus producers like NAI and IBM already have some experience in this area, related to their so–called "immune systems" which also perform some kind of automated computer virus sample gathering. Maybe these companies could share the relevant technology with the WildList Organization in the name of common good.
Third, some measures have to be put in place to prevent the virus writers and the virus collectors from skewing the WildList by scanning large virus collections of viruses which have never been actually found in–the–wild and let the scanners’ automatic reporting feature send its reports. There have been many cases, showing that the authority of the WildList has raised even among the virus writers—nowadays many of them aim for their new creation to "make it onto the WildList", just like before they were aiming for it to be detected by some name by the popular scanners. For some of these people, it is perceived as a kind of "recognition" for the virus they have created.
There is no easy way of solving this last problem. In general, the samples received via the automatic reporting feature should be subjected to the same kind of scrutiny which the manually sent samples are subjected to now. Often it is possible to determine from the contents of the sample and from the image of the virus in it whether this is a real–life infection sample or simply a sample from the virus collection.
This problem is related to the one described in section 2.2. Since new viruses are usually the viruses which the known–virus detection anti–virus products have problems with, and since the known–virus detection anti–virus products are the most widely used kind of anti–virus products, it is not surprising that new viruses are the ones which are most often reported.
And indeed, if one analyses the contents of the WildList over time, one would notice that of the viruses that are newly put on the List (in the sense that they were not on the List before), the vast majority are new viruses. In fact, if one allows for the necessary delay between the time a virus is discovered by the anti–virus researchers for the first time and the time it makes it to the WildList (if at all), one would see that the viruses which are newly put on the WildList are almost exclusively new viruses. The cases when a virus was known for a long time before it made it on the WildList, or when a virus was listed there, then removed because of no reports having been received about it, and then put it back there, are so rare as to be almost non–existent.
The solution to this problem is similar to the one outlined in section 2.2—the human factor must be eliminated as much as possible from the report–gathering process.
The current policy of the WildList is that only those computer virus reports are accepted, which are accompanied with sample(s) of the respective virus(es) being reported. This policy is understandable and correct—it is the only way to determine with certainty which particular virus is being reported—due to the fact that most scanners do not identify exactly all viruses they can detect and are often unable to distinguish between two closely related variants of the same virus family.
However, this policy has also a negative impact on the WildList. Of those few times when the users bother to report a virus to the producer (which, as explained in sections 2.2 and 2.3, happens by far not every time these users have a computer virus incident), they usually just say what the anti–virus product has reported and do not send a sample. Contacting the user and requesting that (s)he also sends a sample of the virus is often infeasible (e.g., because the user cannot be reached easily) and is almost always regarded as an completely unnecessary inconvenience. Unfortunately, this also means that only a minuscule part of the real computer virus incidents are properly reported (accompanied with samples) to the WildList.
We shall illustrate the above with an example from the experience of the company we work for. It involves the macro virus W97M/Groov.A. Every time it replicates, this virus runs the program IPCONFIG (a program for displaying the configuration of the network card of the computer and which program is present on any machine running Windows 9x or NT) and gathers its output to a text file. Due to the way the virus invokes this program, this works correctly only on Windows 98 systems. Then the virus uploads the resulting text file to the ftp site of our company. We are not sure why exactly the virus author has done this—the uploads do not put any significant strain on our ftp server. Maybe it was the virus author’s idea of a practical joke; a way to "thumb his nose" at us and show us that his creature is "alive and well".
From our side, we collect these reports, try to determine the e–mail addresses of the infected users (the IP address of the infected machine is present in each of those reports; the problem is that the infected machine does not necessarily have an e–mail address and even when it does have one, it is not always trivial to determine it).
Since this virus has appeared, we have received several hundreds of thousands of such reports. Granted, one and the same infected machine is usually responsible for more than one report before the infection is discovered (statistical analysis of our data shows an average of 5–7 replications before the infection is discovered). Still, these uploads are a rather reliable evidence that this virus has infected tens of thousands of machines. Yet, so far we have received (and, accordingly reported to the WildList) only 2 (two!) reports of it which reports were accompanied with samples. This gives us an impression about the huge level of under–reporting that the WildList is suffering from.
One of the members of the WildList Organization’s board of directors has made the statement that this isn’t really important, because the W97M/Groov.A virus is listed on the WildList. Therefore, if a virus is actually "out there", it will end on the List sooner or later. However, such an attitude is short–sighted and is missing the point. If several tens of thousands of computer virus incidents result in only two reports verifiable with samples (and, therefore, suitable for the WildList), one cannot help but wonder how many other viruses are "out there", causing "only" thousands of incidents, for which no verifiable reports have been received yet?
Another example is the W97M/ColdApe.A virus. Every time it replicates, this virus sends taunting e–mail to Nick FitzGerald—then editor–in–chief of Virus Bulletin. It does this using VBScript—which means that it will be unable to do it if the Windows Scripting Host is not installed. (It is installed by default only on Windows 98 machines.) By mid–December 1998, when the virus was close to 15 weeks in–the–wild, Nick FitzGerald knew about hundreds of infected sites and had personally discussed the virus and how to remove it with many of the infected victims. Yet, according to the WLO, nobody else of the WildList reporters was reporting this virus. This, despite the fact that many of the infected victims were customers of Network Associates or Symantec and ought to have reported the virus to their anti–virus vendor, hoping to get a "fix" for it. Yet none of those vendors (who have several WildList reporters) had reported the virus. This suggests that some vendors avoid reporting viruses which the shipping version of their anti–virus product does not detect.
Again, the only reliable way to improve the situation is to do everything possible to exclude the human factor from the computer virus reporting business and to replace the unreliable users with automated sample–gathering programs. Unfortunately, as explained in the previous sections, this approach still has a lot of unresolved problems on its own.
Actually, the process of sending virus samples together with the WildList reports has yet another aspect, particularly relevant to the author of this paper. We have a policy of not sending computer viruses to anyone, except to a very limited group of people who are CARO members. (Exceptions are made extremely rarely and only when the person asking for the viruses has a demonstrable need–to–know; for instance, when the virus demonstrates some bug in their anti–virus product.) Normally, this policy should not have been a problem—one of the members of the WLO board of directors and WildList founder, Joe Wells, is a CARO member. So, we were sending any new viruses we were receiving to the CARO members, including to him.
To our surprise, several of our WildList reports were not taken into account—and we were not even informed about it. When we discovered it ourselves and asked for explanation, we were told that these reports of ours have been ignored because they had not been accompanied with samples.
This suggests a very serious lack of communication within the WLO. It turns out that they do not really know whether virus samples have been sent to one of them or not—and that they would not bother to inform the reporter for any perceived problems in (his)her report. This necessarily has a serious negative impact on the quality of the WildList itself.
This issue cannot be resolved by purely technical means. The WLO simply has to get their act together, pay close attention to what has been sent to each one of them, and contact immediately the reporter if they have any problems with (his)her reports.
Some of the lack of accuracy of the WildList and of its failure to reflect properly the viruses which are actually "out there" is caused by the fact that the people maintaining the project have made serious methodological errors when designing the process of information gathering. Errors, which a more professional body of specialists with more experience in such methods as collecting data from large–scale experiments, opinion polls and so on, would not have made.
This problem was present in the original version of the WildList, as created by Joe Wells, and remained in the WildList, as maintained by the WildList Organization until very recently. It was correctly observed that "the list of viruses which are in–the–wild" cannot be a static list. New viruses appear all the time—but also old viruses become extinct and aren’t encountered any more (except in the virus collections). Therefore, for the WildList to reflect the reality properly, there had to be mechanisms both for adding new viruses to it and for removing old viruses from it. The error this section is discussing was made in the implementation of the latter mechanism.
Originally, Joe Wells asked the CARO members which viruses they aren’t seeing any more, so that he could remove them from the List. When the WildList Organization was formed to maintain the WildList, they made the same mistake—but on a much larger scale. Special forms were distributed to the WildList reporters. On these forms, the reporters had to indicate both which viruses they have received samples of this month and which viruses they had not seen during the past 8 months.
Essentially, the methodological mistake made by the WLO was to put data processing tasks on the data gatherers. Furthermore, those were tasks which were difficult for the data gatherers to perform, while they would have been relatively easy to perform by the data processing entity (the WLO). It is difficult enough to keep track of which viruses the reporter has seen the last month—especially having in mind that some of the WildList reporters work for anti–virus companies and see thousands of viruses each month—old, new, from the wild and from virus collections. It is practically impossible for them to remember which viruses they have not seen during the past eight months!
As a result, the reporters simply were not reporting anything like that and once a virus was put on the WildList, it stayed there essentially forever, without any hope of being removed—since the probability that enough people would remember to report it as "not having seen it during the past 8 months" was disappearingly low.
The correct way to solve this problem is for every reporter to submit reports in the following standard format for each virus (s)he is reporting: date, virus name, country where the virus incident has occurred, comments, virus sample. (This is what the author of this paper always did; where in the comments part he indicated whether the virus was received directly by FSI or by one of the partner companies.) Then the WLO can easily process this data automatically, determine which viruses have not been reported by any of the reporters during the past 8 months and remove them from the next issue of the WildList.
It is not until a couple of months before the writing of this paper that this approach was implemented to some extent by the WLO—and after the author of this paper had pushed them to do it and explaining for years why it has to be done! The existing mess was not fixed until one of the members of the WLO board of directors, Shane Coursen, took the initiative to review manually a whole year of reports, determine which viruses from the then–current issue of the WildList did not figure among those reports, ask each reporter for confirmation that (s)he had indeed not seen these viruses during the past year or so, and remove these viruses from the List.
We do not know whether currently the task of removing old, non–reported viruses has been automated in the way we have been suggesting for years, and if yes—to what degree. Fact is that the reporting forms distributed to the WildList reporters still do not reflect the format suggested by the author of this paper. We wish to hope that the problem will be eventually satisfactorily resolved in the future.
There is another methodologically wrong requirement imposed on the WildList reporters—but we do not know to what degree it is imposed, so we are uncertain of the level of negative impact it is causing. The requirement is that the reporter should not submit a report, unless it has been confirmed with samples from two independent sources. This clearly poses unnecessary strain on the data gatherers—they have to keep a number of "pending" singleton reports, waiting to see whether they would be confirmed with a second, independent report, before actually reporting the virus to the WildList. This greatly increases the probability for mistakes.
Again, the issue is resolved much easier on the data processing end of the chain. The reporters should send in each and every report which is confirmed with a sample, clearly labeling it with the date it has been received and its place of origin. When the WLO is processing these reports, it can easily and automatically apply on them any policy they wish—for instance, they can ignore the entries for viruses which appear only once in the monthly report.
When the Win32/SKA virus appeared and quickly became widespread, it caused a significant confusion among the WildList participants. According to some definitions (for instance, the one used by the author of this paper), this virus is a worm. It turned out that it was not clear to the WildList participants—or even to the members of the WLO board of directors—whether it is a worm or not and whether it should be reported to the WildList and listed there. Discussions related to this issue continued for months and consisted of many hundreds of messages posted to a few mailing lists. Meanwhile, the virus was happily spreading in the wild—and, in fact, had become one of the most widespread viruses there.
The discussions revealed again a severe lack of communication between the members of the WLO board of directors, as well as the lack of agreement among the virus experts even on some very basic definitions. For instance, according to some (including the author of this paper), a worm is a virus which explicitly uses the network to spread itself. According to others, a worm is a stand–alone replicating program; a virus without a need for a host. One expert even suggested a third definition—that a worm is a virus which can spread fully automatically from one machine to another—without the need for a human to copy and execute an infected file. Furthermore, the experts disagreed on whether the worms are a subclass of viruses or whether they are a class of their own. Fortunately, all those who insisted that Win32/SKA is a worm, also agreed that worms are a subclass of viruses and, therefore, it is also a virus. However, borderline cases like it will appear more often in the future—in fact several of them have already appeared— X97M/Papa.A, Win95/Pretty_Park, Win32/ExploreZip. They are all in–the–wild, they are all using explicitly the network to spread themselves and they all can be considered worms by at least some of the popular definitions.
But this raised the question of whether worms should be reported to and listed on the WildList or not. One of the members of the WLO board of directors made the statement that worms are not supported by the WildList—just as viruses for platforms other than that IBM PC compatible machines are not supported. In the same time, another member of the WLO board of directors said that "yes, of course if it replicates and is in–the–wild, it should be listed". This reveals an unacceptable level of lack of communication between the WildList organizers, as well as a lack of clear and objective methodology and definitions on which to base the management of the WildList. If such a confusion exists even among the WildList organizers, one can easily imagine how much worse is the situation among the WildList reports. Who knows how many reports have not been sent only because the reporter was not sure whether the replicating thing (s)he has received a sample of should be reported—or, worse, was mistakenly under the impression that it should not be reported and did not bother even to ask.
One of the members of the WLO board of directors made the statement that it is left to the discretion of the reporters to decide what to report. In our opinion, such an attitude is clearly unscientific and unacceptable. The qualification of the WildList reporters varies a lot and it is often not verified to an acceptable degree before a person is accepted as a reporter. The WLO should prepare a set of clear and objectively applicable instructions of what exactly to report and how—and make these instructions available to the WildList reporters. It should be easy for every reporter to apply these instructions and to decide objectively whether the program (s)he has received a sample of should be reported to the WildList or not.
In our opinion, the WildList should cover all self–replicating programs which are reported in–the–wild. This means both regular viruses and worms, and all platforms should be covered—not just the IBM PC but also the Macintosh, Linux, and any other platform for which self–replicating programs are reported from the wild. (The "IBM PC only" approach is particularly indefensible, having in mind that with the appearance of macro and script viruses, one and the same viral program can—and often does—successfully replicate on several different platforms.) On the other hand, the WildList should not include any non–replicating programs—like Trojan horses, jokes, backdoors, etc.
It is acceptable that the WildList does not cover other platforms temporarily—for instance, because the WLO lacks sufficient expertise on those platforms to properly process the incoming reports for them and replicate the received samples. However, this ignoring of the non–IBM PC compatible platforms must not be done as a matter of principle; it must not be a policy. Furthermore, the WLO should actively seek the advice of experts who do have the necessary expertise in the area of non–IBM PC compatible platforms and viruses for them.
If a virus makes its way to the wild, it takes at least a month for it to make it onto the WildList. If it has been reported only by a single reporter, it is put only in the second part of the WildList—which part is normally not used by the anti–virus products testing organizations (see section 4.1). Furthermore, once a virus makes it onto the WildList, it would stay there for at least 8 months, even if nobody ever reports it any more. In our opinion, this level of inertia of the WildList is unacceptable and prevents the List from reflecting the reality properly.
The reality is that most of the infections nowadays are caused by macro viruses. Macro viruses, which are often minor variants of a few "popular" families and which are usually locally created in the companies where they are reported. This virus variant creation is performed either automatically by Word randomly corrupting the virus body (so that the virus is still able to replicate) or by some incompetent user trying to "disable" the virus by commenting out parts of its code—but failing to comment out all the parts that could replicate the resulting new program.
Such newly created variants usually spread very fast within the company (because macro viruses spread very fast)—but they almost never leave the company. Nowadays most anti–virus producers have a very fast reaction time—they can provide the updates of their product which are necessary in order to handle the new virus just in a matter of hours. The new virus is, therefore, quickly exterminated and nobody ever hears about it again. It remains only in the collections of the anti–virus researchers—and one or two of them (who have the infected company as a customer and are WildList reporters) might report it to the WildList.
But those are usually singleton reports—which means that they never make it to the part of the WildList which is used for anti–virus product testing and certification. Or, if they make it there, they stay there long after the virus has been exterminated. In short, the reality is mainly that of fast appearing and fast disappearing new macro viruses—and the WildList fails to reflect that. Needless to say, this has a negative impact on the reliability of the anti–virus product certifications, which certifications are based on the first part of the WildList.
In order to solve this issue, we propose that the WLO treats the macro virus reports differently from the reports of other viruses. A newly reported macro virus should be added to the first part of the WildList immediately after it is reported—even if only a single reporter has reported it. Furthermore, a macro virus should be removed from the WildList much faster than other non–reported viruses—at most 3 months of non–reporting (instead of the standard 8 months), although an even shorter period (e.g., one month) might be advisable. This will make the WildList more dynamic and will make it reflect the reality better than it does now.
In this section we shall consider the different problems the WildList has in respect of classifying and naming the viruses listed there.
One of the most annoying—yet one of the easiest to correct problems the WildList has is that the names of the viruses listed there do not conform 100% to the CARO virus naming scheme. It is bad enough that there exists a major naming confusion among the anti–virus products as far as computer viruses are concerned. There are even such extreme cases as 8 different names being used for one and the same virus by 8 different products (e.g., the names Jerusalem, Friday the 13th, Black Hole, 1813, etc. are all different aliases for the same virus used by different anti–virus products).
The closest thing to a standardization that exists in this area is the so–called CARO virus naming scheme, created by three anti–virus researchers (Dr Vesselin Bontchev, Dr Alan Solomon and Friðrik Skulason) in 1991. Currently most known–virus detection anti–virus products comply with this scheme to some degree (although none achieves 100% compliance) and the scheme has helped greatly to reduce the virus naming confusion problem.
The viruses listed on the WildList are often listed with their standard CARO names—but sometimes they are not. When the List was maintained single–handedly by Joe Wells who was occasionally receiving help in this aspect from the other CARO members, compliance with the CARO virus naming scheme was relatively easy to achieve. Before publishing the next issue of the List, Mr. Wells (a CARO member himself) would send a preliminary version to the other CARO members and they would send him back any corrections (including corrections of the naming of the viruses listed there) they had.
Since the WildList began to be managed by the WildList Organization, none of the people responsible for its maintenance has bothered to consult the CARO members on such issues as virus naming and to send them preliminary issues for corrections. Nevertheless, the author of this paper, every time he saw an issue of the List, he took the trouble to send to the WLO a list of corrections to the names of the viruses listed there, in order to achieve 100% compliance with the CARO naming scheme.
Most of the time, those corrections were simply ignored. One of the members of the WLO board of directors even went as far as to claim that the names used on the WildList intentionally do not comply with the CARO naming scheme; that the WLO prefers to use the names "most commonly used by the scanners" instead. To quote that person, "when given the choice of being pedantically correct or being less correct but not causing a problem for users/vendors, we will choose the latter every time".
In our opinion, this attitude is unscientific and unacceptable for an organization responsible for the maintenance of a document which is supposed to have some scientific value. The WildList should be made as precise as possible—and making it use the exact CARO names of the viruses it lists is definitely possible. Most of the time it can be achieved with minimal effort—and the CARO members should be consulted on every issue which is unclear. In the long run, this will benefit the users by providing them with a reliable source of exact and correct information.
Such benefit cannot be achieved by using a "voting" approach and letting the majority of scanners (many of which are not compliant with the CARO naming scheme) decide under what name to list a virus. Furthermore, any self–respecting scientist should strive to be as precise and objective as possible—instead of letting (his)her results being driven by what is "not causing problems" to the anti–virus vendors.
It is with satisfaction that we have observed that lately a large part of the names on the WildList have been brought to compliance with the CARO virus naming scheme. This was mainly due to the efforts of Shane Coursen who decided to act contrary to the opinion of the person quoted above. He is to be commended for that—but we cannot help but notice that some of the names of the viruses currently listed on the WildList still do not comply 100% with the CARO virus naming scheme and, therefore, there is still some work remaining to be done in this aspect.
It is particularly important that the WildList sticks to some kind of virus naming standard (and the CARO virus naming scheme is the best thing currently available in this aspect). Many anti–virus vendors nowadays have a policy that their product should detect 100% of the viruses listed in the WildList—and they report those viruses with the names under which they are listed there. Therefore, the WildList can contribute to at least reduce the existing confusion in virus naming by exercising pressure on the anti–virus producers to use one and the same standard names at least for the widespread computer viruses.
Another problem the WildList has is that the viruses listed there are not identified exactly. This problem is somehow related to the one discussed in section 3.1. For instance, a virus named, e.g. Foo.1234.A is often listed there just as Foo.1234. If a closely related variant named Foo.1234.B also exists, it would not be clear which of the two variants (or maybe both of them?) are actually in–the–wild.
The main reason behind this problem is because very few scanners identify exactly the viruses they report—and most reporters rely on scanners to tell them which particular virus they have received, simply because they do not have the necessary expertise to identify the virus manually themselves.
Unfortunately, this approach can have very confusing results. For instance, a very popular anti–virus product reported all variants of the CIH virus family as CIH.1075. In reality, there is no variant of the CIH virus with such an infective length. The really widespread one is CIH.1003.A. Such a mistake in the exact variant identification can seriously mislead the users—for instance, the really widespread variant of this virus activates its destructive payload on April 26, while some other variant activates its payload on 26th of any month.
It is understandable that many of the WildList reporters do not have the expertise necessary to identify a virus exactly. But they should not be required to do so. After all, this is the reason why they are required to send a sample together with their report. It is the WildList maintainers who have to perform the virus identification—and not by blindly relying on scanners, either. If they have problems identifying some particular virus, they should ask the CARO members for help—none of the CARO members will ever refuse such kind of help.
Yet we, the CARO members, almost never receive this kind of requests for help from the WLO. Maybe it is caused by ill–understood sense of pride. Maybe there are other reasons. But whatever those reasons are, this attitude is wrong, unacceptable, and, in the long run, causes harm to all users of the information presented in the WildList. This situation must change, if the WildList is to achieve the level of usefulness it aims at.
This problem is somewhat related to the problems listed in sections 2.5.2, 3.1 and 3.2. In short, the WLO seems to be dumping too much responsibility for the correctness of the data gathering process on the shoulders of the WildList reporters. In the same time, it does not seem to take the necessary steps to ensure that only people capable of carrying these responsibilities are selected as WildList reporters. These two approaches are mutually conflicting and greatly reduce the quality of the information present in the WildList.
With the risk of repeating ourselves, we would like to state it again. It is unacceptable to "leave it to the discretion of the reporters to decide what to report". Instead, the reporters should be provided with clear, easily applicable and objective instructions to aid them decide whether to report a malicious program they have received or not to report it. It is unacceptable to let the reporters rely on scanners to tell them what is the virus they are reporting and the WLO to accept this report blindly. Instead, the WLO should identify manually and exactly every virus they receive—asking external experts (such as the CARO members) for help whenever necessary. In the same time, the reporters should be selected not just for their willingness to report and for the geographical regions they reside in (regions, reports from which are desired), but also for their competence in the field of anti–virus research.
Blind reliance on scanners and on semi–competent people using them can lead to truly ridiculous results. For instance, there are 3–4 boot sector viruses currently on the WildList, of which viruses no replicable samples are in possession of the WLO. Worse, we have been unable to locate a live sample of those viruses even in the collections of the anti–virus researchers. One cannot help but wonder what are those viruses doing on the List? Why have reports of them been accepted in the first place, if they haven’t been accompanied with replicable, "live" samples? Why haven’t these viruses been removed from the List, after the author of this paper and other experienced anti–virus researchers have pointed out (numerous times!) to the WLO that these samples are not valid?!
For instance, the blind reliance of some testers on the WildList has resulted in them using in their tests the (often non–replicable) boot sector only file images which are distributed by the WLO, thus giving the users of the tested anti–virus products a false sense of security. As an example of a more responsible approach, Virus Bulletin has a policy not to include in their tests viruses they have been unable to replicate—even if those viruses are rumored to be "in–the–wild". This policy has resulted in some of the WildList–listed viruses being removed from Virus Bulletin’s in–the–wild test set—simply because no "live", replicable samples of them could be found.
The problems, considered in this section, are not problems of the WildList per se, but more problems of the ways the information contained in the WildList is used. They are mainly caused by involuntary (or sometimes intentional) errors made by the testers of anti–virus products, the people presenting the results of those tests, and the end users of those results. Therefore, our criticisms in this section are directed mainly at those people; not at the WildList Organization.
The WildList consists of two parts. The first part lists viruses which have been reported by two or more of the WildList reporters. If only a single reporter has reported a virus, that virus is listed in the second part of the WildList. This does make sense, since it helps isolating the "flukes", carries information of which viruses are clearly more widespread than others, and helps preventing bias intentionally introduced by one of the reporters (e.g., because that reporter is affiliated with a particular anti–virus producer, the anti–virus product of that producer can detect a particular difficult–to–detect–but–not–widespread virus, and he has the interest to see the competing products listed as being unable to detect the virus).
However, few people know that only the viruses listed in the first part of the WildList are actually used in the so–called WildList–based anti–virus product certification tests. Yet the viruses listed in the second part of the WildList are in–the–wild too—just not as widespread as the ones listed in the first part. Simply ignoring their existence seems unacceptable to us. The level of under–reporting the WildList currently has is already unacceptably high (as explained in sections 2.2, 2.3, 2.4 and 2.6); there is no need to make the situation even worse by ignoring additionally the reports which are deemed acceptable even by the current reporting policy.
Therefore, we propose that the anti–virus products testing and certification bodies should use viruses from both parts of the WildList in their tests. They should use two sets of virus collections—one reflecting the first part of the WildList and one reflecting the second part—and test how the different anti–virus products fare against each one of these collections separately. Such an approach would have the advantage that it would not ignore the useful information present in the second part of the WildList, yet it would preserve the usefulness of having these two sets of viruses considered separately. Clearly, detection of viruses in the first set is more important—but detection of the viruses in the second set is also very important.
After all, these viruses are in–the–wild too; just not as widespread as the others. Sometimes a virus is reported only by a single reporter only because this virus is widespread only in the geographical region where this reporters resides (and there are no other reporters in that geographical region)—but the virus is extremely widespread there. Sometimes these regions are rather large and it is very important to the users in those areas to know whether the anti–virus products they are planning to rely on for protection against computer viruses is able to handle the viruses which are widespread in their geographical region. That one of those viruses is not widespread elsewhere in the world would be of little consolation to the people hit by it who were relying on an anti–virus "certified" for its ability to detect "all in–the–wild viruses".
There are several official anti–virus products testing entities that issue different kinds of "certifications" or "rewards" to the anti–virus products which demonstrate the ability to detect 100% of the viruses listed on the WildList— ICSA, Secure Computing, Virus Bulletin. Some testers have additional levels of certification—e.g., the ability to not only detect but also to disinfect the WildList viruses (used by Secure Computing) and so on. The "logos" of these rewards are proudly displayed by the producers of the corresponding anti–virus products on the box the product is shipped in, and/or on their Web sites. The users are let to believe that being awarded one of those signs somehow certifies the superior quality of the anti–virus product.
In reality, this is far from being the case. As explained in section 4.1, only the first part of the WildList is used in those tests. As explained in the previous sections, the WildList itself suffers of many severe completeness and accuracy problems. The users are mislead to believe that the WildList is "a list of the viruses which are in–the–wild"—while, in practice, it is only a list of viruses for which a limited set of reporters have received from the wild reports accompanied with samples. This is something completely different.
Therefore, the ability to detect the viruses listed in the WildList must not be used as a measure for the quality of an anti–virus product. Instead, the opposite kind of measurement should be used—the inability to detect the viruses listed in the WildList should be used to indicate which anti–virus products are of unsatisfactory quality. If a product can detect all these viruses, it only means that it is worth considering; nothing more. It does not necessarily mean that the product is good. Additional tests should be made in order to determine whether the product is indeed good or not. (Of course, if it fails to detect even the viruses listed in the first part of the WildList, it means that the product is so bad that it is not worth even to be considered.)
In order to resolve this issue, the WLO should put some kind of disclaimer, clearly explaining what the WildList is and what it is not. The WLO should encourage the anti–virus product testing bodies to conduct tests in the spirit of the explanation given in the previous paragraph—i.e., that the ability to detect the viruses on the List should be used only to measure which products are clearly not good enough and use much larger and more representative virus collections to measure the actual quality of the products. The testing bodies themselves should impose restrictions on the anti–virus producers in order to prevent the sign of their certifications from being mislead the users to believe that it alone is an insurance for the high quality of the product. Furthermore, testing the ability to detect the viruses on the WildList should be only one of the tests performed by these anti–virus products testing and certification bodies.
The viruses listed on the WildList are relatively few—just two–three hundreds. This is virtually nothing, when compared to the tens of thousands of viruses known to exist. It is, therefore, not surprising that the anti–virus producers enthusiastically embraced the idea of anti–virus products tests based only on the WildList viruses. One major anti–virus producer even made the public statement that their company will be concentrating on these viruses only and will treat the remaining ones as a matter of secondary importance. This is understandable—it is much easier to ensure proper handling of a few hundreds of viruses than of tens of thousands of them.
However, in our opinion, such an attitude is plainly wrong. Even though the other viruses have not been reported to cause real–life incidents yet, most of them are freely available to anyone who wants them from the so–called virus exchange web sites. This means that any attacker who is inclined to do so could take one of them and use it in a virus attack.
Furthermore, incidents do happen. Just recently we had a documented case when a major customer of ours in the USA got 3,500 of their workstations infected with the macro virus W97M/Ded.A. This virus is of Russian origin and was not widespread at all—and especially not in the USA—until that incident. The internal investigation discovered that the infection happened because somebody downloaded the virus from a virus exchange site "to use it in a test of anti–virus products" and accidentally released it. The virus was relatively new, and none of the anti–virus products used by the company was able to detect it, so it succeeded in spreading widely within the company.
Therefore, while the ability to detect the viruses known to be in–the–wild is of utmost importance for the quality of an anti–virus product, the ability to detect the remaining of the known viruses is also very important. This is particularly true for the anti–virus products of the known–virus detection type—that is, the so–called scanners. They are the weakest form of protection against computer viruses—all they can do is to detect known viruses. Simply using a new, unknown virus, is sufficient to get past such a defense. Well, if this is all the scanners are able to do, then they must be able to do it well enough indeed—and detect as many of the known viruses as possible. It is simply that detection of the viruses which are known to be spreading out there is more important—but that does not mean that detection of the remaining known viruses is not important.
There is also another issue related to this point. Since detecting just a couple of hundreds of in–the–wild viruses is relatively easy, most products nowadays are achieving it. Therefore, if the tests of anti–virus products are conducted based on this citerion alone, it would be very difficult for the user to decide, from the results, which products are better than the others—for the simple reason that most products will score basically equally well. This can be seen especially well in the detection rate graphs present in the comparative reviews of anti–virus products conducted by Virus Bulletin—most products score about the same and it is close to the 100% mark.
In order to resolve this problem, the organizations which conduct tests and certifications of anti–virus products should use more than just the viruses listed on the WildList in their test virus collections. Currently, Virus Bulletin and the ICSA make some feeble efforts in that direction. Virus Bulletin uses an additional, so–called "standard" set of about 1,500 viruses ("Standard" by whose standards? There are currently more than 43,000 known viruses; how can a subset of less than 3.5% of them be considered "standard"?) and the ICSA requires, in order to certify an anti–virus product, that product to be able to detect 100% of the viruses listed on the WildList and 90% of an undisclosed set of other viruses (Why undisclosed? How many are they? What criteria has their selection been based on?). Only the tests conducted by VTC–Hamburg use an extensive collection of tens of thousands viruses, in addition to the viruses listed on the WildList.
Our advice is that all anti–virus products testing and certification bodies should use collections of known viruses which are as extensive as possible—although, of course, the results of testing the products against the viruses of the WildList should still be presented separately, because they are of utmost importance. If a product fails to achieve 100% detection of them, the level of detection of other viruses that it achieves is hardly very relevant.
Nowadays many anti–virus products testing and/or certification bodies use the viruses listed in the WildList to test anti–virus products. Yet the results of those tests, even when one and the same version of one and the same product is involved, are often different. Often a product scores 100% detection of the WildList viruses in one test but fails to detect all WildList viruses used in another test. What is the explanation of this rather strange phenomenon?
Well, there are several reasons why this happens. First of all, the competence of the different testers is not the same. Some of the tests are not conducted as meticulously as others and some testers fail to discover that some product does not always detect reliably all replicants of a given virus under all realistic conditions. But there is another reason as well.
It turns out that not all testers who claim to be using the viruses listed on the WildList actually do so. And it is usually not their fault, either. As explained in section 3.2, the WildList does not always identify the particular variants of the viruses listed there with a sufficient degree of exactness. As a result, two testers could use two different virus variants and still believe that they are using one and the same virus—the one listed in the WildList.
As an example, in one of the Virus Bulletin’s comparative reviews, our product (F–PROT) was reported as not detecting the One_Half virus. The virus is listed on the WildList and we had taken the necessary efforts to ensure that the virus there was, indeed, detected by our product. So, why the discrepancy of the results? Well, it turned out that in their tests Virus Bulletin had used a variant of this virus which was new to us and which our product indeed did not detect. However, it was not the variant listed on the WildList.
Solving this kind of problems is more difficult than it seems at a first glance. For instance, even if the WLO solves the problems listed in sections 3.1 and 3.2, this still will not solve the problem for the tester. It is nice if the WildList lists precisely and correctly the particular virus variant that is in–the–wild—but how can the tester make sure that this is indeed the variant they are using in the tests? It would be a grave methodological mistake for the tester to rely on one (or even more) of the scanners tested to tell (him)her the names of any of the viruses used in the test.
It was proposed that the WLO sends a collection of the viruses listed in a particular issue of the WildList to anybody who wants to perform WildList–based anti–virus product tests. In fact, in his speech at the Virus Bulletin Conference 1998, Shane Coursen incorrectly stated that decision had already been taken by the WLO to do so in the future.
However, this "solution", although it does solve the problem described in this section, is dangerous and unacceptable. There is no reliable way of ensuring that the recipients of the virus collection will have indeed the necessary expertise to handle properly the viruses and not leak them due to negligence—or even maliciousness. Fortunately, the WLO has backed away from the decision to do so and has stated that it will not send virus collections out, except to a very limited set of people whose competence and trustworthiness is already known and with whom the WLO already has established a line of trust.
But this leaves opened the question—how to solve the problem raised in this section? One possible solution is for every anti–virus product tester who wants to use a collection of WildList viruses in their tests and already believes that (s)he has such a collection, to send the collection to the WLO for approval. The WLO can then verify whether the viruses present in that collection are indeed those listed in some particular issue of the WildList—asking, if necessary, some experts for help—e.g., the CARO members. If the wrong variant of some virus is present in the collection, or if some virus is missing from it, the WLO should inform the tester about this fact, without sending (him)her any viruses. It should be the responsibility of the tester to obtain the viruses (s)he needs for the test; the WLO should only "certify" (his)her virus collection as indeed being a collection of WildList–listed viruses.
The increased authority of the WildList has lead to increased responsibilities for it. Yet, the list is still plagued with serious problems—problems which raise the doubt whether it is able to meet these responsibilities. It is clear that these problems must be resolved as soon as possible. Yet, in the past few years the WildList Organization has been rather slow in its reaction and problem–fixing activities. Clearly, it is time for a change.
To help stimulate such a change, we have the following proposal. It is well–known that competition stimulates improvement of quality. So far the WildList has had some kind of "monopoly" in the anti–virus products testing field. Maybe it is time for other organizations to start their own lists of viruses known to be in–the–wild. If these lists are of better quality, better maintained, and have fewer problems than the current WildList, they will successfully compete with it for the attention of the testers, despite the big initial advantage the WildList has. This kind of competition will inevitably force the WLO to improve the WildList too. All this will have a clear benefit to the end users by providing them with more accurate information of which viruses are actually in–the–wild and also indirectly providing them with better information about the quality of the anti–virus products.