Vesselin Bontchev

← All papers

The Dark Side of Whitelisting

[Bontchev07a] Virus Bull., August, 2007, pp. 4–8.

1 Introduction

There is a person going by the name Robin Bloor who, for the past year and a half, has consistently posted to his blog and to various other places on the Web articles about how “anti-virus” is dead or dying slowly or at least in serious trouble. One of his articles, in the UK electronic publication The Register ([Bloor07]), caught my attention. There were so many wrong things in it, that I could not resist posting a comment. Later it occurred to me that Mr. Bloor’s articles might mislead some people, so it is worthwhile writing and publishing an article on this subject myself – something people could use as a reference for debunking such delusions.

First of all, like many people who are not anti-virus specialists, by “anti-virus” Mr. Bloor understands essentially known-malware scanners. Of course, this is by far not the only kind of anti-virus technology in existence – it is only the most widely used one. In my Ph.D. thesis ([Bontchev95]) I list seven main types of anti-virus programs – some of which are further subdivided into half a dozen subtypes. And even that source is already obsolete – my thesis was written more than a decade ago and the technology hasn’t stayed in place meanwhile. Nowadays there is sandboxing and a bunch of other anti-virus techniques…

The reason why Mr. Bloor has become disenchanted with “anti-virus” seems to be because the constantly increasing flood of new malware has made the known-malware scanners difficult to cope with the constantly changing threat. For instance, the Storm family of Trojan horses uses what we call “server-side polymorphism” – every time someone downloads one, the server generates a new variant – resulting in about 54,000 variants released in just one week ([Venable07]). Unlike conventional polymorphism found in viruses, this malware is not self-replicating and the polymorphic engine is not present in the malware for the anti-virus researcher to analyze and develop generic detection for it. Of course, this does not mean that there are no efficient techniques against this – see the above reference for details.

It is a fact, however, that conventional scanners are notoriously bad at coping with non-replicating malware. At least a virus replicates – so once the scanner producer implements detection of it, other users will be protected. As opposed to that, Trojans are usually one-short weapons – at the time when the anti-virus people get a sample, the victim is already compromised and the same Trojan is unlikely to be used again against somebody else, so implementing detection of it isn’t of much help to anyone.

Of course, the fact that the number of known malware programs is increasing at an ever faster rate has prompted many people to predict, over the past couple of decades, that at some point of time conventional scanners will “die” – i.e., will become ineffective. Nevertheless, the said conventional scanners are still very much alive and kicking – although the technology behind them has improved significantly (albeit in ways not immediately obvious to the user). There are perfectly good reasons for this – conventional scanners are the kind of anti-virus programs that the average user is most comfortable with, in the sense that he or she can easily understand and use them. (See more about this in section 4.) I am pretty sure that another decade from now conventional scanners will still be in use and people like Mr. Bloor will still be predicting their imminent demise.

Another thing about conventional scanners that Mr. Bloor does not like (and he is certainly not alone in this), is that they require constant updating. That is, the user has to keep paying for them.

Anyway, since Mr. Bloor has decided that conventional “anti-virus” does not work any more, he is left with the obvious question what to replace it with - because something has to be used to protect the user from malware, right? For Mr. Bloor, this “panacea” seems to be the so-called “whitelisting”.

2 What is whitelisting?

If you consider how conventional scanning works, it basically builds a “black list” – a list of known bad programs that shouldn’t be allowed to run on the user’s computer. As new such malicious programs appear, they are constantly added to the black list – this is why scanners need constant updating.

Whitelisting is the opposite of this. Instead of building a list of things that should not be allowed to run, the anti-virus program uses a list of things that should be allowed to run – and denies execution of everything that is not on this “white list”.

The idea of whitelisting is certainly not new. For a long time, people have been joking that the number of known malicious programs is increasing so fast, that at some point of time it will be easier to scan for known good programs instead. As early as in 1990, Dr. Fred Cohen ([Cohen90]) proposed the idea of “integrity shells” – i.e., programs that would allow the execution only of software with known-good integrity (using a database of checksums to control integrity and relying on the fact that viruses have to modify the programs they infect and, therefore, destroy their integrity). The author of this article uses the Kerio personal firewall in “paranoid” mode, which makes it issue an alert every time when an “unapproved” program is executed – this is a kind of whitelisting, too.

Since whitelisting has been around for almost three decades, the natural question to ask is, if it were so much better than conventional scanning, why has it not replaced the latter entirely already? The answer, of course, is because it cannot. It, too, is fraught with problems – only different problems from the ones conventional scanning has. On its own, whitelisting cannot stop malware – just like, on its own, conventional scanning (or other “blacklisting” based approaches) cannot.

3 The problems of whitelisting

So, what are the problems of whitelisting? Basically, they boil down to two – what is a program, and which programs are good ([Wismer06]). (Just like the problems of any blacklisting-based approach boil down to two – what is a program and which programs are bad.)

3.1 Exotic execution exceptions or what is a program

Theoretically, the question of what is a program is an unsolvable one. That is, for any given sequence of symbols, there exists at least one Turing Machine for which this sequence is a program, if it appears somewhere on the machine’s tape.

Of course, real computers are not Turing Machines (a Turing Machine as infinite memory, for instance) – but we hit another snag there. All contemporary computers are based on the von Neumann architecture – and an underlying principle of this architecture is the equivalence between code and data. That is, one program’s code is another program’s data and vice versa. For instance, a JavaScript text is program to the browser – but it is data to the editor used to create it.

The most trivial approach to whitelisting is to build a database of known good frequently used executable files – that is, EXE, COM, BAT and maybe a few kinds of scripts – and deny execution to any other executable program not listed in the database. Sadly, this leaves the door open for many other ways a piece of malicious code could enter.

For instance, what about Office macros? Of course, once the producer of a whitelist-based protection program has though about them, it is possible to make the program monitor macros in documents, too. But how many producers would think of them off-hand? And how many have the competence to handle obscure file formats – because you have to get to the macro bodies; you can’t just whitelist some Word documents and deny access to all other documents. And macros are just the beginning.

What about obscure scripting languages? Most producers would probably think of JavaScript and VBScript – but there are so many others, and most of them are sufficiently powerful to write viruses in them. For instance, what about the scripting languages of the various IRC clients (e.g., mIRC)? There are already many viruses written in these languages. Recently, a virus appeared that was written in the scripting language of the hex editor WinHex – how many producers of whitelist-based protections have heard of this scripting language? Other obscure scripting languages for which viruses exist include (but are not limited to) ABAP, ActionScript, Ami Pro macros, AutoLISP, Corel Draw! Script, Ferite, IDA script, KIX, Lua, MathLab script, One C, WinBatch, REG script, SQL, Perl, Python, Ruby… (OK, please don’t shoot me for calling the latter four “obscure scripting languages”.)

Furthermore, what about Office exploits? They arrive in a Word or Excel or PowerPoint document. Some obscure field in the document is corrupted, causing a buffer overflow somewhere in the Office application that opens it. This (the corruption) is the exploit. The exploit causes control to be transferred to a small piece of code that resides in the document too (usually, but not always – close to the place where the corrupted field is). This small piece of code is called “shellcode”. Then it usually extracts the real malicious program appended (often in encrypted form) after the end of the document – or downloads it from somewhere and runs it.

Now, a whitelist-based approach can prevent the dropped (or downloaded) executable from running. But it cannot stop the execution of the shellcode – not unless it stops the Office applications from running or disallows the opening of foreign documents – both of which would make the machine essentially unusable. And the shellcode doesn’t really have to drop an executable – it’s just easier to implement it this way. The shellcode runs directly in memory, in the context of the user who has opened the malicious document and can do everything that user is allowed to do. There is no hope for a whitelist-based approach of preventing that.

And what about threats that take control before the whitelist-based protection has had the chance to execute and then use stealth to disguise their presence? A boot sector virus would be the obvious example but there are many other convoluted ways in which malware can get itself executed during the computer’s startup process and before any whitelist-based protection.

As if all this wasn’t already bad enough – what about threats that do not exist as files at all? A typical example is the CodeRed virus ([CodeRed]). It enters the attacked system as an HTTP GET request to port 80, exploiting a vulnerability in one of the system DLLs. It is never saved as a file (or any other object on the disk) – instead, it spreads memory-to-memory between the vulnerable machines on the Internet.

How would a whitelist-based program protect against that?! Even a blacklist-based approach (i.e., a scanner) has trouble with this kind of viruses, because there are no obvious files to scan. But at least one can implement a packet scanner (i.e., again a blacklist-based approach) and block the requests sent by the virus before they reach the vulnerable DLL. As opposed to that, whitelisting network packets is a hopeless task.

3.2 Which programs are good?

The question of whether a program performs only legitimate actions is again an unsolvable one, in the general sense – just like, in general, it is impossible to answer the question of whether a program is a virus or whether it will stop after a finite number of steps (the so-called Halting Problem). The proof that these questions are unsolvable are constructive ones. That is, if someone claims to have invented an algorithm that can solve any one of these questions, the proof shows how to construct a program, for which the algorithm will give incorrect results.

But, again, the above is just theory. In practice, there are other, more immediate problems. Let us suppose that you have decided to protect your computer from malware using a whitelist-based approach. For this, you need two things – a program that implements the approach and a whitelist (i.e., a database of legitimate programs that the program would allow to execute, while blocking everything else). OK, the program you purchase from some software producer – but what about the whitelist?

There are, basically, three possible solutions. One, you can purchase such a whitelist from somewhere. Two, you can build one yourself. Three, you can use a combination of the first two approaches. Unfortunately, each of these approaches has its own problems.

3.2.1 Scalability, or the problems of global whitelists

Let us suppose that you decide to obtain your whitelist from a third party. This could be the same producer who sold you the whitelist-based program – or it could be an independed vendor. In both cases, however, you are most certainly not the only customer this vendor has – or they would be out-of-business. And the other customers most certainly don’t use exactly the same “good” programs that you do.

So, in order to serve all its customers, the whitelist vendor will have to compile some kind of global whitelist – a list of all legitimate programs in existence – in order to make sure that all programs their customers are likely to have are covered. In fact, there are already several companies trying to do just that – Securewave, Bit9, AppSense, etc. Unfortunately, as it turns out, their task is much, much more difficult than the task of the anti-virus companies – and for the same reasons, too.

At the recent International Antivirus Testing Workshop in Reykjavik, Iceland, in May this year, I attended a presentation by Mario Vuksan from Bit9 ([Vuksan07]). According to him, the company he works for is trying to build just such a global whitelist – and is having a very hard time doing it.

As it turns out, there are many more legitimate programs than malicious programs out there – many orders of magnitude more, as a matter of fact. Even worse, the rate of creation of legitimate programs by far exceeds the rate of creation of malicious ones – and it keeps increasing. According to Mr. Vuksan, just Microsoft, IBM, SourceForge and Mozilla.Org produce, respectively, 500 K, 100 K, 500 K and 250 K new executables every day! Currently, Bit9 have 2.7 billion files listed in their global whitelist, aiming for 10 billion (that’s US billion) – and are nowhere near finished. Just the index of their database is more than a hundred gigabytes large! So, the joke about the malicious programs soon outnumbering the legitimate ones is nothing more than a silly joke – there is absolutely no hope of this happening any time soon (or even at all).

Clearly, the glut problem faced by a global whitelist producer is much worse than the problem faced by the average scanner producer. After all, we have to deal with “only” about 5,000+ malicious programs per month.

This leads to other problems as well. First of all, how are you going to deliver this database (the global whitelist) to your customers’ computers? Delivering the whole of it is clearly out-of-question – who will be willing to dedicate more than a hundred gigabytes just to the database used by their malware protection? Not to mention that it will have to be updated with a couple of million new entries every day – much worse than the oh-so-hated regular scanner updates. (And the producer is unlikely to be willing to update it for free, either.)

Keeping it on the servers of the database producer and having it accessed remotely is not good, either. What are you going to tell your customers – sorry, you can’t use your computers today, because we (or you) have a network failure and we can’t check right now whether this program you want to run is legitimate or not?

Furthermore, can you honestly believe that a company examining a couple of million new executables per day is not going to make mistakes and put malicious ones on the list? Even the anti-virus people tend to make an occasional mistake when telling the good programs from the bad – and these people are experts and their workload is orders of magnitude lower!

In addition to the technical problems, a global whitelist raises a political one – pretty much like centralized code signing. It is potentially dangerous to allow one entity to control the process. One the one hand, how can you be sure that a small producer’s program will be included? On the other hand, how can you be sure that a big company like Sony BMG will not pay the whitelist producer to include their latest rootkit as a legitimate program in their database?

3.2.2 User (in)competence, or the problems of local whitelists

The alternative to using a whitelist somebody has built for you is to use one you have built yourself. Then you do not have the problem of millions of new legitimate programs per day – because the software installed on your computer is, more or less, constant. Sadly, this approach has problems on its own.

The main problem is that it relies on the user being competent enough to build such a whitelist. It is much more difficult than just running some sort of program that would scan your disks for executable programs and build a database of checksums for them.

For instance, how do you know that your system is not already infected? For most users, the only way to know this is by running a virus scanner. Oops, there goes the hope that conventional anti-virus is “dead”. In addition, most users resort to seeking some kind of protection from malware only after their machines become infected.

Furthermore, what are you going to do when you want to install a new program? Obviously, you will have to update the whitelist in order to record the new program as legitimate and allow it to be executed. Of course, in order to do that, you will have to be able to decide whether the new program is legitimate. But if the users, in general, were able to do that, they wouldn’t get infected in the first place and there wouldn’t be any need for anti-virus software (no matter whether whitelist- or blacklist-based one) – because everybody would be installing only legitimate programs on their computers!

This approach is applicable only to tightly controlled corporate environments, where security is paramount (and takes precedence over usability), where the local whitelist is built by a competent security administrator, contains a relatively small number of programs, and the users are strictly forbidden from installing and running anything new. Sadly, it is totally unusable in home user (and even in most corporate) environments.

3.2.3 The sum is less than its parts, or the problems of the combined approach

If neither of the above two approaches really works, then how about a combination of them? For instance, the whitelist-based protection vendor could supply you with a relatively small database of “most popular legitimate programs” and also give you the possibility to update the database locally with any programs that you use but which are not included in it. Maybe if you do that, the combination of the two approaches will cancel each other’s deficiencies?

Sadly, this is not the case. The combined approach just combines the problems of the above two approaches. For instance, who can decide what are “the most popular programs”? And they are not a static set, either – so the database will have to keep increasing (or at least keep changing) – meaning that the product will have to be constantly updated – just like a conventional scanner.

And if you are allowed to put your programs on the whitelist, you are most likely to “whitelist” some malicious program by mistake or due to lack of competence.

4 Why people use mostly scanners

Malicious programs have been with us for more than a quarter of a century. Yet people still use mostly known-malware scanners to protect themselves from such programs. Now, I would be the first to admit that scanners are the weakest kind of protection from malware. Why, then, do the people still rely almost exclusively on them?

It is not, as some people like Mr. Bloor would have you believe, because of some kind of dark conspiracy among the anti-virus producers who want to keep getting money from you for their updates. It is because this is what the free market has established, no matter whether we like it or not.

Fact is that the average user is not interested in becoming a security expert. The users just want to be left alone doing their jobs, playing games, surfing the Web (mostly porn sites). They start thinking about security only when some malware bites them.

A known-malware scanner is something the user can easily understand and use. It tells the user “no, your computer is not infected” or “yes, your computer is infected with the XYZ virus; do you want me to remove it?”. As opposed to that, the other kinds of anti-malware protection schemes require a significant level of competence from the user, in order to be understood and used correctly.

A heuristic analyzer would say, “The file Foo may contain a virus”. Well, does it, or does it not? A firewall would say, “The program svchost.exe is communicating over port 1900”. What the heck does that mean and should it be permitted? A behavior blocker would say, “The program msvc.exe is trying to write to file Blah.exe”. Is that a virus attack? A whitelist-based protection would either occasionally deny the running of a program the user wants to run, or would ask the user whether the program is legitimate and should be added to the whitelist – a decision the user is generally not competent to make.

This is why most users keep buying scanners – because they are easy to understand and easy to use. Yes, they are the weakest line of defense against malware – but this is what the users want to buy. And since we, the anti-virus producers, have to eat too, this is what we have been making and selling them.

We would gladly sell them something more secure and some of us have tried to do so over the past couple of decades. Does anybody remember Fred Cohen’s Integrity Shell? The program Untouchable? Integrity Master? These were all generic, integrity-based products that were very efficient at stopping virus infections. Nevertheless, none of these products is made any more. The users voted with their wallets and the companies making these products ended up either going out of business or starting to do something else.

The same will happen to whitelisting, which is essentially a form of integrity checking. Any company that makes a product based exclusively on this approach will ultimately fail – because it will sell only to a relatively very small set of customers. Some anti-virus companies will include whitelisting-based protections in their suites – and the users, in general, will keep happily using only the scanner part of these suites – the part that they understand.

5 Use the Force, Luke

From what I have written so far about whitelisting, some readers could be left with the impression that it is a very bad idea, that it does not work and should be avoided like the plague. Nothing could be farther from the truth.

As mentioned in the previous section, whitelisting is essentially a form of integrity checking – and I am a very strong proponent of integrity-based malware protection schemes, because malware (and especially computer viruses) are essentially an integrity problem.

With this article I am just trying to emphasize that whitelisting, alone, is incapable of stopping malware efficiently and is fraught with problems – just like blacklisting, only different kinds of problems. The proper way to protect from malware is by implementing defense in depth, not based on any particular single approach.

Use scanning to ensure that the system is initially malware-free. Use an integrity checker to ensure that it is not modified without authorization at a later date. Use behavior blockers to detect intrusions. Use a personal firewall – not only to protect from external attacks but also as a kind of behavior blocker – to detect if a program on your machine is trying to “phone home” without your knowledge. Use sandboxing to isolate and misdirect potentially troublesome programs. Use encryption to hide the information that does not have to be visible all the time. Use backups!

Unfortunately, I understand very well that this is just wishful thinking from my part. While a handful of geeks like me might be willing to understand what all of the above means and how to install and operate it successfully on one’s computer, it will remain forever way above the head of the average user who will happily keep using inefficient known-malware scanners.

6 Conclusion

Whitelisting, per se, it totally incapable of replacing conventional scanners as far as the general public is concerned. It does work well in small and tightly controlled environments, where security is more important than convenience, but the average home user (and most corporate users, as well) will never be able to rely on it exclusively. It would be best if whitelisting is combined with blacklisting (and will all other anti-virus techniques) to establish defense-in-depth. Sadly, most users do not have the competence required to build, understand, and maintain such defense – while they are able to use and understand scanners. This is why conventional scanning will be with us for a long, long time – Mr. Bloor’s rants notwithstanding.

7 References