I recently got ~40000 malware samples, so i decided to test COMODO AntiViruSpyware Beta 2.0. Set up a VM, downloaded latest AV software distributions and ran the tests. To make it clear, i also decided to test Kaspersky Internet Security 7 and NOD32 v3.0 antivirus capabilities and to compare the results with CAVS ones. I am certainly not trying to pretend being some virus expert. Just been doing these test for myself and thought - why not make them public? I did my best not to be biased.

----------------------------------------------

DISCLAIMER

I am not an employee of COMODO, ESET or Kaspersky Labs. I don't even have a job yet.

The process of testing COMODO AntiViruSpyware and KIS7 was being done on a ultra-fresh (nothing but AV installed) nlited XP Home SP2 inside VM, WITHOUT any network connection - that is, AV and it's DB's were outdated, so the test will be rerun with the remaining undetected files as soon as possible. NOD32 testing was being done on a physical machine with more recent (but still outdated) databases.
Also, all 5 tests were run three times each - one to clear out signature-detected samples, second one to determine how many additional threats AV can find with maximum heuristics enabled, and the last one - to test AV disinfecting capabilities.

I must make some corrections:

• CAVS test was run three times. One without heuristics, one with maximum heuristics and one to check it's disinfecting capabilities.
• KIS7 test was run three times also, but i couldn't determine how many threats KIS7 really detected with heuristics. In fact, i don't even know how much it detected relying solely on signatures (read notes below)! KIS7 test results are about ~90-95% accurate.
• NOD32 test was run three times, but it was different. One - relying only on signatures. Second one - extended signatures (spyware/riskware etc). And third one - with solely heuristics enabled. There is no way to determine how much NOD32 has disinfected since in NOD32's terminology "cleaning" is either disinfecting or deleting. NOD32 test results are probably ~70-90% accurate.

----------------------------------------------

Test results:

* - read notes.

NOD32 v3.0 approximate scanning results:

** - note that these results are based on "remaining" files. But it's obvious that in case of viruses the "detected" samples count is way too low, which makes me think that most part of them are not viruses, but "cleaned" leftovers.

While in testing process, i noted some interesting things thad made me to correct disclaimer and later write lots of other stuff that made this "article" so huge.

Note 1: Some of infections detected by COMODO AntiViruSpyware (CAVS) and NOD32 (probably KAV too) were mislabeled (wrong signature?)

Note 2: I was unable to clean three samples with OneHalf virus with NOD32. AV said they were cleaned when they were not

Note 3: As a relatively new software on the market, given the "rogue" antivirus problem, i had to ensure that CAVS itself is not malware and is not cheating. The testing process revealed that it's highly unlikely that CAVS is cheating in any way (one scan on one VM, three scans on another - no difference) and it's certainly not a malware.

Note 4: CAVS' heuristics give a somewhat false sense of security - NOTHING was detected by heuristics. (Last week i did a quick test and when i enabled high heuristics - CAVS found additional 15 viruses. I don't know why did CAVS detect 15 more viruses after setting heuristics to "high" while it did not detect a single one during the present test, maybe definitions update occured while i was scanning?).

Note 5: Accurate testing of KIS7 (and probably NOD32) is not possible.

I've been doing this test for four nights and discovered some things that i must take into account while interpreting the testing results. The first thing is the fact that KIS7 scans inside packed/joint files. Thus it can detect multiple threats in one file, greatly adding the number of detected threats (e. g. finds 3 threats in 1 packed file, so that counts as 3, not as 1). This makes it impossible to rely on it's report since the detected threats count is even greater than samples count.

I was relying on the "remaining files" count (the ones KIS7 didn't kill, assuming they were not detected) until i found out another interesting thing. I was searching through "Options" and found the option that turns on "Riskware" detection. Then i decided to test how many "riskware" it can detect within the remaining undetected files (by that time the testing was almost finished, i had a list of undetected files which i planned to scan upon definitions update). I knew there were 16 previously undetected files in "other malware" directory and i ran a scan on that. KIS7 detected some additional "riskware", i deleted (not disinfected!) it... BUT! The remaining files count was still 16! That is - KIS7 didn't delete the infected files, it deleted infections inside packed files instead! This means that the "remaining" files are not necessarily malicious! This probably put in doubt all of my test results (which were relying on the remaining files count rather than on scanner's results) and i gave up testing KIS7. However, the overall detection rate was about ~98-99%. (it is generally possible to determine disinfected file from real malware by it's filesize (assuming malware can't be 50 bytes size), but that's not accurate too (who knows, maybe it can!). I have not enough knowledge to determine real malware from the remainings of a packed file).

The problem gets even worse when it comes to NOD32. Though i use NOD32 for more than a year now, i never had an opportunity to test it properly (just "set and forget"). And now i did. This piece of software definitely seriously lacks configurability. There are three major issues with NOD32 version 3.0 (these problems do not apply to NOD32 v2.7 so i'll switch back to 2.7 and retest).

The first one is cleaning mode. I can't set it to delete every infected file (like in KIS or CAVS), i can't set it to only clean and not to delete infected files (like in KIS or CAVS) - i can't customize this at all. All i have is - "no action", "default action" (doesn't delete infected files within archives with legitimate files) or "strict cleaning" (cleanes out everything). If i just want to delete all it can detect - i can't, it still disinfects everything it can disinfect.

It's the nature of viruses - generally they can infect files without destructing them, so it is possible to revert the changes made by virus. That's why NOD32 has ridiculously low detection rate with viruses.

The second one is it's cleaning habits. It is acceptable when AV doesn't attempt to clean files with "system" attributes set (to prevent possible damage to Windows) but there must be an option to clear out literally everything, even if it will kill the system. I spent a night trying to figure out why NOD32 does not automatically clean everything even in "strict cleaning" mode and keeps asking "clean, delete or leave" with almost every threat it detects. Still, even after removing attributes it couldn't clean some threats without my permission, most likely because infected files inside packed/archived files had "system" attributes set. (the more i use console - the more i realize it can't be replaced with all that GUI stuff. Just a simple console command - and all "system" attributes of 23'000 files are cleared, no lags, no hangs. An attempt to remove these from the GUI (through properties window) crashed Explorer (since reading attributes for 23'000 files ain't that easy)

And the third one, finally. The quarantine. NOD32 puts everything in there. After running a test on ~40'000 viruses i have a HUGE quarantine that i can't even clean. Oh yes, i can, but deleting ~30'000 files one by one ain't that much of a fun, you know.

CONCLUSION

Right now the best signature-based AV solution is Kaspersky's Antivirus. Hourly updates, reasonable speed (version 7 MUCH faster but still bloatware IMO, that's why i don't use it), second to none protection from known threats, greatest configurability... Still, i don't have fellow virus writers to ask them to test KAV on new malware that is supposed to be detected by heuristics, and that's where the real challenge must take place.

A couple of words about primary subject of this testing - COMODO AntiViruSpyware Beta 2.0. Overall detection rate is not that good for everyday use and heuristics are nothing more than a tick in the "options" menu. However, it is new, and it has perspectives. And this nice piece of software is free (Never tried free Avira or Avast but probably will. Later). Also i strongly suggest you to try out COMODO firewall. Works fine with NOD32 (probably one has to disable NOD32's proxying to be able to use any firewall (except ESET's) properly), works fine with other AV software (with minor issues sometimes, better check out COMODO forums before installing anything)... And after some spiritual dancing it can even live in peace with Kaspersky (never tried though, but people say they can live together).

As for NOD32... I used to believe it's the perfect solution for me - detects all "in-the-wild" viruses (numerous VB100's), is believed to have best heuristics (latest tests point on Avira, though all tests are different), is fast and lightweight and highly configurable. Not that now i'm disappointed and going to ditch NOD32, no. I just wear no pink glasses anymore. No AV can achieve 100% detection, but NOD32 is nowhere near 100% (unlike KAV). It's heuristics certainly detect something (even with signatures completely disabled it still can see about 40% or so), but i can't really test that so i'll stay neutral here. As for speed - i didn't notice when that happened, but NOD32 ain't that fast and lightweight anymore. Yes, it's scan speeds are still far superior than everything else (including CAVS) but since it places everything in quarantine before doing anything with the infected file, cleaning speeds are now slow as hell! Configurable... Yes, configurable, but again nowhere near KAV in terms of configurability. Still, it's an ideal solution for me since i am a power user and don't have a problem in terms of security conciousness (sometimes i'm a little paranoid though). But, with reasonable detection rates of known threats - why not use CAVS then? It's lighter, somewhat faster (NOD32 will be faster than everything else again when the developers remove that stupid "quarantine everything" feature) and it's free... Of course CAVS has that uber-cool heuristics feature, but nevertheless. Unknown threats are no problem to me since i have a truly powerful COMODO Firewall v3.0 with it's second to none HIPS features, so i kinda don't need a powerful AV software to catch something attempting to crawl inside.

P. S. every single letter in this huge text is my private opinion. I am solely responsible for my own words. No harm was meant, just my own observations, not necessarily 100% correct. Any comments are welcome.