I’ve covered the impact that automated detection systems have on false positives in the past. Hispasec, the makers of VirusTotal, also talked about this issue in their blog post aptly named Antivirus Rumorology. More recently Kaspersky conducted an experiment during a press conference and showed a bunch of journalists how these false positives roll over from one vendor engine to the next. Of course being journalists, they only took home the message “AV copies each other and mostly us” as is shown in the articles published covering the event . Even though the objective of the experiment was put under scrutiny, the fact remains that this is an industry-wide problem and no single vendor is immune to its effects, not even Kaspersky as we will see.

As some of the regular readers of this blog will probably remember, in March 2010 we published a “PandaCloudTestFile.exe” binary file to test the connectivity of Panda products with its cloud-scanning component, Collective Intelligence. This “PandaCloudTestFile.exe” is a completely harmless file that only tells the Panda products to query the cloud. Our cloud-scanning servers have been manually configured to detect this file as malicious with the only objective of showing the end user that the cloud-scanning component of his/her product are working correctly.

Initially this file was only detected by Panda as Trj/CI.A (a Collective Intelligence detection) and Symantec’s Insight (noting that this is not a very common file, even though treating reputation alone as “suspicious” is by itself grounds enough for debate — maybe another future post).

Panda 10.0.2.2 2010.03.10 Trj/CI.A
Symantec 20091.2.0.41 2010.03.11 Suspicious.Insight

Re: Notepad.exe false positive « Reply #10 on: November 27, 2010, 12:48:44 PM » Looks like it has been sorted out in the latest definition update (101127-0) Thanks for verifying the file as safe and adding it to the whitelist. VirusTotal false positive contacts collection Wall of Shame. These vendors don't provide any way to submit a false positive without making an account, or at all. Alibaba (virustotal@list.alibaba-inc.com rejected my mail as spam) AhnLab-V3; ALYac (requires program) Cynet; Elastic; Malwarebytes; Sangfor Engine Zero.

A few days later came the first problematic detection, this time from Kaspersky, who detected the “PandaCloudTestFile.exe” with a signature, specifically calling it a Bredolab backdoor. I call this detection problematic as it is clearly not a suspicious detection nor a reputation signature. It is also clearly an incorrect detection as the file in itself is not related in any way to Bredolab. Soon we will see why this Kaspersky signature is problematic.

Kaspersky 7.0.0.125 2010.03.20 Backdoor.Win32.Bredolab.djl

In the next few days some other AV scanners started detecting it as well, in many cases with the exact same Bredolab name.

McAfee+Artemis 5930 2010.03.24 Artemis!E01A57998BC1
Fortinet 4.0.14.0 2010.03.26 W32/Bredolab.DJL!tr.bdr
TheHacker 6.5.2.0.245 2010.03.26 Backdoor/Bredolab.dmb
Antiy-AVL 2.0.3.7 2010.03.31 Backdoor/Win32.Bredolab.gen
Jiangmin 13.0.900 2010.03.31 Backdoor/Bredolab.bmr
VBA32 3.12.12.4 2010.03.31 Backdoor.Win32.Bredolab.dmb

In the month that follows (April 2010) a bunch of new engines started detecting it, mostly as the Bredolab name we are now familiar with, although some new names started appearing as well (Backdoor.generic, Monder, Trojan.Generic, etc.).

Ahnlab V3 Report False Positive Form

a-squared 4.5.0.50 2010.04.05 Trojan.Win32.Bredolab!IK
AhnLab-V3 2010.04.30.00 2010.04.30 Backdoor/Win32.Bredolab
AVG 9.0.0.787 2010.04.30 BackDoor.Generic12.BHAD
Ikarus T3.1.1.80.0 2010.04.05 Trojan.Win32.Bredolab
CAT-QuickHeal 10.00 2010.04.12 Backdoor.Bredolab.djl
TrendMicro 9.120.0.1004 2010.04.03 TROJ_MONDER.AET
Sunbelt 6203 2010.04.21 Trojan.Win32.Generic!BT
VBA32 3.12.12.4 2010.04.02 Backdoor.Win32.Bredolab.dmb
VirusBuster 5.0.27.0 2010.04.17 Backdoor.Bredolab.BLU

And to top it all off, during this month of May 2010 the following engines started detecting “PandaCloudTestFile.exe” as well. Here we can also even see a “suspicious” detection, probably the only one out of all of them that could make any sense.

Ahnlab V3 Report False Positive Results

Authentium 5.2.0.5 2010.05.15 W32/Backdoor2.GXIM
F-Prot 4.5.1.85 2010.05.15 W32/Backdoor2.GXIM
McAfee 5.400.0.1158 2010.05.05 Bredolab!j
McAfee-GW-Edition 2010.1 2010.05.05 Bredolab!j
Norman 6.04.12 2010.05.13 W32/Suspicious_Gen3.CUGF
PCTools 7.0.3.5 2010.05.14 Backdoor.Bredolab
TrendMicro-HouseCall 9.120.0.1004 2010.05.05 TROJ_MONDER.AET
ViRobot 2010.5.4.2303 2010.05.05 Backdoor.Win32.Bredolab.40960.K

It is worth noting that consumer products have other technologies included in their products, such as white-listing and digital certificate checks, which could cause the file to not be detected on the consumer endpoint, but the fact that there is a signature for such file is a good indicator that it will probably be detected on the endpoint.

So why am I writing about all this? First of all, to emphasize the point I tried to make in the past that automated systems have to be maintained, monitored, tuned and improved so that more in-depth analysis is done through them and not rely so much on “rumorology”.

Secondly, to show that this is an industry-wide problematic that results from having to deal with tens of thousands of new malware variants per day, and no vendor is immune to it. What matters at the end of the day is that the automated systems are supervised and improved constantly to avoid false positives.

I can certainly understand why vendors point to their signatures being “rolled over” to other AV engines, but these same vendors should also take care so that they do not become the source of these “false positive rumors” in the first place.

UPDATE June 3rd, 2010: Reading Larry’s post over at securitywatch, it seems Kaspersky has reacted quickly and has removed their signature for the PandaCloudTestFile.exe file. Thanks Larry & Kaspersky!

VB Test Team

Virus Bulletin

Copyright © 2020 Virus Bulletin

The VB100 set-up
Diversity Test
Upcoming test changes
Products & results
Acronis Cyber Protect
Acronis True Image 2021
Adaware Antivirus Free
Adaware Antivirus Pro
Ad Spider
AhnLab V3 Endpoint Security
Arcabit AntiVirus
Avast Free Antivirus
AVG Internet Security
CMC Malware Detection and Defense
CORE Antivirus
Cynet 360
CyRadar Endpoint Detection and Response
Defenx Security Suite
Emsisoft Anti-Malware
eScan Internet Security Suite for Windows
ESTsecurity ALYac
Exosphere Endpoint Protection
Faronics Anti-Virus
FireEye Endpoint Security
Fortinet FortiClient
G DATA Antivirus
IKARUS anti.virus
Intego AV
K7 Total Security
PCProtect
Private Internet Antivirus
Qi-ANXIN Tianqing Endpoint Security Management System
Rising Enterprise Security Management System
Scanguard
SecureAge SecureAPlus Pro
Systweak Anti-virus
TACHYON Endpoint Security
TeamViewer Endpoint Protection
Tencent PC Manager
TotalAV
Total Defense Premium
TUXGUARD Endpoint Protection
United Endpoint Protector
VIPRE Endpoint Cloud Business
VirIT eXplorer PRO
Appendix 1: products not certified
Appendix 2: testing notes
Appendix 3: sample set sizes
Footnotes

Introduction

The VB100 certification scheme provides a stamp of quality and competence for anti-malware products that satisfy a minimum standard of detecting malicious executables that have recently been seen in the wild, while blocking few to no legitimate programs.

This report details the VB100 certification results of 41 of such products from 36 different vendors during November and December 2020.

The VB100 set-up

In the VB100 test, a copy of the product to be tested is installed on two platforms: Windows 10 and Windows 7. On each platform, and at three different times in the test, the product is asked to scan both the latest version of the WildList¹ and a selection of clean files taken from Virus Bulletin’s own set of files belonging to widely used legitimate software.

A legitimate file that is blocked at least once is considered a false positive, while a WildList file that isn’t blocked is considered a miss.

A product achieves a VB100 certification if:

• No more than 0.5% of WildList samples are missed

and

• No more than 0.01% of legitimate files are blocked

For full details, we refer to the VB100 methodology on the Virus Bulletin website: https://www.virusbulletin.com/testing/vb100/vb100-methodology/vb100-methodology-ver1-1/. This test used version 1.1 of the VB100 methodology.

Diversity Test

The malware part of the VB100 certification uses the WildList, a regularly updated list of extremely well-vetted malware samples, guaranteed to have been spotted in the wild multiple times. This makes them very suitable for a certification test like VB100.

The ‘Diversity Test’ looks at products’ detection of another set of recent malware samples, to acknowledge the fact that products detect malware samples beyond a standard set of samples, and provides a measure of that detection.

Upcoming test changes

As part of planned updates to the VB100 test in 2021, we will be retiring testing on the legacy Windows 7 platform, effective from January 2021.

We are introducing this change because the relevance of Windows 7 has diminished greatly in recent years, and the platform reaches end-of-life in January 2020 – as a consequence of which, a growing number of tested products either lack support for this platform, or struggle to perform properly on the legacy operating system. We expect the retirement to have negligible impact on the relevance of the VB100 reports.

Products & results

Products were allowed to download updates during the course of the test. The version numbers listed in the results that follows refer to those at the start of the test.

Acronis Cyber Protect

Acronis True Image 2021

Ahnlab V3 Internet Security 8.0

Adaware Antivirus Free

Adaware Antivirus Pro

Ad Spider

AhnLab V3 Endpoint Security

Arcabit AntiVirus

Avast Free Antivirus

AVG Internet Security

CMC Malware Detection and Defense

CORE Antivirus

(see notes in Appendix 2)

Cynet 360

CyRadar Endpoint Detection and Response

Defenx Security Suite

Emsisoft Anti-Malware

eScan Internet Security Suite for Windows

ESTsecurity ALYac

Exosphere Endpoint Protection

Faronics Anti-Virus

FireEye Endpoint Security

Fortinet FortiClient

G DATA Antivirus

IKARUS anti.virus

Intego AV

K7 Total Security

PCProtect

Private Internet Antivirus

Ahnlab V3 Report False Positive Blood

Qi-ANXIN Tianqing Endpoint Security Management System

Rising Enterprise Security Management System

Scanguard

SecureAge SecureAPlus Pro

Systweak Anti-virus

(see notes in Appendix 2)

TACHYON Endpoint Security

TeamViewer Endpoint Protection

Tencent PC Manager

TotalAV

Total Defense Premium

TUXGUARD Endpoint Protection

(see notes in Appendix 2)

United Endpoint Protector

(see notes in Appendix 2)

VIPRE Endpoint Cloud Business

VirIT eXplorer PRO

Appendix 1: products not certified

All products achieved VB100 certification in this test.

Appendix 2: testing notes

• CORE Antivirus gained VB100 certification based on measurements taken on Windows 10 only.
• Systweak Anti-virus gained VB100 certification based on measurements taken on Windows 10 only.
• Tabidus Technology’s United Endpoint Protector gained VB100 certification based on measurements taken on Windows 10 only.
• TUXGUARD Endpoint Protection gained VB100 certification based on measurements taken on Windows 10 only and, due to a technical failure, the Diversity Test results for this product were invalidated.

Appendix 3: sample set sizes

Ahnlab V3 Report False Positive Test

The Certification Set contained 1,401 malicious samples. The set of clean samples used for the false positive test contained 100,000 files, of which 29,168 were portable executable (PE) files. The set used for the Diversity Test contained 1000 malicious samples.

Footnotes

Ahnlabs V3 Windows 10

¹ The WildList is an extremely well-vetted set of malware recently observed in the wild by researchers: http://www.wildlist.org/.