Sample quality for the Malware Protection Test

The test set for Malware Protection Test  consisted of about 38,000 samples. As we only use samples that have been analysed by our own in-house automated sandboxes, the quality of our sets is very high. Unlike some other testers, we only use malware in our tests, and do not include PUAs or other controversial software. What is malicious and what is “potentially unwanted” is sometimes debatable. We welcome feedback from vendors; however, the decision as to whether something can or cannot be classified as malware is ultimately up to us, even if our decisions may sometimes be regarded as imperfect.

In selecting samples for the test set, we focus on current, relevant malware and not on exotic or extinct samples. Therefore, high detection/protection rates are to be expected from the participating products.

As usual, AV-Comparatives gives each participating vendor the possibility to review their missed samples AFTER the test using AV-Comparatives’ Feedback System. We continuously verify our test set during the open feedback time. During the feedback process we removed a number of samples even before any vendors had disputed them.

Of 21 vendors tested in this Malware Protection Test, 9 vendors provided feedback. Please note that many of the tested products use a third-party engine, and so they may rely on the engine provider to take care of disputes. We received 38 disputes (an average of just under 2 per participating vendor, median value 1), of which 25 were accepted. All the files disputed by vendors in this test performed suspicious activities (they can be seen as greyware/risky). None of the files were corrupted or clearly benign. Nevertheless, the disputes had no impact on the rankings/awards for this test. Regarding our dispute process in general, we have found that sometimes a vendor will try to dispute a sample, claiming that it is clean, even though it is clearly malicious. In such cases we reject the vendor’s claim, having investigated each individual disputed sample carefully.

Services such as VirusTotal have a valuable role to play, but they cannot replace real testing, as they usually rely on static command line scanners (often without a connection to the vendor’s cloud services). In other words, this means, that a sample not detected by an online multiscanner service, might be detected by the same vendor’s endpoint security product. Equally, detection of a sample by an online scanning engine does not mean that the sample is necessarily suitable for testing, as it cannot guarantee that a file fulfils our criteria for being malicious, or that it will run on a particular test operating system.

Below we have shared the hashes of 20 samples that were disputed in the past (some of those disputes were accepted, some not); users are invited to vote or comment on VirusTotal as to whether they think that the samples are malicious or not (as stated by VirusTotal, please vote “only if you have good evidence for it”).

015aab88e4f4ebd560f1f6c10f9fa889f88daf51d27650c26566ed5924dae3a0

0176f8c0c5113f8c506d56e3814e74b1b2d7854ffd4652d70b47a87302890048

0217232579a244460766fc2b4d8d315b1ea25634e91a50f06ba0549fb2da789d

125effdd6d9a7bae637f912485091a3704931c0b43720d49925448a958d71f70

20da128045c809dd26829bd118b383ca5d2ee970fc4b5e49905ff5ac421be384

22337b12c44cccd90aa9ce25caaeb37ef09950ed0e31f0253b9ecf4f3d4cc722

2289d85b3ff187881959b1d86589b1917926c8b4133c2aae6c2266aa09c97c42

2c9af35c38a0d3f5a352591589955d82b26a73bc59649f25b5dc3aa7769a05a8

3330faf0f57e1e21c32f3b59d9bbf9cc13b33f4558594603cf41686b86e56bce

3df8160a00821b24870762f7aa4a88d10a760ba8b3db4970af79abfb6f05ee7d

4149c2ddfd2deb28956ea0a35852ae73138e6c9210e4a762dfbcb8514a702f98

42140084e3e4d593da02bc83095d4b92ed571db7d8d84f39d3d0d92bf8aa314b

4842c5403372ead4fd28a26f2e1dfc139541e71bcf574e62c7c18b9cfc406674

70c1bd1125e7e9cb11e66e66ac075912cb3f9b808a9cb18acac388e42d1d4851

97c52f29819c5e56aa2bc69abca61cd2908467c4b13890572b4602c63d2666ec

aa7b5a839022552faef51c662fbbeee429b21060e1653502a6bdf4e0d81cefeb

b9200de9e177b2c332bb2027d2b92acb6a0aaa02429d299790b463f9578bfe0b

ccb800062567c3adcdf917b29500625bc2f266969a59b7d1d7f98694a756589d

d0848a012e6fd4d4b202541aefef7e66d7dbcc75e059e274836841b1197f8ca6

d98d193ca3bab3d6ba91d14e714927ee3df6ed5c9771f88524713436807f7ba3