Queue mechanism on Metadefender Core v4

The below article will help understand the queue mechanism in MetaDefender Core V4

The main topics that are covered are the following:

  • Queue mechanism in general

  • Queue size for requests

  • Limit of concurrent connections

  • Max file size allowed

Queue mechanism in general

The files are stored on the Node side, Core simply proxies them to the Node. Each item in the queue is handled/managed by a workflow (a thread) on the Core side.
If there is a free engine slot then Core instructs the Node to scan a specific item in the queue by the engine. The core starts processing the queue on the "first come first served" basis, however, this doesn't determine the end time of the processing.
Node uses max 1/4 of the max queue size for archive processing (this applies to all archives processed at a time, not to each one ). This means if you send only one file into the queue which is an archive, the extraction fills the queue only up to 1/4 of the queue size, to leave room for further files, but provide parallel processing also.

Queue size for requests

There are no separate queues for Core and Node. Node is the one that handles the queue.
The default queue size is set to 500. To increased/decreased this value please refer to the following KB: How can I configure the maximum queue size in MetaDefender Core v4?
The number of items in the queue can be extracted from the results of the REST API call /stat/nodes.
There is not significant usage of memory by the items found in the queue.
We have one thread per queue item on the Core side (this was tested with 20k parallel threads).

Limit of concurrent connections

The limit of concurrent connections is based on OS limitations:
*Windows has a 4K limit
*Linux has a 65K limit

The practical amount of concurrent connections is about 1K.
There is no limitation of concurrent connections set on the license.

Max file size allowed

Limited by the available disk size of the Node.

This article applies to MetaDefender Core v4
This article was last updated on 2019-10-06
VM