Queue mechanism on Metadefender Core v4

The below article will help understand the queue mechanism in MetaDefender Core V4

The main topics that are covered are the following:

  • Queue mechanism in general

  • Queue size for requests

  • Limit of concurrent connections

  • Max file size allowed

Queue mechanism in general

The files are stored on Node side, Core simply proxies them to the Node. Each item in the queue is handled/managed by a workflow (a thread) on Core side.
If there is a free engine slot then Core instruct the Node to scan a specific item in the queue by the engine. Core starts processing the queue in the first come first served basis, however this doesn't determine the end time of the processing.
Node uses max 1/4 of the max queue size for archive processing (this applies to all archives processed at a time, not to each one ). This means if you send only one file into the queue which is an archive, the extraction fills the queue only up to 1/4 of the queue size, to leave room for further files, but provide parallel processing also.

Queue size for requests

There are no separate queues for Core and Node. Node is the one that handles the queue.
The default queue size is set to 500. To increased/decreased this value please refer to the following KB: How can I configure the maximum queue size in MetaDefender Core v4 ?
The number of items in the queue can be extracted from the results of the REST API call /stats/nodes.
There is not significant usage of memory by the items found in the queue.
We have one thread per queue item on Core side (this was tested with 20k parallel threads).

Limit of concurrent connections

The limit of concurrent connections isbased on OS limitations:
*Windows has a 4K limit
*Linux has a 65K limit

The practical amount of concurrent conenctions is at about 1K.
There is not limitation of concurrent connections set on the license.

Max file size allowed

Limited by the available disk size of the Node.