Hi all,
I don't know if this is the right forum to ask this question but here it is!
The context is a c++ application implementing a classic producer/consumer pattern using two threads and the std::queue. The implementation comes from one of many thread-safe examples available on the web. The queue has no control on maximum size, that is the producer can push as many elements as it wants.
The producer receives real-time sample buffers from an optical network interface (something like 50/100M samples per second) and pushes them in the queue. Buffers are 40k bytes size.
The consumer extracts buffers from the queue and writes them on SSD disk.
Running the application, "top" shows that the kernel buffers start to eat the available memory (16GB) until there's about 0.5GB available. The situation then remains stable, with the producer pushing millions of samples in the queue and the consumer slowly writing on the disk.
After all samples (for 5 minutes of streaming something like 120GB at 100M samples per second) are received, the consumer has yet to write to disk much more GB than the ones available in memory.
"top" also shows that no disk swapping is performed.
When the consumer finishes its job several minutes after, the file contaning all samples has correct size.
The question is: how can this happen? I thought that after allocating all memory for kernel buffers, the OS began to swap or the kernel killed the application because of out of memory. Where is this huge queue memorized while the consumer slowly writes its elements to file? I can't explain this...maybe some kernel expert can help me understand.
Thank you very much!
Regards,
Elvio
I don't know if this is the right forum to ask this question but here it is!
The context is a c++ application implementing a classic producer/consumer pattern using two threads and the std::queue. The implementation comes from one of many thread-safe examples available on the web. The queue has no control on maximum size, that is the producer can push as many elements as it wants.
The producer receives real-time sample buffers from an optical network interface (something like 50/100M samples per second) and pushes them in the queue. Buffers are 40k bytes size.
The consumer extracts buffers from the queue and writes them on SSD disk.
Running the application, "top" shows that the kernel buffers start to eat the available memory (16GB) until there's about 0.5GB available. The situation then remains stable, with the producer pushing millions of samples in the queue and the consumer slowly writing on the disk.
After all samples (for 5 minutes of streaming something like 120GB at 100M samples per second) are received, the consumer has yet to write to disk much more GB than the ones available in memory.
"top" also shows that no disk swapping is performed.
When the consumer finishes its job several minutes after, the file contaning all samples has correct size.
The question is: how can this happen? I thought that after allocating all memory for kernel buffers, the OS began to swap or the kernel killed the application because of out of memory. Where is this huge queue memorized while the consumer slowly writes its elements to file? I can't explain this...maybe some kernel expert can help me understand.
Thank you very much!
Regards,
Elvio
