Troubles with memory leak and OOM-killer
Sometimes you can find that services on your server are shut down without any error or warning message. Only one message you can find in main logs is something like this:
xxx invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Where "xxx" may by any process - Java, Apache, MySQL, Postgesql etc.
Use of memory is got critical and OOM-Killer (System kernel's feature) simply killed the process which used too much memory and had long execution time. It may be caused by bugs in software, bugs in PHP-code, memory leaks in software or simply heavy load on the server.
What to do?
Try to find software which causes this or bugs in your code. It may be difficult task and there is no typical solution for this.
When You are sure that everything is OK or You cannot find cause of this trouble You may try following:
1. Check what memory allocation strategy is in use on your system.
sudo cat /proc/sys/vm/overcommit_memory
In most cases You will get "0" in output. What does it mean?
It is default memory allocation strategy. For today we have three of them:
0 - OVERCOMMIT_GUESS - Heuristic approach to the allocation of memory. This strategy gives as much memory to the process as it need. The system will reject requests only, which could not be satisfied. Swap is mostly unused.
1 - OVERCOMMIT_ALWAYS - This strategy means that kernel will always satisfy all requests for memory allocation. In practice almost the same as previous. Is used with very unstable software.
2 - OVERCOMMIT_NEVER - In this case a total amount of memory will be total_swap + total_ram * overcommit_ratio / 100. By default overcommit_ratio is set to 50 and is unused.
So, We need the las strategy to resolve our problem.
2. Set new memory allocation strategy.
sudo echo 2 > /proc/sys/vm/overcommit_memory sudo echo 95 > /proc/sys/vm/overcommit_ratio
Setting overcommit_ratio to 95 will allow us to use the most amount of RAM and SWAP. System will reserve 3% of memory for root processes.
Now we can test the system with new allocation strategy. If everything is OK go to step 3. If not - try to investigate software You have.
3. Change system configuration
To use chosen strategy after reboot You should add following lines to the sysctl ( /etc/sysctl.conf in most of Linux OSs):
vm.overcommit_memory = 2 vm.overcommit_ratio = 95
From now we have much more stable system!