Odi's astoundingly incomplete notes

I have been running Oracle 11.2 on x86_64 linux with no swap and transparent hugepage support. The system has 6GB RAM and vm.overcommit_memory=0. Oracle is configured to using automatic memory management with MEMORY_TARGET = 4000M. There is nothing else on the system. So 6GB vs 4GB should be plenty, right? Wrong.

While Oracle would run fine most of the time, sometimes connect fails with:

ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory

If you go to the server and check the memory usage, all seems fine however:

~ # free -m
             total       used       free     shared    buffers     cached
Mem:          5972       5714        257          0         11       4078
-/+ buffers/cache:       1624       4347
Swap:            0          0          0

After adding a little swap file (1 GB) the problem goes away. With vm.swappiness=40 the swap usage remains stable at around 250MB. So what's going on here?

Obviously free's output of 1.6GB used memory is wrong. The 4GB SGA lies in the shm filesystem, and obviously doesn't show up here. Why? Because in Linux shm is mapped to the filesystem cache*. I guess this is a trick so that reads from shm do not get copied to cache. So to get the real memory usage you have to add the shm usage to free's output. And that sum is 5.6GB. Which is admittedly a bit much for a 6GB swapless system.

So to fix this, I reduced the SGA to  3.5GB. This results in a bit less than 5GB used memory. And now it works fine without swap!

*) This is a nice example why the linux ate my ram website is not right, when it claims
Disk cache [...] NEVER EVER takes memory away from [applications]. Another example is locked pages in disk cache.

posted on 2011-11-25 13:44 UTC in Code | 2 comments | permalink