Nginx 1.4.1 Modsecurity 2.7.4 CRS 2.2.7 Standard configuration uses up all memory and it then killed

[rcbarnett-zz]

MODSEC-402: When I try to do a login to our web application with ModSecurity set for DetectionOnly Nginx uses up all the memory and I see the following via dmesg:-

nginx invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
nginx cpuset=/ mems_allowed=0
Pid: 1535, comm: nginx Not tainted 2.6.32-358.6.2.el6.i686 #1
Call Trace:
[] ? dump_header+0x84/0x190
[] ? oom_kill_process+0x68/0x280
[] ? oom_badness+0x92/0xf0
[] ? out_of_memory+0xc8/0x1e0
[] ? __alloc_pages_nodemask+0x7fd/0x810
[] ? handle_pte_fault+0xa6f/0xdf0
[] ? kmap_atomic_prot+0x120/0x150
[] ? handle_mm_fault+0x131/0x1d0
[] ? __do_page_fault+0xfb/0x430
[] ? __rcu_process_callbacks+0x212/0x2f0
[] ? rcu_process_callbacks+0x35/0x40
[] ? __do_softirq+0xae/0x1a0
[] ? do_page_fault+0x2a/0x90
[] ? smp_apic_timer_interrupt+0x53/0x90
[] ? do_page_fault+0x0/0x90
[] ? error_code+0x73/0x78
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 30
HighMem per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 30
active_anon:305984 inactive_anon:158195 isolated_anon:32
active_file:19 inactive_file:34 isolated_file:0
unevictable:0 dirty:0 writeback:2577 unstable:0
free:12163 slab_reclaimable:823 slab_unreclaimable:2265
mapped:10 shmem:0 pagetables:1672 bounce:0
DMA free:7736kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 863 2016 2016
Normal free:40500kB min:3724kB low:4652kB high:5584kB active_anon:338012kB inactive_anon:337864kB active_file:68kB inactive_file:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:883912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3292kB slab_unreclaimable:9060kB kernel_stack:928kB pagetables:1412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:21 all_unreclaimable? no
lowmem_reserve[]: 0 0 9222 9222
HighMem free:416kB min:512kB low:1752kB high:2996kB active_anon:885924kB inactive_anon:294916kB active_file:8kB inactive_file:76kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:1180532kB mlocked:0kB dirty:0kB writeback:10308kB mapped:68kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:5276kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:18 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 6_4kB 4_8kB 4_16kB 2_32kB 2_64kB 2_128kB 2_256kB 1_512kB 2_1024kB 2_2048kB 0_4096kB = 7736kB
Normal: 213_4kB 280_8kB 118_16kB 36_32kB 11_64kB 5_128kB 3_256kB 3_512kB 4_1024kB 1_2048kB 6_4096kB = 40500kB
HighMem: 2_4kB 3_8kB 0_16kB 2_32kB 1_64kB 0_128kB 1_256kB 0_512kB 0_1024kB 0_2048kB 0*4096kB = 416kB
4581 total pagecache pages
4524 pages in swap cache
Swap cache stats: add 1045423, delete 1040899, find 3325/3893
Free swap = 0kB
Total swap = 1048568kB
524272 pages RAM
297474 pages HighMem
39613 pages reserved
1043 pages shared
471123 pages non-shared
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[ 481] 0 481 623 1 1 -17 -1000 udevd
[ 1073] 0 1073 3233 7 1 -17 -1000 auditd
[ 1090] 0 1090 9267 55 0 0 0 rsyslogd
[ 1113] 32 1113 644 14 1 0 0 rpcbind
[ 1131] 29 1131 710 1 1 0 0 rpc.statd
[ 1159] 0 1159 653 1 0 0 0 rpc.idmapd
[ 1180] 0 1180 737 20 0 0 0 lldpad
[ 1204] 0 1204 547 4 0 0 0 fcoemon
[ 1216] 81 1216 750 1 0 0 0 dbus-daemon
[ 1253] 0 1253 2367 8 0 -17 -1000 sshd
[ 1261] 38 1261 1285 29 0 0 0 ntpd
[ 1270] 0 1270 5380 6 0 0 0 bacula-fd
[ 1284] 93 1284 3666 1 0 0 0 exim
[ 1293] 496 1293 1507 8 0 0 0 nrpe
[ 1315] 0 1315 502 1 0 0 0 mingetty
[ 1317] 0 1317 502 1 0 0 0 mingetty
[ 1319] 0 1319 502 1 0 0 0 mingetty
[ 1321] 0 1321 502 1 1 0 0 mingetty
[ 1323] 0 1323 502 1 0 0 0 mingetty
[ 1326] 0 1326 622 1 0 -17 -1000 udevd
[ 1327] 0 1327 622 1 1 -17 -1000 udevd
[ 1328] 0 1328 502 1 1 0 0 mingetty
[ 1330] 0 1330 3321 40 1 0 0 sshd
[ 1332] 0 1332 1314 52 1 0 0 bash
[ 1513] 0 1513 4718 26 1 0 0 nginx
[ 1535] 48 1535 722420 460390 1 0 0 nginx
Out of memory: Kill process 1535 (nginx) score 965 or sacrifice child
Killed process 1535, UID 48, (nginx) total-vm:2889680kB, anon-rss:1841500kB, file-rss:60kB

[rcbarnett-zz]

Original reporter: amismb

[rcbarnett-zz]

bpinto: ModSecurity will at least duplicate the size of allocated memory for each nginx process you have (if you enable only base_rules).

OOM killer works :

• Processes that have the PF_SWAPOFF flag set will be killed first
• Processes which fork a lot of child processes are next in line
• Kill off niced processes, since they are typically less important
• Superuser processes are usually more important, so try to avoid killing those

Could you please try two tests ?

1 - Reduce the number of nginx workers you are starting
2 - Change the user to root and let me know if it continue killing nginx process.

Thanks

Breno

[rcbarnett-zz]

bpinto: Another question. How much free memory you have before starting nginx+modsecurity ?

Thanks

[ifastnet]

I can reproduce this exactly on centos 6.4 with the owsap core rule set in detection only mode, server with 64GB of ram in seconds goes to swap after nginx eats all the rams available in the system (have tested with trunk mod_sec , and nginx version 1.4.1 stable )

[zimmerle]

No longer a concern in libModSecurity. Marking it as won't fix for 2.x. Further information about libModSecurity available here:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master