NginX binary with modsecurity consume all memory and crashes

[intelbg]

Hello again and sorry for the inconvenience,
I am using libmodsecurity V3 (nginx_refactoring branch) + nginx connector with CRSv3 and when I put this binary of nginx in production with around 150 -200 vhosts, after I make even service nginx configtest (or restart) the server is on 150-200 load immediately (strace show that there are a lot of wait4 sys calls) and the process takes around 20-30 seconds. I tried to remove the CRS rules folder to see if the problem is in the number of lines/files or in the binary but no big difference. With binary without modsec everything is ok. Also, with the modsec nginx binary there is a memory allocation problems on restart and nginx does not want to start. Removed all limits on the operating systems including cgroups, ulimit, sysctl etc but no effect. If I back the binary without modsec everything is fine.

Is here anyone that faced such a problems and is there a chance to:

1. The binary to be not compiled with the right options, by the right way or something other?
2. The nginx to trigger all the modsec functions on restart even when mod security on is not present on the vhosts?
3. I tried to optimize everything that I found but it's the same.

The machines are with 32GB of ram, 12 or 16 cpu cores and normal HDDs. On SSD it's a little bit better.

[met3or]

Hi intelbg,

I've so far been unable to replicate the issue that you're experiencing with my setup, I'm compiling the v3/master branch of ModSecurity (with a few tweaks to work-around SELinux but nothing major) and thrown an arbitrary 1000 vhosts at nginx, reloaded & restarted fine albeit slow.

The machine being used is physical, 32G RAM with 12 core cpu aswell as SSD for disk.

For you to reach the load averages you mention are you performing any specific tasks or are the vhosts particularly active?

[intelbg]

Thank you for your reply! I didn't perform any specific tasks - only yum update of the nginx and service nginx configtest or restart. The vhosts are particularly active, but with nginx without modsec there is not an issue. Maybe for every request by some way there is an overhead even if modsecurity is not on for the ghosts? Some calculations in the nginx binary?

[met3or]

Hi @intelbg ,

The domains I've added aren't in use so perhaps that is the reason for being unable to replicate, I will attempt to run some tests against these hosts and get back to you soon if I can replicate it or not.

[intelbg]

Thank you again. Meanwhile, is it possible the problem to come from different versions of gcc, g++, make, automate on the compile server and the production server to which transfer the binary later? Or in the parameters of the machine of the compilation server and the production servers (production servers are 10 times faster than this for the compilation)? But I compiled before that nginx in the same environment and without modsec no problems.

[met3or]

Hi @intelbg - Interestingly, my server load has not massively increased but it has exhausted it's memory and I'm getting a -

fork: Cannot allocate memory error from the system.

I will look at this a bit more now and see if I can see any obvious culprits.

[intelbg]

Did you found anything unexpected?

[met3or]

Hi @intelbg - I've not found the reason for this yet, but with further tests I have found an nginx out-of-memory event in the system messages. I will be trying to trace the cause of this, but from my preliminary checks It seems to be due to the large quantity of configs.

[intelbg]

But why this affects the system loads as these configs has not modsecurity enabled respectively they does not invoke modsecurity functions?

[sethinsd]

I am having this issue currently. we have about 1500 vhosts. Works fine as long as I don't send any big traffic at it, but with a moderate 20requests per second or so this is what the log ends up looking like the following (I've fixed the PCRE limits exceeded issue, but still having it crash out

2017/09/13 12:24:50 [alert] 1556#1556: worker process 3665 exited on signal 11 (core dumped)
2017/09/13 12:24:50 [alert] 1556#1556: worker process 3667 exited on signal 11 (core dumped)
2017/09/13 12:24:51 [alert] 1556#1556: worker process 3661 exited on signal 11 (core dumped)
2017/09/13 12:24:52 [alert] 1556#1556: worker process 3671 exited on signal 11 (core dumped)
2017/09/13 12:24:53 [alert] 1556#1556: worker process 3669 exited on signal 11 (core dumped)
2017/09/13 12:24:53 [alert] 1556#1556: worker process 3673 exited on signal 11 (core dumped)
2017/09/13 12:24:54 [alert] 1556#1556: worker process 3677 exited on signal 11 (core dumped)
2017/09/13 12:24:55 [alert] 1556#1556: worker process 3675 exited on signal 11 (core dumped)
2017/09/13 12:24:56 [alert] 1556#1556: worker process 3681 exited on signal 11 (core dumped)
2017/09/13 12:24:56 [alert] 1556#1556: worker process 3685 exited on signal 11 (core dumped)

[met3or]

Hi @sethinsd - For the error you're seeing here:

2017/09/13 12:24:48 [error] 3659#3659: [client 172.31.19.26] ModSecurity: Rule 7f4136252c70 [id "-"][file "/etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"][line "272"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "ip-172-31-8-158"] [uri "/"] [unique_id "AcA8AIAcAcANtcAcAcAc8cAc"]

You may need to configure the following settings:

SecPcreMatchLimit 500000
SecPcreMatchLimitRecursion 500000

To a value of your requirements, the reason this has a limit is to protect from DoS type attacks.

[sethinsd]

ah yes I fixed that one already. But the other issue is persisting with it crashing under load

[intelbg]

@sethinsd how did you fixed it? Can you please share all of your experience? I am not sure why until now nobody has reported these problems as they are serious and now I hope as this number of people is increasing to find a solution. If I can provide another information that can be useful just let me know.

[sethinsd]

I only fixed that one PCRE issue - the crashes and memory with fork problem persists and I cannot enable modsecurity in my production environment right now. I think it's likely related to both of us having so many host configuration files. I have about 1600 server {} declarations if not more.

[sethinsd]

One thing we are noticing in our setup even with modsec disabled is the active connections climb to unreal levels over a few days until we restart nginx non gracefully. Like upwards of 60k when really we only have 100-200 active connections. So something is seriously wrong with the nginx compile.

[intelbg]

Any progress found?