Segfaults in kern.log

[GNU-Plus-Windows-User]

Describe the bug

ModSec has segfaults which causes CrowdSec Nginx Bouncer to stop working.

Logs and dumps

Feb  2 12:10:43 redacted kernel: [991891.066133] nginx[2386946]: segfault at 8 ip 00007f31d846001e sp 00007ffc5832da30 error 4 in libmodsecurity.so.3.0.8[7f31d8360000+114000]
Feb  2 12:10:43 redacted kernel: [991891.066146] Code: 83 c4 08 4c 89 e0 5d 41 5c c3 66 90 f3 0f 1e fa 41 57 41 56 41 55 4c 63 ee 41 54 49 89 fc 55 53 48 89 d3 48 81 ec b8 00 00 00 <4c> 8b 42 08 64 48 8b 04 25 28 00 00 00 48 89 84 24 a8 00 00 00 31
Feb  2 12:10:43 redacted kernel: [991891.109427] nginx[2422833]: segfault at 8 ip 00007f31d846001e sp 00007ffc5832da30 error 4 in libmodsecurity.so.3.0.8[7f31d8360000+114000]
Feb  2 12:10:43 redacted kernel: [991891.109441] Code: 83 c4 08 4c 89 e0 5d 41 5c c3 66 90 f3 0f 1e fa 41 57 41 56 41 55 4c 63 ee 41 54 49 89 fc 55 53 48 89 d3 48 81 ec b8 00 00 00 <4c> 8b 42 08 64 48 8b 04 25 28 00 00 00 48 89 84 24 a8 00 00 00 31

To Reproduce

Steps to reproduce the behavior:

1. Install ModSec 3.0.8 with CRS
2. Install Lua and CrowdSec Nginx Bouncer
3. wait for segfaults to show up in logs

Expected behavior

Segfaults should not be showing up in logs

Server

• ModSec v3.0.8
• Nginx 1.22.1
• Ubuntu 22.04 Proxmox LXC container
• Lua 5.1.5
• CrowdSec Nginx Bouncer

Rule Set

• CRS 3.3.4 (Sep 21, 2022)

Additional context

CrowdSec Nginx Bouncer will stop receiving decisions (IP Bans) if seg faults occur. I made a ticket on the CrowdSec Discord regarding the issue, they provided a fix for that but, if a segfault happens around the same time the bouncer is querying the CrowdSec Agent, the CrowdSec Bouncer will stop working.

Restarting Nginx temporarily fixes the issue

[martinhsv]

Hello @GNU-Plus-Windows-User ,

"Lua 5.1.5"

ModSecurity documentation states: "Note : ModSecurity v3 is compatible with Lua 5.2+."

[GNU-Plus-Windows-User]

@martinhsv I've updated Lua to 5.2 and I'm still seeing segfaults in my logs, I've also tried disabling Lua and CrowdSec Nginx Bouncer altogether but there is still segfaults.

[martinhsv]

Can you produce a core dump, and at least supply the backtrace?

[GNU-Plus-Windows-User]

@martinhsv I've generated a core dump, but I couldn't figure out how to get a backtrace.
Is there anywhere I can safely share the coredump, I don't want to post it publicly on the internet?

[GNU-Plus-Windows-User]

@martinhsv I've managed to generate a backtrace but I don't know if I did it correctly

Core was generated by `nginx: worker process                           '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f65815f3a4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=<optimized out>) at /usr/include/c++/11/bits/basic_string.h:927
927           { return _M_string_length; }

I still have that coredump if you want it, but I don't want to post it publicly.

[martinhsv]

Hi @GNU-Plus-Windows-User ,

That looks like the start of a backtrace. Was there more output? Usually a backtrace will show that #0 and then move up the call stack with entries for #1, #2, etc.

[GNU-Plus-Windows-User]

@martinhsv Sorry about that, I didn't read the documentation correctly.
This should be what you're after

Core was generated by `nginx: worker process                           '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f65815f3a4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=<optimized out>) at /usr/include/c++/11/bits/basic_string.h:927
927           { return _M_string_length; }
[Current thread is 1 (Thread 0x7f65816b3740 (LWP 1342))]
(gdb) backtrace
#0  0x00007f65815f3a4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=<optimized out>) at /usr/include/c++/11/bits/basic_string.h:927
#1  modsecurity::utils::string::limitTo (amount=amount@entry=200, str=<error reading variable: Cannot access memory at address 0x8>) at utils/string.cc:100
#2  0x00007f658158ea88 in modsecurity::RuleMessage::_details[abi:cxx11](modsecurity::RuleMessage const*) (rm=0x55ce45e32160) at /usr/include/c++/11/bits/shared_ptr_base.h:1295
#3  0x00007f658158f61b in modsecurity::RuleMessage::log[abi:cxx11](modsecurity::RuleMessage const*, int, int) (rm=0x55ce45e32160, props=props@entry=0, code=code@entry=-1) at rule_message.cc:88
#4  0x00007f6581577abf in modsecurity::RuleMessage::log[abi:cxx11](modsecurity::RuleMessage const*, int) (props=0, rm=<optimized out>) at ../headers/modsecurity/rule_message.h:177
#5  modsecurity::RuleMessage::log[abi:cxx11]() (this=<optimized out>) at ../headers/modsecurity/rule_message.h:162
#6  modsecurity::ModSecurity::serverLog (this=0x55ce37ac6d00, data=0x55ce463bedb0, rm=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...}) at modsecurity.cc:205
#7  0x00007f658155e06c in modsecurity::Transaction::serverLog (this=this@entry=0x55ce4644c660, rm=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...}) at transaction.cc:1858
#8  0x00007f6581584cbd in modsecurity::RuleWithActions::performLogging (this=this@entry=0x55ce397f1000, trans=trans@entry=0x55ce4644c660,
    ruleMessage=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...}, lastLog=lastLog@entry=true, chainedParentNull=chainedParentNull@entry=true) at rule_with_actions.cc:521
#9  0x00007f658158b030 in modsecurity::RuleWithOperator::evaluate (this=0x55ce397f1000, trans=<optimized out>, ruleMessage=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...})
    at rule_with_operator.cc:372
#10 0x00007f65815851d5 in modsecurity::RuleWithActions::evaluate (this=0x55ce397f1000, transaction=0x55ce4644c660) at rule_with_actions.cc:177
#11 0x00007f658157a4bd in modsecurity::RulesSet::evaluate (this=<optimized out>, phase=phase@entry=5, t=t@entry=0x55ce4644c660) at rules_set.cc:210
#12 0x00007f6581562c70 in modsecurity::Transaction::processResponseBody (this=0x55ce4644c660) at transaction.cc:1264
#13 0x00007f65816a5cea in ngx_http_modsecurity_body_filter (r=0x55ce46424f50, in=0x55ce46425ed0) at ../ModSecurity-nginx/src/ngx_http_modsecurity_body_filter.c:161
#14 0x000055ce378e16d1 in ngx_output_chain ()
#15 0x000055ce37959cf7 in ?? ()
#16 0x000055ce3791213e in ngx_http_output_filter ()
#17 0x000055ce3799785d in ?? ()
#18 0x000055ce37923f82 in ngx_http_finalize_request ()
#19 0x000055ce3791726f in ngx_http_core_rewrite_phase ()
#20 0x000055ce3790fd9d in ngx_http_core_run_phases ()
#21 0x000055ce37917a6e in ngx_http_internal_redirect ()
#22 0x000055ce37921384 in ngx_http_special_response_handler ()
#23 0x000055ce37923f82 in ngx_http_finalize_request ()
#24 0x000055ce37924735 in ngx_http_process_request_uri ()
#25 0x000055ce379259c5 in ?? ()
#26 0x000055ce379263e9 in ?? ()
#27 0x000055ce379075ab in ?? ()
#28 0x000055ce378fde99 in ngx_process_events_and_timers ()
#29 0x000055ce3790429d in ?? ()
#30 0x000055ce378f9e08 in ngx_spawn_process ()
#31 0x000055ce378fb348 in ?? ()
#32 0x000055ce3790303b in ngx_master_process_cycle ()
#33 0x000055ce378d5c37 in main ()

[martinhsv]

@GNU-Plus-Windows-User

Yes, that's it. Thanks.

One point to clarify: You mentioned that you are using ModSecurity v3.0.8. Is that the official, published version from last September? Or is that a build from more recent v3/master (which would also report its version as v3.0.8)?

[GNU-Plus-Windows-User]

@martinhsv I followed this guide to compile ModSecurity for Nginx https://www.nginx.com/blog/compiling-and-installing-modsecurity-for-open-source-nginx/
I assume this guide uses the latest published release but I'm not very knowledgeable with github

[martinhsv]

Thanks. That implies that you were using a later v3/master.

There were some recent changes to the implicated function -- although I don't immediately see how those changes could have caused a segfault.

If you are willing to do a couple of experiments, it might be interesting to know if:

During step 3 ("Download and Compile the ModSecurity 3.0 Source Code") ...

• during substep 1, use this git clone instead: git clone -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
• during substep 2, immediately after 'cd ModSecurity' add this git checkout v3.0.8

Repeat the experiment above, but instead of the 'git checkout v3.0.8' specified above, try it with each of these:

• git checkout 62ec4edc4258971deec677b4f5e6bda188d27f26
• git checkout dabf79eec251bfcf2c24c8b19295d40ef16ce2be

Under which (if any) of those three tests the problem still occurs, could be very useful information.

[GNU-Plus-Windows-User]

@martinhsv I've recompiled ModSec using git checkout v3.0.8, I'll test all 3 and let you know if anything changes.

[martinhsv]

By the way, what compiler and version are you using?

[GNU-Plus-Windows-User]

@martinhsv I just finished all 3 tests, all of them had segfaults.
Regarding what compiler I'm using and version, I'm using whatever the Nginx guide uses along with the latest packages for Ubuntu 22.04.
The installed version of the packages listed on the Nginx guide is

• apt-utils: 2.4.8
• autoconf: 2.71
• automake: 1.16.5
• build-essentials: 12.9
• git: 2.34.1
• libcurl4-openssl-dev: 7.81.0
• libgeoip-dev: 1.6.12
• liblmdb-dev: 0.9.24
• libpcre++-dev: 0.95
• libtool: 2.4.6
• libxml2-dev: 2.9.13
• libyajl-dev: 2.1.0
• pkgconf: 1.8.0
• wget: 1.21.2
• zlib1g-dev: 1.2.11

[martinhsv]

Hi @GNU-Plus-Windows-User ,

In your original report, you mentioned your nginx version as Nginx 1.22.1.

I don't know that this is what is causing your problem, but: as of nginx 1.21.5, nginx has changed to use pcre2 by default, whereas the default in ModSecurity is pcre1.

To manage this particular difference, you need to do one of three things:

1. use an earlier version of nginx (it looks like the version of nginx available via apt in Ubuntu 22.04 is 1.18.0)
2. use a >= 1.21.5 version of nginx, use the default for ModSecurity, but make sure you build the connector with the --without-pcre2 flag (Add support for PCRE2 #2668 , Module compilation error with NGINX 1.21.5 ModSecurity-nginx#261 (comment))
3. use a >= 1.21.5 version of nginx, use the default for building the connector, but make sure you use the --with-pcre2 flag during the configure step for building ModSecurity itself (Support PCRE2 #2719).

[GNU-Plus-Windows-User]

@martinhsv I've recompiled ModSec using option 2 --without-pcre2 for the Nginx Connector, but I still encountered segfaults. I tried to do option 3 using with-prce2 but it compailed about not being able to find prce2, I using this repository for Lua and Nginx 1.22.1 https://ppa.launchpadcontent.net/ondrej/nginx/ubuntu/, not sure if it's relevant but thought I'd mention just in case.