mlogc hangs and/or fails to process data

[rcbarnett-zz]

MODSEC-20: when my httpd server is down :

root@trix:/$ ps -ef | grep mlogc
root 17974 17119 0 18:14 pts/0 00:00:00 grep mlogc
18:14:04 root@trix:/$

after service httpd start

root@trix:/$ps -ef | grep mlogc
root 18097 18096 0 18:15 ? 00:00:00 /opt/mlogc/mlogc /etc/mlogc.conf
root 18099 18096 0 18:15 ? 00:00:00 /opt/mlogc/mlogc /etc/mlogc.conf
root 18122 17119 0 18:15 pts/0 00:00:00 grep mlogc
18:15:50 root@trix$

[rcbarnett-zz]

Original reporter: zoroo

[rcbarnett-zz]

brectanus: On my system I get this:

$ ps -ef|grep mlogc
root 15746 15745 0 09:30 ? 00:00:00 /bin/sh -c /usr/local/bin/mlogc /usr/local/apache2/conf/mlogc-console.conf
root 15747 15746 0 09:30 ? 00:00:00 /usr/local/bin/mlogc /usr/local/apache2/conf/mlogc-console.conf

$ ps -fp15745
UID PID PPID C STIME TTY TIME CMD
root 15745 1 0 09:30 ? 00:00:00 /usr/local/apache2/bin/httpd -e D

What flavor and version of bourne shell is /bin/sh on your system? On mine it is "dash", but some it may be bash, ksh, pdksh, etc.

$ ls -lad /bin/*sh
-rwxr-xr-x 1 root root 813912 2008-05-12 11:36 /bin/bash
-rwxr-xr-x 1 root root 100856 2008-03-12 04:52 /bin/dash
lrwxrwxrwx 1 root root 4 2008-07-18 17:11 /bin/rbash -> bash
lrwxrwxrwx 1 root root 4 2008-07-18 17:12 /bin/sh -> dash

You might try linking /bin/sh to bash (or another shell) and see if that helps.

[rcbarnett-zz]

zoroo: if you need another information I can give :)

[rcbarnett-zz]

zoroo: $ ps -fp18096
UID PID PPID C STIME TTY TIME CMD
root 18096 1 0 Sep16 ? 00:00:01 /opt/httpd/bin/httpd

it's good ???

$ ls -lad /bin/*sh
-rwxr-xr-x 1 root root 801504 mai 24 23:18 /bin/bash
lrwxrwxrwx 1 root root 4 jui 23 10:55 /bin/csh -> tcsh
-rwxr-xr-x 1 root root 1107216 mai 24 18:46 /bin/ksh
lrwxrwxrwx 1 root root 4 jui 23 11:26 /bin/sh -> bash
-rwxr-xr-x 1 root root 352912 mar 15 2007 /bin/tcsh

bash --version
GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

[rcbarnett-zz]

brectanus: I cannot duplicate this.

ModSecurity calls an Apache API function (ap_open_piped_log()) to execute mlogc. So, the bug seems to lie within this API or, more likely, in an underlying APR/APR-Util call for your platform. What apr/apr-util are you using (httpd -V)?

I suggest you try the latest Apache httpd with the latest APU/APR-Util and see if this fixes the issue. These are working fine for me:

Linux Ubuntu 8.04.1 (hardy) x86_64 with httpd 2.2.9 with either of these sets of apr libs:

• APR 1.2.11, APR-Util 1.2.12 (package installed)
• APR 1.3.2, APR-Util 1.3.2 (src installed)

If the upgrade does not solve it, then I'll do some more digging.

[rcbarnett-zz]

zoroo: hello,

/opt/httpd/bin/httpd -V
Server version: Apache/2.2.8 (Unix)
Server built: Sep 5 2008 10:32:20
Server's Module Magic Number: 20051115:11
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/opt/httpd228"
-D SUEXEC_BIN="/opt/httpd228/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

[rcbarnett-zz]

brectanus: Have you been able to try a more recent Apache and/or APR/APR-Util to see if this resolves the issue?

[rcbarnett-zz]

brectanus: Dropping priority as I cannot duplicate and am no longer getting feedback.

[rcbarnett-zz]

michaelcaplan: I'm working with a similar environment (centOS 5.2, custom compiled Apache 2.2.10, PHP 5.2.6, mlogc compiled from ModSecurity v2.5.7).

I'm seeing mlogc processes growing with each apache start and stop:

[root@linux-dev sloccount-2.26]# ps aux | grep mlogc
root 26306 0.0 0.2 4296 648 pts/2 S+ 13:46 0:00 grep mlogc
[root@linux-dev sloccount-2.26]# /usr/local/apache2/bin/apachectl start
[root@linux-dev sloccount-2.26]# ps aux | grep mlogc
root 26325 0.1 0.6 5984 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26327 0.1 0.6 6616 1764 ? S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26350 0.0 0.2 4356 644 pts/2 R+ 13:47 0:00 grep mlogc
[root@linux-dev sloccount-2.26]# /usr/local/apache2/bin/apachectl stop
[root@linux-dev sloccount-2.26]# ps aux | grep mlogc
root 26325 0.0 0.6 5984 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26327 0.0 0.6 6616 1764 ? S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26354 0.0 0.6 5052 1768 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26356 0.0 0.2 5376 644 pts/2 R+ 13:47 0:00 grep mlogc
[root@linux-dev sloccount-2.26]# /usr/local/apache2/bin/apachectl start
[root@linux-dev sloccount-2.26]# ps aux | grep mlogc
root 26325 0.0 0.6 5984 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26327 0.0 0.6 6616 1764 ? S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26354 0.0 0.6 5052 1768 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26360 0.3 0.6 5900 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26362 0.3 0.6 6652 1760 ? S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26385 0.0 0.2 4812 648 pts/2 S+ 13:47 0:00 grep mlogc
[root@linux-dev sloccount-2.26]# /usr/local/apache2/bin/apachectl stop
[root@linux-dev sloccount-2.26]# ps aux | grep mlogc
root 26325 0.0 0.6 5984 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26327 0.0 0.6 6616 1764 ? S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26354 0.0 0.6 5052 1768 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26360 0.1 0.6 5900 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26362 0.1 0.6 6652 1760 ? S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26389 0.3 0.6 4748 1760 pts/2 S 13:47 0:00 /usr/local/bin/mlogc /etc/mlogc.conf
root 26404 0.0 0.2 4940 640 pts/2 R+ 13:47 0:00 grep mlogc

[rcbarnett-zz]

brectanus: What APR/APR-Util is your Apache instance using (httpd -V)?

How was Apache compiled (should be at the top of your config.log where you compiled apache)?

Are you using any NFS mounts for the audit logs?

[rcbarnett-zz]

michaelcaplan: # /usr/local/apache2/bin/httpd -V
Server version: Apache/2.2.10 (Unix)
Server built: Oct 23 2008 09:05:23
Server's Module Magic Number: 20051115:18
Server loaded: APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2"
-D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

apache was configured as follows:

./configure --enable-so --with-mpm=prefork --enable-ssl --enable-rewrite --enable-unique-id --disable-include --disable-userdir --enable-deflate --with-pcre=/usr

No NSF mounts

modsecurity configured as follows:

./configure --with-apxs=/usr/local/apache2/bin/apxs --with-lua=/usr/local --with-pcre=/usr --with-apr=/usr/local/apache2 --with-apu=/usr/local/apache2

Anything else I can provide to better diagnose the problem?

[rcbarnett-zz]

brectanus: Thanks. I'll build a setup as you have described and see if I can duplicate it.

[rcbarnett-zz]

brectanus: I built a centos 5.2 box from scratch, built Apache httpd from source and built ModSecurity 2.5.7 from source:

$ uname -a
Linux brian-centos 2.6.18-92.el5 #1 SMP Tue Jun 10 18:49:47 EDT 2008 i686 i686 i386 GNU/Linux

$ cat /etc/redhat-release
CentOS release 5.2 (Final)

$ /usr/local/apache2/bin/httpd -V
Server version: Apache/2.2.10 (Unix)
Server built: Oct 28 2008 11:30:07
Server's Module Magic Number: 20051115:18
Server loaded: APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2"
-D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

Here is what I did:

sudo yum install pcre-devel
sudo yum install libxml2-devel
tar zxf apr-1.3.3.tar.gz
tar zxf apr-util-1.3.4.tar.gz
tar zxf httpd-2.2.10.tar.gz
tar zxf modsecurity-apache_2.5.7.tar.gz

cd apr-1.3.3
./configure
make && sudo make install

cd ../apr-util-1.3.4
./configure --with-apr=/usr/local/apr

cd ../httpd-2.2.10
./configure --enable-so --with-mpm=prefork --enable-ssl --enable-rewrite --enable-unique-id --disable-include --disable-userdir --enable-deflate --with-pcre=/usr
make && sudo make install

cd ../modsecurity-apache_2.5.7/apache2
./configure --with-apxs=/usr/local/apache2/bin/apxs --without-lua --with-apr=/usr/local/apr/ --with-apu=/usr/local/apr
make && make test && sudo make install
make mlogc
sudo cp ../tools/mlogc /usr/local/bin
sudo chown root:root /usr/local/bin/mlogc
sudo chmod 755 /usr/local/bin/mlogc

I will attach a tarball of the config I am using (contents of /home/user/httpd_root).

This install works just fine for me. The mlogc binary is executed (and stopped) correctly and alerts are sent to the console correctly.

I get this after starting httpd:

$ sudo /usr/local/apache2/bin/apachectl -f /home/user/httpd_root/conf/httpd.conf -k start
$ ps -ef|grep 'mlogc|httpd'
root 19480 1 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
root 19481 19480 0 12:31 ? 00:00:00 /usr/local/bin/mlogc /home/user/httpd_root/conf/mlogc.conf
user 19484 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19485 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19486 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19487 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19488 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19489 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19490 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start
user 19491 19480 0 12:31 ? 00:00:00 /usr/local/apache2/bin/httpd -f /home/user/httpd_root/conf/httpd.conf -k start

(fixed a typo in my original comment)

[rcbarnett-zz]

brectanus: Attached the httpd_root I used in my previous example. Should work if you extract to /home/user.