Json audit logging #914

p0pr0ck5 · 2015-07-16T22:38:11Z

Implement generation of audit logging data as JSON (available as a compile-time option).

* Write Stopwatch2 values into a separate map * Remove legacy Stopwatch * Proper sanitization of request/response headers * Lazily open maps for keys that may not have content

Remove compile-time setting for generating audit logs as JSON, creating a new config option (SecAuditLogFormat). sec_audit_logger is now a wrapper for sec_audit_logger_json or sec_audit_logger_native. This has the disadvantage of making the audit log generation code harder to maintain, but the logger function itself now is no longer pepper with binary branches.

p0pr0ck5 · 2015-07-19T05:48:31Z

Refactored this to make it a configurable option (instead of set at compile time)

Create a separate map for each matched rule chain, making it easier to identify chains in which only a portion of rules actually matched.

* Escape rule actionset metadata * Escape and truncate logdata * Lazily add actionset tags as an array * Add negated rule op_param * Add unparsed rule representation

p0pr0ck5 · 2015-07-24T17:59:50Z

Any updates or notes on whether this is an acceptable patch? If there are stylistic concerns or missing functionality I will be happy to review them. Thanks.

zimmerle · 2015-07-27T12:17:54Z

Hi @p0pr0ck5, thanks for the patch!

I did not reviewed it yet, working on libmodsecurity -
https://github.com/SpiderLabs/ModSecurity/tree/libmodsecurity

Is this patch following the format discussed at #897 ?

p0pr0ck5 · 2015-07-27T20:22:46Z

@zimmerle, the JSON structure is somewhat different, in that the 'messages' key does not contain detailed info about the matched rules; instead msr->matched_rules->elts is used to grab detailed info of the msre_rule and msre_actionset structs to form the details of matched rules. This means that we don't have to stomp on the signatures of msc_alert_message and msc_alert, which I think makes the patch much more maintainable. You can see an example of a log generated here: https://gist.github.com/p0pr0ck5/02687caa5ac0af80d6d4 as well as some discussion here: https://www.cryptobells.com/mod_security-json-audit-logs-revisited/

The reason the patch is so large at this point is because we duplicated sec_audit_logger in order to make it more readable (rather than littering the function with if-blocks based on whether or not JSON logging is configured).

p0pr0ck5 · 2015-08-24T01:23:51Z

Hi @zimmerle, any idea if this patch is likely to make it into mainline? I understand most development time is being put into libmodsecurity, just looking for an idea if this is maybe in the ballpark.

zimmerle · 2015-09-29T18:27:52Z

Hi @p0pr0ck5,

Sorry for the delay, the development of libmodsecurity is taking more time than what I was expecting.

We need to have both versions 2.9 and libModSecurity with the exactly same format, does not make sense to have two different things. In order to do that, I was hoping to have into the discussion about the format the authors of AuditConsole [1] and WAF FLE [2].

I will start a discussion via email. Also, not sure if you are in our development mailing list, but we are going to have a community meeting to discuss the open issues, etc... tell me if you are interested to participate...

[1] https://jwall.org/web/audit/console/index.jsp
[2] http://waf-fle.org/

p0pr0ck5 · 2015-09-30T19:18:45Z

Thanks for the reply and the off-thread discussion; I've joined the development mailing list and look forward to further discussion on this.

'is_chained' should be true for an actionset when the is_chained member of the struct is true, or when its rule has a valid chain_starter member.

zimmerle · 2016-01-29T16:29:38Z

Merged! :)

Thanks @p0pr0ck5!!!

jurgenweber · 2017-04-27T05:54:41Z

is this going to be in version 3?

zimmerle · 2017-04-27T11:02:18Z

@jurgenweber version 3 is JSON by default.

jurgenweber · 2017-04-27T20:17:10Z

I am not experiencing that at all.

My Goal is to get this data into elasticsearch/kibana in a structured manner. Right now I am getting the logs per section, line by line.

For further context, I am using this: https://hub.docker.com/r/elisiano/nginx-modsecurity/~/dockerfile/

umarfarook882 · 2017-05-06T11:43:06Z

Mod Security Log (JSON) Structure is not proper key:pair?

Hi all I was happy about mod security support JSON log format but why log is in not proper key:pair structure? I am planing to pull that log into Elastic search and Kibana but i am stuck because log are not structure (key:pair}. so let me know is any solution to make that possible? i was reading the one blog in which he get the log in key pair combination.so check this blog. Let me know is there any solution for the problem.

{"transaction":{"time":"05/May/2017:10:50:14 +0530","transaction_id":"WQwLjn8AAQEAAAwdUykAAAAA","remote_address":"127.0.0.1","remote_port":56642,"local_address":"127.0.0.1","local_port":80},"request":{"request_line":"GET / HTTP/1.1","headers":{"Host":"127.0.0.1","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language":"en-US,en;q=0.5","Accept-Encoding":"gzip, deflate","Connection":"keep-alive","Upgrade-Insecure-Requests":"1"}},"response":{"protocol":"HTTP/1.0","status":500,"headers":{"Content-Length":"0","Connection":"close","Content-Type":"text/html; charset=UTF-8"},"body":""},"audit_data":{"error_messages":["[file \"/build/php5-tXxPwu/php5-5.6.7+dfsg/sapi/apache2handler/sapi_apache2.c\"] [line 325] [level 3] %s","[file \"/build/php5-tXxPwu/php5-5.6.7+dfsg/sapi/apache2handler/sapi_apache2.c\"] [line 325] [level 3] %s","[file \"/build/php5-tXxPwu/php5-5.6.7+dfsg/sapi/apache2handler/sapi_apache2.c\"] [line 325] [level 3] %s"],"handler":"application/x-httpd-php","stopwatch":{"p1":315,"p2":258,"p3":2,"p4":83,"p5":56,"sr":37,"sw":0,"l":0,"gc":0},"response_body_dechunked":true,"producer":["ModSecurity for Apache/2.9.1 (http://www.modsecurity.org/)","OWASP_CRS/2.2.9"],"server":"Apache/2.4.25 (Debian)","engine_mode":"ENABLED"}}

umarfarook882 · 2017-05-06T20:59:41Z

Mod Security Log monitoring Dashboard

Finally Mod Security Log monitoring Dashboard is ready, I have integrated Mod security Audit Log with ELK(latest version) for real time analysis and it's working good. So i am now working on WAF rules development. i need some guidance from others regarding WAF rule development. So please give some ideas to move further ?

p0pr0ck5 · 2017-05-08T15:24:40Z

@umarfarook882 please do not hijack unrelated issues for your own personal questions, it's impolite and confusing for other members of the community. Consider using the mailing list to ask your own questions. Thank you! :)

p0pr0ck5 added 3 commits July 15, 2015 13:57

first pass at JSON logging implementation

d1c12d2

Additional updates for JSON logging

7305381

* Write Stopwatch2 values into a separate map * Remove legacy Stopwatch * Proper sanitization of request/response headers * Lazily open maps for keys that may not have content

p0pr0ck5 added 2 commits July 19, 2015 00:09

Update JSON structure for matched rules

806e8df

Create a separate map for each matched rule chain, making it easier to identify chains in which only a portion of rules actually matched.

Clean up JSON rule writer

60640eb

* Escape rule actionset metadata * Escape and truncate logdata * Lazily add actionset tags as an array * Add negated rule op_param * Add unparsed rule representation

zimmerle self-assigned this Oct 16, 2015

zimmerle added the RIP - release-2.9.1 label Oct 16, 2015

p0pr0ck5 added 5 commits November 10, 2015 11:12

Do not compile in JSON logging support if yajl is not found

42b9d24

Fix 'is_chained' value for final rule in chain

e7124dd

'is_chained' should be true for an actionset when the is_chained member of the struct is true, or when its rule has a valid chain_starter member.

Whitespace fix for pull request

973e883

Add JSON log parse script

1903a42

Updates to parse_modsec.pl

19a7655

zimmerle pushed a commit that referenced this pull request Jan 29, 2016

Adds information about the pull request #914 on the CHANGES file

c131dcc

This was referenced Jan 29, 2016

Machine Parseable Audit Log Formats #656

Closed

Sending Logs in JSON format #897

Closed

zimmerle closed this Jan 29, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Json audit logging #914

Json audit logging #914

Uh oh!

p0pr0ck5 commented Jul 16, 2015

Uh oh!

p0pr0ck5 commented Jul 19, 2015

Uh oh!

p0pr0ck5 commented Jul 24, 2015

Uh oh!

zimmerle commented Jul 27, 2015

Uh oh!

p0pr0ck5 commented Jul 27, 2015

Uh oh!

p0pr0ck5 commented Aug 24, 2015

Uh oh!

zimmerle commented Sep 29, 2015

Uh oh!

p0pr0ck5 commented Sep 30, 2015

Uh oh!

zimmerle commented Jan 29, 2016

Uh oh!

jurgenweber commented Apr 27, 2017

Uh oh!

zimmerle commented Apr 27, 2017

Uh oh!

jurgenweber commented Apr 27, 2017 •

edited

Loading

Uh oh!

umarfarook882 commented May 6, 2017

Uh oh!

umarfarook882 commented May 6, 2017

Uh oh!

p0pr0ck5 commented May 8, 2017

Uh oh!

Uh oh!

Json audit logging #914

Json audit logging #914

Uh oh!

Conversation

p0pr0ck5 commented Jul 16, 2015

Uh oh!

p0pr0ck5 commented Jul 19, 2015

Uh oh!

p0pr0ck5 commented Jul 24, 2015

Uh oh!

zimmerle commented Jul 27, 2015

Uh oh!

p0pr0ck5 commented Jul 27, 2015

Uh oh!

p0pr0ck5 commented Aug 24, 2015

Uh oh!

zimmerle commented Sep 29, 2015

Uh oh!

p0pr0ck5 commented Sep 30, 2015

Uh oh!

zimmerle commented Jan 29, 2016

Uh oh!

jurgenweber commented Apr 27, 2017

Uh oh!

zimmerle commented Apr 27, 2017

Uh oh!

jurgenweber commented Apr 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

umarfarook882 commented May 6, 2017

Mod Security Log (JSON) Structure is not proper key:pair?

Uh oh!

umarfarook882 commented May 6, 2017

Mod Security Log monitoring Dashboard

Uh oh!

p0pr0ck5 commented May 8, 2017

Uh oh!

Uh oh!

jurgenweber commented Apr 27, 2017 •

edited

Loading