Support "data streams"

[phudson]

Elastic search has had data streams for a while now. However, the logging exporter doesn't send things in the right way.

According to here: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/use-a-data-stream.html

"To add multiple documents with a single request, use the bulk API. Only create actions are supported."

curl -X PUT "localhost:9200/my-data-stream/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d'
{"create":{ }}
{ "@timestamp": "2099-03-08T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" }
{"create":{ }}
{ "@timestamp": "2099-03-08T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" }
{"create":{ }}
{ "@timestamp": "2099-03-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" }
'
But the exporter sends:

POST /wls/doc/_bulk?pretty HTTP/1.1
Accept: application/json
Content-Type: application/json
User-Agent: Jersey/2.22.4 (HttpUrlConnection 1.8.0_281)
Host: localhost:9200
Connection: Keep-Alive
Content-Length: 1512

{ "index" : { }}
{"messageID": "BEA-2162611","message": "Creating ManagedScheduledExecutorService "DefaultManagedScheduledExecutorService" (partition="DOMAIN", module="null", application="bea_wls_deployment_internal", workmanager="default")","timestamp": 1641037065050,"serverName": "server_8080","threadName": "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'","severity": "Info","userId": "","level": "Info","loggerName": "CONCURRENCY","formattedDate": "Jan 1, 2022 11:37:45,050 AM UTC","subSystem": "CONCURRENCY","machineName": "server","transactionId": "","diagnosticContextId": "7434236c-676a-4857-839f-9b4b00bc7fc7-000000a5","sequenceNumber": 5865,"domainUID": "domainid"}

and ES produces an error:

{
"took" : 0,
"errors" : true,
"items" : [
{
"index" : {
"_index" : "wls",
"_type" : "doc",
"_id" : null,
"status" : 400,
"error" : {
"type" : "illegal_argument_exception",
"reason" : "only write ops with an op_type of create are allowed in data streams"
}
}
},
{
"index" : {
"_index" : "wls",
"_type" : "doc",
"_id" : null,
"status" : 400,
"error" : {
"type" : "illegal_argument_exception",
"reason" : "only write ops with an op_type of create are allowed in data streams"
}
}
}
]
}

Since (I assume) the exported never needs to update a doc, it could use "create" here. I'll try this out and submit a pull request if it works

[phudson]

This change was pretty easy:

In LogExportHandler.java, change

private static final String DOC_TYPE = "doc";
private static final String INDEX = " { \"index\" : { }} ";

to

private static final String DOC_TYPE = "_doc";
private static final String CREATE = " { \"create\" : { }} ";

and the use:

buffer.append(INDEX);

to

 buffer.append(CREATE);

And the name of the time stamp in the assembly of the event

    + dataAsJson("timestamp", wlLogRecord.getMillis())

to

    + dataAsJson("@timestamp", wlLogRecord.getMillis())

and in createMappings:

        + "        \"timestamp\": {"

to
+ " "@timestamp": {"

Note: The doc -> _doc change is a generally applicable bug fix, I think - the doc type should be _doc. In later ES versions, doc types is deprecated anyway, so it could be removed entirely. I didn't make that change.

The index->create change will also work for "straight" indicies as well as data streams, as far as I can see so this change too could be made unconditionally.

[phudson]

Nothing happened in a month. Is this project dead?