Elasticsearch 2.0 update by query

Unfortunately ES 2.0  doesn’t have _update_by_query function support even via plugins. So lest’s bash it!

Get data

First of all fetch all data (I expect not more 9999) need to be deleted: host is «UA-web1», message contains «Response from bank» and not contans «censored». I’ll put all data in pretty (field by line) mode to the file raw_list.
curl -XGET http://localhost:9200/logstash-2016.03*/_search?pretty=true -d {'
"query": {
"bool": {
"must": [
{ "match": { "host": "UA-web1"}},
{ "match": { "message": "Response"}},
{ "match": { "message": "from"}},
{ "match": { "message": "bank"}}
],
"must_not": {
"match": { "message": "censored" }
}}},
"fields": "message",
"size": 9999
'} > raw_list

Format list to update

Then parse raw_list to format «id index»:
AVON8Q5Hv9AoJbRnorhQ logstash-2016.03.19
AVON-P9Av9AoJbRnot7r logstash-2016.03.19
AVOOIA5hv9AoJbRno6Nu logstash-2016.03.19
AVOOFnFVv9AoJbRno2wT logstash-2016.03.19
AVOOKeW1v9AoJbRno9SO logstash-2016.03.19
Here is my AWK script:
cat raw_list |awk -F'"' '/_id/{id=$4} /_index/{ind=$4} /_score/{print id,ind}' > list_to_delete

Update records

Now file list_to_delete contains lines with IDs of records to update.

CURL query will update a record with _id=$ID and _index=$INDEX:

curl -XPOST http://localhost:9200/$INDEX/esyslog/$ID/_update?pretty -d '{
doc: {"message": "This message was fixed" }
}'

So read list_to_delete in cycle and update records:

cat list_to_delete|while read line; do arr=($line);curl -XPOST http://localhost:9200/${arr[1]}/esyslog/${arr[0]}/_update?pretty -d '{
doc: {"message": "This message was fixed" }
}'; done

That’s all!

Elasticsearch direct update

Try to set field «message» to «_deleted_by_XPOST» where _id=AVOdxbLJv9AoJbRn2TDI and type=esyslog (required field):

url -XPOST http://localhost:9200/logstash-2016.03.22/esyslog/AVOdxbLJv9AoJbRn2TDI/_update?pretty -d {'
"doc": { "message": "_deleted_by_XPOST" }
'}

And successful result is

{
  "_index" : "logstash-2016.03.22",
  "_type" : "esyslog",
  "_id" : "AVOdxbLJv9AoJbRn2TDI",
  "_version" : 2,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  }

Elasticsearch direct queries

Try to search entries from Logstash host UA-web1 with message including «Data» and not including «censored». Filter to fetch «path» field only:

curl -XGET  http://localhost:9200/logstash-2016.03.22/_search?pretty=true -d {'
"query": {
  "bool": {
    "should": [
      { "match": { "host": "UA-web1"}},
      { "match": { "message": "Data"}}
      ],
    "must_not": { 
      "match": { "message": "censored" }
    }
    }
  },
"fields": "path"
'}