Flag in _field_caps to return only fields with values in index

[piergm]

We are adding a query parameter to the field_caps api in order to filter out fields with no values.
The parameter is called include_empty_fields and defaults to true, and if set to false it
will filter out from the field_caps response all the fields that has no value in the index.
We keep track of FieldInfos during refresh in order to know which field has value in an index.
We added also a sys.prop es.field_caps_empty_fields_filter in order to disable this feature if needed.

[elasticsearchmachine]

Hi @piergm, I've created a changelog YAML for you.

[kertal]

A quick question, in a CCS scenario, when there are old CCS clusters targeted, and just the requesting machine has the include_empty_fields capabilities (using set to false in the request ). Then the CCS clusters return all the fields, including the empty ones, right? So all CCS clusters need to support include_empty_fields to get the desired result of excluding empty ones. But apart from that, such a request should work, so no error is thrown, because the CCS ones haven't heard of include_empty_fields before?

[piergm]

@kertal, yes, in CCS if we target es version prior to 8.13 we will not consider the flag and therefore we will be returning all fields, empty and non-empty, from those non updated clusters. That said the response per se is bwc and can be handled from the updated cluster, therefore we will not return any error from this miss match, instead we will return also empty fields from non updated clusters.

[kertal]

@kertal, yes, in CCS if we target es version prior to 8.13 we will not consider the flag and therefore we will be returning all fields, empty and non-empty, from those non updated clusters. That said the response per se is bwc and can be handled from the updated cluster, therefore we will not return any error from this miss match, instead we will return also empty fields from non updated clusters.

thx for the confirmation @piergm 👍

[javanna]

I left some cosmetic comments, mostly around testing refinements. LGTM otherwise.

[javanna]

cool thanks for the update. Could you check that we have test coverage about the new flag used in combination with index_filter, as well as not? Thanks!

[javanna]

This assessment does not sound accurate to me. The mapping hash is used to deduplicate mappings, and decide to send back one mapping per hash instead one mapping per index. But the whole mapping needs to be sent back?

Using the flag, the index mapping hash deduplication is entirely disabled. We do have now field level deduplication across indices though, and the benefit that not returning empty fields will make responses smaller and decrease overhead on the merging side. But again, it is an entirely different trade-off and we need to measure the impact in the worst case scenario, compared to the previous index mapping hash based deduplication. Makes sense?

[original-brownbear]

Performance wise this is just fine for the non-ccs case. The logic is optimal enough at this point it seems to make serialisation fo the response the most important aspect of this. I could not easily reproduce a case where the flag had much measurable overhead (maybe a couple percent when all fields have data but that's it, not a crazy scientific benchmark run here but still :)).

E.g.

Server Software:        
Server Hostname:        elasticsearch-4
Server Port:            9200
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name:        elasticsearch-4

Document Path:          /_field_caps?fields=*
Document Length:        2067221 bytes

Concurrency Level:      1
Time taken for tests:   39.806 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      206733400 bytes
HTML transferred:       206722100 bytes
Requests per second:    2.51 [#/sec] (mean)
Time per request:       398.058 [ms] (mean)
Time per request:       398.058 [ms] (mean, across all concurrent requests)
Transfer rate:          5071.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        7    7   0.1      7       7
Processing:   376  391  19.1    386     516
Waiting:      371  386  19.1    382     511
Total:        383  398  19.1    393     523

Percentage of the requests served within a certain time (ms)
  50%    393
  66%    395
  75%    398
  80%    400
  90%    411
  95%    440
  98%    478
  99%    523
 100%    523 (longest request)

vs

Server Software:        
Server Hostname:        elasticsearch-4
Server Port:            9200
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name:        elasticsearch-4

Document Path:          /_field_caps?fields=*?include_empty_fields=false
Document Length:        1891399 bytes

Concurrency Level:      1
Time taken for tests:   39.314 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      189151200 bytes
HTML transferred:       189139900 bytes
Requests per second:    2.54 [#/sec] (mean)
Time per request:       393.138 [ms] (mean)
Time per request:       393.138 [ms] (mean, across all concurrent requests)
Transfer rate:          4698.55 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        7    7   0.1      7       8
Processing:   372  386  16.9    380     484
Waiting:      369  383  16.9    377     481
Total:        379  393  17.0    387     491

Percentage of the requests served within a certain time (ms)
  50%    387
  66%    395
  75%    401
  80%    402
  90%    411
  95%    425
  98%    452
  99%    491
 100%    491 (longest request)

[original-brownbear]

Did not re-read everything again, but performance is fine in my testing and a quick read of the code looks just fine still -> LGTM

[kertal]

🎉 👋 🙇 !

[ninoslavmiskovic]

Amazing 🤩 great job all 👍

[elasticsearchmachine]

@piergm according to this PR's labels, I need to update the changelog YAML, but I can't because the PR is closed. Please either update the changelog yourself on the appropriate branch, or adjust the labels. Specifically:

• The PR is labelled release highlight but the changelog has no highlight section