Skip to content

Improve Analyzer definitions and fix various classes #3215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Feb 5, 2025
Merged

Improve Analyzer definitions and fix various classes #3215

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
16979d3
Improve `Analyzer` definitions
flobernd Dec 3, 2024
6b0a1c3
Version is deprecated
flobernd Dec 3, 2024
387138a
Minor `Processor` related fixes
flobernd Dec 3, 2024
c2cb5c4
Generate
flobernd Dec 3, 2024
108d411
Fix `SpanQuery`
flobernd Dec 3, 2024
179e327
`PinnedDoc._index` is optional
flobernd Dec 3, 2024
6a12ba9
Fix `graph.explore` `Hop` definition
flobernd Dec 3, 2024
e40cf28
`AzureRepository.settings` is optional
flobernd Dec 3, 2024
0de0a84
Fix `Hop`
flobernd Dec 3, 2024
1a513e6
Fix query validation
flobernd Dec 4, 2024
3b08e5f
Fix shortcut properties nullability
flobernd Dec 13, 2024
bf02b19
Regenerate
flobernd Dec 13, 2024
757f86d
Make linter happy
flobernd Feb 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21,670 changes: 5,555 additions & 16,115 deletions output/schema/schema.json
View file
Open in desktop

Large diffs are not rendered by default.

34 changes: 18 additions & 16 deletions output/typescript/types.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

92 changes: 85 additions & 7 deletions specification/_types/analysis/analyzers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,21 +36,42 @@ export class CustomAnalyzer {

export class FingerprintAnalyzer {
type: 'fingerprint'
/** @deprecated 7.14.0 */
version?: VersionString
max_output_size: integer
preserve_original: boolean
separator: string
stopwords?: StopWords
/**
* The maximum token size to emit. Tokens larger than this size will be discarded.
* Defaults to `255`
*
* @server_default 255
*/
max_output_size?: integer
/**
* The character to use to concatenate the terms.
* Defaults to a space.
*/
separator?: string
/**
* A pre-defined stop words list like `_english_` or an array containing a list of stop words.
* Defaults to `_none_`.
*
* @server_default _none_
*/
stopwords?: StopWords
/**
* The path to a file containing stop words.
*/
stopwords_path?: string
}

export class KeywordAnalyzer {
type: 'keyword'
/** @deprecated 7.14.0 */
version?: VersionString
}

export class LanguageAnalyzer {
type: 'language'
/** @deprecated 7.14.0 */
version?: VersionString
language: Language
stem_exclusion: string[]
Expand Down Expand Up @@ -311,6 +332,7 @@ export class ThaiAnalyzer {

export class NoriAnalyzer {
type: 'nori'
/** @deprecated 7.14.0 */
version?: VersionString
decompound_mode?: NoriDecompoundMode
stoptags?: string[]
Expand All @@ -319,40 +341,96 @@ export class NoriAnalyzer {

export class PatternAnalyzer {
type: 'pattern'
/** @deprecated 7.14.0 */
version?: VersionString
flags?: string
/**
* Java regular expression flags. Flags should be pipe-separated, eg "CASE_INSENSITIVE|COMMENTS".
*/
flags?: string // TODO: Use PipeSeparatedFlags<T> and proper enum
/**
* Should terms be lowercased or not.
* Defaults to `true`.
*
* @server_default true
*/
lowercase?: boolean
pattern: string
stopwords?: StopWords
/**
* A Java regular expression.
* Defaults to `\W+`.
*
* @server_default \W+
*/
pattern?: string
/**
* A pre-defined stop words list like `_english_` or an array containing a list of stop words.
* Defaults to `_none_`.
*
* @server_default _none_
*/
stopwords?: StopWords
/**
* The path to a file containing stop words.
*/
stopwords_path?: string
}

export class SimpleAnalyzer {
type: 'simple'
/** @deprecated 7.14.0 */
version?: VersionString
}

// TODO: This one seems undocumented!?
export class SnowballAnalyzer {
type: 'snowball'
/** @deprecated 7.14.0 */
version?: VersionString
language: SnowballLanguage
stopwords?: StopWords
}

export class StandardAnalyzer {
type: 'standard'
/**
* The maximum token length. If a token is seen that exceeds this length then it is split at `max_token_length` intervals.
* Defaults to `255`.
*
* @server_default 255
*/
max_token_length?: integer
/**
* A pre-defined stop words list like `_english_` or an array containing a list of stop words.
* Defaults to `_none_`.
*
* @server_default _none_
*/
stopwords?: StopWords
/**
* The path to a file containing stop words.
*/
stopwords_path?: string
}

export class StopAnalyzer {
type: 'stop'
/** @deprecated 7.14.0 */
version?: VersionString
/**
* A pre-defined stop words list like `_english_` or an array containing a list of stop words.
* Defaults to `_none_`.
*
* @server_default _none_
*/
stopwords?: StopWords
/**
* The path to a file containing stop words.
*/
stopwords_path?: string
}

export class WhitespaceAnalyzer {
type: 'whitespace'
/** @deprecated 7.14.0 */
version?: VersionString
}

Expand Down
5 changes: 3 additions & 2 deletions specification/_types/query_dsl/span.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
*/

import { SingleKeyDictionary } from '@spec_utils/Dictionary'
import { Field } from '@_types/common'
import { Field, FieldValue } from '@_types/common'
import { integer } from '@_types/Numeric'
import { QueryBase, QueryContainer } from './abstractions'

Expand Down Expand Up @@ -136,7 +136,8 @@ export class SpanOrQuery extends QueryBase {
* @ext_doc_id query-dsl-span-term-query
*/
export class SpanTermQuery extends QueryBase {
value: string
/** @aliases term */
value: FieldValue
}

/**
Expand Down
2 changes: 1 addition & 1 deletion specification/_types/query_dsl/specialized.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ export class PinnedDoc {
/**
* The index that contains the document.
*/
_index: IndexName
_index?: IndexName
}

export class RankFeatureFunction {}
Expand Down
2 changes: 1 addition & 1 deletion specification/graph/_types/Hop.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ export class Hop {
/**
* An optional guiding query that constrains the Graph API as it explores connected terms.
*/
query: QueryContainer
query?: QueryContainer
/**
* Contains the fields you are interested in.
*/
Expand Down
3 changes: 2 additions & 1 deletion specification/graph/_types/Vertex.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ export class VertexDefinition {
size?: integer
}

/** @shortcut_property term */
export class VertexInclude {
boost: double
boost?: double
term: string
}
5 changes: 3 additions & 2 deletions specification/ingest/_types/Processors.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import { UserDefinedValue } from '@spec_utils/UserDefinedValue'
import { Field, Fields, GrokPattern, Id, Name } from '@_types/common'
import { GeoShapeRelation } from '@_types/Geo'
import { double, integer, long } from '@_types/Numeric'
import { Script } from '@_types/Scripting'
import { SortOrder } from '@_types/sort'

/**
Expand Down Expand Up @@ -309,7 +310,7 @@ export class ProcessorBase {
/**
* Conditionally execute the processor.
*/
if?: string
if?: Script
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flobernd I'm curious about whether this change is correct. I thought the if conditions for all ingest processors were represented only string values (that hold Painless expressions). If I understand correctly, this makes the if value into a struct which runs contrary to the pipelines processors I have used. Could you double check this please?

Copy link
Member Author

@flobernd flobernd Feb 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @andrewkroh, sure I will double check.

The driver for this change are some examples in our REST API documentation which do use a stroed script for the condition.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation says:

You can also specify a stored script as the if condition.

The corresponding example is this one:

PUT _ingest/pipeline/my-pipeline
{
  "processors": [
    {
      "drop": {
        "description": "Drop documents that don't contain 'prod' tag",
        "if": { "id": "my-prod-tag-script" }
      }
    }
  ]
}

Our current Script type is defined like this:

/**
 * @shortcut_property source
 * */
export class Script {
  /**
   * The script source.
   */
  source?: string
  /**
   * The `id` for a stored script.
   */
  id?: Id
  // ...
}

Because of the "shortcut" to source, any JSON string literal is eqivalent to initializing a Script object and setting the source property to the value.

For example:

"if": "test"

is eqivalent to

"if": {
  "source": "test"
}

I just double checked in the Kibana dev console again and both requests (+ the one that uses id instead of source) do indeed work fine 🙂

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic. My missing piece of knowledge was this @shortcut_property. Thank you!

/**
* Ignore failures for the processor.
*/
Expand Down Expand Up @@ -731,7 +732,7 @@ export class DateIndexNameProcessor extends ProcessorBase {
* An array of the expected date formats for parsing dates / timestamps in the document being preprocessed.
* Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
*/
date_formats: string[]
date_formats?: string[]
/**
* How to round the date when formatting the date into the index name. Valid values are:
* `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second).
Expand Down
4 changes: 2 additions & 2 deletions specification/ml/_types/DataframeAnalytics.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,9 +238,9 @@ export class DataframeAnalysisClassification extends DataframeAnalysis {
/** @shortcut_property includes */
export class DataframeAnalysisAnalyzedFields {
/** An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically. */
includes: string[]
includes?: string[]
/** An array of strings that defines the fields that will be included in the analysis. */
excludes: string[]
excludes?: string[]
}

/** @variants container */
Expand Down
2 changes: 1 addition & 1 deletion specification/snapshot/_types/SnapshotRepository.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ export class AzureRepository extends RepositoryBase {
/**
* The repository settings.
*/
settings: AzureRepositorySettings
settings?: AzureRepositorySettings
}

export class GcsRepository extends RepositoryBase {
Expand Down
0 specification/sql/types.ts → specification/sql/_types/types.ts
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion specification/sql/get_async/SqlGetAsyncResponse.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
*/

import { Id } from '@_types/common'
import { Column, Row } from '../types'
import { Column, Row } from '../_types/types'

export class Response {
body: {
Expand Down
2 changes: 1 addition & 1 deletion specification/sql/query/QuerySqlResponse.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
*/

import { Id } from '@_types/common'
import { Column, Row } from '../types'
import { Column, Row } from '../_types/types'

export class Response {
body: {
Expand Down
Loading