Skip to content

Fixes automatic import and AI Assistant pages #756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 12, 2025
Merged

Fixes automatic import and AI Assistant pages #756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8566db5
Cleans up automatic import and AI Assistant pages
benironside Mar 12, 2025
c16c32d
Merge branch 'main' into ben-cleanup-final
benironside Mar 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 19 additions & 64 deletions solutions/security/ai/ai-assistant-knowledge-base.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,16 @@ mapped_urls:

# AI Assistant Knowledge Base

% What needs to be done: Align serverless/stateful

% Use migrated content from existing pages that map to this page:

% - [x] ./raw-migrated-files/security-docs/security/ai-assistant-knowledge-base.md
% - [ ] ./raw-migrated-files/docs-content/serverless/ai-assistant-knowledge-base.md

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):

$$$enable-knowledge-base$$$

$$$knowledge-base-add-knowledge-index$$$

AI Assistant’s Knowledge Base feature enables AI Assistant to recall specific documents and other specified information. This information, which can include everything from the location of your datacenters to the latest threat research, provides additional context that can improve the quality of AI Assistant’s responses to your queries. This topic describes how to enable and add information to Knowledge Base.

::::{note}
When you upgrade from {{elastic-sec}} version 8.15 to a newer version, information previously stored by AI Assistant will be lost.
{{stack}} users: when you upgrade from {{elastic-sec}} version 8.15 to a newer version, information previously stored by AI Assistant will be lost.
::::


::::{admonition} Requirements
* To use Knowledge Base, you need the `Elastic AI Assistant: All` privilege. To edit global Knowledge Base entries (information that will affect the AI Assistant experience for other users in the {{kib}} space), you need the `Allow Changes to Global Entries` privilege.
* To use Knowledge Base, the `Elastic AI Assistant: All` privilege.
* To edit global Knowledge Base entries (information that will affect the AI Assistant experience for other users in the {{kib}} space), the `Allow Changes to Global Entries` privilege.
* You must [enable machine learning](/solutions/security/advanced-entity-analytics/machine-learning-job-rule-requirements.md) with a minimum ML node size of 4 GB.

::::
Expand Down Expand Up @@ -114,20 +102,9 @@ Add an individual document to Knowledge Base when you want AI Assistant to remem
5. In the **Markdown text** field, enter the information you want AI Assistant to remember.
6. If it should be **Required knowledge**, select the option. Otherwise, leave it blank. Alternatively, you can simply send a message to AI Assistant that instructs it to "Remember" the information. For example, "Remember that I changed my password today, October 24, 2024", or "Remember we always use the Threat Hunting Timeline template when investigating potential threats". Entries created in this way are private to you. By default they are not required knowledge, but you can make them required by instructing AI Assistant to "Always remember", for example "Always remember to address me as madam", or "Always remember that our primary data center is located in Austin, Texas".

Refer to the following video for an example of adding a document to Knowledge Base from the settings menu.

::::{admonition}
<script type="text/javascript" async src="https://play.vidyard.com/embed/v4.js"></script>
<img
style="width: 100%; margin: auto; display: block;"
class="vidyard-player-embed"
src="https://play.vidyard.com/rQsTujEfikpx3vv1vrbfde.jpg"
data-uuid="rQsTujEfikpx3vv1vrbfde"
data-v="4"
data-type="inline"
/>
</br>
::::
Refer to the following video for an example of adding a document to Knowledge Base from the settings menu (click to play video).

[![Add knowledge document video](https://play.vidyard.com/rQsTujEfikpx3vv1vrbfde.jpg)](https://videos.elastic.co/watch/rQsTujEfikpx3vv1vrbfde?)



Expand All @@ -154,20 +131,10 @@ Indices added to Knowledge Base must have at least one field mapped as [semantic
:alt: Knowledge base's Edit index entry menu
:::

Refer to the following video for an example of adding an index to Knowledge Base.

::::{admonition}
<script type="text/javascript" async src="https://play.vidyard.com/embed/v4.js"></script>
<img
style="width: 100%; margin: auto; display: block;"
class="vidyard-player-embed"
src="https://play.vidyard.com/Q5CjXMN4R2GYLGLUy5P177.jpg"
data-uuid="Q5CjXMN4R2GYLGLUy5P177"
data-v="4"
data-type="inline"
/>
</br>
::::
Refer to the following video for an example of adding an index to Knowledge Base (click to play video).


[![Add knowledge index video](https://play.vidyard.com/Q5CjXMN4R2GYLGLUy5P177.jpg)](https://videos.elastic.co/watch/Q5CjXMN4R2GYLGLUy5P177?)



Expand All @@ -185,23 +152,22 @@ First, you’ll need to set up a web crawler to add the desired data to an index
1. From the **Search** section of {{kib}}, find **Web crawlers** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
2. Click **New web crawler**.

1. Under **Index name**, name the index where the data from your new web crawler will be stored, for example `threat_intelligence_feed_1`. Click **Create index**.
2. Under **Domain URL**, enter the URL where the web crawler should collect data. Click **Validate Domain** to test it, then **Add domain**.
* Under **Index name**, name the index where the data from your new web crawler will be stored, for example `threat_intelligence_feed_1`. Click **Create index**.
* Under **Domain URL**, enter the URL where the web crawler should collect data. Click **Validate Domain** to test it, then **Add domain**.

3. The previous step opens a page with the details of your new index. Go to its **Mappings** tab, then click **Add field**.

::::{note}
Remember, each index added to Knowledge Base must have at least one semantic text field.
::::

::::{note}
Remember, each index added to Knowledge Base must have at least one semantic text field.
::::

1. Under **Field type**, select `Semantic text`. Under **Select an inference endpoint***, select `elastic-security-ai-assistant-elser2`. Click ***Add field**, then **Save mapping**.
* Under **Field type**, select `Semantic text`. Under **Select an inference endpoint**, select `elastic-security-ai-assistant-elser2`. Click **Add field**, then **Save mapping**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these be separate steps?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I see how they could be, but they're like this on purpose as of now!


4. Go to the **Scheduling** tab. Enable the **Enable recurring crawls with the following schedule** setting, and define your desired schedule.
5. Go to the **Manage Domains** tab. Select the domain associated with your new web crawler, then go the its **Crawl rules** tab and click **Add crawl rule**. For more information, refer to [Web crawler content extraction rules](https://www.elastic.co/guide/en/enterprise-search/current/crawler-extraction-rules.html).

1. Click **Add crawl rule** again. Under **Policy***, select `Disallow`. Under ***Rule***, select `Regex`. Under ***Path pattern**, enter `.*`. Click **Save**.
2. Under **Policy**, select `Allow`. Under **Rule***, select `Contains`. Under ***Path pattern**, enter your path pattern, for example `threat-intelligence`. Click **Save**. Make sure this rule appears below the rule created in the previous step on the list.
1. Click **Add crawl rule** again. Under **Policy**, select `Disallow`. Under **Rule**, select `Regex`. Under **Path pattern**, enter `.*`. Click **Save**.
2. Under **Policy**, select `Allow`. Under **Rule**, select `Contains`. Under **Path pattern**, enter your path pattern, for example `threat-intelligence`. Click **Save**. Make sure this rule appears below the rule created in the previous step on the list.
3. Click **Crawl**, then **Crawl all domains on this index**. A success message appears. The crawl process will take longer for larger data sources. Once it finishes, your new web crawler’s index will contain documents provided by the crawler.

6. Finally, follow the instructions to [add an index to Knowledge Base](/solutions/security/ai/ai-assistant-knowledge-base.md#knowledge-base-add-knowledge-index). Add the index that contains the data from your new web crawler (`threat_intelligence_feed_1` in this example).
Expand All @@ -210,15 +176,4 @@ Your new threat intelligence data is now included in Knowledge Base and can info

Refer to the following video for an example of creating a web crawler to ingest threat intelligence data and adding it to Knowledge Base.

::::{admonition}
<script type="text/javascript" async src="https://play.vidyard.com/embed/v4.js"></script>
<img
style="width: 100%; margin: auto; display: block;"
class="vidyard-player-embed"
src="https://play.vidyard.com/eYo1e1ZRwT2mjfM7Yr9MuZ.jpg"
data-uuid="eYo1e1ZRwT2mjfM7Yr9MuZ"
data-v="4"
data-type="inline"
/>
</br>
::::
[![Add knowledge via web crawler video](https://play.vidyard.com/eYo1e1ZRwT2mjfM7Yr9MuZ.jpg)](https://videos.elastic.co/watch/eYo1e1ZRwT2mjfM7Yr9MuZ?)
26 changes: 7 additions & 19 deletions solutions/security/ai/ai-assistant.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,6 @@ mapped_urls:

# AI Assistant

% What needs to be done: Align serverless/stateful

% Use migrated content from existing pages that map to this page:

% - [x] ./raw-migrated-files/security-docs/security/security-assistant.md
% - [ ] ./raw-migrated-files/docs-content/serverless/security-ai-assistant.md

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):

$$$configure-ai-assistant$$$

$$$ai-assistant-anonymization$$$

The Elastic AI Assistant utilizes generative AI to bolster your cybersecurity operations team. It allows users to interact with {{elastic-sec}} for tasks such as alert investigation, incident response, and query generation or conversion using natural language and much more.

:::{image} ../../../images/security-assistant-basic-view.png
Expand All @@ -32,11 +19,12 @@ The Elastic AI Assistant is designed to enhance your analysis with smart dialogu


::::{admonition} Requirements
* The Elastic AI Assistant and Generative AI connector are available in {{stack}} versions 8.8.1 and later. The Generative AI connector is renamed to OpenAI connector in 8.11.0.
* This feature requires an [Enterprise subscription](https://www.elastic.co/pricing).
* To use AI Assistant, you need at least the **Elastic AI Assistant : All** and **Actions and Connectors : Read** [privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md).
* To set up AI Assistant, you need the **Actions and Connectors : All** [privilege](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md).
* You need a [generative AI connector](/solutions/security/ai/set-up-connectors-for-large-language-models-llm.md), which AI Assistant uses to generate responses.
* {{stack}} users: {{stack}} version 8.8.1 or later. Also note the Generative AI connector was renamed to OpenAI connector in 8.11.0.
* {{stack}} users: an [Enterprise subscription](https://www.elastic.co/pricing).
* {{serverless-short}} users: a [Security Analytics Complete subscription](/deploy-manage/deploy/elastic-cloud/project-settings.md).
* To use AI Assistant, the **Elastic AI Assistant : All** and **Actions and Connectors : Read** [privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md).
* To set up AI Assistant, the **Actions and Connectors : All** [privilege](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges.md).
* A [generative AI connector](/solutions/security/ai/set-up-connectors-for-large-language-models-llm.md), which AI Assistant uses to generate responses.

::::

Expand Down Expand Up @@ -148,7 +136,7 @@ To modify Anonymization settings, you need the **Elastic AI Assistant: All** pri
The **Anonymization** tab of the Security AI settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed*** toggled on are included in events provided to AI Assistant. ***Allowed*** fields with ***Anonymized** set to **Yes** are included, but with their values obfuscated.

::::{note}
You can access anonymization settings directly from the **Attack Discovery** page by clicking the settings (![Settings icon](../../../images/security-icon-settings.png "")) button next to the model selection dropdown menu.
You can access anonymization settings directly from the **Attack Discovery** page by clicking the settings (![Settings icon](../../../images/security-icon-settings.png "title=70%")) button next to the model selection dropdown menu.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually resizing all the icons in our docs in this bugbash PR, so this will be covered there :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're a legend

::::


Expand Down
34 changes: 16 additions & 18 deletions solutions/security/get-started/automatic-import.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,10 @@ Click [here](https://elastic.navattic.com/automatic-import) to access an interac

::::{admonition} Requirements
* A working [LLM connector](/solutions/security/ai/set-up-connectors-for-large-language-models-llm.md). Recommended models: `Claude 3.5 Sonnet`; `GPT-4o`; `Gemini-1.5-pro-002`.
* An [Enterprise](https://www.elastic.co/pricing) subscription.
* {{stack}} users: An [Enterprise](https://www.elastic.co/pricing) subscription.
* {{serverless-short}} users: a [Security Analytics Complete subscription](/deploy-manage/deploy/elastic-cloud/project-settings.md).
* A sample of the data you want to import, in a structured or unstructured format (including JSON, NDJSON, and Syslog).
* To import data from a REST API, have its OpenAPI specification (OAS) file ready.
* To import data from a REST API: its OpenAPI specification (OAS) file.

::::

Expand All @@ -47,32 +48,29 @@ Using Automatic Import allows users to create new third-party data integrations
1. In {{elastic-sec}}, click **Add integrations**.
2. Under **Can’t find an integration?** click **Create new integration**.

:::{image} ../../../images/security-auto-import-create-new-integration-button.png
:alt: The Integrations page with the Create new integration button highlighted
:::
:::{image} ../../../images/security-auto-import-create-new-integration-button.png
:alt: The Integrations page with the Create new integration button highlighted
:::

3. Click **Create integration**.
4. Select an [LLM connector](/solutions/security/ai/set-up-connectors-for-large-language-models-llm.md).
5. Define how your new integration will appear on the Integrations page by providing a **Title**, **Description***, and ***Logo**. Click **Next**.
5. Define how your new integration will appear on the Integrations page by providing a **Title**, **Description**, and **Logo**. Click **Next**.
6. Define your integration’s package name, which will prefix the imported event fields.
7. Define your **Data stream title**, **Data stream description**, and **Data stream name**. These fields appear on the integration’s configuration page to help identify the data stream it writes to.
8. Select your [**Data collection method**](asciidocalypse://docs/beats/docs/reference/filebeat/configuration-filebeat-options.md). This determines how your new integration will ingest the data (for example, from an S3 bucket, an HTTP endpoint, or a file stream).

::::{admonition} Importing CEL data
:class: note

If you select **API (CEL input)**, you’ll have the additional option to upload the API’s OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume and your authentication method before uploading your sample data.

::::
::::{admonition} Importing CEL data
:class: note
If you select **API (CEL input)**, you’ll have the additional option to upload the API’s OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume and your authentication method before uploading your sample data.
::::

9. Upload a sample of your data. Make sure to include all the types of events that you want the new integration to handle.

::::{admonition} Best practices for sample data
* For JSON and NDJSON samples, each object in your sample should represent an event, and you should avoid deeply nested object structures.
* The more variety in your sample, the more accurate the pipeline will be. Include a wide range of unique log entries instead of just repeating the same type of entry. Automatic Import will select up to 100 different events from your sample to use as the basis for the new integration.
* Ideally, each field name should describe what the field does.

::::
::::{admonition} Best practices for sample data
* For JSON and NDJSON samples, each object in your sample should represent an event, and you should avoid deeply nested object structures.
* The more variety in your sample, the more accurate the pipeline will be. Include a wide range of unique log entries instead of just repeating the same type of entry. Automatic Import will select up to 100 different events from your sample to use as the basis for the new integration.
* Ideally, each field name should describe what the field does.
::::

10. Click **Analyze logs**, then wait for processing to complete. This may take several minutes.
11. After processing is complete, the pipeline’s field mappings appear, including ECS and custom fields.
Expand Down