Skip to content

Add package ml jobs #2496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Aug 14, 2023
Merged

Add package ml jobs #2496

Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
c0d31ce
Add dga and lotl. Todo: datafeeds
susan-shu-c Jul 31, 2023
aaf0b15
Add ded and lmd draft
susan-shu-c Jul 31, 2023
3ad6b5c
Add lmd descriptions
susan-shu-c Jul 31, 2023
218a993
Add package descriptions and subsections
susan-shu-c Jul 31, 2023
84b6512
Update introduction
susan-shu-c Aug 2, 2023
d0eff16
Add datafeeds
susan-shu-c Aug 2, 2023
4fa01e4
Fix typo
susan-shu-c Aug 2, 2023
5242435
Merge branch 'main' into add-package-ml-jobs
susan-shu-c Aug 2, 2023
5dc8ea8
Update headers and address review feedback
susan-shu-c Aug 2, 2023
62ea2d4
Merge branch 'main' into add-package-ml-jobs
susan-shu-c Aug 8, 2023
463960a
Update docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc
susan-shu-c Aug 9, 2023
66997a4
Update docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc
susan-shu-c Aug 9, 2023
5b5bb7d
Update docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc
susan-shu-c Aug 9, 2023
d33917a
Update docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc
susan-shu-c Aug 9, 2023
bf11faa
Update docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc
susan-shu-c Aug 9, 2023
ff9c725
Batch address feedback
susan-shu-c Aug 9, 2023
77555a3
Batch address feedback
susan-shu-c Aug 9, 2023
f542563
Merge branch 'main' into add-package-ml-jobs
susan-shu-c Aug 10, 2023
cd357e6
Merge branch 'main' into add-package-ml-jobs
jmikell821 Aug 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 191 additions & 0 deletions docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,197 @@ they are listed for each job.
|https://github.com/elastic/kibana/blob/{branch}/x-pack/plugins/ml/server/models/data_recognizer/modules/security_windows/ml/v3_windows_rare_user_type10_remote_login.json[image:images/link.svg[A link icon]]
|https://github.com/elastic/kibana/blob/{branch}/x-pack/plugins/ml/server/models/data_recognizer/modules/security_windows/ml/datafeed_v3_windows_rare_user_type10_remote_login.json[image:images/link.svg[A link icon]]

|===
// end::security-windows-jobs[]

[discrete]
[[security-integrations-jobs]]
== Security: Elastic Integrations

https://docs.elastic.co/integrations[Elastic Integrations] are a streamlined way to add Elastic assets to your environment, such as data ingestion, {transforms}, and in this case, {ml} capabilities for Security.

The following Integrations use {ml} to analyze patterns of user and entity behavior, and help detect and alert when there is related suspicious activity in your environment.

* https://docs.elastic.co/integrations/ded[Data Exfiltration Detection]
* https://docs.elastic.co/integrations/dga[Domain Generation Algorithm Detection]
* https://docs.elastic.co/integrations/lmd[Lateral Movement Detection]
* https://docs.elastic.co/integrations/problemchild[Living off the Land Attack Detection]

// dga

*Domain Generation Algorithm (DGA) Detection*

{ml-cap} solution package to detect domain generation algorithm (DGA) activity in your network data. Refer to the {subscriptions}[subscription page] to learn more about the required subscription.

To download, refer to the https://docs.elastic.co/integrations/dga[documentation].

|===
|Name |Description |Job |Datafeed

|dga_high_sum_probability
|Detect domain generation algorithm (DGA) activity in your network data.
|https://github.com/elastic/integrations/blob/{branch}/packages/dga/kibana/ml_module/dga-ml.json#L23[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/dga/kibana/ml_module/dga-ml.json#L58[image:images/link.svg[A link icon]]

|===

// LotL

*Living off the Land Attack (LotL) Detection*

{ml-cap} solution package to detect Living off the Land (LotL) attacks in your environment. Refer to the {subscriptions}[subscription page] to learn more about the required subscription. (Also known as ProblemChild).

To download, refer to the https://docs.elastic.co/integrations/problemchild[documentation].

|===
|Name |Description |Job |Datafeed

|problem_child_rare_process_by_host
|Looks for a process that has been classified as malicious on a host that does not commonly manifest malicious process activity (experimental).
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#L29[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#262[image:images/link.svg[A link icon]]

|problem_child_high_sum_by_host
|Looks for a set of one or more malicious child processes on a single host (experimental).
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#L64[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#262[image:images/link.svg[A link icon]]

|problem_child_rare_process_by_user
|Looks for a process that has been classified as malicious where the user context is unusual and does not commonly manifest malicious process activity (experimental).
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#L106[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#262[image:images/link.svg[A link icon]]

|problem_child_rare_process_by_parent
|Looks for rare malicious child processes spawned by a parent process (experimental).
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#L141[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#262[image:images/link.svg[A link icon]]

|problem_child_high_sum_by_user
|Looks for a set of one or more malicious processes, started by the same user (experimental).
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#L177[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#262[image:images/link.svg[A link icon]]

|problem_child_high_sum_by_parent
|Looks for a set of one or more malicious child processes spawned by the same parent process (experimental).
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#L219[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/problemchild/kibana/ml_module/problemchild-ml.json#262[image:images/link.svg[A link icon]]

|===

// ded

*Data Exfiltration Detection (DED)*

{ml-cap} package to detect data exfiltration in your network and file data. Refer to the {subscriptions}[subscription page] to learn more about the required subscription.

To download, refer to the https://docs.elastic.co/integrations/ded[documentation].

|===
|Name |Description |Job |Datafeed

|high-sent-bytes-destination-geo-country_iso_code
|Detects data exfiltration to an unusual geo-location (by country iso code).
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L44[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|high-sent-bytes-destination-ip
|Detects data exfiltration to an unusual geo-location (by IP address).
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L83[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|high-sent-bytes-destination-port
|Detects data exfiltration to an unusual destination port.
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L119[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|high-sent-bytes-destination-region_name
|Detects data exfiltration to an unusual geo-location (by region name).
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L156[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|high-bytes-written-to-external-device
|Detects data exfiltration activity by identifying high bytes written to an external device.
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L194[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|rare-process-writing-to-external-device
|Detects data exfiltration activity by identifying a file write started by a rare process to an external device.
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L231[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|high-bytes-written-to-external-device-airdrop
|Detects data exfiltration activity by identifying high bytes written to an external device via Airdrop.
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L268[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/ded/kibana/ml_module/ded-ml.json#L304[image:images/link.svg[A link icon]]

|===

// lmd

*Lateral Movement Detection (LMD)*

{ml-cap} package to detect lateral movement based on file transfer activity and Windows RDP events. Refer to the {subscriptions}[subscription page] to learn more about the required subscription.

To download, refer to the https://docs.elastic.co/integrations/lmd[documentation].

|===
|Name |Description |Job |Datafeed

|high-count-remote-file-transfer
|Detects unusually high file transfers to a remote host in the network.
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L24[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-file-size-remote-file-transfer
|Detects unusually high size of files shared with a remote host in the network.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L58[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|rare-file-extension-remote-transfer
|Detects data exfiltration to an unusual destination port.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L92[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|rare-file-path-remote-transfer
|Detects unusual folders and directories on which a file is transferred.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L126[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-mean-rdp-session-duration
|Detects unusually high mean of RDP session duration.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L160[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-var-rdp-session-duration
|Detects unusually high variance in RDP session duration.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L202[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-sum-rdp-number-of-processes
|Detects unusually high number of processes started in a single RDP session.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L244[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|unusual-time-weekday-rdp-session-start
|Detects an RDP session started at an usual time or weekday.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L286[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-rdp-distinct-count-source-ip-for-destination
|Detects a high count of source IPs making an RDP connection with a single destination IP.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L326[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-rdp-distinct-count-destination-ip-for-source
|Detects a high count of destination IPs establishing an RDP connection with a single source IP.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L360[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|high-mean-rdp-process-args
|Detects unusually high number of process arguments in an RDP session.
|https://github.com/elastic/integrations/blob/main/packages/lmd/kibana/ml_module/lmd-ml.json#L394[image:images/link.svg[A link icon]]
|https://github.com/elastic/integrations/blob/{branch}/packages/lmd/kibana/ml_module/lmd-ml.json#L436[image:images/link.svg[A link icon]]

|===
// end::security-windows-jobs[]
// end::siem-jobs[]