Skip to content

Some cleanup for Manage data -> Ingest from apps #722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 2 additions & 12 deletions manage-data/ingest/ingesting-data-from-applications.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,12 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-ingest-guides.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-ingest-guides.html
applies_to:
stack: ga
---

# Ingesting data from applications

% What needs to be done: Refine

% GitHub issue: docs-projects#387

% Scope notes: The ESS and ECE versions should be identical, but they'll still need to be combined.

% Use migrated content from existing pages that map to this page:

% - [ ] ./raw-migrated-files/cloud/cloud/ec-ingest-guides.md
% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-ingest-guides.md


The following tutorials demonstrate how you can use the Elasticsearch language clients to ingest data from an application into a deployment in {{ech}} or {{ece}}.

[Ingest data with Node.js](ingesting-data-from-applications/ingest-data-with-nodejs-on-elasticsearch-service.md)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,61 +2,12 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-getting-started-search-use-cases-beats-logstash.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-search-use-cases-beats-logstash.html
applies_to:
stack: ga
---

# Ingest data from Beats with Logstash as a proxy

% What needs to be done: Refine

% Scope notes: Merge ESS and ECE versions (should be pretty much identical)

% Use migrated content from existing pages that map to this page:

% - [ ] ./raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-beats-logstash.md
% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-beats-logstash.md

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):

$$$ec-beats-logstash-connect-securely$$$

$$$ec-beats-logstash-elasticsearch$$$

$$$ec-beats-logstash-filebeat$$$

$$$ec-beats-logstash-listen$$$

$$$ec-beats-logstash-logstash$$$

$$$ec-beats-logstash-metricbeat-send$$$

$$$ec-beats-logstash-metricbeat$$$

$$$ec-beats-logstash-stdout$$$

$$$ec-beats-logstash-trial$$$

$$$ec-beats-logstash-view-kibana$$$

$$$ece-beats-logstash-connect-securely$$$

$$$ece-beats-logstash-deployment$$$

$$$ece-beats-logstash-elasticsearch$$$

$$$ece-beats-logstash-filebeat$$$

$$$ece-beats-logstash-listen$$$

$$$ece-beats-logstash-logstash$$$

$$$ece-beats-logstash-metricbeat-send$$$

$$$ece-beats-logstash-metricbeat$$$

$$$ece-beats-logstash-stdout$$$

$$$ece-beats-logstash-view-kibana$$$

This guide explains how to ingest data from Filebeat and Metricbeat to {{ls}} as an intermediary, and then send that data to your {{ech}} or {{ece}} deployment. Using {{ls}} as a proxy limits your Elastic stack traffic through a single, external-facing firewall exception or rule. Consider the following features of this type of setup:

* You can send multiple instances of Beats data through your local network’s demilitarized zone (DMZ) to {{ls}}. {{ls}} then acts as a proxy through your firewall to send the Beats data to your deployment, as shown in the following diagram:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,44 +2,12 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-getting-started-search-use-cases-db-logstash.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-search-use-cases-db-logstash.html
applies_to:
stack: ga
---

# Ingest data from a relational database

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):

$$$ec-db-logstash-connect-securely$$$

$$$ec-db-logstash-database-structure$$$

$$$ec-db-logstash-database$$$

$$$ec-db-logstash-driver$$$

$$$ec-db-logstash-output$$$

$$$ec-db-logstash-pipeline$$$

$$$ec-db-logstash-prerequisites$$$

$$$ec-db-logstash-trial$$$

$$$ece-db-logstash-connect-securely$$$

$$$ece-db-logstash-database-structure$$$

$$$ece-db-logstash-database$$$

$$$ece-db-logstash-deployment$$$

$$$ece-db-logstash-driver$$$

$$$ece-db-logstash-output$$$

$$$ece-db-logstash-pipeline$$$

$$$ece-db-logstash-prerequisites$$$

This guide explains how to ingest data from a relational database into {{ecloud}} through [{{ls}}](logstash://reference/index.md), using the Logstash [JDBC input plugin](logstash://reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an {{ech}} or {{ece}} deployment.

The code and methods presented here have been tested with MySQL. They should work with other relational databases.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-getting-started-node-js.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-node-js.html
applies_to:
stack: ga
---

# Ingest data with Node.js
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-getting-started-python.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-python.html
applies_to:
stack: ga
---

# Ingest data with Python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,12 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-getting-started-search-use-cases-node-logs.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-search-use-cases-node-logs.html
applies_to:
stack: ga
---

# Ingest logs from a Node.js web application using Filebeat

% What needs to be done: Refine

% Scope notes: Merge ESS and ECE versions (should be pretty much identical)

% Use migrated content from existing pages that map to this page:

% - [ ] ./raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-node-logs.md
% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-node-logs.md

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):

$$$ec-node-logs-connect-securely$$$

$$$ec-node-logs-create-request-script$$$

$$$ec-node-logs-create-server-script$$$

$$$ec-node-logs-filebeat$$$

$$$ec-node-logs-prerequisites$$$

$$$ec-node-logs-send-ess$$$

$$$ec-node-logs-trial$$$

$$$ec-node-logs-view-kibana$$$

$$$ece-node-logs-connect-securely$$$

$$$ece-node-logs-create-request-script$$$

$$$ece-node-logs-create-server-script$$$

$$$ece-node-logs-deployment$$$

$$$ece-node-logs-filebeat$$$

$$$ece-node-logs-prerequisites$$$

$$$ece-node-logs-send-ess$$$

$$$ece-node-logs-view-kibana$$$

This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an {{ech}} or {{ece}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md#_get_started).

*Time required: 1.5 hours*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,37 +2,12 @@
mapped_urls:
- https://www.elastic.co/guide/en/cloud/current/ec-getting-started-search-use-cases-python-logs.html
- https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-search-use-cases-python-logs.html
applies_to:
stack: ga
---

# Ingest logs from a Python application using Filebeat

% What needs to be done: Refine

% Scope notes: Merge ESS and ECE versions (should be pretty much identical)

% Use migrated content from existing pages that map to this page:

% - [ ] ./raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-python-logs.md
% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-python-logs.md

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):

$$$ec-python-logs-create-script$$$

$$$ec-python-logs-filebeat$$$

$$$ec-python-logs-send-ess$$$

$$$ec-python-logs-view-kibana$$$

$$$ece-python-logs-create-script$$$

$$$ece-python-logs-filebeat$$$

$$$ece-python-logs-send-ess$$$

$$$ece-python-logs-view-kibana$$$

This guide demonstrates how to ingest logs from a Python application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md).

*Time required: 1 hour*
Expand Down
Loading