|
| 1 | +.. _ak8so-stream-processing: |
| 2 | + |
| 3 | +============================== |
| 4 | +Manage {+atlas-sp+} |
| 5 | +============================== |
| 6 | + |
| 7 | +.. default-domain:: mongodb |
| 8 | + |
| 9 | +.. contents:: On this page |
| 10 | + :local: |
| 11 | + :backlinks: none |
| 12 | + :depth: 1 |
| 13 | + :class: singlecol |
| 14 | + |
| 15 | +You can use |ak8so| to manage stream processing instances and |
| 16 | +connections for {+atlas-sp+}. {+atlas-sp+} enables you to process |
| 17 | +streams of complex data using the same :manual:`MongoDB Query API |
| 18 | +</query-api>` that {+service+} databases use. {+atlas-sp+} allows you |
| 19 | +to do the following tasks: |
| 20 | + |
| 21 | +- Build :ref:`aggregation pipelines <stream-aggregation>` to |
| 22 | + continuously operate on streaming data without the delays inherent in |
| 23 | + batch processing. |
| 24 | +- Perform continuous :ref:`schema validation |
| 25 | + <streams-agg-pipeline-validate>` to check that messages are properly |
| 26 | + formed, detect message corruption, and detect late-arriving data. |
| 27 | +- Continuously publish results to |
| 28 | + {+service+} collections or {+kafka+} clusters, ensuring up-to-date |
| 29 | + views and analysis of data. |
| 30 | + |
| 31 | +{+atlas-sp+} components belong directly to |service| projects and |
| 32 | +operate independent of |service| {+clusters+}. To learn more, see |
| 33 | +:ref:`what-is-atlas-sp`. |
| 34 | + |
| 35 | +.. _ak8so-spi-connections: |
| 36 | + |
| 37 | +Stream Processing Instances and Connections |
| 38 | +------------------------------------------- |
| 39 | + |
| 40 | +{+atlas-sp+} instances provide the context for all of your operations |
| 41 | +on streaming data. You can configure a sample connection, a connection |
| 42 | +to an {+service+} :manual:`change stream <changeStreams>`, or a |
| 43 | +connection to an {+kafka+} system. Then you can add the connection to |
| 44 | +the Connection Registry for your stream processing instance. To learn |
| 45 | +more, see :ref:`manage-spi`. |
| 46 | + |
| 47 | +Prerequisites |
| 48 | +------------- |
| 49 | + |
| 50 | +To use |ak8so| to manage stream processing instances and connections, |
| 51 | +you must: |
| 52 | + |
| 53 | +- Have an |service| user with the :authrole:`Project Owner` or |
| 54 | + the :authrole:`Project Stream Processing Owner` role to manage a |
| 55 | + Connection Registry and {+SPI+}. To learn more about the differences |
| 56 | + between the two roles, see :ref:`Project Roles <project-roles>`. |
| 57 | + |
| 58 | +- Have a running |k8s| cluster with |ak8so| :ref:`deployed |
| 59 | + <ak8so-quick-start-ref>`. |
| 60 | + |
| 61 | +- Ensure your |ipaddr| address is in the organization's |api| |
| 62 | + :ref:`access list <access-list>`. |
| 63 | + |
| 64 | +Procedure |
| 65 | +--------- |
| 66 | + |
| 67 | +To use |ak8so| to manage a stream processing instance and its |
| 68 | +connections, do the following steps: |
| 69 | + |
| 70 | +.. procedure:: |
| 71 | + :style: normal |
| 72 | + |
| 73 | + .. step:: Configure the :ref:`atlasstreaminstance-custom-resource`. |
| 74 | + |
| 75 | + **Example:** |
| 76 | + |
| 77 | + .. code-block:: |
| 78 | + |
| 79 | + cat <<EOF | kubectl apply -f - |
| 80 | + apiVersion: atlas.mongodb.com/v1 |
| 81 | + kind: AtlasStreamInstance |
| 82 | + metadata: |
| 83 | + name: my-stream-instance |
| 84 | + spec: |
| 85 | + name: my-stream-instance |
| 86 | + clusterConfig: |
| 87 | + provider: AWS |
| 88 | + region: VIRGINIA_USA |
| 89 | + tier: SP30 |
| 90 | + EOF |
| 91 | + |
| 92 | + To learn more about the available parameters, see the |
| 93 | + :ref:`atlasstreaminstance-custom-resource`. |
| 94 | + |
| 95 | + .. include:: /includes/fact-ak8so-atlas-sp-provider-region.rst |
| 96 | + |
| 97 | + .. step:: Configure the :ref:`atlasstreamconnection-custom-resource`. |
| 98 | + |
| 99 | + You can configure a sample connection, a connection to an |
| 100 | + {+service+} :manual:`change stream <changeStreams>`, or a |
| 101 | + connection to an {+kafka+} system. |
| 102 | + |
| 103 | + **Examples:** |
| 104 | + |
| 105 | + .. tabs:: |
| 106 | + |
| 107 | + .. tab:: Sample |
| 108 | + :tabid: sample |
| 109 | + |
| 110 | + .. code-block:: sh |
| 111 | + |
| 112 | + apiVersion: atlas.mongodb.com/v1 |
| 113 | + kind: AtlasStreamConnection |
| 114 | + metadata: |
| 115 | + name: my-stream-connection |
| 116 | + spec: |
| 117 | + name: sample_stream_solar |
| 118 | + type: Sample |
| 119 | + |
| 120 | + .. include:: /includes/fact-ak8so-atlas-sp-sample.rst |
| 121 | + |
| 122 | + .. tab:: Change Stream |
| 123 | + :tabid: cluster |
| 124 | + |
| 125 | + .. code-block:: sh |
| 126 | + |
| 127 | + apiVersion: atlas.mongodb.com/v1 |
| 128 | + kind: AtlasStreamConnection |
| 129 | + metadata: |
| 130 | + name: my-stream-connection |
| 131 | + spec: |
| 132 | + name: my-stream-connection |
| 133 | + type: Cluster |
| 134 | + clusterConfig: |
| 135 | + name: my-cluster |
| 136 | + role: |
| 137 | + name: my-db-role |
| 138 | + type: CUSTOM |
| 139 | + |
| 140 | + .. tab:: Kafka |
| 141 | + :tabid: kafka |
| 142 | + |
| 143 | + .. code-block:: sh |
| 144 | + apiVersion: atlas.mongodb.com/v1 |
| 145 | + kind: AtlasStreamConnection |
| 146 | + metadata: |
| 147 | + name: my-stream-connection |
| 148 | + spec: |
| 149 | + name: my-stream-connection |
| 150 | + type: Kafka |
| 151 | + kafkaConfig: |
| 152 | + bootstrapServers: "comma,separated,list,of,server,addresses" |
| 153 | + authentication: |
| 154 | + mechanism: SCRAM-512 |
| 155 | + credentials: |
| 156 | + name: ref-to-creds-secret |
| 157 | + namespace: default |
| 158 | + security: |
| 159 | + protocol: SSL |
| 160 | + certificate: |
| 161 | + name: ref-to-certificate-secret |
| 162 | + namespace: default |
| 163 | + |
| 164 | + To learn more about the available parameters, see the |
| 165 | + :ref:`atlasstreamconnection-custom-resource`. |
| 166 | + |
| 167 | + .. step:: Add your connections to the Connection Registry in the :ref:`atlasstreaminstance-custom-resource`. |
| 168 | + |
| 169 | + **Example:** |
| 170 | + |
| 171 | + .. code-block:: sh |
| 172 | + |
| 173 | + cat <<EOF | kubectl apply -f - |
| 174 | + apiVersion: atlas.mongodb.com/v1 |
| 175 | + kind: AtlasStreamInstance |
| 176 | + metadata: |
| 177 | + name: my-stream-instance |
| 178 | + spec: |
| 179 | + name: my-stream-instance |
| 180 | + clusterConfig: |
| 181 | + provider: AWS |
| 182 | + region: VIRGINIA_USA |
| 183 | + tier: SP30 |
| 184 | + connectionRegistry: |
| 185 | + - name: ref-my-connection-1 |
| 186 | + namespace: my-namespace1 |
| 187 | + - name: ref-my-connection-2 |
| 188 | + namespace: my-namespace2 |
| 189 | + - name: ref-my-connection-3 |
| 190 | + namespace: my-namespace1 |
| 191 | + EOF |
| 192 | + |
| 193 | + .. include:: /includes/fact-ak8so-atlas-sp-provider-region.rst |
| 194 | + |
| 195 | + To learn more about the available parameters, see the |
| 196 | + :ref:`atlasstreaminstance-custom-resource`. |
0 commit comments