You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
*[Advanced control during rolling upgrades](#k8s-advanced-upgrade-control)
18
19
19
20
## NodeSets overview [k8s-nodesets]
20
21
@@ -179,89 +180,16 @@ Operations that reduce the number of nodes in the cluster cannot make progress w
179
180
* Adjust the Elasticsearch [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to a number of replicas that allow the desired node removal.
180
181
* Use [`auto_expand_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md) to automatically adjust the replicas to the number of data nodes in the cluster.
181
182
182
-
183
183
## Advanced control during rolling upgrades [k8s-advanced-upgrade-control]
184
184
185
-
The rules (otherwise known as predicates) that the ECK operator follows during an Elasticsearch upgrade can be selectively disabled for certain scenarios where the ECK operator will not proceed with an Elasticsearch cluster upgrade because it deems it to be "unsafe".
186
-
187
-
::::{warning}
188
-
Selectively disabling the predicates listed in this section are extremely risky, and carry a high chance of either data loss, or causing a cluster to become completely unavailable. Use them only if you are sure that you are not causing permanent damage to an Elasticsearch cluster. These predicates might change in the future. We will be adding, removing, and renaming these over time, so be careful in adding these to any automation. Also, make sure you remove them after use. `kublectl annotate elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch-sample eck.k8s.elastic.co/disable-upgrade-predicates-`
189
-
::::
190
-
191
-
192
-
* The following named predicates control the upgrade process
Upgrade the frozen tier first, then the cold tier, then the warm tier, and the hot tier last. This ensures ILM can continue to move data through the tiers during the upgrade.
During a rolling upgrade, primary shards assigned to a node running a new version cannot have their replicas assigned to a node with the old version. Therefore we must allow some Pods to be restarted even if cluster health is yellow so the replicas can be assigned.
213
-
214
-
* require_started_replica
215
-
216
-
If a cluster is yellow, allow deleting a node, but only if they do not contain the only replica of a shard since it would make the cluster go red.
217
-
218
-
* one_master_at_a_time
219
-
220
-
Only allow a single master to be upgraded at a time.
Force an upgrade of all the master-ineligible nodes before upgrading the last master-eligible node.
185
+
During {{es}} rolling upgrades, ECK follows a set of rules (also known as predicates) to ensure the upgrade process is safe and does not put the cluster at risk. For example, one of these predicates ensures that only a single master node is upgraded at a time, while another prevents nodes from being restarted if the cluster is in a red state.
225
186
226
-
* do_not_delete_pods_with_same_shards
227
-
228
-
Do not allow two pods containing the same shard to be deleted at the same time.
229
-
230
-
* do_not_delete_all_members_of_a_tier
231
-
232
-
Do not delete all nodes that share the same node roles at once. This ensures that there is always availability for each configured tier of nodes during a rolling upgrade.
233
-
234
-
235
-
Any of these predicates can be disabled by adding an annotation with the key of `eck.k8s.elastic.co/disable-upgrade-predicates` to the Elasticsearch metadata, specifically naming the predicate that is needing to be disabled. Also, all predicates can be disabled by replacing the name of any predicatae with "*".
236
-
237
-
* Example use case
238
-
239
-
Assume a given Elasticsearch cluster is a "red" state because of an un-allocatable shard setting that was applied to the cluster:
240
-
241
-
```json
242
-
{
243
-
"settings": {
244
-
"index.routing.allocation.include._id": "does not exist"
245
-
}
246
-
}
247
-
```
248
-
249
-
This cluster would never be allowed to be upgraded with the standard set of upgrade predicates in place, as the cluster is in a "red" state, and the named predicate `only_restart_healthy_node_if_green_or_yellow` prevents the upgrade.
250
-
251
-
If the following annotation was added to the cluster specification, and the version was increased from 7.15.2 → 7.15.3
# Also note that eck.k8s.elastic.co/disable-upgrade-predicates: "*" would work as well, but is much less selective.
261
-
spec:
262
-
version: 7.15.3 # previously set to 7.15.2, for example
263
-
```
264
-
265
-
The ECK operator would allow this upgrade to proceed, even though the cluster was in a "red" state during this upgrade process.
187
+
These predicates can be selectively disabled for certain scenarios where the ECK operator will not proceed with an Elasticsearch cluster upgrade because it deems it to be "unsafe".
266
188
189
+
For a complete list of available predicates, their meaning, and example usage, refer to [ECK upgrade predicates](cloud-on-k8s://reference/upgrade-predicates.md).
267
190
191
+
::::{warning}
192
+
* Selectively disabling the predicates is extremely risky, and carry a high chance of either data loss, or causing a cluster to become completely unavailable. Use them only if you are sure that you are not causing permanent damage to an Elasticsearch cluster.
193
+
* These predicates might change in the future. We will be adding, removing, and renaming these over time, so be careful in adding these to any automation.
194
+
* Also, make sure you remove them after use by running `kublectl annotate elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch-sample eck.k8s.elastic.co/disable-upgrade-predicates-`
0 commit comments