edit the proposal

Author: charleszheng44

proposals/20201026-creating-control-plane.md

@@ -5,6 +5,8 @@ authors:
 reviewers:
   - "@christopherhein"
   - "@Fei-Guo"
+  - "@vincepri"
+  - "@brightzheng100"
 creation-date: 2020-10-26
 last-updated: 2020-11-09
 status: provisional
@@ -46,6 +48,12 @@ The followings are terms that will be used in the proposal. Whether including th
 
 * **NestedControlPlane(NCP)** - The control plane that are hosted on the super cluster. 
 
+* **NestedEtcd(NEtcd)** - The etcd that belongs to the control plane of the nested cluster.
+
+* **NestedAPIServer(NKAS)** - The kube-apiserver which belongs to the control plane of the nested cluster.
+
+* **NestedControllerManager(NKCM)** - The kube-control-manager which belongs to the control plane of the nested cluster.
+
 * **Component Controller** - Operators that create components of the NCP, including the Etcd controller, KAS controller and KCM controller.
 
 To better illustrate the creation process, we define two Roles with different responsibilities:
@@ -61,7 +69,7 @@ The goal of this proposal is to define CRDs of the three major components (kube-
 
 ## Motivation
 
-CAPN aims at providing control-plane level isolation while sharing physical resources among control planes. There exist various approaches to creating isolated control-planes. For example, one can run components of the nested control-plane as pods on the underlying clusters, create NCPs through cloud providers' Kubernetes services or use third-party component providers to create each component. In this proposal, we try to define CRDs of the NCP's three major components and a standard process of creating the three components regardless of which underlying approach is used. As examples, we introduce two setups that 1) creating each component natively, 2) creating KAM and KCM natively while using the [Etcd-cluster-operator](https://github.com/improbable-eng/etcd-cluster-operator) to create the Etcd.
+CAPN aims at providing control plane level isolation while sharing physical resources among control planes. There exist various approaches to creating isolated control planes. For example, one can run components of the nested control plane as pods on the underlying clusters, create NCPs through cloud providers' Kubernetes services or use third-party component providers to create each component. In this proposal, we try to define CRDs of the NCP's three major components and a standard process of creating the three components regardless of which underlying approach is used. As examples, we introduce two setups that 1) creating each component natively, 2) creating KAM and KCM natively while using the [Etcd-cluster-operator](https://github.com/improbable-eng/etcd-cluster-operator) to create the Etcd.
 
 ### Goals
 
@@ -77,17 +85,17 @@ CAPN aims at providing control-plane level isolation while sharing physical reso
 
 - Define how NCP controller works.
 
-- Discuss the details of how the third-party component provider work.
+- Discuss the implementation details of the out-of-tree component controllers.
 
 ## Proposal
 
 ### Portability and Customizability
 
-Generally, creating the three major components requires similar high-level information, like the components' version, the number of replicas, and the amount of computing resources. Meanwhile, end-users should be able to customize NCP components, i.e., specifying the component image, version, and command-line options. Therefore, we define a new struct `ComponentSpec` that contains common information required by different providers as well as customized information specified by the end-users. The `ComponentSpec` will look like the following
+Generally, creating the three major components requires similar high-level information, like the components' version, the number of replicas, and the amount of computing resources. Meanwhile, end-users should be able to customize NCP components, i.e., specifying the component image, version, and command-line options. Therefore, we define a new struct `NestedComponentSpec` that contains common information required by different providers as well as customized information specified by the end-users. The `NestedComponentSpec` will look like the following
 
 ```go
-type ComponentSpec struct {
-    // ComponentSpec defines the common information for creating the component
+type NestedComponentSpec struct {
+    // NestedComponentSpec defines the common information for creating the component
     // +optional
     addonv1alpha1.CommonSpec `json:",inline"`
 
@@ -129,19 +137,58 @@ type PatchSpec struct {
 
 #### Create prerequisites 
 
-We assume that the APIServer, ContollerManager, Etcd and the NCP CR are located in the same namespace. To create an NCP, we need to first create APIserver CR, ControllerManager CR, Etcd CR, NCP CR, and a namespace that holds all the CRs, then the sub-controllers can cooperate with each other to create components for the NCP. 
+We assume that the APIServer, ContollerManager, Etcd and the NCP CR are located in the same namespace. To create an NCP, we need to first create APIserver CR, ControllerManager CR, Etcd CR, NCP CR, and a namespace that holds all the CRs, then the component controller can cooperate with each other to create components for the NCP. 
 
-As there exist dependencies between components, i.e., KAS cannot run without Etcd, KCM cannot work without KAS, when creating the NCP component, sub-controllers will need to get information and status of other CRs. Also, as all CRs of an NCP will be located in the same namespace, we will set `metav1.OwnerReference` in each component CR as the NCP that owns the CR, which can help the CR to find its belonging NCP. As the NCP status will include all required [CAPI status](https://cluster-api.sigs.k8s.io/developer/architecture/controllers/control-plane.html#required-status-fields), we can then use the `Selector` field to get the desired CR.
+As there exist dependencies between components, i.e., KAS cannot run without Etcd, KCM cannot work without KAS, when creating NCP components, component controllers will need to get information and status of other CRs. Also, as all CRs of an NCP will be located in the same namespace, we will set `metav1.OwnerReference` in each component CR as the NCP that owns the CR, which can help the CR to find its belonging NCP. As the NCP status will include all required [CAPI status](https://cluster-api.sigs.k8s.io/developer/architecture/controllers/control-plane.html#required-status-fields), we can then use the `Selector` field to get the desired CR.
 
 ### Creation
 
-End-users can create component CRs manually and apply them to the cluster with an NCP to create the resources. In the future, we might introduce the `Template` CR, which will handle the creation of the component CRs in it's controller. We assume that on a super cluster, there will be only one sub-controller for each component CR, and it's the cluster admin's responsibility to setup sub-controllers. 
+End-users can create component CRs manually and apply them to the cluster with an NCP to create the resources. In the future, we might introduce the `Template` CR, which will handle the creation of the component CRs in it's controller. If the end-user intend to use out-of-tree controllers, they can specify them in the NCP spec, we define three new types:
+
+```go
+type NestedKASProvider string
+
+type NestedEtcdProvider string
+
+type NestedKCMProvider string
+
+const (
+    NativeKASProvider NestedKASProvider = "NativeKASProvider"
+
+    NativeEtcdProvider  NestedEtcdProvider = "NativeEtcdProvider"
+    EtcdClusterOperator NestedEtcdProvider = "EtcdClusterOperator"
+    EtcdOperator        NestedEtcdProvider = "EtcdOperator"
+
+    NativeKCMProvider NestedKCMProvider = "NativeKCMProvider"
+) 
+```
+
+and adds three new fields into the NCP spec:
+
+```go
+type NestedControlPlaneSpec struct {
+    // other fields ...
+
+    // EtcdProvider specifies which EtcdProvider will be used to create the Etcd
+    // +optional
+    EtcdProvider NestedEtcdProvider `json:"etcdProvider,omitempty"`
+     
+    // ApiServerProvider specifies which KASProvider will be used to create the kube-apiserver
+    // +optional
+    ApiServerProvider NestedKASProvider `json:"apiServerProvider,omitempty"`
+ 
+    // ControllerManagerProvider specifies which KCMProvider will be used to create the kube-controller-manager
+    // +optional
+    ControllerManagerProvider NestedKCMProvider `json:"controllerManagerProvider,omitempty"`
+}
+```
+
 
 #### Native way 
 
-If no component provider is specified, the sub-controller will create the component under the native mode, which will create the component using the default manifests. The readiness and liveness probe will be used, and we will mark each component as ready only when the corresponding workload is ready. As the KAS cannot work without available Etcd and the KCM cannot run without KAS, the three components need to be created by their respective controllers in the order of Etcd, KAS, and KCM. Creation order is maintained using cross resource status checks to "wait" until the dependencies are provisioned before booting.
+If no component provider is specified, the component controller will create the component under the native mode, which will create the component using the default manifests. The readiness and liveness probe will be used, and we will mark each component as ready only when the corresponding workload is ready. As the KAS cannot work without available Etcd and the KCM cannot run without KAS, the three components need to be created by their respective controllers in the order of Etcd, KAS, and KCM. Creation order is maintained using cross resource status checks to "wait" until the dependencies are provisioned before booting.
 
-Except using `OwnerReference` and `CAPI.Status.Selector`, another option of sharing status across components are adding initContainer to each component. The sub-controllers will then create the component workload without polling the status of other CRs while having the initContainer to check and wait for other components to be ready. 
+Except using `OwnerReference` and `CAPI.Status.Selector`, another option of sharing status across components are adding initContainer to each component. The component controllers will then create the component workload without polling the status of other CRs while having the initContainer to check and wait for other components to be ready. 
 
 We will host sets of default templates in this repositor. Users can specify which set of templates they intend to use by specifying the corresponding `version` or `channel` in the embedded `CommonSpec` in the component's CR.
 
@@ -185,18 +232,18 @@ In the following example, we assume that the user intend to use Etcd-cluster-ope
 
 The followings are CRDs of the three components.
 
-### Etcd CRD
+### NestedEtcd CRD
 ```go
-// EtcdSpec defines the desired state of Etcd
-type EtcdSpec struct {
-    // ComponentSpec contains the common and user-specified information that are 
+// NestedEtcdSpec defines the desired state of Etcd
+type NestedEtcdSpec struct {
+    // NestedComponentSpec contains the common and user-specified information that are 
     // required for creating the component
     // +optional
-    ComponentSpec ComponentSpec  `json:",inline"`
+    NestedComponentSpec  `json:",inline"`
 }
 
-// EtcdStatus defines the observed state of Etcd
-type EtcdStatus struct {
+// NestedEtcdStatus defines the observed state of Etcd
+type NestedEtcdStatus struct {
     // Ready is set if all resources have been created
     Ready bool `json:"ready,omitempty"`
 
@@ -207,14 +254,14 @@ type EtcdStatus struct {
     ProviderObject *corev1.ObjectReference `json:referralObject,omitempty`
 
     // EtcdDomain defines how to address the etcd instance
-    Addresses []EtcdAddress `json:"addresses,omitempty"`
+    Addresses []NestedEtcdAddress `json:"addresses,omitempty"`
 
     // CommonStatus allows addons status monitoring 
     CommonStatus addonv1alpha1. CommonStatus `json:",inline"`
 }
 
 // EtcdAddress defines the observed addresses for etcd
-type EtcdAddress struct {
+type NestedEtcdAddress struct {
     // IP Address of the etcd instance.
     // +optional
     IP string `json:"ip,omitempty"`
@@ -227,8 +274,8 @@ type EtcdAddress struct {
     Port int32 `json:"port"`
 }
 
-// Etcd is the Schema for the Etcd API
-type Etcd struct {
+// NestedEtcd is the Schema for the Etcd API
+type NestedEtcd struct {
     metav1.TypeMeta   `json:",inline"`
     metav1.ObjectMeta `json:"metadata,omitempty"`
 
@@ -237,18 +284,18 @@ type Etcd struct {
 }
 ```
 
-### APIServer CRD
+### NestedAPIServer CRD
 
 ```go
-type APIServerSpec struct {
-    // ComponentSpec contains the common and user-specified information that are 
+type NestedAPIServerSpec struct {
+    // NestedComponentSpec contains the common and user-specified information that are 
     // required for creating the component
     // +optional
-    ComponentSpec ComponentSpec  `json:",inline"`
+    NestedComponentSpec  `json:",inline"`
 }
 
-// APIServerStatus defines the observed state of APIServer 
-type APIServerStatus struct {
+// NestedAPIServerStatus defines the observed state of APIServer 
+type NestedAPIServerStatus struct {
     // Ready is set if all resources have been created
     // +kubebuilder:default=false
     Ready bool `json:"ready,omitempty"`
@@ -266,29 +313,29 @@ type APIServerStatus struct {
     CommonStatus addonv1alpha1. CommonStatus `json:",inline"`
 }
 
-// APIServer is the Schema for the APIServers API
-type APIServer struct {
+// NestedAPIServer is the Schema for the APIServers API
+type NestedAPIServer struct {
     metav1.TypeMeta   `json:",inline"`
     metav1.ObjectMeta `json:"metadata,omitempty"`
 
-    Spec   APIServerSpec   `json:"spec,omitempty"`
-    Status APIServerStatus `json:"status,omitempty"`
+    Spec   NestedAPIServerSpec   `json:"spec,omitempty"`
+    Status NestedAPIServerStatus `json:"status,omitempty"`
 }
 ```
 
-### ControllerManager CRD
+### NestedControllerManager CRD
 
 ```go
-// ControllerManagerSpec defines the desired state of ControllerManager
-type ControllerManagerSec struct {
-    // ComponentSpec contains the common and user-specified information that are 
+// NestedControllerManagerSpec defines the desired state of ControllerManager
+type NestedControllerManagerSec struct {
+    // NestedComponentSpec contains the common and user-specified information that are 
     // required for creating the component
     // +optional
-    ComponentSpec ComponentSpec  `json:",inline"`
+    NestedComponentSpec  `json:",inline"`
 }
 
-// ControllerManagerStatus defines the observed state of ControllerManager
-type ControllerManagerStatus struct {
+// NestedControllerManagerStatus defines the observed state of ControllerManager
+type NestedControllerManagerStatus struct {
     // Ready is set if all resources have been created
     Ready bool `json:"ready,omitempty"`
 
@@ -301,13 +348,13 @@ type ControllerManagerStatus struct {
     CommonStatus addonv1alpha1. CommonStatus `json:",inline"`
 }
 
-// ControllerManager is the Schema for the ControllerManagers API
-type ControllerManager struct {
+// NestedControllerManager is the Schema for the ControllerManagers API
+type NestedControllerManager struct {
     metav1.TypeMeta   `json:",inline"`
     metav1.ObjectMeta `json:"metadata,omitempty"`
 
-    Spec   ControllerManagerSpec   `json:"spec,omitempty"`
-    Status ControllerManagerStatus `json:"status,omitempty"`
+    Spec   NestedControllerManagerSpec   `json:"spec,omitempty"`
+    Status NestedControllerManagerStatus `json:"status,omitempty"`
 }
 ```