adding initial NCP + NC proposal

christopherhein · christopherhein · commit 9ce568e8e4e2 · 2021-01-29T11:30:28.000-08:00
Signed-off-by: Chris Hein &lt;me@chrishein.com&gt;
diff --git a/proposals/20210126-nc-and-ncp.md b/proposals/20210126-nc-and-ncp.md
@@ -0,0 +1,218 @@
+---
+title: CAPN NestedCluster & NestedControlPlane 
+authors:
+  - "@christopherhein"
+reviewers:
+  - "@Fei-Guo"
+  - "@charleszheng44"
+  - "@salaxander"
+  - "@vincepri"
+creation-date: 2020-10-21
+last-updated: 2021-01-26
+status: provisional
+see-also:
+- https://sigs.k8s.io/cluster-api-provider-nested/proposals/20201026-creating-control-plane-components.md
+replaces: []
+superseded-by: []
+---
+
+# CAPN NestedCluster & NestedControlPlane
+
+## Table of Contents
+
+<!--ts-->
+
+   * [CAPN NestedCluster &amp; NestedControlPlane](#capn-nestedcluster--nestedcontrolplane)
+      * [Table of Contents](#table-of-contents)
+      * [Glossary](#glossary)
+      * [Summary](#summary)
+      * [Motivation](#motivation)
+         * [Goals](#goals)
+         * [Non-Goals/Future Work](#non-goalsfuture-work)
+      * [Proposal](#proposal)
+         * [User Stories](#user-stories)
+         * [Features from user stories](#features-from-user-stories)
+            * [NestedCluster](#nestedcluster)
+            * [NestedControlPlane](#nestedcontrolplane)
+         * [Requirements (Optional)](#requirements-optional)
+         * [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
+         * [Risks and Mitigations](#risks-and-mitigations)
+      * [Upgrade Strategy](#upgrade-strategy)
+         * [Version Skew Strategy](#version-skew-strategy)
+      * [Implementation History](#implementation-history)
+
+<!--te-->
+
+## Glossary
+
+Refer to the [Cluster API Provider Nested Glossary](/proposals/00_capn-glossary.md).
+
+If this proposal adds new terms, or defines some, make the changes to the book's glossary when in the PR stage.
+
+## Summary
+
+This proposal outlines the base architecture for Cluster API Provider Nested (CAPN). CAPN  enables you to use Cluster API to manage control planes that are run as Pods "nested" within a [super cluster](/proposals/00_capn-glossary.md#super-cluster). These nested control planes, paired with a "sync controller" (PROPOSAL LINK TBD), that allows the super cluster to run workloads that have been scheduled against the CAPN provided control planes.
+
+Doing this allows you to take a multi-tenant super cluster and break it up into multiple single-tenant control planes, thus allowing the users of those control planes the ability to use cluster level resources like CRDs, AdmissionWebhooks, Cluster RBAC and more, while still getting the benefits of utilization of multi-tenancy clusters.
+
+## Motivation
+
+This project is based after the work from the [multi-tenancy working group](http://sigs.k8s.io/multi-tenancy) on `virtualcluster`. Virtual cluster was implemented with a component internally called `vc-manager` which allowed you to provision a pod based control plane with the exception of the `kube-scheduler`. The motivation for CAPN is to reimplement this design with CAPI in-mind.
+
+### Goals
+
+- To design a new custom resource definition to orchestrate the creation of the nested control planes.
+- To design a new custom resource definition to orchestrate the creation of the "cluster"
+- To enable declarative orchestrated control plane upgrades for apiserver and controller manager pods in built-in controllers.
+- To support managing custom cluster add ons in the CAPN control plane, such as DNS, auth, etc using CAPI's `ClusterResourceSet`.
+
+### Non-Goals/Future Work
+
+Non-goals are limited to the scope of this document, these features will evolve
+over time.
+
+- To support managing real nodes in CAPN. _Technically, CAPN can use real nodes. But we have to incorporate an entire new machine provision mechanism/workflow which we leave as the future work._
+- To support managing components running in the super cluster worker nodes. _All the node plugins such as Kubeproxy, CNI/CSI will be managed in the super cluster by the super cluster administrator. They cannot be configured through the CAPN APIs._
+- To support a separate scheduler in each CAPN control plane. _There are few use cases which require two level schedulers but it is out of the scope for CAPN as of now._ 
+- Define how each [component controller](/proposals/00_capn-glossary.md#component-controller) operates, this is defined in [Creating Control Plane Components](/proposals/20201026-creating-control-plane-components.md)
+
+
+## Proposal
+
+### User Stories
+
+1. As a control plane operator, I want to be able to set the namespace where my cluster is provisioned
+2. As a control plane operator, I want to be able to set the name of my control plane
+3. As a control plane operator, I want to be able to specify the [component controllers](/proposals/00_capn-glossary.md#component-controller) for my cluster.
+4. As a machine, I want to know how to address my control plane.
+
+### Features from user stories
+
+#### NestedCluster
+
+Kubernetes API Group: `infrastructure.cluster.x-k8s.io/v1alpha4`
+
+This resource is responsible for getting the Cluster API `Cluster` type and setting the `spec.infrasteructureRef` pointing back to to this `NestedCluster`, this allows the CAPI controllers to understand the state of the cluster. As well It listens to the `NestedControlPlane` resource which is associated and keeps it's status in sync as the cluster is provided.
+
+
+```go=
+type NestedClusterSpec struct {
+	// ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.
+	// +optional
+	ControlPlaneEndpoint clusterv1.APIEndpoint `json:"controlPlaneEndpoint"`
+}
+
+type NestedClusterStatus struct {
+	// Ready is when the NestedControlPlane has a API server URL.
+	// +optional
+	Ready bool `json:"ready,omitempty"`
+}
+
+type NestedCluster struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   NestedClusterSpec   `json:"spec,omitempty"`
+	Status NestedClusterStatus `json:"status,omitempty"`
+}
+```
+
+#### NestedControlPlane
+
+Kubernetes API Group: `controlplane.cluster.x-k8s.io/v1alpha4`
+
+The NestedControlPlane resource is responsible for orchestrating the overarching cluster, this controller doesn't create any of the downstream objects instead it creates a place where the downstream objects can look up shared values. 
+
+**Question:**
+1. should we have information necessary for the syncer within this object? 
+
+```go=
+type NestedControlPlaneSpec struct {
+	// NamespacePrefix is the namespace where the control plane is deployed as well as the prefix ++ tenant control plane namespace name for all workloads 
+	// +optional
+	NamespacePrefix string `json:"namespacePrefix,omitempty"`
+    
+	// ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.
+	// +optional
+	ControlPlaneEndpoint clusterv1.APIEndpoint `json:"controlPlaneEndpoint"`
+
+    
+	// EtcdRef is the reference to the NestedEtcd 
+	EtcdRef *corev1.ObjectReference `json:"etcd,omitempty"` 
+    
+	// APIServerRef is the reference to the NestedAPIServer 
+	// +optional
+	APIServerRef *corev1.ObjectReference `json:"apiserver,omitempty"` 
+    
+	// ContollerManagerRef is the reference to the NestedControllerManager
+	// +optional
+	ControllerManagerRef *corev1.ObjectReference `json:"controllerManager,omitempty"`
+}
+
+type NestedControlPlaneStatus struct {
+	// ExternalManagedControlPlane indicates to cluster-api that the control plane
+	// is managed by an external service such as AKS, EKS, GKE, etc.
+	// +kubebuilder:default=true
+	ExternalManagedControlPlane *bool `json:"externalManagedControlPlane,omitempty"`
+    
+	// Initialized denotes whether or not the control plane has the
+	// uploaded kubernetes config-map.
+	// +optional
+	Initialized bool `json:"initialized"`
+    
+	// Ready denotes that the AWSManagedControlPlane API Server is ready to
+	// receive requests and that the VPC infra is ready.
+	// +kubebuilder:default=false
+	Ready bool `json:"ready"`
+    
+	// ErrorMessage indicates that there is a terminal problem reconciling the
+	// state, and will be set to a descriptive error message.
+	// +optional
+	FailureMessage *string `json:"failureMessage,omitempty"`
+    
+	// Conditions specifies the cpnditions for the managed control plane
+	Conditions clusterv1.Conditions `json:"conditions,omitempty"`
+}
+```
+
+
+### Requirements (Optional)
+
+Each NestedControlPlane MUST have `ExternalManagedControlPlane` set to true.
+
+### Implementation Details/Notes/Constraints
+
+- What are some important details that didn't come across above.
+- What are the caveats to the implementation?
+- Go in to as much detail as necessary here.
+- Talk about core concepts and how they releate.
+
+
+### Risks and Mitigations
+
+- What are the risks of this proposal and how do we mitigate? Think broadly.
+- How will UX be reviewed and by whom?
+- How will security be reviewed and by whom?
+- Consider including folks that also work outside the SIG or subproject.
+
+## Upgrade Strategy
+
+Both `NestedCluster` & `NestedControlPlane` resources aren't actually responsible for the upgrading of the physical components, this is left to the downstream component controllers. When they recieve an updated CR for `NestedEtcd` or `NestedAPIserver` those controllers are responsible for keeping the control planes updated. The `NestedControlPlane` and `NestedCluster` both need to be kept insync if update do happen to make sure `spec.controlPlaneEndpoint` is still pointed at the right access.
+
+
+### Version Skew Strategy 
+
+Version skew is a problem that can easily happen, since the NCP doesn't prescribe the overall version of the stack, these musy me managed manually by the individual component controllers and the 
+
+## Implementation History
+
+- [x] 10/21/2020: Proposed idea in an issue or [community meeting]
+- [x] 01/11/2021: Compile a Google Doc following the CAEP template (link here)
+- [ ] MM/DD/YYYY: First round of feedback from community
+- [ ] MM/DD/YYYY: Present proposal at a [community meeting]
+- [ ] MM/DD/YYYY: Open proposal PR
+
+<!-- Links -->
+[community meeting]: https://docs.google.com/document/d/10aTeq2lhXW_3aFQAd_MdGjY8PtZPslKhZCCcXxFp3_Q/edit#heading=h.ejz1103gmaij
+
+
