-
Notifications
You must be signed in to change notification settings - Fork 115
Add API to list resources under namespace tree #281
Add API to list resources under namespace tree #281
Conversation
Skipping CI for Draft Pull Request. |
034f097
to
2dd6590
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some early comments. I understand and like the overall approach, thanks! I'm not sure what code is boilerplate and what's not, can you maybe add some comments to indicate?
The only thing I'm not seeing (at a quick review) are the RBAC checks - how do we know that the user is authorized to list resource X in namespace Y? (It is safe to assume that this also gives you access to all descendants of Y).
Thanks for this, looks good so far!
// Package consts provides shared static data for the extension server. | ||
package consts | ||
|
||
const ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/master/api/v1alpha2/groupversion_info.go, and use v1alpha2
instead for consistency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed for now. This would be the first introduction of a new API that might iterate differently from the main hnc.x-k8s.io
API. Are you sure you want it to start at 2? If we want to call it part of the original API then maybe we should move the whole thing to v1alpha3
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what code is boilerplate and what's not, can you maybe add some comments to indicate?
I've split the vendor code and the kustomize yaml into separate commits for easier reviewing. Let me know if you'd ultimately like to keep them split or squash them back into one once this is ready to merge.
The only thing I'm not seeing (at a quick review) are the RBAC checks - how do we know that the user is authorized to list resource X in namespace Y? (It is safe to assume that this also gives you access to all descendants of Y).
Good catch, this is intentional for the moment and an open question in the design doc. As it is right now, k8s blocks users from using the new resource endpoints unless they are explicitly granted permission with a role like kubectl -n parentns create role hnc-pods --verb list,watch --resource pods.resources.hnc.x-k8s.io
. This is the least opinionated path and allows for the finest granularity of permission delegation. But it's obviously also a little user-hostile.
There's nothing that can be done from inside the handlers about this because kube-api won't even let the user touch the endpoint without this permission. So one option would to add a routine that auto-creates a catch-all role and rolebinding in each namespace that allows system:authenticated
users to access all these *.resources.hnc.x-k8s.io
endpoints, then have the handlers perform the real RBAC check for the target resource. This would make the new APIs "just work" but it's a little extra complexity and it moves the security boundary away from kube-api and into HNC, which might not be every user's preference.
// Package consts provides shared static data for the extension server. | ||
package consts | ||
|
||
const ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed for now. This would be the first introduction of a new API that might iterate differently from the main hnc.x-k8s.io
API. Are you sure you want it to start at 2? If we want to call it part of the original API then maybe we should move the whole thing to v1alpha3
?
2dd6590
to
1bb67d8
Compare
1bb67d8
to
abb534f
Compare
560a7b0
to
7b8369b
Compare
LMK when you want me to review this again, thanks for your work on it! |
@adrianludwin thanks for your patience! It's been a busy summer. I'm working on unit tests and then this should be ready for review. |
Lol I'm replying to some of these issues months late, no need to explain or
apologize!
…On Fri, Sept 15, 2023, 3:43 p.m. Colleen Murphy ***@***.***> wrote:
@adrianludwin <https://github.com/adrianludwin> thanks for your patience!
It's been a busy summer. I'm working on unit tests and then this should be
ready for review.
—
Reply to this email directly, view it on GitHub
<#281 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE43PZFOJOUNSIMA22SVLNTX2SVXZANCNFSM6AAAAAAYMUNRZY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
7b8369b
to
150b7e4
Compare
Marking this ready for review. It still needs integration tests, but enough of the core functionality is there to get feedback on. Happy to make any changes needed. The commits are divided by components:
I know it's a massive code dump, let me know if there's any way I can organize it to make it easier to review. Happy to jump on a call to work through it as well. Some open questions:
|
150b7e4
to
0307ef0
Compare
0307ef0
to
29cfb4c
Compare
c3c3e5c
to
035f6f0
Compare
You have this right, you could create a role with rules like
or be more fine-grained as in my example
depending on how selective the cluster admin wants to be.
It's technically possible to change the endpoint path but there are a few reasons I don't think it would be a good idea:
I'm not seeing any inline comments, is it possible you didn't submit the review?
Will add in next comment
I've updated it to add some comments on where I've pulled inspiration from in places where I didn't come up with it myself. I no longer have all my research for this in my browser history but I did my best to find sources again. The only parts that I think of as "boilerplate" are things like parsing flags where I've copied from other parts of HNC, and I didn't explicitly comment those parts. Let me know if you want more background on any other parts.
I'll squash this into one commit once we think it's close to ready to merge. |
Output ExamplesDiscoveryAPI
Kubectl
Global Resource ListAPIList
Watch(Secret child1-s2 is created in another session after watch is started)
KubectlList
Watch(Secret child1-s3 is created in another session after watch is started)
Namespaced (Tree) Resource ListAPIList
Watch(Secret grandchild1-s2 is created in another session after watch is started)
KubectlList
Watch(Secret grandchild1-s3 is created in another session after watch is started)
|
c8294b8
to
b78d28d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Uuuuuuuggggh, sorry, forgot to hit submit! I'll reply to your latest comments soon
apiregv1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" | ||
"path/filepath" | ||
"sigs.k8s.io/controller-runtime/pkg/log/zap" | ||
api "sigs.k8s.io/hierarchical-namespaces/api/v1alpha2" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Newline between the external imports and the HNC-internal stuff (here and elsewhere)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -0,0 +1,214 @@ | |||
package apiresources |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all looks very boilerplatey - if it is, do you mind adding a comment that says where this comes from or some kind of resource for it all?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -0,0 +1,122 @@ | |||
package clients |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto for my last comment, this is largely boilerplate, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some comments explaining inspiration/sources
|
||
const ( | ||
allowedCNKey = "requestheader-allowed-names" | ||
workers = int64(3) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not leave this untyped?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
} | ||
|
||
// Forwarder returns an HTTP handler that handles requests for namespaced resources that have no particular namespace specified | ||
// (e.g., kubectl get pods -A). The result is the equivalent of calling list or watch on the original endpoint. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dumb question: can we redirect back to the equivalent apiserver endpoint? Or, what would be the consequence of not implementing this? Just wondering if we can cut down on a bit of implementation (though tbh it wouldn't really save us a ton).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to implement this, and when running it got the error message
Error: an error on the server ("the backend attempted to redirect this request, which is not permitted") has prevented the request from succeeding
It looks like kube-aggregator blocks this by default as of kubernetes/apimachinery@1c318b6 due to https://discuss.kubernetes.io/t/security-advisory-cve-2022-3172-aggregated-api-server-can-cause-clients-to-be-redirected-ssrf/21322
metadata: | ||
labels: | ||
app: resourcelist | ||
name: resourcelist |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: maybe get the term "api extension" in there so it's more obvious what this is?
Also, I think you need name namespace system
in here (Kustomize will add the hnc-
prefix).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
groupPriorityMinimum: 10 | ||
versionPriority: 10 | ||
service: | ||
namespace: hnc-system |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be system
, kustomize should add hnc-
I think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't work, I'm not sure why. Maybe kustomize doesn't know about APIService
?
@@ -0,0 +1,14 @@ | |||
--- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add some comments about what all these RBAC roles are for
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
internal/kubectl/get.go
Outdated
func init() { | ||
getCmd.Flags().BoolVarP(&watch, "watch", "w", false, "watch for changes") | ||
getCmd.PersistentFlags().StringVarP(&parent, "namespace", "n", "default", "parent namespace to scope request to") | ||
getCmd.PersistentFlags().BoolVarP(&allNamespaces, "all-namespaces", "A", false, "list across all namespaces") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add to help text: equivalent to kubectl get <resource> -A
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
apiregv1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" | ||
"path/filepath" | ||
"sigs.k8s.io/controller-runtime/pkg/log/zap" | ||
api "sigs.k8s.io/hierarchical-namespaces/api/v1alpha2" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -0,0 +1,214 @@ | |||
package apiresources |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -0,0 +1,122 @@ | |||
package clients |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some comments explaining inspiration/sources
|
||
const ( | ||
allowedCNKey = "requestheader-allowed-names" | ||
workers = int64(3) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
if objI.GetNamespace() < objJ.GetNamespace() { | ||
return true | ||
} | ||
if objI.GetNamespace() < objJ.GetNamespace() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
internal/kubectl/get.go
Outdated
func init() { | ||
getCmd.Flags().BoolVarP(&watch, "watch", "w", false, "watch for changes") | ||
getCmd.PersistentFlags().StringVarP(&parent, "namespace", "n", "default", "parent namespace to scope request to") | ||
getCmd.PersistentFlags().BoolVarP(&allNamespaces, "all-namespaces", "A", false, "list across all namespaces") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
metadata: | ||
labels: | ||
app: resourcelist | ||
name: resourcelist |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
groupPriorityMinimum: 10 | ||
versionPriority: 10 | ||
service: | ||
namespace: hnc-system |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't work, I'm not sure why. Maybe kustomize doesn't know about APIService
?
@@ -0,0 +1,14 @@ | |||
--- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
name: default | ||
namespace: hnc-system |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI this allows both the new deployment and the existing HNC manager deployment to read resources, even though the latter doesn't need it. Would there be a preference to use a separate service account or separate namespace for the API extension deployment?
b78d28d
to
20d1de5
Compare
@adrianludwin will you be able to take another look here? If not I'll dig in this week. |
/lgtm |
I'll rebase this and take a look at the most recent comment this week. Thanks @rjbez17 for your attention on this. |
This change adds a new command and configuration to register an API endpoint which will allow users to query for resources belonging to descendants of any namespace. This means that finding all resources under a parent namespace can be done in one request, instead finding and iterating over all descendants. The new endpoints are: * /apis/resources.hnc.x-k8s.io/v1alpha2 - discovery endpoint * /apis/resources.hnc.x-k8s.io/v1alpha2/{resource} - global resource query, equivalent to the same global request for the original resource * /apis/resources.hnc.x-k8s.io/v1alpha2/namespaces/{namespace}/{resource} - returns resources in all namespaces under {namespace}, including {namespace}.
Add dependencies and run 'go mod vendor'.
Add a subcommand to list resources by their parent namespace: kubectl hns get -n <parent ns> <resource>
Add kustomization resources for the new Deployment, Service, APIService and cert-manager configuration for the resource list extension.
Add cert-rotator to the setup for the new resource list extension to remove the dependency on cert-manager.
Add integration tests to list and watch resources under a tree. Add test utilities to enable killing a watch command gracefully/
20d1de5
to
78037b5
Compare
Rebased |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cmurphy, rjbez17 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
volumes: | ||
- secret: | ||
defaultMode: 420 | ||
secretName: hnc-resourcelist-apiextension |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmurphy Hi, it seems this secret doesn't exist in the manifest. Where is it?
This change adds a new command and configuration to register an API endpoint which will allow users to query for resources belonging to descendants of any namespace. This means that finding all resources under a parent namespace can be done in one request, instead finding and iterating over all descendants. The new endpoints are:
Design: https://docs.google.com/document/d/1WpnAJ3442v93G4Wi7SnoEiLZVLJJO-AWCkl5GO4YLl4/edit?usp=sharing
Fixes: #235
This is based on the POC in https://github.com/cmurphy/hns-list.
To do: