KEP-4633: Only allow anonymous auth for configured endpoints.

[vinayakankugoyal]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds the functionality of kubernetes/enhancements#4633 behind a feature gate. When the feature gate is enabled this PR allows a restricted anonymous auth mode that only allows a set of configured endpoints to be reached anonymously.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/issues/4633

[vinayakankugoyal]

Testing Notes:

1. Created an image with the changes from my branch using

$ kind build node-image .

2. Created a kind cluster with this node image

kind create cluster --image kindest/node:latest

3. Get the IP for the API server

kubectl cluster-info                                                                                
Kubernetes control plane is running at https://127.0.0.1:33559

4. Anonymous request to readyz should succeed

curl -k https://127.0.0.1:33559/readyz                                                                                   
ok%                                                                                                                                                                    

5. But listing all pods should fail

curl -k https://127.0.0.1:33559/api/v1/pods                                                                               ok 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}%                                                   

6. create a cluster role and cluster role binding that allows system:anonymous to view pods

kubectl create clusterrole --resource pods --verb get --verb list podview                                  
clusterrole.rbac.authorization.k8s.io/podview created

kubectl create clusterrolebinding --clusterrole=podview --user system:anonymous anon                      
clusterrolebinding.rbac.authorization.k8s.io/anon created

7. now try to list pods again

curl -k https://127.0.0.1:33559/api/v1/pods | jq ".items[] | .metadata.name"                                              ok 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  134k    0  134k    0     0  10.3M      0 --:--:-- --:--:-- --:--:-- 10.9M
"coredns-7db6d8ff4d-c5272"
"coredns-7db6d8ff4d-qv8j2"
"etcd-kind-control-plane"
"kindnet-q7mfw"
"kube-apiserver-kind-control-plane"
"kube-controller-manager-kind-control-plane"
"kube-proxy-jjlnd"
"kube-scheduler-kind-control-plane"
"local-path-provisioner-c9cbdf4f9-5hlt8"

8. now lets set --anonymous-auth-health-only=true in API server without setting the feature gate. This should fail.

docker exec -it $(docker container ls | grep kind-control-plane | awk '{print $1}') /bin/bash 

root@kind-control-plane:/# apt-get update

root@kind-control-plane:/# apt-get install vim

root@kind-control-plane:/# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

I updated the kube-apiserver container with the argument --anonymous-auth-health-only=true

crictl ps -a --name  kube-apiserver
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
9de3341a21eb8       d9ccda0175e15       11 seconds ago      Exited              kube-apiserver      2                   e3d2f027fab3a       kube-apiserver-kind-control-plane
root@kind-control-plane:/# crictl logs 9de3341a21eb8
I0517 01:06:38.409000       1 options.go:225] external host was not specified, using 192.168.8.2
E0517 01:06:38.409476       1 run.go:72] "command failed" err="--anonymous-auth-health-only cannot be set to true unless HealthOnlyAnonymousAuth feature gate is enabled"

as expected it fails.

9. now I updated the manifest to include the --feature-gates flag and enabled HealthOnlyAnonymousAuth feature. kube-apiserver is now healthy.

root@kind-control-plane:/# crictl ps -a --name  kube-apiserver
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
881b6d0919900       d9ccda0175e15       7 seconds ago       Running             kube-apiserver      0                   232a52b6522dd       kube-apiserver-kind-control-plane

10. now lets check the readyz endpoint

curl -k https://127.0.0.1:33559/readyz                                                                           ok  10m 44s 
ok%                                        

anonymous request to readyz succeeded (public-info-viewer clusterrolebinding works)

11. now lets check what bindings are there which have system:anonymous

kubectl get clusterrolebindings,rolebindings -o json | jq '.items[] | select(.subjects[]? | .name == "system:anonymous") | {kind: .kind, name:.metadata.name, namespace:.metadata.namespace, role:.roleRef.name, subjects:.subjects}'

{
  "kind": "ClusterRoleBinding",
  "name": "anon",
  "namespace": null,
  "role": "podview",
  "subjects": [
    {
      "apiGroup": "rbac.authorization.k8s.io",
      "kind": "User",
      "name": "system:anonymous"
    }
  ]
}

12. according to rbac system:anonymous has role podview, lets check the permissions

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2024-05-17T00:57:23Z"
  name: podview
  resourceVersion: "679"
  uid: 5478cffe-f78f-48a3-b299-a6cc32f349f0
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list

13. system:anonymous should be able to view pods according to rbac, but we have --anonymous-auth-health-only enabled so this request shouldn't authenticate

curl -k https://127.0.0.1:33559/api/v1/pods                                                                               
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}%                                            

[vinayakankugoyal]

/cc @liggitt

[vinayakankugoyal]

Testing Notes 2:

Instead of docker execing and changing the kube-apiserver manifest I can just create a new kind cluster and use the kind cluster config to set extra args and feature-gates on the cluster. I used the following kind cluster config.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  "HealthOnlyAnonymousAuth": true
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          anonymous-auth-health-only: "true"

[seans3]

/triage accepted
/priority important-longterm

[k8s-triage-robot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vinayakankugoyal
Once this PR has been reviewed and has the lgtm label, please assign andrewsykim, deads2k, mrunalp for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cmd/kube-apiserver/OWNERS
• cmd/kube-controller-manager/OWNERS
• cmd/kubelet/OWNERS
• pkg/controlplane/OWNERS
• pkg/kubeapiserver/OWNERS
• staging/src/k8s.io/apiserver/pkg/apis/OWNERS
• staging/src/k8s.io/apiserver/pkg/authentication/OWNERS
• staging/src/k8s.io/apiserver/pkg/endpoints/OWNERS
• staging/src/k8s.io/apiserver/pkg/features/OWNERS
• staging/src/k8s.io/apiserver/pkg/server/options/OWNERS
• staging/src/k8s.io/cloud-provider/OWNERS
• test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vinayakankugoyal]

Testing note 3:

After reworking the KEP a bit here is the new kubeadm config required to use this feature.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  # any feature gate can be enabled here with "Name": true
  # or disabled here with "Name": false
  # not all feature gates are tested, however
  "AnonymousAuthConfigurableEndpoints": true
  "StructuredAuthenticationConfiguration": true
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /usr/local/google/home/vinaygo/go/src/k8s.io/kubernetes/authconfig.yaml
    containerPath: /etc/kubernetes/pki/authconfig.yaml
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          authentication-config: /etc/kubernetes/pki/authconfig.yaml

and here is how you need to setup the auth config file

apiVersion: apiserver.config.k8s.io/v1alpha1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  restrictToPaths:
  - /livez
  - /readyz
  - /healthz

[k8s-ci-robot]

@vinayakankugoyal: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-unit	b0213b7	link	true	/test pull-kubernetes-unit
pull-kubernetes-verify	b0213b7	link	true	/test pull-kubernetes-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.