In need to run k3d as amd on mac m1

[ahsan-n]

Hi, big fan of k3d. One query I had is often we see images that only built on amd64 arch. Any idea how we can run k3d cluster as amd platform on macos m1 chips.

like in docker you can provide --platform to override default arch.

[iwilltry42]

Hey @ahsan-n , thanks for starting this dicussion!
I guess you can just do k3d cluster create --image rancher/k3s:v1.22.7-k3s1-amd64 🤔
You can see the tags on DockerHub, where for every version and platform there is a specific tag: https://hub.docker.com/r/rancher/k3s/tags

[ahsan-n]

yes there are the logs shared above I tried setting engine globally to amd64 platform but that didn't help either.

export DOCKER_DEFAULT_PLATFORM=linux/amd64

[iwilltry42]

I was referring to the container logs, not the k3d logs.
i.e. docker logs k3d-mycluster-server-0

[Shaked]

Hi, did @iwilltry42's reply has worked for you?

I am running k3d on a MacBook MAX M2, and I tried using the above:

k3d cluster create tests-amd --image rancher/k3s:v1.22.7-k3s1-amd64

At first, everything seems fine and the cluster was created successfully, but then nothing works. Checking the logs I see this:

$ docker logs k3d-tests-amd-server-0
...
...

I0501 10:00:16.075717      23 controllermanager.go:577] Started "pvc-protection"
I0501 10:00:16.076404      23 pvc_protection_controller.go:110] "Starting PVC protection controller"
I0501 10:00:16.076435      23 shared_informer.go:240] Waiting for caches to sync for PVC protection
I0501 10:00:16.098594      23 controller.go:611] quota admission added evaluator for: jobs.batch
time="2023-05-01T10:00:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"traefik-crd\", UID:\"00a6ccad-2f1e-4f62-a3ce-8abc38cfe79d\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"398\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job kube-system/helm-install-traefik-crd"
time="2023-05-01T10:00:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"3060c464-697f-405c-8c9b-ed7681f7ea95\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"399\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job kube-system/helm-install-traefik"
time="2023-05-01T10:00:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"3060c464-697f-405c-8c9b-ed7681f7ea95\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"401\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job kube-system/helm-install-traefik"
time="2023-05-01T10:00:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"traefik-crd\", UID:\"00a6ccad-2f1e-4f62-a3ce-8abc38cfe79d\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"400\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job kube-system/helm-install-traefik-crd"
time="2023-05-01T10:00:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"traefik-crd\", UID:\"00a6ccad-2f1e-4f62-a3ce-8abc38cfe79d\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"400\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job kube-system/helm-install-traefik-crd"
time="2023-05-01T10:00:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"3060c464-697f-405c-8c9b-ed7681f7ea95\", APIVersion:\"helm.cattle.io/v1\", ResourceVersion:\"401\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyJob' Applying HelmChart using Job kube-system/helm-install-traefik"
I0501 10:00:16.225097      23 node_ipam_controller.go:91] Sending events to api server.
E0501 10:00:16.357107      23 controllermanager.go:419] unable to get all supported resources from server: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0501 10:00:16.359392      23 node_controller.go:115] Sending events to api server.
I0501 10:00:16.359523      23 controllermanager.go:285] Started "cloud-node"
I0501 10:00:16.359710      23 node_controller.go:154] Waiting for informer caches to sync
I0501 10:00:16.361018      23 node_lifecycle_controller.go:76] Sending events to api server
I0501 10:00:16.361071      23 controllermanager.go:285] Started "cloud-node-lifecycle"
time="2023-05-01T10:00:16Z" level=info msg="Waiting for control-plane node k3d-tests-amd-server-0 startup: nodes \"k3d-tests-amd-server-0\" not found"
E0501 10:00:16.534088      23 server.go:290] "Failed to run kubelet" err="failed to run Kubelet: could not detect clock speed from output: \"processor\\t: 0\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 1\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 2\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 3\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 4\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 5\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\n\""
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
..
...

And of course kubectl just hangs.

╰─$ k3d version
k3d version v5.4.9
k3s version v1.25.7-k3s1 (default)

Any idea?

[iwilltry42]

@Shaked , googling that kubelet error brings me to this K3s issue which in turn points to a qemu issue that may be causing this.

[corneliusroemer]

Getting same error half a year later even though Rosetta is now used instead of QEMU by docker desktop. Has someone found a workaround, maybe using something other than k3d?

[froy001]

I tried the above approach. I.e. k3d cluster create --image rancher/k3s:v1.22.17-k3s1-amd64 running on docker desktop on an apple m1 silicon and I am getting the error below from my k3d-qwilt-test-cluster-server-0 container. What I need to do is run images built on amd64 inside my k3d cluster. Since running the images just freezes at a certain point without oom errors. Thought it might be an architecture issue.

time="2023-04-10T06:50:00Z" level=info msg="Starting k3s v1.22.17+k3s1 (3ed243df)"
time="2023-04-10T06:50:01Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2023-04-10T06:50:01Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2023-04-10T06:50:01Z" level=info msg="Database tables and indexes are up to date"
time="2023-04-10T06:50:01Z" level=info msg="Kine available at unix://kine.sock"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
time="2023-04-10T06:50:01Z" level=info msg="generated self-signed CA certificate CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01.256147971 +0000 UTC notAfter=2033-04-07 06:50:01.256147971 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="generated self-signed CA certificate CN=k3s-server-ca@1681109401: notBefore=2023-04-10 06:50:01.304238054 +0000 UTC notAfter=2033-04-07 06:50:01.304238054 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="generated self-signed CA certificate CN=k3s-request-header-ca@1681109401: notBefore=2023-04-10 06:50:01.312925513 +0000 UTC notAfter=2033-04-07 06:50:01.312925513 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="generated self-signed CA certificate CN=etcd-server-ca@1681109401: notBefore=2023-04-10 06:50:01.320135513 +0000 UTC notAfter=2033-04-07 06:50:01.320135513 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="generated self-signed CA certificate CN=etcd-peer-ca@1681109401: notBefore=2023-04-10 06:50:01.331333388 +0000 UTC notAfter=2033-04-07 06:50:01.331333388 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:01 +0000 UTC"
time="2023-04-10T06:50:01Z" level=warning msg="dynamiclistener [::]:6443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2023-04-10T06:50:01Z" level=info msg="Active TLS secret / (ver=) (count 12): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.20.0.3:172.20.0.3 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3d-qwilt-test-cluster-server-0:k3d-qwilt-test-cluster-server-0 listener.cattle.io/cn-k3d-qwilt-test-cluster-serverlb:k3d-qwilt-test-cluster-serverlb listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=57092D4173CECF123BB96EB34C69E12826726F2C]"
time="2023-04-10T06:50:01Z" level=info msg="Tunnel server egress proxy mode: agent"
time="2023-04-10T06:50:01Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-04-10T06:50:01Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-04-10T06:50:01Z" level=info msg="Waiting for API server to become available"
time="2023-04-10T06:50:01Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-04-10T06:50:01Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0410 06:50:01.802004      54 server.go:581] external host was not specified, using 172.20.0.3
time="2023-04-10T06:50:01Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
I0410 06:50:01.827614      54 server.go:175] Version: v1.22.17+k3s1
time="2023-04-10T06:50:01Z" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
time="2023-04-10T06:50:01Z" level=info msg="To join server node to cluster: k3s server -s https://172.20.0.3:6443 -t ${SERVER_NODE_TOKEN}"
time="2023-04-10T06:50:01Z" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
time="2023-04-10T06:50:01Z" level=info msg="To join agent node to cluster: k3s agent -s https://172.20.0.3:6443 -t ${AGENT_NODE_TOKEN}"
time="2023-04-10T06:50:01Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2023-04-10T06:50:01Z" level=info msg="Run: k3s kubectl"
time="2023-04-10T06:50:02Z" level=info msg="certificate CN=k3d-qwilt-test-cluster-server-0 signed by CN=k3s-server-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:02 +0000 UTC"
time="2023-04-10T06:50:02Z" level=info msg="certificate CN=system:node:k3d-qwilt-test-cluster-server-0,O=system:nodes signed by CN=k3s-client-ca@1681109401: notBefore=2023-04-10 06:50:01 +0000 UTC notAfter=2024-04-09 06:50:02 +0000 UTC"
time="2023-04-10T06:50:02Z" level=warning msg="Host resolv.conf includes loopback or multicast nameservers - kubelet will use autogenerated resolv.conf with nameserver 8.8.8.8"
time="2023-04-10T06:50:02Z" level=info msg="Module overlay was already loaded"
time="2023-04-10T06:50:02Z" level=info msg="Module nf_conntrack was already loaded"
time="2023-04-10T06:50:02Z" level=warning msg="Failed to load kernel module br_netfilter with modprobe"
time="2023-04-10T06:50:02Z" level=warning msg="Failed to load kernel module iptable_nat with modprobe"
W0410 06:50:02.188170      54 sysinfo.go:203] Nodes topology is not available, providing CPU topology
time="2023-04-10T06:50:02Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 229376"
time="2023-04-10T06:50:02Z" level=error msg="Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied"
time="2023-04-10T06:50:02Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
time="2023-04-10T06:50:02Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
time="2023-04-10T06:50:02Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2023-04-10T06:50:02Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2023-04-10T06:50:03Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
I0410 06:50:03.825248      54 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0410 06:50:03.846716      54 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0410 06:50:03.846874      54 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0410 06:50:03.859531      54 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0410 06:50:03.859586      54 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0410 06:50:03.991551      54 genericapiserver.go:455] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0410 06:50:04.000461      54 instance.go:278] Using reconciler: lease
I0410 06:50:04.162623      54 rest.go:130] the default service ipfamily for this cluster is: IPv4
time="2023-04-10T06:50:04Z" level=info msg="Containerd is now running"
time="2023-04-10T06:50:04Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2023-04-10T06:50:04Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=k3d-qwilt-test-cluster-server-0 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --kubelet-cgroups=/k3s --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/var/lib/rancher/k3s/agent/etc/resolv.conf --runtime-cgroups=/k3s --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
time="2023-04-10T06:50:04Z" level=info msg="Handling backend connection request [k3d-qwilt-test-cluster-server-0]"
time="2023-04-10T06:50:04Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
W0410 06:50:05.622130      54 genericapiserver.go:455] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.631104      54 genericapiserver.go:455] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.687548      54 genericapiserver.go:455] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.695892      54 genericapiserver.go:455] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.724613      54 genericapiserver.go:455] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.741390      54 genericapiserver.go:455] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0410 06:50:05.778573      54 genericapiserver.go:455] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.778650      54 genericapiserver.go:455] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0410 06:50:05.787391      54 genericapiserver.go:455] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0410 06:50:05.787451      54 genericapiserver.go:455] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0410 06:50:05.844177      54 genericapiserver.go:455] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0410 06:50:05.860705      54 genericapiserver.go:455] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0410 06:50:05.892617      54 genericapiserver.go:455] Skipping API apps/v1beta2 because it has no resources.
W0410 06:50:05.892806      54 genericapiserver.go:455] Skipping API apps/v1beta1 because it has no resources.
W0410 06:50:05.905989      54 genericapiserver.go:455] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
I0410 06:50:05.928185      54 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0410 06:50:05.928265      54 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0410 06:50:05.962546      54 genericapiserver.go:455] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
Error from server (NotFound): nodes "k3d-qwilt-test-cluster-server-0" not found
time="2023-04-10T06:50:06Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-04-10T06:50:09Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
Error from server (NotFound): nodes "k3d-qwilt-test-cluster-server-0" not found
time="2023-04-10T06:50:11Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
I0410 06:50:11.980417      54 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
I0410 06:50:11.980445      54 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
I0410 06:50:11.982993      54 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
I0410 06:50:11.987649      54 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0410 06:50:11.987692      54 secure_serving.go:266] Serving securely on 127.0.0.1:6444
I0410 06:50:11.989784      54 available_controller.go:491] Starting AvailableConditionController
I0410 06:50:11.990688      54 apf_controller.go:312] Starting API Priority and Fairness config controller
I0410 06:50:11.992039      54 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0410 06:50:11.993284      54 autoregister_controller.go:141] Starting autoregister controller
I0410 06:50:11.993570      54 cache.go:32] Waiting for caches to sync for autoregister controller
I0410 06:50:11.994608      54 customresource_discovery_controller.go:209] Starting DiscoveryController
I0410 06:50:11.994979      54 dynamic_serving_content.go:129] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key"
I0410 06:50:11.995269      54 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0410 06:50:11.995448      54 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0410 06:50:11.996194      54 crdregistration_controller.go:111] Starting crd-autoregister controller
I0410 06:50:11.996398      54 controller.go:85] Starting OpenAPI controller
I0410 06:50:11.996926      54 naming_controller.go:291] Starting NamingConditionController
I0410 06:50:12.000004      54 establishing_controller.go:76] Starting EstablishingController
I0410 06:50:12.000756      54 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0410 06:50:12.003426      54 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0410 06:50:12.003969      54 crd_finalizer.go:266] Starting CRDFinalizer
I0410 06:50:11.996683      54 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I0410 06:50:12.010562      54 controller.go:83] Starting OpenAPI AggregationController
I0410 06:50:12.019424      54 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0410 06:50:12.019517      54 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0410 06:50:12.032695      54 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
I0410 06:50:12.033125      54 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
I0410 06:50:12.191528      54 apf_controller.go:317] Running API Priority and Fairness config worker
I0410 06:50:12.192645      54 cache.go:39] Caches are synced for AvailableConditionController controller
I0410 06:50:12.194146      54 cache.go:39] Caches are synced for autoregister controller
I0410 06:50:12.199802      54 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0410 06:50:12.205169      54 shared_informer.go:247] Caches are synced for crd-autoregister
I0410 06:50:12.228612      54 shared_informer.go:247] Caches are synced for node_authorizer
I0410 06:50:12.229030      54 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
I0410 06:50:12.263992      54 controller.go:611] quota admission added evaluator for: namespaces
E0410 06:50:12.418665      54 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time
E0410 06:50:12.428753      54 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.20.0.3, ResourceVersion: 0, AdditionalErrorMsg:
I0410 06:50:12.979228      54 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0410 06:50:13.013243      54 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
I0410 06:50:13.028503      54 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0410 06:50:13.028600      54 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
time="2023-04-10T06:50:14Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
I0410 06:50:14.477460      54 controller.go:611] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0410 06:50:14.624324      54 controller.go:611] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0410 06:50:14.890511      54 lease.go:234] Resetting endpoints for master service "kubernetes" to [172.20.0.3]
I0410 06:50:14.898238      54 controller.go:611] quota admission added evaluator for: endpoints
time="2023-04-10T06:50:14Z" level=info msg="Kube API server is now running"
time="2023-04-10T06:50:14Z" level=info msg="ETCD server is now running"
time="2023-04-10T06:50:14Z" level=info msg="k3s is up and running"
time="2023-04-10T06:50:14Z" level=info msg="Waiting for cloud-controller-manager privileges to become available"
I0410 06:50:14.932636      54 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
time="2023-04-10T06:50:14Z" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2023-04-10T06:50:15Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2023-04-10T06:50:15Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
time="2023-04-10T06:50:15Z" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
I0410 06:50:15.406899      54 server.go:436] "Kubelet version" kubeletVersion="v1.22.17+k3s1"
E0410 06:50:15.503239      54 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden: User "system:k3s-controller" cannot list resource "endpoints" in API group "" in the namespace "default"
I0410 06:50:15.587567      54 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt"
W0410 06:50:15.588606      54 manager.go:159] Cannot detect current cgroup on cgroup v2
time="2023-04-10T06:50:15Z" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2023-04-10T06:50:15Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-10.19.300.tgz"
time="2023-04-10T06:50:15Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-10.19.300.tgz"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
time="2023-04-10T06:50:15Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
time="2023-04-10T06:50:16Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2023-04-10T06:50:16Z" level=info msg="Creating deploy event broadcaster"
time="2023-04-10T06:50:16Z" level=info msg="Waiting for control-plane node k3d-qwilt-test-cluster-server-0 startup: nodes \"k3d-qwilt-test-cluster-server-0\" not found"
time="2023-04-10T06:50:16Z" level=info msg="Creating svccontroller event broadcaster"
time="2023-04-10T06:50:16Z" level=info msg="Cluster dns configmap has been set successfully"
E0410 06:50:16.398643      54 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden: User "system:k3s-controller" cannot list resource "endpoints" in API group "" in the namespace "default"
Error from server (NotFound): nodes "k3d-qwilt-test-cluster-server-0" not found
time="2023-04-10T06:50:16Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting apps/v1, Kind=Deployment controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=Service controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
time="2023-04-10T06:50:16Z" level=info msg="Starting /v1, Kind=Node controller"
time="2023-04-10T06:50:16Z" level=info msg="Updating TLS secret for kube-system/k3s-serving (count: 12): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.20.0.3:172.20.0.3 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3d-qwilt-test-cluster-server-0:k3d-qwilt-test-cluster-server-0 listener.cattle.io/cn-k3d-qwilt-test-cluster-serverlb:k3d-qwilt-test-cluster-serverlb listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=57092D4173CECF123BB96EB34C69E12826726F2C]"
I0410 06:50:17.066300      54 serving.go:354] Generated self-signed cert in-memory
time="2023-04-10T06:50:17Z" level=info msg="Active TLS secret kube-system/k3s-serving (ver=220) (count 12): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.20.0.3:172.20.0.3 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3d-qwilt-test-cluster-server-0:k3d-qwilt-test-cluster-server-0 listener.cattle.io/cn-k3d-qwilt-test-cluster-serverlb:k3d-qwilt-test-cluster-serverlb listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=57092D4173CECF123BB96EB34C69E12826726F2C]"
time="2023-04-10T06:50:17Z" level=info msg="Waiting for control-plane node k3d-qwilt-test-cluster-server-0 startup: nodes \"k3d-qwilt-test-cluster-server-0\" not found"
I0410 06:50:18.145127      54 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"5759996c-4a66-41d2-bc27-3d5da2e1f0e0\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"221\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
E0410 06:50:18.225338      54 reflector.go:138] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden: User "system:k3s-controller" cannot list resource "endpoints" in API group "" in the namespace "default"
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"5759996c-4a66-41d2-bc27-3d5da2e1f0e0\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"221\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
time="2023-04-10T06:50:18Z" level=info msg="Waiting for control-plane node k3d-qwilt-test-cluster-server-0 startup: nodes \"k3d-qwilt-test-cluster-server-0\" not found"
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"9aa68646-7160-4e7d-8d3e-e75bf453e101\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"229\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
I0410 06:50:18.405747      54 controller.go:611] quota admission added evaluator for: serviceaccounts
I0410 06:50:18.580931      54 controller.go:611] quota admission added evaluator for: deployments.apps
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"9aa68646-7160-4e7d-8d3e-e75bf453e101\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"229\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"local-storage\", UID:\"43b825a9-cd05-4280-8f9c-bddea3ad0a58\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"240\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"local-storage\", UID:\"43b825a9-cd05-4280-8f9c-bddea3ad0a58\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"240\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"aggregated-metrics-reader\", UID:\"80a34fbe-8279-484b-9705-72b7d8bbe7b8\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"250\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
time="2023-04-10T06:50:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"aggregated-metrics-reader\", UID:\"80a34fbe-8279-484b-9705-72b7d8bbe7b8\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"250\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-delegator\", UID:\"b7bbb388-3850-4dd2-a861-350bc05d06b2\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"255\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-delegator\", UID:\"b7bbb388-3850-4dd2-a861-350bc05d06b2\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"255\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-reader\", UID:\"726a0219-7d1c-44ae-a57e-171a2986b933\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"260\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-reader\", UID:\"726a0219-7d1c-44ae-a57e-171a2986b933\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"260\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-apiservice\", UID:\"e2b90fdc-24a4-49d8-a459-b9b64e0bab06\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"265\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-apiservice\", UID:\"e2b90fdc-24a4-49d8-a459-b9b64e0bab06\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"265\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-deployment\", UID:\"259ec69d-afea-4d55-bc59-4ea39c626a18\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"271\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-deployment\", UID:\"259ec69d-afea-4d55-bc59-4ea39c626a18\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"271\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=k3d-qwilt-test-cluster-server-0 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
W0410 06:50:19.327375      54 server.go:224] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
E0410 06:50:19.330251      54 proxier.go:645] "Failed to read builtin modules file. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules" err="open /lib/modules/5.15.49-linuxkit/modules.builtin: no such file or directory" filePath="/lib/modules/5.15.49-linuxkit/modules.builtin"
I0410 06:50:19.354192      54 proxier.go:655] "Failed to load kernel module with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs"
time="2023-04-10T06:50:19Z" level=info msg="Waiting for control-plane node k3d-qwilt-test-cluster-server-0 startup: nodes \"k3d-qwilt-test-cluster-server-0\" not found"
I0410 06:50:19.377494      54 proxier.go:655] "Failed to load kernel module with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_rr"
I0410 06:50:19.401098      54 proxier.go:655] "Failed to load kernel module with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_wrr"
I0410 06:50:19.423901      54 proxier.go:655] "Failed to load kernel module with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_sh"
I0410 06:50:19.444749      54 proxier.go:655] "Failed to load kernel module with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="nf_conntrack"
E0410 06:50:19.506321      54 node.go:161] Failed to retrieve node info: nodes "k3d-qwilt-test-cluster-server-0" not found
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-service\", UID:\"c1a132b6-a7c0-4f97-be78-d9614fa17186\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"277\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-service\", UID:\"c1a132b6-a7c0-4f97-be78-d9614fa17186\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"277\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\""
time="2023-04-10T06:50:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"resource-reader\", UID:\"c1043261-79b3-43c8-b38e-a8c1e8a447dc\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"284\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\""
I0410 06:50:20.016125      54 controllermanager.go:186] Version: v1.22.17+k3s1
I0410 06:50:20.034728      54 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0410 06:50:20.035029      54 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0410 06:50:20.035177      54 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0410 06:50:20.035190      54 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0410 06:50:20.035285      54 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0410 06:50:20.036297      54 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0410 06:50:20.036677      54 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0410 06:50:20.036687      54 secure_serving.go:200] Serving securely on 127.0.0.1:10257
time="2023-04-10T06:50:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"resource-reader\", UID:\"c1043261-79b3-43c8-b38e-a8c1e8a447dc\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"284\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\""
I0410 06:50:20.135205      54 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0410 06:50:20.136353      54 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0410 06:50:20.136590      54 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
W0410 06:50:20.313549      54 handler_proxy.go:107] no RequestInfo found in the context
E0410 06:50:20.314260      54 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0410 06:50:20.314351      54 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
time="2023-04-10T06:50:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"d3a01653-fe66-4a9c-b232-17a4540f1aec\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"290\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\""
time="2023-04-10T06:50:20Z" level=info msg="Waiting for control-plane node k3d-qwilt-test-cluster-server-0 startup: nodes \"k3d-qwilt-test-cluster-server-0\" not found"
time="2023-04-10T06:50:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"d3a01653-fe66-4a9c-b232-17a4540f1aec\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"290\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\""
E0410 06:50:20.615536      54 server.go:290] "Failed to run kubelet" err="failed to run Kubelet: could not detect clock speed from output: \"processor\\t: 0\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 1\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 2\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 3\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 4\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 5\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\nprocessor\\t: 6\\nBogoMIPS\\t: 48.00\\nFeatures\\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\\nCPU implementer\\t: 0x00\\nCPU architecture: 8\\nCPU variant\\t: 0x0\\nCPU part\\t: 0x000\\nCPU revision\\t: 0\\n\\n\""
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

[GabriFedi97]

Hey have you found any workaround?

[iwilltry42]

The line below quoted from your logs output indicates an error with running the Kubelet itself, rendering the cluster unusable.

See my comment up here: #1031 (reply in thread)
-> K3s issue -> QEMU Issue

E0410 06:50:20.615536 54 server.go:290] "Failed to run kubelet" err="failed to run Kubelet: could not detect clock speed from output: "processor\t: 0\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 1\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 2\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 3\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 4\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 5\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\nprocessor\t: 6\nBogoMIPS\t: 48.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint\nCPU implementer\t: 0x00\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0x000\nCPU revision\t: 0\n\n""