[QUESTION/HELP] How to use k3d-managed registry as a pull-through-cache registry for docker hub

[schlichtanders]

Question / Where do you need Help?

The documentation states that registries.yaml can be used to define redirects like normally in k3s. the --registry-config param is used here. I found some tutorials online how to define your own pull-through-cache registries mainly using the Environment Variable REGISTRY_PROXY_REMOTEURL and similar to the docker image registry:2 or similar.

However, the documentation recommends to use the k3s-managed registries, where the --registry-use param is used. Unfortunately the documentation of k3d registry create does not have any extra parameters which would indicate how to enable pull-through-cache.

How then to use k3d-managed registries as a pull-through-cache? I.e. how to set variables like REGISTRY_PROXY_REMOTEURL.

Scope of your Question

k3d documentation

[venkatamutyala]

Here is an example k3d-config.yaml I use. Feel free to tag me if you need any help to get this to work

k3d cluster create --config k3d-config.yaml

apiVersion: k3d.io/v1alpha5 
kind: Simple 
metadata:
  name: yolooooo
servers: 1
agents: 3
registries:
#   create:
#     name: docker-io # name of the registry container
#     proxy:
#       remoteURL: https://registry-1.docker.io # proxy DockerHub
#       username: <DOCKER_USERNAME>
#       password: <DOCKER_PASSWORD>
#     volumes:
#       - /tmp/reg:/var/lib/registry # persist data locally in /tmp/reg
#   config: | # tell K3s to use this registry when pulling from DockerHub
#     mirrors:
#       "docker.io":
#         endpoint:
#           - http://docker-io:5000
  config: | 
    mirrors:
      "docker.io":
        endpoint:
          - http://192.168.0.101:5000
      "quay.io":
        endpoint:
          - http://192.168.0.101:5001
      "ghcr.io":
        endpoint:
          - http://192.168.0.101:5002
options:
  k3d:
    disableLoadbalancer: true
  k3s:
    extraArgs:
      - arg: --disable=cloud-controller
        nodeFilters:
          - server:*
      - arg: --disable=helm-controller
        nodeFilters:
          - server:*
      - arg: --disable=traefik
        nodeFilters:
          - server:*
      - arg: --disable=servicelb
        nodeFilters:
          - server:*

[schlichtanders]

Thank you for sharing your k3d-config.yaml

I am using k3d to create clusters when I need them for testing and developing purposes. Hence I like the registry to be separate from the cluster. Unfortunately k3d registry create does not have a --config option. Is there an alternative?

[schlichtanders]

Now I understood, that the volume directive /tmp/reg:/var/lib/registry is probably already enough to preserve registry cache across cluster starts.

[schlichtanders]

I now tried to setup the cluster+registry at once using the following configuration

apiVersion: k3d.io/v1alpha5
kind: Simple
# using default cluster name
kubeAPI:  # same as `--api-port jolin.localhost:6445` (where the name would resolve to 127.0.0.1)
    host: jolin.localhost
    hostIP: "127.0.0.1"
    hostPort: "6445"
registries:
    create:
        name: registry.localhost
        hostPort: "5000"
        proxy:
            # proxy DockerHub
            remoteURL: https://registry-1.docker.io
        volumes:
            # persist data locally in /tmp/reg
            - /tmp/reg:/var/lib/registry
    # tell K3s to use this registry when pulling from DockerHub
    config: |
        mirrors:
          "docker.io":
            endpoint:
              - http://registry.localhost:5000
options:
    kubeconfig:
        updateDefaultKubeconfig: false
        switchCurrentContext: false
image: rancher/k3s:v1.29.1-k3s1
ports:
    # same as `--port '80:80@loadbalancer'`
    - port: 80:80
      nodeFilters:
        - loadbalancer

Unfortunately I couldn't find a way how to push into this registry using docker. With the externally k3d-managed registry it worked. I tried to set 127.0.0.1 registry.localhost in my hostfile and tried to push using docker buildx --tag registry.localhost:5000/mydocker:latest --push .. Or with also using localhost:5000/mydocker:latest. Nothing worked.

I get an ERROR: unsupported.

Also when trying the official registry test, it just cannot connect, retrying and retrying and retrying.

docker pull nginx:latest  
docker tag nginx:latest registry.localhost:5000/nginx:latest
docker push registry.localhost:5000/nginx:latest

The push refers to repository [registry.localhost:5000/nginx]
fd31601f0be4: Retrying in 10 seconds 
93b4c8c4ac05: Retrying in 10 seconds 
b7df9f234b50: Retrying in 10 seconds 
ab75a0b61bd1: Retrying in 10 seconds 
c1b1bf2f95dc: Retrying in 10 seconds 
4d99aab1eed4: Waiting 
a483da8ab3e9: Waiting 

@venkatamutyala can you help?

[schlichtanders]

This seems to be a docker problem - a registry with pull-through-cache does not support pushing to it so it seems

https://stackoverflow.com/questions/33610215/unable-to-push-image-to-a-docker-registry-configured-as-proxy-cache

[schlichtanders]

@venkatamutyala a new problem appeared: the username and password are not correctly recognized
[Deleted logs]

EDIT: The problem was on my side, I used a wrong image tag, referring to a wrong docker user.
Now it works :)

[schlichtanders]

Really the only thing left is how to update the registry locally. I found something like

docker stop registry.localhost
docker run -d -p 5000:5000 -v /tmp/reg:/var/lib/registry --name registry.pushable registry:2
docker push k3d-registry.localhost:5000/nginx:latest  # only an example
docker stop registry.pushable
docker rm registry.pushable
docker start registry.localhost

But I am wondering whether it would get picked up correctly ... as the tag cannot include the docker username, but the image name itself obviously needs to be unique...

I guess the above should roughly. Maybe there is some additional interaction with the "latest" version... I hope to be able test this soon.

[schlichtanders]

It would be great if the documentation gets updated so that the k3d-config.yaml description as well as the k3s registry create command include these new proxy arguments

[venkatamutyala]

Hey @schlichtanders in case it helps, here is how i create the registry using docker compose. I use a separate VM for this since all our developers leverage it. To be clear I only use it as a pull through cache. I have not tried pushing images to it.

version: '3'
services:
  registry-docker:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5000:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://registry-1.docker.io
    restart: always
  registry-quay:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5001:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://quay.io
    restart: always
  registry-ghcr:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5002:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://ghcr.io
    restart: always
  registry-gcr:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5003:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://gcr.io
    restart: always
  registry-aws:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5004:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://public.ecr.aws
    restart: always
  registry-mcr:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5005:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://mcr.microsoft.com
    restart: always
  registry-gitlab:
    image: registry:2
    volumes:
      - ./data:/var/lib/registry
    ports:
      - "5006:5000"
    environment:
      REGISTRY_PROXY_REMOTEURL: https://registry.gitlab.com
    restart: always


#https://distribution.github.io/distribution/recipes/mirror/
#https://distribution.github.io/distribution/recipes/mirror/#configure-the-cache
#https://distribution.github.io/distribution/about/deploying/#customize-the-storage-location