How to Setup Private Docker Registry in Kubernetes (k8s)

It is always recommended to have private docker registry or repository in your Kubernetes cluster. Docker private registry allows the developers to push and pull their private container images. Once the application’s containers are pushed to private registry then developers can use the path of their private registry while creating and deploying their yaml files.

In this article, we will learn how we can deploy private docker registry as a deployment on top of Kubernetes cluster. I am assuming Kubernetes cluster is already up and running.

Also ReadHow to Install Kubernetes (k8s) Cluster on RHEL 8

Kubernetes lab details for setting up private docker registry

  • k8s-master – 192.168.1.40 – RHEL 8 / Rocky Linux 8
  • k8s-worker-1 – 192.168.1.41 – RHEL 8 / Rocky Linux 8
  • k8s-worker-2 – 192.168.1.42  – RHEL 8 / Rocky Linux 8
  • kadmin user with sudo rights
  • NFS share ‘/opt/certs’ & ‘/opt/registry’

Note:  In my case, I have setup nfs server on master node and exported /opt/certs and /opt/registry as nfs share. Don’t forget to set following permissions on your nfs share.

$ sudo chown -R nobody: /opt/certs/
$ sudo chown -R nobody: /opt/registry/

Before starting the deployment of private registry, please make sure these nfs shares are mounted on each worker nodes. Run the following commands on each worker node.

$ sudo mkdir /opt/certs /opt/registry
$ sudo mount 192.168.1.40:/opt/certs /opt/certs
$ sudo mount 192.168.1.40:/opt/registry /opt/registry

For permanent mount, add nfs entries in /etc/fstab file.

In place of mounting these nfs shares, we can also create nfs based persistent volumes and later we can use these persistent volumes in yaml file.

Let’s deep dive into installation and configuration steps of private docker registry in Kubernetes.

Step 1) Generate self-signed certificates for private registry

Login to your control plane or master node and use openssl command to generate self-signed certificates for private docker repository.

$ cd /opt 
$ sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout ./certs/registry.key -addext "subjectAltName = DNS:master-node-k8" -x509 -days 365 -out ./certs/registry.crt

Private-Docker-Repo-Key-Cerificate-k8s

Once the key and certificate file are generated, use ls command to verify them,

[kadmin@k8s-master opt]$ ls -l certs/
total 8
-rw-r--r--. 1 nobody nobody 2175 Mar 21 08:06 registry.crt
-rw-------. 1 nobody nobody 3272 Mar 21 08:05 registry.key
[kadmin@k8s-master opt]$

Step 2) Deploy private registry as deployment via yaml file

On your master node, create a private-registry.yaml file with the following contents

[kadmin@k8s-master ~]$ mkdir docker-repo
[kadmin@k8s-master ~]$ cd docker-repo/
[kadmin@k8s-master docker-repo]$ vi private-registry.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: private-repository-k8s
  labels:
    app: private-repository-k8s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: private-repository-k8s
  template:
    metadata:
      labels:
        app: private-repository-k8s
    spec:
      volumes:
      - name: certs-vol
        hostPath:
          path: /opt/certs
          type: Directory
      - name: registry-vol
        hostPath:
          path: /opt/registry
          type: Directory

      containers:
        - image: registry:2
          name: private-repository-k8s
          imagePullPolicy: IfNotPresent
          env:
          - name: REGISTRY_HTTP_TLS_CERTIFICATE
            value: "/certs/registry.crt"
          - name: REGISTRY_HTTP_TLS_KEY
            value: "/certs/registry.key"
          ports:
            - containerPort: 5000
          volumeMounts:
          - name: certs-vol
            mountPath: /certs
          - name: registry-vol
            mountPath: /var/lib/registry

save and close the yaml file

private-registry-deployment-yaml-k8s

Run the following kubectl command deploy the private registry using above created yaml file,

[kadmin@k8s-master docker-repo]$ kubectl create -f private-registry.yaml
deployment.apps/private-repository-k8s created
[kadmin@k8s-master docker-repo]$

Execute below kubectl commands to verify status of registry deployment and its pod.

[kadmin@k8s-master ~]$ kubectl get deployments private-repository-k8s
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
private-repository-k8s   1/1     1            1           3m32s
[kadmin@k8s-master ~]$
[kadmin@k8s-master ~]$ kubectl get pods | grep -i private-repo
private-repository-k8s-85cf76b9d7-qsjxq   1/1     Running   0          5m14s
[kadmin@k8s-master ~]$

Perfect, above output confirms that registry has been deployed successfully, Now copy the registry certificate file to worker nodes and master node under the folder “/etc/pki/ca-trust/source/anchors“. Execute the following commands on master node and each worker nodes

$ sudo cp /opt/certs/registry.crt /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust

Step 3) Expose registry deployment as a nodeport service type

To expose registry deployment as a nodeport service type, create the following yaml file with the beneath contents,

[kadmin@k8s-master ~]$ cd docker-repo/
[kadmin@k8s-master docker-repo]$ vi private-registry-svc.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: private-repository-k8s
  name: private-repository-k8s
spec:
  ports:
  - port: 5000
    nodePort: 31320
    protocol: TCP
    targetPort: 5000
  selector:
    app: private-repository-k8s
  type: NodePort

save and close the file.

Now deploy the service by running following kubectl command,

$ kubectl create -f private-registry-svc.yaml
service/private-repository-k8s created
$

Run below kubectl command to verify the service status,

[kadmin@k8s-master ~]$ kubectl get svc private-repository-k8s
NAME                   TYPE     CLUSTER-IP    EXTERNAL-IP PORT(S)        AGE
private-repository-k8s NodePort 10.100.113.39 <none>      5000:31320/TCP 2m1s
[kadmin@k8s-master ~]$

Step 4) Test and Use private docker registry in k8s

To test private registry, we will download nginx image locally and then will upload that image to private registry, from the master node run the following set of commands,

$ sudo docker pull nginx
$ sudo docker tag nginx:latest k8s-master:31320/nginx:1.17
$ sudo docker push k8s-master:31320/nginx:1.17

Output of above command would like below:

Docker-push-command-example

Run below docker command to verify whether nginx is uploaded to private repository or not.

[kadmin@k8s-master ~]$ sudo docker image ls | grep -i nginx
nginx                     latest   7e4d58f0e5f3        2 weeks ago      133MB
k8s-master:31320/nginx    1.17     7e4d58f0e5f3        2 weeks ago      133MB
[kadmin@k8s-master ~]$

Now, let’s deploy a nginx based deployment and in the yaml file specify the image’s path as our private docker registry. Example is shown below:

[kadmin@k8s-master ~]$ vi nginx-test-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-1-17
        image: k8s-master:31320/nginx:1.17
        ports:
        - containerPort: 80

Save and Close the file

private-repository-deployment-yaml

Run following kubectl commands,

[kadmin@k8s-master ~]$ kubectl create -f nginx-test-deployment.yaml
deployment.apps/nginx-test-deployment created
[kadmin@k8s-master ~]$ kubectl get deployments  nginx-test-deployment
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
nginx-test-deployment   3/3     3            3           13s
[kadmin@k8s-master ~]$
[kadmin@k8s-master ~]$ kubectl get  pods | grep nginx-test-deployment
nginx-test-deployment-f488694b5-2rvmv     1/1     Running   0      80s
nginx-test-deployment-f488694b5-8kb6c     1/1     Running   0      80s
nginx-test-deployment-f488694b5-dgcxl     1/1     Running   0      80s
[kadmin@k8s-master ~]$

Try to describe any pod using ‘kubectl describe‘ command and verify image path

$ kubectl describe pod nginx-test-deployment-f488694b5-2rvmv

Output of above command would be,

kubectl-describe-pod-k8s

Above output confirms that container’s image path is our private docker registry, so it means nginx image has been downloaded from private registry. That’s all from this article, I hope these steps help you to setup private docker registry on your Kubernetes cluster. Please do share your feedback and comments in the comments section below.

Also ReadHow to Setup Kubernetes Cluster on Google Cloud Platform (GCP)

Also ReadHow to Setup NGINX Ingress Controller in Kubernetes

Share Now!

15 thoughts on “How to Setup Private Docker Registry in Kubernetes (k8s)”

  1. I tried this deployment but everytime i deploy the docker registry deployment i get CrashLoopBackOff. What is the image: registry:2 ? Where in this guide does this get created?

    Reply
    • Hey Ross,

      Docker registry image is downloaded from internet or web automatically when you deploy the yaml file. You have to make sure all required certificates and volume is created for registry deployment

      Reply
  2. Hello, thank for your tutorial. I have managed to deploy the docker-registry service.

    I am wondering if it is possible to get an auth token from that service?
    Let’s say I want to use kaniko to build some image inside of k8s.

    Reply
  3. Is it possible to create an alias for the whole thing k8s-master:31320 including port, like companycontainerregistry? I just would like yml files know as less as possible

    Reply
  4. I had to use a workaround to add the certs (cat yourrootCAcertificate.cer >> /etc/pki/tls/certs/ca-bundle.crt) as mentioned in the comment above. When I try to deploy nginx as a test I get this (kubectl describe pod nginxpod):

    Warning Failed 15s (x2 over 27s) kubelet Failed to pull image “k8s-server:31320/nginx:1.17”: rpc error: code = Unknown desc = failed to pull and unpack image “k8s-server:31320/nginx:1.17”: failed to resolve reference “k8s-server:31320/nginx:1.17”: failed to do request: Head “https://k8s-server:31320/v2/nginx/manifests/1.17”: x509: certificate is not valid for any names, but wanted to match k8s-server

    (Note that my master is called k8s-server).

    Any help greatly appreciated!

    Reply
  5. Hello All,

    I need help regarding the deployment please check the below error.

    time=”2022-12-12T16:59:44.476783058Z” level=warning msg=”No HTTP secret provided – generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.” go.version=go1.16.15 instance.id=f3dec7ac-3acc-4c2a-8960-6a6f2ad975b9 service=registry version=”v2.8.1+unknown”
    time=”2022-12-12T16:59:44.476867924Z” level=info msg=”redis not configured” go.version=go1.16.15 instance.id=f3dec7ac-3acc-4c2a-8960-6a6f2ad975b9 service=registry version=”v2.8.1+unknown”
    time=”2022-12-12T16:59:44.477645779Z” level=info msg=”Starting upload purge in 11m0s” go.version=go1.16.15 instance.id=f3dec7ac-3acc-4c2a-8960-6a6f2ad975b9 service=registry version=”v2.8.1+unknown”
    time=”2022-12-12T16:59:44.484680116Z” level=info msg=”using inmemory blob descriptor cache” go.version=go1.16.15 instance.id=f3dec7ac-3acc-4c2a-8960-6a6f2ad975b9 service=registry version=”v2.8.1+unknown”
    time=”2022-12-12T16:59:44.484947066Z” level=info msg=”restricting TLS version to tls1.2 or higher” go.version=go1.16.15 instance.id=f3dec7ac-3acc-4c2a-8960-6a6f2ad975b9 service=registry version=”v2.8.1+unknown”
    time=”2022-12-12T16:59:44.484967702Z” level=info msg=”restricting TLS cipher suites to: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_256_GCM_SHA384″ go.version=go1.16.15 instance.id=f3dec7ac-3acc-4c2a-8960-6a6f2ad975b9 service=registry version=”v2.8.1+unknown”
    time=”2022-12-12T16:59:44.4850104Z” level=fatal msg=”open /opt/certs/registry.crt: no such file or directory”

    Reply
  6. Dear sir,
    I followed the steps in your document, but it didn’t work.

    [root@k8s-master docker-repo]# kubectl get pods | grep nginx-test-deployment
    nginx-test-deployment-846f59c7c9-bjhmw 0/1 ErrImagePull 0 65s
    nginx-test-deployment-846f59c7c9-hr7hb 0/1 ErrImagePull 0 65s
    nginx-test-deployment-846f59c7c9-r48p2 0/1 ErrImagePull 0 65s
    thank you in advance for your response

    Reply
    • Hi Fadhel,

      Your nodes must trust the registry from which you want to pull images. I mean, your k8s nodes must trust self-signed certificates of your registry.

      Reply
  7. Hello, thank for your tutorial.
    I am testing installation of kubernetes on RHEL 9.1 and I want make a private registry (kubernetes, cri-o, calico). Kubernetes work fine.
    My self signed certificate (with self signed CA) seems to work if I believe the returns of the openssl command (openssl s_client -showcerts -connect k8s-master:3120)
    However my pod is in ImagePullBackOff. With error : Failed to pull image “k8s-master.levieux.fr:31320/ap_nginx”: rpc error: code = Unknown desc = pinging container registry k8s-master.levieux.fr:3120: Get “https://k8s-master:31320/v2/”: dial tcp 192.168.112.75:31320: connect: connection refused

    In this tutorial you don’t describe your docker installation. Have you make a particular configuration? Have you installed docker on workers ? At some point you say “restart docker on all nodes.

    Reply
    • Hi Alain,

      When I wrote this article at that time, Docker was supported as container runtime in kubernetes cluster. So now a days , it is not not supported.So no need to restart docker. Better go with Crio or Containerd runtime.

      Reply

Leave a Comment