Nashpush

aws-ecr-registry-creds

4 min read

Guide to Using registry-creds on On-Premises Kubernetes Clusters

This guide provides step-by-step instructions for deploying and troubleshooting registry-creds on on-premises Kubernetes clusters using registry-creds as a ReplicationController in the kube-system namespace. This tool helps in managing Docker registry credentials, particularly for AWS ECR.

Prerequisites

  • Kubernetes cluster running on-premises
  • Access to the Kubernetes cluster (kubectl configured)
  • AWS CLI configured with necessary permissions
  • registry-creds repository: registry-creds GitHub

Deployment

  1. Clone the repository:
git clone https://github.com/upmc-enterprises/registry-creds.git
cd registry-creds
  1. Edit the configuration file:
  • Edit the deployment/kubernetes/replicationcontroller.yaml file to include your AWS ECR credentials and relevant information.
  • Example:
env:
- name: ECR_REGION
  value: "your-aws-region"
- name: ECR_URL
  value: "your-aws-account-id.dkr.ecr.your-aws-region.amazonaws.com"
- name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: registry-creds-ecr
      key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: registry-creds-ecr
      key: AWS_SECRET_ACCESS_KEY
  1. Create a Kubernetes secret. Create a secret to store AWS credentials.
apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-ecr
  namespace: kube-system
  labels:
    app: registry-creds
    cloud: ecr
data:
  AWS_ACCESS_KEY_ID: <base64-encoded-access-key-id>
  AWS_SECRET_ACCESS_KEY: <base64-encoded-secret-access-key>
  aws-account: <base64-encoded-aws-account-id>
  aws-region: <base64-encoded-aws-region>
type: Opaque
  1. Deploy registry-creds:
kubectl apply -f deployment/kubernetes/replicationcontroller.yaml

Verify Deployment:

kubectl get pods -n kube-system | grep registry-creds

  1. Check the status of the registry-creds pod:

    kubectl get pods -n kube-system -l app=registry-creds

  2. Verify the logs to ensure there are no errors:

    kubectl logs -n kube-system $(kubectl get pods -n kube-system -l app=registry-creds -o jsonpath="{.items[0].metadata.name}")

Troubleshooting

Troubleshooting Image Pull Errors

If you encounter an error while pulling an image, similar to the one shown below, follow the troubleshooting steps:

Error Message:

Events:                                                                                                                                  
│   Type     Reason     Age                 From               Message                                                                                                           │
│   ----     ------     ----                ----               -------                                                                                                           │
│   Normal   Scheduled  21m                 default-scheduler  Successfully assigned webpush/webpush-go-spammer-master-7ffb4f87d6-5p77n to node14                                │
│   Normal   Pulling    20m (x4 over 21m)   kubelet            Pulling image "589378948933.dkr.ecr.eu-central-1.amazonaws.com/webpush-go-sender:1.18.3.0"                        │
│   Warning  Failed     20m (x4 over 21m)   kubelet            Failed to pull image "589378948933.dkr.ecr.eu-central-1.amazonaws.com/webpush-go-sender:1.18.3.0": failed to pull │
│  and unpack image "589378948933.dkr.ecr.eu-central-1.amazonaws.com/webpush-go-sender:1.18.3.0": failed to resolve reference "589378948933.dkr.ecr.eu-central-1.amazonaws.com/w │
│ ebpush-go-sender:1.18.3.0": unexpected status from HEAD request to https://589378948933.dkr.ecr.eu-central-1.amazonaws.com/v2/webpush-go-sender/manifests/1.18.3.0: 403 Forbidden                                                                                                   
│   Warning  Failed     20m (x4 over 21m)   kubelet            Error: ErrImagePull                                                                                               │
│   Warning  Failed     20m (x6 over 21m)   kubelet            Error: ImagePullBackOff                                                                                           │
│   Normal   BackOff    95s (x88 over 21m)  kubelet            Back-off pulling image "589378948933.dkr.ecr.eu-central-1.amazonaws.com/webpush-go-sender:1.18.3.0"

Steps to Resolve

  1. Check AWS ECR Permissions:

Ensure that the AWS IAM user or role associated with the credentials has sufficient permissions to access the ECR repository.

  1. Update Credentials:
apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-ecr
  namespace: kube-system
  labels:
    app: registry-creds
    cloud: ecr
data:
  AWS_ACCESS_KEY_ID: <new-base64-encoded-access-key-id>
  AWS_SECRET_ACCESS_KEY: <new-base64-encoded-secret-access-key>
  aws-account: <base64-encoded-aws-account-id>
  aws-region: <base64-encoded-aws-region>
type: Opaque
  1. Apply the updated secret:
kubectl apply -f secret.yaml
  1. Reboot registry-creds ReplicationController:
  • Rebooting the ReplicationController will refresh the credentials used by the nodes.
kubectl delete pod -l app=registry-creds -n kube-system
  • Wait approximately 10 minutes for the new credentials to propagate and the registry-creds to update on all nodes.
  1. Verify Image Pull:
  • After waiting, attempt to pull the image again. Check the pod status to confirm the image is pulled successfully.
kubectl describe pod <your-pod-name> -n <your-namespace>
  1. Check registry-creds Pod Logs:

To ensure that Docker ECR credentials were updated after rebooting the registry-creds ReplicationController, check the logs of the registry-creds pod:

kubectl logs <registry-creds-pod-name> -n kube-system

Example log entries indicating successful credential processing:

time="2024-07-25T08:40:22Z" level=info msg="Processing secret for namespace webpush, secret awsecr-cred"
 
time="2024-07-25T08:40:22Z" level=info msg="Updated secret awsecr-cred in namespace webpush" function=processNamespace
 
time="2024-07-25T08:40:22Z" level=info msg="Updating ServiceAccount default in namespace webpush" function=processNamespace
 
time="2024-07-25T08:40:22Z" level=info msg="Finished processing secret for namespace webpush, secret awsecr-cred"
 
  
 
time="2024-07-25T08:40:39Z" level=info msg="------------------ [awsecr-cred] ------------------"
 
time="2024-07-25T08:40:39Z" level=info msg="Getting secret; try #1 of 4"
 
time="2024-07-25T08:40:39Z" level=info msg="Successfully got secret for provider awsecr-cred after trying 1 time(s)"

Conclusion

By following this guide, you should be able to deploy and manage registry-creds on your on-premises Kubernetes cluster, ensuring seamless Docker registry credential management. For more detailed information, refer to the registry-creds GitHub repository.

Graph View