Skaffold/Kaniko with ECR

[jicowan]

This is a feature request. I'd like to be able to use Skaffold to create a Kaniko container/pod on KOPS cluster for building container images and then pushing them to an ECR registry. I'd also like to be able to use an S3 bucket for the build context.

[dlorenc]

Using s3 for the build context needs to be implemented first in kaniko. I think there's a feature request over in that repo detailing more flexible build context locations.

[bhack]

@dlorenc GoogleContainerTools/kaniko#121

[priyawadhwa]

Pushing to ECR with kaniko is tricky because the docker config has to specify a specific ECR registry which looks like this:

{
	"credHelpers": {
		"aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login"
	}
}

where the user has to specify their own aws account id and region. For this to work in skaffold, a user would need some way of mounting in their own docker config.

[jicowan]

If Kaniko is running on an EC2 instance, you can assign the instance an IAM role that grants it access to ECR. You shouldn't need to use the credential helper in that instance.

[bhack]

On EC2 it is solvable also with https://github.com/jtblin/kube2iam

[dlorenc]

Do you have any pointers to how docker actually fetches access tokens in an environment like that?

gcr works the same way from gce, but the access token fetching happens from inside the credential helper.

Basically auth works like this currently:

• docker or kaniko parse the registry out of the image name
• they look up the credential helper for that registry from the docker config.json
• they call the credential helper to get an access token
• the credential helper determines it is running on gce and gets an access token using the IAM roles the VM service account is bound to

This all works easily with gcr because there is a small set of valid registry urls (us.gcr.io, eu.gcr.io, etc.) that we can hardcode inside the config.json, instructing the calling tool to use the gcr credential helper.

[bhack]

Simply you need to give the listed permission in https://kubernetes.io/docs/concepts/containers/images/#using-aws-ec2-container-registry to the pod with kube2iam

[bhack]

Of course these are for read you need to add the write ones for pushing

[dlorenc]

In this case it looks like the Kubelet has direct support for ec2 in it:

The kubelet will fetch and periodically refresh ECR credentials. It needs the following permissions to do this:

We would need to do something similar in https://github.com/google/go-containerregistry to make this work.

[bhack]

@dlorenc https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go#L76.
But I think that if you can have the aws ecr credential helper and you authorize i.e. with kube2iam it can work with the basic auth pipeline that you have described.

[dlorenc]

Kaniko has the ECR credential helper now: https://github.com/GoogleContainerTools/kaniko/blob/master/deploy/Dockerfile

Do you think we need to do anything else now?