Running a task on ECS from an image in a private repository

by Madeleine Thompson, 2015–12–01

This document describes how I ran an ECS task using an image from a private Docker repository on Docker Hub using only the AWS CLI. There is a similar tutorial at Using the AWS CLI with Amazon ECS that tells you how to do this with a public image.

I'm going to use the AWS CLI because the ECS CLI, which attempts to be a simpler interface to ECS, does not give you enough control over the instances it creates to use images from a private Docker repository.

There might be better ways to do this; I'm writing this as a record of my own exploration.

After some commands, I'm going to assign values from the output to variables. If you see something like:

task_arn=...

after a command I say to run, find something that looks like it could be task_arn and assign it to that variable. Read about bash shell variables if this is confusing.

The AWS CLI is Baroque. You have been warned.

Preliminaries

I'm using Ubuntu 14.04 on EC2, but these instructions should work anywhere you can install Docker and the AWS CLI.

Create a private image on Docker Hub

Go to Docker Hub, log in, and create a new repository. I'm going to call mine ecs-hello. On the Create Repository page, make the visibility "private" to keep things interesting.

I've created a test image and some configuration files to work with. Obtain them with:

git clone git@github.com:madeleineth/ecs-hello.git
cd ecs-hello

Set docker_hub_username to your username. Build the image:

docker_hub_username=...
docker_image=${docker_hub_username}/ecs-hello
docker build -t ${docker_hub_username}/ecs-hello .

Test it locally by running:

docker run --rm -p 4080:80 ${docker_image}

and connecting to http://localhost:4080. Confirm that it serves a page that says "Hello, World!" Use ^C to stop the container.

Push the image to your private repository with:

docker push "${docker_image}"

Create a node configuration file

The strangest part of ECS, to me, is how instances are configured. You start a compute node by choosing an AMI that is "ECS-optimized." When any such node starts, it'll join the default cluster with no privileges by default. You use the EC2 "user data" feature to, at instance creation time, copy in a configuration file that tells it to join a different cluster and use credentials when it starts.

We're going to create a cluster named ecs-hello-cluster.

Create a config file named ecs.config that looks like this:

ECS_CLUSTER=ecs-hello-cluster
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"...","email":"..."}}

Replace the value of ECS_ENGINE_AUTH_DATA with the contents of the "auths" block of $HOME/.docker/config.json (which should exist after you ran docker login above).

You can see all the variables you can put in that file in Amazon ECS Container Agent Configuration.

Now, you need to put this file in S3 so it can be copied to the instances when they start. Suppose you want to use a new bucket with a name of your username followed by -ecs-hello. Create the bucket and copy the config file there:

s3_bucket=${docker_hub_username}-ecs-hello
aws s3 mb s3://${s3_bucket}
aws s3 cp ecs.config s3://${s3_bucket}

Next, create a script, copy-ecs-config.sh, that copies this to /etc/etc/ecs.config on your instance. It should have contents like this:

#!/bin/sh
yum install -y aws-cli
aws s3 cp s3://USERNAME-ecs-hello/ecs.config /etc/ecs/ecs.config

Make sure you match the bucket name with the bucket you created above. You have to actually put the name there. It will run on a different machine, so your shell variables will not be seen.

Pick an AMI for your cluster instances

Next, pick an AMI located in your AWS region. Launching an Amazon ECS Container Instance has a list. I'm us-east-1, so I'm using the AMI ami-ddc7b6b7.

ami=ami-ddc7b6b7

Create an IAM role for your cluster instances

An IAM role specifies a collection of privileges an entity on AWS (like an EC2 instance) can have. We'll create one named ecs-hello-role that can manage ECS tasks running on it and can read the object we just created from S3. Read role-policy.json, which defines what sort of role it is, in this case one for EC2 instances.

aws iam create-role --role-name ecs-hello-role --assume-role-policy-document file://role-policy.json

Next, we want to give this role access to the S3 bucket containing ecs.config. Edit s3-policy.json so it refers to the S3 bucket you created. Then, put that policy on the role:

aws iam put-role-policy --role-name ecs-hello-role --policy-name 's3-policy' --policy-document file://s3-policy.json

Now, attach an AWS-provided policy that will let it manage ECS tasks:

aws iam attach-role-policy --role-name ecs-hello-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

There's more discussion and instructions for doing this on the web console at Amazon ECS Container Instance IAM Role.

Create an instance profile

An instance profile specifies the permissions that an EC2 instance has. It is a collection of IAM roles. We want our instances to have a single role, ecs-hello-role, so run this to create the instance profile ecs-hello-profile:

aws iam create-instance-profile --instance-profile-name ecs-hello-profile
aws iam add-role-to-instance-profile --instance-profile-name ecs-hello-profile --role-name ecs-hello-role

Pick a VPC and subnet

I'm going to assume you're putting everything in a VPC. This has been the default on AWS for a while now. You can see the subnets and associated VPCs you can choose from with aws ec2 describe-subnets. Choose a subnet id from the output of that command and remember it for later along with its associated VPC id.

aws ec2 describe-subnets
subnet=...
vpc=...

Create a security group

You'll also need a security group that tells you which network traffic can connect to your cluster nodes. For this guide, I recommend allowing any IP to connect to ports 22 (ssh, for debugging) and 80 (HTTP, so you can connect to the test server from your workstation). Using the VPC from the last step:

aws ec2 create-security-group --group-name ecs-hello-group --vpc-id $vpc --description 'test group for ecs-hello'
sg=...

Assign rules to the security group $sg for incoming network traffic:

aws ec2 authorize-security-group-ingress --group-id $sg --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $sg --protocol tcp --port 80 --cidr 0.0.0.0/0

You can verify that your security group is set up correctly by looking at the output of:

aws ec2 describe-security-groups --group-ids $sg

Create a cluster and put an instance in it

You don't actually need to create the cluster; that happens automatically if you skip this step. However, I wish it were required, so I'm going to do it for style. Our cluster is ecs-hello-cluster.

aws ecs create-cluster --cluster-name ecs-hello-cluster

Now, create the instance and save its instance id (which starts with i-):

aws ec2 run-instances --key-name KEYNAME --image-id $ami --security-group-ids $sg --instance-type t2.small --iam-instance-profile Name=ecs-hello-profile --user-data "`base64 -w 0 < copy-ecs-config.sh`" --subnet-id $subnet
ec2_instance=...

Make sure you've set $ami, $sg, and $subnet as above. If you have uploaded an ssh key to EC2, pass its name as KEYNAME. Otherwise, you can drop the --key-name KEYNAME phrase, but you won't be able to log in to debug the instance if anything goes wrong.

It's easy to lose track of what any given EC2 instance was created for. Tag it with a useful name (like ecs-hello-node-1):

aws ec2 create-tags --resources $ec2_instance --tags Key=Name,Value=ecs-hello-node-1

If you did everything right with the ecs.config file, the instance should eventually be part of your cluster. Wait until this command shows the state (in the field Reservations.Instances.State.Name) as "running":

aws ec2 describe-instances --instance-ids $ec2_instance

Then, you can confirm that it is part of your cluster with:

aws ecs list-container-instances --cluster ecs-hello-cluster

This will contain a single entry if everything so far worked. If it didn't, and if you specified --key-name, you can get the instance's public IP from aws ec2 describe-instances and ssh in as the user ec2-user and poke around the logs for what might've gone wrong. The file /var/log/cloud-init.log is often useful.

Run your task on the cluster

A task is defined by a task definition, expressed in JSON. The contents roughly correspond to the parameters of docker run. Different versions of a definition are referred to as a family. Version 3 of family foo would be expressed as foo:3 in command line arguments.

In the repository you checked out, edit task-def.json to refer to the image you built above, putting your Docker Hub username in the marked location. Create a definition with:

aws ecs register-task-definition --cli-input-json file://task-def.json

task-def.json specified the name as ecs-hello-task; you should see the versioned task definition name as ecs-hello-task:1 in the output of aws ecs register-task-definition at the end of the taskDefinitionArn field. (The version might be different if this is not your first time through this process.)

You can run this task with:

aws ecs run-task --cluster ecs-hello-cluster --task-definition ecs-hello-task:1

This will output, amongst other things, a task ARN in the tasks.taskArn field. It will have the form arn:aws:ecs:...:...:task/.... Save this.

task_arn=...

Visit your new server in a web browser

There's only one instance, so you could cheat, find its public IP address in the EC2 web console, and visit that IP address in your browser. Why do that, when AWS offers a much more complicated way?

First, find the ECS container instance your task is (hopefully) running on:

aws ecs describe-tasks --cluster ecs-hello-cluster --tasks $task_arn

Somewhere in the output is containerInstanceArn field, which specifies a single ECS container agent running on an EC2 instance. It'll look something like arn:aws:ecs:us-east-1:NNN:container-instance/.... Save this:

container_instance_arn=...

Find the EC2 instance it's running on with:

aws ecs describe-container-instances --cluster ecs-hello-cluster --container-instances $container_instance_arn

In that command's output is the ec2InstanceId field, which will have a value like i-01234567. This will match the instance you created above. Save this.

task_ec2_instance=...

Finally, find the public IP address of the instance it's running on:

aws ec2 describe-instances --instance-ids $task_ec2_instance

The output will have a PublicIp field. If you go to that address in a browser, and you see a page with "Hello, World!" everything is working.

What next: services and cleanup

If you want a stable replicated service with a name you can point a loadbalancer to, read AWS's Services documentation.

When you're done exploring services (or if you don't want to), you'll probably want to shut down the resources you created for this article. These should be obvious:

aws ecs stop-task --cluster ecs-hello-cluster --task $task_arn
aws ecs deregister-task-definition --task-definition ecs-hello-task:1
aws ec2 terminate-instances --instance-ids $ec2_instance

Wait a little while for the EC2 instance to shut down, so it's possible to delete the security group.

aws ec2 delete-security-group --group-id $sg
aws ecs delete-cluster --cluster ecs-hello-cluster

aws s3 rm s3://${s3_bucket}/ecs.config
aws s3 rb s3://${s3_bucket}

aws iam remove-role-from-instance-profile --instance-profile-name ecs-hello-profile --role-name ecs-hello-role
aws iam delete-instance-profile --instance-profile-name ecs-hello-profile
aws iam detach-role-policy --role-name ecs-hello-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
aws iam delete-role-policy --role-name ecs-hello-role --policy-name s3-policy
aws iam delete-role --role-name ecs-hello-role