Deploy Docker containers using AWS Opsworks

Posted in: Cloud, DevOps, Technical Track

Introduction

This post is about how to deploy Docker containers on AWS using Opsworks and Docker Composer.
For AWS and Docker, the introduction isn’t required. So, let’s quickly introduce Opsworks and Docker Composer.

Opsworks

Opsworks is a great tool provided by AWS, which runs Chef recipes on your Instances. If the instance is an AWS instance, you don’t pay anything for using Opsworks, but you can also manage instances outside of AWS with a flat cost just by installing the Agent and registering the instance on Opsworks.

Opsworks Instances type

We have three different types of instances on Opsworks:

1. 24x7x365
Run with no stop

2. Time based
Run in a predefined time. Such as work hours.

3. Load based
Scale up and down according to the metrics preconfigured.

You can find more details here.

Custom JSON

Opsworks provides Chef Databags (variables to be used in your recipes) via Custom JSON, and that’s the key to this solution. We will manage everything just changing a JSON file. This file can become a member of your development pipeline easily.

Life cycle

Opsworks has five life cycles:
1. Setup
2. Configure
3. Deploy
4. Undeploy
5. Shutdown
We will use setup, deploy, and shutdown. You can find more details about Opsworks life cycle here.

Docker Compose

Docker Compose was originally developed under the Fig project. Nowadays, the fig is deprecated, and docker-compose is a built-in component of Docker.
Using docker-compose, you can manage all containers and their attributes (links, share volumes, etc.) in a Docker host. Docker-compose can only manage containers on the local host where it is deployed. It cannot orchestrate Docker containers between hosts.
All configuration is specified inside of a YML file.

Chef recipes

Using Opsworks, you will manage all hosts using just one small Chef cookbook. All the magic is in translating Custom JSON file from Opsworks to YML file to be used by docker-compose.
The cookbook will install all components (Docker, pip, and docker-compose), translate Custom JSON to YML file and send commands to docker-compose.

Hands ON

Let’s stop talking and see things happen.

We can split it into five steps:

  1. Resources creation
    1. Opsworks Stack
        1. Log into your AWS account
        2. Go to Services -> Management Tools -> Opsworks
          Accessing Opsworks menu
        3. Click on Add stack (if you already have stacks on Opsworks) or Add your first stack (if it’s the first time you are creating stacks on opsworks)
        4. Select type Chef 12 stack
          Note: The Chef cookbook used in this example only supports Chef12
        5. Fill out stack information
          aws_opsworks_docker_image02
          Note:
          – You can use any name as stack name
          – Make sure VPC selected are properly configured
          – This solution supports Amazon Linux and Ubuntu
          – Repository URL https://bitbucket.org/tnache/opsworks-recipes.git
        6. Click on advanced if you want to change something. Changing “Use OpsWorks security groups” to No can be a good idea when you need to communicate with instances which are running outside of Opsworks
        7. Click on “Add stack”
    2. Opsworks layer
        1. Click on “Add a layer”
        2. Set Name, Short name and Security groups. I will use webserver

      Note:
      Use a simple name because we will use this name in next steps
      The Name web is reserved for AWS internal use

        1. Click on “Add layer”

      aws_opsworks_docker_image03

    3. Opsworks Instance
        1. Click on “Instances” on left painel
        2. Click on “Add an instance”
        3. Select the size (instance type)
        4. Select the subnet
        5. Click on “Add instance”

      aws_opsworks_docker_image05

  2. Resources configuration
    1. Opsworks stack
        1. Click on “Stack” on left painel
        2. Click on “Stack Settings”
        3. Click on “Edit”
        4. Find Custom JSON field and paste the content of the file bellow

      custom_json_1

      1. Click on “Save”
    2. Opsworks layer
        1. Click on “Layers” on left painel
        2. Click on “Recipes”
        3. Hit docker-compose and press enter on Setup
        4. Hit docker-compose::deploy and press enter on Deploy
        5. Hit docker-compose::stop and press enter on Deploy
        6. Click on “Save”

      aws_opsworks_docker_image04

  3. Start
    1. Start instance
        1. Click on start

      aws_opsworks_docker_image06

  4. Tests
    Note: Wait until instance get online state

      1. Open your browser and you should be able to see It works!
      2. Checking running containers

    aws_opsworks_docker_image07

  5. Management
      1. Change custom json to file bellow (See resources configuration=>Opsworks stack)

    custom_json_2

      1. Click on “Deployments” on left painel
      2. Click on “Run Command”
      3. Select “Execute Recipes” as “Command”
      4. Hit “docker-compose::deploy” as “Recipes to execute”
      5. Click on “Execute Recipes”

    Note: Wait until deployment finish

      1. Checking running containers

    aws_opsworks_docker_image08

email
Want to talk with an expert? Schedule a call with our team to get the conversation started.

10 Comments. Leave new

Nice explanation about AWS. This guy has a domain about this tool. Nice job.

Reply
Thiago Carvalho
April 28, 2016 4:58 pm

Thanks, Roberson. I hope it would help you.

Reply

Yes, it has helped.

Reply
ISABEL C NACHE BORGES
April 29, 2016 7:22 pm

s2 – the best ;)

Reply

Hi! Thanks for the simple tutoria. I am very new to Docker and AWS.

I am getting an error when starting the instance; not sure where i went wrong; error as below;

Mixlib::ShellOut::ShellCommandFailed
————————————
Expected process to exit with [0], but received ‘1’
—- Begin output of docker-compose pull —-
STDOUT:
STDERR: Top level object in ‘./docker-compose.yml’ needs to be an object not ”.
—- End output of docker-compose pull —-
Ran docker-compose pull returned 1

Resource Declaration:
———————
# In /var/chef/runs/ae3c1b70-8f6a-4425-a28d-2737e83ed68a/local-mode-cache/cache/cookbooks/docker-compose/recipes/pull.rb

1: execute ‘pull containers images’ do
2: cwd ‘/docker-compose/’
3: command ‘docker-compose pull’
4: case node[:platform]
5: when ‘ubuntu’
6: environment ‘COMPOSE_API_VERSION’ => ‘1.18’
7: end
8: end

Reply

additional information:

I used an m3-medium instance running Ubuntu 14.04. I am not also using a VPC.

Reply
Thiago Carvalho
August 5, 2016 1:24 pm

Hi Philip,
Thanks for your comment.
It seems to be an issue in Custom JSON. Have you used your own or provided by me? If you created your own CustomJson make sure the layer name is matching.
If possible, share your custom JSON privately with me and I can try to help you.

Best regards,

Thiago

Reply

I found the exact same issue as Phillip, above. Turns out the layer name in the custom json didn’t match the Opsworks layer, easily fixed. So, the “layer1” in the custom json needs to change to match the name of your web server layer in OpsWorks.

Reply

Seems when i do this, throw ELB in front of the server. ELB does 443 -> 80 TCP and then run everything I run into some networking issues. I noticed with docker network inspect bridge that no Gateway is being created. Is this normal?

Reply
Thiago Carvalho
April 25, 2017 2:48 pm

Hello there,

Sorry about the delay.

What kind of network issue are you running into? We do SSL termination at ELB, ELB doing 443 -> 80 isn’t a problem due to it.

Regards,

Thiago

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *