Amazon EC2 Container Service is the container management solution from AWS. It allows you to run a container cluster over Amazon EC2 instances. Amazon EC2 Container Service is capable of orchestrating containers based on resource requirements.
This post explains the steps required to run logstash in Amazon EC2 Container Service. In this example, logstash receives logs through an ELB and forwards to Amazon Elasticsearch service.
Components
The following are the main Amazon EC2 Container Service Components:
Tasks
Tasks work the same way as a docker-compose file. It defines the application containers. Tasks can be defined using AWS console or can be built from JSON representation.
Task definitions are versioned and cannot be edited. Updated task definitions create a new task version.
Services
Services try to maintain a specific number of tasks in the cluster and move around the tasks based on resource requirement.
Cluster
Cluster is a logical grouping of EC2 instances where tasks can be run.
Container instances
EC2 instances running in a Cluster are called container instances. Container instances must run Amazon ECS container agent to register into the clusters. Amazon ECS-optimized AMIs comes with the agent already installed. Agent can be manually installed if another AMI needs to be used.
The Amazon ECS container agent makes calls to Amazon ECS. Container instances need to be launched with an IAM role that authenticates to the account and provides the required resource permissions.
For more details on Amazon ECS, refer to AWS documentation.
Extra Features
Factors that need to be considered while running logstash in Amazon EC2 Container Service include the following:
IAM Role in Tasks
To avoid connecting from logstash to Elasticsearch service using AWS keys, an IAM role needs to be created and applied to logstash Tasks. This is the best practice for controlling application access to ECS and other areas of AWS.
Load Balancing
Elastic Load Balancing options can be selected when an ECS Service is created. ELB needs to be created before service configuration. ELB load balance traffic across containers. It can perform health checks and recovery of unhealthy containers. If a service is updated while running in an ELB, it would first drain the existing connections then will start new tasks. Classic Elastic Load Balancing and Application Load Balancing are supported in AWS ECS.
With Classic Elastic Load Balancing, there can only be one container with same port per instance. With Application Load Balancing, multiple tasks per container instance can be used, but it only allows http and https connections.
In this setup, Classic Elastic Load Balancing is used, as incoming logs are not using http/https.
Implementation steps
The following are the basic implementation steps:
- Build and export image
- Create IAM Role, Cluster and Task Definition
- Start Container Instances and Service.
Container will send the logs to AWS Elasticsearch service and Instances will be autoscaled as more task numbers are added.
1. Build and export image
Container images can be built from Dockerfile, running container or packer. The process can be moved to a continuous workflow using triggered automatic builds and even automatic deployments with jenkins.
In this example, container image is built from Dockerfile. Here’s how to create a Dockerfile:
# ls
Dockerfile logstash.conf logstash-forwarder.crt logstash-forwarder.key
# cat Dockerfile
FROM logstash:2.4
COPY logstash.conf /config-dir/
COPY logstash.crt /config-dir/
COPY logstash.key /config-dir/
RUN /opt/logstash/bin/logstash-plugin install logstash-output-amazon_es
RUN /opt/logstash/bin/logstash-plugin install logstash-patterns-core
RUN /opt/logstash/bin/logstash-plugin install logstash-output-datadog
CMD ["-f", "/config-dir/logstash.conf"]
This Dockerfile does following,
- Use standard docker image logstash.
- Copy logstash certificates.
- Copy configuration file
- Install logstash plugins.
Amazon EC2 Container Registry can be used as a private registry to store images. A ECR repo name can look like this. https://6666690109902.dkr.ecr.us-east-1.amazonaws.com/abc/logstash.
To export the built image,
Login to ECR.
# aws ecr get-login --region us-east-1
Run the build command.
# docker build -t abc/logstash .
Set tag to the image
# docker tag abc/logstash:latest 6666690109902.dkr.ecr.us-east-1.amazonaws.com/abc/logstash:latest
Push to ECR.
# docker push 6666690109902.dkr.ecr.us-east-1.amazonaws.com/abc/logstash:latest
2. Create IAM Role, Cluster and Task Definition
Create an IAM Role and create a policy with following rules and attach it to the role. This role will allow access to elasticsearch service with arn value aws:es:us-east-1:6666690109902:domain/es2.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:6666690109902:domain/es2*"
}
]
}
Create a Cluster under Amazon EC2 Container Service home and create a task definition with logstash:latest image and specify the previously created IAM role as ‘Task Role’.
3. Start Container Instances and Service
Create an instances with default role ecsInstanceRole using an ECS Optimized AMI. Add following User Data so that the container instances will be joined under the correct cluster,
#!/bin/bash
echo ECS_CLUSTER=utils >> /etc/ecs/ecs.config
Create an ELB with following Health Check details.
Ping Target : TCP:12346
Timeout : 30 seconds
Interval : 60 seconds
Unhealthy threshold : 2
Healthy threshold : 4
Listeners are following,
TCP 12345
TCP 12346
12345 and 12346 are the ports which logstash listens for incoming logs. A service needs to be created with ECSServiceRole as IAM Role and new ELB.
To scale, edit the service and increase “Desired count”. For stopping a task started by a service, the “Desired count” in service need to be updated to 0.
ECS autoscaling can be enabled while creating the service. ECS autoscaling autoscales tasks.
ECS Instance (EC2) autocaling can also be enabled along with ECS autocaling so that new EC2 instances will be spawned when a watermark is hit. For this to function, in ECS autoscaling, policy watermark (eg: MemoryReservation watermark) needs to be tuned and reduced below ECS Instance autoscaling.
As logstash is using the classic load balancer in this setup, only one logstash contaner will be run in an EC2 instance. This means that complete autocaling setup described above would not work. When ECS autoscaling tries to start a new logstash task, it would not be able to do so as the task would need a new EC2 instance and autocaling would only create a new EC2 instance when its watermark is hit.
AWS command line tools and ECS CLI can be used to create and manage Amazon EC2 Container Service.
No comments