Mustapha Awwal's Blog

ECS Managed Scaling Not Working (EC2 Capacity Provider)

Mustapha Awwal's photo
Mustapha Awwal
·

3 min read

Intro

Amazon Elastic Container Service (ECS) is a managed AWS orchestration service that simplifies running containerized applications. Compared to Amazon Elastic Kubernetes Service (EKS), ECS is relatively easier to use and integrates seamlessly with other AWS services, making it a popular choice for developers and teams looking for efficient container management.

However, while using ECS for the first time, I ran into a surprising issue: the managed scaling went out of control! 😂 Specifically, ECS was setting the desired capacity in my Auto Scaling Group (ASG) far beyond what I expected. This issue arose while I was using the binpack placement strategy, which is supposed to optimize EC2 instance usage by packing tasks onto one EC2 instance until it's fully utilized before spinning up additional instances. Instead of achieving efficient resource utilization, I found ECS adding more instances prematurely. 🤦‍♂️

As shown in the image, ECS is setting the desired capacity in the Auto Scaling Group (ASG) to 3 😅, while I was expecting it to stay at just 1.

Thanks for the clarification! Here’s the corrected section:

The Issue

Let’s cut the long story short. The root of the problem turned out to be the awsvpc mode in my task definition. When you use awsvpc mode, each task running on an instance requires its own Elastic Network Interface (ENI).

The instance type I was using only supported 3 ENIs, which is why ECS was setting the desired capacity abnormally. This behavior occurred because ECS managed scaling adjusts the desired capacity based on the infrastructure's defined requirements, and in this case, my setup needed more ENIs than my instance type could provide.

To check how many ENIs your instance type supports, you can use the following command:

aws ec2 describe-instance-types --instance-types <INSTANCE_TYPE> --query "InstanceTypes[0].NetworkInfo.MaximumNetworkInterfaces" --output table

This command will give you the maximum number of ENIs your instance type can handle, which is a crucial factor when using awsvpc mode in ECS.

The Fix

To resolve the issue, here are your options:

  1. Switch to Bridge Mode
    If your application doesn’t specifically require awsvpc mode, switching to bridge mode can help avoid ENI limitations. Additionally, when using bridge mode, I recommend enabling dynamic port mapping by setting the host port to 0 in your task definition. This ensures that ECS assigns ports dynamically, preventing conflicts when running multiple replicas of the same task.

  2. Upgrade to a Larger Instance Type
    If awsvpc mode is essential for your setup, consider switching to an instance type that supports a higher number of ENIs. This allows you to run more tasks per instance without hitting network interface limits.

Final Advice

When troubleshooting issues like this, it’s essential to go straight to the root cause. Ask, “What exactly isn’t working?” and systematically analyze the error messages or unexpected behavior. Avoid making assumptions—they can lead to wasted time and unnecessary frustration. Stay focused, and work step by step. 🚀

AWSCloud