Posted on July 31, 2026 by Albena Galabova, Itgix

CNCF projects highlighted in this post

KEDA logo Kubernetes logo

In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in an Amazon SQS queue. In other cases, pods may continue running long after a traffic spike has passed.

For asynchronous, queue-based workloads, backlog is the true scaling signal – not infrastructure utilization.

In this article, you’ll learn:

Why this matters:
In queue-driven systems, delayed message processing directly impacts users and downstream systems. Scaling based on SQS depth aligns autoscaling with actual demand, enabling faster burst handling and lower idle costs when queues are empty.

A graphic featuring KEDA, Kubernetes and Itigix logos

1. Prerequisites

Before implementing KEDA-based autoscaling with Amazon SQS, ensure you have:

2. Architecture and Request Flow

This architecture relies on KEDA observing Amazon SQS queue metrics and managing a Kubernetes Horizontal Pod Autoscaler (HPA) based on backlog.

Request flow:

  1. Producers send messages to the Amazon SQS queue
  2. KEDA polls queue attributes to determine backlog
  3. KEDA updates the target metrics in the HPA
  4. The HPA scales the worker Deployment
  5. Kubernetes schedules additional Pods to process messages
  6. As the queue drains, replicas scale back down

This model keeps autoscaling decisions tied directly to outstanding work.

3. Implementation Steps

Install KEDA via Helm

The recommended way to install KEDA is via Helm:

# Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts
helm repo update

# Install KEDA into a dedicated namespace
helm install keda kedacore/keda \
  --namespace keda \
  --create-namespace

What this does:
Deploys the KEDA operator and metrics API server, along with CRDs such as ScaledObject and TriggerAuthentication.

Deploy the Worker with AWS Identity Attached

An SQS consumer typically requires permissions such as:

Example IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ConsumeFromQueue",
      "Effect": "Allow",
      "Action": [
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:orders-queue"
    }
  ]
}

What this does:
Enforces least-privilege access by granting permissions only for the required SQS queue.

Configure KEDA Authentication and ScaledObject

KEDA connects the deployment to the SQS queue using TriggerAuthentication and ScaledObject. Using queueURLFromEnv avoids hardcoding the queue URL.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: sqs-processor-auth
  namespace: workers
spec:
  podIdentity:
    provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-processor
  namespace: workers
spec:
  scaleTargetRef:
    name: sqs-processor
  pollingInterval: 10
  cooldownPeriod: 120
  minReplicaCount: 0
  maxReplicaCount: 30
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: sqs-processor-auth
      metadata:
        queueURLFromEnv: QUEUE_URL
        awsRegion: us-east-1
        queueLength: "10"
        activationQueueLength: "1"

What this does:
Defines how KEDA authenticates with AWS and how replicas are calculated. Each pod targets 10 messages, and scale-to-zero is enabled when the queue is empty.

4. Replica Calculation Logic

KEDA calculates outstanding work as:

ApproximateNumberOfMessages
+ ApproximateNumberOfMessagesNotVisible
(+ delayed messages, if enabled)

Replica calculation:

desired replicas = ceil(outstanding messages / queueLength)

Example (queueLength: “10”):

Final replica counts are bounded by minReplicaCount and maxReplicaCount.

5. Verification

After applying the manifests:

1. Send 50 messages to the SQS queue

2. Inspect the ScaledObject:

kubectl get scaledobject sqs-processor -n workers

3. Watch pod scaling:

kubectl get pods -n workers -w

4. Confirm pods scale back to zero after processing completes

6. Troubleshooting Common Issues

No scale-out

Too many pods

Never scales to zero

HPA exists but no scaling

7. Production Tuning Tips

Conclusion

Queue-based workloads benefit most when autoscaling is driven by the amount of work waiting to be processed rather than traditional infrastructure metrics. Backlog-aware scaling enables applications to react more quickly to changing demand while reducing unnecessary resource consumption during quieter periods.

Although this article demonstrated the approach using Amazon SQS, Amazon EKS, and KEDA, the same design pattern applies across many event-driven architectures and messaging platforms. The key is selecting a scaling signal that accurately represents workload demand and tuning the autoscaling behaviour to match the characteristics of the application.

As organisations increasingly adopt asynchronous, event-driven systems, workload-aware autoscaling becomes an important part of building resilient, efficient, and cost-effective Kubernetes deployments.