Why my load balancer returned 502 (the task was fine)

2025/03/23
awsecsload-balancingtarget-grouphttps

I had a Go server running on AWS ECS, and I wanted to put it behind a domain name with HTTPS. Getting the whole thing deployed, from Docker to ECR to task definitions to the load balancer, is its own story, and I wrote that part up in from docker push to HTTPS. This post is about one wall I hit near the end: a 502 error that made no sense.

For HTTPS, I put a load balancer (an ALB) in front of the service. Between the load balancer and my service sits a target group. The target group is how the load balancer knows where to send traffic. You can think of it as a label that points at your running tasks, similar to a selector.

My Go app was listening on port 80, plain HTTP.

When I configured the load balancer, I chose HTTPS on port 443. That is correct: users should reach me over HTTPS.

Then I created the target group, and I set it to HTTPS:443 as well, because that felt consistent. That was the mistake.

The part that made no sense

Here is what I saw:

  • The task's public IP worked. I opened it directly and the Go server responded.
  • The target group was pointing at the correct private IP of my task.
  • But every request through the load balancer returned 502 Bad Gateway.

Each piece looked right, and it still failed. That is the worst kind of bug.

The fix

The target group's protocol has to match what your service speaks, not what the load balancer speaks.

My service speaks plain HTTP on port 80. So the target group must be HTTP:80, even though the load balancer is HTTPS:443.

wrong:  user → LB (HTTPS 443) → target group (HTTPS 443) → task (HTTP 80)
right:  user → LB (HTTPS 443) → target group (HTTP 80)   → task (HTTP 80)

I changed the target group to HTTP:80, and the 502 was gone.

Why it happens

The thing I had wrong: HTTPS stops at the load balancer.

The load balancer terminates TLS. It receives the encrypted request from the user, decrypts it, and then makes a separate, fresh request to the task, using the protocol set in the target group. That second request travels inside the VPC, a private network.

So when the target group was HTTPS:443, the load balancer tried to start a TLS handshake with my Go app. But my Go app only speaks plain HTTP on port 80. No certificate, no TLS on its side. The handshake failed, the load balancer got nothing usable back, and it reported 502 Bad Gateway to the user.

My wrong picture was "the whole path is HTTPS." It is not. HTTPS covers only the first hop, from the user to the load balancer. The second hop, from the load balancer to the task, uses whatever the target group says. For most apps, that is plain HTTP.

So what is a target group, really

A target group is not a server or a machine. It is a logical pointer. It tells the load balancer two things: where the tasks are, and how to reach them, meaning the protocol and the port. Set that protocol wrong, and the load balancer ends up speaking a language your service does not understand.

What if you want it encrypted all the way?

The fix above is the default: TLS ends at the ALB, and the LB-to-task hop runs plain HTTP. That hop is inside your VPC, so plain HTTP there is fine for most apps.

If a compliance rule wants that hop encrypted too, two ways to go further:

  • Re-encryption (target group on HTTPS): your Go server listens on TLS with its own certificate (self-signed is fine, the ALB doesn't validate it). The ALB still decrypts to route the request, then re-encrypts to the server. You keep the ALB's L7 routing; the cost is a certificate on the task and two handshakes.
  • NLB passthrough (L4 TCP): swap the ALB for a Network Load Balancer. It forwards bytes without decrypting, so your Go server terminates TLS itself. One TLS session the balancer never touches; the cost is losing L7 routing (no path or header rules).