Why my load balancer returned 502 (the task was fine)

2025/03/23
awsecsload-balancingtarget-grouphttps

I had a Go server running on AWS ECS, and I wanted to put it behind a domain name with HTTPS. Getting the whole thing deployed, from Docker to ECR to task definitions to the load balancer, is its own story, and I wrote that part up in from docker push to HTTPS. This post is about one wall I hit near the end: a 502 error that made no sense.

For HTTPS, I put a load balancer (an ALB) in front of the service. Between the load balancer and my service sits a target group. The target group is how the load balancer knows where to send traffic. It holds a list of registered targets, here my running tasks.

My Go app was listening on port 80, plain HTTP.

When I configured the load balancer, I chose HTTPS on port 443. That is correct: users should reach me over HTTPS.

Then I created the target group, and I set it to HTTPS:443 as well, because that felt consistent. That was the mistake.

The part that made no sense

Here is what I saw:

  • The task's public IP worked. I opened it directly and the Go server responded.
  • The target group was pointing at the correct private IP of my task.
  • But every request through the load balancer returned 502 Bad Gateway.

Each piece looked right, and it still failed. That is the worst kind of bug.

The fix

The target group's protocol has to match what your service speaks, not what the load balancer speaks.

My service speaks plain HTTP on port 80. So the target group must be HTTP:80, even though the load balancer is HTTPS:443.

wrong:  user → LB (HTTPS 443) → target group (HTTPS 443) → task (HTTP 80)
right:  user → LB (HTTPS 443) → target group (HTTP 80)   → task (HTTP 80)

I changed the target group to HTTP:80, and the 502 was gone.

Why it happens

The thing I had wrong: HTTPS stops at the load balancer.

The load balancer terminates TLS. It receives the encrypted request from the user, decrypts it, and then makes a separate, fresh request to the task, using the protocol set in the target group. That second request travels inside the VPC, a private network.

So when the target group was HTTPS:443, the load balancer switched to TLS to reach the task. On which port? The ECS service's load balancer config says containerPort: 80, so ECS should register each task on 80; the target group's 443 only applies when a registration names no port. So the load balancer most likely tried a TLS handshake with my Go app on port 80. But my Go app only speaks plain HTTP there. No certificate, no TLS on its side. The handshake failed, the load balancer got nothing usable back, and it reported 502 Bad Gateway to the user. AWS's ALB troubleshooting page lists an SSL handshake error with the target as one cause of 502.

The docs don't say outright that ECS registers on 80. To check, the target group's Targets tab lists the port each target is registered on.

My wrong picture was "the whole path is HTTPS." It is not. HTTPS covers only the first hop, from the user to the load balancer. The second hop, from the load balancer to the task, uses whatever the target group says. For most apps, that is plain HTTP.

So what is a target group, really

A target group is not a server or a machine. It is a logical pointer. It tells the load balancer two things: where the tasks are, and how to reach them, meaning the protocol and the port. Set that protocol wrong, and the load balancer ends up speaking a language your service does not understand.

What if you want it encrypted all the way?

The fix above is the default: TLS ends at the ALB, and the LB-to-task hop runs plain HTTP. That hop is inside your VPC, so plain HTTP there is fine for most apps.

If a compliance rule wants that hop encrypted too, two ways to go further:

  • Re-encryption (target group on HTTPS): your Go server listens on TLS with its own certificate (self-signed is fine, the ALB doesn't validate it). The ALB still decrypts to route the request, then re-encrypts to the server. You keep the ALB's L7 routing; the cost is a certificate on the task and two handshakes.
  • NLB passthrough (L4 TCP): swap the ALB for a Network Load Balancer. With a TCP listener it forwards bytes without decrypting, so your Go server terminates TLS itself. One TLS session the balancer never touches; the cost is losing L7 routing (no path or header rules).

Where the certificate goes in each setup

A certificate is really two files, kept on whichever end of a TLS connection acts as the server:

  • certificate (say fullchain.pem): public. It holds the domain name (say api.example.com), an expiry date, and the signature of the CA that issued it (Let's Encrypt, ACM and the like).
  • private key (say privkey.pem): stays on the server only, and proves the certificate belongs to it.

When a browser connects, the server hands over the certificate, and the browser checks that a trusted CA signed it, that the domain matches, and that it hasn't expired. If any check fails, the browser shows a "connection not secure" warning.

But the certificate is public, and anyone can hand a copy to a browser, so handing it over isn't enough. After sending fullchain.pem, the server signs this handshake with privkey.pem. The browser checks the signature with the public key inside the certificate; if it holds, the other side really has the certificate's private key.

The requests and responses after that are encrypted with a separate session key, not the private key. During the handshake, the browser and the server each make a temporary key pair and send only the public half to the other. Each side combines its own unsent half with the other's public half and arrives at the same session key (this is ECDHE). Someone watching the network sees only the two public halves and can't compute the key.

When the connection ends, both sides throw those temporary keys away; privkey.pem was only ever used to sign. So even if someone recorded the encrypted traffic and later steals privkey.pem, they still can't compute that session key or decrypt the recording. What they can do is impersonate the server to new connections.

The certificate goes on the end that terminates TLS. In the three setups below, that is the ALB, the ALB and the task, and the task:

default:          browser ══TLS══> ALB ──HTTP──> task
re-encryption:    browser ══TLS══> ALB ══TLS══> task
NLB passthrough:  browser ═════════TLS═════════> task (NLB only forwards)
CertificatesWho hands the browser a certificateCertificate on the task?What the task's certificate must be
default1 (ALB)the ALB (ACM certificate)nonone
re-encryption2 (one on the ALB, one on the task)the ALB (ACM certificate)yesself-signed is fine
NLB passthrough1 (task)the taskyesa real certificate browsers trust

Pick HTTPS under "Target group protocol" and HTTP :80 under "Go server listens on" in the embed below to reproduce the 502 from above (in the embed, a sealed letter is TLS and a postcard is plain HTTP).

"The certificate on the task" means the two files the Go server reads. The task is the container ECS runs, and the Go server is the process inside it. Change the line that listens on HTTP to listen on HTTPS; net/http supports it out of the box, no extra package:

// before
http.ListenAndServe(":80", mux)
 
// after
http.ListenAndServeTLS(":443", "/etc/tls/fullchain.pem", "/etc/tls/privkey.pem", mux)

The Go code is identical in both setups; only the source of those two files differs.

Re-encryption: a self-signed certificate on the task is enough

Re-encryption has two TLS hops, so it has two certificates:

  • Browser to ALB: the ALB presents the ACM certificate on its listener, same as the default setup. This is the only one the browser checks.
  • ALB to task: the task is the server on this hop, so it needs its own certificate.

Only the ALB reads the second one, and the ALB target group docs say the ALB doesn't validate target certificates, so self-signed or expired ones work. The docs' reason: the ALB and its targets are in a VPC, where AWS authenticates traffic at the packet level, so an invalid certificate still doesn't open the door to man-in-the-middle attacks.

So generate one yourself and put it on the task. No CA, no renewals:

openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
  -keyout privkey.pem -out fullchain.pem -subj "/CN=internal"

A real certificate works here too, and the ALB accepts it the same way. But since the ALB doesn't check this certificate anyway, a real one makes nothing safer, and adds a certificate fee and renewals to handle.

The two files openssl makes, and how they get onto the task

After that openssl line, the directory has two new text files:

./
├── fullchain.pem   18 lines, the certificate, public
└── privkey.pem     28 lines, the private key, never share it

Here is a real pair generated in November 2026, with the middle lines cut. It's a demo pair and isn't used on any machine:

$ cat fullchain.pem
-----BEGIN CERTIFICATE-----
MIIC5jCCAc6gAwIBAgIUOYPZWAAWIVIqs1YLbgDPV1kip0IwDQYJKoZIhvcNAQEL
BQAwEzERMA8GA1UEAwwIaW50ZXJuYWwwHhcNMjYxMTEwMDgzMzM1WhcNMzYxMTA3
(13 lines cut)
K/IMkBE8ar53A+947AKWd/nhiAN6rMfwcMY=
-----END CERTIFICATE-----

$ cat privkey.pem
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCcwwzKL8zvvAzv
WLTH/AXdmX8MiMI4yPxreyf0+VrXKUzYJ1jPkewz2tSXmLO0Vdgo/JVE8LepzqVo
(23 lines cut)
IIloJ7SAvHT3RqQXCrWLL/I=
-----END PRIVATE KEY-----

The certificate is encoded; openssl x509 shows what it says:

$ openssl x509 -in fullchain.pem -noout -subject -issuer -dates
subject=CN=internal
issuer=CN=internal
notBefore=Nov 10 08:33:35 2026 GMT
notAfter=Nov  7 08:33:35 2036 GMT

subject is who the certificate belongs to, issuer is who signed it. Both are internal: it signed itself, which is what self-signed means. One from Let's Encrypt would show a Let's Encrypt CA as the issuer.

In the embed above, the browser, ALB and Go server boxes list the files each machine holds, and they change with the options. Set the target group protocol to HTTPS and Go to "HTTPS self" (self-signed) for re-encryption: the Go server box shows these two files under /etc/tls/, and the ALB box shows only an ACM certificate ARN.

So where does the ALB's private key come from? ACM's data protection page lays out the flow. ACM stores every private key encrypted under a KMS (Key Management Service) key called aws/acm. When you attach the certificate to a listener, ACM sends the ALB the certificate and the encrypted private key, and creates a KMS grant, a permission that lets the ALB use aws/acm to decrypt. The ALB decrypts the private key and uses it for the handshake with the browser.

All of this happens inside AWS. The ALB has no file you could open, which is why its box in the embed lists only an ARN, and whoever deploys the service never gets the private key either. That is also why a regular ACM certificate can't be put on the task.

Two ways to get the files into the container:

  • COPY them into the image: add COPY fullchain.pem privkey.pem /etc/tls/ to the Dockerfile. Simplest, but the private key goes into ECR with the image, and anyone who can pull the image has it.

  • Store them in Secrets Manager and inject at task start: save each file's full contents, -----BEGIN and -----END lines included, as its own secret:

    aws secretsmanager create-secret --name alb-target-cert --secret-string file://fullchain.pem
    aws secretsmanager create-secret --name alb-target-key  --secret-string file://privkey.pem

    Then list both ARNs under secrets in the task definition. The ECS docs inject them as environment variables, not files, so Go reads them from the environment:

    "secrets": [
      { "name": "TLS_CERT", "valueFrom": "arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:alb-target-cert-AbCdEf" },
      { "name": "TLS_KEY",  "valueFrom": "arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:alb-target-key-AbCdEf" }
    ]
    cert, err := tls.X509KeyPair([]byte(os.Getenv("TLS_CERT")), []byte(os.Getenv("TLS_KEY")))
    if err != nil {
        log.Fatal(err)
    }
    srv := &http.Server{
        Addr:      ":443",
        Handler:   mux,
        TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}},
    }
    log.Fatal(srv.ListenAndServeTLS("", ""))

    The same page lists two limits. The secret is injected once, when the container starts; change it later and running tasks keep the old value until a force new deployment starts fresh ones. And the task execution role needs permission to read from Secrets Manager.

NLB passthrough: the task needs a real certificate

With passthrough, the browser does TLS directly with the task and checks the certificate the task presents, so self-signed won't do. A real certificate can come from two places:

  • Let's Encrypt: free. Go's official golang.org/x/crypto/acme/autocert requests and renews certificates automatically. It isn't in the standard library, so go get it:
    m := &autocert.Manager{
        Prompt:     autocert.AcceptTOS,
        HostPolicy: autocert.HostWhitelist("api.example.com"),
        Cache:      autocert.DirCache("/var/cache/certs"),
    }
    srv := &http.Server{Addr: ":443", Handler: mux, TLSConfig: m.TLSConfig()}
    srv.ListenAndServeTLS("", "")
    With several tasks behind the NLB, DirCache keeps a separate copy per task, so each task requests its own. To share one certificate, point DirCache at an EFS directory every task mounts, or implement the autocert.Cache interface (Get, Put, Delete) and store certificates in S3.
  • ACM exportable public certificate: a regular ACM certificate can only be attached to AWS services such as an ALB or CloudFront, and its private key can't be taken out. In June 2025 AWS launched an exportable version: mark it exportable when you request it, then download the certificate and private key and put them on the task. Per the ACM docs:
    • it is valid for 198 days, and ACM renews it 45 days before it expires;
    • after renewal you deploy the new certificate to the task yourself; ACM only sends a notification;
    • it costs extra; see the ACM pricing page for the amount.