The Use Case
Sometimes we cannot reproduce a bug in our GoLang software within our CI/CD pipelines, they only seem to occur within our dev or staging environments.
The What
This tutorial will give you a basic outline of how to deploy a debug container, which will contain the Delve aka dlv debugger along with the software in a binary that has been compiled specifically for the purpose of debugging. This is extremely handy for debugging containerized, headless processes.
Use Caution
Take precautions and always be thinking of security. Only deploy in closed, test-oriented environments.
Okay Now What
Debugging a headless process within a container is essentially the same as debugging locally, there only needs to be certain considerations made within our Dockerfile’s compilation commands, but more on that later.
To get started, create a server of your choosing. I’m a fan of Go-Chi, here’s an example straight from their repo:
package main
import (
"net/http"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
)
func main() {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.Get("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("welcome"))
})
http.ListenAndServe(":3000", r)
}
Docker Magic
Docker File
# Build ENV
FROM GoLang:1.15-alpine AS build-env
RUN apk update && apk upgrade
RUN apk add git g++
RUN go get github.com/go-Delve/dlv/cmd/dlv
ADD . /dockerdev
WORKDIR /dockerdev
RUN go build -gcflags="all =-N -l" -o server
# Running Container
FROM GoLang:1.15-alpine
EXPOSE 8001
WORKDIR /
COPY --from=build-env /go/bin/dlv /
COPY --from=build-env /server /
CMD["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--continue", "--check-go-version", "--accept-multiclient", "exec", "/server"]
Uuuuuhhhh, what?
This is a multistage Dockerfile, using two different containers; one for building and one for deployment. Why use a multistage Docker build? Because the container that we end up deploying is very small, if we were to try to use the GoLang-Alpine image or, God forbid, the stock GoLang image, the container we would end up deploying would be MASSIVE, something in the neighborhood of 500+ MB, which we definitely don’t want since we’re genuinely concerned about the efficiency of our containers. This isn’t Java, have some standards! Do note that a specific version of GoLang is used here, I always use the previous release with my builds, to ensure Delve compatibility and overall stability.
The top half of the Dockerfile above is the build container, which will do all the actual building of our software and drag in the Delve debugger so that we can use it in our running container. The magic really happens on RUN go build -gcflags="all =-N -l" -o server, as this compiles the Go binary into a manner that makes it compatible with Delve.
The bottom half of the Dockerfile lays out the structure and commands of the container that we’ll end up running. Essentially all this end does is expose a port, copy over our compiled binary along with Delve, and then execute a long command with lots of flags. That last line, CMD["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--continue=true", "--check-go-version", "--accept-multiclient", "exec", "/server"] is divided into two halves, the first half being dedicated to running Delve, and the second half executing the compiled binary of our server. Actually, the latter portion of running the server is literally just the command /server, everything else before it is Delve.
In nutshell, this is basically telling Delve to run our program, not the typical go run that you’re used to. The Delve headless server will accept multiple client connections and be listening over port 40000 for them. Your docker run command will remain the exact same as it did before, just ensure that you allow interactive terminals within the container (default behavior).
Getting into the Container
Start an interactive terminal with the alpine container using this command: docker exec -it <container-name> /bin/ash
Getting into Delve
Remember that by using that long Delve command within our run container, we’re actually using Delve to run our software. All we need to do is attach to the process itself, using the command ./dlv connect :40000. From there you should get the Delve prompt (dlv).
Conclusion
From there, you can start debugging. Enjoy! The real magic of all of this is the docker build, Delve is smart enough that we can use it to actually run our software for us, and be able to peer into it headlessly. Please give many kudos to Derek Parker who wrote Delve.