Go on Steve Sun

A General Approach to Server-Side Performance Issues in Go

Tue, 06 May 2025 10:35:41 +0800

I recently ran into a performance issue. A customer reported that two Go-based service processes running in the background of their IPC device kept climbing in memory usage, peaking at 40% of total memory. One of those processes was our log-collection agent.

My first suspicion was a memory leak, because we’d had memory leaks caused by goroutine blocking in the past (I discussed this in Common Patterns of Memory Leaks in Go), so I started by reviewing everywhere we created and released goroutines.

After the last incident, we’d added goroutine leak detection at the unit-test level using go.uber.org/goleak. You only need to add a single line at the start of a test:

func TestXXX(t *testing.T) {
 defer goleak.VerifyNone(t)
 // ...
}

It automatically checks for lingering goroutines after the test finishes. For background goroutines that run on a delay, you can use wait or sleep in the test to wait for them to be released before the test case ends.

The first round of investigation ruled out problems caused by goroutines in the code itself. So I turned my attention to another suspect: scheduled tasks.

According to the customer, memory would slowly climb even with no foreground activity.

In our code, we use the third-party package github.com/robfig/cron/v3, which orchestrates scheduled tasks. Usage looks like this:

c = cron.New()
c.AddFunc("@every 10s", callbackFunc)

This structure defines a scheduled task. Its implementation is also based on goroutines, so I added Go’s built-in pprof to the dependencies in main.go, rebuilt the project binary, and deployed it to a test environment using the same hardware configuration as the customer. This way, once the project started, I could pull memory information from a specific port. (For more on pprof, see Profiling Go Programs.)

I used pprof’s interface to grab heap data at different intervals:

curl -o heap.1.out http://127.0.0.1:6060/debug/pprof/heap

Then used:

go tool pprof -http=:8099 -base heap.1.out heap.2.out

to compare the two results. In the web UI, I selected the In Use Space option, which let me see what memory hadn’t been released.

Even after this second round, I still didn’t find a memory leak. But this time I noticed that one of the scheduled tasks ran every 10 seconds, and CPU usage clearly spiked during execution. Looking at the code for this task, it used the third-party library github.com/shirou/gopsutil/process to query system process IDs and process names.

Looking at the library’s source code, I found that the way it queries process IDs is by loading all process information on the system into memory and then matching the ID or name. So, if the customer’s device has a lot of processes, each query consumes a large amount of memory.

Calling this library from a scheduled task that runs every 10 seconds is clearly very inefficient.

After further communication with the customer, we discovered that of the two high-memory processes, the other one also showed high CPU usage. So we had the customer send us a screenshot of the top command. The moment I saw the screenshot, the truth came into focus:

The customer’s IPC device was a lower-performance version—while it had plenty of memory, the CPU was struggling. When multiple processes run background tasks simultaneously, the CPU periodically maxes out, causing tasks to block. And the third-party library we use implements scheduled tasks on top of goroutines. When the previous task is blocked, the next task still creates a new background goroutine, causing goroutines to pile up in memory.

This was a goroutine blocking problem caused by high CPU usage and overly short intervals in the scheduled tasks.

Once we knew the cause, the rest was straightforward: optimize the code logic, ship a new version, and explain the issue to the customer…

That’s the full process of debugging this Go service performance issue. If you run into something similar, I hope this helps.

Dependency Inversion in Go

Thu, 21 Nov 2024 11:26:22 +0800

This article is fairly basic; it was material I used when training Java programmers in Go.

Why Dependency Inversion Principle (DIP)?

Dependency Inversion, also called dependency inversion or DIP, is a very important design principle in software development. Many programmers have never learned about it, or only know the general idea from Java Spring. Today I’d like to use a brief article and a simple Go example to explain how to implement dependency inversion in the simplest way.

If you don’t yet know what it is, you can refer to the description in Wikipedia, or read Martin Fowler’s article on DIP.

The Dependency Inversion Principle addresses a common risk in software development: dependency.

Try to recall:

When you try to use mocks to shield underlying details for testing, you find that the class you want to test references a large number of framework-provided interfaces, requiring you to mock many underlying implementations.
When you try to modify an old low-level class, but there are too many upper-layer service classes depending on it, you worry about side effects while refactoring the upper-layer code at every dependency point.

Let’s analyze these two scenarios:

In scenario 1, the application class depends on the implementation provided by the framework, making it difficult to separate the application class from the framework. The industry method for dealing with this problem is called Inversion of Control (IoC). The application class should not depend on the framework; instead, the framework provides slots, registering the application class with the framework, and the framework uniformly dispatches the application to execute the corresponding methods.

In scenario 2, the service class depends on the low-level class, making modifications to the low-level class increasingly difficult. The solution is Dependency Injection (DI). The upper-layer class does not directly reference the low-level class, but instead, the low-level class on which the upper-layer class depends is injected at the point of use.

Combining these two scenarios captures the core of the Dependency Inversion Principle:

High-level modules should not depend on low-level modules; both should depend on abstractions.
Abstractions should not depend on details. Details should depend on abstractions.

These two principles ensure high cohesion and low coupling among modules, while also creating the conditions for mocking and iteratively updating modules.

Implementing It in Go

Suppose we need to query user information from a user service. There are two interfaces: UserRepository serves as the data layer responsible for querying the database, and UserService handles business logic and depends on UserRepository. At the same time, to facilitate testing, we also need to write a mock data layer implementation. The entire structure is shown in the figure below.

Go example

Next, very easily, we implement the two interfaces and write their implementation classes. At the same time, we also write a NewUserService in the UserService implementation class to inject the UserRepository implementation it depends on.

// Implement the specific interface in user_repository.go
type UserRepository interface {
 GetByID(id int) (*User, error)
 Save(user *User) error
}

// ... specific implementation of UserRepository, omitted

// Implemented in user_service.go
type UserService interface {
 GetUser(id int) (*User, error)
 CreateUser(name string, age int) error
}

// ... specific implementation of UserService, omitted

func NewUserService(repo UserRepository) UserService {
 return &UserServiceImpl{
 repo: repo,
 }
}

So the question arises: can we directly reference the repository in user_service.go? Obviously not, because this would create a dependency between the two modules.

This is the core of dependency inversion: the upper-layer module does not directly reference the lower-layer module; instead, the executing class initializes the Service and injects the dependent lower-layer service.

// In main.go
func main() {
 repo := &MySQLUserRepository{}
 userService := NewUserService(repo)
}

This way, when writing test mock code, you don’t need to modify any code logic. You can simply replace the parameter of NewUserService in the test with a fake test instance.

// In user_service_test.go
func TestUserService() {
 repo := &MockTestUserRepository{}
 userService := NewUserService(repo)
}

In addition, if the data layer changes its implementation or migrates to another database, you only need to modify two places: the data layer’s implementer and the dependency injector. The caller UserService is completely unaffected. The entire project won’t form a dependency trap.

Summary

The two core principles of the Dependency Inversion Principle:

Modules do not depend on other modules, but both depend on abstract interfaces.
Abstract interfaces do not depend on implementations; implementations depend on abstract interfaces.

Implementing these two principles in Go isn’t difficult—you just need to transform the original caller-implementer relationship into a registrar-caller-implementer relationship. There are also some libraries and frameworks in Go that implement dependency inversion, but the core ideas are not different.

Go Server-Side Events

Wed, 13 Sep 2023 09:19:18 +0800

Server-Sent Events (SSE) is a technology that enables real-time communication between a web browser and a server. It allows the server to send updates or event notifications to the browser without the need for the browser to repeatedly make requests.

SSE is often used in applications that require real-time updates, such as chat applications, social media feeds, stock market tickers, or notification systems. It provides a lightweight and efficient way to deliver server-side updates to clients, improving responsiveness and reducing unnecessary network traffic.

Below is an example of SSE in the Go web framework Gin:

The Gin API handler SSEDemoGetStatus will call service.CheckStatus() every 10 seconds and respond to the client with a message event.

func SSEDemoGetStatus(c *gin.Context) {
	chanStream := make(chan interface{}, StreamBufferSize)
	clientGone := c.Writer.CloseNotify()

	go func() {
		defer close(chanStream)

		ticker := time.NewTicker(10*time.Second)
		defer ticker.Stop()
		for {
			status := service.CheckStatus()
			chanStream <- status

			select {
			case <-ticker.C:
				continue
			case <-clientGone:
				return
			}
		}
	}()

	c.Writer.Header().Set("Connection", "keep-alive")
	c.Writer.Header().Set("X-Accel-Buffering", "no")
	c.Writer.Header().Set("Cache-Control", "no-cache")
	c.Stream(func(w io.Writer) bool {
		if msg, ok := <-chanStream; ok {
			c.SSEvent("message", msg)
			return true
		}
		c.SSEvent("status", "Done")
		return false
	})
}

In this way, the client receives an HTTP request that establishes a TCP connection through which the server continuously sends data.

What must be noted are the HTTP headers. In my case, I need to tell the web server (e.g., Nginx) not to cache the TCP traffic. For the JavaScript EventSource API, the header Connection: keep-alive is important.

A better place to put these headers is a Gin middleware function:

func HeadersMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		c.Writer.Header().Set("Content-Type", "text/event-stream")
		c.Writer.Header().Set("Cache-Control", "no-cache")
		c.Writer.Header().Set("Connection", "keep-alive")
		c.Writer.Header().Set("Transfer-Encoding", "chunked")
		c.Next()
	}
}

For more details, please head over to Gin SSE example.

Common Patterns of Go Memory Leaks

Tue, 27 Jun 2023 14:46:59 +0800

While investigating a Go memory leak at work recently, I came across this blog post by Uber, which shares several common goroutine memory leak patterns. I’ve organized my notes on goroutine issues here, hoping more people searching for this topic will find this article and quickly track down memory leaks.

Causes of Goroutine Memory Leaks

Memory leaks in Go are usually caused by incorrect use of goroutines and channels. For example:

Opening a connection (e.g., gRPC) inside a goroutine but forgetting to close it
Failing to release a global variable’s object inside a goroutine
Reading from a channel inside a goroutine, but no one is writing to it, so it blocks
Writing to an unbuffered channel inside a goroutine, but the read end of the channel is closed by another goroutine, causing a block
Writing to a buffered channel inside a goroutine, but the channel’s buffer is already full

These cases are usually tangled up in complex code logic and are very hard to spot through debugging. As a result, the following most error-prone patterns have emerged in everyday work.

Common Goroutine Memory Leak Patterns

Premature Function Return

A goroutine wants to write to a channel, but the other end exits unexpectedly, so the code that reads from the channel never runs.

func Example() {
 a := 1
 c := make(chan error)
 go func() {
 c <- err
 return
 }()

 // do something

 if a > 0 {
 return
 }

 // do something

 err := <-c
}

In this code, the main process returns at if a > 0, so the write to the channel never happens and the goroutine blocks.

A way to fix this is to convert the unbuffered channel to a buffered channel with capacity 1.

c := make(chan error, 1)

A buffered channel won’t block even if no one reads from it.

The Timeout Leak

This is a problem we ran into at work, often encountered when running an async operation that might time out.

func Example() {
 timeoutOption := SomeTimeoutOption()
 done := make(chan any)
 go func() {
 done <- result
 }()

 select {
 case <- done:
 return
 case <- timeoutOption.Timeout():
 return
 }
}

In this code, once the timeoutOption operation times out, it signals the select, and the program exits, leaving the goroutine blocked on its write to done and unable to exit.

The fix is the same as the previous pattern: replace the unbuffered channel with a buffered one.

The NCast Leak

This happens when a channel has only one reader but multiple writers.

func Example() {
 c := make(chan any)
 for _, i := range items {
 go func(c chan any) {
 c <- result
 }(c)
 }
 data := <- c
 return
}

This also applies to “multiple writers, one reader.” The fix is to size the channel’s buffer to match the number of writers or readers.

c := make(chan any, len(items))

Channel Iteration Misuse

Go supports the “Range over channels” feature, which lets you use range to loop over a channel’s contents.

But once there’s nothing to read, range waits for a write to the channel. If the range happens to be inside a goroutine, that goroutine will be blocked.

func Example() {
 wg := &sync.WaitGroup{}
 c := make(chan any, 1)

 for _, i := items {
 wg.Add(1)
 go func() {
 c <- data
 }()
 }

 go func() {
 for data := range c {
 wg.Done()
 }
 }()

 wg.Wait()
}

The fix is to explicitly close the channel.

wg := &sync.WaitGroup{}
c := make(chan any, 1)
defer close(c)
//...

This way, once the WaitGroup is done, the main program closes the channel, allowing the range inside the async goroutine to exit its wait.

Summary

Goroutine memory leaks are the most common form of memory leak in Go, and they usually come hand-in-hand with incorrect use of goroutines and channels. Special channel constructs like select and range make blocking even more subtle and harder to spot, which raises the difficulty of debugging memory leaks.

When writing goroutines and debugging memory leak issues, pay particular attention to channel operations—especially the four patterns listed in this article: premature function return, the timeout leak, the NCast leak, and channel iteration misuse.