Manage Goroutine Work Queues Efficiently in Go

Goroutines are one of Go’s defining features. They make concurrent work inexpensive to start, but creating an uncontrolled number of goroutines can still overwhelm memory, downstream services, file descriptors, or database connections.

A common solution is a bounded worker pool: tasks are placed on a channel and a fixed number of goroutines consume them.

Why Use a Queue?

A controlled work queue helps you:

• limit the number of active workers;
• apply backpressure when work arrives faster than it can be processed;
• centralize task scheduling;
• shut down cleanly after queued work finishes.

1. A Basic Channel-Based Queue

type Task struct {
    ID int
}

func worker(id int, tasks <-chan Task) {
    for task := range tasks {
        fmt.Printf("Worker %d processing task %d\n", id, task.ID)
        time.Sleep(time.Second)
    }
}

A channel can hold pending tasks while several workers read from it.

2. Worker Pool with sync.WaitGroup

Use a WaitGroup instead of sleeping for an arbitrary amount of time:

package main

import (
    "fmt"
    "sync"
    "time"
)

const numWorkers = 3

type Task struct {
    ID int
}

func worker(id int, tasks <-chan Task, wg *sync.WaitGroup) {
    defer wg.Done()

    for task := range tasks {
        fmt.Printf("Worker %d processing task %d\n", id, task.ID)
        time.Sleep(time.Second)
    }
}

func main() {
    tasks := make(chan Task, 10)
    var wg sync.WaitGroup

    for i := 1; i <= numWorkers; i++ {
        wg.Add(1)
        go worker(i, tasks, &wg)
    }

    for i := 1; i <= 10; i++ {
        tasks <- Task{ID: i}
    }

    close(tasks)
    wg.Wait()
}

The important sequence is:

1. Start a bounded number of workers.
2. Send tasks to the channel.
3. Close the channel when no more tasks will be produced.
4. Wait for every worker to finish.

Buffered vs. Unbuffered Channels

A buffered channel such as:

tasks := make(chan Task, 10)

allows a producer to enqueue a limited amount of work before it blocks. The capacity is part of your backpressure strategy; making it arbitrarily large can simply move the overload problem into memory.

Add Cancellation for Real Services

Long-running servers usually need cancellation. Pass a context.Context to workers and stop when it is canceled:

func worker(ctx context.Context, tasks <-chan Task) {
    for {
        select {
        case <-ctx.Done():
            return
        case task, ok := <-tasks:
            if !ok {
                return
            }
            process(task)
        }
    }
}

This makes worker pools easier to integrate with graceful shutdown.

More Advanced Patterns

• Rate limiting controls how quickly tasks are started.
• Priority queues can schedule urgent work before ordinary work.
• Retries should normally include limits and backoff so persistent failures do not create an infinite loop.
• Metrics such as queue depth, task duration, errors, and active workers make production behavior observable.

Conclusion

Goroutines are lightweight, but concurrency should still be bounded. A channel plus a fixed worker pool is a simple, idiomatic pattern for controlling resource usage while processing many independent jobs. Add context cancellation, rate limits, retries, and metrics as the workload becomes more demanding.