Tag Archives: golang internals

Golang’s sync.Map internals

Great article on the topic — https://victoriametrics.com/blog/go-sync-map/index.html. The whole series of articles is solid, so please take a look if you are interested. Let me share the most important insights. How it works Internal structure is:

type Map struct {
	mu Mutex
	read atomic.Pointer[readOnly]
	dirty map[any]*entry
	misses int
}

type Map struct {

mu Mutex

read atomic.Pointer[readOnly]

dirty map[any]*entry

misses int

}

Maps internals in Go

Idea Map is passed as a value, but it consists of a pointer to hmap which has all the details on map implementation. So, if you change/add to map, it will be reflected everywhere. And that’s why you cannot assign to an uninitialized map (no memory allocated, no hash seed generated yet). runtime/map.go

type hmap struct {
	count     int // # live cells == size of map.  Must be first (used by len() builtin)
	B         uint8  // log_2 of # of buckets (can hold up to loadFactor * 2^B items)
	hash0     uint32 // hash seed

	buckets    unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.
	oldbuckets unsafe.Pointer // previous bucket array of half the size, non-nil only when growing
	nevacuate  uintptr        // progress counter for evacuation (buckets less than this have been evacuated)
}

type hmap struct {

count int // # live cells == size of map. Must be first (used by len() builtin)

B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items)

hash0 uint32 // hash seed

buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.

oldbuckets unsafe.Pointer // previous bucket array of half the size, non-nil only when growing

nevacuate uintptr // progress counter for evacuation (buckets less than this have been evacuated)

}

Buckets …

Go’s Concurrency and Channel Internals

Go is implementing CSP (Communicating Sequential Processing): processes are communicating through channels, they can block each other while waiting for read/writes to channels. Actor model makes inter-process communications more explicit and non-blocking. CSP vs Actor explained — https://dev.to/karanpratapsingh/csp-vs-actor-model-for-concurrency-1cpg. Channels requirements goroutine-safe store and pass data across goroutines FIFO can block/unblock goroutines

Go scheduler details

Scheduler was implemented by Dmitry Vyukov in go 1.1 and lives in runtime/proc.go. Good resources https://www.youtube.com/watch?v=-K11rY57K7k — video by Dmitry Vyukov (slides as pdf) https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part2.html — nice article by Bill Kennedy in 3 parts. Most of the images below are taken from this article. Main ideas Goroutines are very light weight (~2KB+) and very cheap …

Go memory hints

Below hints are related to very hot places of CPU/memory bound programs. And in regular (i/o bound) programs do not make much sense. Stack vs heap Go’s compiler goal is to allocate on the stack as much as possible. As it will reduce the amount of data that needs to be cleaned by the Garbage …

Go internals

Channels https://youtu.be/KBZlN0izeiY Go scheduler https://youtu.be/YHRO5WQGh0k

How golang garbage collection works

It is a Mark-and-Sweep GC. Phases Mark Stop-the-World: Set write barrier (to know how much was allocated during maark phase) Concurrent: Mark all memory which is still in use by the app Stop-the-World: Remove write barrier Concurrent: Sweep (it actually happens on new allocations) Details/Algorithm It’s a tri-color process: grey for objects to check, black …