ItemIterator.Next() high memory usage

[EmasXP]

I've created a rather large database of 5.9 GiB. And then I run this code:

func printMemoryInfo() {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("Alloc = %v MiB, Sys = %v MiB\n", m.Alloc/1024/1024, m.Sys/1024/1024)
}

func main() {
	db, err := pogreb.Open("database", nil)
	if err != nil {
		panic(err)
	}
	defer db.Close()

	printMemoryInfo() // => Alloc = 0 MiB, Sys = 6 MiB

	it := db.Items()

	printMemoryInfo() // => Alloc = 0 MiB, Sys = 6 MiB

	for {
		_, _, err := it.Next()
		if err == pogreb.ErrIterationDone {
			break
		}
	}

	printMemoryInfo() // Alloc = 2 MiB, Sys = 11 MiB
}

So far so good, but htop reports that 5789 MB residential memory is in use. In case something is freed after the iteration has finished, I modified the code a bit:

	count := 0

	for {
		_, _, err := it.Next()
		if err == pogreb.ErrIterationDone {
			break
		}

		count ++
		if count == 5000 {
			printMemoryInfo()
			count = 0
		}
	}

But the Sys memory never exceeds 12 MiB.

As a test I limited the iteration to 1000 entries:

	for i := 0; i < 1000; i++  {
		_, _, err := it.Next()
		if err == pogreb.ErrIterationDone {
			break
		}
	}

When doing that, htop reports a residential memory usage of 23 MB. That is what makes me believe that it's Next() that is causing the high memory usage.

I have dug into the code, trying to find the cause, but I am not able to find anything myself. I realize that this can be caused by lack of knowledge from my side, too.

Any ideas what this could be?

Details:

• OS: openSUSE Tumbleweed 20240918
• Kernel: Linux 6.10.9-1-default
• Architecture: x86-64
• Go version: go1.23.0 linux/amd64