cmd/compile: Go periodically gets stuck for a very long time after a build

[connorszczepaniak-wk]

Go version

go1.25.4

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE='on'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/connorszczepaniak/.local/share/mise/installs/go/1.25.4/bin'
GOCACHE='/Users/connorszczepaniak/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/connorszczepaniak/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/14/dl88pgbs11766nqnslc5ry8h0000gs/T/go-build3441948075=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/connorszczepaniak/go/src/github.com/Workiva/skaardb/go.mod'
GOMODCACHE='/Users/connorszczepaniak/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/connorszczepaniak/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/connorszczepaniak/.local/share/mise/installs/go/1.25.4'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/connorszczepaniak/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/connorszczepaniak/.local/share/mise/installs/go/1.25.4/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.4'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Ran go test -run "TestSomething" ./path/to/package

This also occurs when running go tool sometool, so I suspect it happens any time a build is invoked. Unfortunately I don't have a consistent way to repro, but it happens probably at least once a day.

What did you see happen?

The command hung for a long time (7 minutes 33 seconds) after the test finished. The process appears to have been in a "stuck" state

What did you expect to see?

The command to exit promptly.

[seankhliao]

maybe something macos gatekeeper holding the binary?
either way, without a reliable reproducer, it won't be very actionable

[randall77]

Take a look through #37885, it sounds very similar.

[connorszczepaniak-wk]

Thanks, next time it happens I will try the same (top -o cpu and kill -ABRT <pid>) to see if that gets us any more info.

[connorszczepaniak-wk]

I got this to happen again; Go was bouncing around between stuck and running, and also bouncing in and out of the top 3 on top -o cpu.

One more interesting (albeit qualitative) aspect of this is that when Go gets stuck, I can get it to be consistently stuck after I kill it and try again. In this case, it first happened when running a test; but then I was able to subsequently get Go stuck with go tool gosimports --help (which printed the help of the tool and then hung).

Here's the output of top -o cpu:

PID    COMMAND      %CPU TIME     #TH    #WQ  #PORT MEM    PURG   CMPRS  PGRP  PPID  STATE
0      kernel_task  25.5 01:41:18 764/12 0    0     15M    0B     0B     0     0     running
4485   com.apple.Vi 23.6 09:37:16 30     3    102   21G    0B     28G-   4485  1     sleeping
57997  go           23.3 00:11.85 25     0    41    96M    0B     0B     57997 31588 stuck
406    WindowServer 20.1 68:12.50 27     6    3796- 768M+  27M-   82M    406   1     stuck

Result of `kill -ABRT [pid]`

SIGABRT: abort
PC=0x19e89a904 m=0 sigcode=0

goroutine 0 gp=0x10517cf60 m=0 mp=0x10517e7a0 [idle]:
runtime.sigNoteSleep(0x0?)
        runtime/os_darwin.go:133 +0x20 fp=0x14000cf0f90 sp=0x14000cf0f50 pc=0x10459ff60
os/signal.signal_recv()
        runtime/sigqueue.go:149 +0x2c fp=0x14000cf0fb0 sp=0x14000cf0f90 pc=0x1045dcc6c
os/signal.loop()
        os/signal/signal_unix.go:23 +0x1c fp=0x14000cf0fd0 sp=0x14000cf0fb0 pc=0x10473674c
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000cf0fd0 sp=0x14000cf0fd0 pc=0x1045e2bb4
created by os/signal.Notify.func1.1 in goroutine 16458
        os/signal/signal.go:152 +0x28

goroutine 16397 gp=0x1400109ba40 m=0 mp=0x10517e7a0 [syscall, 1 minutes]:
runtime.sigNoteSleep(0x0?)
        runtime/os_darwin.go:133 +0x20 fp=0x14000cf0f90 sp=0x14000cf0f50 pc=0x10459ff60
os/signal.signal_recv()
        runtime/sigqueue.go:149 +0x2c fp=0x14000cf0fb0 sp=0x14000cf0f90 pc=0x1045dcc6c
os/signal.loop()
        os/signal/signal_unix.go:23 +0x1c fp=0x14000cf0fd0 sp=0x14000cf0fb0 pc=0x10473674c
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000cf0fd0 sp=0x14000cf0fd0 pc=0x1045e2bb4
created by os/signal.Notify.func1.1 in goroutine 16458
        os/signal/signal.go:152 +0x28

goroutine 1 gp=0x140000021c0 m=nil [runnable]:
internal/filepathlite.Clean({0x14003884000, 0x76})
        internal/filepathlite/path.go:65 +0x854 fp=0x14004018ef0 sp=0x14004018ef0 pc=0x1046511b4
path/filepath.Clean(...)
        path/filepath/path.go:56
path/filepath.join({0x14004018fd8?, 0xee0a6e498?, 0x10517c5a0?})
        path/filepath/path_unix.go:36 +0x74 fp=0x14004018f30 sp=0x14004018ef0 pc=0x1046848a4
path/filepath.Join(...)
        path/filepath/path.go:131
cmd/go/internal/cache.(*DiskCache).trimSubdir(0x140040190a8?, {0x14002e60040, 0x33}, {0x1?, 0x1?, 0x10517c5a0?})
        cmd/go/internal/cache/cache.go:436 +0x16c fp=0x14004019000 sp=0x14004018f30 pc=0x10475845c
cmd/go/internal/cache.(*DiskCache).Trim(0x1400000e2d0)
        cmd/go/internal/cache/cache.go:403 +0x1b8 fp=0x14004019110 sp=0x14004019000 pc=0x1047581c8
cmd/go/internal/cache.(*DiskCache).Close(0x140015eb119?)
        cmd/go/internal/cache/cache.go:375 +0x1c fp=0x14004019130 sp=0x14004019110 pc=0x104757fec
cmd/go/internal/work.(*Builder).Do.func1()
        cmd/go/internal/work/exec.go:81 +0x24 fp=0x14004019180 sp=0x14004019130 pc=0x104a063e4
cmd/go/internal/work.(*Builder).Do(0x14000892fc0, {0x104d6a420, 0x1051a5220}, 0x14003aea900)
        cmd/go/internal/work/exec.go:238 +0x378 fp=0x140040192a0 sp=0x14004019180 pc=0x104a05748
cmd/go/internal/test.runTest({0x104d6a420, 0x1051a5220}, 0x10516f060, {0x14000020070?, 0x104c75440?, 0x0?})
        cmd/go/internal/test/test.go:1082 +0x177c fp=0x140040199a0 sp=0x140040192a0 pc=0x104a92d0c
main.invoke(0x10516f060, {0x14000020060, 0x4, 0x4})
        cmd/go/main.go:341 +0x63c fp=0x14004019cb0 sp=0x140040199a0 pc=0x104ab161c
main.main()
        cmd/go/main.go:220 +0xd58 fp=0x14004019f40 sp=0x14004019cb0 pc=0x104ab0948
runtime.main()
        runtime/proc.go:285 +0x278 fp=0x14004019fd0 sp=0x14004019f40 pc=0x1045a5918
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14004019fd0 sp=0x14004019fd0 pc=0x1045e2bb4

goroutine 2 gp=0x14000002700 m=nil [force gc (idle), 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006cf90 sp=0x1400006cf70 pc=0x1045da8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.forcegchelper()
        runtime/proc.go:373 +0xb4 fp=0x1400006cfd0 sp=0x1400006cf90 pc=0x1045a5c64
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006cfd0 sp=0x1400006cfd0 pc=0x1045e2bb4
created by runtime.init.7 in goroutine 1
        runtime/proc.go:361 +0x24

goroutine 3 gp=0x14000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006d760 sp=0x1400006d740 pc=0x1045da8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.bgsweep(0x1400007e000)
        runtime/mgcsweep.go:323 +0x104 fp=0x1400006d7b0 sp=0x1400006d760 pc=0x104590894
runtime.gcenable.gowrap1()
        runtime/mgc.go:212 +0x28 fp=0x1400006d7d0 sp=0x1400006d7b0 pc=0x1045844f8
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006d7d0 sp=0x1400006d7d0 pc=0x1045e2bb4
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:212 +0x6c

goroutine 4 gp=0x14000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0x155a70e?, 0x145d1b2?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006df60 sp=0x1400006df40 pc=0x1045da8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.(*scavengerState).park(0x10517c620)
        runtime/mgcscavenge.go:425 +0x5c fp=0x1400006df90 sp=0x1400006df60 pc=0x10458e3bc
runtime.bgscavenge(0x1400007e000)
        runtime/mgcscavenge.go:658 +0xac fp=0x1400006dfb0 sp=0x1400006df90 pc=0x10458e95c
runtime.gcenable.gowrap2()
        runtime/mgc.go:213 +0x28 fp=0x1400006dfd0 sp=0x1400006dfb0 pc=0x104584498
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006dfd0 sp=0x1400006dfd0 pc=0x1045e2bb4
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:213 +0xac

goroutine 5 gp=0x14000003340 m=nil [GOMAXPROCS updater (idle), 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006e770 sp=0x1400006e750 pc=0x1045da8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.updateMaxProcsGoroutine()
        runtime/proc.go:6720 +0xf4 fp=0x1400006e7d0 sp=0x1400006e770 pc=0x1045b4554
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006e7d0 sp=0x1400006e7d0 pc=0x1045e2bb4
created by runtime.defaultGOMAXPROCSUpdateEnable in goroutine 1
        runtime/proc.go:6708 +0x48

goroutine 6 gp=0x14000003500 m=nil [finalizer wait, 1 minutes]:
runtime.gopark(0x1400006c5b8?, 0x1046e100c?, 0x20?, 0x8e?, 0x33?)
        runtime/proc.go:460 +0xc0 fp=0x1400006c580 sp=0x1400006c560 pc=0x1045da8e0
runtime.runFinalizers()
        runtime/mfinal.go:210 +0x104 fp=0x1400006c7d0 sp=0x1400006c580 pc=0x1045834e4
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006c7d0 sp=0x1400006c7d0 pc=0x1045e2bb4
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:172 +0x78

goroutine 7 gp=0x140000036c0 m=nil [cleanup wait, 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006ef40 sp=0x1400006ef20 pc=0x1045da8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.(*cleanupQueue).dequeue(0x10517cda0)
        runtime/mcleanup.go:439 +0x110 fp=0x1400006ef80 sp=0x1400006ef40 pc=0x1045809d0
runtime.runCleanups()
        runtime/mcleanup.go:635 +0x40 fp=0x1400006efd0 sp=0x1400006ef80 pc=0x1045811e0
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006efd0 sp=0x1400006efd0 pc=0x1045e2bb4
created by runtime.(*cleanupQueue).createGs in goroutine 1
        runtime/mcleanup.go:589 +0x108

goroutine 33 gp=0x14000702380 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d83e520?, 0x1?, 0x3f?, 0x20?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140017b8f10 sp=0x140017b8ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x140017b8fb0 sp=0x140017b8f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140017b8fd0 sp=0x140017b8fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140017b8fd0 sp=0x140017b8fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 34 gp=0x14000702540 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d7e16a2?, 0x3?, 0xf4?, 0x1?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140017b9f10 sp=0x140017b9ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x140017b9fb0 sp=0x140017b9f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140017b9fd0 sp=0x140017b9fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140017b9fd0 sp=0x140017b9fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 35 gp=0x14000702700 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d85767e?, 0x3?, 0x55?, 0x89?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x14001cd6f10 sp=0x14001cd6ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x14001cd6fb0 sp=0x14001cd6f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x14001cd6fd0 sp=0x14001cd6fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14001cd6fd0 sp=0x14001cd6fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 36 gp=0x140007028c0 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d8593ca?, 0x3?, 0xfd?, 0x46?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400078ff10 sp=0x1400078fef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x1400078ffb0 sp=0x1400078ff10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x1400078ffd0 sp=0x1400078ffb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400078ffd0 sp=0x1400078ffd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 37 gp=0x14000702a80 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d86bef1?, 0x3?, 0xbd?, 0xd9?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140006bef10 sp=0x140006beef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x140006befb0 sp=0x140006bef10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140006befd0 sp=0x140006befb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140006befd0 sp=0x140006befd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 38 gp=0x14000702c40 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d837e6c?, 0x3?, 0xf4?, 0x1?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x14001678f10 sp=0x14001678ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x14001678fb0 sp=0x14001678f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x14001678fd0 sp=0x14001678fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14001678fd0 sp=0x14001678fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 39 gp=0x14000702e00 m=nil [GC worker (idle)]:
runtime.gopark(0x1051a6660?, 0x3?, 0xfa?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140016bcf10 sp=0x140016bcef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x140016bcfb0 sp=0x140016bcf10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140016bcfd0 sp=0x140016bcfb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140016bcfd0 sp=0x140016bcfd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 40 gp=0x14000702fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d83d774?, 0x3?, 0xd2?, 0x62?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x14000790f10 sp=0x14000790ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x14000790fb0 sp=0x14000790f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x14000790fd0 sp=0x14000790fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000790fd0 sp=0x14000790fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 41 gp=0x14000703180 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d86e4dd?, 0x3?, 0xca?, 0x1?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140017d9f10 sp=0x140017d9ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x140017d9fb0 sp=0x140017d9f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140017d9fd0 sp=0x140017d9fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140017d9fd0 sp=0x140017d9fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 42 gp=0x14000703340 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d86ddde?, 0x1?, 0x18?, 0xfd?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x14000791f10 sp=0x14000791ef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x14000791fb0 sp=0x14000791f10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x14000791fd0 sp=0x14000791fb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000791fd0 sp=0x14000791fd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 43 gp=0x14000703500 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d85a028?, 0x3?, 0xe3?, 0x6c?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140006baf10 sp=0x140006baef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x140006bafb0 sp=0x140006baf10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140006bafd0 sp=0x140006bafb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140006bafd0 sp=0x140006bafd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 44 gp=0x140007036c0 m=nil [GC worker (idle)]:
runtime.gopark(0x39e27d85ad81?, 0x3?, 0x8d?, 0x9d?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400078ef10 sp=0x1400078eef0 pc=0x1045da8e0
runtime.gcBgMarkWorker(0x1400070e2a0)
        runtime/mgc.go:1463 +0xe0 fp=0x1400078efb0 sp=0x1400078ef10 pc=0x104586b70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x1400078efd0 sp=0x1400078efb0 pc=0x104586a58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400078efd0 sp=0x1400078efd0 pc=0x1045e2bb4
created by runtime.gcBgMarkStartWorkers in goroutine 18
        runtime/mgc.go:1373 +0x140

goroutine 16434 gp=0x140016908c0 m=nil [chan receive, 1 minutes]:
runtime.gopark(0x14000ebef38?, 0x104694010?, 0x30?, 0x82?, 0x14000ebef80?)
        runtime/proc.go:460 +0xc0 fp=0x14000ebef00 sp=0x14000ebeee0 pc=0x1045da8e0
runtime.chanrecv(0x1400325c310, 0x0, 0x1)
        runtime/chan.go:667 +0x428 fp=0x14000ebef80 sp=0x14000ebef00 pc=0x1045744a8
runtime.chanrecv1(0x14001f99d61?, 0x1?)
        runtime/chan.go:509 +0x14 fp=0x14000ebefb0 sp=0x14000ebef80 pc=0x104574044
cmd/go/internal/base.processSignals.func1()
        cmd/go/internal/base/signal.go:21 +0x24 fp=0x14000ebefd0 sp=0x14000ebefb0 pc=0x104739384
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000ebefd0 sp=0x14000ebefd0 pc=0x1045e2bb4
created by cmd/go/internal/base.processSignals in goroutine 16458
        cmd/go/internal/base/signal.go:20 +0x84

goroutine 16396 gp=0x14001c22c40 m=nil [select, 1 minutes, locked to thread]:
runtime.gopark(0x14001448fa0?, 0x2?, 0xa8?, 0x8e?, 0x14001448f90?)
        runtime/proc.go:460 +0xc0 fp=0x14001448e10 sp=0x14001448df0 pc=0x1045da8e0
runtime.selectgo(0x14001448fa0, 0x14001448f8c, 0x0?, 0x0, 0x0?, 0x1)
        runtime/select.go:351 +0x6bc fp=0x14001448f50 sp=0x14001448e10 pc=0x1045b962c
runtime.ensureSigM.func1()
        runtime/signal_unix.go:1085 +0x154 fp=0x14001448fd0 sp=0x14001448f50 pc=0x1045d5484
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14001448fd0 sp=0x14001448fd0 pc=0x1045e2bb4
created by runtime.ensureSigM in goroutine 16458
        runtime/signal_unix.go:1068 +0xcc

r0      0x4
r1      0x14000cf0f83
r2      0x1
r3      0x10517cf60
r4      0x110
r5      0x2350100023500
r6      0x2340000023500
r7      0x0
r8      0x10517e7a0
r9      0x14000cf0f58
r10     0x70000000700
r11     0x70000000702
r12     0x100
r13     0x16b899b40
r14     0x23500
r15     0x23400
r16     0x3
r17     0x19e89a900
r18     0x0
r19     0x16b899868
r20     0x7d
r21     0x3
r22     0x3a
r23     0x0
r24     0x1
r25     0x1400004ed10
r26     0x104d5dc30
r27     0x810
r28     0x10517cf60
r29     0x16b899b28
lr      0x1045e38ec
sp      0x16b899b30
pc      0x19e89a904
fault   0x19e89a904

Result of `kill -ABRT [pid]` for the `go tool` process

SIGABRT: abort
PC=0x19e89d4f8 m=0 sigcode=0

goroutine 0 gp=0x1055b0f60 m=0 mp=0x1055b27a0 [idle]:
runtime.pthread_cond_wait(0x1055b2d08, 0x1055b2cc8)
        runtime/sys_darwin.go:547 +0x20 fp=0x16b465850 sp=0x16b465820 pc=0x1049f8fe0
runtime.semasleep(0xffffffffffffffff)
        runtime/os_darwin.go:72 +0x80 fp=0x16b4658b0 sp=0x16b465850 pc=0x1049d3cf0
runtime.notesleep(0x1055b28f0)
        runtime/lock_sema.go:62 +0xac fp=0x16b4658e0 sp=0x16b4658b0 pc=0x1049ad27c
runtime.mPark(...)
        runtime/proc.go:1974
runtime.stopm()
        runtime/proc.go:3013 +0x84 fp=0x16b465910 sp=0x16b4658e0 pc=0x1049de9f4
runtime.findRunnable()
        runtime/proc.go:3784 +0x1358 fp=0x16b465ae0 sp=0x16b465910 pc=0x1049e0b08
runtime.schedule()
        runtime/proc.go:4152 +0x90 fp=0x16b465b20 sp=0x16b465ae0 pc=0x1049e1e00
runtime.park_m(0x140002eb500)
        runtime/proc.go:4280 +0x2ac fp=0x16b465b80 sp=0x16b465b20 pc=0x1049e22dc
runtime.mcall()
        runtime/asm_arm64.s:241 +0x54 fp=0x16b465b90 sp=0x16b465b80 pc=0x104a147b4

goroutine 1 gp=0x140000021c0 m=nil [runnable]:
runtime.slicebytetostring(0x0?, 0x1400068c915?, 0x42?)
        runtime/string.go:133 +0xf0 fp=0x14000267310 sp=0x14000267310 pc=0x1049f5ab0
os.(*File).readdir(0x1400010e068, 0xffffffffffffffff, 0x0)
        os/dir_darwin.go:89 +0x650 fp=0x14000267470 sp=0x14000267310 pc=0x104a8ef40
os.(*File).Readdirnames(0x14000544080?, 0x33?)
        os/dir.go:73 +0x24 fp=0x140002674a0 sp=0x14000267470 pc=0x104a8e614
cmd/go/internal/cache.(*DiskCache).trimSubdir(0x14000267618?, {0x14000544080, 0x33}, {0x1?, 0x1?, 0x1055b05a0?})
        cmd/go/internal/cache/cache.go:428 +0x5c fp=0x14000267570 sp=0x140002674a0 pc=0x104b8c34c
cmd/go/internal/cache.(*DiskCache).Trim(0x1400000e3c0)
        cmd/go/internal/cache/cache.go:403 +0x1b8 fp=0x14000267680 sp=0x14000267570 pc=0x104b8c1c8
cmd/go/internal/cache.(*DiskCache).Close(0x140004e7619?)
        cmd/go/internal/cache/cache.go:375 +0x1c fp=0x140002676a0 sp=0x14000267680 pc=0x104b8bfec
cmd/go/internal/work.(*Builder).Do.func1()
        cmd/go/internal/work/exec.go:81 +0x24 fp=0x140002676f0 sp=0x140002676a0 pc=0x104e3a3e4
cmd/go/internal/work.(*Builder).Do(0x140000c8240, {0x10519e420, 0x1055d9220}, 0x1400077d200)
        cmd/go/internal/work/exec.go:238 +0x378 fp=0x14000267810 sp=0x140002676f0 pc=0x104e39748
cmd/go/internal/tool.buildAndRunTool({0x10519e420, 0x1055d9220}, {0x140001b3320, 0x2d}, {0x140000200b0, 0x1, 0x1}, 0x1400000e2e8)
        cmd/go/internal/tool/tool.go:355 +0x208 fp=0x14000267890 sp=0x14000267810 pc=0x104ed2ce8
cmd/go/internal/tool.buildAndRunModtool({0x10519e420, 0x1055d9220}, {0x16b4665b0, 0xa}, {0x140001b3320, 0x2d}, {0x140000200b0, 0x1, 0x1})
        cmd/go/internal/tool/tool.go:335 +0xa4 fp=0x140002678e0 sp=0x14000267890 pc=0x104ed2944
cmd/go/internal/tool.runTool({0x10519e420, 0x1055d9220}, 0x1400002a660?, {0x140000200a0, 0x2, 0x2})
        cmd/go/internal/tool/tool.go:118 +0x2f0 fp=0x140002679a0 sp=0x140002678e0 pc=0x104ed1420
main.invoke(0x1055a32a0, {0x14000020090, 0x3, 0x3})
        cmd/go/main.go:341 +0x63c fp=0x14000267cb0 sp=0x140002679a0 pc=0x104ee561c
main.main()
        cmd/go/main.go:220 +0xd58 fp=0x14000267f40 sp=0x14000267cb0 pc=0x104ee4948
runtime.main()
        runtime/proc.go:285 +0x278 fp=0x14000267fd0 sp=0x14000267f40 pc=0x1049d9918
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000267fd0 sp=0x14000267fd0 pc=0x104a16bb4

goroutine 2 gp=0x14000002700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006cf90 sp=0x1400006cf70 pc=0x104a0e8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.forcegchelper()
        runtime/proc.go:373 +0xb4 fp=0x1400006cfd0 sp=0x1400006cf90 pc=0x1049d9c64
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006cfd0 sp=0x1400006cfd0 pc=0x104a16bb4
created by runtime.init.7 in goroutine 1
        runtime/proc.go:361 +0x24

goroutine 3 gp=0x14000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x1055af201?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006d760 sp=0x1400006d740 pc=0x104a0e8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.bgsweep(0x1400007e000)
        runtime/mgcsweep.go:323 +0x104 fp=0x1400006d7b0 sp=0x1400006d760 pc=0x1049c4894
runtime.gcenable.gowrap1()
        runtime/mgc.go:212 +0x28 fp=0x1400006d7d0 sp=0x1400006d7b0 pc=0x1049b84f8
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006d7d0 sp=0x1400006d7d0 pc=0x104a16bb4
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:212 +0x6c

goroutine 4 gp=0x14000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0x1a99e5?, 0x17614b?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006df60 sp=0x1400006df40 pc=0x104a0e8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.(*scavengerState).park(0x1055b0620)
        runtime/mgcscavenge.go:425 +0x5c fp=0x1400006df90 sp=0x1400006df60 pc=0x1049c23bc
runtime.bgscavenge(0x1400007e000)
        runtime/mgcscavenge.go:658 +0xac fp=0x1400006dfb0 sp=0x1400006df90 pc=0x1049c295c
runtime.gcenable.gowrap2()
        runtime/mgc.go:213 +0x28 fp=0x1400006dfd0 sp=0x1400006dfb0 pc=0x1049b8498
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006dfd0 sp=0x1400006dfd0 pc=0x104a16bb4
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:213 +0xac

goroutine 5 gp=0x14000003340 m=nil [GOMAXPROCS updater (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006e770 sp=0x1400006e750 pc=0x104a0e8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.updateMaxProcsGoroutine()
        runtime/proc.go:6720 +0xf4 fp=0x1400006e7d0 sp=0x1400006e770 pc=0x1049e8554
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006e7d0 sp=0x1400006e7d0 pc=0x104a16bb4
created by runtime.defaultGOMAXPROCSUpdateEnable in goroutine 1
        runtime/proc.go:6708 +0x48

goroutine 6 gp=0x14000003500 m=nil [finalizer wait]:
runtime.gopark(0x1400006c5b8?, 0x104b1500c?, 0x20?, 0xce?, 0x33?)
        runtime/proc.go:460 +0xc0 fp=0x1400006c580 sp=0x1400006c560 pc=0x104a0e8e0
runtime.runFinalizers()
        runtime/mfinal.go:210 +0x104 fp=0x1400006c7d0 sp=0x1400006c580 pc=0x1049b74e4
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006c7d0 sp=0x1400006c7d0 pc=0x104a16bb4
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:172 +0x78

goroutine 7 gp=0x140000036c0 m=nil [cleanup wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x1400006ef40 sp=0x1400006ef20 pc=0x104a0e8e0
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.(*cleanupQueue).dequeue(0x1055b0da0)
        runtime/mcleanup.go:439 +0x110 fp=0x1400006ef80 sp=0x1400006ef40 pc=0x1049b49d0
runtime.runCleanups()
        runtime/mcleanup.go:635 +0x40 fp=0x1400006efd0 sp=0x1400006ef80 pc=0x1049b51e0
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006efd0 sp=0x1400006efd0 pc=0x104a16bb4
created by runtime.(*cleanupQueue).createGs in goroutine 1
        runtime/mcleanup.go:589 +0x108

goroutine 20 gp=0x140002eae00 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b713337?, 0x3?, 0xa6?, 0xea?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x14000120f10 sp=0x14000120ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x14000120fb0 sp=0x14000120f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x14000120fd0 sp=0x14000120fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000120fd0 sp=0x14000120fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 21 gp=0x140002eafc0 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b708d87?, 0x3?, 0x2d?, 0x37?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140003b6f10 sp=0x140003b6ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140003b6fb0 sp=0x140003b6f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140003b6fd0 sp=0x140003b6fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140003b6fd0 sp=0x140003b6fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 22 gp=0x140002eb180 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b705466?, 0x3?, 0x35?, 0x7c?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140008e8f10 sp=0x140008e8ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140008e8fb0 sp=0x140008e8f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140008e8fd0 sp=0x140008e8fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140008e8fd0 sp=0x140008e8fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 23 gp=0x140002eb340 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b71a35c?, 0x1?, 0xfd?, 0x30?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140008b8f10 sp=0x140008b8ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140008b8fb0 sp=0x140008b8f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140008b8fd0 sp=0x140008b8fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140008b8fd0 sp=0x140008b8fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 24 gp=0x140002eb500 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b7130f0?, 0x1?, 0x41?, 0xdf?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140008b4f10 sp=0x140008b4ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140008b4fb0 sp=0x140008b4f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140008b4fd0 sp=0x140008b4fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140008b4fd0 sp=0x140008b4fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 25 gp=0x140002eb6c0 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b70630c?, 0x1?, 0x9a?, 0x77?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140003b7f10 sp=0x140003b7ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140003b7fb0 sp=0x140003b7f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140003b7fd0 sp=0x140003b7fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140003b7fd0 sp=0x140003b7fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 26 gp=0x140002eb880 m=nil [GC worker (idle)]:
runtime.gopark(0x1055da660?, 0x1?, 0x22?, 0x94?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140008ebf10 sp=0x140008ebef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140008ebfb0 sp=0x140008ebf10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140008ebfd0 sp=0x140008ebfb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140008ebfd0 sp=0x140008ebfd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 27 gp=0x140002eba40 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b7055b3?, 0x3?, 0xdb?, 0x91?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140003b8f10 sp=0x140003b8ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140003b8fb0 sp=0x140003b8f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140003b8fd0 sp=0x140003b8fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140003b8fd0 sp=0x140003b8fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 28 gp=0x140002ebc00 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b71b613?, 0x3?, 0xab?, 0x16?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x14000174f10 sp=0x14000174ef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x14000174fb0 sp=0x14000174f10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x14000174fd0 sp=0x14000174fb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x14000174fd0 sp=0x14000174fd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 29 gp=0x140002ebdc0 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b70601e?, 0x3?, 0x8a?, 0x58?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140003bdf10 sp=0x140003bdef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140003bdfb0 sp=0x140003bdf10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140003bdfd0 sp=0x140003bdfb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140003bdfd0 sp=0x140003bdfd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 30 gp=0x140002fe000 m=nil [GC worker (idle)]:
runtime.gopark(0x3a27c2bb16b4?, 0x3?, 0xd0?, 0x1?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140003bbf10 sp=0x140003bbef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140003bbfb0 sp=0x140003bbf10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140003bbfd0 sp=0x140003bbfb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140003bbfd0 sp=0x140003bbfd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 31 gp=0x140002fe1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x3a2b0b712dd8?, 0x14000800200?, 0x1c?, 0xa?, 0x0?)
        runtime/proc.go:460 +0xc0 fp=0x140003bcf10 sp=0x140003bcef0 pc=0x104a0e8e0
runtime.gcBgMarkWorker(0x140000924d0)
        runtime/mgc.go:1463 +0xe0 fp=0x140003bcfb0 sp=0x140003bcf10 pc=0x1049bab70
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x28 fp=0x140003bcfd0 sp=0x140003bcfb0 pc=0x1049baa58
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x140003bcfd0 sp=0x140003bcfd0 pc=0x104a16bb4
created by runtime.gcBgMarkStartWorkers in goroutine 8
        runtime/mgc.go:1373 +0x140

goroutine 442 gp=0x140003248c0 m=nil [select, locked to thread]:
runtime.gopark(0x1400006f7a0?, 0x2?, 0xa8?, 0xf6?, 0x1400006f790?)
        runtime/proc.go:460 +0xc0 fp=0x1400006f610 sp=0x1400006f5f0 pc=0x104a0e8e0
runtime.selectgo(0x1400006f7a0, 0x1400006f78c, 0x0?, 0x0, 0x0?, 0x1)
        runtime/select.go:351 +0x6bc fp=0x1400006f750 sp=0x1400006f610 pc=0x1049ed62c
runtime.ensureSigM.func1()
        runtime/signal_unix.go:1085 +0x154 fp=0x1400006f7d0 sp=0x1400006f750 pc=0x104a09484
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006f7d0 sp=0x1400006f7d0 pc=0x104a16bb4
created by runtime.ensureSigM in goroutine 738
        runtime/signal_unix.go:1068 +0xcc

goroutine 441 gp=0x14000383500 m=8 mp=0x14000500008 [syscall]:
runtime.sigNoteSleep(0x0?)
        runtime/os_darwin.go:133 +0x20 fp=0x1400006ff90 sp=0x1400006ff50 pc=0x1049d3f60
os/signal.signal_recv()
        runtime/sigqueue.go:149 +0x2c fp=0x1400006ffb0 sp=0x1400006ff90 pc=0x104a10c6c
os/signal.loop()
        os/signal/signal_unix.go:23 +0x1c fp=0x1400006ffd0 sp=0x1400006ffb0 pc=0x104b6a74c
runtime.goexit({})
        runtime/asm_arm64.s:1268 +0x4 fp=0x1400006ffd0 sp=0x1400006ffd0 pc=0x104a16bb4
created by os/signal.Notify.func1.1 in goroutine 738
        os/signal/signal.go:152 +0x28

r0      0x104
r1      0x0
r2      0xf00
r3      0x0
r4      0x0
r5      0xa0
r6      0x0
r7      0x0
r8      0x16b465738
r9      0x0
r10     0x0
r11     0x2
r12     0x2
r13     0x0
r14     0x0
r15     0x0
r16     0x131
r17     0x20c8bdfc0
r18     0x0
r19     0x1055b2cc8
r20     0x1055b2d08
r21     0x20b2888e0
r22     0x0
r23     0x0
r24     0xf00
r25     0x1201
r26     0x1300
r27     0x105590000
r28     0x1055b0f60
r29     0x16b4657b0
lr      0x19e8dd0dc
sp      0x16b465720
pc      0x19e89d4f8
fault   0x19e89d4f8

Both have a stack sitting on cmd/go/internal/cache.(*DiskCache).trimSubdir, I wonder if something on my system is blocking access to disk somehow.

[randall77]

What does "stuck" mean as a top state?
That sounds like the kernel is preventing execution for some reason.

[adonovan]

I guessed from the title that this was a macOS issue and that the stack would include (*DiskCache).Trim, and I was right on both counts. I see this problem a lot on my mac. I suspect (based on no evidence) that macOS' Spotlight search index, which uses hooks into file system calls, is getting backed up, slowing down the big file tree walk done by the go command's periodic cache trimming.

[ianlancetaylor]

I wonder how much it would help to use getattrlistbulk to fetch a bunch of file names and modification times from the disk in one system call, rather than looping through readdir_r and stat.

[adonovan]

I wonder how much it would help to use getattrlistbulk to fetch a bunch of file names and modification times from the disk in one system call, rather than looping through readdir_r and stat.

Good to know, thanks.

Of course, if my hypothesis is right, it's the modifications to files (caused by deleting stale cache entries) that causes Spotlight to become involved, not the reading of metadata.

[ianlancetaylor]

In that case I wonder if it would make a difference to delete the files in a goroutine, rather than serializing the deletes. Or at least have one goroutine collect file names while another one deletes them.

It's a bit surprising to me that any single run would delete all that many files.

[adonovan]

It's a bit surprising to me that any single run would delete all that many files.

The trim occurs only once per day. After a day's work there's always plenty of garbage in a typical cache. That's what we saw with gopls/internal/filecache.gc, which had similar characteristics.

[connorszczepaniak-wk]

Interesting additional info.

The trim occurs only once per day.

This empirically matches my experience; I usually experience this problem in the morning and then never again (if my memory serves properly).

I don't know the workings of the build cache, but in my workflow I very often switch branches and in my main (very large) repo master changes quite often as well; given the slightly different merge bases of the various branches I'm switching to, I wouldn't doubt if I'm constantly invalidating the cache and creating large amounts of garbage.