nil pointer dereference when deploying CSI node plugin

[olljanat]

Nomad version

Nomad v1.11.0
BuildDate 2025-11-11T16:18:19Z
Revision 9103d93

Operating system and Environment details

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.3 LTS"

Issue

We where testing CSI node plugin deployment to our Nomad environment and managed to get it in state that Nomad UI was unstable and there was this error in log:

Dec 15 11:59:40 server1 nomad[1616110]:     2025-12-15T11:59:40.025Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=91d4ced8-81aa-1aa1-82f5-46f02e8fa290 task=z-platform_csi-ntnx-controller type="Plugin became healthy" msg="plugin: nutanix" failed=false
Dec 15 11:59:42 server1 nomad[1616110]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 15 11:59:42 server1 nomad[1616110]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19717d1]
Dec 15 11:59:42 server1 nomad[1616110]: goroutine 358 [running]:
Dec 15 11:59:42 server1 nomad[1616110]: github.com/hashicorp/nomad/nomad/structs.(*DeploymentState).Copy(...)
Dec 15 11:59:42 server1 nomad[1616110]:         github.com/hashicorp/nomad/nomad/structs/structs.go:10831
Dec 15 11:59:42 server1 nomad[1616110]: github.com/hashicorp/nomad/nomad/structs.(*Deployment).Copy(0xc00880a000)
Dec 15 11:59:42 server1 nomad[1616110]:         github.com/hashicorp/nomad/nomad/structs/structs.go:10677 +0x1b1
Dec 15 11:59:42 server1 nomad[1616110]: github.com/hashicorp/nomad/nomad.evaluatePlanPlacements(0xc008513260, 0xc002cfd590, 0xc0057023f0, {0x434ee28, 0xc000bf0de0})
Dec 15 11:59:42 server1 nomad[1616110]:         github.com/hashicorp/nomad/nomad/plan_apply.go:517 +0x70
Dec 15 11:59:42 server1 nomad[1616110]: github.com/hashicorp/nomad/nomad.evaluatePlan(0xc008513260, 0xc002cfd590, 0xc0057023f0, {0x434ee28, 0xc000bf0de0})
Dec 15 11:59:42 server1 nomad[1616110]:         github.com/hashicorp/nomad/nomad/plan_apply.go:506 +0x274
Dec 15 11:59:42 server1 nomad[1616110]: github.com/hashicorp/nomad/nomad.(*planner).planApply(0xc0008b1040)
Dec 15 11:59:42 server1 nomad[1616110]:         github.com/hashicorp/nomad/nomad/plan_apply.go:165 +0x3bf
Dec 15 11:59:42 server1 nomad[1616110]: created by github.com/hashicorp/nomad/nomad.(*Server).establishLeadership in goroutine 273
Dec 15 11:59:42 server1 nomad[1616110]:         github.com/hashicorp/nomad/nomad/leader.go:407 +0x17f

Reproduction steps

Unfortunately we don't have full reproduction steps. We where testing this CSI plugin but even that started to working after removing job with API and creating it again so this looks like some kind of race condition.

[tgross]

Hi @olljanat! Sorry to hear about this. I've fixed the immediate problem in #27262, but I'd like to try to root cause why we'd have a nil deployment state here. Was there any chance this job existed prior to your 1.11.0 upgrade and this happened during the first update?

[olljanat]

@tgross this cluster has 3 servers without Consul which was created directly with 1.11.0 and two other servers was joined to it with nomad server join ... command.

After my first message, the cluster has crashed multiple times and it got stuck to reboot loop until CSI node plugin is removed but we where not able to track that which exact change causes that.

In parallel with CSI plugin test I was doing stress testing by deploying job which contains 10 groups and each of those was starting 100 Docker containers and I have been re-deploying clients multiple times. Some point it looked that shutdown old client would also cause of this but I wasn't able to reliably reproduce that either.

Also there where multiple minutes without any log messages before first crash so seems that what ever causes this does not get logged.