Unusual NUMA Access Cost Matrix - Node 01 Local Access Slower Than Remote?
Hi everyone,
I'm seeing some confusing NUMA topology results from coreinfo and hoping someone can help explain what's happening.
**System specs:**
* 32 physical cores (64 logical with hyperthreading) * 2 sockets, 2 NUMA nodes
**The issue:** My NUMA access cost matrix shows:
Approximate Cross-NUMA Node Access Cost (relative to fastest): 00 01 00: 1.0 1.0 01: 1.0 1.4
This doesn't make sense to me:
1. **Node 00→01 access shows 1.0** \- Shouldn't remote memory access be slower than local (>1.0)? 2. **Node 01→01 access shows 1.4** \- This is local memory access within the same NUMA node, so why isn't it 1.0 like Node 00→00?
**Full coreinfo output:**
Logical to Physical Processor Map: **------------------------------ Physical Processor 0 (Hyperthreaded) --**---------------------------- Physical Processor 1 (Hyperthreaded) ----**-------------------------- Physical Processor 2 (Hyperthreaded) ------**------------------------ Physical Processor 3 (Hyperthreaded) --------**---------------------- Physical Processor 4 (Hyperthreaded) ----------**-------------------- Physical Processor 5 (Hyperthreaded) ------------**------------------ Physical Processor 6 (Hyperthreaded) --------------**---------------- Physical Processor 7 (Hyperthreaded) ----------------**-------------- Physical Processor 8 (Hyperthreaded) ------------------**------------ Physical Processor 9 (Hyperthreaded) --------------------**---------- Physical Processor 10 (Hyperthreaded) ----------------------**-------- Physical Processor 11 (Hyperthreaded) ------------------------**------ Physical Processor 12 (Hyperthreaded) --------------------------**---- Physical Processor 13 (Hyperthreaded) ----------------------------**-- Physical Processor 14 (Hyperthreaded) ------------------------------** Physical Processor 15 (Hyperthreaded) **------------------------------ Physical Processor 16 (Hyperthreaded) --**---------------------------- Physical Processor 17 (Hyperthreaded) ----**-------------------------- Physical Processor 18 (Hyperthreaded) ------**------------------------ Physical Processor 19 (Hyperthreaded) --------**---------------------- Physical Processor 20 (Hyperthreaded) ----------**-------------------- Physical Processor 21 (Hyperthreaded) ------------**------------------ Physical Processor 22 (Hyperthreaded) --------------**---------------- Physical Processor 23 (Hyperthreaded) ----------------**-------------- Physical Processor 24 (Hyperthreaded) ------------------**------------ Physical Processor 25 (Hyperthreaded) --------------------**---------- Physical Processor 26 (Hyperthreaded) ----------------------**-------- Physical Processor 27 (Hyperthreaded) ------------------------**------ Physical Processor 28 (Hyperthreaded) --------------------------**---- Physical Processor 29 (Hyperthreaded) ----------------------------**-- Physical Processor 30 (Hyperthreaded) ------------------------------** Physical Processor 31 (Hyperthreaded)
Logical Processor to Socket Map: ******************************** Socket 0 ******************************** Socket 1
Logical Processor to NUMA Node Map: ******************************** NUMA Node 0 ******************************** NUMA Node 1
Approximate Cross-NUMA Node Access Cost (relative to fastest): 00 01 00: 1.0 1.5 01: 1.0 1.4
Logical Processor to Cache Map: **------------------------------ Data Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64 **------------------------------ Instruction Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64 **------------------------------ Unified Cache 0, Level 2, 1 MB, Assoc 16, LineSize 64 ******************************** Unified Cache 1, Level 3, 33 MB, Assoc 11, LineSize 64 --**---------------------------- Data Cache 1, Level 1, 32 KB, Assoc 8, LineSize 64 --**---------------------------- Instruction Cache 1, Level 1, 32 KB, Assoc 8, LineSize 64 --**---------------------------- Unified Cache 2, Level 2, 1 MB, Assoc 16, LineSize 64 ----**-------------------------- Data Cache 2, Level 1, 32 KB, Assoc 8, LineSize 64 ----**-------------------------- Instruction Cache 2, Level 1, 32 KB, Assoc 8, LineSize 64 ----**-------------------------- Unified Cache 3, Level 2, 1 MB, Assoc 16, LineSize 64 ------**------------------------ Data Cache 3, Level 1, 32 KB, Assoc 8, LineSize 64 ------**------------------------ Instruction Cache 3, Level 1, 32 KB, Assoc 8, LineSize 64 ------**------------------------ Unified Cache 4, Level 2, 1 MB, Assoc 16, LineSize 64 --------**---------------------- Data Cache 4, Level 1, 32 KB, Assoc 8, LineSize 64 --------**---------------------- Instruction Cache 4, Level 1, 32 KB, Assoc 8, LineSize 64 --------**---------------------- Unified Cache 5, Level 2, 1 MB, Assoc 16, LineSize 64 ----------**-------------------- Data Cache 5, Level 1, 32 KB, Assoc 8, LineSize 64 ----------**-------------------- Instruction Cache 5, Level 1, 32 KB, Assoc 8, LineSize 64 ----------**-------------------- Unified Cache 6, Level 2, 1 MB, Assoc 16, LineSize 64 ------------**------------------ Data Cache 6, Level 1, 32 KB, Assoc 8, LineSize 64 ------------**------------------ Instruction Cache 6, Level 1, 32 KB, Assoc 8, LineSize 64 ------------**------------------ Unified Cache 7, Level 2, 1 MB, Assoc 16, LineSize 64 --------------**---------------- Data Cache 7, Level 1, 32 KB, Assoc 8, LineSize 64 --------------**---------------- Instruction Cache 7, Level 1, 32 KB, Assoc 8, LineSize 64 --------------**---------------- Unified Cache 8, Level 2, 1 MB, Assoc 16, LineSize 64 ----------------**-------------- Data Cache 8, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------**-------------- Instruction Cache 8, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------**-------------- Unified Cache 9, Level 2, 1 MB, Assoc 16, LineSize 64 ------------------**------------ Data Cache 9, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------**------------ Instruction Cache 9, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------**------------ Unified Cache 10, Level 2, 1 MB, Assoc 16, LineSize 64 --------------------**---------- Data Cache 10, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------**---------- Instruction Cache 10, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------**---------- Unified Cache 11, Level 2, 1 MB, Assoc 16, LineSize 64 ----------------------**-------- Data Cache 11, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------**-------- Instruction Cache 11, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------**-------- Unified Cache 12, Level 2, 1 MB, Assoc 16, LineSize 64 ------------------------**------ Data Cache 12, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------**------ Instruction Cache 12, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------**------ Unified Cache 13, Level 2, 1 MB, Assoc 16, LineSize 64 --------------------------**---- Data Cache 13, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------------**---- Instruction Cache 13, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------------**---- Unified Cache 14, Level 2, 1 MB, Assoc 16, LineSize 64 ----------------------------**-- Data Cache 14, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------------**-- Instruction Cache 14, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------------**-- Unified Cache 15, Level 2, 1 MB, Assoc 16, LineSize 64 ------------------------------** Data Cache 15, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------------** Instruction Cache 15, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------------** Unified Cache 16, Level 2, 1 MB, Assoc 16, LineSize 64 **------------------------------ Data Cache 16, Level 1, 32 KB, Assoc 8, LineSize 64 **------------------------------ Instruction Cache 16, Level 1, 32 KB, Assoc 8, LineSize 64 **------------------------------ Unified Cache 17, Level 2, 1 MB, Assoc 16, LineSize 64 ******************************** Unified Cache 18, Level 3, 33 MB, Assoc 11, LineSize 64 --**---------------------------- Data Cache 17, Level 1, 32 KB, Assoc 8, LineSize 64 --**---------------------------- Instruction Cache 17, Level 1, 32 KB, Assoc 8, LineSize 64 --**---------------------------- Unified Cache 19, Level 2, 1 MB, Assoc 16, LineSize 64 ----**-------------------------- Data Cache 18, Level 1, 32 KB, Assoc 8, LineSize 64 ----**-------------------------- Instruction Cache 18, Level 1, 32 KB, Assoc 8, LineSize 64 ----**-------------------------- Unified Cache 20, Level 2, 1 MB, Assoc 16, LineSize 64 ------**------------------------ Data Cache 19, Level 1, 32 KB, Assoc 8, LineSize 64 ------**------------------------ Instruction Cache 19, Level 1, 32 KB, Assoc 8, LineSize 64 ------**------------------------ Unified Cache 21, Level 2, 1 MB, Assoc 16, LineSize 64 --------**---------------------- Data Cache 20, Level 1, 32 KB, Assoc 8, LineSize 64 --------**---------------------- Instruction Cache 20, Level 1, 32 KB, Assoc 8, LineSize 64 --------**---------------------- Unified Cache 22, Level 2, 1 MB, Assoc 16, LineSize 64 ----------**-------------------- Data Cache 21, Level 1, 32 KB, Assoc 8, LineSize 64 ----------**-------------------- Instruction Cache 21, Level 1, 32 KB, Assoc 8, LineSize 64 ----------**-------------------- Unified Cache 23, Level 2, 1 MB, Assoc 16, LineSize 64 ------------**------------------ Data Cache 22, Level 1, 32 KB, Assoc 8, LineSize 64 ------------**------------------ Instruction Cache 22, Level 1, 32 KB, Assoc 8, LineSize 64 ------------**------------------ Unified Cache 24, Level 2, 1 MB, Assoc 16, LineSize 64 --------------**---------------- Data Cache 23, Level 1, 32 KB, Assoc 8, LineSize 64 --------------**---------------- Instruction Cache 23, Level 1, 32 KB, Assoc 8, LineSize 64 --------------**---------------- Unified Cache 25, Level 2, 1 MB, Assoc 16, LineSize 64 ----------------**-------------- Data Cache 24, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------**-------------- Instruction Cache 24, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------**-------------- Unified Cache 26, Level 2, 1 MB, Assoc 16, LineSize 64 ------------------**------------ Data Cache 25, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------**------------ Instruction Cache 25, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------**------------ Unified Cache 27, Level 2, 1 MB, Assoc 16, LineSize 64 --------------------**---------- Data Cache 26, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------**---------- Instruction Cache 26, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------**---------- Unified Cache 28, Level 2, 1 MB, Assoc 16, LineSize 64 ----------------------**-------- Data Cache 27, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------**-------- Instruction Cache 27, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------**-------- Unified Cache 29, Level 2, 1 MB, Assoc 16, LineSize 64 ------------------------**------ Data Cache 28, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------**------ Instruction Cache 28, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------**------ Unified Cache 30, Level 2, 1 MB, Assoc 16, LineSize 64 --------------------------**---- Data Cache 29, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------------**---- Instruction Cache 29, Level 1, 32 KB, Assoc 8, LineSize 64 --------------------------**---- Unified Cache 31, Level 2, 1 MB, Assoc 16, LineSize 64 ----------------------------**-- Data Cache 30, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------------**-- Instruction Cache 30, Level 1, 32 KB, Assoc 8, LineSize 64 ----------------------------**-- Unified Cache 32, Level 2, 1 MB, Assoc 16, LineSize 64 ------------------------------** Data Cache 31, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------------** Instruction Cache 31, Level 1, 32 KB, Assoc 8, LineSize 64 ------------------------------** Unified Cache 33, Level 2, 1 MB, Assoc 16, LineSize 64
Logical Processor to Group Map: ******************************** Group 0
Thanks in advance for any help!