965 瀏覽數

NUMA (Non-uniform memory access)

多核心的處理器最早是透過對稱多處理 SMP (Symmetric multiprocessing) 的方式,所有的 CPU 核心對於記憶體的存取是共用的,但是當 CPU 核心數太多時反而是一個限制,當不同的處理器需要交換資料時都是透過系統匯流排將資料儲存在記憶體中,但當核心數多時,交換資料變成常態, CPU 與記憶體之間的速度跟不上 CPU 處理的速度.越多的核心反而讓整體效能降低.

因此有了 Intel 的 NUMA (Non-uniform memory access),他把 CPU 與記憶體區分成不同的結點 Node (不同的 CPU 各自擁有記憶體),彼此的 CPU 節點再透過 QPI (Intel QuickPath Interconnect) 這個介面做溝通.

關於 CPU 的演進可以參考鳥哥網站 http://linux.vbird.org/linux_enterprise/cputune.php

測試環境為 Ubuntu 16.04 64bits

來看一下我系統的下的 NUMA 狀態,可以使用 numactl , numastat 這兩個指令,為非預設安裝,需要透過 apt 來安裝.

root@ubuntu:~# apt-get install numactl
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  ubuntu-core-launcher
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  numactl
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 30.2 kB of archives.
After this operation, 117 kB of additional disk space will be used.
Get:1 http://tw.archive.ubuntu.com/ubuntu xenial/universe amd64 numactl amd64 2.0.11-1ubuntu1 [30.2 kB]
Fetched 30.2 kB in 0s (40.9 kB/s)
Selecting previously unselected package numactl.
(Reading database ... 205238 files and directories currently installed.)
Preparing to unpack .../numactl_2.0.11-1ubuntu1_amd64.deb ...
Unpacking numactl (2.0.11-1ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up numactl (2.0.11-1ubuntu1) ...

透過 numastat 可以看到我的系統有兩個 Node (Node0 與 Node1)

root@ubuntu:~# numastat 
                           node0           node1
numa_hit                 6018617         4385882
numa_miss                      0               0
numa_foreign                   0               0
interleave_hit              4800           15116
local_node               6003635         4380820
other_node                 14982            5062

上面的數值所代表

  • numa_hit
    Memory successfully allocated on this node as intended.
    記憶體成功配置至此節點
  • numa_miss
    Memory allocated on this node despite the process preferring some different node. Each numa_miss has a numa_foreign on another node.
    原先預定的節點的記憶體不足,而配置至此節點. numa_miss 與另一個節點的 numa_foreign 是相對應的.
  • numa_foreign
    Memory intended for this node, but actually allocated on some different node. Each numa_foreign has a numa_miss on another node.
    原先預定至此節點的記憶體但被配置至其他節點上. numa_foreign 與另一個節點的 numa_miss 是相對應的.
  • interleave_hit
    Interleaved memory successfully allocated on this node as intended.
    The number of interleave policy allocations that were intended for a specific node and succeeded there.
  • local_node
    Memory allocated on this node while a process was running on it.
    該節點上的程序成功配置到該節點的記憶體空間.
  • other_node
    Memory allocated on this node while a process was running on some other node.
    該節點上的程序,成功配置到另一個節點的記憶體空間.

numactl 指令參考
–hardware , -H : Show inventory of available node on the system.

root@ubuntu:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 8 9 10 11
node 0 size: 3939 MB
node 0 free: 3034 MB
node 1 cpus: 4 5 6 7 12 13 14 15
node 1 size: 4029 MB
node 1 free: 3468 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

–show, -s : Show NUMA policy setting of the current process.

root@ubuntu:~# numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 
cpubind: 0 1 
nodebind: 0 1 
membind: 0 1 

command {arguments …}
透過 numacl 可以限制後面接的指令要使用哪一顆處理器或是記憶體.

# numactl  [ --all ] [ --interleave nodes ] [ --preferred node ] [ --mem‐bind nodes  ]  [  --cpunodebind  nodes  ]  [  --physcpubind  cpus  ]  [--localalloc ] [--] command {arguments ...}

利用 lmbench 的工具 lat_mem_rd 來進行測試時,限制 MEM (–membind=0), CPU (–cpunodebind=0) ,關於 lmbench 請參考 http://benjr.tw/98076

root@ubuntu:~# numactl --membind=0 --cpunodebind=0 ./lat_mem_rd 2000 128

One Reply to “NUMA (Non-uniform memory access)”

  1. 自動參照通知: Linux – Stressful Application Test (NUMA) – Benjr.tw

發表迴響