Performance notice
Device under test
All data was collected using the next device under test:
- Dell PowerEdge R630
- Intel(R) Xeon(R) CPU E5-2680 v4, two CPUs installed, used cores only local NUMA node (14 cores total), HT disabled
- 64 GiB RAM, DDR4-2133, 16 GiB x 4
- NVIDIA/Mellanox ConnectX-6 Dx, 100GbE, Dual-port QSFP56
- VyOS version:
1.5-rolling-202402070021
💡 Pay attention
PCIe 3.0 limits available bandwidth. That is why we cannot get the full 100 Gbps in any test.
VFIO
Enabling VFIO affects non-VPP interfaces performance, reducing it by approximately 30%.
Summary
| Test case | 64 byte Mpps | 64 byte Gbps | IMIX Mpps | IMIX Gbps | 1400 byte Mpps | 1400 byte Gbps |
|---|---|---|---|---|---|---|
| Without VPP flowtables disabled | 10.01 | 4.95 | 9.01 | 25.63 | 6 | 64.88 |
| Without VPP flowtables enabled | 19.02 | 9.01 | 16.01 | 43.16 | 6 | 65.77 |
| With VPP | 72.05 | 33.87 | 23.01 | 60.65 | 7.01 | 76.69 |
Tests details
Without VPP, flowtables disabled
Kernel options: nosmt mitigations=off
CPU cores disabled:
for i in 1 3 5 7 9 11 13 15 17 19 21 23 25 27 ; do echo 0 > /sys/devices/system/cpu/cpu${i}/online ; done
Tuned queues count to fit the available amount of CPU cores:
for i in 6 7 ; do ethtool -L eth${i} combined 14 ; done
Configuration:
set interfaces ethernet eth6 address '192.168.2.1/24'
set interfaces ethernet eth6 ring-buffer rx '8192'
set interfaces ethernet eth6 ring-buffer tx '8192'
set interfaces ethernet eth7 address '192.168.3.1/24'
set interfaces ethernet eth7 ring-buffer rx '8192'
set interfaces ethernet eth7 ring-buffer tx '8192'
set protocols static arp interface eth6 address 192.168.2.2 mac '<TREX_MAC_IF1>'
set protocols static arp interface eth7 address 192.168.3.2 mac '<TREX_MAC_IF2>'
set protocols static route 16.0.0.0/8 next-hop 192.168.2.2
set protocols static route 48.0.0.0/8 next-hop 192.168.3.2
set system conntrack # empty section
set system sysctl parameter net.core.rmem_default value '134217728'
set system sysctl parameter net.core.rmem_max value '536870912'
set system sysctl parameter net.core.wmem_default value '134217728'
set system sysctl parameter net.core.wmem_max value '536870912'
Results
64-byte packets:
trex>start -f stl/bench.py -m 5mpps -t size=64,vm=random
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 1 streams to port(s) [0._]: [SUCCESS]
Attaching 1 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
38.08 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 64 | Continuous | 1 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 64 | Continuous | 1 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 5.13 Gbps
version : STL @ v3.03 total_tx_L1 : 6.73 Gbps
cpu_util. : 39.05% @ 4 cores (4 per dual port) total_rx : 4.95 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 10.01 Mpps
async_util. : 0% / 22.48 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 39.05% | 39.05% |
-- | | |
Tx bps L2 | 2.57 Gbps | 2.56 Gbps | 5.13 Gbps
Tx bps L1 | 3.37 Gbps | 3.36 Gbps | 6.73 Gbps
Tx pps | 5.01 Mpps | 5 Mpps | 10.01 Mpps
Line Util. | 3.37 % | 3.36 % |
--- | | |
Rx bps | 2.47 Gbps | 2.47 Gbps | 4.95 Gbps
Rx pps | 4.55 Mpps | 4.54 Mpps | 9.09 Mpps
---- | | |
opackets | 1602866107 | 1602923661 | 3205789768
ipackets | 1455468098 | 1455520499 | 2910988597
obytes | 102583430848 | 102587114304 | 205170545152
ibytes | 98971830664 | 98975393932 | 197947224596
tx-pkts | 1.6 Gpkts | 1.6 Gpkts | 3.21 Gpkts
rx-pkts | 1.46 Gpkts | 1.46 Gpkts | 2.91 Gpkts
tx-bytes | 102.58 GB | 102.59 GB | 205.17 GB
rx-bytes | 98.97 GB | 98.98 GB | 197.95 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
IMIX packets:
trex>start -f stl/bench.py -m 4.5mpps -t size=imix,vm=random
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 3 streams to port(s) [0._]: [SUCCESS]
Attaching 3 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
65.32 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 68 | Continuous | 28 pps | - | -
2 | - | _ | Ethernet:IP:UDP:Raw | 590 | Continuous | 16 pps | - | -
3 | - | _ | Ethernet:IP:UDP:Raw | 1514 | Continuous | 4 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 68 | Continuous | 28 pps | - | -
2 | - | _ | Ethernet:IP:UDP:Raw | 590 | Continuous | 16 pps | - | -
3 | - | _ | Ethernet:IP:UDP:Raw | 1514 | Continuous | 4 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 26.12 Gbps
version : STL @ v3.03 total_tx_L1 : 27.56 Gbps
cpu_util. : 39.75% @ 4 cores (4 per dual port) total_rx : 25.63 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 9.01 Mpps
async_util. : 0% / 19.66 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 39.75% | 39.75% |
-- | | |
Tx bps L2 | 13.03 Gbps | 13.08 Gbps | 26.12 Gbps
Tx bps L1 | 13.75 Gbps | 13.81 Gbps | 27.56 Gbps
Tx pps | 4.49 Mpps | 4.51 Mpps | 9.01 Mpps
Line Util. | 13.75 % | 13.81 % |
--- | | |
Rx bps | 12.79 Gbps | 12.84 Gbps | 25.63 Gbps
Rx pps | 4.36 Mpps | 4.38 Mpps | 8.74 Mpps
---- | | |
opackets | 84763466 | 84795286 | 169558752
ipackets | 82217710 | 82285561 | 164503271
obytes | 30726758326 | 30738293210 | 61465051536
ibytes | 30132697170 | 30158724804 | 60291421974
tx-pkts | 84.76 Mpkts | 84.8 Mpkts | 169.56 Mpkts
rx-pkts | 82.22 Mpkts | 82.29 Mpkts | 164.5 Mpkts
tx-bytes | 30.73 GB | 30.74 GB | 61.47 GB
rx-bytes | 30.13 GB | 30.16 GB | 60.29 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
1400-byte packets:
trex>start -f stl/bench.py -m 3mpps -t size=1500,vm=random
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 1 streams to port(s) [0._]: [SUCCESS]
Attaching 1 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
46.61 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 1500 | Continuous | 1 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 1500 | Continuous | 1 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 72 Gbps
version : STL @ v3.03 total_tx_L1 : 72.96 Gbps
cpu_util. : 26.79% @ 4 cores (4 per dual port) total_rx : 64.88 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 6 Mpps
async_util. : 0% / 36.31 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 26.79% | 26.79% |
-- | | |
Tx bps L2 | 35.96 Gbps | 36.04 Gbps | 72 Gbps
Tx bps L1 | 36.44 Gbps | 36.52 Gbps | 72.96 Gbps
Tx pps | 3 Mpps | 3 Mpps | 6 Mpps
Line Util. | 36.44 % | 36.52 % |
--- | | |
Rx bps | 31.49 Gbps | 33.4 Gbps | 64.88 Gbps
Rx pps | 2.62 Mpps | 2.78 Mpps | 5.39 Mpps
---- | | |
opackets | 37040494 | 37062600 | 74103094
ipackets | 32204510 | 34393870 | 66598380
obytes | 55560741000 | 55593900000 | 111154641000
ibytes | 48435583040 | 51728380480 | 100163963520
tx-pkts | 37.04 Mpkts | 37.06 Mpkts | 74.1 Mpkts
rx-pkts | 32.2 Mpkts | 34.39 Mpkts | 66.6 Mpkts
tx-bytes | 55.56 GB | 55.59 GB | 111.15 GB
rx-bytes | 48.44 GB | 51.73 GB | 100.16 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
Without VPP, flowtables enabled
Kernel options: nosmt mitigations=off
CPU cores disabled:
for i in 1 3 5 7 9 11 13 15 17 19 21 23 25 27 ; do echo 0 > /sys/devices/system/cpu/cpu${i}/online ; done
Tuned queues count to fit the available amount of CPU cores:
for i in 6 7 ; do ethtool -L eth${i} combined 14 ; done
Configuration:
set firewall flowtable ft1 interface 'eth6'
set firewall flowtable ft1 interface 'eth7'
set firewall flowtable ft1 offload 'software'
set firewall group interface-group connectx6 interface 'eth6'
set firewall group interface-group connectx6 interface 'eth7'
set firewall ipv4 forward filter default-action 'accept'
set firewall ipv4 forward filter rule 10 action 'jump'
set firewall ipv4 forward filter rule 10 inbound-interface group 'connectx6'
set firewall ipv4 forward filter rule 10 jump-target 'off_ft1'
set firewall ipv4 name off_ft1 default-action 'accept'
set firewall ipv4 name off_ft1 rule 10 action 'offload'
set firewall ipv4 name off_ft1 rule 10 offload-target 'ft1'
set interfaces ethernet eth6 address '192.168.2.1/24'
set interfaces ethernet eth6 ring-buffer rx '8192'
set interfaces ethernet eth6 ring-buffer tx '8192'
set interfaces ethernet eth7 address '192.168.3.1/24'
set interfaces ethernet eth7 ring-buffer rx '8192'
set interfaces ethernet eth7 ring-buffer tx '8192'
set protocols static arp interface eth6 address 192.168.2.2 mac '<TREX_MAC_IF1>'
set protocols static arp interface eth7 address 192.168.3.2 mac '<TREX_MAC_IF2>'
set protocols static route 16.0.0.0/8 next-hop 192.168.2.2
set protocols static route 48.0.0.0/8 next-hop 192.168.3.2
set system conntrack # empty section
set system sysctl parameter net.core.rmem_default value '134217728'
set system sysctl parameter net.core.rmem_max value '536870912'
set system sysctl parameter net.core.wmem_default value '134217728'
set system sysctl parameter net.core.wmem_max value '536870912'
Results:
64-byte packets:
trex>start -f stl/bench.py -m 9.5mpps -t size=64,vm=random
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 1 streams to port(s) [0._]: [SUCCESS]
Attaching 1 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
51.92 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 64 | Continuous | 1 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 64 | Continuous | 1 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 9.74 Gbps
version : STL @ v3.03 total_tx_L1 : 12.78 Gbps
cpu_util. : 79.49% @ 4 cores (4 per dual port) total_rx : 9.01 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 19.02 Mpps
async_util. : 0% / 14.03 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 79.49% | 79.49% |
-- | | |
Tx bps L2 | 4.87 Gbps | 4.87 Gbps | 9.74 Gbps
Tx bps L1 | 6.39 Gbps | 6.39 Gbps | 12.78 Gbps
Tx pps | 9.51 Mpps | 9.51 Mpps | 19.02 Mpps
Line Util. | 6.39 % | 6.39 % |
--- | | |
Rx bps | 4.5 Gbps | 4.51 Gbps | 9.01 Gbps
Rx pps | 8.28 Mpps | 8.28 Mpps | 16.56 Mpps
---- | | |
opackets | 420893819 | 420968024 | 841861843
ipackets | 366796248 | 366943397 | 733739645
obytes | 26937204416 | 26941953536 | 53879157952
ibytes | 24942144864 | 24952150996 | 49894295860
tx-pkts | 420.89 Mpkts | 420.97 Mpkts | 841.86 Mpkts
rx-pkts | 366.8 Mpkts | 366.94 Mpkts | 733.74 Mpkts
tx-bytes | 26.94 GB | 26.94 GB | 53.88 GB
rx-bytes | 24.94 GB | 24.95 GB | 49.89 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
IMIX packets:
trex>start -f stl/bench.py -m 8mpps -t size=imix,vm=random
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 3 streams to port(s) [0._]: [SUCCESS]
Attaching 3 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
64.51 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 68 | Continuous | 28 pps | - | -
2 | - | _ | Ethernet:IP:UDP:Raw | 590 | Continuous | 16 pps | - | -
3 | - | _ | Ethernet:IP:UDP:Raw | 1514 | Continuous | 4 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 68 | Continuous | 28 pps | - | -
2 | - | _ | Ethernet:IP:UDP:Raw | 590 | Continuous | 16 pps | - | -
3 | - | _ | Ethernet:IP:UDP:Raw | 1514 | Continuous | 4 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 46.44 Gbps
version : STL @ v3.03 total_tx_L1 : 49 Gbps
cpu_util. : 73.02% @ 4 cores (4 per dual port) total_rx : 43.16 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 16.01 Mpps
async_util. : 0% / 23.93 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 73.02% | 73.02% |
-- | | |
Tx bps L2 | 23.2 Gbps | 23.23 Gbps | 46.44 Gbps
Tx bps L1 | 24.48 Gbps | 24.51 Gbps | 49 Gbps
Tx pps | 8 Mpps | 8.01 Mpps | 16.01 Mpps
Line Util. | 24.48 % | 24.51 % |
--- | | |
Rx bps | 21.57 Gbps | 21.59 Gbps | 43.16 Gbps
Rx pps | 7.35 Mpps | 7.36 Mpps | 14.72 Mpps
---- | | |
opackets | 179052555 | 179107649 | 358160204
ipackets | 164708883 | 164836260 | 329545143
obytes | 64906551348 | 64926523378 | 129833074726
ibytes | 60369421326 | 60416771286 | 120786192612
tx-pkts | 179.05 Mpkts | 179.11 Mpkts | 358.16 Mpkts
rx-pkts | 164.71 Mpkts | 164.84 Mpkts | 329.55 Mpkts
tx-bytes | 64.91 GB | 64.93 GB | 129.83 GB
rx-bytes | 60.37 GB | 60.42 GB | 120.79 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
1400-byte packets:
trex>start -f stl/bench.py -m 3mpps -t size=1400,vm=random
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 1 streams to port(s) [0._]: [SUCCESS]
Attaching 1 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
44.84 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 1400 | Continuous | 1 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
1 | - | _ | Ethernet:IP:UDP:Raw | 1400 | Continuous | 1 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 67.19 Gbps
version : STL @ v3.03 total_tx_L1 : 68.15 Gbps
cpu_util. : 26.5% @ 4 cores (4 per dual port) total_rx : 65.77 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 6 Mpps
async_util. : 0% / 28.19 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 26.5% | 26.5% |
-- | | |
Tx bps L2 | 33.56 Gbps | 33.63 Gbps | 67.19 Gbps
Tx bps L1 | 34.04 Gbps | 34.11 Gbps | 68.15 Gbps
Tx pps | 3 Mpps | 3 Mpps | 6 Mpps
Line Util. | 34.04 % | 34.11 % |
--- | | |
Rx bps | 32.63 Gbps | 33.15 Gbps | 65.77 Gbps
Rx pps | 2.9 Mpps | 2.95 Mpps | 5.86 Mpps
---- | | |
opackets | 39684257 | 39682341 | 79366598
ipackets | 38432786 | 39025804 | 77458590
obytes | 55557959800 | 55555277400 | 111113237200
ibytes | 53959631544 | 54792228816 | 108751860360
tx-pkts | 39.68 Mpkts | 39.68 Mpkts | 79.37 Mpkts
rx-pkts | 38.43 Mpkts | 39.03 Mpkts | 77.46 Mpkts
tx-bytes | 55.56 GB | 55.56 GB | 111.11 GB
rx-bytes | 53.96 GB | 54.79 GB | 108.75 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
With VPP
Kernel options: nosmt mitigations=off isolcpus=1-27 nohz_full=1-27 rcu_nocbs=1-27 rcu_nocb_poll audit=0 intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable intel_iommu=on iommu=pt modules_load=vfio,vfio_iommu_type1,vfio_pci
Flow Control disabled:
for i in 6 7 ; do ethtool -A defunct_eth${i} rx off tx off ; done
Disabled limit for realtime scheduling:
echo -1 > /proc/sys/kernel/sched_rt_runtime_us
Configuration:
set interfaces ethernet eth6 address '192.168.2.1/24'
set interfaces ethernet eth7 address '192.168.3.1/24'
set protocols static arp interface eth6 address 192.168.2.2 mac '<TREX_MAC_IF1>'
set protocols static arp interface eth7 address 192.168.3.2 mac '<TREX_MAC_IF2>'
set protocols static route 16.0.0.0/8 next-hop 192.168.2.2
set protocols static route 48.0.0.0/8 next-hop 192.168.3.2
set system sysctl parameter net.core.rmem_default value '134217728'
set system sysctl parameter net.core.rmem_max value '536870912'
set system sysctl parameter net.core.wmem_default value '134217728'
set system sysctl parameter net.core.wmem_max value '536870912'
set vpp settings buffers buffers-per-numa '65536'
set vpp settings buffers data-size '2048'
set vpp settings buffers page-size 'default-hugepage'
set vpp settings cpu corelist-workers '4'
set vpp settings cpu corelist-workers '6'
set vpp settings cpu corelist-workers '8'
set vpp settings cpu corelist-workers '10'
set vpp settings cpu corelist-workers '12'
set vpp settings cpu corelist-workers '16'
set vpp settings cpu corelist-workers '18'
set vpp settings cpu corelist-workers '20'
set vpp settings cpu corelist-workers '22'
set vpp settings cpu corelist-workers '24'
set vpp settings cpu corelist-workers '26'
set vpp settings cpu main-core '2'
set vpp settings host-resources max-map-count '6176'
set vpp settings host-resources nr-hugepages '8192'
set vpp settings host-resources shmmax '6442450944'
set vpp settings interface eth6 dpdk-options num-rx-queues '12'
set vpp settings interface eth6 dpdk-options num-tx-queues '12'
set vpp settings interface eth6 driver 'dpdk'
set vpp settings interface eth7 dpdk-options num-rx-queues '12'
set vpp settings interface eth7 dpdk-options num-tx-queues '12'
set vpp settings interface eth7 driver 'dpdk'
set vpp settings lcp netlink rx-buffer-size '536870912'
set vpp settings memory main-heap-page-size 'default-hugepage'
set vpp settings memory main-heap-size '4G'
set vpp settings statseg page-size 'default-hugepage'
set vpp settings statseg size '256M'
Results:
64-byte packets:
trex>start -f stl/bench.py -m 36mpps -t size=64,vm=cached
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 1 streams to port(s) [0._]: [SUCCESS]
Attaching 1 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
20.94 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
10 | - | _ | Ethernet:IP:UDP:Raw | 64 | Continuous | 1 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
10 | - | _ | Ethernet:IP:UDP:Raw | 64 | Continuous | 1 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 36.89 Gbps
version : STL @ v3.03 total_tx_L1 : 48.42 Gbps
cpu_util. : 49.23% @ 7 cores (7 per dual port) total_rx : 33.87 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 72.05 Mpps
async_util. : 0% / 10.54 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 49.23% | 49.23% |
-- | | |
Tx bps L2 | 18.43 Gbps | 18.46 Gbps | 36.89 Gbps
Tx bps L1 | 24.19 Gbps | 24.22 Gbps | 48.42 Gbps
Tx pps | 36 Mpps | 36.05 Mpps | 72.05 Mpps
Line Util. | 24.19 % | 24.22 % |
--- | | |
Rx bps | 16.92 Gbps | 16.94 Gbps | 33.87 Gbps
Rx pps | 31.11 Mpps | 31.15 Mpps | 62.25 Mpps
---- | | |
opackets | 959158591 | 959511833 | 1918670424
ipackets | 828779015 | 829084198 | 1657863213
obytes | 61386149824 | 61408757312 | 122794907136
ibytes | 56356973020 | 56377725464 | 112734698484
tx-pkts | 959.16 Mpkts | 959.51 Mpkts | 1.92 Gpkts
rx-pkts | 828.78 Mpkts | 829.08 Mpkts | 1.66 Gpkts
tx-bytes | 61.39 GB | 61.41 GB | 122.79 GB
rx-bytes | 56.36 GB | 56.38 GB | 112.73 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
IMIX packets:
trex>start -f stl/bench.py -m 11.5mpps -t size=imix,vm=cached
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 3 streams to port(s) [0._]: [SUCCESS]
Attaching 3 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
67.94 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
35 | - | _ | Ethernet:IP:UDP:Raw | 68 | Continuous | 28 pps | - | -
36 | - | _ | Ethernet:IP:UDP:Raw | 590 | Continuous | 16 pps | - | -
37 | - | _ | Ethernet:IP:UDP:Raw | 1514 | Continuous | 4 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
35 | - | _ | Ethernet:IP:UDP:Raw | 68 | Continuous | 28 pps | - | -
36 | - | _ | Ethernet:IP:UDP:Raw | 590 | Continuous | 16 pps | - | -
37 | - | _ | Ethernet:IP:UDP:Raw | 1514 | Continuous | 4 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 66.74 Gbps
version : STL @ v3.03 total_tx_L1 : 70.43 Gbps
cpu_util. : 18.85% @ 7 cores (7 per dual port) total_rx : 60.65 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 23.01 Mpps
async_util. : 0% / 18.37 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 18.85% | 18.85% |
-- | | |
Tx bps L2 | 33.36 Gbps | 33.39 Gbps | 66.74 Gbps
Tx bps L1 | 35.2 Gbps | 35.23 Gbps | 70.43 Gbps
Tx pps | 11.5 Mpps | 11.51 Mpps | 23.01 Mpps
Line Util. | 35.2 % | 35.23 % |
--- | | |
Rx bps | 30.31 Gbps | 30.34 Gbps | 60.65 Gbps
Rx pps | 10.24 Mpps | 10.24 Mpps | 20.48 Mpps
---- | | |
opackets | 584183690 | 584248776 | 1168432466
ipackets | 520213601 | 520153718 | 1040367319
obytes | 211766581816 | 211790173068 | 423556754884
ibytes | 192466280166 | 192494896980 | 384961177146
tx-pkts | 584.18 Mpkts | 584.25 Mpkts | 1.17 Gpkts
rx-pkts | 520.21 Mpkts | 520.15 Mpkts | 1.04 Gpkts
tx-bytes | 211.77 GB | 211.79 GB | 423.56 GB
rx-bytes | 192.47 GB | 192.49 GB | 384.96 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
1400-byte packets:
trex>start -f stl/bench.py -m 3.5mpps -t size=1400,vm=cached
Removing all streams from port(s) [0._, 1._]: [SUCCESS]
Attaching 1 streams to port(s) [0._]: [SUCCESS]
Attaching 1 streams to port(s) [1._]: [SUCCESS]
Starting traffic on port(s) [0._, 1._]: [SUCCESS]
43.83 [ms]
trex>streams
Port 0:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
46 | - | _ | Ethernet:IP:UDP:Raw | 1400 | Continuous | 1 pps | - | -
Port 1:
ID | name | profile | packet type | length | mode | rate | PG ID | next
-----------+-----------------+-----------------+---------------------+----------+------------------+-----------------+--------------+-------------
46 | - | _ | Ethernet:IP:UDP:Raw | 1400 | Continuous | 1 pps | - | -
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 78.49 Gbps
version : STL @ v3.03 total_tx_L1 : 79.61 Gbps
cpu_util. : 5.66% @ 7 cores (7 per dual port) total_rx : 76.69 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 7.01 Mpps
async_util. : 0% / 24.97 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | TRANSMITTING |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 5.66% | 5.66% |
-- | | |
Tx bps L2 | 39.23 Gbps | 39.26 Gbps | 78.49 Gbps
Tx bps L1 | 39.79 Gbps | 39.82 Gbps | 79.61 Gbps
Tx pps | 3.5 Mpps | 3.51 Mpps | 7.01 Mpps
Line Util. | 39.79 % | 39.82 % |
--- | | |
Rx bps | 38.46 Gbps | 38.22 Gbps | 76.69 Gbps
Rx pps | 3.42 Mpps | 3.4 Mpps | 6.83 Mpps
---- | | |
opackets | 120705811 | 120701086 | 241406897
ipackets | 118039259 | 117166761 | 235206020
obytes | 168988134074 | 168981520400 | 337969654474
ibytes | 165727119636 | 164502132444 | 330229252080
tx-pkts | 120.71 Mpkts | 120.7 Mpkts | 241.41 Mpkts
rx-pkts | 118.04 Mpkts | 117.17 Mpkts | 235.21 Mpkts
tx-bytes | 168.99 GB | 168.98 GB | 337.97 GB
rx-bytes | 165.73 GB | 164.5 GB | 330.23 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0