Help! NetworkManager High CPU Usage.

CrazyWes

New Member
Joined
Jul 15, 2022
Messages
21
Reaction score
3
Credits
517
Hello there,

I have a 3 node Docker Swarm running and managed by Portainer.
On my docker nodes, running the top command in the CLI I notice that NetworkManager is using 100% of the CPU. It seems to hop around from node to node but is only ever using 100% of the CPU on 2 out of the 3 nodes at any given time. All 3 nodes are manager nodes.
I also notice that when NetworkManager is using 100% CPU the nmcli general status command shows it in a disabled state.

This is the output from Syslog that stands out to me. Has anyone seen this before and can help me debug and fix this? Google is not helping me. Thanks. If anymore info is needed let me know.
The part of the output that catches my attention starts at line 1555 where it shows config failed.

Line 27: Dec 12 14:21:31 docker2 NetworkManager[537]: <info> [1670883691.7578] manager: (veth96604d0): new Veth device (/org/freedesktop/NetworkManager/Devices/185430)
Line 53: Dec 12 14:21:32 docker2 NetworkManager[537]: <info> [1670883692.0109] manager: (vethe7b8d47): new Veth device (/org/freedesktop/NetworkManager/Devices/185431)
Line 60: Dec 12 14:21:32 docker2 NetworkManager[537]: <info> [1670883692.1554] manager: (vethf79351b): new Veth device (/org/freedesktop/NetworkManager/Devices/185432)
Line 67: Dec 12 14:21:32 docker2 NetworkManager[537]: <info> [1670883692.2636] manager: (vethd87b3ca): new Veth device (/org/freedesktop/NetworkManager/Devices/185433)
Line 68: Dec 12 14:21:32 docker2 NetworkManager[537]: <info> [1670883692.4540] device (vethf79351b): carrier: link connected
Line 78: Dec 12 14:21:34 docker2 NetworkManager[537]: <info> [1670883694.7430] manager: (veth96604d0): new Veth device (/org/freedesktop/NetworkManager/Devices/185434)
Line 90: Dec 12 14:21:34 docker2 NetworkManager[537]: <info> [1670883694.9119] manager: (vethd87b3ca): new Veth device (/org/freedesktop/NetworkManager/Devices/185435)
Line 107: Dec 12 14:21:35 docker2 NetworkManager[537]: <info> [1670883695.1546] device (vethf79351b): released from master device docker_gwbridge
Line 125: Dec 12 14:21:58 docker2 NetworkManager[537]: <info> [1670883718.0835] manager: (veth962659b): new Veth device (/org/freedesktop/NetworkManager/Devices/185436)
Line 151: Dec 12 14:21:58 docker2 NetworkManager[537]: <info> [1670883718.3448] manager: (veth0b3b4f8): new Veth device (/org/freedesktop/NetworkManager/Devices/185437)
Line 152: Dec 12 14:21:58 docker2 NetworkManager[537]: <info> [1670883718.4444] manager: (veth3adce94): new Veth device (/org/freedesktop/NetworkManager/Devices/185438)
Line 161: Dec 12 14:21:58 docker2 NetworkManager[537]: <info> [1670883718.5411] manager: (veth369021f): new Veth device (/org/freedesktop/NetworkManager/Devices/185439)
Line 166: Dec 12 14:21:58 docker2 NetworkManager[537]: <info> [1670883718.7766] device (veth3adce94): carrier: link connected
Line 185: Dec 12 14:22:01 docker2 NetworkManager[537]: <info> [1670883721.1692] manager: (veth962659b): new Veth device (/org/freedesktop/NetworkManager/Devices/185440)
Line 211: Dec 12 14:22:01 docker2 NetworkManager[537]: <info> [1670883721.4571] device (veth3adce94): released from master device docker_gwbridge
Line 235: Dec 12 14:22:35 docker2 NetworkManager[537]: <info> [1670883755.0677] manager: (vethd8145a4): new Veth device (/org/freedesktop/NetworkManager/Devices/185441)
Line 259: Dec 12 14:22:35 docker2 NetworkManager[537]: <info> [1670883755.2826] manager: (vethfdad821): new Veth device (/org/freedesktop/NetworkManager/Devices/185442)
Line 260: Dec 12 14:22:35 docker2 NetworkManager[537]: <info> [1670883755.4134] manager: (vethe88e4d9): new Veth device (/org/freedesktop/NetworkManager/Devices/185443)
Line 273: Dec 12 14:22:35 docker2 NetworkManager[537]: <info> [1670883755.5236] manager: (vethf8c56bb): new Veth device (/org/freedesktop/NetworkManager/Devices/185444)
Line 274: Dec 12 14:22:35 docker2 NetworkManager[537]: <info> [1670883755.7014] device (vethe88e4d9): carrier: link connected
Line 284: Dec 12 14:22:37 docker2 NetworkManager[537]: <info> [1670883757.8895] manager: (vethd8145a4): new Veth device (/org/freedesktop/NetworkManager/Devices/185445)
Line 299: Dec 12 14:22:38 docker2 NetworkManager[537]: <info> [1670883758.0648] manager: (vethf8c56bb): new Veth device (/org/freedesktop/NetworkManager/Devices/185446)
Line 315: Dec 12 14:22:38 docker2 NetworkManager[537]: <info> [1670883758.2782] device (vethe88e4d9): released from master device docker_gwbridge
Line 364: Dec 12 14:23:02 docker2 NetworkManager[537]: <info> [1670883782.2111] manager: (veth585b47c): new Veth device (/org/freedesktop/NetworkManager/Devices/185447)
Line 369: Dec 12 14:23:02 docker2 NetworkManager[537]: <info> [1670883782.3467] manager: (veth27d817a): new Veth device (/org/freedesktop/NetworkManager/Devices/185448)
Line 370: Dec 12 14:23:02 docker2 NetworkManager[537]: <info> [1670883782.4489] manager: (veth4457ed5): new Veth device (/org/freedesktop/NetworkManager/Devices/185449)
Line 379: Dec 12 14:23:02 docker2 NetworkManager[537]: <info> [1670883782.5447] manager: (vethc9a02a1): new Veth device (/org/freedesktop/NetworkManager/Devices/185450)
Line 384: Dec 12 14:23:02 docker2 NetworkManager[537]: <info> [1670883782.7343] device (veth4457ed5): carrier: link connected
Line 403: Dec 12 14:23:05 docker2 NetworkManager[537]: <info> [1670883785.2460] manager: (vethc9a02a1): new Veth device (/org/freedesktop/NetworkManager/Devices/185451)
Line 423: Dec 12 14:23:05 docker2 NetworkManager[537]: <info> [1670883785.4431] device (veth4457ed5): released from master device docker_gwbridge
Line 438: Dec 12 14:23:10 docker2 NetworkManager[537]: <info> [1670883790.6623] manager: (veth06655b6): new Veth device (/org/freedesktop/NetworkManager/Devices/185452)
Line 464: Dec 12 14:23:10 docker2 NetworkManager[537]: <info> [1670883790.9244] manager: (veth7aab232): new Veth device (/org/freedesktop/NetworkManager/Devices/185453)
Line 466: Dec 12 14:23:11 docker2 NetworkManager[537]: <info> [1670883791.0483] manager: (veth4faa5fc): new Veth device (/org/freedesktop/NetworkManager/Devices/185454)
Line 475: Dec 12 14:23:11 docker2 NetworkManager[537]: <info> [1670883791.1766] manager: (vethf6cd508): new Veth device (/org/freedesktop/NetworkManager/Devices/185455)
Line 476: Dec 12 14:23:11 docker2 NetworkManager[537]: <info> [1670883791.3444] device (veth4faa5fc): carrier: link connected
Line 489: Dec 12 14:23:13 docker2 NetworkManager[537]: <info> [1670883793.6980] manager: (vethf6cd508): new Veth device (/org/freedesktop/NetworkManager/Devices/185456)
Line 514: Dec 12 14:23:13 docker2 NetworkManager[537]: <info> [1670883793.9869] device (veth4faa5fc): released from master device docker_gwbridge
Line 556: Dec 12 14:23:19 docker2 NetworkManager[537]: <info> [1670883799.4203] manager: (veth9d877d2): new Veth device (/org/freedesktop/NetworkManager/Devices/185457)
Line 557: Dec 12 14:23:19 docker2 NetworkManager[537]: <info> [1670883799.5311] manager: (veth5f8685f): new Veth device (/org/freedesktop/NetworkManager/Devices/185458)
Line 559: Dec 12 14:23:19 docker2 NetworkManager[537]: <info> [1670883799.6309] manager: (vethf124818): new Veth device (/org/freedesktop/NetworkManager/Devices/185459)
Line 571: Dec 12 14:23:19 docker2 NetworkManager[537]: <info> [1670883799.7525] manager: (vethd814c56): new Veth device (/org/freedesktop/NetworkManager/Devices/185460)
Line 572: Dec 12 14:23:19 docker2 NetworkManager[537]: <info> [1670883799.9382] device (vethf124818): carrier: link connected
Line 585: Dec 12 14:23:22 docker2 NetworkManager[537]: <info> [1670883802.0677] manager: (veth9d877d2): new Veth device (/org/freedesktop/NetworkManager/Devices/185461)
Line 614: Dec 12 14:23:22 docker2 NetworkManager[537]: <info> [1670883802.3767] device (vethf124818): released from master device docker_gwbridge
Line 655: Dec 12 14:23:27 docker2 NetworkManager[537]: <info> [1670883807.7600] manager: (vetha6f4c3b): new Veth device (/org/freedesktop/NetworkManager/Devices/185462)
Line 656: Dec 12 14:23:27 docker2 NetworkManager[537]: <info> [1670883807.8703] manager: (veth9300717): new Veth device (/org/freedesktop/NetworkManager/Devices/185463)
Line 663: Dec 12 14:23:28 docker2 NetworkManager[537]: <info> [1670883808.0245] manager: (veth86fe2e9): new Veth device (/org/freedesktop/NetworkManager/Devices/185464)
Line 667: Dec 12 14:23:28 docker2 NetworkManager[537]: <info> [1670883808.1407] manager: (veth68d5ed6): new Veth device (/org/freedesktop/NetworkManager/Devices/185465)
Line 668: Dec 12 14:23:28 docker2 NetworkManager[537]: <info> [1670883808.3370] device (veth86fe2e9): carrier: link connected
Line 687: Dec 12 14:23:30 docker2 NetworkManager[537]: <info> [1670883810.5872] manager: (vetha6f4c3b): new Veth device (/org/freedesktop/NetworkManager/Devices/185466)
Line 707: Dec 12 14:23:30 docker2 NetworkManager[537]: <info> [1670883810.7959] device (veth86fe2e9): released from master device docker_gwbridge
Line 752: Dec 12 14:23:36 docker2 NetworkManager[537]: <info> [1670883816.1947] manager: (veth46d4e19): new Veth device (/org/freedesktop/NetworkManager/Devices/185467)
Line 753: Dec 12 14:23:36 docker2 NetworkManager[537]: <info> [1670883816.3209] manager: (veth6877f05): new Veth device (/org/freedesktop/NetworkManager/Devices/185468)
Line 757: Dec 12 14:23:36 docker2 NetworkManager[537]: <info> [1670883816.4432] manager: (vethe2c851d): new Veth device (/org/freedesktop/NetworkManager/Devices/185469)
Line 764: Dec 12 14:23:36 docker2 NetworkManager[537]: <info> [1670883816.5503] manager: (veth42840b1): new Veth device (/org/freedesktop/NetworkManager/Devices/185470)
Line 765: Dec 12 14:23:36 docker2 NetworkManager[537]: <info> [1670883816.7438] device (vethe2c851d): carrier: link connected
Line 784: Dec 12 14:23:39 docker2 NetworkManager[537]: <info> [1670883819.2023] manager: (veth46d4e19): new Veth device (/org/freedesktop/NetworkManager/Devices/185471)
Line 804: Dec 12 14:23:39 docker2 NetworkManager[537]: <info> [1670883819.4068] device (vethe2c851d): released from master device docker_gwbridge
Line 1548: > Dec 12 13:13:54 docker1 NetworkManager[520]: <info> [1670879634.3788] policy: auto-activating connection 'Wired connection 5958' (99a48012-c467-3662-8bc0-108a96f9600b)
Line 1549: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.4219] platform-linux: netlink: read: too many netlink events. Need to resynchronize platform cache
Line 1550: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.5834] device (veth901b797): Activation: starting connection 'Wired connection 5819' (2c47bef4-6fbe-3d0c-a4e4-428b241582d9)
Line 1551: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.5836] device (vethceebbdb): Activation: starting connection 'Wired connection 5954' (8eeacf1a-a66f-3b2e-aa4a-58464b88fe67)
Line 1552: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.5838] device (veth8d90788): Activation: starting connection 'Wired connection 5958' (99a48012-c467-3662-8bc0-108a96f9600b)
Line 1553: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.5882] device (vethc9456c7): carrier: link connected
Line 1554: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.5901] manager: (vethc9456c7): new Veth device (/org/freedesktop/NetworkManager/Devices/136032)
Line 1555: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.6730] device (veth901b797): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Line 1556: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.6819] device (veth901b797): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Line 1557: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.6844] device (veth901b797): state change: config -> failed (reason 'config-failed', sys-iface-state: 'managed')
Line 1558: > Dec 12 13:13:55 docker1 NetworkManager[520]: <warn> [1670879635.7072] device (veth901b797): Activation: failed for connection 'Wired connection 5819'
Line 1559: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7074] device (vethceebbdb): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Line 1560: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7151] device (vethceebbdb): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Line 1561: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7165] device (vethceebbdb): state change: config -> failed (reason 'config-failed', sys-iface-state: 'managed')
Line 1562: > Dec 12 13:13:55 docker1 NetworkManager[520]: <warn> [1670879635.7363] device (vethceebbdb): Activation: failed for connection 'Wired connection 5954'
Line 1563: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7364] device (veth8d90788): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Line 1564: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7441] device (veth8d90788): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Line 1565: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7456] device (veth8d90788): state change: config -> failed (reason 'config-failed', sys-iface-state: 'managed')
Line 1566: > Dec 12 13:13:55 docker1 NetworkManager[520]: <warn> [1670879635.7651] device (veth8d90788): Activation: failed for connection 'Wired connection 5958'
Line 1567: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7655] device (veth901b797): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Line 1568: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.7694] device (vethceebbdb): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Line 1569: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.9100] device (veth8d90788): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Line 1570: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.9566] policy: auto-activating connection 'Wired connection 5819' (2c47bef4-6fbe-3d0c-a4e4-428b241582d9)
Line 1571: > Dec 12 13:13:55 docker1 NetworkManager[520]: <info> [1670879635.9978] policy: auto-activating connection 'Wired connection 5954' (8eeacf1a-a66f-3b2e-aa4a-58464b88fe67)
Line 2364: > Dec 12 13:18:08 docker1 NetworkManager[520]: <info> [1670879888.6745] policy: auto-activating connection 'Wired connection 5958' (99a48012-c467-3662-8bc0-108a96f9600b)
Line 2365: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.5598] platform-linux: netlink: read: too many netlink events. Need to resynchronize platform cache
Line 2366: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.6717] device (veth901b797): Activation: starting connection 'Wired connection 5819' (2c47bef4-6fbe-3d0c-a4e4-428b241582d9)
Line 2367: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.6724] device (vethceebbdb): Activation: starting connection 'Wired connection 5954' (8eeacf1a-a66f-3b2e-aa4a-58464b88fe67)
Line 2368: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.6728] device (veth8d90788): Activation: starting connection 'Wired connection 5958' (99a48012-c467-3662-8bc0-108a96f9600b)
Line 2369: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.6745] device (veth901b797): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Line 2370: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.6837] device (veth901b797): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Line 2371: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.6860] device (veth901b797): state change: config -> failed (reason 'config-failed', sys-iface-state: 'managed')
Line 2372: > Dec 12 13:18:09 docker1 NetworkManager[520]: <warn> [1670879889.7061] device (veth901b797): Activation: failed for connection 'Wired connection 5819'
Line 2373: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7065] device (vethceebbdb): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Line 2374: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7143] device (vethceebbdb): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Line 2375: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7162] device (vethceebbdb): state change: config -> failed (reason 'config-failed', sys-iface-state: 'managed')
Line 2376: > Dec 12 13:18:09 docker1 NetworkManager[520]: <warn> [1670879889.7362] device (vethceebbdb): Activation: failed for connection 'Wired connection 5954'
Line 2377: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7366] device (veth8d90788): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Line 2378: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7436] device (veth8d90788): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Line 2379: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7454] device (veth8d90788): state change: config -> failed (reason 'config-failed', sys-iface-state: 'managed')
Line 2380: > Dec 12 13:18:09 docker1 NetworkManager[520]: <warn> [1670879889.7655] device (veth8d90788): Activation: failed for connection 'Wired connection 5958'
Line 2381: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7662] device (veth901b797): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Line 2382: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.7704] device (vethceebbdb): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Line 2383: > Dec 12 13:18:09 docker1 NetworkManager[520]: <info> [1670879889.8969] device (veth8d90788): state change: failed -> disconnected (reason 'none', sys-iface-s
 


Moved to Networking - good luck.

Wizard
 

Members online


Latest posts

Top