NUC with Centos 7 crashing: Microcode SW Error Detected












1















I am trying to figure out why the WiFi on my Intel NUC with CentOS 7 continues to die. As info, I have 5 node Hadoop cluster and they are all configured the same (as far as I can tell), however, the other machines which are on WiFi do not crash. I don't know what is wrong with this particular machine.



Here is the error from /var/log/messages. It is the same message that I see regularly as I've been watching this problem for days.




Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000022CE | ADVANCED_SYSASSERT
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x05900280 | trm_hw_status0
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | interruptlink2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000012C | data1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | data3
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD28011F1 | beacon time
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x72F4FDDD | tsf low
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000182 | tsf hi
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xCA511FA7 | time gp2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A96001C | hcmd
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xA7F93882 | isr0
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00050000 | isr1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0020180A | isr2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x40417DCD | isr3
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A95001C | last cmd Id
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00007888 | flow_handler
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00DF019C | last host cmd
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
Jan 2 08:41:06 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: FW error in SYNC CMD STATISTICS_CMD
Jan 2 08:41:06 mapr04 kernel: CPU: 0 PID: 4898 Comm: NetworkManager Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Jan 2 08:41:06 mapr04 kernel: Hardware name: Intel Corporation NUC7i7BNH/NUC7i7BNB, BIOS BNKBL357.86A.0049.2017.0724.1541 07/24/2017
Jan 2 08:41:06 mapr04 kernel: Call Trace:
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb61e41>] dump_stack+0x19/0x1b
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0afa983>] iwl_trans_pcie_send_hcmd+0x563/0x580 [iwlwifi]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae4c2d00>] ? wake_up_atomic_t+0x30/0x30
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0b060fc>] iwl_trans_send_cmd+0x5c/0xe0 [iwlwifi]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6d312>] iwl_mvm_send_cmd+0x32/0xb0 [iwlmvm]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6e632>] iwl_mvm_request_statistics+0x72/0x100 [iwlmvm]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c616fe>] iwl_mvm_mac_sta_statistics+0xbe/0x100 [iwlmvm]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bb68f7>] sta_set_sinfo+0xb7/0x800 [mac80211]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bcd052>] ieee80211_get_station+0x52/0x80 [mac80211]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffc08cae41>] nl80211_get_station+0xa1/0x240 [cfg80211]
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae794d0d>] ? list_del+0xd/0x30
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bdf1a>] ? __rmqueue+0x8a/0x460
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77918>] genl_family_rcv_msg+0x208/0x430
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bf134>] ? free_one_page+0x2e4/0x310
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b9b>] genl_rcv_msg+0x5b/0xc0
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73ec0>] ? __netlink_lookup+0xc0/0x110
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b40>] ? genl_family_rcv_msg+0x430/0x430
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75bab>] netlink_rcv_skb+0xab/0xc0
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea760e8>] genl_rcv+0x28/0x40
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75530>] netlink_unicast+0x170/0x210
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae78c042>] ? memcpy_fromiovec+0x62/0xb0
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea758d8>] netlink_sendmsg+0x308/0x420
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73112>] ? netlink_recvmsg+0x212/0x490
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea193a6>] sock_sendmsg+0xb6/0xf0
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea194f5>] ? sock_recvmsg+0xc5/0x100
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1a269>] ___sys_sendmsg+0x3e9/0x400
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae656fe0>] ? __pollwait+0xf0/0xf0
Jan 2 08:41:06 mapr04 kernel: [<ffffffffae68ee1e>] ? ep_poll+0x31e/0x360
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b921>] __sys_sendmsg+0x51/0x90
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b972>] SyS_sendmsg+0x12/0x20
Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb74ddb>] system_call_fastpath+0x22/0x27



Where should I start trying to debug? I can edit the original post with updates.



Here are some things which I think might be helpful:



uname -a:

Linux mapr04.wired.carnoustie 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



dmesg | grep iwlwifi:

[ 3.822041] iwlwifi 0000:3a:00.0: irq 132 for MSI/MSI-X
[ 3.831295] iwlwifi 0000:3a:00.0: loaded firmware version 34.0.1 op_mode iwlmvm
[ 3.924043] iwlwifi 0000:3a:00.0: Detected Intel(R) Dual Band Wireless AC 8265, REV=0x230
[ 3.984049] iwlwifi 0000:3a:00.0: base HW address: f8:94:c2:5c:07:24



Here is the latest error:

Here is the latest error:
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Error sending STATISTICS_CMD: time out after 2000ms.
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Current CMD queue read_ptr 246 write_ptr 247
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000280 | trm_hw_status0
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | data1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000080 | data2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data3
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD4C029D9 | beacon time
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x456535F1 | tsf low
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000001BA | tsf hi
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x6D3BFE27 | time gp2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000001C | hcmd
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00012000 | isr0
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000180A | isr2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00417CC0 | isr3
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A89001C | last cmd Id
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00002838 | flow_handler
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00F6019C | last host cmd
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
Jan 5 03:17:01 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.










share|improve this question





























    1















    I am trying to figure out why the WiFi on my Intel NUC with CentOS 7 continues to die. As info, I have 5 node Hadoop cluster and they are all configured the same (as far as I can tell), however, the other machines which are on WiFi do not crash. I don't know what is wrong with this particular machine.



    Here is the error from /var/log/messages. It is the same message that I see regularly as I've been watching this problem for days.




    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000022CE | ADVANCED_SYSASSERT
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x05900280 | trm_hw_status0
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | interruptlink2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000012C | data1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | data3
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD28011F1 | beacon time
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x72F4FDDD | tsf low
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000182 | tsf hi
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xCA511FA7 | time gp2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A96001C | hcmd
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xA7F93882 | isr0
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00050000 | isr1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0020180A | isr2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x40417DCD | isr3
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A95001C | last cmd Id
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00007888 | flow_handler
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00DF019C | last host cmd
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
    Jan 2 08:41:06 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
    Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: FW error in SYNC CMD STATISTICS_CMD
    Jan 2 08:41:06 mapr04 kernel: CPU: 0 PID: 4898 Comm: NetworkManager Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
    Jan 2 08:41:06 mapr04 kernel: Hardware name: Intel Corporation NUC7i7BNH/NUC7i7BNB, BIOS BNKBL357.86A.0049.2017.0724.1541 07/24/2017
    Jan 2 08:41:06 mapr04 kernel: Call Trace:
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb61e41>] dump_stack+0x19/0x1b
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0afa983>] iwl_trans_pcie_send_hcmd+0x563/0x580 [iwlwifi]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae4c2d00>] ? wake_up_atomic_t+0x30/0x30
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0b060fc>] iwl_trans_send_cmd+0x5c/0xe0 [iwlwifi]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6d312>] iwl_mvm_send_cmd+0x32/0xb0 [iwlmvm]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6e632>] iwl_mvm_request_statistics+0x72/0x100 [iwlmvm]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c616fe>] iwl_mvm_mac_sta_statistics+0xbe/0x100 [iwlmvm]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bb68f7>] sta_set_sinfo+0xb7/0x800 [mac80211]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bcd052>] ieee80211_get_station+0x52/0x80 [mac80211]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffc08cae41>] nl80211_get_station+0xa1/0x240 [cfg80211]
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae794d0d>] ? list_del+0xd/0x30
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bdf1a>] ? __rmqueue+0x8a/0x460
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77918>] genl_family_rcv_msg+0x208/0x430
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bf134>] ? free_one_page+0x2e4/0x310
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b9b>] genl_rcv_msg+0x5b/0xc0
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73ec0>] ? __netlink_lookup+0xc0/0x110
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b40>] ? genl_family_rcv_msg+0x430/0x430
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75bab>] netlink_rcv_skb+0xab/0xc0
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea760e8>] genl_rcv+0x28/0x40
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75530>] netlink_unicast+0x170/0x210
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae78c042>] ? memcpy_fromiovec+0x62/0xb0
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea758d8>] netlink_sendmsg+0x308/0x420
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73112>] ? netlink_recvmsg+0x212/0x490
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea193a6>] sock_sendmsg+0xb6/0xf0
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea194f5>] ? sock_recvmsg+0xc5/0x100
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1a269>] ___sys_sendmsg+0x3e9/0x400
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae656fe0>] ? __pollwait+0xf0/0xf0
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffae68ee1e>] ? ep_poll+0x31e/0x360
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b921>] __sys_sendmsg+0x51/0x90
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b972>] SyS_sendmsg+0x12/0x20
    Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb74ddb>] system_call_fastpath+0x22/0x27



    Where should I start trying to debug? I can edit the original post with updates.



    Here are some things which I think might be helpful:



    uname -a:

    Linux mapr04.wired.carnoustie 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



    dmesg | grep iwlwifi:

    [ 3.822041] iwlwifi 0000:3a:00.0: irq 132 for MSI/MSI-X
    [ 3.831295] iwlwifi 0000:3a:00.0: loaded firmware version 34.0.1 op_mode iwlmvm
    [ 3.924043] iwlwifi 0000:3a:00.0: Detected Intel(R) Dual Band Wireless AC 8265, REV=0x230
    [ 3.984049] iwlwifi 0000:3a:00.0: base HW address: f8:94:c2:5c:07:24



    Here is the latest error:

    Here is the latest error:
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Error sending STATISTICS_CMD: time out after 2000ms.
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Current CMD queue read_ptr 246 write_ptr 247
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000280 | trm_hw_status0
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | data1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000080 | data2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data3
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD4C029D9 | beacon time
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x456535F1 | tsf low
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000001BA | tsf hi
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x6D3BFE27 | time gp2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000001C | hcmd
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00012000 | isr0
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000180A | isr2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00417CC0 | isr3
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A89001C | last cmd Id
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00002838 | flow_handler
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00F6019C | last host cmd
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
    Jan 5 03:17:01 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
    Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.










    share|improve this question



























      1












      1








      1








      I am trying to figure out why the WiFi on my Intel NUC with CentOS 7 continues to die. As info, I have 5 node Hadoop cluster and they are all configured the same (as far as I can tell), however, the other machines which are on WiFi do not crash. I don't know what is wrong with this particular machine.



      Here is the error from /var/log/messages. It is the same message that I see regularly as I've been watching this problem for days.




      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000022CE | ADVANCED_SYSASSERT
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x05900280 | trm_hw_status0
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | interruptlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000012C | data1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | data3
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD28011F1 | beacon time
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x72F4FDDD | tsf low
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000182 | tsf hi
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xCA511FA7 | time gp2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A96001C | hcmd
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xA7F93882 | isr0
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00050000 | isr1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0020180A | isr2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x40417DCD | isr3
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A95001C | last cmd Id
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00007888 | flow_handler
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00DF019C | last host cmd
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
      Jan 2 08:41:06 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: FW error in SYNC CMD STATISTICS_CMD
      Jan 2 08:41:06 mapr04 kernel: CPU: 0 PID: 4898 Comm: NetworkManager Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
      Jan 2 08:41:06 mapr04 kernel: Hardware name: Intel Corporation NUC7i7BNH/NUC7i7BNB, BIOS BNKBL357.86A.0049.2017.0724.1541 07/24/2017
      Jan 2 08:41:06 mapr04 kernel: Call Trace:
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb61e41>] dump_stack+0x19/0x1b
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0afa983>] iwl_trans_pcie_send_hcmd+0x563/0x580 [iwlwifi]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae4c2d00>] ? wake_up_atomic_t+0x30/0x30
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0b060fc>] iwl_trans_send_cmd+0x5c/0xe0 [iwlwifi]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6d312>] iwl_mvm_send_cmd+0x32/0xb0 [iwlmvm]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6e632>] iwl_mvm_request_statistics+0x72/0x100 [iwlmvm]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c616fe>] iwl_mvm_mac_sta_statistics+0xbe/0x100 [iwlmvm]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bb68f7>] sta_set_sinfo+0xb7/0x800 [mac80211]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bcd052>] ieee80211_get_station+0x52/0x80 [mac80211]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc08cae41>] nl80211_get_station+0xa1/0x240 [cfg80211]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae794d0d>] ? list_del+0xd/0x30
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bdf1a>] ? __rmqueue+0x8a/0x460
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77918>] genl_family_rcv_msg+0x208/0x430
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bf134>] ? free_one_page+0x2e4/0x310
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b9b>] genl_rcv_msg+0x5b/0xc0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73ec0>] ? __netlink_lookup+0xc0/0x110
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b40>] ? genl_family_rcv_msg+0x430/0x430
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75bab>] netlink_rcv_skb+0xab/0xc0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea760e8>] genl_rcv+0x28/0x40
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75530>] netlink_unicast+0x170/0x210
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae78c042>] ? memcpy_fromiovec+0x62/0xb0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea758d8>] netlink_sendmsg+0x308/0x420
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73112>] ? netlink_recvmsg+0x212/0x490
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea193a6>] sock_sendmsg+0xb6/0xf0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea194f5>] ? sock_recvmsg+0xc5/0x100
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1a269>] ___sys_sendmsg+0x3e9/0x400
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae656fe0>] ? __pollwait+0xf0/0xf0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae68ee1e>] ? ep_poll+0x31e/0x360
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b921>] __sys_sendmsg+0x51/0x90
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b972>] SyS_sendmsg+0x12/0x20
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb74ddb>] system_call_fastpath+0x22/0x27



      Where should I start trying to debug? I can edit the original post with updates.



      Here are some things which I think might be helpful:



      uname -a:

      Linux mapr04.wired.carnoustie 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



      dmesg | grep iwlwifi:

      [ 3.822041] iwlwifi 0000:3a:00.0: irq 132 for MSI/MSI-X
      [ 3.831295] iwlwifi 0000:3a:00.0: loaded firmware version 34.0.1 op_mode iwlmvm
      [ 3.924043] iwlwifi 0000:3a:00.0: Detected Intel(R) Dual Band Wireless AC 8265, REV=0x230
      [ 3.984049] iwlwifi 0000:3a:00.0: base HW address: f8:94:c2:5c:07:24



      Here is the latest error:

      Here is the latest error:
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Error sending STATISTICS_CMD: time out after 2000ms.
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Current CMD queue read_ptr 246 write_ptr 247
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000280 | trm_hw_status0
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | data1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000080 | data2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data3
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD4C029D9 | beacon time
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x456535F1 | tsf low
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000001BA | tsf hi
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x6D3BFE27 | time gp2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000001C | hcmd
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00012000 | isr0
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000180A | isr2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00417CC0 | isr3
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A89001C | last cmd Id
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00002838 | flow_handler
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00F6019C | last host cmd
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
      Jan 5 03:17:01 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.










      share|improve this question
















      I am trying to figure out why the WiFi on my Intel NUC with CentOS 7 continues to die. As info, I have 5 node Hadoop cluster and they are all configured the same (as far as I can tell), however, the other machines which are on WiFi do not crash. I don't know what is wrong with this particular machine.



      Here is the error from /var/log/messages. It is the same message that I see regularly as I've been watching this problem for days.




      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000022CE | ADVANCED_SYSASSERT
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x05900280 | trm_hw_status0
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | interruptlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000012C | data1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | data3
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD28011F1 | beacon time
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x72F4FDDD | tsf low
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000182 | tsf hi
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xCA511FA7 | time gp2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A96001C | hcmd
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xA7F93882 | isr0
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00050000 | isr1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0020180A | isr2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x40417DCD | isr3
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A95001C | last cmd Id
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00007888 | flow_handler
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00DF019C | last host cmd
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
      Jan 2 08:41:06 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
      Jan 2 08:41:06 mapr04 kernel: iwlwifi 0000:3a:00.0: FW error in SYNC CMD STATISTICS_CMD
      Jan 2 08:41:06 mapr04 kernel: CPU: 0 PID: 4898 Comm: NetworkManager Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
      Jan 2 08:41:06 mapr04 kernel: Hardware name: Intel Corporation NUC7i7BNH/NUC7i7BNB, BIOS BNKBL357.86A.0049.2017.0724.1541 07/24/2017
      Jan 2 08:41:06 mapr04 kernel: Call Trace:
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb61e41>] dump_stack+0x19/0x1b
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0afa983>] iwl_trans_pcie_send_hcmd+0x563/0x580 [iwlwifi]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae4c2d00>] ? wake_up_atomic_t+0x30/0x30
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0b060fc>] iwl_trans_send_cmd+0x5c/0xe0 [iwlwifi]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6d312>] iwl_mvm_send_cmd+0x32/0xb0 [iwlmvm]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c6e632>] iwl_mvm_request_statistics+0x72/0x100 [iwlmvm]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0c616fe>] iwl_mvm_mac_sta_statistics+0xbe/0x100 [iwlmvm]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bb68f7>] sta_set_sinfo+0xb7/0x800 [mac80211]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc0bcd052>] ieee80211_get_station+0x52/0x80 [mac80211]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffc08cae41>] nl80211_get_station+0xa1/0x240 [cfg80211]
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae794d0d>] ? list_del+0xd/0x30
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bdf1a>] ? __rmqueue+0x8a/0x460
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77918>] genl_family_rcv_msg+0x208/0x430
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae5bf134>] ? free_one_page+0x2e4/0x310
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b9b>] genl_rcv_msg+0x5b/0xc0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73ec0>] ? __netlink_lookup+0xc0/0x110
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea77b40>] ? genl_family_rcv_msg+0x430/0x430
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75bab>] netlink_rcv_skb+0xab/0xc0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea760e8>] genl_rcv+0x28/0x40
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea75530>] netlink_unicast+0x170/0x210
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae78c042>] ? memcpy_fromiovec+0x62/0xb0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea758d8>] netlink_sendmsg+0x308/0x420
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea73112>] ? netlink_recvmsg+0x212/0x490
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea193a6>] sock_sendmsg+0xb6/0xf0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea194f5>] ? sock_recvmsg+0xc5/0x100
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1a269>] ___sys_sendmsg+0x3e9/0x400
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae656fe0>] ? __pollwait+0xf0/0xf0
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffae68ee1e>] ? ep_poll+0x31e/0x360
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b921>] __sys_sendmsg+0x51/0x90
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaea1b972>] SyS_sendmsg+0x12/0x20
      Jan 2 08:41:06 mapr04 kernel: [<ffffffffaeb74ddb>] system_call_fastpath+0x22/0x27



      Where should I start trying to debug? I can edit the original post with updates.



      Here are some things which I think might be helpful:



      uname -a:

      Linux mapr04.wired.carnoustie 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



      dmesg | grep iwlwifi:

      [ 3.822041] iwlwifi 0000:3a:00.0: irq 132 for MSI/MSI-X
      [ 3.831295] iwlwifi 0000:3a:00.0: loaded firmware version 34.0.1 op_mode iwlmvm
      [ 3.924043] iwlwifi 0000:3a:00.0: Detected Intel(R) Dual Band Wireless AC 8265, REV=0x230
      [ 3.984049] iwlwifi 0000:3a:00.0: base HW address: f8:94:c2:5c:07:24



      Here is the latest error:

      Here is the latest error:
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Error sending STATISTICS_CMD: time out after 2000ms.
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Current CMD queue read_ptr 246 write_ptr 247
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 6
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Loaded firmware version: 34.0.1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000280 | trm_hw_status0
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | trm_hw_status1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00023FDC | branchlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0003915A | interruptlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | data1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000080 | data2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x03830000 | data3
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xD4C029D9 | beacon time
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x456535F1 | tsf low
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000001BA | tsf hi
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | time gp1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x6D3BFE27 | time gp2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000001 | uCode revision type
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | uCode version major
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | uCode version minor
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000230 | hw version
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00C89000 | board version
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000001C | hcmd
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00012000 | isr0
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000180A | isr2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00417CC0 | isr3
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr4
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0A89001C | last cmd Id
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | wait_event
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00004288 | l2p_control
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00018024 | l2p_duration
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | l2p_mhvalid
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x000000EF | l2p_addr_match
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x0000000D | lmpm_pmg_sel
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x30101345 | timestamp
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00002838 | flow_handler
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Start IWL Error Log Dump:
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Status: 0x00000100, count: 7
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000070 | ADVANCED_SYSASSERT
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac branchlink1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0086964 | umac branchlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac interruptlink2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000800 | umac data1
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC0083A94 | umac data2
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xDEADBEEF | umac data3
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000022 | umac major
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | umac minor
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | frame pointer
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0xC088628C | stack pointer
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00F6019C | last host cmd
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: 0x00000000 | isr status reg
      Jan 5 03:17:01 mapr04 kernel: ieee80211 phy0: Hardware restart was requested
      Jan 5 03:17:01 mapr04 kernel: iwlwifi 0000:3a:00.0: Microcode SW error detected. Restarting 0x2000000.







      linux centos linux-kernel intel iwlwifi






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Jan 5 at 20:04







      Chris C

















      asked Jan 4 at 14:50









      Chris CChris C

      213




      213






















          1 Answer
          1






          active

          oldest

          votes


















          0














          It seems the wifi drivers cannot manage the wifi hardware in your NUCs.




          • Several linux distros can be tried live without installing. I think NUCs have Intel wifi, which should work with built-in linux drivers, but they must be new enough.


          • I have a NUC with Intel 6th generation hardware. I have noticed that older versions of operating systems cannot manage the wifi hardware, but newer versions manage it without any tweaks, 'out of the box'.


          • Edit 1: I tested with live systems: Ubuntu 18.04.1 LTS can manage both the wired and wireless hardware of my NUC6i3SYH. Debian 9, Stretch, can manage the wired network automatically. I failed with the wifi, but other people might fix it (I don't know if there is a driver problem or if I cannot manage the user interface for wifi in Debian.)

          • Edit 2: I downloaded CentOS-7-x86_64-LiveGNOME-1810.iso ran it live and it can manage both the wired and wireless hardware of my NUC6i3SYH. It was started as easily as with Ubuntu 18.04.1 LTS. But I have not tested the stability during a long time.


          Edit 3: You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.




          • When was your NUC hardware developed, and when was the CentOS 7 software developed?


          • Centos 7 has an old kernel series, 3.10; the kernel version in the live system '1810' is 3.10.0-957.el7.x86_64 #1 SMP. Ubuntu 18.04.1 live has kernel version 4.15.0-29 and an up to date installed system has 4.15.0-43.


          • Please try with another operating system with a newer linux kernel with newer hardware drivers.






          share|improve this answer


























          • You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

            – sudodus
            Jan 5 at 23:02











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "106"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492496%2fnuc-with-centos-7-crashing-microcode-sw-error-detected%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          0














          It seems the wifi drivers cannot manage the wifi hardware in your NUCs.




          • Several linux distros can be tried live without installing. I think NUCs have Intel wifi, which should work with built-in linux drivers, but they must be new enough.


          • I have a NUC with Intel 6th generation hardware. I have noticed that older versions of operating systems cannot manage the wifi hardware, but newer versions manage it without any tweaks, 'out of the box'.


          • Edit 1: I tested with live systems: Ubuntu 18.04.1 LTS can manage both the wired and wireless hardware of my NUC6i3SYH. Debian 9, Stretch, can manage the wired network automatically. I failed with the wifi, but other people might fix it (I don't know if there is a driver problem or if I cannot manage the user interface for wifi in Debian.)

          • Edit 2: I downloaded CentOS-7-x86_64-LiveGNOME-1810.iso ran it live and it can manage both the wired and wireless hardware of my NUC6i3SYH. It was started as easily as with Ubuntu 18.04.1 LTS. But I have not tested the stability during a long time.


          Edit 3: You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.




          • When was your NUC hardware developed, and when was the CentOS 7 software developed?


          • Centos 7 has an old kernel series, 3.10; the kernel version in the live system '1810' is 3.10.0-957.el7.x86_64 #1 SMP. Ubuntu 18.04.1 live has kernel version 4.15.0-29 and an up to date installed system has 4.15.0-43.


          • Please try with another operating system with a newer linux kernel with newer hardware drivers.






          share|improve this answer


























          • You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

            – sudodus
            Jan 5 at 23:02
















          0














          It seems the wifi drivers cannot manage the wifi hardware in your NUCs.




          • Several linux distros can be tried live without installing. I think NUCs have Intel wifi, which should work with built-in linux drivers, but they must be new enough.


          • I have a NUC with Intel 6th generation hardware. I have noticed that older versions of operating systems cannot manage the wifi hardware, but newer versions manage it without any tweaks, 'out of the box'.


          • Edit 1: I tested with live systems: Ubuntu 18.04.1 LTS can manage both the wired and wireless hardware of my NUC6i3SYH. Debian 9, Stretch, can manage the wired network automatically. I failed with the wifi, but other people might fix it (I don't know if there is a driver problem or if I cannot manage the user interface for wifi in Debian.)

          • Edit 2: I downloaded CentOS-7-x86_64-LiveGNOME-1810.iso ran it live and it can manage both the wired and wireless hardware of my NUC6i3SYH. It was started as easily as with Ubuntu 18.04.1 LTS. But I have not tested the stability during a long time.


          Edit 3: You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.




          • When was your NUC hardware developed, and when was the CentOS 7 software developed?


          • Centos 7 has an old kernel series, 3.10; the kernel version in the live system '1810' is 3.10.0-957.el7.x86_64 #1 SMP. Ubuntu 18.04.1 live has kernel version 4.15.0-29 and an up to date installed system has 4.15.0-43.


          • Please try with another operating system with a newer linux kernel with newer hardware drivers.






          share|improve this answer


























          • You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

            – sudodus
            Jan 5 at 23:02














          0












          0








          0







          It seems the wifi drivers cannot manage the wifi hardware in your NUCs.




          • Several linux distros can be tried live without installing. I think NUCs have Intel wifi, which should work with built-in linux drivers, but they must be new enough.


          • I have a NUC with Intel 6th generation hardware. I have noticed that older versions of operating systems cannot manage the wifi hardware, but newer versions manage it without any tweaks, 'out of the box'.


          • Edit 1: I tested with live systems: Ubuntu 18.04.1 LTS can manage both the wired and wireless hardware of my NUC6i3SYH. Debian 9, Stretch, can manage the wired network automatically. I failed with the wifi, but other people might fix it (I don't know if there is a driver problem or if I cannot manage the user interface for wifi in Debian.)

          • Edit 2: I downloaded CentOS-7-x86_64-LiveGNOME-1810.iso ran it live and it can manage both the wired and wireless hardware of my NUC6i3SYH. It was started as easily as with Ubuntu 18.04.1 LTS. But I have not tested the stability during a long time.


          Edit 3: You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.




          • When was your NUC hardware developed, and when was the CentOS 7 software developed?


          • Centos 7 has an old kernel series, 3.10; the kernel version in the live system '1810' is 3.10.0-957.el7.x86_64 #1 SMP. Ubuntu 18.04.1 live has kernel version 4.15.0-29 and an up to date installed system has 4.15.0-43.


          • Please try with another operating system with a newer linux kernel with newer hardware drivers.






          share|improve this answer















          It seems the wifi drivers cannot manage the wifi hardware in your NUCs.




          • Several linux distros can be tried live without installing. I think NUCs have Intel wifi, which should work with built-in linux drivers, but they must be new enough.


          • I have a NUC with Intel 6th generation hardware. I have noticed that older versions of operating systems cannot manage the wifi hardware, but newer versions manage it without any tweaks, 'out of the box'.


          • Edit 1: I tested with live systems: Ubuntu 18.04.1 LTS can manage both the wired and wireless hardware of my NUC6i3SYH. Debian 9, Stretch, can manage the wired network automatically. I failed with the wifi, but other people might fix it (I don't know if there is a driver problem or if I cannot manage the user interface for wifi in Debian.)

          • Edit 2: I downloaded CentOS-7-x86_64-LiveGNOME-1810.iso ran it live and it can manage both the wired and wireless hardware of my NUC6i3SYH. It was started as easily as with Ubuntu 18.04.1 LTS. But I have not tested the stability during a long time.


          Edit 3: You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.




          • When was your NUC hardware developed, and when was the CentOS 7 software developed?


          • Centos 7 has an old kernel series, 3.10; the kernel version in the live system '1810' is 3.10.0-957.el7.x86_64 #1 SMP. Ubuntu 18.04.1 live has kernel version 4.15.0-29 and an up to date installed system has 4.15.0-43.


          • Please try with another operating system with a newer linux kernel with newer hardware drivers.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Jan 6 at 0:20

























          answered Jan 5 at 21:15









          sudodussudodus

          1,24816




          1,24816













          • You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

            – sudodus
            Jan 5 at 23:02



















          • You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

            – sudodus
            Jan 5 at 23:02

















          You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

          – sudodus
          Jan 5 at 23:02





          You should consider that the hardware might be damaged (for example failing when getting hot). But if it works well with another operating system, you can conclude that the hardware is good.

          – sudodus
          Jan 5 at 23:02


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Unix & Linux Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492496%2fnuc-with-centos-7-crashing-microcode-sw-error-detected%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Morgemoulin

          Scott Moir

          Souastre