nvme 硬盘I/O读写能力下降的问题

[xutao123777]

root@ghnode:/var/log# cd /var/log
root@ghnode:/var/log# cat syslog
Dec 10 06:48:39 ghnode kernel: [7222436.263648] nvme nvme1: I/O 691 QID 32 timeout, completion polled
Dec 10 07:00:53 ghnode snapd[2715773]: storehelpers.go:710: cannot refresh: snap has no updates available: "core18", "core20", "lxd"
Dec 10 07:00:53 ghnode snapd[2715773]: autorefresh.go:536: auto-refresh: all snaps are up-to-date
Dec 10 07:17:01 ghnode CRON[2253567]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 10 07:21:31 ghnode kernel: [7224408.470712] nvme nvme3: I/O 256 QID 19 timeout, completion polled
Dec 10 07:56:00 ghnode kernel: [7226476.933010] nvme nvme1: I/O 516 QID 68 timeout, completion polled
Dec 10 07:56:33 ghnode kernel: [7226509.700739] nvme nvme4: I/O 399 QID 92 timeout, completion polled
Dec 10 07:57:05 ghnode kernel: [7226541.700429] nvme nvme1: I/O 780 QID 56 timeout, completion polled
Dec 10 07:57:46 ghnode kernel: [7226583.428105] nvme nvme0: I/O 130 QID 93 timeout, completion polled
Dec 10 08:14:32 ghnode kernel: [7227589.499523] nvme nvme0: I/O 425 QID 98 timeout, completion polled
Dec 10 08:17:01 ghnode CRON[2254017]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 10 08:41:11 ghnode kernel: [7229188.462013] nvme nvme0: I/O 375 QID 119 timeout, completion polled
Dec 10 08:44:06 ghnode kernel: [7229362.796498] nvme nvme0: I/O 827 QID 99 timeout, completion polled
Dec 10 08:48:34 ghnode kernel: [7229630.826114] nvme nvme1: I/O 497 QID 8 timeout, completion polled
Dec 10 08:54:56 ghnode systemd[1]: Started Session 2559 of user milan.
Dec 10 08:54:56 ghnode systemd[1]: Started Session 2560 of user milan.
Dec 10 09:13:39 ghnode kernel: [7231136.093326] nvme nvme3: I/O 170 QID 112 timeout, completion polled
Dec 10 09:13:44 ghnode systemd[1]: Started Session 2561 of user milan.
Dec 10 09:13:45 ghnode systemd[1]: Started Session 2562 of user milan.
Dec 10 09:14:47 ghnode systemd[1]: session-2541.scope: Succeeded.
Dec 10 09:17:01 ghnode CRON[2272322]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)



nvme nvme1: I/O 497 QID 8 timeout, completion polled 这个是什么问题, 该怎么解决

[astolia]

内核版本？出现问题前有没有休眠过机器？

[xutao123777]

cat /etc/issue
Ubuntu 20.04.3 LTS \n \l

[xutao123777]

没有休眠过 cat /proc/version
Linux version 5.4.0-84-generic (buildd@lgw01-amd64-050) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021

[astolia]

你这个有可能是写入操作较多，触发了trim操作。
看下你的/etc/fstab中那个分区的挂载参数，如果没有discard，就加上去。或者你时不时手动运行fstrim一下。