前提
环境说明
1. 操作系统: centos 7.2 2. 应用: docker
现象:
1. / 磁盘不足 2. 实际上磁盘还有很多空间 3. 应用开始报错( out of space)
分析
利用 find 分析,没有找到特殊的大文件
如:
find /apps -size +50M
利用 df 进行分析,发现 / 已经用光
[root@hh-yun-k8s-128151 /]# df -ah Filesystem Size Used Avail Use% Mounted on rootfs - - - - / sysfs 0 0 0 - /sys proc 0 0 0 - /proc devtmpfs 95G 0 95G 0% /dev securityfs 0 0 0 - /sys/kernel/security tmpfs 95G 0 95G 0% /dev/shm devpts 0 0 0 - /dev/pts tmpfs 95G 4.1G 91G 5% /run tmpfs 95G 0 95G 0% /sys/fs/cgroup cgroup 0 0 0 - /sys/fs/cgroup/systemd pstore 0 0 0 - /sys/fs/pstore cgroup 0 0 0 - /sys/fs/cgroup/cpuset cgroup 0 0 0 - /sys/fs/cgroup/blkio cgroup 0 0 0 - /sys/fs/cgroup/perf_event cgroup 0 0 0 - /sys/fs/cgroup/cpu,cpuacct cgroup 0 0 0 - /sys/fs/cgroup/hugetlb cgroup 0 0 0 - /sys/fs/cgroup/devices cgroup 0 0 0 - /sys/fs/cgroup/net_cls cgroup 0 0 0 - /sys/fs/cgroup/freezer cgroup 0 0 0 - /sys/fs/cgroup/memory configfs 0 0 0 - /sys/kernel/config /dev/sda2 80G 80G 286M 100% / systemd-1 - - - - /proc/sys/fs/binfmt_misc debugfs 0 0 0 - /sys/kernel/debug mqueue 0 0 0 - /dev/mqueue hugetlbfs 0 0 0 - /dev/hugepages /dev/sda1 397M 135M 263M 34% /boot tmpfs 19G 0 19G 0% /run/user/0 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/fa7f705d-9a82-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/32d9731a-bdfc-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-dqn5d tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/c131bddd-c367-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/27779000-c5cf-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/277f9d4b-c5cf-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/72505ec1-d570-11e6-b3fc-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b50b4c1c-d570-11e6-b3fc-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/29fa587e-ecdc-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/56e3e992-ed03-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/85342d9b-ece3-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/0e10352d-edbd-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-dqn5d tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df078f2d-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/deab6a67-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df71a3a2-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df165896-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df62fbed-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/ef590f38-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/ffe50f62-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b0cacb34-f1af-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/025dc80b-f1b9-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/de7dc25d-f272-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2 tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b07a6d58-f297-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cji2k tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b084b8a9-f297-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cji2k
利用 df 进行分析,实际上 / 只占用了少量空间 (4.4G)
[root@hh-yun-k8s-128151 /]# mount /dev/sda2 /mnt [root@hh-yun-k8s-128151 /]# du -sh /mnt/ 4.4G /mnt/
结果
分析可能性
1. df 与 du 显示不一致 2. 常见为进程正在占用文件,而运维人员直接删除文件,导致文件句柄没有释放 3. 上述第二点会导致 df 读取时候也把删除的文件的空间计算进去
获取进程占用而又被删除的文件方法
lsof | grep deleted > /tmp/files
查询结果
[root@hh-yun-k8s-128151 /]# head -n 30 /tmp/files monitor 1389 root 5w REG 8,2 588 404101751 /var/log/openvswitch/ovsdb-server.log-20160802 (deleted) monitor 1389 root 7u REG 8,2 141 1285894 /tmp/tmpfvWpBig (deleted) ovsdb-ser 1391 root 7u REG 8,2 141 1285894 /tmp/tmpfvWpBig (deleted) monitor 1438 root 3w REG 8,2 3478 404108801 /var/log/openvswitch/ovs-vswitchd.log-20160802 (deleted) docker 56894 root 12r REG 8,2 2000037 269953568 /var/lib/docker/containers/42a0c6add5d3b86037bef74d955fd968f4af853bf350ca1cb681094e0f60f816/42a0c6add5d3b86037bef74d955fd968f4af853bf350ca1cb681094e0f60f816-json.log.4 (deleted) docker 56894 root 16r REG 8,2 2000010 342880030 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 20r REG 8,2 2000089 343187052 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 22r REG 8,2 2000074 343187085 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 23r REG 8,2 2000022 342328837 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 25r REG 8,2 2000001 346233868 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 27r REG 8,2 2000117 345268401 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 30r REG 8,2 2000154 1770081 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted) docker 56894 root 31r REG 8,2 2001891 1761668 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted) docker 56894 root 32r REG 8,2 2000077 1761664 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted) docker 56894 root 33r REG 8,2 2001938 1762263 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted) docker 56894 root 34r REG 8,2 2000154 3736653 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted) docker 56894 root 39r REG 8,2 2000039 345268394 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 41r REG 8,2 2000243 404129948 /var/lib/docker/containers/a03a8e43513ea060224cf6658680157488c452d3c0096471e4a4bfd4d0e3394b/a03a8e43513ea060224cf6658680157488c452d3c0096471e4a4bfd4d0e3394b-json.log.4 (deleted) docker 56894 root 42r REG 8,2 2000044 345268392 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 43r REG 8,2 2000138 270873518 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 44r REG 8,2 2000109 342328852 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 49r REG 8,2 2003399 2935891 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted) docker 56894 root 51r REG 8,2 2000258 345268396 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 57r REG 8,2 2000024 345177666 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 58r REG 8,2 2000141 270873521 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 59r REG 8,2 2000243 346233846 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 60r REG 8,2 2000103 270873517 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 65r REG 8,2 2000009 346233853 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted) docker 56894 root 66r REG 8,2 2001743 269761873 /var/lib/docker/containers/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331-json.log.4 (deleted) docker 56894 root 67r REG 8,2 2000473 269761869 /var/lib/docker/containers/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331-json.log.4 (deleted)
总结
根据结果,我们得到下面分析:
1. 当需要清除日志时候,请使用 logrotate 等工具进行日志截断 2. 如果真的怕麻烦,那么建议使用 cat /dev/null > file.log 的方法进行文件截断 (但文件多过也会占用 inode 空间,可以利用 df -Th 方法查询 ) 3. 建议使用标准日志格式,利用 logstash 等工具进行远程备份与日志分析,避免类似麻烦