DevOps / Sys Admin Q & A #15 : Why is the Server slow?
Tracking down the sluggish/unresponsive host server issue could be very tricky. That's because any networking problems often create the same set of symptoms as caused by the server itself.
When a machine is slow, it is often because we have consumed a particular resource on the system.
The main resources are CPU, RAM, disk I/O, and network.
System load average is probably the fundamental metric we can use when troubleshooting a sluggish system.
One of the first commands is uptime:
$ uptime 21:21:09 up 2 days, 11:23, 3 users, load average: 0.92, 1.33, 1.30
The three numbers after load average - 0.92, 1.33, and 1.30 - represent the 1-, 5-, and 15-minute load averages on the machine, respectively.
A system load average is equal to the average number of processes in a runnable or uninterruptible state.
Runnable processes are either currently using the CPU or waiting to do so, and uninterruptible processes are waiting for I/O.
Picture info DevOps / Sys Admin Q & A #10 : Trouble Shooting
Here are the metrics for each type of loads:
- CPU-bound (processes waiting on CPU resources)
- RAM-bound (specifically, high RAM usage that has moved into swap)
- I/O-bound (processes fighting for disk or network I/O)
Typically systems seem to be more responsive when under CPU-bound load than when under I/O-bound load.
Even though a system is CPU-bound, and it could still run diagnostic tools on the system with pretty good response times.
On the other hand, a system with relatively low I/O-bound loads on which just logging in took a while because the disk I/O was completely saturated.
A system that runs out of RAM resources often appears to have I/O-bound load, since once the system starts using swap storage on the disk, it can consume disk resources and cause a downward spiral as processes slow to a halt.
The basic steps are to examine the top output to identify what resources we're running out of (CPU, RAM, disk I/O).
Once we have figured that out, we can try to identify what processes are consuming those resources the most.
- High User CPU Time - If we see high user CPU time but low I/O wait times, we simply need to identify which processes on the system are consuming the most CPU.
- High I/O Wait - When we see high I/O wait, one of the first things we should check is whether the machine is using a lot of swap. If we have plenty of RAM, we will need to figure out which program is consuming the most I/O.
It can sometimes be difficult to figure out exactly which process is using the I/O, but if we have multiple partitions on our system, we can narrow it down by figuring out which partition most of the I/O is on.
To do this, we can use iostat program, which is provided by the sysstat:
$ iostat -p sda Linux 3.13.0-40-generic (laptop) 11/30/2015 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 29.46 0.40 8.94 1.38 0.00 59.82 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 10.46 87.83 69.78 19248132 15291117 sda1 0.01 0.04 0.00 9751 1 sda2 8.91 64.39 49.27 14109557 10797156 sda3 1.24 23.37 20.51 5120684 4493960When we have a system under heavy I/O load, the first step is to look at each of the partitions and identify which partition is getting the heaviest I/O load.
In addition to iostat, we have a much simpler tool available in newer distributions called iotop. In effect, it is a blend of top and iostat in that it shows us all of the running processes on the system sorted by their I/O statistics.
For more on trouble shooting tools, check DevOps / Sys Admin Q & A (10) - Trouble Shooting: Load, Throughput, Response time and Leaks and DevOps / Sys Admin Q & A (9) - Linux System / Application Monitoring, Performance Tuning, Profiling Methods & Tools.
DevOps / Sys Admin Q & A
Linux - system, cmds & shell programming
- Linux Tips - links, vmstats, rsync
- Linux Tips 2 - ctrl a, curl r, tail -f, umask
- Linux - bash I
- Linux - bash II
- Linux - Uncompressing 7z file
- Linux - sed I (substitution: sed 's///', sed -i)
- Linux - sed II (file spacing, numbering, text conversion and substitution)
- Linux - sed III (selective printing of certain lines, selective definition of certain lines)
- Linux - 7 File types : Regular, Directory, Block file, Character device file, Pipe file, Symbolic link file, and Socket file
- Linux shell programming - introduction
- Linux shell programming - variables (readonly, unset)
- Linux shell programming - special shell variables
- Linux shell programming : arrays - three different ways of declaring arrays & looping with $*/$@
- Linux shell programming : operations on array
- Linux shell programming : variables & commands substitution
- Linux shell programming : metacharacters & quotes
- Linux shell programming : input/output redirection & here document
- Linux shell programming : loop control - for, while, break, and break n
- Linux shell programming : for-loop
- Linux shell programming : if/elif/else/fi
- Managing User Account - useradd, usermod, and userdel
- Linux Secure Shell (SSH) I : key generation, private key and public key
- Linux Secure Shell (SSH) II : ssh-agent & scp
- Linux Secure Shell (SSH) III : SSH Tunnel as Proxy - Dynamic Port Forwarding (SOCKS Proxy)
- Linux Secure Shell (SSH) IV : Local port forwarding (outgoing ssh tunnel)
- Linux Secure Shell (SSH) V : Reverse SSH Tunnel (remote port forwarding / incoming ssh tunnel) /)
- Linux Processes and Signals
- Linux Drivers 1
- Linux Debugging using gdb
- Embedded Systems Programming I - Introduction
- Embedded Systems Programming II - gcc ARM Toolchain and Simple Code on Ubuntu/Fedora
- LXC (Linux Container) Install and Run
- Hadoop - 1. Setting up on Ubuntu for Single-Node Cluster
- Hadoop - 2. Runing on Ubuntu for Single-Node Cluster
- ownCloud 7 install
- Ubuntu 14.04 guest on Mac OSX host using VirtualBox I
- Ubuntu 14.04 guest on Mac OSX host using VirtualBox II
- Windows 8 guest on Mac OSX host using VirtualBox I
- Ubuntu Package Management System (apt-get vs dpkg)
- How to Make a Self-Signed SSL Certificate
- DevOps / Sys Admin interview questions
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization