Linux Performance [High Load Alert] Troubleshooting issues on servers (TOOLS to use)

What do you do when you get an alert that your system load is high? Tracking down the cause of high load just takes some time, some experience and a few Linux tools.

CPU LOAD-

1. uptime (1,5,10 minuits CPU loads)
2. top
   

   Cpu(s): 11.4%us, 29.6%sy, 0.0%ni, 58.3%id, .7%wa, 0.0%hi, 0.0%si, 0.0%st  

       
   us: user CPU time. More often than not, when you have CPU-bound load, it's due to a process run by a user on the system, such as Apache, MySQL or maybe a shell script. If this percentage is high, a user process such as those is a likely cause of the load.
       
   sy: system CPU time. The system CPU time is the percentage of the CPU tied up by kernel and other system processes. CPU-bound load should manifest either as a high percentage of user or high system CPU time.
       
   id: CPU idle time. This is the percentage of the time that the CPU spends idle. The higher the number here the better! In fact, if you see really high CPU idle time, it's a good indication that any high load is not CPU-bound.
       

• wa: I/O wait. The I/O wait value tells the percentage of time the CPU is spending waiting on I/O (typically disk I/O). If you have high load and this value is high, it's likely the load is not CPU-bound but is due to either RAM issues or high disk I/O.

A little below it tells you which process is hogging the CPU usage.  TOP by default sorts based on cpu usage so the high cpu consuming processes on top.

**NOTE__?: There are instances where an applications spawning up multiple threads on a single CPU server causing the server to have lot of wait cycles and high CPU load average usage.

1. iostat
   
   
   

Memory LOAD

Check SWAP memory usage

Once all the memory is used up the Swap space is used; usually on a hard drive and is much slower than RAM. Causing processes that load from swap to slow down dramatically. This is a downward spiral causing more wait for other processes and slowing the system to its crawling state.  Its easy to mis diagnose swap issues is high disk I/O.

After all, if your disk is being used as RAM, any processes that actually want to access files on the disk are going to have to wait in line. So, if I see high I/O wait in the CPU row in top, I check RAM next and rule it out before I troubleshoot any other I/O issues.

    Mem: 1024176k total, 997408k used, 26768k free, 85520k buffers  Swap: 1004052k total, 4360k used, 999692k free, 286040k cached  

.This tells us who much swap memory is used and how much is free

1. more to come....

HIGH DISK I/O Bound LOAD

1. more to come.....

hxxp:<slash><slash>www.linuxjournal.com<slash>magazine<slash>hack-and-linux-troubleshooting-part-i-high-load