Tech Tip du Jour: CPU load

Today I'm going to take a brief step outside the Informix box and talk about sar and vmstat and one very particular issue I've seen: if sar or vmstat (or whatever) reports that you have less than an order of magnitude more %user than %sys, you have got a problem.

Let me illustrate:


informix@freebnort:/home/informix
pts/4:ksh$ sar 5 12
Linux 2.6.16.42-0.10-smp (freebnort) 07/08/08

22:25:12 CPU %user %nice %sys %iowait %idle
22:25:17 all 10.66 0.00 12.52 0.00 76.83
22:25:22 all 9.13 0.00 12.30 0.00 78.57
22:25:27 all 12.04 0.00 12.76 0.00 75.20
22:25:32 all 11.46 0.00 12.69 0.00 75.85
22:25:37 all 10.14 0.00 12.69 0.00 77.17
22:25:42 all 12.81 0.00 13.17 0.00 74.02
22:25:47 all 12.15 0.00 12.71 0.00 75.14
22:25:52 all 13.76 0.00 13.10 0.00 73.14
22:25:57 all 21.79 0.00 12.89 0.00 65.32
22:26:02 all 14.84 0.00 12.64 0.00 72.53
22:26:07 all 12.11 0.00 12.47 0.00 75.42
22:26:12 all 13.43 0.00 13.18 0.00 73.38
Average: all 12.86 0.00 12.76 0.00 74.38

informix@freebnort:/home/informix
pts/4:ksh$


(Pardon the formatting, blogger is doing my head in on this.)

As you can see, the %sys is the same order of magnitude as the %user. Given that this server is only running IDS, something is wrong. The %user should be at least an order of magnitude greater than than %sys. In general, if a database server is only a database server, then if %user is 10%, then %sys should be about 1%. If the server has a lot of connections, this might get a bit higher, but %sys shouldn't generally be more than about 3% if %user is about 10%.

Things to watch out for are:

1. Swapping, especially if %sys is significantly greater than %user and the server doesn't seem to be doing anything.
2. Very high rates of network activity, such as lots of connect / disconnect activity; or poorly configured connectivity, leading to excessive transmission of small packets, rather than fewer transmissions of larger packets.

Obviously, there could be many other causes, but these are the most common.