The following demonstrates running the cputimes program on an idle system. We use an interval of 1 second and a count of 3, # ./cputimes 1 3 2005 Apr 27 23:37:58, THREADS TIME (ns) KERNEL 10795499 PROCESS 20941091 IDLE 970707443 2005 Apr 27 23:37:59, THREADS TIME (ns) KERNEL 8919418 PROCESS 77446789 IDLE 910555040 2005 Apr 27 23:38:00, THREADS TIME (ns) KERNEL 8615123 PROCESS 78314246 IDLE 810100417 In the above output, we can see a breakdown of CPU time into the catagories KERNEL, PROCESS and IDLE. The time is measured in nanoseconds. Most of the time is in the IDLE category, as the system is idle. Very little time was spent serving the kernel. In the following example, several programs are run to hog the CPUs, # ./cputimes 1 3 2005 Apr 27 23:40:58, THREADS TIME (ns) KERNEL 11398807 PROCESS 992254664 2005 Apr 27 23:40:59, THREADS TIME (ns) KERNEL 9205260 PROCESS 987561182 2005 Apr 27 23:41:00, THREADS TIME (ns) KERNEL 9196669 PROCESS 877850474 Now there is no IDLE category, as the system is 100% utilised. The programs were the following, while :; do :; done & which keeps the CPU busy. In the following example a different style of program is run to hog the CPUs, while :; do date; done This causes many processes to be created and destroyed in a hurry, and can be difficult to troubleshoot (tools like prstat cannot sample quick enough to easily identify what is going on). The following is the cputimes output, # ./cputimes 1 3 2005 Apr 27 23:45:30, THREADS TIME (ns) KERNEL 192647392 PROCESS 835397568 2005 Apr 27 23:45:31, THREADS TIME (ns) KERNEL 168773713 PROCESS 810825730 2005 Apr 27 23:45:32, THREADS TIME (ns) KERNEL 151676122 PROCESS 728477272 Now the kernel is doing a substantial amount of work to create and destroy these processes. In the following example, a large amount of network activity occurs while cputimes is running, # ./cputimes 1 6 2005 Apr 27 23:49:29, THREADS TIME (ns) KERNEL 10596399 PROCESS 21793920 IDLE 974395713 2005 Apr 27 23:49:30, THREADS TIME (ns) KERNEL 251465759 IDLE 357436576 PROCESS 508986422 2005 Apr 27 23:49:31, THREADS TIME (ns) IDLE 9758227 KERNEL 367645318 PROCESS 385427847 2005 Apr 27 23:49:32, THREADS TIME (ns) IDLE 28351679 KERNEL 436022725 PROCESS 451304688 2005 Apr 27 23:49:33, THREADS TIME (ns) KERNEL 262586158 PROCESS 325238896 IDLE 358243503 2005 Apr 27 23:49:34, THREADS TIME (ns) KERNEL 10075578 PROCESS 238170506 IDLE 647956998 Initially the system is idle. A command is run to cause heavy network activity, which peaks during the fourth sample - during which the kernel is using around 40% of the CPU. The Solaris 10 command "intrstat" can help to analyse this activity further. Longer samples are possible. The following is a 60 second sample, # ./cputimes 60 1 2005 Apr 27 23:53:02, THREADS TIME (ns) KERNEL 689808449 PROCESS 8529562214 IDLE 50406951876 # cputimes has a "-a" option to print all processes. The following is a single 1 second sample with -a, # ./cputimes -a 1 1 2005 Apr 28 00:00:32, THREADS TIME (ns) svc.startd 51042 nautilus 130645 in.routed 131823 fmd 152822 nscd 307042 dsdm 415799 mixer_applet2 551066 gnome-smproxy 587234 xscreensaver 672270 fsflush 1060196 java_vm 1552988 wnck-applet 2060870 dtrace 2398658 acroread 2614687 soffice.bin 2825117 mozilla-bin 5497488 KERNEL 13541120 metacity 28924204 gnome-terminal 74304348 Xorg 289631407 IDLE 465054209 The times are in nanoseconds, and multiple processes with the same name have their times aggregated. The above output is at an amazing resolution - svc.startd ran for 51 microseconds, and soffice.bin ran for 28 milliseconds. The following is a 10 second sample on an idle desktop, # ./cputimes -a 10 1 2005 Apr 28 00:03:57, THREADS TIME (ns) snmpd 127859 fmd 171897 inetd 177134 svc.configd 185006 mapping-daemon 197674 miniserv.pl 305603 gconfd-2 330511 xscreensaver 443207 sendmail 473434 nautilus 506799 gnome-vfs-daemon 549037 gnome-panel 770631 nscd 885353 svc.startd 1181286 gnome-netstatus- 4329671 mixer_applet2 4833519 dtrace 6244366 in.routed 6556075 fsflush 9553155 soffice.bin 13954327 java_vm 16285243 acroread 32126193 gnome-terminal 34891991 Xorg 35553412 mozilla-bin 67855629 KERNEL 94834997 IDLE 9540941846 Wow, maybe not as idle as I thought!