A Sysadmin’s Guide to Linux Performance Monitoring from the Command Line
When a Linux server slows down, every second counts. While graphical user interfaces have their place, the command line remains the most powerful and direct way to diagnose performance bottlenecks. For system administrators, developers, and power users, mastering a core set of command-line tools is not just a valuable skill—it’s essential for maintaining system health and resolving issues quickly.
These tools provide unfiltered, real-time access to the inner workings of your system, from CPU and memory utilization to disk I/O and network traffic. Let’s explore the most critical command-line utilities you should have in your arsenal, categorized by their primary function.
The All-Rounders: High-Level System Overview
Before diving deep, you need a bird’s-eye view of your system’s health. These tools provide a comprehensive snapshot of what’s happening right now.
top: The undisputed classic. Present on virtually every Unix-like system,topprovides a dynamic, real-time view of running processes. It displays a summary of system uptime, load average, CPU usage, and memory status, along with a list of the most resource-intensive processes. While basic, its universal availability makes it an indispensable first step in any investigation.htop: The modern, colorful successor totop.htopis a significant upgrade, offering a more intuitive and interactive experience. It allows you to scroll vertically and horizontally to see all processes and their full command lines. You can also kill processes, change their priority, and filter the list directly from the interface without typing commands. If it’s not installed by default, it’s worth the quickapt install htoporyum install htop.glances: Ifhtopis an upgrade,glancesis a full-featured dashboard in your terminal. Written in Python, it presents an enormous amount of information in a single, well-organized screen.glancesprovides at-a-glance data on CPU, memory, load, disk I/O, network interfaces, and running processes. It also uses color-coded alerts to warn you of potential issues, making it excellent for continuous monitoring.
Deep Dive into CPU Performance
When the load average is high, you need to pinpoint what’s consuming your CPU cycles.
mpstat: Part of thesysstatpackage,mpstatis crucial for multi-core systems. Whiletopshows overall CPU usage,mpstatbreaks down CPU utilization on a per-processor or per-core basis. This helps you identify imbalances, such as a single-threaded application maxing out one core while others sit idle. Usempstat -P ALL 1to see a continuous, one-second update for all cores.sar: The System Activity Reporter,sar, is a powerful tool for historical analysis. It collects and archives performance data over time, allowing you to investigate issues that occurred in the past. Unlike real-time tools,saris perfect for tracking trends and identifying patterns in CPU usage, memory, and I/O over hours or days. For example,sar -uwill show historical CPU usage data.
Understanding Memory Usage
Out-of-memory errors can bring a system to its knees. These tools help you see how your RAM is being used.
free: The simplest and quickest way to check system memory. Runningfree -h(-hfor human-readable) provides a clear summary of total, used, and free memory. Crucially, it also shows thebuff/cachememory, which is RAM used by the kernel for disk caching. A highbuff/cachevalue is normal and healthy; the kernel will release this memory if applications need it.vmstat: The Virtual Memory Statistics reporter gives you a broader view of system performance. Runningvmstat 1provides a one-second rolling update on processes, memory (including swap activity), block I/O, and CPU activity. Its key strength is showing the interaction between different subsystems. High numbers in thesi(swap-in) andso(swap-out) columns are a clear indicator of memory pressure.
Monitoring Disk I/O Bottlenecks
A fast CPU and ample RAM are useless if your storage can’t keep up. Disk I/O is a common and often overlooked bottleneck.
iostat: This command provides detailed statistics for your storage devices. It shows transfers per second (tps), kilobytes read/written per second, and average wait times. A high%iowaitvalue in its CPU report indicates the processor is spending significant time waiting for disk operations to complete—a sure sign of a storage bottleneck.iotop: Just ashtopshows which processes are using the most CPU,iotopshows which processes are responsible for the most disk I/O. This is incredibly useful for finding a rogue process that is constantly reading from or writing to the disk. It requires root privileges to run and provides a real-time, process-by-process breakdown of disk activity.
Keeping an Eye on Network Traffic
For web servers, databases, and any networked application, understanding network traffic is vital.
ss: The modern replacement for the classicnetstatcommand. Thesstool is faster and provides more detailed information about network sockets. You can use it to list all open TCP, UDP, and UNIX sockets and the processes using them. A command likess -tulpnis a great way to see all listening ports and their associated services.iptraf-ng: An interactive, full-screen network monitoring tool.iptraf-ngprovides a real-time view of network traffic passing through your interfaces. You can see detailed statistics on traffic by protocol (TCP, UDP, ICMP), packet size, and port. It’s an excellent tool for visualizing network flow and diagnosing connectivity or bandwidth issues on the fly.
A Practical Monitoring Workflow
Knowing the tools is one thing; using them effectively is another. Here is a simple, actionable workflow for diagnosing a slow system:
- Start Broad: Begin with
htoporglancesfor a high-level overview. Check the load average, CPU usage, and memory status. - Identify the Constraint:
- High CPU? Use
mpstat -P ALL 1to check for core imbalances. Identify the offending process inhtop. - High Memory Usage? Use
free -hto check available memory. Runvmstat 1to see if the system is actively swapping. - System Laggy but CPU/Memory OK? It’s likely I/O. Run
iostat -x 1to check for high%iowaitor busy devices. Then, useiotopto find the process causing the disk activity.
- High CPU? Use
- Investigate Network Issues: If the problem seems network-related, use
ss -sfor a connection summary andiptraf-ngto watch traffic in real time.
By mastering these essential command-line utilities, you can move from guessing to knowing, transforming yourself into a more efficient and effective system administrator capable of diagnosing and resolving performance issues with confidence.
Source: https://www.tecmint.com/command-line-tools-to-monitor-linux-performance/
