Advanced Tips to Improve Disk IO Performance in Linux

In today's data-driven world, efficient data handling and storage are paramount. If you've ever faced lags, long wait times, or system freezes, there's a good chance that the bottleneck could be related to Disk IO performance.

 

What exactly is Disk I/O?

Disk Input/Output, commonly referred to as Disk IO, pertains to the activities that involve reading data from or writing data to a storage medium, usually a hard drive or SSD, within a computing environment. Especially in Linux, a preferred choice for many servers and data centers, optimizing Disk IO can lead to significant performance enhancements. This article aims to guide users on how to improve disk IO performance in Linux, ensuring swift data transactions and a smoother computing experience.

Below flow chart gives you an overview of how IO request flows between User Space and Device.

In this sequence diagram:

1. A user initiates an IO request via an application in User Space.
2. The application makes the necessary system call for the IO operation.
3. The System Call Interface (SCI) in the operating system passes the IO request to the Kernel Space.
4. Within the Kernel Space, the IO Scheduler and the File System (FS) process the request and interact with the Device to execute the IO operations.

 

Understand Basic Concepts: Disk IO, Throughput, Latency

Disk IO: This refers to the input/output operations on a physical or virtual disk in a system. In simple terms, it's how data is read from or written to storage devices. Improving disk I/O performance means making these read and write operations faster and more efficient, which can lead to quicker data access, faster application response times, and better overall system performance.

Throughput: his is the amount of data that can be transferred to or from the disk in a specific time frame, often expressed in MB/s or GB/s. When you improve disk IO performance, throughput typically increases, meaning more data can be processed in less time.

Latency: Latency represents the time it takes for a single I/O request to be completed, from the moment it's issued to the disk until it's fulfilled. It's usually measured in milliseconds. Low latency is desired, as it means that individual read/write requests are completed quickly. When you focus on methods to improve disk IO performance, reducing latency is often a primary goal.

 

Tools to Monitor Disk IO

iostat: This tool provides statistics for I/O operations, showcasing read/write operations per second, throughput, and latency, among other metrics. By analyzing the data from iostat, you can pinpoint areas that need tweaking to improve disk IO performance.

iostat -xz 5

This command will display extended statistics (-x) every 5 seconds, providing a real-time view of how your storage devices are performing.

vmstat: While vmstat is often associated with virtual memory statistics, it also provides valuable information about block I/O, allowing you to see how often the system is reading from or writing to disk. Observing this can help in formulating strategies to improve disk IO performance.

vmstat 5

This will show a summary of system resources including disk I/O every 5 seconds.

iotop: This tool offers a top-like interface to monitor real-time disk I/O and identify processes that are the most intensive in terms of disk usage. By understanding which processes are heavy on I/O, you can make targeted optimizations to improve disk IO performance.

$ sudo iotop

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
...

This will display a real-time dashboard of processes and their disk I/O activities.

 

Common Causes of Poor Disk IO Performance

Hardware Limitations:

• Drive Speed: Traditional hard disk drives (HDDs) use spinning platters, and the speed at which these platters spin (often measured in RPM) can influence the speed of data access. A 5400 RPM drive is typically slower than a 7200 RPM drive. Use hdparm to test drive read speeds: hdparm -Tt /dev/sda
• Drive Type: Solid-state drives (SSDs) are typically much faster than HDDs due to their lack of moving parts. However, not all SSDs are created equal; differences in technology like NVMe vs. SATA can significantly impact and also improve Disk IO performance. For SSDs, check wear leveling count and other attributes with smartctl: smartctl -a /dev/sda
• Interface Bottlenecks: The connection between the drive and the motherboard, such as SATA or NVMe, can become a bottleneck if the drive is capable of faster speeds than the interface allows. lspci and dmesg can be used to view hardware interfaces and check if they are operating at the expected speeds or if there are any reported issues.
• Age and Wear: Drives, especially SSDs, have a limited number of write cycles. As a drive ages or if it undergoes an excessive number of writes, its performance can degrade.

Software Misconfigurations:

• File System: Using an inappropriate or outdated file system can impact performance. For example, using FAT32 for large drives or not using a journaling filesystem where it's beneficial can hinder speed. Use df -T to check which file systems are in use and use fsck (e.g., fsck.ext4) to check for file system errors.
• Incorrect Drive Settings: Features like write caching or AHCI mode can affect performance. If they're misconfigured, it can lead to sub-optimal drive speeds. Verify write caching settings with hdparm: hdparm -W /dev/sda. Check AHCI mode settings in BIOS/UEFI or inspect kernel logs with dmesg | grep -i ahci.
• Disk Fragmentation: Particularly a concern with HDDs. When files are fragmented, the disk needs to work harder to gather all the parts of a file, slowing down read/write operations.

System Bottlenecks:

• CPU Limitations: If the CPU is consistently at high usage, it might not be able to handle I/O requests promptly. Use top, htop, or glances to monitor CPU usage.
• Memory Bottlenecks: Insufficient RAM can lead to excessive use of swap space on the disk, leading to increased I/O operations and reduced performance. Monitor RAM and swap usage with commands like free -m or tools like vmstat. Examine swap activity with swapon -s or by checking the /proc/swaps file.
• Bus Congestion: If many devices are communicating over the same bus, it can limit the bandwidth available for each device, slowing down disk operations. Use lspci to view devices connected through PCI. Check dmesg for any reported issues.

 

DISK I/O Optimization Techniques to Improve Performance

1. Basic Optimizations

1.1 Filesystem Choice:

• Different file systems have varied performance characteristics and are suited for different workloads.
• ext4: A widely-used journaling file system. It provides good performance for a wide range of workloads. Example: mkfs.ext4 /dev/sda1
• XFS: It's known for high performance with large files and filesystems and can help improve Disk IO Performance. Great for media storage or databases. Example: mkfs.xfs /dev/sda1
• Btrfs: A modern filesystem that offers advanced features like snapshots, RAID configurations, and more. However, it may have performance implications in certain scenarios. Example: mkfs.btrfs /dev/sda1

1.2 Proper Partition Alignment:

• Especially crucial for SSDs, aligning partitions correctly ensures that block and page boundaries align with the underlying storage's physical architecture.
• Most modern partition tools like gdisk and parted will align partitions correctly by default. If unsure, you can check alignment with parted /dev/sda align-check optimal 1, where 1 is the partition number.

1.3 Mount Options and Tuning:

• noatime: Disables the updating of the inode access time, reducing write operations.
• nodiratime: Similar to noatime but specifically for directories. Often redundant if noatime is set.
• barrier: Controls whether write barriers are enabled (useful for protecting data integrity but can have a performance cost). Example: To mount with specific options, modify /etc/fstab:

/dev/sda1  /  ext4  defaults,noatime,nodiratime  0  1

1.4 Using SSDs and Enabling TRIM:

• SSDs offer faster access times and better overall performance compared to HDDs. If you haven't already, consider migrating your system's core partitions (like the root partition) to an SSD as it can help improve Disk IO Performance.
• TRIM: Helps SSDs manage unused blocks, maintaining their performance over time.
  • Ensure your SSD supports TRIM with lsblk --discard /dev/sda
  • Enable periodic TRIM by enabling the fstrim.timer service:
    systemctl enable fstrim.timer && systemctl start fstrim.timer

 

2. Advanced Optimizations

2.1 Logical Volume Management (LVM):

• Overview: LVM allows for a more flexible disk management scheme than traditional partitioning. It abstracts the storage, enabling dynamic resizing of disk space and easy addition of new disks. It may not directly improve Disk IO Performance but it has it's own advantages.
• Benefits:
  • Flexibility: Easily resize logical volumes.
  • Snapshots: Create quick backups of the current state of a volume.
  • Span Multiple Disks: Create a logical volume that spans multiple physical devices.
  • Disk Replacement: Can replace a disk without downtime.
• Example Commands:
  • Create a physical volume: pvcreate /dev/sdb
  • Create a volume group: vgcreate myVolumeGroup /dev/sdb
  • Create a logical volume: lvcreate -L 10G -n myLogicalVolume myVolumeGroup

2.2 RAID Configurations:

• RAID 0 (Striping): Distributes data evenly across two or more disks without parity information. Increases performance but doesn't offer redundancy.
• RAID 1 (Mirroring): Duplicates the same data on two or more disks. Offers redundancy at the cost of space.
• RAID 5 (Distributed Parity): Distributes data and parity information across three or more disks. Offers a balance between performance and redundancy.
• RAID 10 (1+0): Combines RAID 1 and RAID 0. Provides redundancy and performance.
• Example Commands: Using mdadm for RAID:
  • Create a RAID 1 array:
  • mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1

2.3 Disk Caching Tools:

• EnhanceIO: Uses SSDs as cache for traditional hard drives. Allows for read and write caching.
• bcache: Transforms an SSD into a cache for another drive. Requires formatting and setting up the SSD as a caching device.
• dm-cache: A component of the Linux device mapper, it also uses faster devices as cache for slower devices.
• Example Commands:
  • For bcache:
    • Set up SSD as a cache: make-bcache -C /dev/sda
    • Attach a backing device: make-bcache -B /dev/sdb

2.4 IO Schedulers:

I/O schedulers play a critical role in system and to improve Disk IO performance. They determine the order and the manner in which I/O operations are passed to the underlying storage devices. Different schedulers have different algorithms, optimized for specific workloads and devices. Let's delve deeper:

CFQ (Completely Fair Queuing):

• Description: CFQ maintains a separate I/O queue for each process and provides I/O bandwidth in a round-robin manner. By offering a balanced I/O bandwidth allocation, it can improve disk IO performance, particularly in multitasking environments.
• Best For: General-purpose workloads on rotational HDDs.
• Recommendation: While CFQ used to be the default scheduler for many Linux distributions aiming to improve disk IO performance, it might not be the best choice for SSDs or high-performance storage.

noop:

• Description: The noop scheduler is straightforward, making no efforts to reorder or schedule requests. It queues each new I/O request sequentially. This simplicity can improve disk IO performance on devices with their own sophisticated I/O mechanisms.
• Best For: Devices that have their own sophisticated I/O handling or reordering mechanisms, such as SSDs or SAN storage.
• Recommendation: For storage systems that natively improve disk IO performance, such as certain SSDs, using noop can reduce unnecessary CPU overhead.

deadline:

• Description: The deadline scheduler prioritizes I/O request deadlines to enhance consistency and reduce I/O latency, aiming to improve disk IO performance. With separate read and write queues, it prevents I/O request starvation.
• Best For: Database workloads or scenarios where consistent I/O latency is crucial.
• Recommendation: If you're looking to improve disk IO performance and you face I/O request delays with CFQ or noop, the deadline scheduler might be the solution.

mq-deadline:

• Description: As a multiqueue version of the deadline scheduler, mq-deadline was introduced to accommodate modern hardware like NVMe drives. By catering to multi-queue capable devices, it seeks to further improve disk IO performance.
• Best For: Modern NVMe drives.
• Recommendation: If your goal is to improve disk IO performance on NVMe drives, mq-deadline should be your primary consideration.

BFQ (Budget Fair Queuing):

• Description: BFQ allocates specific time budgets for tasks, processing I/O requests within these limits. This design strikes a balance to improve disk IO performance, ensuring low latency for interactive tasks while maintaining higher throughput for background operations
• Best For: Desktop systems or interactive workloads that need responsive performance.
• Recommendation: If you prioritize system responsiveness in your efforts to improve disk IO performance, BFQ might be your optimal choice.

Kyber:

• Description: Kyber maintains separate FIFO queues for read, write, and other requests, aiming to achieve a balance between I/O latency and throughput.
• Best For: High-performance storage devices like NVMe SSDs.
• Recommendation: For systems with fast storage and a mix of foreground and background tasks, Kyber can be a good choice.

2.5 How to check and change current I/O Scheduler

You can check the current I/O scheduler for a specific device, for example sda, with the following command:

cat /sys/block/sda/queue/scheduler

The output might look something like this

[mq-deadline] kyber bfq none

The name inside the brackets [...] indicates the currently active scheduler. On newer kernel version (probably 5.x or later) it is possible you may see only multi-queue block layer:

none [mq-deadline]

In such configurations, traditional I/O schedulers like cfq, noop, and deadline are not available. Instead, multi-queue schedulers like mq-deadline and others (like bfq if compiled with the kernel) are used.

If you need a different scheduler, such as bfq, you'd typically need to ensure your kernel has that scheduler compiled in. Some distributions might provide it by default, while others might require a custom kernel or a specific kernel module.

To switch between multi-queue schedulers (assuming more than one is available):

Set Scheduler Temporarily to validate the performance (This change will be lost after a system reboot.):

echo bfq | sudo tee /sys/block/sda/queue/scheduler

To set the I/O scheduler permanently (so it persists across reboots), you'll need to modify the kernel boot parameters:

Open the GRUB configuration file:

sudo vim /etc/default/grub

On Debian based OS find the line starting with GRUB_CMDLINE_LINUX_DEFAULT and on RHEL based OS look for GRUB_CMDLINE_LINUX. This line contains the default kernel boot parameters. Append the desired I/O scheduler setting for your device. For example, to set the sda device to use the noop scheduler, add elevator=noop. The line might look something like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash elevator=noop" # On Debian based OS
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap elevator=noop" # On RHEL based OS

Save and close the file. Update GRUB with the new configuration:

sudo update-grub # On Ubuntu based distro
sudo grub2-mkconfig -o /boot/grub2/grub.cfg # On RHEL based distro for BIOS systems
sudo grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg  # On RHEL based distro for UEFI systems

NOTE: Replace centos with redhat or appropriate name if you're on RHEL or another RHEL-based distribution.

Finally, reboot your machine for the changes to take effect:

sudo reboot

 

Tuning the Linux Kernel

1. Optimize sysctl to Improve Disk IO Performance

sysctl is a utility to modify kernel parameters at runtime, and some of its settings can directly influence disk I/O performance:

• vm.dirty_ratio: It defines the percentage of total system memory that can contain dirty pages before a process is forced to write these changes to disk. By optimizing this value, you can control how long dirty pages remain in memory before being flushed to disk.
• vm.dirty_background_ratio: This parameter determines the percentage of total system memory containing dirty pages at which the pdflush/flush/kdmflush background processes begin writing these pages to disk. Adjusting this can optimize when the system starts the writeback process.

Example for Improving Disk I/O Performance:

To potentially improve disk IO performance, you might want to adjust these values so that dirty pages are written to disk less frequently:

sudo sysctl -w vm.dirty_ratio=60
sudo sysctl -w vm.dirty_background_ratio=20

Persist these settings across reboots by adding them to /etc/sysctl.conf:

echo "vm.dirty_ratio=60" | sudo tee -a /etc/sysctl.conf
echo "vm.dirty_background_ratio=20" | sudo tee -a /etc/sysctl.conf

Apply the changes

sysctl --system

By delaying the write operations, you can sometimes achieve better I/O throughput, but the trade-off is the potential for increased data loss in case of a system crash.

 

2. Use ulimit for Improve Disk IO Performance

ulimit sets user-level limits, which can indirectly improve disk IO performance, especially when dealing with operations that require a lot of file handles or processes:

• -n (open file descriptors): If an application is I/O-intensive, it might require opening many files simultaneously. Increasing the limit can improve its performance.

Example for Improving Disk I/O Performance:

First, check the current limit of open file descriptors:

ulimit -n

To potentially improve disk IO performance for applications requiring many simultaneous file operations, increase the file descriptor limit:

ulimit -n 4096

To make this change permanent for a user, you can add it to their .bashrc or .bash_profile:

echo "ulimit -n 4096" >> ~/.bashrc

For a system-wide change affecting all users, adjust the settings in /etc/security/limits.conf.

 

Improve Disk IO Performance in Virtual Environments

Each virtualization platform, whether it's KVM, VMware, Hyper-V, or others, offers its own set of tools and best practices for I/O optimization.

KVM:

Cache Settings: In KVM, when you create a virtual disk for a VM, you can specify cache settings. To improve disk IO performance, setting the cache to none can sometimes be beneficial, especially on hosts with battery-backed write caches. This avoids double caching.

qemu-img create -f qcow2 -o cache=none mydisk.img 10G

VMware:

Paravirtualized SCSI Controller: For I/O-intensive applications, VMware recommends using the VMware Paravirtual SCSI controller. It's designed to improve disk IO performance by offloading I/O processing to the guest operating system.

Example: Within the VM's settings in vSphere, you can change the SCSI controller type to "VMware Paravirtual."

Virtio Drivers and Benefits:

virtio is a paravirtualization standard for network and disk device drivers. It's optimized for performance, delivering data directly from the guest VM to the physical system, bypassing the host's emulation layer.

Benefits:

• Reduced I/O Latency: By bypassing the emulation layer, I/O operations are faster, which can significantly "improve disk IO performance."
• Efficient Use of Resources: virtio drivers typically use fewer host resources compared to fully emulated drivers.
• Scalability: Improved performance under high I/O workloads, allowing VMs to handle more simultaneous I/O requests.

Example for KVM:

To use virtio for disk devices in KVM:

1. Ensure your guest OS supports virtio. Most modern Linux distributions do.
2. When creating a new virtual disk or modifying an existing one, set the disk bus to virtio.

virt-install ... --disk path=/path/to/disk.img,bus=virtio ...

Or, in the VM's XML definition (used with virsh edit or similar):

<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none'/>
  <source file='/path/to/disk.img'/>
  <target dev='vda' bus='virtio'/>
</disk>

The use of virtio drivers, combined with appropriate VM disk and network settings, can drastically "improve disk IO performance" in a virtualized environment

 

Frequently Asked Questions

 

Conclusion

Improving disk I/O performance is vital for ensuring smooth operations, especially in data-intensive applications and virtualized environments. Effective strategies include optimizing kernel parameters with sysctl, controlling user-level limits using ulimit, and leveraging the benefits of virtio drivers in virtualized scenarios. By fine-tuning these parameters and being aware of the tools at your disposal, you can achieve significant improvements in disk I/O throughput and latency.