DRBD Users Guide 9.0

The DRBD9 User’s Guide

Lars Ellenberg

Andreas Grünbacher

Florian Haas

Brian Hellman

Roland Kammerer

Philipp Marek

Philipp Reisner

This guide has been released to the DRBD community, and its authors strive to improve it permanently. Feedback from readers is always welcome and encouraged. Please use the DRBD public mailing list for enhancements, suggestions and corrections.

September 2016

License information

The text of and illustrations in this document are licensed under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA", brief explanation, full license text).

In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.

Trademarks used in this guide

DRBD®, the DRBD logo, LINBIT®, and the LINBIT logo are trademarks or registered trademarks of LINBIT Information Technologies GmbH in Austria, the United States and other countries.

AMD is a registered trademark of Advanced Micro Devices, Inc.

Citrix is a registered trademark of Citrix, Inc.

Debian is a registered trademark of Software in the Public Interest, Inc.

Dolphin Interconnect Solutions and SuperSockets are trademarks or registered trademarks of Dolphin Interconnect Solutions ASA.

IBM is a registered trademark of International Business Machines Corporation.

Intel is a registered trademark of Intel Corporation.

Linux is a registered trademark of Linus Torvalds.

Oracle, MySQL, and MySQL Enterprise are trademarks or registered trademarks of Oracle Corporation and/or its affiliates.

Red Hat, Red Hat Enterprise Linux, and RPM are trademarks or registered trademarks of Red Hat, Inc.

SuSE, SUSE, and SUSE Linux Enterprise Server are trademarks or registered trademarks of Novell, Inc.

Xen is a registered trademark of Citrix, Inc.

Other names mentioned in this guide may be trademarks or registered trademarks of their respective owners.

Table of Contents

Please Read This First
I. Introduction to DRBD
1. DRBD Fundamentals
1.1. Kernel module
1.2. User space administration tools
1.3. Resources
1.4. Resource roles
2. DRBD Features
2.1. Single-primary mode
2.2. Dual-primary mode
2.3. Replication modes
2.4. More than 2-way redundancy
2.5. Automatic Promotion of Resources
2.6. Multiple replication transports
2.6.1. TCP Transport
2.6.2. RDMA Transport
2.6.3. Conclusion
2.7. Efficient synchronization
2.7.1. Variable-rate synchronization
2.7.2. Fixed-rate synchronization
2.7.3. Checksum-based synchronization
2.8. Suspended replication
2.9. On-line device verification
2.10. Replication traffic integrity checking
2.11. Split brain notification and automatic recovery
2.12. Support for disk flushes
2.13. Trim/Discard support
2.14. Disk error handling strategies
2.15. Strategies for dealing with outdated data
2.16. Three-way replication via stacking
2.17. Long-distance replication via DRBD Proxy
2.18. Truck based replication
2.19. Floating peers
2.20. Data rebalancing (horizontal storage scaling)
2.21. DRBD client
II. Building and installing the DRBD software
3. Installing pre-built DRBD binary packages
3.1. Packages supplied by LINBIT
3.2. Packages supplied by distribution vendors
3.2.1. SUSE Linux Enterprise Server
3.2.2. CentOS
3.2.3. Ubuntu Linux
III. Working with DRBD
4. Common administrative tasks - DRBD Manage
4.1. Initializing your cluster
4.2. Adding nodes to your cluster
4.2.1. Types of DRBD Manage nodes
4.2.2. Adding a control node
4.2.3. Adding a pure controller node
4.2.4. Adding a satellite node
4.2.5. Adding a pure client node
4.2.6. Adding an external node
4.3. Cluster configuration
4.4. Configuring storage plugins
4.4.1. Configuring ZFS
4.4.2. Discussion of the storage plugins
4.5. Creating and deploying resources/volumes
4.6. Managing snapshots
4.6.1. Creating a snapshot
4.6.2. Restoring a snapshot
4.6.3. Removing a snapshot
4.7. Checking the state of your cluster
4.8. Setting options for resources
4.9. Rebalancing data with DRBD Manage
4.10. Getting help
5. Common administrative tasks - Manually
5.1. Configuring DRBD
5.1.1. Preparing your lower-level storage
5.1.2. Preparing your network configuration
5.1.3. Configuring your resource
5.1.4. Defining network connections
5.1.5. Configuring transport implementations
5.1.6. Enabling your resource for the first time
5.1.7. The initial device synchronization
5.1.8. Using truck based replication
5.1.9. Example configuration for four nodes
5.2. Checking DRBD status
5.2.1. Retrieving status with drbd-overview
5.2.2. Status information in /proc/drbd
5.2.3. Status information via drbdadm
5.2.4. One-shot or realtime monitoring via drbdsetup events2
5.2.5. Connection states
5.2.6. Resource roles
5.2.7. Disk states
5.2.8. Connection information data
5.2.9. Performance indicators
5.3. Enabling and disabling resources
5.3.1. Enabling resources
5.3.2. Disabling resources
5.4. Reconfiguring resources
5.5. Promoting and demoting resources
5.6. Basic Manual Fail-over
5.7. Upgrading DRBD
5.7.1. General overview
5.7.2. Updating your repository
5.7.3. Checking the DRBD state
5.7.4. Pausing the cluster
5.7.5. Upgrading the packages
5.7.6. Loading the new Kernel module
5.7.7. Migrating your configuration files
5.7.8. Changing the meta-data
5.7.9. Starting DRBD again
5.7.10. From DRBD 9 to DRBD 9
5.8. Enabling dual-primary mode
5.8.1. Permanent dual-primary mode
5.8.2. Temporary dual-primary mode
5.9. Using on-line device verification
5.9.1. Enabling on-line verification
5.9.2. Invoking on-line verification
5.9.3. Automating on-line verification
5.10. Configuring the rate of synchronization
5.10.1. Estimating a synchronization speed
5.10.2. Variable sync rate configuration
5.10.3. Permanent fixed sync rate configuration
5.10.4. Some more hints about synchronization
5.11. Configuring checksum-based synchronization
5.12. Configuring congestion policies and suspended replication
5.13. Configuring I/O error handling strategies
5.14. Configuring replication traffic integrity checking
5.15. Resizing resources
5.15.1. Growing on-line
5.15.2. Growing off-line
5.15.3. Shrinking on-line
5.15.4. Shrinking off-line
5.16. Disabling backing device flushes
5.17. Configuring split brain behavior
5.17.1. Split brain notification
5.17.2. Automatic split brain recovery policies
5.18. Creating a stacked three-node setup
5.18.1. Device stacking considerations
5.18.2. Configuring a stacked resource
5.18.3. Enabling stacked resources
5.19. Data rebalancing
5.19.1. Prepare a bitmap slot
5.19.2. Preparing and activating the new node
5.19.3. Starting the initial sync
5.19.4. Check connectivity
5.19.5. After the initial sync
5.19.6. Cleaning up
5.19.7. Conclusion and further steps
6. Using DRBD Proxy
6.1. DRBD Proxy deployment considerations
6.2. Installation
6.3. License file
6.4. Configuration
6.5. Controlling DRBD Proxy
6.6. About DRBD Proxy plugins
6.6.1. Using a WAN Side Bandwidth Limit
6.7. Troubleshooting
7. Troubleshooting and error recovery
7.1. Dealing with hard drive failure
7.1.1. Manually detaching DRBD from your hard drive
7.1.2. Automatic detach on I/O error
7.1.3. Replacing a failed disk when using internal meta data
7.1.4. Replacing a failed disk when using external meta data
7.2. Dealing with node failure
7.2.1. Dealing with temporary secondary node failure
7.2.2. Dealing with temporary primary node failure
7.2.3. Dealing with permanent node failure
7.3. Manual split brain recovery
IV. DRBD-enabled applications
8. Integrating DRBD with Pacemaker clusters
8.1. Pacemaker primer
8.2. Using DRBD as a background service in a pacemaker cluster
8.3. Adding a DRBD-backed service to the cluster configuration, including a master-slave resource
8.4. Using resource-level fencing in Pacemaker clusters
8.4.1. Resource-level fencing with dopd
8.4.2. Resource-level fencing using the Cluster Information Base (CIB)
8.5. Using stacked DRBD resources in Pacemaker clusters
8.5.1. Adding off-site disaster recovery to Pacemaker clusters
8.5.2. Using stacked resources to achieve 4-way redundancy in Pacemaker clusters
8.6. Configuring DRBD to replicate between two SAN-backed Pacemaker clusters
8.6.1. DRBD resource configuration
8.6.2. Pacemaker resource configuration
8.6.3. Site fail-over
9. Integrating DRBD with Red Hat Cluster
9.1. Red Hat Cluster background information
9.1.1. Fencing
9.1.2. The Resource Group Manager
9.2. Red Hat Cluster configuration
9.2.1. The cluster.conf file
9.3. Using DRBD in Red Hat Cluster fail-over clusters
9.3.1. Setting up your cluster configuration
10. Using LVM with DRBD
10.1. LVM primer
10.2. Using a Logical Volume as a DRBD backing device
10.3. Using automated LVM snapshots during DRBD synchronization
10.4. Configuring a DRBD resource as a Physical Volume
10.5. Adding a new DRBD volume to an existing Volume Group
10.6. Nested LVM configuration with DRBD
10.6.1. Switching the VG to the other node
10.7. Highly available LVM with Pacemaker
11. Using GFS with DRBD
11.1. GFS primer
11.2. Creating a DRBD resource suitable for GFS
11.3. Configuring LVM to recognize the DRBD resource
11.4. Configuring your cluster to support GFS
11.5. Creating a GFS filesystem
11.6. Using your GFS filesystem
12. Using OCFS2 with DRBD
12.1. OCFS2 primer
12.2. Creating a DRBD resource suitable for OCFS2
12.3. Creating an OCFS2 filesystem
12.4. Pacemaker OCFS2 management
12.4.1. Adding a Dual-Primary DRBD resource to Pacemaker
12.4.2. Adding OCFS2 management capability to Pacemaker
12.4.3. Adding an OCFS2 filesystem to Pacemaker
12.4.4. Adding required Pacemaker constraints to manage OCFS2 filesystems
12.5. Legacy OCFS2 management (without Pacemaker)
12.5.1. Configuring your cluster to support OCFS2
12.5.2. Using your OCFS2 filesystem
13. Using Xen with DRBD
13.1. Xen primer
13.2. Setting DRBD module parameters for use with Xen
13.3. Creating a DRBD resource suitable to act as a Xen VBD
13.4. Using DRBD VBDs
13.5. Starting, stopping, and migrating DRBD-backed domU’s
13.6. Internals of DRBD/Xen integration
13.7. Integrating Xen with Pacemaker
14. DRBD volumes in Openstack
14.1. Openstack Overview
14.2. DRBD for Openstack Installation
14.2.1. Additional Configuration
14.3. Choosing the Transport Protocol
14.3.1. iSCSI Transport
14.3.2. DRBD Transport
14.3.3. Configuring the Transport Protocol
14.4. Some further notes
14.4.1. Free space reporting
14.4.2. Hyperconverged Setups
15. DRBD Volumes in OpenNebula
15.1. OpenNebula Overview
15.2. OpenNebula Installation
15.3. Deployment Policies
15.4. Live Migration
15.5. Free Space Reporting
16. DRBD Volumes in Docker
16.1. Docker Overview
16.2. Docker Plugin Installation
16.3. Some examples
17. DRBD Volumes in Proxmox VE
17.1. Proxmox VE Overview
17.2. Proxmox Plugin Installation
17.3. Proxmox Plugin Configuration
V. Optimizing DRBD performance
18. Measuring block device performance
18.1. Measuring throughput
18.2. Measuring latency
19. Optimizing DRBD throughput
19.1. Hardware considerations
19.2. Throughput overhead expectations
19.3. Tuning recommendations
19.3.1. Setting max-buffers and max-epoch-size
19.3.2. Tuning the TCP send buffer size
19.3.3. Tuning the Activity Log size
19.3.4. Disabling barriers and disk flushes
19.4. Achieving better Read Performance via increased Redundancy
20. Optimizing DRBD latency
20.1. Hardware considerations
20.2. Latency overhead expectations
20.3. Latency vs. IOPs
20.4. Tuning recommendations
20.4.1. Setting DRBD’s CPU mask
20.4.2. Modifying the network MTU
20.4.3. Enabling the deadline I/O scheduler
VI. Learning more
21. DRBD Internals
21.1. DRBD meta data
21.1.1. Internal meta data
21.1.2. External meta data
21.1.3. Estimating meta data size
21.2. Generation Identifiers
21.2.1. Data generations
21.2.2. The generation identifier tuple
21.2.3. How generation identifiers change
21.2.4. How DRBD uses generation identifiers
21.3. The Activity Log
21.3.1. Purpose
21.3.2. Active extents
21.3.3. Selecting a suitable Activity Log size
21.4. The quick-sync bitmap
21.5. The Peer-Fencing interface
22. More Information about DRBD Manage
22.1. Free Space reporting
22.2. Policy plugin, waiting for deployment
22.2.1. More detailed information for driver-writers
23. Getting more information
23.1. Commercial DRBD support
23.2. Public mailing list
23.3. Public IRC Channels
23.4. Official Twitter account
23.5. Publications
23.6. Other useful resources
VII. Appendices
A. Recent changes
A.1. Connections
A.2. Auto-Promote Feature
A.3. Increased Performance
A.4. Multiple Volumes in one Resource
A.4.1. Changes to udev symlinks
A.5. Changes to the configuration syntax
A.5.1. Boolean configuration options
A.5.2. syncer section no longer exists
A.5.3. protocol option is no longer special
A.5.4. New per-resource options section
A.6. On-line changes to network communications
A.6.1. Changing the replication protocol
A.6.2. Changing from single-Primary to dual-Primary replication
A.7. Changes to the drbdadm command
A.7.1. Changes to pass-through options
A.7.2. --force option replaces --overwrite-data-of-peer
A.8. Changed default values
A.8.1. Number of concurrently active Activity Log extents (al-extents)
A.8.2. Run-length encoding (use-rle)
A.8.3. I/O error handling strategy (on-io-error)
A.8.4. Variable-rate synchronization
A.8.5. Number of configurable DRBD devices (minor-count)
B. DRBD system manual pages
drbd.conf — DRBD Configuration Files
drbdadm — Utility for DRBD administration
drbdmeta — Manipulate the DRBD on-disk metadata