DRBD Users Guide 8.0-8.3

Chapter 6. Common administrative tasks

This chapter outlines typical administrative tasks encountered during day-to-day operations. It does not cover troubleshooting tasks, these are covered in detail in Chapter 7, Troubleshooting and error recovery.

Checking DRBD status

Retrieving status with drbd-overview

The most convenient way to look at DRBD's status is the drbd-overview utility. It is available since DRBD-8.0.15 and 8.3.0. In 8.3.0 it is installed as drbd-overview.pl.

drbd-overview
  0:home                 Connected Primary/Secondary   UpToDate/UpToDate C r--- /home        xfs  200G 158G 43G  79%
  1:data                 Connected Primary/Secondary   UpToDate/UpToDate C r--- /mnt/ha1     ext3 9.9G 618M 8.8G 7%
  2:nfs-root             Connected Primary/Secondary   UpToDate/UpToDate C r--- /mnt/netboot ext3 79G  57G  19G  76%

Status information in /proc/drbd

/proc/drbd is a virtual file displaying real-time status information about all DRBD resources currently configured. You may interrogate this file's contents using this command:

cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by buildsystem@linbit, 2008-12-18 16:02:26
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r---
    ns:0 nr:8 dw:8 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
 1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r---
    ns:0 nr:12 dw:12 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
 2: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r---
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

The first line, prefixed with version:, shows the DRBD version used on your system. The second line contains information about this specific build.

The other four lines in this example form a block that is repeated for every DRBD device configured, prefixed by the device minor number. In this case, this is 0, corresponding to the device /dev/drbd0.

The resource-specific output from /proc/drbd contains various pieces of information about the resource:

  • cs (connection state). Status of the network connection. See the section called “Connection states” for details about the various connection states.

  • ro (roles). Roles of the nodes. The role of the local node is displayed first, followed by the role of the partner node shown after the slash. See the section called “Resource roles” for details about the possible resource roles.

    [Note]Note

    Prior to DRBD 8.3, /proc/drbd used the st field (referring to the ambiguous term state) when referring to resource roles.

  • ds (disk states). State of the hard disks. Prior to the slash the state of the local node is displayed, after the slash the state of the hard disk of the partner node is shown. See the section called “Disk states” for details about the various disk states.

  • ns (network send).  Volume of net data sent to the partner via the network connection; in Kibyte.

  • nr (network receive).  Volume of net data received by the partner via the network connection; in Kibyte.

  • dw (disk write). Net data written on local hard disk; in Kibyte.

  • dr (disk read). Net data read from local hard disk; in Kibyte.

  • al (activity log). Number of updates of the activity log area of the meta data.

  • bm (bit map).  Number of updates of the bitmap area of the meta data.

  • lo (local count). Number of open requests to the local I/O sub-system issued by DRBD.

  • pe (pending). Number of requests sent to the partner, but that have not yet been answered by the latter.

  • ua (unacknowledged). Number of requests received by the partner via the network connection, but that have not yet been answered.

  • ap (application pending). Number of block I/O requests forwarded to DRBD, but not yet answered by DRBD.

  • ep (epochs). Number of epoch objects. Usually 1. Might increase under I/O load when using either the barrier or the none write ordering method. Since 8.2.7.

  • wo (write order). Currently used write ordering method: b (barrier), f (flush), d (drain) or n (none). Since 8.2.7.

  • oos (out of sync). Amount of storage currently out of sync; in Kibibytes. Since 8.2.6.

Connection states

A resource's connection state can be observed either by monitoring /proc/drbd, or by issuing the drbdadm cstate command:

drbdadm cstate resource
Connected

A resource may have one of the following connection states:

  • StandAloneNo network configuration available. The resource has not yet been connected, or has been administratively disconnected (using drbdadm disconnect), or has dropped its connection due to failed authentication or split brain.

  • Disconnecting Temporary state during disconnection. The next state is StandAlone.

  • Unconnected Temporary state, prior to a connection attempt. Possible next states: WFConnection and WFReportParams.

  • TimeoutTemporary state following a timeout in the communication with the peer. Next state: Unconnected.

  • BrokenPipeTemporary state after the connection to the peer was lost. Next state: Unconnected.

  • NetworkFailureTemporary state after the connection to the partner was lost. Next state: Unconnected.

  • ProtocolErrorTemporary state after the connection to the partner was lost. Next state: Unconnected.

  • TearDownTemporary state. The peer is closing the connection. Next state: Unconnected.

  • WFConnectionThis node is waiting until the peer node becomes visible on the network.

  • WFReportParamsTCP connection has been established, this node waits for the first network packet from the peer.

  • ConnectedA DRBD connection has been established, data mirroring is now active. This is the normal state.

  • StartingSyncSFull synchronization, initiated by the administrator, is just starting. The next possible states are: SyncSource or PausedSyncS.

  • StartingSyncTFull synchronization, initiated by the administrator, is just starting. Next state: WFSyncUUID.

  • WFBitMapSPartial synchronization is just starting. Next possible states: SyncSource or PausedSyncS.

  • WFBitMapTPartial synchronization is just starting. Next possible state: WFSyncUUID.

  • WFSyncUUIDSynchronization is about to begin. Next possible states: SyncTarget or PausedSyncT.

  • SyncSourceSynchronization is currently running, with the local node being the source of synchronization.

  • SyncTargetSynchronization is currently running, with the local node being the target of synchronization.

  • PausedSyncSThe local node is the source of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by drbdadm pause-sync.

  • PausedSyncTThe local node is the target of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by drbdadm pause-sync.

  • VerifySOn-line device verification is currently running, with the local node being the source of verification.

  • VerifyTOn-line device verification is currently running, with the local node being the target of verification.

Resource roles

A resource's role can be observed either by monitoring /proc/drbd, or by issuing the drbdadm role command:

drbdadm role resource
Primary/Secondary

The local resource role is always displayed first, the remote resource role last.

[Note]Note

Prior to DRBD 8.3, the drbdadm state command provided the same information. Since state is an ambigious term, DRBD uses role in its stead from version 8.3.0 forward. drbdadm state is also still available, albeit only for compatibility reasons. You should use drbdadm role.

You may see one of the following resource roles:

  • PrimaryThe resource is currently in the primary role, and may be read from and written to. This role only occurs on one of the two nodes, unless dual-primary node is enabled.

  • SecondaryThe resource is currently in the secondary role. It normally receives updates from its peer (unless running in disconnected mode), but may neither be read from nor written to. This role may occur on one node or both nodes.

  • UnknownThe resource's role is currently unknown. The local resource role never has this status. It is only displayed for the peer's resource role, and only in disconnected mode.

Disk states

A resource's disk state can be observed either by monitoring /proc/drbd, or by issuing the drbdadm dstate command:

drbdadm dstate resource
UpToDate/UpToDate

The local disk state is always displayed first, the remote disk state last.

Both the local and the remote disk state may be one of the following:

  • DisklessNo local block device has been assigned to the DRBD driver. This may mean that the resource has never attached to its backing device, that it has been manually detached using drbdadm detach, or that it automatically detached after a lower-level I/O error.

  • AttachingTransient state while reading meta data.

  • FailedTransient state following an I/O failure report by the local block device. Next state: Diskless.

  • NegotiatingTransient state when an Attach is carried out on an already-connected DRBD device.

  • InconsistentThe data is inconsistent. This status occurs immediately upon creation of a new resource, on both nodes (before the initial full sync). Also, this status is found in one node (the synchronization target) during synchronization.

  • OutdatedResource data is consistent, but outdated.

  • DUnknownThis state is used for the peer disk if no network connection is available.

  • ConsistentConsistent data of a node without connection. When the connection is established, it is decided whether the data are UpToDate or Outdated.

  • UpToDateConsistent, up-to-date state of the data. This is the normal state.