determining whether the two nodes are in fact members of the same cluster (as opposed to two nodes that were connected accidentally),
determining the direction of background re-synchronization (if necessary),
determining whether full re-synchronization is necessary or whether partial re-synchronization is sufficient,
DRBD marks the start of a new data generation at each of the following occurrences:
The initial device full sync,
a disconnected resource switching to the primary role,
a resource in the primary role disconnecting.
Thus, we can summarize that whenever a resource is in the
Connected connection state, and both nodes' disk state is
UpToDate, the current data generation on both nodes is the same. The inverse is also true.
Every new data generation is identified by a 8-byte, universally unique identifier (UUID).
DRBD keeps four pieces of information about current and historical data generations in the local resource meta data:
Current UUID. This is the generation identifier for the current data generation, as seen from the local node's perspective. When a resource is
Connected and fully synchronized, the current UUID is identical between nodes.
“Bitmap” UUID. This is the UUID of the generation against which the on-disk sync bitmap is tracking changes. As the on-disk sync bitmap itself, this identifier is only relevant while in disconnected mode. If the resource is
Connected, this UUID is always empty (zero).
Two “Historical” UUID's. These are the identifiers of the two data generations preceding the current one.
Collectively, these four items are referred to as the generation identifier tuple, or “GI tuple” for short.
When a node loses connection to its peer (either by network failure or manual intervention), DRBD modifies its local generation identifiers in the following manner:
A new UUID is created for the new data generation. This becomes the new current UUID for the primary node.
The previous UUID now refers to the generation the bitmap is tracking changes against, so it becomes the new bitmap UUID for the primary node.
On the secondary node, the GI tuple remains unchanged.
Upon the initiation of re-synchronization, DRBD performs these modifications on the local generation identifiers:
The current UUID on the synchronization source remains unchanged.
The bitmap UUID on the synchronization source is rotated out to the first historical UUID.
A new bitmap UUID is generated on the synchronization source.
This UUID becomes the new current UUID on the synchronization target.
The bitmap and historical UUID's on the synchronization target remain unchanged.
When re-synchronization concludes, the following changes are performed:
The current UUID on the synchronization source remains unchanged.
The bitmap UUID on the synchronization source is rotated out to the first historical UUID, with that UUID moving to the second historical entry (any existing second historical entry is discarded).
The bitmap UUID on the synchronization source is then emptied (zeroed).
The synchronization target adopts the entire GI tuple from the synchronization source.
When a connection between nodes is established, the two nodes exchange their currently available generation identifiers, and proceed accordingly. A number of possible outcomes exist:
Current UUID's empty on both nodes. The local node detects that both its current UUID and the peer's current UUID are empty. This is the normal occurrence for a freshly configured resource that has not had the initial full sync initiated. No synchronization takes place; it has to be started manually.
Current UUID's empty on one node. The local node detects that the peer's current UUID is empty, and its own is not. This is the normal case for a freshly configured resource on which the initial full sync has just been initiated, the local node having been selected as the initial synchronization source. DRBD now sets all bits in the on-disk sync bitmap (meaning it considers the entire device out-of-sync), and starts synchronizing as a synchronization source.
If the opposite case (local current UUID empty, peer's non-empty), DRBD performs the same steps, except that the local node becomes the synchronization target.
Equal current UUID's. The local node detects that its current UUID and the peer's current UUID are non-empty and equal. This is the normal occurrence for a resource that went into disconnected mode at a time when it was in the secondary role, and was not promoted on either node while disconnected. No synchronization takes place, as none is necessary.
Bitmap UUID matches peer's current UUID. The local node detects that its bitmap UUID matches the peer's current UUID, and that the peer's bitmap UUID is empty. This is the normal and expected occurrence after a secondary node failure, with the local node being in the primary role. It means that the peer never became primary in the meantime and worked on the basis of the same data generation all along. DRBD now initiates a normal, background re-synchronization, with the local node becoming the synchronization source.
If, conversely, the local node detects that its bitmap UUID is empty, and that the peer's bitmap matches the local node's current UUID, then that is the normal and expected occurrence after a failure of the local node. Again, DRBD now initiates a normal, background re-synchronization, with the local node becoming the synchronization target.
Current UUID matches peer's historical UUID. The local node detects that its current UUID matches one of the peer's historical UUID's. This implies that while the two data sets share a common ancestor, and the local node has the up-to-date data, the information kept in the local node's bitmap is outdated and not useable. Thus, a normal synchronization would be insufficient. DRBD now marks the entire device as out-of-sync and initiates a full background re-synchronization, with the local node becoming the synchronization source.
In the opposite case (one of the local node's historical UUID matches the peer's current UUID), DRBD performs the same steps, except that the local node becomes the synchronization target.
Bitmap UUID's match, current UUID's do not. The local node detects that its current UUID differs from the peer's current UUID, and that the bitmap UUID's match. This is split brain, but one where the data generations have the same parent. This means that DRBD invokes split brain auto-recovery strategies, if configured. Otherwise, DRBD disconnects and waits for manual split brain resolution.
Neither current nor bitmap UUID's match. The local node detects that its current UUID differs from the peer's current UUID, and that the bitmap UUID's do not match. This is split brain with unrelated ancestor generations, thus auto-recovery strategies, even if configured, are moot. DRBD disconnects and waits for manual split brain resolution.
No UUID's match. Finally, in case DRBD fails to detect even a single matching element in the two nodes' GI tuples, it logs a warning about unrelated data and disconnects. This is DRBD's safeguard against accidental connection of two cluster nodes that have never heard of each other before.