Troubleshoot 389 Directory Server Replication Errors and Synchronization Failures

Replication can report green status on one agreement while another path is unreachable, credentials fail, or lag climbs quietly. This guide walks through symptom-driven diagnosis: capture a baseline before you change anything, follow the connection path from supplier to consumer, enable targeted debug logging when status output is not enough, and choose the least disruptive recovery action.

For routine monitoring commands and field definitions, see monitor replication and lag. For agreement creation and modification, see manage replication agreements. For initialization and reinitialization procedures, see initialize and reinitialize replicas. For stale supplier replica IDs after permanent removal, see CleanAllRUV.

Tested on: Rocky Linux 10.2; 389 Directory Server 3.2.0.

Troubleshoot by replication symptom

Start from what you observe, not from a generic command checklist. The table below maps common symptoms to the layer you should inspect first.

Reinitialization is the right answer only after you rule out transient network, TLS, bind, schedule, and agreement-state problems and you have identified an authoritative supplier.

Lab environment

Examples use the shared replication lab from manage replication agreements and monitor replication and lag.

Password files: /root/dm.pw on ldap1, /root/consumer1-dm.pw on the consumer. For ds-replcheck online, /root/dm.pw works in this lab because Directory Manager uses the same password on both instances. Both instances run on one Rocky Linux host; use the agreement hostname and port in production.

Capture a replication baseline before making changes

Record the current state before you restart services, change passwords, or reinitialize anything. A short baseline makes rollback and post-fix comparison possible.

Confirm the instance is running and note the package build:

systemctl is-active dirsrv@ldap1

Sample output:

active

rpm -q 389-ds-base

Sample output:

389-ds-base-3.2.0-8.el10_2.x86_64

Check disk space on the database volume. A full disk can stop changelog writes and leave replicas unable to catch up:

df -h /var/lib/dirsrv

Sample output:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rlm-root   14G   12G  2.0G  86% /

Verify time synchronization. CSNs embed time, and skew can produce misleading lag or session failures:

timedatectl status

Sample output (trimmed):

System clock synchronized: no
              NTP service: active

If System clock synchronized is no, fix NTP on every topology member before you chase replication errors.

Record replica role, suffix, and replica ID on the supplier:

dsconf -y /root/dm.pw ldap1 replication get --suffix "dc=example,dc=com"

Sample output (trimmed):

nsDS5ReplicaId: 1
nsDS5ReplicaRoot: dc=example,dc=com
nsDS5ReplicaType: 3

nsDS5ReplicaType: 3 is a supplier. Consumers use type 2 and replica ID 65535.

List outgoing agreements:

dsconf -y /root/dm.pw ldap1 repl-agmt list --suffix "dc=example,dc=com"

Sample output:

cn: ldap1-to-consumer1
cn: ldap1-to-supplier2

Capture agreement status for each name. On the healthy path:

dsconf -y /root/dm.pw ldap1 repl-agmt status --suffix "dc=example,dc=com" --bind-dn "cn=Directory Manager" --bind-passwd-file /root/consumer1-dm.pw ldap1-to-consumer1

Sample output (trimmed):

Last Update Status: Error (0) Replica acquired successfully: Incremental update succeeded
Last Init Status: Error (0) Total update succeeded
Replication Status: In Synchronization
Replication Lag Time: 00:00:00

Error (0) with Incremental update succeeded is success. The word Error is legacy labeling; read the message text.

On the unreachable secondary agreement in this lab, supplier2.example.com is intentionally absent:

dsconf -y /root/dm.pw ldap1 repl-agmt status --suffix "dc=example,dc=com" ldap1-to-supplier2

Because supplier2 is intentionally absent, this test only exercises the connectivity-failure path. For an existing destination whose Directory Manager password differs, provide that destination's bind DN and password file with --bind-dn and --bind-passwd-file.

Sample output (trimmed):

Last Update Status: Error (0) No replication sessions started since server startup
Replication Status: Not in Synchronization: supplier (6a5db4ba000000010000) consumer (Unavailable) State (green) Reason (error (0) no replication sessions started since server startup)
Replication Lag Time: unavailable

consumer (Unavailable) and Replication Lag Time: unavailable point to connectivity or service state on the remote host, not a successful incremental session.

Save RUV output from the supplier:

dsconf -y /root/dm.pw ldap1 replication get-ruv --suffix "dc=example,dc=com"

Sample output (trimmed):

RUV:        {replica 1 ldap://ldap1.example.com:389} 6a5d05cc000000010000 6a5db9e2000000010000
Replica ID: 1
Max CSN:    2026-07-20 06:02:10 (6a5db9e2000000010000)

Run automated replication health checks:

dsctl ldap1 healthcheck --check replication

Sample output:

Healthcheck complete.
No issues found.

Run one controlled test update and confirm it reaches the consumer. Add a disposable description on the supplier:

ldapmodify -x -H ldap://127.0.0.1:389 -D "cn=Directory Manager" -y /root/dm.pw <<'EOF'
dn: uid=user1,ou=People,dc=example,dc=com
changetype: modify
add: description
description: repl-troubleshoot-test-20260720
EOF

The modify succeeds with no output beyond modifying entry.

After a few seconds, confirm replication on the agreement:

dsconf -y /root/dm.pw ldap1 repl-agmt status --suffix "dc=example,dc=com" --bind-dn "cn=Directory Manager" --bind-passwd-file /root/consumer1-dm.pw ldap1-to-consumer1

Sample output (trimmed):

Last Update Status: Error (0) Replica acquired successfully: Incremental update succeeded
Replication Status: In Synchronization
Replication Lag Time: 00:00:00

Read the value on the consumer:

ldapsearch -LLL -x -H ldap://127.0.0.1:1389 -D "cn=Directory Manager" -y /root/consumer1-dm.pw -b "uid=user1,ou=People,dc=example,dc=com" -s base description

Sample output:

dn: uid=user1,ou=People,dc=example,dc=com
description: repl-troubleshoot-test-20260720

Remove the test value when you finish:

ldapmodify -x -H ldap://127.0.0.1:389 -D "cn=Directory Manager" -y /root/dm.pw <<'EOF'
dn: uid=user1,ou=People,dc=example,dc=com
changetype: modify
delete: description
description: repl-troubleshoot-test-20260720
EOF

Store repl-agmt get output, monitor results, and the current nsslapd-errorlog-level with this baseline.

Check network connectivity, TLS, and authentication

Replication agreements use the same LDAP connection path you would test manually. Start at the supplier and work toward the consumer or hub named in the agreement.

Test the same connection used by the agreement

Read the agreement target host and port:

dsconf -y /root/dm.pw ldap1 repl-agmt get --suffix "dc=example,dc=com" ldap1-to-consumer1

Sample output (trimmed):

nsDS5ReplicaHost: consumer1.example.com
nsDS5ReplicaPort: 1389
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsds5ReplicaEnabled: on

When an administrator configured a replication window, the same repl-agmt get output can also include:

nsDS5ReplicaUpdateSchedule: 0800-2200 0246

nsds5ReplicaEnabled: off means the agreement is disabled. A defined nsDS5ReplicaUpdateSchedule limits when replication runs. If the schedule attribute is absent, updates are not restricted to a configured window.

Confirm the hostname resolves from the supplier host:

getent hosts consumer1.example.com

An empty result means DNS or /etc/hosts is wrong before you touch Directory Server.

Test the agreement endpoint with the same protocol the agreement uses. Agreement endpoint checks use the ldapsearch command with the same -H, -D, and -b values the agreement uses. This lab agreement uses plain LDAP:

ldapsearch -LLL -x -H ldap://consumer1.example.com:1389 -D "cn=Directory Manager" -y /root/consumer1-dm.pw -b "dc=example,dc=com" -s base

Sample output:

dn: dc=example,dc=com
dc: example

A successful search confirms routing, port access, and a working LDAP bind with known credentials.

To simulate a stopped consumer, repl-agmt status on the supplier reports the failure path. After stopping dirsrv@ldap2 in the lab:

Replication Status: Not in Synchronization: supplier (6a5db4ba000000010000) consumer (Unavailable) State (green) Reason (error (0) replica acquired successfully: incremental update succeeded)
Replication Lag Time: unavailable

Manual ldapsearch to the same host returns:

ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Restart the consumer, confirm that its LDAP endpoint responds, and poke the agreement before continuing.

Diagnose TLS and certificate failures

When the agreement uses LDAPS or StartTLS, test with the same security settings the agreement uses. Certificate trust, expiry, hostname or SAN mismatch, and protocol version mismatches appear in the supplier errors log and often as bind or handshake failures in repl-agmt status.

The healthy ldap1-to-consumer1 agreement in this lab uses plain LDAP on port 1389. The commands below are TLS connectivity templates for agreements that use consumer1.example.com on the lab's LDAPS port 1636 or StartTLS on 1389. Substitute your CA path and hostname when your deployment differs.

For LDAPS, point ldapsearch at the secure port and trust the issuing CA:

LDAPTLS_CACERT=/path/to/ca.crt ldapsearch -LLL -x -H ldaps://consumer1.example.com:1636 -D "cn=Directory Manager" -y /root/consumer1-dm.pw -b "dc=example,dc=com" -s base

For StartTLS on the plain LDAP port:

LDAPTLS_CACERT=/path/to/ca.crt ldapsearch -LLL -x -ZZ -H ldap://consumer1.example.com:1389 -D "cn=Directory Manager" -y /root/consumer1-dm.pw -b "dc=example,dc=com" -s base

Sample output when TLS and trust are correct:

dn: dc=example,dc=com
dc: example

-ZZ requires StartTLS to succeed. The command fails instead of silently continuing without TLS. Replace /path/to/ca.crt with the CA certificate your deployment trusts.

Keep certificate issuance, renewal, and mapping procedures in the certificate-based replication auth guide and certificate management. This troubleshooting guide stops at identifying TLS as the failure layer.

Diagnose bind failures

Replication bind failures usually mean the agreement bind DN, password, or authentication method does not match what the consumer expects.

Test with intentionally wrong credentials to see the client-side error:

ldapsearch -LLL -x -H ldap://consumer1.example.com:1389 -D "cn=Directory Manager" -w wrongpassword -b "dc=example,dc=com" -s base

Sample output:

ldap_bind: Invalid credentials (49)

Compare the agreement nsDS5ReplicaBindDN with the identity configured on the consumer, confirm bind-group membership when used, and verify the password file on the supplier matches the consumer's replication manager entry. After you change credentials, update every agreement that uses the old password and poke the agreement to start a new session.

Enable replication debug logging with level 8192

When status output is inconclusive, temporarily add replication debug logging. Log levels are additive: read the current value and add 8192, do not replace the existing level with 8192 alone.

Read the current errors-log level:

dsconf -y /root/dm.pw ldap1 config get nsslapd-errorlog-level

Sample output:

nsslapd-errorlog-level: 16384

Add 8192 to that value (16384 + 8192 = 24576 in this lab) and apply it:

dsconf -y /root/dm.pw ldap1 config replace nsslapd-errorlog-level=24576

Sample output:

Successfully replaced value(s) for 'nsslapd-errorlog-level': '24576'

Reproduce the failure — for example poke the agreement:

dsconf -y /root/dm.pw ldap1 repl-agmt poke --suffix "dc=example,dc=com" ldap1-to-consumer1

Inspect replication debug lines in the errors log:

grep -i NSMMReplicationPlugin /var/log/dirsrv/slapd-ldap1/errors | tail -5

Sample output (trimmed):

[20/Jul/2026:12:01:12.822276702 +0530] - DEBUG - NSMMReplicationPlugin - repl5_inc_run - sid="SZ9g6knPq3X  24" - agmt="cn=ldap1-to-consumer1" (consumer1:1389): State: start -> ready_to_acquire_replica
[20/Jul/2026:12:01:12.827822173 +0530] - DEBUG - NSMMReplicationPlugin - repl5_inc_run - sid="SZ9g6knPq3X  24" - agmt="cn=ldap1-to-consumer1" (consumer1:1389): State: ready_to_acquire_replica -> wait_for_changes

Restore the original level when you finish collecting data:

dsconf -y /root/dm.pw ldap1 config replace nsslapd-errorlog-level=16384

Replication debug logging can generate substantial output and affect performance. Use it briefly during diagnosis, not as a permanent setting.

Trace an update using the replication session ID

The sid value in debug lines identifies one replication session. Correlate a supplier errors-log entry with the matching consumer access-log operation when you need to follow one update through the hop.

Search the supplier errors log for the session ID you saw during the failure window, then search the consumer access log for the same sid around the same timestamp. That pairing confirms whether the consumer received and applied the update or rejected it during acquisition.

Validate replica roles, IDs, suffixes, and agreements

Many failures are configuration mismatches rather than database corruption.

Confirm every peer uses the same replicated suffix on the supplier:

dsconf -y /root/dm.pw ldap1 replication get --suffix "dc=example,dc=com"

Repeat on the consumer:

dsconf -y /root/consumer1-dm.pw ldap2 replication get --suffix "dc=example,dc=com"

Sample consumer output (trimmed):

nsDS5ReplicaId: 65535
nsDS5ReplicaRoot: dc=example,dc=com
nsDS5ReplicaType: 2

Suppliers need unique replica IDs between 1 and 65534. Consumers and hubs use 65535.

Verify agreement direction, enabled state, host, port, protocol, and schedule on the supplier side:

dsconf -y /root/dm.pw ldap1 repl-agmt get --suffix "dc=example,dc=com" ldap1-to-consumer1

Confirm nsds5ReplicaEnabled: on and check whether nsDS5ReplicaUpdateSchedule restricts the current time window. For cascading layouts, confirm the downstream consumer has a working upstream source before you delete an old hub agreement — see cascading replication.

For duplicate supplier replica IDs, isolate the incorrectly configured host, assign a verified unique ID, and reinitialize that server from a healthy supplier. Do not casually change the replica ID of an active supplier that already holds production writes.

Fix "Replica has a different generation ID"

The errors log reports this condition clearly:

The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.

Common causes:

• The replica was never initialized
• It was initialized from a different topology lineage
• The database was restored or replaced independently
• Initialization did not complete successfully

Inspect initialization status on the affected agreement:

dsconf -y /root/dm.pw ldap1 repl-agmt init-status --suffix "dc=example,dc=com" ldap1-to-consumer1

Sample output when initialization previously succeeded:

Agreement successfully initialized.

Use ds-replcheck online to compare the database RUVs. Under Supplier RUV and Replica RUV, the {replicageneration} values should match:

ds-replcheck online -D "cn=Directory Manager" -y /root/dm.pw -m ldap://127.0.0.1:389 -r ldap://127.0.0.1:1389 -b "dc=example,dc=com"

Sample output (trimmed):

Supplier RUV:
  {replicageneration} 6a5d05b4000000010000

Replica RUV:
  {replicageneration} 6a5d05b4000000010000

If {replicageneration} differs after initialization has completed, determine which replica contains the authoritative data and reinitialize the other replica — see initialize and reinitialize replicas. repl-agmt get shows agreement settings such as destination, schedule, protocol, and bind configuration; it does not compare database generation IDs.

Diagnose missing CSNs and changelog gaps

The database RUV, replication changelog, and CSNs work together. A replica that missed updates may report missing CSN errors when the changelog no longer contains the required history.

Investigate:

• Disk-full events that stopped changelog writes
• Crashes or ungraceful shutdowns
• Database reloads or restores from backup
• Changelog retention expiring before a long-offline replica reconnects — see manage the replication changelog
• Updates present on one supplier but not another in multi-supplier layouts

Compare replication get-ruv on suppliers and inspect whether the required CSN still exists in the changelog on an authoritative supplier. A genuine missing update requires reinitialization or recovery from a supplier that still holds the history.

An obsolete replica ID left after permanent supplier removal is different from a missing CSN. That metadata problem belongs in CleanAllRUV, not in changelog gap repair.

Resolve clock skew, replica busy, and growing lag

Fix "Too much time skew"

Large clock differences between topology members can break replication sessions and make CSN comparisons unreliable. Verify NTP or chrony on every host and correct skew before you tune timeouts or reinitialize.

Investigate recurring "Replica busy" errors

An occasional replica busy response can be normal when another replication session holds the replica. The errors log may show:

Unable to acquire replica: error: replica busy locked by conn=111 id=1367 for incremental update

When busy errors repeat and lag increases, look for a supplier monopolizing the replica, an unresponsive consumer, or unsuitable session timing. Retry after a short interval before you treat the condition as a hard failure.

Diagnose increasing replication lag

When Replication Lag Time climbs, check:

• Pending changes waiting to ship
• CPU, disk I/O, and memory pressure on supplier and consumer
• Network latency and packet loss
• Agreement schedules that defer updates
• Downstream replicas that are offline or failing bind/TLS checks
• Whether the changelog still contains the updates required for catch-up

Fix the bottleneck first. Avoid increasing timeout values before you know why updates are delayed.

Verify whether replicas are actually consistent

Agreement status and lag are necessary but not sufficient.

Re-check agreement status after you apply a fix:

dsconf -y /root/dm.pw ldap1 repl-agmt status --suffix "dc=example,dc=com" --bind-dn "cn=Directory Manager" --bind-passwd-file /root/consumer1-dm.pw ldap1-to-consumer1

For topology-wide context, run replication monitor as described in monitor replication and lag.

Compare RUV and max CSN on the supplier:

dsconf -y /root/dm.pw ldap1 replication get-ruv --suffix "dc=example,dc=com"

Repeat on the consumer:

dsconf -y /root/dm.pw ldap2 replication get-ruv --suffix "dc=example,dc=com"

Matching per-origin maximum CSNs are a good sign that the path has caught up. An RUV element belonging to a permanently retired supplier points to the CleanAllRUV workflow; ordinary maximum-CSN differences indicate lag and should not be treated as stale-replica-ID cleanup.

Run a controlled write on the supplier and read it on every expected replica, as in the baseline section.

For entry-level comparison, use ds-replcheck online — see compare replicas with ds-replcheck:

ds-replcheck online -D "cn=Directory Manager" -y /root/dm.pw -m ldap://127.0.0.1:389 -r ldap://127.0.0.1:1389 -b "dc=example,dc=com"

-D and -y supply one bind identity and password for both endpoints. The command works only when that identity has the same valid password on both instances. In this lab, /root/dm.pw satisfies that condition even though routine consumer checks use /root/consumer1-dm.pw.

Sample output when replication state and entries align (trimmed):

Replication State: Supplier and Replica are in perfect synchronization

Result
=====================================================
No replication differences between Supplier and Replica

When entries or attributes under the search base differ, the report lists the mismatches and ends with There are replication differences between Supplier and Replica instead.

Zero lag does not prove identical directory contents when fractional replication excludes attributes or when naming conflicts exist. Account for intentional differences before you reinitialize.

Decide whether to retry, repair, or reinitialize

Finish every recovery with verification in both replication directions when the topology is bidirectional: agreement status on each hop, a controlled update test, RUV comparison, and restoration of the original nsslapd-errorlog-level.

References

• Red Hat Directory Server 13: Troubleshooting replication-related problems
• Red Hat Directory Server 13: Monitoring the replication topology from the command line
• Red Hat Directory Server 13: Solving common replication problems
• Red Hat Directory Server 13: Log files reference
• Red Hat Directory Server 13: Comparing two Directory Server instances
• 389 Project: Replication update status reference

Summary

Troubleshoot 389 Directory Server replication by symptom, capture a baseline before you change configuration, and follow the connection path from supplier to consumer. Treat Error (0) messages as success only when the full text confirms it. Add 8192 to the existing nsslapd-errorlog-level for short debug runs, then restore the original value. Prefer connection and credential repair over reinitialization; when reinitialization is required, choose an authoritative supplier first and verify the fix with agreement status, controlled writes, and ds-replcheck online.