Troubleshoot Slow LDAP Searches in 389 Directory Server

Slow LDAP searches in production usually arrive as a timeout in SSSD, a sluggish application login, or an ldapsearch that hangs before results appear. The fix depends on where the delay occurs: the client, the network, the directory search engine, caches, disk, or one replica that drifted from the rest of the topology.

This guide measures one slow query first, separates client elapsed time from server wtime, optime, and etime, reads notes=A and notes=U in the access log, corrects indexes and filters, and compares before-and-after results. Complete index planning stays in configure LDAP indexes; cache sizing stays in LMDB and cache tuning; workload benchmarking stays in performance testing with ldclt and logconv.

Examples use instance ldap1 on ldap1.example.com with suffix dc=example,dc=com and backend userroot. Test directly against an individual server rather than a load-balanced address when you compare access-log timings. If Directory Manager binds return error 49 because the password file is wrong or the secret was lost, confirm the instance hostname first, then follow reset the Directory Manager password.

Causes and fixes at a glance

Establish where the LDAP delay occurs

Measure the slow query from the client with wall-clock timing. The ldapsearch command examples below time the full client experience.

In Bash, use the time keyword with TIMEFORMAT:

TIMEFORMAT='elapsed=%R seconds'
time ldapsearch -LLL -x \
  -H ldap://ldap1.example.com:389 \
  -D "cn=Directory Manager" \
  -y /root/dm.pw \
  -b "dc=example,dc=com" \
  "(&(objectClass=nsOrgPerson)(employeeNumber=10042))" \
  1.1 >/dev/null

Requesting 1.1 returns no attributes while still measuring search and result-entry processing.

Alternatively, use the external time utility:

/usr/bin/time -f 'elapsed=%e seconds' \
ldapsearch -LLL -x \
  -H ldap://ldap1.example.com:389 \
  -D "cn=Directory Manager" \
  -y /root/dm.pw \
  -b "dc=example,dc=com" \
  "(&(objectClass=nsOrgPerson)(employeeNumber=10042))" \
  1.1 >/dev/null

Client wall-clock time includes process startup, DNS resolution, TCP connection establishment, bind processing, the search, result transfer, and client-side handling. The etime on the search RESULT line measures only that server-side LDAP operation. Check the BIND and SEARCH records on the same connection before you attribute the full client delay to the search engine. RHDS defines etime as the interval from receiving an operation until sending its result, while wtime and optime split that individual operation into queue and execution time.

Repeat the same command several times. On a small lab suffix both indexed and unindexed filters can finish in under a second; on production data the gap between a selective indexed filter and a broad unindexed filter is where this measurement matters.

When client elapsed time stays high but the access log shows low etime, investigate outside the search engine before you add indexes.

Find the slow search in the access log

Follow the access log while you reproduce the query:

sudo tail -f /var/log/dirsrv/slapd-ldap1/access

Access logging is buffered by default. If the expected operation does not appear immediately, wait briefly and search again before concluding that the request reached another replica.

Correlate the SEARCH and RESULT records through the same conn and op values. A partially unindexed filter on employeeNumber before an equality index existed produced:

conn=28 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(&(objectClass=nsOrgPerson)(employeeNumber=10042))" attrs="distinguishedName"
conn=28 op=1 RESULT err=0 tag=101 nentries=1 wtime=0.000135833 optime=0.000344278 etime=0.000476964 notes=U details="Partially Unindexed Filter"

After adding an equality index on employeeNumber and reindexing, the same filter returned without notes=U:

conn=45 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(&(objectClass=nsOrgPerson)(employeeNumber=10042))" attrs="distinguishedName"
conn=45 op=1 RESULT err=0 tag=101 nentries=1 wtime=0.000126403 optime=0.000574154 etime=0.000698428

Practical patterns:

• High optime: filter, index, cache, database, plug-in, or result-generation cost
• High wtime: the operation waited for a worker thread
• Low server etime but high client time: investigate network, TLS, DNS, or client processing
• High nentries: result size may dominate the cost

Analyze a longer window on busy servers:

sudo logconv.pl /var/log/dirsrv/slapd-ldap1/access

Sample output (trimmed):

Access Log Analyzer 8.2
Total Log Lines Analysed:  4069
Searches:                      820           (0.08/sec)  (4.67/min)

Use logconv.pl -S and -E for a time range and -T for a minimum etime threshold. See logconv.pl --help for the exact syntax on your build. Deeper reporting workflows are in performance testing with ldclt and logconv.

Find unindexed and partially indexed searches

Search the access log for unindexed operations:

sudo grep -E 'notes=.*(A|U)' /var/log/dirsrv/slapd-ldap1/access

Example filter that logged notes=U before indexing:

(&(objectClass=nsOrgPerson)(employeeNumber=10042))

List current indexes on the backend:

dsconf ldap1 backend index list userroot

For a compound filter such as (&(objectClass=nsOrgPerson)(employeeNumber=10042)), employeeNumber might have an equality index while another filter term is unindexed or exceeds its ID-list limit. RHDS records notes=U when at least one search term is unindexed, when the operation exceeds nsslapd-idlistscanlimit, when no index file exists, or when the available index is not configured for the filter type.

Match the index type to the filter:

notes=U does not always mean the attribute has no index. It can also appear when the candidate ID list exceeds nsslapd-idlistscanlimit or when another component of a compound filter remains unindexed. See configure LDAP search limits before you raise the limit or add more indexes. Custom matching-rule and schema changes belong in the indexes chapter, not in this troubleshooting flow.

Add the correct index and rebuild it

List current indexes before you change anything:

dsconf ldap1 backend index list userroot

This prevents running index add against an index entry that already exists.

Configure and rebuild the index separately on every server where the search can run.

When no index entry exists for the attribute, add one and reindex:

dsconf ldap1 backend index add \
  --attr employeeNumber \
  --index-type eq \
  --reindex \
  userroot

When an index entry exists but lacks the required type, add the type and reindex the attribute:

dsconf ldap1 backend index set \
  --attr employeeNumber \
  --add-type eq \
  userroot

dsconf ldap1 backend index reindex \
  --attr employeeNumber \
  --wait \
  userroot

Sample output from a new equality index:

Index task index_attrs_2026-07-22T09:43:25.479450 completed successfully
Successfully added index

Add a substring index only when the application truly needs middle or trailing wildcards:

dsconf ldap1 backend index add \
  --attr displayName \
  --index-type sub \
  --reindex \
  userroot

Verify the index definition:

dsconf ldap1 backend index list userroot

Repeat the original search and confirm that:

• notes=A or notes=U disappears where appropriate
• optime and etime decrease on representative data
• The result set remains correct

Practical cautions:

• Do not index every attribute
• Add only the index types your real filters use
• Substring indexes increase storage and write overhead
• Do not add substring indexes to member or uniqueMember as a generic performance fix
• Reindex after creating or changing an index
• Test indexes independently on every replica

Fix expensive LDAP filters and large results

Even an indexed search can be slow when it produces a very large candidate list or result set.

Compare a broad filter:

(objectClass=nsOrgPerson)

with a selective filter:

(&(objectClass=nsOrgPerson)(uid=user1))

Check for:

• Searches beginning at an unnecessarily broad base DN
• Subtree scope when one-level or base scope is sufficient
• Leading and middle wildcards such as (cn=*admin*)
• Large OR filters or presence filters that match most entries
• Applications requesting every attribute
• Clients repeatedly performing the same broad search

Request only the attributes the application needs:

ldapsearch -LLL -x \
  -H ldap://ldap1.example.com:389 \
  -D "cn=Directory Manager" \
  -y /root/dm.pw \
  -b "ou=People,dc=example,dc=com" \
  "(uid=user1)" \
  uid cn mail

For large result sets, test simple paged results:

ldapsearch -LLL -x \
  -H ldap://ldap1.example.com:389 \
  -D "cn=Directory Manager" \
  -y /root/dm.pw \
  -b "dc=example,dc=com" \
  -E pr=500/noprompt \
  "(objectClass=nsOrgPerson)" \
  uid

Paging controls result delivery but does not repair an inefficient filter.

Check caches and server load

Run the same query twice and compare access-log optime and etime. A pattern such as a multi-second first search and a sub-second repeat often indicates cache warming rather than a missing index.

Inspect cache statistics:

dsconf ldap1 monitor dbmon

Sample output (trimmed):

Normalized DN Cache:
 - Cache Hit Ratio:     83%
Entry Cache Hit Ratio:        80%
Entry Cache Count:            10

Review entry-cache and DN-cache hit ratios, current use, and whether the working dataset fits available memory. These ratios are cumulative since the instance started. Treat cache hit ratios as cumulative context rather than proof for one slow request. Compare the first and second query optime, then use dbmon to determine whether the broader workload also shows poor cache efficiency. Do not increase cache sizes immediately. First determine which cache has a poor hit ratio, whether the host is swapping, and whether automatic cache sizing is enabled. Cache-sizing commands belong in LMDB and cache tuning.

Check server health and queueing:

dsconf ldap1 monitor server

Sample output (trimmed):

currentconnectionsatmaxthreads: 0
maxthreadsperconnhits: 0
opsinitiated: 456
opscompleted: 455

dsctl ldap1 healthcheck

During a slow query, watch CPU, disk, and memory on the directory server host with tools such as pidstat, iostat, and vmstat. High wtime means the operation waited in the server work queue before a worker thread processed it. Check concurrent workload, CPU and disk pressure, currentconnectionsatmaxthreads, and maxthreadsperconnhits. Do not treat readwaiters as worker-queue depth; it counts threads waiting to read client data. Do not raise nsslapd-threadnumber merely because searches are slow. Thread tuning is covered in configure threads and connections.

Investigate large groups, plugins, and replica differences

Large group searches

Test membership filters such as:

(&(objectClass=groupOfUniqueNames)(uniqueMember=uid=user1,ou=People,dc=example,dc=com))

A uniqueMember equality index is normally required, but searches can still be expensive when groups contain very large multi-valued membership attributes. Check member count per group, cache warmth, and whether memberOf can satisfy the practical lookup more efficiently. Complete large-group design is in large-group performance.

Plugin and virtual searches

Internal or application-driven searches may involve MemberOf, Referential Integrity, roles, Class of Service, views, Auto Membership, or dynamic groups. dsctl ldap1 healthcheck can flag missing indexes used by plug-in operations and problematic membership index choices.

One replica is slower

On every server, compare:

dsconf ldap1 backend index list userroot

dsconf ldap1 monitor dbmon

Also compare the exact search filter, entry count, plug-in configuration, available memory, disk latency, 389 DS version, background tasks, and replication state. Identical replicated data does not guarantee identical performance configuration.

Verify the improvement

Repeat the same query with the same client, server, bind DN, base, scope, filter, and requested attributes. Record before-and-after values:

The timing columns illustrate a typical indexed fix on a larger suffix. On the lab host the same filter showed notes=U before indexing and no notes flag after indexing, with sub-millisecond server etime in both cases.

Also test:

• Cold and warm cache behaviour
• Normal and peak load
• Every replica in the topology
• The original application, not only ldapsearch

Troubleshooting

References

• 389 Directory Server — Access log reference
• Red Hat Directory Server 13 — Log files
• Red Hat Directory Server 13 — Monitoring database and cache use
• Red Hat Directory Server 13 — Tuning cache settings
• dsconf backend index commands

Summary

When LDAP searches are slow in 389 Directory Server, measure client elapsed time, reproduce the query while you read the access log, and compare wtime, optime, etime, and notes. Treat notes=A and notes=U as signals to fix indexes, filters, or ID-list scan limits rather than as reasons to raise threads or cache sizes blindly. Verify improvements with the same client, server, and filter on every replica and under both cold and warm cache conditions.