Skip to content

ipa: avoid ipa-restore and copy whole LDAP database instead - #288

Merged
pbrezina merged 2 commits into
SSSD:masterfrom
pbrezina:ipa-copy-backup
Oct 16, 2026
Merged

ipa: avoid ipa-restore and copy whole LDAP database instead#288
pbrezina merged 2 commits into
SSSD:masterfrom
pbrezina:ipa-copy-backup

Conversation

@pbrezina

@pbrezina pbrezina commented Oct 15, 2026

Copy link
Copy Markdown
Member

ipa-restore --data stops dirsrv, imports data and start dirstv. It
does not do anything else, but it is prone to race conditions inside
389-ds that can render 389-ds unusable and break ipa.

See: 389ds/389-ds-base#7998

Copying the files is easier, should be stable as we do not depend on
any component and as a benefit it is blazingly fast - ~1-2s instead of
~10-20s.

@pbrezina

pbrezina commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

PR CI failures are unrelated. It was already fixed upstream but the nightly copr built is outdated.

@justin-stephenson justin-stephenson self-assigned this Oct 15, 2026
@justin-stephenson
justin-stephenson self-requested a review October 15, 2026 16:45
justin-stephenson
justin-stephenson reviewed Oct 15, 2026
View reviewed changes
Comment thread sssd_test_framework/hosts/ipa.py
systemctl stop dirsrv.target
backup /etc/dirsrv "$path/dirsrv-etc"
backup /var/lib/dirsrv "$path/dirsrv-lib"
systemctl start dirsrv.target

@justin-stephenson justin-stephenson Oct 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there is any downside of this approach (just copying the dirsrv db files/config) compared to the docs recommended dsctl <instance_name> db2bak ?

https://docs.redhat.com/en/documentation/red_hat_directory_server/13/html/management_configuration_and_operations/backing-up-and-restoring-rhds

@pbrezina pbrezina Oct 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should not be any downside and will probably be faster as it literally does no extra operation. So far I did not run into any issue and why would we? Unless we miss some special folder this is just plain restart for dirsrv.

justin-stephenson
justin-stephenson previously approved these changes Oct 15, 2026
View reviewed changes

@justin-stephenson justin-stephenson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added one comment to consider but these changes LGTM still, so ack from me.

@justin-stephenson

justin-stephenson commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Please also check:

$ grep -irn 'ipa-backup' sssd_test_framework/hosts/ipa.py 
43:        ``ipa-backup`` and ``ipa-restore`` commands which can take several
132:        This is done by calling ``ipa-backup --data`` on the server and can

@pbrezina
ipa: avoid ipa-restore and copy whole LDAP database instead
5d1dc07 
ipa-restore --data stops dirsrv, imports data and start dirstv. It
does not do anything else, but it is prone to race conditions inside
389-ds that can render 389-ds unusable and break ipa.

See: 389ds/389-ds-base#7998

Copying the files is easier, should be stable as we do not depend on
any component and as a benefit it is blazingly fast - ~1-2s instead of
~10-20s.
@pbrezina
keycloak: avoid keycloak import/export and copy data instead
6bb134f 
import and export operation takes a long time and from time to time we
hit the following crash. Switching to manually copying data will
hopefully avoid this crash completely and it is also blazing fast.

```
Oct 15 13:41:02 master.keycloak.test kc.sh[14413]: Changes detected in configuration. Updating the server image.
Oct 15 13:41:02 master.keycloak.test kc.sh[14413]: WARNING: Usage of the default value for the db option in the production profile is deprecated. Please explicitly set the db instead.
Oct 15 13:41:02 master.keycloak.test kc.sh[14413]: Updating the configuration and installing your custom providers, if any. Please wait.
Oct 15 13:41:05 master.keycloak.test kc.sh[14413]: 2026-10-15 13:41:05,696 WARN  [org.keycloak.common.Profile] (build-71) Deprecated features identity-brokering-api:v1, twitter-broker:v1 enabled by default. Check the upgrading guide for steps to use later versions if available.
Oct 15 13:41:06 master.keycloak.test kc.sh[14413]: 2026-10-15 13:41:06,347 INFO  [org.freedesktop.dbus.connections.transports.TransportBuilder] (build-71) Using transport dbus-java-transport-native-unixsocket for address unix:path=/var/run/dbus/system_bus_socket
Oct 15 13:41:07 master.keycloak.test kc.sh[14413]: [5.290s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Oct 15 13:41:07 master.keycloak.test kc.sh[14413]: [5.290s][warning][os,thread] Failed to start the native thread for java.lang.Thread "build-168"
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]: ERROR: Failed to run 'start' command.
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]: ERROR: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         [error]: Build step io.quarkus.deployment.steps.ClassTransformingBuildStep#handleClassTransformation threw an exception: java.util.concurrent.RejectedExecutionException
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:20)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor.rejectException(EnhancedQueueExecutor.java:2249)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor.doStartThread(EnhancedQueueExecutor.java:1833)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor.execute(EnhancedQueueExecutor.java:883)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:149)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at io.quarkus.deployment.steps.ClassTransformingBuildStep.handleClassTransformation(ClassTransformingBuildStep.java:215)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:735)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:811)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at io.quarkus.builder.BuildContext.run(BuildContext.java:242)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at java.base/java.lang.Thread.run(Thread.java:1474)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:         Suppressed: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:                 at java.base/java.lang.Thread.start0(Native Method)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:                 at java.base/java.lang.Thread.start(Thread.java:1417)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:                 at org.jboss.threads.JBossThread.start(JBossThread.java:558)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:                 at org.jboss.threads.EnhancedQueueExecutor.doStartThread(EnhancedQueueExecutor.java:1829)
Oct 15 13:41:08 master.keycloak.test kc.sh[14413]:                 ... 13 more
```
@pbrezina

pbrezina commented Oct 15, 2026

Copy link
Copy Markdown
Member Author

Thank Justin, fixed. I also added a commit that does the same with Keycloak to avoid a crash there. It's also much faster then keycloak's import and export as well.

Maybe I will do the same for LDAP. Previsouly, the database was preallocated and had over GB but now with the switch to mdb it has just few MBs.

@aborah-sudo

aborah-sudo commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Looks good to me.

Copying the database files instead of going through ipa-restore avoids depending on IPA tooling for something that is really just a file swap, so the 389-ds race in 389ds/389-ds-base#7998 cannot bite us. Dropping the two retry_command workarounds along with it makes sense, both were specific to ipa-backup/ipa-restore (pagure 9584 and 9669) and have nothing left to guard now.

The speedup matters too, backup and restore run for every test, so 1-2s instead of 10-20s adds up across a full run. Nice that the Keycloak path gets the same treatment and both now look the same.

@pbrezina
pbrezina merged commit ff93df7 into SSSD:master Oct 16, 2026
6 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@justin-stephenson justin-stephenson justin-stephenson approved these changes
@aborah-sudo aborah-sudo aborah-sudo approved these changes
@thalman thalman Awaiting requested review from thalman

Assignees

@thalman thalman

@justin-stephenson justin-stephenson

@aborah-sudo aborah-sudo

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pbrezina @justin-stephenson @aborah-sudo @thalman