You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ETCD database growing up quite quickly, found that there were no compaction. So, service has been modified to perform autocompaction - start service with those flags: --auto-compaction-retention=10 --auto-compaction-mode=periodic
Now status showing db size 1.4 GB and DB IN USE - only 8MB. It is normal, cause ETCD cluster serves only 2 patroni clusters, 2 nodes in each.
Tried to do manual defrag locally:
etcdctl defrag --endpoints 127.0.0.1:2379 returns Error: context deadline exceeded
When has tried to stop etcd on one member and use etcdutl defrag --data-dir succeeded, but as result gave an error during service start that database has been locked already. lsof shows that db file is locking only by starting service, no other locks. Then I should fix problem by removing and adding member to cluster.
But because of size of database defrag using --cluster flag returns for each node "context deadline exceeded".
And as result cluster getting unhealth status for some time.
On other ETCD cluster with growing problems situation was much better, but anyway, local defrag did not work, returns absolutely the same error: Error: context deadline exceeded
But execution with --cluster key works fine and really decreased size of database 145M -> 16M.
I understood that problem with size of database: is too bug and can't be done during timeout.
Suggested variant with etcdctl defrag --command-timeout=1m - gives the same error, as before. And error returns during ~5 seconds, not after 1 minute. Looks like timeout set by flag has been ignored.
Is there any solution beside re-creating cluster from the scratch?
Is it really, that online defrag works with --cluster flag only for version 3.6?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
ETCD database growing up quite quickly, found that there were no compaction. So, service has been modified to perform autocompaction - start service with those flags: --auto-compaction-retention=10 --auto-compaction-mode=periodic
Now status showing db size 1.4 GB and DB IN USE - only 8MB. It is normal, cause ETCD cluster serves only 2 patroni clusters, 2 nodes in each.
Tried to do manual defrag locally:
etcdctl defrag --endpoints 127.0.0.1:2379 returns Error: context deadline exceeded
When has tried to stop etcd on one member and use etcdutl defrag --data-dir succeeded, but as result gave an error during service start that database has been locked already. lsof shows that db file is locking only by starting service, no other locks. Then I should fix problem by removing and adding member to cluster.
But because of size of database defrag using --cluster flag returns for each node "context deadline exceeded".
And as result cluster getting unhealth status for some time.
On other ETCD cluster with growing problems situation was much better, but anyway, local defrag did not work, returns absolutely the same error: Error: context deadline exceeded
But execution with --cluster key works fine and really decreased size of database 145M -> 16M.
I understood that problem with size of database: is too bug and can't be done during timeout.
Suggested variant with etcdctl defrag --command-timeout=1m - gives the same error, as before. And error returns during ~5 seconds, not after 1 minute. Looks like timeout set by flag has been ignored.
Is there any solution beside re-creating cluster from the scratch?
Is it really, that online defrag works with --cluster flag only for version 3.6?
Beta Was this translation helpful? Give feedback.
All reactions