Skip to content

[data] Prevent download deadlock on wide directory trees#287

Merged
ldmonster merged 3 commits intomainfrom
fix/data-download-deadlock
Mar 3, 2026
Merged

[data] Prevent download deadlock on wide directory trees#287
ldmonster merged 3 commits intomainfrom
fix/data-download-deadlock

Conversation

@kkozoriz
Copy link
Contributor

Description

Problem

recursiveDownload used a blocking semaphore acquire (sem <- struct{}{}) before
launching each child goroutine. When the directory tree had more direct children
than the semaphore capacity (default: 10), all running goroutines blocked trying
to schedule the next level while holding their semaphore slot - causing a deadlock.

Root Cause

// Before - blocks unconditionally, causing deadlock on wide trees
sem <- struct{}{}
wg.Add(1)
	go func(sp string) {
		defer func() { <-sem; wg.Done() }()
		subErr := recursiveDownload(ctx, sClient, log, sem, url, srcPath+sp, filepath.Join(dstPath, sp))
		if subErr != nil {
			mu.Lock()
			if firstErr == nil {
				firstErr = fmt.Errorf("Download %s: %w", filepath.Join(srcPath, sp), subErr)
			}
			mu.Unlock()		
        }
	}(subPath)

Every active goroutine tried to acquire a new semaphore slot before yielding its
own, which deadlocked as soon as all 10 slots were occupied by tasks that each
needed a free slot to proceed.

Fix

Replace the blocking send with a non-blocking select:

  • if a semaphore slot is available → run child in a new goroutine;
  • if the semaphore is full → run child synchronously in the current goroutine.

Before fix

image

After fix

image image

Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
@kkozoriz kkozoriz requested a review from ldmonster as a code owner February 25, 2026 18:33
@kkozoriz kkozoriz changed the title [data]: prevent download deadlock on wide directory trees [data]: Prevent download deadlock on wide directory trees Feb 25, 2026
@kkozoriz kkozoriz self-assigned this Feb 26, 2026
@kkozoriz kkozoriz changed the title [data]: Prevent download deadlock on wide directory trees [data] Prevent download deadlock on wide directory trees Feb 26, 2026
AleksZimin
AleksZimin previously approved these changes Feb 26, 2026
@kkozoriz kkozoriz requested a review from kneumoin February 26, 2026 08:20
kkozoriz added 2 commits March 2, 2026 18:36
Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
@ldmonster ldmonster merged commit e80a417 into main Mar 3, 2026
5 checks passed
@ldmonster ldmonster deleted the fix/data-download-deadlock branch March 3, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants