Can "unknown blob" errors be caused by testcontainers?

[bjansen]

Hi,

We're experiencing the following error quite frequently on our CI servers since last month (full stack trace at the end of this post):

Can't get Docker image: RemoteDockerImage(imageName=artifactory.local/confluentinc/cp-kafka:7.5.2, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@2381a972)
...
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: error pulling image configuration: download failed after attempts=1: unknown blob

Context: we have a Jenkins infrastructure consisting of multiple workers. Each worker can run concurrent builds in "docker-in-docker" mode, meaning Jenkins pipelines spin their own Docker container to run maven builds. These builds are run using -T1C to build maven child modules in parallel, so in theory there's a chance that testcontainers might download the same image from multiple builds and/or multiple maven child modules at the same time.

A docker system prune will usually remove broken layers and force a re-download during the next build, but after a few hours/days the problem comes back again and again. It happens on multiple images (ryuk, kafka, postgresql…), and images in different versions (postgresql 16.4, 16.8). It will eventually happen on any of our 8 workers.

Pulling these images from one of our development machine always succeeds, suggesting that the corruption is not in our internal Artifactory instance, only in local caches in Jenkins workers.

From what I understand, a standard "docker pull" prevents cache corruption when run multiple times in parallel. Copilot and Gemini both seem to suggest that testcontainers works differently and is more "aggressive" when pulling images. It also states that we should rather use docker pull commands at the beginning of our build before starting testcontainers, which is not ideal since different projects will use different images in different versions.

I can't really find anything regarding unknown blobs and testcontainers using a google search. I'm wondering if this kind of problems rings a bell for anyone in this community. Help would be really appreciated :)

Full stack trace:

java.util.concurrent.ExecutionException: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=artifactory.local/confluentinc/cp-kafka:7.5.2, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@2381a972)
	at java.base/java.util.concurrent.CompletableFuture.wrapInExecutionException(CompletableFuture.java:345)
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:440)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2094)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:314)
	at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:276)
	at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:263)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$startContainers$4(TestcontainersExtension.java:83)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$startContainers$5(TestcontainersExtension.java:83)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.startContainers(TestcontainersExtension.java:83)
	at org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:57)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
	Suppressed: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=artifactory.local/postgres:15.8, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@2381a972)
		at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1308)
		at org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizerFactory.createSource(ServiceConnectionContextCustomizerFactory.java:90)
		at org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizerFactory.lambda$collectSources$1(ServiceConnectionContextCustomizerFactory.java:65)
		at org.springframework.core.annotation.TypeMappedAnnotations$AggregatesSpliterator.tryAdvance(TypeMappedAnnotations.java:591)
		at org.springframework.core.annotation.TypeMappedAnnotations$AggregatesSpliterator.tryAdvance(TypeMappedAnnotations.java:558)
		at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332)
		at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:803)
		at org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizerFactory.lambda$collectSources$0(ServiceConnectionContextCustomizerFactory.java:65)
		at org.springframework.util.ReflectionUtils.doWithLocalFields(ReflectionUtils.java:691)
		at org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizerFactory.collectSources(ServiceConnectionContextCustomizerFactory.java:62)
		at org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizerFactory.collectSources(ServiceConnectionContextCustomizerFactory.java:73)
		at org.springframework.boot.testcontainers.service.connection.ServiceConnectionContextCustomizerFactory.createContextCustomizer(ServiceConnectionContextCustomizerFactory.java:54)
		at org.springframework.test.context.support.AbstractTestContextBootstrapper.getContextCustomizers(AbstractTestContextBootstrapper.java:420)
		at org.springframework.test.context.support.AbstractTestContextBootstrapper.buildMergedContextConfiguration(AbstractTestContextBootstrapper.java:393)
		at org.springframework.test.context.support.AbstractTestContextBootstrapper.buildDefaultMergedContextConfiguration(AbstractTestContextBootstrapper.java:268)
		at org.springframework.test.context.support.AbstractTestContextBootstrapper.buildMergedContextConfiguration(AbstractTestContextBootstrapper.java:226)
		at org.springframework.test.context.support.AbstractTestContextBootstrapper.buildTestContext(AbstractTestContextBootstrapper.java:117)
		at org.springframework.boot.test.context.SpringBootTestContextBootstrapper.buildTestContext(SpringBootTestContextBootstrapper.java:102)
		at org.springframework.test.context.TestContextManager.<init>(TestContextManager.java:142)
		at org.springframework.test.context.TestContextManager.<init>(TestContextManager.java:126)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
		at org.springframework.test.context.junit.jupiter.SpringExtension.getTestContextManager(SpringExtension.java:466)
		at org.springframework.test.context.junit.jupiter.SpringExtension.afterAll(SpringExtension.java:220)
		... 1 more
	Caused by: org.testcontainers.containers.ContainerFetchException: Failed to get Docker client for artifactory.local/postgres:15.8
		at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:128)
		at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:35)
		at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
		at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
		at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1306)
		... 23 more
	Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: error pulling image configuration: download failed after attempts=1: unknown blob
		at com.github.dockerjava.api.command.PullImageResultCallback.checkDockerClientPullSuccessful(PullImageResultCallback.java:97)
		at com.github.dockerjava.api.command.PullImageResultCallback.throwFirstError(PullImageResultCallback.java:112)
		at com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:93)
		at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.awaitCompletion(TimeLimitedLoggedPullImageResultCallback.java:58)
		at org.testcontainers.images.RemoteDockerImage.pullImage(RemoteDockerImage.java:167)
		at org.testcontainers.images.RemoteDockerImage.lambda$tryImagePullCommand$1(RemoteDockerImage.java:142)
		at org.testcontainers.shaded.org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
		at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
		at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
		at org.testcontainers.shaded.org.awaitility.core.SameThreadExecutorService.execute(SameThreadExecutorService.java:79)
		at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:149)
		at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:101)
		at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
		at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
		at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
		at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
		at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:109)
		... 27 more
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=artifactory.local/confluentinc/cp-kafka:7.5.2, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@2381a972)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1308)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:346)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:808)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:503)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: org.testcontainers.containers.ContainerFetchException: Failed to get Docker client for artifactory.local/confluentinc/cp-kafka:7.5.2
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:128)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:35)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1306)
	... 7 more
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: error pulling image configuration: download failed after attempts=1: unknown blob
	at com.github.dockerjava.api.command.PullImageResultCallback.checkDockerClientPullSuccessful(PullImageResultCallback.java:97)
	at com.github.dockerjava.api.command.PullImageResultCallback.throwFirstError(PullImageResultCallback.java:112)
	at com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:93)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.awaitCompletion(TimeLimitedLoggedPullImageResultCallback.java:58)
	at org.testcontainers.images.RemoteDockerImage.pullImage(RemoteDockerImage.java:167)
	at org.testcontainers.images.RemoteDockerImage.lambda$tryImagePullCommand$1(RemoteDockerImage.java:142)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99)
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at org.testcontainers.shaded.org.awaitility.core.SameThreadExecutorService.execute(SameThreadExecutorService.java:79)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:149)
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:101)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:109)
	... 11 more