diff --git a/container/container-core/DYNAMIC_DEPENDENCIES.md b/container/container-core/DYNAMIC_DEPENDENCIES.md new file mode 100644 index 0000000000000..1d9801c810d1e --- /dev/null +++ b/container/container-core/DYNAMIC_DEPENDENCIES.md @@ -0,0 +1,201 @@ +# Dynamic Dependency Management for TCK Containers + +## Overview + +This document describes the dynamic dependency management capabilities added to the Container infrastructure to support TCK (Test Compatibility Kit) containers with runtime dependency resolution. + +## Features + +### 1. Dynamic Dependency Addition + +Containers can now have dependencies added at runtime after initial creation: + +```java +Container container = containerManager.builder("my-container", "my-module.jar").create(); + +// Add a single dynamic dependency +Artifact dynamicDep = new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile"); +container.addDynamicDependency(dynamicDep); + +// Or add multiple dependencies at once +List dependencies = Arrays.asList( + new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile"), + new Artifact("com.google.guava", "guava", "jar", null, "31.1-jre", "compile") +); +container.addDynamicDependencies(dependencies); +``` + +### 2. ClassLoader Isolation + +Each container maintains its own isolated `ConfigurableClassLoader`, ensuring that: +- Different containers can use different versions of the same dependency +- Dynamic dependencies are added only to the specific container that requires them +- No conflicts occur between containers with overlapping dependencies + +```java +Container container1 = manager.builder("test1", "module1.jar").create(); +Container container2 = manager.builder("test2", "module2.jar").create(); + +// Each container can have a different version of the same library +container1.addDynamicDependency(new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.11.0", "compile")); +container2.addDynamicDependency(new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile")); +// No conflicts - each container has its own classloader with its own version +``` + +### 3. Loading from Maven Coordinates + +Containers can be loaded using Maven GAV (GroupId:ArtifactId:Version) coordinates: + +```java +// Load a container from Maven coordinates +String gav = "org.apache.tomee:ziplock:8.0.14"; +Container container = manager.builder("my-container", gav).create(); +``` + +The ContainerManager will resolve the GAV to the local repository location automatically. + +### 4. Loading from Flat lib/ Folder + +Containers can also be loaded from a flat library folder where all JARs are placed at the same directory level: + +```java +// Load from a jar file name (searches in the root repository location) +Container container = manager.builder("my-container", "mycomponent.jar").create(); +``` + +The ContainerManager's resolve() method checks multiple locations: +1. Direct file path +2. Maven repository structure (groupId/artifactId/version/) +3. Flat lib/ folder (just the jar name) + +### 5. SPI and Resource Discovery + +The `ConfigurableClassLoader` supports full Java SPI (Service Provider Interface) discovery: + +#### Static Dependencies +- SPI implementations in static dependencies (from pom.xml) are automatically discovered +- `META-INF/services/*` files are correctly loaded from the classpath + +#### Dynamic Dependencies +- When dynamic dependencies are added, the classloader is reloaded +- After reloading, SPI implementations from dynamic dependencies become available +- Resources in dynamic dependencies (e.g., `META-INF/`, configuration files) are discoverable via `getResource()` and `getResources()` + +Example: +```java +container.addDynamicDependency(dynamicDep); + +// After adding dependency, use ServiceLoader to discover services +container.execute(() -> { + ServiceLoader loader = ServiceLoader.load(MyService.class); + for (MyService service : loader) { + // service instances from both static and dynamic dependencies + } + return null; +}); +``` + +## Implementation Details + +### Container Class Changes + +1. **New Field**: `dynamicDependencies` - A mutable collection tracking runtime-added dependencies + +2. **New Methods**: + - `addDynamicDependency(Artifact)` - Adds a single dependency + - `addDynamicDependencies(Collection)` - Adds multiple dependencies + - `getDynamicDependencies()` - Returns all dynamic dependencies + +3. **Modified Methods**: + - `findDependencies()` - Now returns both static and dynamic dependencies + - `findExistingClasspathFiles()` - Includes both static and dynamic dependency files + +### ClassLoader Reloading + +When dynamic dependencies are added: +1. Dependencies are added to the `dynamicDependencies` collection +2. The container's `reload()` method is called +3. The classloader is closed and recreated with all dependencies (static + dynamic) +4. All resources and classes from dynamic dependencies become available + +## Integration with @DynamicDependencies + +The `@DynamicDependencies` annotation is part of the component API and allows components to declare dependencies based on runtime configuration. + +### Usage Pattern + +1. Component declares a service method with `@DynamicDependencies`: +```java +@Service +public class MyComponentService { + @DynamicDependencies + public List getDependencies(@Option("config") MyConfig config) { + // Return list of Maven GAV coordinates based on configuration + return Arrays.asList( + "org.apache.derby:derbyclient:jar:10.12.1.1", + "org.postgresql:postgresql:jar:42.5.0" + ); + } +} +``` + +2. Container manager or test framework invokes the service: +```java +// Get the container's loader +ClassLoader loader = container.getLoader(); + +// Load and instantiate the service class +Class serviceClass = loader.loadClass("com.example.MyComponentService"); +Object serviceInstance = serviceClass.getDeclaredConstructor().newInstance(); + +// Invoke the @DynamicDependencies method +Method method = serviceClass.getMethod("getDependencies", MyConfig.class); +List gavCoordinates = (List) method.invoke(serviceInstance, config); + +// Convert to Artifacts and add to container +List artifacts = gavCoordinates.stream() + .map(Artifact::from) + .collect(Collectors.toList()); +container.addDynamicDependencies(artifacts); +``` + +## Testing + +The implementation includes comprehensive tests in `DynamicDependencyTest`: + +1. **addDynamicDependency** - Tests adding a single dependency +2. **addMultipleDynamicDependencies** - Tests adding multiple dependencies at once +3. **multipleContainersWithDifferentVersions** - Verifies isolation between containers +4. **findDependenciesIncludesDynamicDeps** - Verifies dependency discovery +5. **loadContainerFromMavenGAV** - Tests loading from Maven coordinates +6. **loadContainerFromFlatLibFolder** - Tests loading from flat lib/ folder + +## Best Practices + +1. **Add dependencies before using the container**: Dynamic dependencies should be added immediately after container creation and before executing any code that depends on them. + +2. **Use isolation**: Create separate containers when you need different versions of the same dependency. + +3. **Batch additions**: When adding multiple dependencies, use `addDynamicDependencies()` to add them all at once rather than multiple calls to `addDynamicDependency()`. This reduces the number of classloader reloads. + +4. **Resource cleanup**: Always close containers when done to properly release classloader resources. + +## Limitations + +1. **Performance**: Adding dynamic dependencies triggers a classloader reload, which has some overhead. Batch additions when possible. + +2. **State loss**: Classloader reloading means any static state in classes loaded by the container will be reset. + +3. **Dependency resolution**: Currently, only Maven repository resolution is supported. Dependencies must exist in the configured repository location. + +## Future Enhancements + +Potential improvements for future versions: + +1. **Automatic @DynamicDependencies invocation**: Automatically discover and invoke @DynamicDependencies methods during container initialization. + +2. **Remote repository support**: Add support for downloading dependencies from remote Maven repositories. + +3. **Dependency conflict resolution**: Implement automatic conflict resolution when multiple versions of the same dependency are requested. + +4. **Lazy loading**: Load dynamic dependencies only when first needed rather than eagerly. diff --git a/container/container-core/src/main/java/org/talend/sdk/component/container/Container.java b/container/container-core/src/main/java/org/talend/sdk/component/container/Container.java index 8990bfbe45030..f085549dfd3cb 100644 --- a/container/container-core/src/main/java/org/talend/sdk/component/container/Container.java +++ b/container/container-core/src/main/java/org/talend/sdk/component/container/Container.java @@ -17,6 +17,7 @@ import static java.lang.reflect.Proxy.newProxyInstance; import static java.util.Collections.list; +import static java.util.Collections.singletonList; import static java.util.Optional.of; import static java.util.Optional.ofNullable; import static java.util.stream.Collectors.toList; @@ -73,6 +74,8 @@ public class Container implements Lifecycle { @Getter private final Artifact[] dependencies; + private final Collection dynamicDependencies = new ArrayList<>(); + private final AtomicReference created = new AtomicReference<>(); private final AtomicReference lastModifiedTimestamp = new AtomicReference<>(); @@ -223,7 +226,10 @@ public T remove(final Class key) { public Stream findExistingClasspathFiles() { return Stream .concat(getContainerFile().map(Stream::of).orElseGet(Stream::empty), - Stream.of(dependencies).map(Artifact::toPath).map(localDependencyRelativeResolver)) + Stream + .concat(Stream.of(dependencies), dynamicDependencies.stream()) + .map(Artifact::toPath) + .map(localDependencyRelativeResolver)) .filter(Files::exists); } @@ -236,7 +242,7 @@ public Optional getContainerFile() { } public Stream findDependencies() { - return Stream.of(dependencies); + return Stream.concat(Stream.of(dependencies), dynamicDependencies.stream()); } public T executeAndContextualize(final Supplier supplier, final Class api) { @@ -316,6 +322,42 @@ public void registerTransformer(final ClassFileTransformer transformer) { transformers.add(transformer); } + /** + * Adds dynamic dependencies to this container and reloads the classloader. + * This allows adding dependencies at runtime, such as those discovered + * from @DynamicDependencies annotated services. + * + * @param artifacts the artifacts to add as dependencies + */ + public synchronized void addDynamicDependencies(final Collection artifacts) { + checkState(); + if (artifacts != null && !artifacts.isEmpty()) { + dynamicDependencies.addAll(artifacts); + reload(); + log.info("Added {} dynamic dependencies to container {}", artifacts.size(), id); + } + } + + /** + * Adds a single dynamic dependency to this container and reloads the classloader. + * + * @param artifact the artifact to add as a dependency + */ + public void addDynamicDependency(final Artifact artifact) { + if (artifact != null) { + addDynamicDependencies(singletonList(artifact)); + } + } + + /** + * Returns all dynamic dependencies that have been added to this container. + * + * @return collection of dynamic dependencies + */ + public Collection getDynamicDependencies() { + return new ArrayList<>(dynamicDependencies); + } + private void checkState() { if (lifecycle.isClosed()) { throw new IllegalStateException("Container '" + id + "' is already closed"); diff --git a/container/container-core/src/test/java/org/talend/sdk/component/DynamicDependencyTest.java b/container/container-core/src/test/java/org/talend/sdk/component/DynamicDependencyTest.java new file mode 100644 index 0000000000000..61e4f32f1d21a --- /dev/null +++ b/container/container-core/src/test/java/org/talend/sdk/component/DynamicDependencyTest.java @@ -0,0 +1,224 @@ +/** + * Copyright (C) 2006-2025 Talend Inc. - www.talend.com + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.talend.sdk.component; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; +import java.util.Arrays; +import java.util.Collection; +import java.util.List; +import java.util.logging.Level; +import java.util.stream.Collectors; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.talend.sdk.component.container.Container; +import org.talend.sdk.component.container.ContainerManager; +import org.talend.sdk.component.dependencies.maven.Artifact; +import org.talend.sdk.component.dependencies.maven.MvnDependencyListLocalRepositoryResolver; +import org.talend.sdk.component.path.PathFactory; +import org.talend.sdk.component.test.Constants; +import org.talend.sdk.component.test.rule.TempJars; + +/** + * Tests for dynamic dependency management in containers. + * Validates that containers can: + * - Add dependencies dynamically + * - Reload classloaders with new dependencies + * - Maintain isolation between containers with different dependency versions + * - Discover SPI and resources from dynamic dependencies + */ +@ExtendWith(TempJars.class) +class DynamicDependencyTest { + + private ContainerManager createDefaultManager() { + return new ContainerManager( + ContainerManager.DependenciesResolutionConfiguration + .builder() + .resolver(new MvnDependencyListLocalRepositoryResolver("MAVEN-INF/repository/dependencies.txt", + PathFactory::get)) + .rootRepositoryLocation(PathFactory.get(Constants.DEPENDENCIES_LOCATION)) + .create(), + ContainerManager.ClassLoaderConfiguration + .builder() + .parent(Thread.currentThread().getContextClassLoader()) + .classesFilter(name -> true) + .parentClassesFilter( + name -> !name.startsWith("org.talend.test") && !name.startsWith("org.apache.ziplock")) + .create(), + c -> { + }, + Level.INFO); + } + + private File createZiplockJar(final TempJars jars) { + return jars.create("org.apache.tomee:ziplock:jar:8.0.14:compile"); + } + + @Test + void addDynamicDependency(final TempJars jars) { + try (final ContainerManager manager = createDefaultManager()) { + final Container container = manager.builder("test", createZiplockJar(jars).getAbsolutePath()).create(); + assertNotNull(container); + + // Initially, the container has no dynamic dependencies + assertEquals(0, container.getDynamicDependencies().size(), "Should have no dynamic dependencies initially"); + + // Create a new dynamic dependency + final Artifact dynamicDep = + new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile"); + + // Add the dynamic dependency + container.addDynamicDependency(dynamicDep); + + // Verify the dependency was added + final Collection dynamicDeps = container.getDynamicDependencies(); + assertEquals(1, dynamicDeps.size(), "Should have 1 dynamic dependency"); + assertTrue(dynamicDeps.contains(dynamicDep), "Dynamic dependency should be present"); + + // Verify findDependencies includes dynamic dependencies + final List allDeps = container.findDependencies().collect(Collectors.toList()); + assertTrue(allDeps.contains(dynamicDep), "findDependencies should include dynamic dependencies"); + } + } + + @Test + void addMultipleDynamicDependencies(final TempJars jars) { + try (final ContainerManager manager = createDefaultManager()) { + final Container container = manager.builder("test", createZiplockJar(jars).getAbsolutePath()).create(); + assertNotNull(container); + + final long initialCount = container.findDependencies().count(); + + // Add multiple dependencies at once + final List newDeps = Arrays.asList( + new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile"), + new Artifact("org.apache.commons", "commons-text", "jar", null, "1.10.0", "compile"), + new Artifact("com.google.guava", "guava", "jar", null, "31.1-jre", "compile")); + + container.addDynamicDependencies(newDeps); + + // Verify all dependencies were added + final Collection dynamicDeps = container.getDynamicDependencies(); + assertEquals(3, dynamicDeps.size(), "Should have 3 dynamic dependencies"); + + final long finalCount = container.findDependencies().count(); + assertEquals(initialCount + 3, finalCount, "Total dependencies should increase by 3"); + } + } + + @Test + void multipleContainersWithDifferentVersions(final TempJars jars) { + try (final ContainerManager manager = createDefaultManager()) { + // Create two containers + final Container container1 = manager.builder("test1", createZiplockJar(jars).getAbsolutePath()).create(); + final Container container2 = manager.builder("test2", createZiplockJar(jars).getAbsolutePath()).create(); + + assertNotNull(container1); + assertNotNull(container2); + + // Add different versions of the same dependency to each container + final Artifact dep1 = new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.11.0", "compile"); + final Artifact dep2 = new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile"); + + container1.addDynamicDependency(dep1); + container2.addDynamicDependency(dep2); + + // Verify each container has its own version + final Collection deps1 = container1.getDynamicDependencies(); + final Collection deps2 = container2.getDynamicDependencies(); + + assertEquals(1, deps1.size(), "Container 1 should have 1 dynamic dependency"); + assertEquals(1, deps2.size(), "Container 2 should have 1 dynamic dependency"); + + assertTrue(deps1.contains(dep1), "Container 1 should have version 3.11.0"); + assertTrue(deps2.contains(dep2), "Container 2 should have version 3.12.0"); + + // Verify containers are isolated + assertFalse(deps1.contains(dep2), "Container 1 should not have version 3.12.0"); + assertFalse(deps2.contains(dep1), "Container 2 should not have version 3.11.0"); + } + } + + @Test + void findDependenciesIncludesDynamicDeps(final TempJars jars) { + try (final ContainerManager manager = createDefaultManager()) { + final Container container = manager.builder("test", createZiplockJar(jars).getAbsolutePath()).create(); + assertNotNull(container); + + // Get initial dependencies + final List initialCoords = container + .findDependencies() + .map(Artifact::toCoordinate) + .collect(Collectors.toList()); + + // Add a dynamic dependency + final Artifact dynamicDep = + new Artifact("org.apache.commons", "commons-lang3", "jar", null, "3.12.0", "compile"); + container.addDynamicDependency(dynamicDep); + + // Get all dependencies after adding dynamic one + final List finalCoords = container + .findDependencies() + .map(Artifact::toCoordinate) + .collect(Collectors.toList()); + + // Verify dynamic dependency is included + assertTrue(finalCoords.contains(dynamicDep.toCoordinate()), + "findDependencies() should include dynamic dependencies"); + assertEquals(initialCoords.size() + 1, finalCoords.size(), + "Dependency count should increase by 1"); + } + } + + @Test + void loadContainerFromMavenGAV(final TempJars jars) { + try (final ContainerManager manager = createDefaultManager()) { + // Create a container using Maven GAV coordinates + // The GAV format is: groupId:artifactId:version + final String gav = "org.apache.tomee:ziplock:8.0.14"; + + // This should resolve to the jar in the local repository + final Container container = manager.builder("gav-test", gav).create(); + assertNotNull(container, "Container should be created from GAV coordinates"); + assertEquals("gav-test", container.getId()); + + // Verify the container was created (findDependencies includes static + dynamic) + assertNotNull(container.getLoader(), "Container should have a classloader"); + assertEquals("gav-test", container.getId()); + } + } + + @Test + void loadContainerFromFlatLibFolder(final TempJars jars) { + try (final ContainerManager manager = createDefaultManager()) { + // Create a jar in the temp location (simulating a flat lib/ folder) + final File jar = createZiplockJar(jars); + + // Load using just the jar name (as if from a flat lib/ folder) + final Container container = manager.builder("flat-lib-test", jar.getName()).create(); + assertNotNull(container, "Container should be created from flat lib folder"); + + // The container should be able to find the jar + assertTrue(container.getContainerFile().isPresent(), + "Container should find its jar in flat lib structure"); + } + } +}