What happened?
On Windows, submitting any portable pipeline to a job server fails during artifact staging, before the pipeline runs. The staged filename contains a colon, which is not a legal path character on Windows.
Job server log:
java.util.concurrent.ExecutionException: java.nio.file.InvalidPathException:
Illegal char <:> at index 3: 1-0:ref_Environment_default-submission_environment_dependencies.txt
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 3: 1-0:ref_Environment_default-submission_environment_dependencies.txt
at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:170)
at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
Reproduced on Windows 11, JDK 11, with the Spark job server (--spark-master-url local[4]) and --environment_type=LOOPBACK, SDK and job server both built from the same commit. The staging code is in runners/java-fn-execution and is shared by the portable runners, so I would expect this to affect the other portable runners too, though I only tried Spark.
It is not specific to any pipeline shape
Both of these fail identically and produce no output:
- an unbounded
Read (splittable DoFn)
Create([1, 2, 3]) | Map , a bounded pipeline with no source and no SDF
Across five submissions the job server logged the same Illegal char <:> ten times, so it is deterministic rather than intermittent.
Where the colon comes from
ArtifactStagingService.createFilename builds the name out of the environment id:
return clip(
String.format("%s-%s-%s", idGenerator.getId(), clip(environment, 25), base), 100);For the observed name 1-0:ref_Environment_default-submission_environment_dependencies.txt the parts are id 1, clip(environment, 25) = 0:ref_Environment_default, which is exactly 25 characters, and base submission_environment_dependencies.txt. So the colon comes from the environment id being embedded verbatim.
getDestination then resolves that name against the staging directory, and LocalResourceId rejects it on Windows.
Sanitizing characters that are not legal in a filename before building the name would fix it. Colon is the one that bites here, but *, ?, ", <, > and | are also illegal on Windows.
While reading that method, the splitter just above it also looks off:
// Limit to the last contiguous alpha-numeric sequence. In particular, this will exclude
// all path separators.
List<String> components = Splitter.onPattern("[^A-Za-z-_.]]").splitToList(path);The pattern has a trailing ] outside the character class, so it matches a disallowed character followed by a literal ] rather than any disallowed character. That does not cause this bug, since the staged name is already a bare filename, but it means the comment's intent is not met for the other artifact roles.
Related
#21025 is the same area on Windows but a different failure: an illegal * in removeStagedArtifacts, which happens during cleanup after the job reaches DONE. In that report the pipeline still ran. This one fails staging, so the job never starts.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
On Windows, submitting any portable pipeline to a job server fails during artifact staging, before the pipeline runs. The staged filename contains a colon, which is not a legal path character on Windows.
Job server log:
Reproduced on Windows 11, JDK 11, with the Spark job server (
--spark-master-url local[4]) and--environment_type=LOOPBACK, SDK and job server both built from the same commit. The staging code is inrunners/java-fn-executionand is shared by the portable runners, so I would expect this to affect the other portable runners too, though I only tried Spark.It is not specific to any pipeline shape
Both of these fail identically and produce no output:
Read(splittable DoFn)Create([1, 2, 3]) | Map, a bounded pipeline with no source and no SDFAcross five submissions the job server logged the same
Illegal char <:>ten times, so it is deterministic rather than intermittent.Where the colon comes from
ArtifactStagingService.createFilenamebuilds the name out of the environment id:For the observed name
1-0:ref_Environment_default-submission_environment_dependencies.txtthe parts are id1,clip(environment, 25)=0:ref_Environment_default, which is exactly 25 characters, and basesubmission_environment_dependencies.txt. So the colon comes from the environment id being embedded verbatim.getDestinationthen resolves that name against the staging directory, andLocalResourceIdrejects it on Windows.Sanitizing characters that are not legal in a filename before building the name would fix it. Colon is the one that bites here, but
*,?,",<,>and|are also illegal on Windows.While reading that method, the splitter just above it also looks off:
The pattern has a trailing
]outside the character class, so it matches a disallowed character followed by a literal]rather than any disallowed character. That does not cause this bug, since the staged name is already a bare filename, but it means the comment's intent is not met for the other artifact roles.Related
#21025 is the same area on Windows but a different failure: an illegal
*inremoveStagedArtifacts, which happens during cleanup after the job reaches DONE. In that report the pipeline still ran. This one fails staging, so the job never starts.Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components