NIFI-16243 Added DetectBase64Content Processor - #11583
Conversation
Determines whether FlowFile content is Base64 encoded by inspecting the characters instead of attempting a decode, avoiding the false failures that occur when a decode attempt is used as a detection mechanism. The result is written as true or false to a configurable attribute and the FlowFile is always routed to success, allowing downstream branching with RouteOnAttribute. Detection Scope selects between reading the entire content for a definitive result and reading a configurable Sample Size for a bounded check on large content. Content is streamed in both cases, so heap usage remains constant regardless of FlowFile size.
exceptionfactory
left a comment
exceptionfactory
left a comment
There was a problem hiding this comment.
Thanks for proposing this new Processor @Scrooge-McDucks.
Can you provide some intended use cases for this Processor? Although it could be used in conjunction with EncodeContent, it might actually be better to consider including Base64 detection in that Processor as a feature.
Beyond than use case, however, most components may decide how to handle encoded content through type detection in other ways. Other file formats have standard headers or footers.
For these reasons, I'm not sure about adding this as a standard Processor, but it would be helpful to hear some additional background.
|
Our use case: we receive payloads that are a mix of Base64-encoded and plain content within the same flow, and we don’t know ahead of time which is which. Today we handle this with a script Processor that inspects each FlowFile and decides how to route it. DetectBase64 was meant to replace that with a purpose-built component. The reason EncodeContent doesn’t quite work for us: it assumes the content is already Base64, so on plain content it just fails or garbles it. Happy to take a direction on what you think is best ! |
|
Thanks for the reply! The use case makes sense, but I'm not sure if it fits as a generalized capability for the project. As the additional details highlights, there are differences in Base64 alphabets, depending on the encoding style. Although this could be handled through additional configuration properties, this introduces another level of complexity. The detection strategy is notable, highlighting the challenges of attempting to infer Base64 status based on partial content or entire content. Apache Tika supports a variety of content detection strategies, although it does not appear to have anything out of the box for Base64 detection. Base64 detection can also be tuned if the content follows a standard pattern, which may be more reliable and efficient that general strategies. For all of these reasons, it seems like this would be better left as a custom Processor as opposed to included with the project for now. Another option is publishing it directly as a NAR to Maven Central, which could provide opportunity for additional use and feedback. |
2 participants
Adds DetectBase64Content, a processor that determines whether FlowFile content is Base64 encoded by inspecting its characters rather than attempting a decode. The result is written as true or false to a configurable attribute, and the FlowFile is always routed to success — downstream flows can branch with RouteOnAttribute instead of relying on a decode failure as the signal.
Detection Scope selects between two modes:
Entire Content — reads the full FlowFile for a definitive result.
Sample Size — reads a configurable number of bytes for a bounded check on large content.
Content is streamed in both modes, so heap usage stays constant regardless of FlowFile size.
Detection rules
Content is reported as Base64 when all of the following hold:
It contains at least one character from the standard alphabet defined in RFC 4648 §4.
It contains no characters outside that alphabet, other than line separators and terminating padding.
It has no more than two padding characters.
When the entire content is read, the character count is a multiple of four.
Alternatives considered
ValidatingBase64InputStream — considered, but doesn't fit this use case. It delegates to Commons Codec's Base64.isBase64(), which only checks alphabet membership and treats whitespace as valid — so plain text like hello world passes. It also doesn't check padding placement, padding count, or content length. Detection still needs to inspect every byte against those additional rules, so the wrapper would only be useful for rejecting out-of-alphabet characters — not for the rest of the validation.
Summary
NIFI-16243
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation