Tika mimetype whitelist is checked against the HTTP header while the parser is chosen from the bytes

[rzo1]

What happens

ParserBolt.execute() matches parser.mimetype.whitelist against parse.Content-Type and, when that key is absent, against the Content-Type response header. Parsing then calls tika.getParser().parse(...), where the auto-detect parser picks a parser from the content itself. When the two disagree, the whitelist decides whether the document is parsed but not which parser runs. parse.Content-Type is written by JSoupParserBolt when detect.mimetype is true, so the gap only opens where that key is missing: detect.mimetype: false, or a topology that feeds the Tika bolt without a JSoup parser in front of it.

Where

external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:163-183, config key parser.mimetype.whitelist.

String mimeType = metadata.getFirstValue("parse.Content-Type");
// otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
    mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE, this.protocolMDprefix);
}

Why it matters

The whitelist reads as a way of choosing which document types this bolt handles, and the archetype configuration uses it that way. Where parse.Content-Type is absent, the value being matched is whatever the fetched server said, so the crawled host decides which side of the gate its bytes land on, while the parser that runs is chosen from those bytes. The practical effect is limited: the archetypes do wire JSoupParserBolt ahead of Tika with detection on, and getting further than a mismatched parser needs a defect inside a Tika parser. Still, a check that does not bind to the parser being selected is not doing the job it appears to do.

Reproduction

Save as external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltWhitelistDetectionTest.java.

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to you under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.stormcrawler.tika;

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.http.HttpHeaders;
import org.apache.storm.task.OutputCollector;
import org.apache.stormcrawler.Constants;
import org.apache.stormcrawler.Metadata;
import org.apache.stormcrawler.TestUtil;
import org.apache.stormcrawler.parse.ParsingTester;
import org.apache.stormcrawler.persistence.Status;
import org.apache.stormcrawler.protocol.ProtocolResponse;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

/**
 * When no parse.Content-Type is present, ParserBolt matches parser.mimetype.whitelist against the
 * Content-Type response header, while tika.getParser() dispatches on the bytes. The two can
 * disagree, so the whitelist does not decide which parser runs.
 */
class ParserBoltWhitelistDetectionTest extends ParsingTester {

    @BeforeEach
    void setupParserBolt() {
        bolt = new ParserBolt();
        setupParserBolt(bolt);
    }

    @Test
    void whitelistAppliesToTheDetectedType() throws IOException {
        Map<String, Object> conf = new HashMap<>();
        // the whitelist shipped by the archetypes
        conf.put("parser.mimetype.whitelist", "application/.+word.*");
        conf.put(ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "http.");
        bolt.prepare(conf, TestUtil.getMockedTopologyContext(), new OutputCollector(output));

        // no parse.Content-Type: no JSoupParserBolt upstream, or detect.mimetype disabled
        Metadata metadata = new Metadata();
        metadata.addValue(
                "http." + HttpHeaders.CONTENT_TYPE,
                "application/vnd.openxmlformats-officedocument.wordprocessingml.document");

        // the body is not a word document
        byte[] content = "<html><body><p>not a word document</p></body></html>"
                .getBytes(StandardCharsets.UTF_8);
        parse("https://example.org/doc.docx", content, metadata);

        System.out.println("detected type: " + metadata.getFirstValue("parse.Content-Type"));
        System.out.println("emitted documents: " + output.getEmitted().size());

        List<List<Object>> status = output.getEmitted(Constants.StatusStreamName);
        Assertions.assertEquals(
                1, status.size(), "content not matching the whitelist should be rejected");
        Assertions.assertEquals(Status.ERROR, status.get(0).get(2));
    }
}

Run it:

mvn -pl external/tika test -Dtest=ParserBoltWhitelistDetectionTest

It fails on main. It whitelists the archetype's application/.+word.*, sets the response header to the Word type, supplies an HTML body and no parse.Content-Type, and asserts the document is rejected.

[INFO] Running org.apache.stormcrawler.tika.ParserBoltWhitelistDetectionTest
detected type: text/html; charset=ISO-8859-1
emitted documents: 1
[ERROR] ParserBoltWhitelistDetectionTest.whitelistAppliesToTheDetectedType -- Time elapsed: 1.483 s <<< FAILURE!
org.opentest4j.AssertionFailedError: content not matching the whitelist should be rejected ==> expected: <1> but was: <0>
	at org.apache.stormcrawler.tika.ParserBoltWhitelistDetectionTest.whitelistAppliesToTheDetectedType(ParserBoltWhitelistDetectionTest.java:73)

The detected type is printed from parse.Content-Type after the parse, so the test also shows what the bolt itself ended up dispatching on.

Suggested fix

In ParserBolt.execute(), run Tika detection on the content bytes when parse.Content-Type is absent and match the whitelist against the detected type rather than the response header. Keep the header only as a detection hint, which is what it already is at line 204. Detection on bytes costs one extra pass over the content, and documents whose header and bytes disagree will start being rejected where they were parsed before, so mention it in the release notes. Restricting the parser set in tika-config.xml is worth documenting alongside, since that bounds which parsers can be selected at all.

[Riddhish1]

Hi! I'd like to work on this.

Plan: I'll update the MIME type validation to detect the type directly from the content when parse.Content-Type is absent, while keeping the HTTP header only as a hint. I'll also add unit tests to verify the whitelist uses the detected content type correctly.

Could you please assign this to me? Thanks!

/assign
take