Use batch endpoint in URLFrontier module for DISCOVERED urls

external/urlfrontier/README.md

@@ -24,6 +24,23 @@ urlfrontier.max.buckets: 10
 urlfrontier.max.urls.per.bucket:10
 ```
 
+## Sending discovered URLs in batches
+
+`StatusUpdaterBolt` sends known URLs (fetched, redirections, errors...) to the frontier's
+`PutURLs` endpoint one message at a time, and discovered URLs, which are the bulk of what a
+crawl writes, through the batched `PutDiscovered` endpoint introduced in
+[URLFrontier 2.6](https://github.com/crawler-commons/url-frontier/releases/tag/2.6). Up to
+
+```
+urlfrontier.batch.size: 100
+```
+
+discovered URLs are grouped into one message, which amortises the per-message cost that limits
+the ingestion rate. A partially filled batch is sent after a second at the latest, so that acks
+are not delayed when the crawl tails off. Set `urlfrontier.batch.size` to 0 to send every URL
+individually. If the frontier predates 2.6 and does not implement `PutDiscovered`, the bolt
+detects it and falls back to sending discovered URLs individually on the streaming endpoint.
+
 ## Robots crawl-delay pacing
 
 `QueueRegulatorBolt` can pace host queues when a robots.txt Crawl-delay exceeds the fetcher's local

external/urlfrontier/pom.xml

@@ -37,7 +37,7 @@ under the License.
     <description>URL Frontier resources for StormCrawler</description>
 
     <properties>
-        <urlfrontier.version>2.5</urlfrontier.version>
+        <urlfrontier.version>2.6</urlfrontier.version>
         <jacoco.haltOnFailure>true</jacoco.haltOnFailure>
         <jacoco.classRatio>0.50</jacoco.classRatio>
         <jacoco.instructionRatio>0.49</jacoco.instructionRatio>

external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Constants.java

@@ -45,6 +45,16 @@ private Constants() {}
             "urlfrontier.updater.max.messages";
     public static final String URLFRONTIER_CRAWL_ID_KEY = "urlfrontier.crawlid";
 
+    /**
+     * Maximum number of discovered URLs sent in one message on the batched PutDiscovered endpoint
+     * added in URLFrontier 2.6. The outlinks of a page form a natural batch: grouping them
+     * amortises the per-message cost which limits the ingestion rate. A value of 0 sends the
+     * discovered URLs individually on the streaming PutURLs endpoint. Defaults to 100.
+     */
+    public static final String URLFRONTIER_BATCH_SIZE_KEY = "urlfrontier.batch.size";
+
+    public static final int URLFRONTIER_BATCH_SIZE_DEFAULT = 100;
+
     /**
      * Maximum delay in seconds honoured when a server requests a back-off via the Retry-After HTTP
      * response header. {@code -1} disables the cap. Defaults to 86400 (24h).