HIVE-29724: Provide a Metastore tool for de-duplicating the columns

[dengzhhu653]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[Copilot]

Pull request overview

Adds a new standalone-metastore “metatool” command to de-duplicate column descriptors (CDs) for partitioned tables, aiming to reduce metastore metadata bloat (e.g., after replication) by merging identical per-partition schemas.

Changes:

• Introduces -dedupColumns CLI command (with optional catalog/db/table filters), including -dryRun support and a -verbose mode for progress/details.
• Implements the core CD de-duplication logic (ColumnDeduplicator) and exposes it via MetaToolObjectStore.
• Adds/extends unit tests to cover CLI parsing and validates de-duplication behavior via an HMS test.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/metatool/TestHiveMetaToolCommandLine.java	Updates CLI parsing tests for new -dedupColumns / -verbose behavior and error messages.
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHMSColumnDescriptorReuse.java	Adds an integration-style test that exercises the de-dup tool against ObjectStore state.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/MetaToolObjectStore.java	Adds dedupColumns() entrypoint and result container type.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/MetaToolTaskDedupColumns.java	New metatool task implementation for running and reporting de-dup results.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/HiveMetaToolCommandLine.java	Wires new CLI options (-dedupColumns, -verbose) and updates validation.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/HiveMetaTool.java	Dispatches the new MetaToolTaskDedupColumns.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/ColumnDeduplicator.java	New implementation of CD de-duplication logic over metastore JDO models.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/impl/TableStoreImpl.java	Adjusts CD-reference checking helper to be reusable from the new tool path.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Suppressed comments (4)

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/HiveMetaToolCommandLine.java:92

• The -dedupColumns option arg name shown in metatool -help is malformed ("catalog> <db> <table"). This looks like a typo and will confuse users reading the CLI help.

  private static final Option DEDUP_COLUMNS = OptionBuilder
      .withArgName("catalog> " + "<db> " + "<table")
      .hasArgs(3)

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHMSColumnDescriptorReuse.java:56

• Unused static import assertNotEquals is never referenced in this test class; Java treats unused imports as a compilation error.

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotNull;

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/metatool/TestHiveMetaToolCommandLine.java:151

• Add coverage for -dedupColumns with no filter args (e.g. new String[]{"-dedupColumns"}), since the option is declared with optional args and is expected to work without requiring catalog/db/table filters.

  public void testParseDedupColumns() throws ParseException {
    HiveMetaToolCommandLine cl = new HiveMetaToolCommandLine(
        new String[] {"-dedupColumns", "hive", "default", "person", "-dryRun", "-verbose"});
    assertTrue(cl.isDedupColumns());
    assertTrue(cl.isDryRun());
    assertTrue(cl.isVerbose());
    assertEquals("hive", cl.getDedupColumnsParams()[0]);
    assertEquals("default", cl.getDedupColumnsParams()[1]);
    assertEquals("person", cl.getDedupColumnsParams()[2]);
  }

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/ColumnDeduplicator.java:234

• findPartitionedTables() executes an extra MPartition query per table (isPartitionedTable(...)), creating an N+1 query pattern that will be very slow on large metastores. Consider rewriting this to fetch partitioned table IDs in one query (or query MPartition and derive the distinct tables) to reduce DB round trips.

      List<MTable> mTables = (List<MTable>) query.executeWithArray(parameterVals.toArray(new String[0]));
      pm.retrieveAll(mTables);
      for (MTable mTable : mTables) {
        if (!isPartitionedTable(mTable.getId())) {
          continue;
        }
        pm.retrieve(mTable.getDatabase());

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Suppressed comments (4)

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHMSColumnDescriptorReuse.java:57

• Unused static import assertNotEquals causes a compilation failure (unused imports are errors in Java). Remove the import or use it.

import static org.apache.hadoop.hive.metastore.Warehouse.DEFAULT_CATALOG_NAME;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHMSColumnDescriptorReuse.java:222

• This line exceeds the standalone-metastore Checkstyle LineLength limit (120) and will fail the build. Wrap the getPartitionsByNames call (and avoid hardcoding the catalog when DEFAULT_CATALOG_NAME is already imported).

    partitions = objectStore.getPartitionsByNames("hive", "default", "person", GetPartitionsArgs.from(request));

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/HiveMetaToolCommandLine.java:92

• The withArgName string for -dedupColumns is malformed (missing > for the last token and inconsistent formatting), which makes the help/usage output confusing. Use a single well-formed argName string.

  @SuppressWarnings("static-access")
  private static final Option DEDUP_COLUMNS = OptionBuilder
      .withArgName("catalog> " + "<db> " + "<table")
      .hasArgs(3)

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/ColumnDeduplicator.java:156

• In -dryRun mode, columnDescriptorsRemoved is computed via hasRemainingCDReference(...) against the current DB state (before any SDs are remapped), so it will almost always report 0 even when the tool would remove CDs in a real run. Compute the dry-run removal count by simulating the post-remap CD reference counts within the table.

    if (!isDryRun) {
      applyTableChanges(partSdUpdates, result);
    } else {
      Set<Long> candidateCdIds = new HashSet<>();
      for (Map.Entry<PartitionSdInfo, Long> update : partSdUpdates) {

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/HiveMetaToolCommandLine.java:92

• The -dedupColumns Option arg name shown in help output is malformed (missing leading '<' for catalog and trailing '>' for table). This makes the CLI usage confusing.

  @SuppressWarnings("static-access")
  private static final Option DEDUP_COLUMNS = OptionBuilder
      .withArgName("catalog> " + "<db> " + "<table")
      .hasArgs(3)

[sonarqubecloud]

Quality Gate passed

Issues
21 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.3% Duplication on New Code

See analysis details on SonarQube Cloud