feat: Support PostgreSQL export for export-metadata

[JWuCines]

Description

The export-metadata tool currently only supports exporting from Derby metadata stores. This PR adds support for exporting from PostgreSQL by implementing a generic JDBC-based exportTable in SQLMetadataConnector, which PostgreSQL (and any future connector) inherits automatically.

Added generic JDBC export in SQLMetadataConnector

Implemented exportTableWithJdbc, a protected method that exports any table to CSV using standard JDBC. Binary/BLOB columns (including PostgreSQL BYTEA) are hex-encoded, booleans are written as true/false strings, and string values containing commas, quotes, \n, or \r are properly CSV-escaped per RFC 4180. The base exportTable delegates to this method, while DerbyConnector continues to override it with Derby's native SYSCS_EXPORT_TABLE.

Updated ExportMetadata for PostgreSQL compatibility

• Table names are no longer unconditionally uppercased; uppercasing is now applied only for Derby (detected via jdbc:derby URI prefix), since PostgreSQL uses lowercase table names.
• Updated the @Command description to mention PostgreSQL support.

Added unit tests

Four new tests in SQLMetadataConnectorTest exercise the generic JDBC export path via TestDerbyConnector.exportTableGeneric():

• testExportTable — verifies hex-encoded BLOBs and true/false boolean strings
• testExportTableWithSpecialCharacters — verifies CSV quoting/escaping for commas, double quotes, and plain values
• testExportTableWithNullValues — verifies NULL columns produce empty CSV fields
• testExportTablePreservesAllColumns — verifies all columns (including nullable trailing columns like used_status_last_updated) are exported

Updated documentation

• export-metadata.md — removed the Derby-only limitation, added a "PostgreSQL" section under "Running the tool" with the required -Ddruid.extensions.loadList and -Ddruid.metadata.storage.type flags, clarified _raw.csv description
• metadata-migration.md — updated intro and export tool reference to include PostgreSQL
• deep-storage-migration.md — updated export tool reference, added note about no running processes needed when migrating from PostgreSQL

Release note

The export-metadata tool now supports exporting from PostgreSQL metadata stores in addition to Derby. When exporting from PostgreSQL, pass -Ddruid.extensions.loadList='["postgresql-metadata-storage"]' -Ddruid.metadata.storage.type=postgresql on the command line along with the appropriate --connectURI.

Fixed streaming, column preservation, and CSV escaping in export

• SQLMetadataConnector.exportTableWithJdbc — switched from retryWithHandle (auto-commit) to retryTransaction so the connection runs with autoCommit=false, and applied getStreamingFetchSize() to the Statement. This enables PostgreSQL cursor-based streaming instead of buffering the entire ResultSet in memory.
• ExportMetadata.rewriteSegmentsExport — the rewrite stage previously hardcoded columns 0–8, dropping used_status_last_updated, indexing_state_fingerprint, upgraded_from_segment_id, and optional schema columns. Added a loop to pass through all columns after payload.
• ExportMetadata rewrite methods — all five rewrite methods (rewriteDatasourceExport, rewriteRulesExport, rewriteConfigExport, rewriteSupervisorExport, rewriteSegmentsExport) now re-escape non-payload fields via a new csvEscapeField() helper (RFC 4180). Previously, fields containing commas or double quotes were parsed correctly by opencsv but written back without quoting, producing malformed output.
• ExportMetadataTest — added 10 tests covering csvEscapeField, segments rewrite with all columns, special characters round-trip, and backward compatibility with 9-column tables.

Deterministic segments column order and updated import commands

• SQLMetadataConnector — added exportTable(tableName, outputPath, columns) and getTableColumns(tableName). When a column list is given, the export query selects those columns in that order, quoting each identifier with the database's identifier quote string so reserved words such as end work. DerbyConnector uses SYSCS_EXPORT_QUERY in that case.
• ExportMetadata — the segments table is now exported with a canonical column order (id, dataSource, created_date, start, end, partitioned, version, used, payload, used_status_last_updated, indexing_state_fingerprint, upgraded_from_segment_id[, schema_fingerprint, num_rows]), instead of relying on the physical column order, which depends on the order in which ALTER TABLE added the newer columns. Unknown columns are appended at the end.
• export-metadata.md — documented the exported segments column order and updated the MySQL LOAD DATA and PostgreSQL COPY commands to list all exported segments columns.

Preserved backslashes through the CSV rewrite

• ExportMetadata — the rewrite stage read the intermediate CSV with the default OpenCSV CSVParser, which treats backslash as an escape character and silently dropped it from values such as segment ids and datasource names (backslashes are permitted by Druid's id validation). The rewrite now uses RFC4180Parser, matching the RFC 4180 output written by the export stage.
• ExportMetadataTest — added testRewriteSegmentsExport_preservesBackslashes, and existing assertions now parse the output with RFC4180Parser.

Record-aware CSV parsing in the rewrite stage

• ExportMetadata — all five rewrite methods read one physical line at a time, so a quoted value containing a newline or carriage return (which the export stage writes as a record spanning several lines) was misparsed. They now use a shared openCsvReader() helper that builds a CSVReader with an RFC4180Parser and withKeepCarriageReturn(true), and iterate with readNext() so multi-line records are handled as single records.
• ExportMetadataTest — added testExportAndRewriteSegments_withMultilineFields, an end-to-end test that inserts values containing newlines, carriage returns, and commas into a Derby segments table, exports it through the generic JDBC path, runs the rewrite, and verifies every record and field round-trips.

Review fixes

• SQLMetadataConnector.getTableColumns — the lookup passed the raw table name as a DatabaseMetaData search pattern with a null schema, so _ acted as a wildcard and columns from same-named tables in other schemas could leak in. It is now scoped to Connection.getSchema() (the schema an unqualified table name resolves to, consistent with PostgreSQLConnector.tableExists) and both names are escaped with getSearchStringEscape().
• SQLMetadataConnector.makeExportSelectList — doubles any identifier quote character inside a column name.
• ExportMetadata.exportSegmentsTable — fails with an ISE instead of silently falling back to SELECT * (and thus an unstable column order) when the column list cannot be read.
• ExportMetadata.readRecord — all five rewrites now validate the field count of each record and fail with the row number, file name, and expected arity instead of throwing ArrayIndexOutOfBoundsException on a malformed raw CSV.
• export-metadata.md — the PostgreSQL segments import uses FORCE_NULL for the nullable columns, since NULLs are exported as empty fields which COPY would otherwise import as empty strings (and reject outright for non-string columns).
• SQLMetadataConnectorTest — added testExportTableWithDerbyNativeExport, covering DerbyConnector's native SYSCS_EXPORT_QUERY path with an explicit column list, a reserved-word column, and a BLOB payload (written as lowercase hex).
• ExportMetadataTest — added testRewriteSegmentsExport_failsOnTruncatedRow.

---

Key changed/added classes in this PR

• SQLMetadataConnector — added exportTable and exportTableWithJdbc for generic JDBC CSV export; switched export to transactional streaming
• ExportMetadata — added isDerby() helper, conditional table name casing, csvEscapeField(), all-column preservation in segments rewrite
• TestDerbyConnector — added exportTableGeneric() to test the generic JDBC path
• SQLMetadataConnectorTest — added 4 export tests
• ExportMetadataTest — added 10 tests for CSV escaping and segments rewrite

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	1
P3	0
Total	3

Reviewed 7 of 7 changed files.

Found three export-migration risks: buffered PostgreSQL reads, dropped current-schema segment columns, and lost CSV escaping during rewrites.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Stream PostgreSQL results instead of buffering the table

PostgreSQL JDBC buffers the complete ResultSet by default. This plain Statement runs on an auto-commit connection and never sets a fetch size; PostgreSQL cursor-based fetching requires autoCommit=false and a positive fetch size. Exporting a production segments table can therefore exhaust heap before rows are written. Run the query in a transaction and apply getStreamingFetchSize().

[JWuCines]

Fixed in 6b73b7e

[FrankChen021]

Confirmed: retryTransaction gives PostgreSQL an auto-commit-disabled connection, and getStreamingFetchSize() is applied before query execution, enabling cursor-based fetching. Reviewed 8 of 8 changed files.

[FrankChen021]

[P1] Preserve current segment columns in the importable CSV

The generic raw export preserves all columns, but run() immediately passes the file through rewriteSegmentsExport, which emits only columns 0 through 8. Current segment tables also contain required used_status_last_updated and additional fingerprint/upgrade columns. A PostgreSQL migration therefore loses those values, and the documented COPY into a freshly created current table fails because used_status_last_updated is NOT NULL without a default. The new test covers only the raw file; the final CSV and import commands need to preserve the current schema.

[JWuCines]

Fixed in 6b73b7e

[FrankChen021]

Thanks; the rewrite now preserves trailing values, but the documented PostgreSQL COPY druid_segments(...) command still names only the original nine columns. Current exports contain 12–14 fields, so COPY fails with extra data; upgraded source tables can also have added columns in a different physical order. Please export segments in an explicit stable column order and update the import column lists accordingly. Reviewed 8 of 8 changed files.

[FrankChen021]

[P2] Preserve CSV escaping through the rewrite stage

Although the raw writer correctly quotes commas and double quotes, each ExportMetadata rewrite parses those fields and concatenates them into the final CSV without re-escaping. Legal datasource and metadata identifiers may contain these characters, so the final importable file gains extra columns or malformed quoting even though the raw file is valid. Use a CSV writer or shared escaping routine in the rewrite stage and add an end-to-end test.

[JWuCines]

Fixed in 6b73b7e

[FrankChen021]

The comma/quote round-trip is fixed, but multiline fields are still split because each rewrite uses BufferedReader.readLine() plus CSVParser.parseLine() on one physical line. exportTableWithJdbc writes quoted newline or carriage-return values as CSV records spanning lines, so the rewrite misparses them before csvEscapeField runs. Please use a record-aware CSVReader/readNext path and cover this end to end. Reviewed 8 of 8 changed files.

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 8 of 8 changed files. PostgreSQL streaming is fixed, but segment import compatibility and CSV backslash preservation remain broken.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Update imports for preserved segment columns

Current segment tables emit at least 12 fields, and this loop now retains all of them, but the documented PostgreSQL and MySQL import commands still declare only the original nine columns. PostgreSQL COPY rejects every such row as extra data, while MySQL truncates or ignores trailing fields with warnings. Emit a deterministic schema-compatible order and update import commands to include all exported columns.

[FrankChen021]

[P2] Preserve backslashes through CSV rewrites

The writer emits RFC 4180 data and leaves backslashes unchanged, but ExportMetadata reads it with default OpenCSV CSVParser, where backslash is the escape character. Consequently, a valid identifier such as foo\bar is parsed as foobar before csvEscapeField runs; Druid's ID validation explicitly permits backslashes. Use an RFC 4180 reader/parser or configure a null escape character, and add a round-trip test.

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 9 of 9 changed files. Stable segment ordering and RFC CSV parsing fix the prior issues, but legacy schemas still conflict with the documented import list and mixed-case PostgreSQL bases fail column discovery.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Keep import lists aligned with older segment schemas

orderSegmentsColumns skips every absent column, and the new test explicitly supports legacy nine-column tables, but the documented PostgreSQL COPY always expects the three later columns. A nine-field export therefore fails with missing data. The instructions or generated output must identify every optional post-payload column and omit absent ones from the import list and FORCE_NULL.

[FrankChen021]

[P2] Normalize PostgreSQL identifiers before metadata lookup

PostgreSQL folds unquoted identifiers to lowercase, while DatabaseMetaData.getColumns uses a case-sensitive LIKE. With a valid mixed-case --base, tableExists succeeds via ILIKE and ordinary SQL resolves the lowercase table, but this lookup returns no columns, causing segment export to abort. Normalize according to JDBC identifier rules or resolve the actual table name first.

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 9 of 9 changed files. Mixed-case PostgreSQL identifiers are now handled, but legacy Derby imports and configured PostgreSQL schema discovery remain incorrect.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Provide a column-mapped Derby import for legacy exports

The new guidance says to omit optional columns from the import list, but the Derby example uses SYSCS_IMPORT_TABLE, which has no column list and imports every destination column in physical order. A nine-column legacy export therefore still cannot be loaded into a current 12/14-column Derby segments table, and the documented adjustment cannot be performed. Use SYSCS_IMPORT_DATA with an explicit mapping, or provide another working legacy-schema procedure.

[FrankChen021]

[P2] Look up columns in the configured PostgreSQL schema

PostgreSQLConnector.tableExists scopes the relation to druid.metadata.postgres.dbTableSchema, but getTableColumns scopes DatabaseMetaData.getColumns to Connection.getSchema(). These can differ, for example when a role-named schema precedes public in search_path while the Druid tables remain in public, so tableExists succeeds but column discovery returns empty and exportSegmentsTable aborts. Use the connector's configured table schema or resolve the actual schema containing the unqualified relation.