GH-22232: [C++][Python] Introduce optional default_column_type parameter

[vladborovtsov]

Rationale for this change

Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.

What changes are included in this PR?

• c++: new convert option "default_column_type" to augment logic around column_types parameter
• 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by PARTIAL FIX: Improve leading zeros preservation with dtype=str for dict-based dtypes pandas-dev/pandas#62242 and BUG: pyarrow read_csv engine stripping leading zeros with dtype=str pandas-dev/pandas#57666
• python: corresponding changes to make cpp change consumable from python
• python: extended test_convert_options test - include. default_column_type
• python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
• relevant documentation update for python component;

Are these changes tested?

Yes. Existing and new tests are passing.

C++:

> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN      ] ReaderTests.DefaultColumnTypePartialDefault
> [       OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN      ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [       OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN      ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [       OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [  PASSED  ] 3 tests.

All:

> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [  PASSED  ] 264 tests.

pyarrow:
New tests are passing.

Are there any user-facing changes?

I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.

Maybe relevant: #22232

Relates to #47502

• GitHub Issue: [C++] CSV reader: add a default column type (or sentinel mapping) to avoid per-column enumeration #47502

• GitHub Issue: [C++] CSV reader: Ability to not infer column types. #22232

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[vladborovtsov]

@github-actions crossbow submit preview-docs

[github-actions]

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/18062577036

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

Thank you @vladborovtsov for the contribution.
I will add info about the proposed solution in the original issue (#22232) so I can see opinions from C++ devs on the proposed solution.

[github-actions]

⚠️ GitHub issue #22232 has been automatically assigned in GitHub to PR creator.

[vladborovtsov]

Hi @AlenkaF
I'm happy to continue the labour and discussion to get that merged.
As for AI, it wasn't used much, although I tried :) With such huge codebase the generation quality is quite low.

[AlenkaF]

Happy to see a response!
All good, it is totally OK to use gen AI wisely ;)

I will wait for an opinion from a C++ dev and in the meantime try to look at the Python part.

[vladborovtsov]

Hi @AlenkaF
Any feedback?

[AlenkaF]

The Python part looks good to me. I only added a comment for the docstring examples.

@rok @raulcd do you have any opinions on the C++ approach to handle this use case?

[pitrou]

This looks good to me, just two minor comments below.

[AlenkaF]

Thank you for your the work on this!
I only have two extra comments for the Python part.

First is connected to the repr method for the ConvertOptions. It needs default_column_type to be added in the list:

arrow/python/pyarrow/_csv.pyx

Lines 1144 to 1164 in 0600621

	def _repr_base(self):
	return (f"""
	check_utf8={self.check_utf8},
	column_types={self.column_types},
	null_values={self.null_values},
	true_values={self.true_values},
	false_values={self.false_values},
	decimal_point={self.decimal_point!r},
	strings_can_be_null={self.strings_can_be_null},
	quoted_strings_can_be_null={self.quoted_strings_can_be_null},
	include_columns={self.include_columns},
	include_missing_columns={self.include_missing_columns},
	auto_dict_encode={self.auto_dict_encode},
	auto_dict_max_cardinality={self.auto_dict_max_cardinality},
	timestamp_parsers={[str(i) for i in self.timestamp_parsers]}""")
	def __repr__(self):
	return (f"<pyarrow.csv.ConvertOptions>({self._repr_base()})")
	def __str__(self):
	return (f"ConvertOptions({self._repr_base()})")

and in the test:

arrow/python/pyarrow/tests/test_csv.py

Line 301 in 0600621

def test_convert_options(pickle_module):

The pickling compatibility is done ( __getstate__ and __setstate__).

The second comment is one extra assert in the test. Otherwise LGTM!

[AlenkaF]

@vladborovtsov the CI failures seem to be related. Could you have a look?

[vladborovtsov]

Hi @AlenkaF
I believe now it should be fine?
I'm running tests locally in docker using pytest -v --pyargs pyarrow.tests.test_csv.
Am I missing something?

[AlenkaF]

Yes, everything looking good. I will just run the extended builds before merging. Thanks!

[AlenkaF]

@github-actions crossbow submit -g python

[github-actions]

Revision: c1ad5f6

Submitted crossbow builds: ursacomputing/crossbow @ actions-9de857b990

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[AlenkaF]

The failing emscripten build is not related. Will merge.
Thanks!