Skip to content

docs: Clarify Known Users criteria and clean up active project list - #26926

Merged
kosiew merged 5 commits into
apache:mainfrom
kosiew:known-users-26803
Nov 3, 2026
Merged

docs: Clarify Known Users criteria and clean up active project list#26926
kosiew merged 5 commits into
apache:mainfrom
kosiew:known-users-26803

Conversation

@kosiew

@kosiew kosiew commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #26803.

Rationale for this change

The "Known Users" section previously defined an active project using only a six-month commit activity criterion, documented in an HTML comment rather than visible to readers.

This made inclusion requirements unclear and allowed inactive or minimally adopted projects to remain in the active list. Some entries also contained outdated repository links.

This PR establishes visible, measurable inclusion criteria and reorganizes the list to better distinguish active projects from other projects using or having used DataFusion.

What changes are included in this PR?

  • Clarify active-list criteria: Public GitHub repositories must meet all three requirements:
    • At least one commit on the default branch within the last six calendar months.
    • At least 100 GitHub stars.
    • At least five contributors, according to the GitHub repository sidebar.
  • Document exceptions: Projects without public repositories must link to public evidence of DataFusion use, such as documentation or blog posts.
  • Clarify scope: These criteria apply only to Known Users, not Integrations and Extensions.
  • Reorganize project listings: Move datapress, EDB Postgres Lakehouse, GlareDB, HoraeDB, IceGate, KalamDB, ROAPI, and VegaFusion from the active list to the secondary list. Remove the Infino entry from the active list.
  • Improve supporting references: Add DataFusion usage links for Funnel, hiop, Hotdata, Massive.com, Mycorr, Supermetal, and Telemetry.
  • Fix repository links: Update Lance, Restate, and HoraeDB repository URLs. Mark HoraeDB as archived.
  • Clarify the secondary list: Rename its introduction to cover less active projects and projects that do not meet the active-list criteria.

What is the testing strategy for this PR?

Documentation-only change. No automated tests are added or modified in this patch.

The changes can be reviewed by inspecting the Markdown, checking the updated links, and verifying project placement against the documented inclusion criteria.

Are there any user-facing changes?

Yes. Readers of the DataFusion user guide will see explicit inclusion criteria for the Known Users active list, an updated classification of projects, and corrected or additional project references.

There are no changes to DataFusion runtime behavior or public APIs.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

@kosiew
Update documentation: criteria, links, and repositories
dab0cd3 
- Visible criteria and use-evidence rule added.
- Eight entries moved; canonical links fixed.
- Public repositories re-audited.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 30, 2026
kosiew added 2 commits October 30, 2026 18:22
@kosiew
fix: update Massive.com link in introduction.md
80e104d
@kosiew
Update documentation: inline links and remove URL definitions
4246c48 
- Inlined 33 links in the user guide introduction.
- Removed 25 URL definitions from the documentation.
- Preserved the invitation text in the updated guide.
@kosiew
kosiew marked this pull request as ready for review October 30, 2026 11:26
kumarUjjawal
kumarUjjawal reviewed Oct 31, 2026
View reviewed changes

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kosiew

This looks like a nice improvement to me.

alamb
alamb approved these changes Nov 2, 2026
View reviewed changes

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kosiew and @kumarUjjawal

Comment thread docs/source/user-guide/introduction.md Resolved
Comment thread docs/source/user-guide/introduction.md Resolved
Comment thread docs/source/user-guide/introduction.md Resolved
Comment thread docs/source/user-guide/introduction.md Resolved
Comment thread docs/source/user-guide/introduction.md Resolved
Comment thread docs/source/user-guide/introduction.md Resolved
kosiew added 2 commits November 3, 2026 10:54
@kosiew
Update introduction examples and project references
3fd6512 
- Expand Apache references, eg Apache Auron
- Expand Cloudflare reference
@kosiew
Update documentation for OpenObserve and Cloudflare R2
807ae68 
- Replaced HoraeDB with OpenObserve in Use Cases.
- Added Cloudflare R2 Query Engine with deep-dive link.
@kosiew

kosiew commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

🚀
@kumarUjjawal, @alamb
Thank you for your reviews.

@kosiew
kosiew enabled auto-merge November 3, 2026 02:55
@kosiew
kosiew added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@kosiew
kosiew added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
This was referenced Nov 3, 2026
Closed
Merged
@kosiew
kosiew added this pull request to the merge queue Nov 3, 2026
Merged via the queue into apache:main with commit 05296c7 Nov 3, 2026
45 checks passed
@kosiew
kosiew deleted the known-users-26803 branch November 3, 2026 05:50
Phoenix500526 pushed a commit to Phoenix500526/datafusion that referenced this pull request Nov 3, 2026
@kumarUjjawal
test: skip completed_groups_release_first_value_bitmap under force_ha…
7d7d6b1 
…sh_collisions (apache#27034)

## Which issue does this PR close?

- Closes apache#27033.

## Rationale for this change

`completed_groups_release_first_value_bitmap` (added in apache#26782) takes
about 10 minutes in the `cargo test hash collisions (amd64)` merge-queue
job. It aggregates 131,072 distinct keys in one batch. Under
`force_hash_collisions`, every key has the hash 0, so the aggregate is
quadratic. The test checks that `FIRST_VALUE(v ORDER BY v)` releases its
per-batch bitmap, which does not depend on hashing.

The long job gives spot runner shutdowns more time to hit it. Evidence
from merge-queue runs:

- In the passing rerun for apache#26926
([job](https://github.com/apache/datafusion/actions/runs/41571721394/job/125710730373)),
the physical-plan unit test binary took 609.89s. All other tests in it
finished by 03:40:31. This test ran from about 03:40:07 to 03:50:17.
- The median job time went from 6.2 minutes (230 jobs, Oct 21 to Oct 30)
to 12.0 minutes (70 jobs after apache#26782 merged).
- All 5 failed attempts of the job since then were runner shutdowns
while this test was running: apache#26929
([job](https://github.com/apache/datafusion/actions/runs/41114163010/job/124263887540)),
apache#26898
([job](https://github.com/apache/datafusion/actions/runs/41115099549/job/124267135386)),
apache#26883
([job](https://github.com/apache/datafusion/actions/runs/41120437208/job/124285496814)),
apache#26974
([job](https://github.com/apache/datafusion/actions/runs/41559552415/job/125666576696)),
apache#26926
([job](https://github.com/apache/datafusion/actions/runs/41571721394/job/125705896210)).

## What changes are included in this PR?

Skip the test under `force_hash_collisions`, as apache#26594 did for another
test. The imports that only this test uses move into its body, so they
are not unused under the feature.

## What is the testing strategy for this PR?

Test config change only. The regular `cargo test` job still runs the
test. Locally (`--profile ci`), the test takes 0.92s by default and 309s
with `force_hash_collisions`. With the feature, it is now skipped, and
Clippy passes with and without the feature.

## Are there any user-facing changes?

No.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb approved these changes
@kumarUjjawal kumarUjjawal kumarUjjawal approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation v56.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add star and contributor thresholds to "Known Users" inclusion criteria

3 participants

@kosiew @alamb @kumarUjjawal