Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions modules/gds/pages/dataloaders.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,41 @@ Fetch neighborhood subgraphs for specific vertices.
Each vertex corresponds to a dict with two mandatory keys
{"primary_id": ..., "type": ...}

=== HTTP vs. Kafka transfer behavior

The `NeighborLoader` can transfer data from the graph to the Python runtime
over either **HTTP** (default) or **Kafka**, configured using the
`kafka_address` parameter of `neighborLoader()`.

The two transfer modes handle batches differently:

* **HTTP:** All batches are transferred from the graph to the Python runtime
before they are iterated through locally. Because the entire result set is
transferred up front, the total number of vertices fetched should be kept
small. As a general guideline, keep the total number of fetched vertices
to approximately 10,000 or fewer.
* **Kafka:** Data is transferred one batch at a time as the
`NeighborLoader` is iterated. This avoids transferring the entire result
set to the Python runtime at once.

The total number of vertices fetched can increase rapidly with the number of
hops and neighbors per hop. For example, with 10,000 seed vertices, 2 hops,
and 10 neighbors sampled per hop:

|===
| Seeds | Hops | Neighbors per hop | Approximate total vertices fetched

| 10,000 | 2 | 10 | 10,000 + (10,000 x 10 x 10) = 1,010,000
|===

When using HTTP, reduce the total number of vertices fetched by reducing the
number of seed vertices, the number of neighbors, or the number of hops, or
by applying appropriate filters or graph downsampling.

The amount of vertex attribute data also affects the total amount of data
transferred. If a workload requires a large result set, Kafka can be used to
transfer the data one batch at a time.


== EdgeLoader

Expand Down
Loading