From 38260be90ad146db05cab6fe1f69074cca6da592 Mon Sep 17 00:00:00 2001 From: priyankagunaki-cloud Date: Tue, 1 Sep 2026 15:01:46 +0530 Subject: [PATCH 1/3] DOC-1283 Update dataloaders.adoc --- modules/gds/pages/dataloaders.adoc | 40 ++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/modules/gds/pages/dataloaders.adoc b/modules/gds/pages/dataloaders.adoc index 935e657..e0d7e1b 100644 --- a/modules/gds/pages/dataloaders.adoc +++ b/modules/gds/pages/dataloaders.adoc @@ -15,6 +15,46 @@ You can declare a `NeighborLoader` instance with the factory function `neighborL A neighbor loader is an iterable. When you loop through a neighbor loader instance, it loads one batch of data from the graph to which you established a connection. +[NOTE] +==== +*HTTP vs. Kafka transfer behavior* + +The `NeighborLoader` can transfer data from the graph to the Python runtime +over either **HTTP** (default) or **Kafka**, configured using the +`kafka_address` parameter of +xref:factory-functions.adoc#_neighborloader[`neighborLoader()`]. + +The two transfer modes handle batches differently: + +* **HTTP:** All batches are transferred from the graph to the Python runtime + before they are iterated through locally. Because the entire result set is + transferred up front, the total number of vertices fetched should be kept + small. As a general guideline, keep the total number of fetched vertices + to approximately 10,000 or fewer. +* **Kafka:** Data is transferred one batch at a time as the + `NeighborLoader` is iterated. This avoids transferring the entire result + set to the Python runtime at once. + +The total number of vertices fetched can increase rapidly with the number of +hops and neighbors per hop. For example, with 10,000 seed vertices, 2 hops, +and 10 neighbors sampled per hop: + +[stripes=none] +|=== +| Seeds | Hops | Neighbors/hop | Approximate total vertices fetched + +| 10,000 | 2 | 10 | 10,000 + (10,000 x 10 x 10) = 1,010,000 +|=== + +When using HTTP, reduce the total number of vertices fetched by reducing the +number of seed vertices, the number of neighbors, or the number of hops, or +by applying appropriate filters or graph downsampling. + +The amount of vertex attribute data also affects the total amount of data +transferred. If a workload requires a large result set, Kafka can be used to +transfer the data one batch at a time. +==== + In every iteration, it first chooses a specified number of vertices as seeds, then picks a specified number of neighbors of each seed at random, then the same number of neighbors of each neighbor, and repeat for a specified number of hops. From 7d64646e7b09b780294d6ef3ff10267820dcee69 Mon Sep 17 00:00:00 2001 From: priyankagunaki-cloud Date: Tue, 1 Sep 2026 16:07:16 +0530 Subject: [PATCH 2/3] Update dataloaders.adoc --- modules/gds/pages/dataloaders.adoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/modules/gds/pages/dataloaders.adoc b/modules/gds/pages/dataloaders.adoc index e0d7e1b..6d05a40 100644 --- a/modules/gds/pages/dataloaders.adoc +++ b/modules/gds/pages/dataloaders.adoc @@ -21,8 +21,7 @@ When you loop through a neighbor loader instance, it loads one batch of data fro The `NeighborLoader` can transfer data from the graph to the Python runtime over either **HTTP** (default) or **Kafka**, configured using the -`kafka_address` parameter of -xref:factory-functions.adoc#_neighborloader[`neighborLoader()`]. +`kafka_address` parameter of `neighborLoader()`. The two transfer modes handle batches differently: From 5fb612b9b831ea343dbeeceda63830c76ae08be8 Mon Sep 17 00:00:00 2001 From: priyankagunaki-cloud Date: Fri, 4 Sep 2026 10:45:03 +0530 Subject: [PATCH 3/3] Update dataloaders.adoc --- modules/gds/pages/dataloaders.adoc | 74 ++++++++++++++---------------- 1 file changed, 35 insertions(+), 39 deletions(-) diff --git a/modules/gds/pages/dataloaders.adoc b/modules/gds/pages/dataloaders.adoc index 6d05a40..e04ecd2 100644 --- a/modules/gds/pages/dataloaders.adoc +++ b/modules/gds/pages/dataloaders.adoc @@ -15,45 +15,6 @@ You can declare a `NeighborLoader` instance with the factory function `neighborL A neighbor loader is an iterable. When you loop through a neighbor loader instance, it loads one batch of data from the graph to which you established a connection. -[NOTE] -==== -*HTTP vs. Kafka transfer behavior* - -The `NeighborLoader` can transfer data from the graph to the Python runtime -over either **HTTP** (default) or **Kafka**, configured using the -`kafka_address` parameter of `neighborLoader()`. - -The two transfer modes handle batches differently: - -* **HTTP:** All batches are transferred from the graph to the Python runtime - before they are iterated through locally. Because the entire result set is - transferred up front, the total number of vertices fetched should be kept - small. As a general guideline, keep the total number of fetched vertices - to approximately 10,000 or fewer. -* **Kafka:** Data is transferred one batch at a time as the - `NeighborLoader` is iterated. This avoids transferring the entire result - set to the Python runtime at once. - -The total number of vertices fetched can increase rapidly with the number of -hops and neighbors per hop. For example, with 10,000 seed vertices, 2 hops, -and 10 neighbors sampled per hop: - -[stripes=none] -|=== -| Seeds | Hops | Neighbors/hop | Approximate total vertices fetched - -| 10,000 | 2 | 10 | 10,000 + (10,000 x 10 x 10) = 1,010,000 -|=== - -When using HTTP, reduce the total number of vertices fetched by reducing the -number of seed vertices, the number of neighbors, or the number of hops, or -by applying appropriate filters or graph downsampling. - -The amount of vertex attribute data also affects the total amount of data -transferred. If a workload requires a large result set, Kafka can be used to -transfer the data one batch at a time. -==== - In every iteration, it first chooses a specified number of vertices as seeds, then picks a specified number of neighbors of each seed at random, then the same number of neighbors of each neighbor, and repeat for a specified number of hops. @@ -100,6 +61,41 @@ Fetch neighborhood subgraphs for specific vertices. Each vertex corresponds to a dict with two mandatory keys {"primary_id": ..., "type": ...} +=== HTTP vs. Kafka transfer behavior + +The `NeighborLoader` can transfer data from the graph to the Python runtime +over either **HTTP** (default) or **Kafka**, configured using the +`kafka_address` parameter of `neighborLoader()`. + +The two transfer modes handle batches differently: + +* **HTTP:** All batches are transferred from the graph to the Python runtime + before they are iterated through locally. Because the entire result set is + transferred up front, the total number of vertices fetched should be kept + small. As a general guideline, keep the total number of fetched vertices + to approximately 10,000 or fewer. +* **Kafka:** Data is transferred one batch at a time as the + `NeighborLoader` is iterated. This avoids transferring the entire result + set to the Python runtime at once. + +The total number of vertices fetched can increase rapidly with the number of +hops and neighbors per hop. For example, with 10,000 seed vertices, 2 hops, +and 10 neighbors sampled per hop: + +|=== +| Seeds | Hops | Neighbors per hop | Approximate total vertices fetched + +| 10,000 | 2 | 10 | 10,000 + (10,000 x 10 x 10) = 1,010,000 +|=== + +When using HTTP, reduce the total number of vertices fetched by reducing the +number of seed vertices, the number of neighbors, or the number of hops, or +by applying appropriate filters or graph downsampling. + +The amount of vertex attribute data also affects the total amount of data +transferred. If a workload requires a large result set, Kafka can be used to +transfer the data one batch at a time. + == EdgeLoader