[SPARK-57787][CONNECT] Reuse a persistent local Spark Connect server managed by sbin/start-connect-server.sh

dev/sparktestsupport/modules.py

@@ -1162,6 +1162,7 @@ def __hash__(self):
         "pyspark.sql.tests.connect.test_connect_readwriter",
         "pyspark.sql.tests.connect.test_connect_retry",
         "pyspark.sql.tests.connect.test_connect_session",
+        "pyspark.sql.tests.connect.test_connect_local_server",
         "pyspark.sql.tests.connect.test_connect_stat",
         "pyspark.sql.tests.connect.test_parity_geographytype",
         "pyspark.sql.tests.connect.test_parity_geometrytype",

docs/spark-connect-overview.md

@@ -277,6 +277,40 @@ The connection may also be programmatically created using _SparkSession#builder_
 </div>
 </div>
 
+## Faster local iteration with a persistent Connect server
+
+When you develop or test locally with
+
+```python
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote("local[*]").getOrCreate()
+```
+
+PySpark boots a fresh in-process Spark Connect server in **every** process. Each
+`python script.py` run (or each forked test JVM) therefore re-pays the one-time startup cost --
+JVM warmup, `SparkContext` construction, and Connect server boot -- which can take a few seconds and
+makes a quick edit/run loop feel slow.
+
+To amortize that cost across runs, start one persistent local Spark Connect server and point
+every run at it:
+
+```bash
+# Start once; it stays up across runs.
+$SPARK_HOME/sbin/start-connect-server.sh --master "local[*]"
+
+# Every run reconnects instead of booting a new server.
+python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("sc://localhost:15002").getOrCreate()'
+
+# Stop it when you are done.
+$SPARK_HOME/sbin/stop-connect-server.sh
+```
+
+Each run connects as its own Connect session, so session-local state -- temp views, runtime SQL
+configurations, and session artifacts -- is fresh on every run and never leaks between runs. State
+backed by the shared `SparkContext` (the persistent catalog/warehouse, global temp views, and
+cached datasets) *is* shared across runs, so namespace per-run databases or clear that state
+yourself if your runs must be fully isolated.
+
 ## Use Spark Connect in standalone applications
 
 <div class="codetabs">
@@ -371,7 +405,7 @@ one may implement their own class extending `ClassFinder` for customized search
 </div>
 
 For more information on application development with Spark Connect as well as extending Spark Connect
-with custom functionality, see [Application Development with Spark Connect](app-dev-spark-connect.html). 
+with custom functionality, see [Application Development with Spark Connect](app-dev-spark-connect.html).
 # Client application authentication
 
 While Spark Connect does not have built-in authentication, it is designed to

python/packaging/classic/setup.py

@@ -345,7 +345,9 @@ def run(self):
             "pyspark.sbin": [
                 "spark-config.sh",
                 "spark-daemon.sh",
+                "start-connect-server.sh",
                 "start-history-server.sh",
+                "stop-connect-server.sh",
                 "stop-history-server.sh",
             ],
             "pyspark.python.lib": ["*.zip"],

python/pyspark/errors/error-conditions.json

@@ -546,6 +546,11 @@
       "<arg1> and <arg2> should be of the same length, got <arg1_length> and <arg2_length>."
     ]
   },
+  "LOCAL_CONNECT_SERVER_START_FAILED": {
+    "message": [
+      "Failed to start a persistent local Spark Connect server: <reason>."
+    ]
+  },
   "LOCAL_RELATION_SIZE_LIMIT_EXCEEDED": {
     "message": [
       "Local relation size (<actualSize> bytes) exceeds the limit (<sizeLimit> bytes)."