From ef3c0c044d84ad776acb230c8aa0183c8716773e Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Thu, 21 May 2026 14:32:48 +0200 Subject: [PATCH 01/26] Intro notebooks --- src/main/python/notebooks/python_e2e_l1.ipynb | 520 ++++++++++++++++++ src/main/python/notebooks/quick_start.ipynb | 234 ++++++++ src/main/python/requirements.txt | 9 + 3 files changed, 763 insertions(+) create mode 100644 src/main/python/notebooks/python_e2e_l1.ipynb create mode 100644 src/main/python/notebooks/quick_start.ipynb create mode 100644 src/main/python/requirements.txt diff --git a/src/main/python/notebooks/python_e2e_l1.ipynb b/src/main/python/notebooks/python_e2e_l1.ipynb new file mode 100644 index 00000000000..34564cffbd6 --- /dev/null +++ b/src/main/python/notebooks/python_e2e_l1.ipynb @@ -0,0 +1,520 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "26ef529d", + "metadata": {}, + "source": [ + "This example shows how one can work the SystemDS framework. More precisely, we will make use of the built-in DataManager, Multinomial Logistic Regression function, and the Confusion Matrix function. The dataset used in this tutorial is a preprocessed version of the “UCI Adult Data Set”. If one wants to skip the explanation then the full script is available at the end of this level.\n", + "\n", + "We will train a Multinomial Logistic Regression model on the training dataset and subsequently use the test dataset to assess how well our model can predict if the income is above or below $50K/yr based on the features." + ] + }, + { + "cell_type": "markdown", + "id": "18ac25e2", + "metadata": {}, + "source": [ + "# Step 1: Load and prepare data\n", + "First, we get our training and testing data from the built-in DataManager. Since the multiLogReg function requires the labels (Y) to be > 0, we add 1 to all labels. This ensures that the smallest label is >= 1. Additionally we will only take a fraction of the training and test set into account to speed up the execution." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "40c4453f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Warning: Scuro dependencies missing or wrong version installed: torch 2.4.1, torchvision 0.19.1, librosa 0.10.2, opencv-python 4.10.0.84, opt-einsum 3.3.0, h5py 3.11.0, transformers 4.46.3, nltk 3.9.1, gensim 4.3.3\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Using incubator modules: jdk.incubator.vector\n", + "26/05/06 15:19:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" + ] + } + ], + "source": [ + "from systemds.context import SystemDSContext\n", + "from systemds.examples.tutorials.adult import DataManager\n", + "from systemds.operator.algorithm import multiLogReg\n", + "from systemds.operator.algorithm import multiLogRegPredict\n", + "from systemds.operator.algorithm import confusionMatrix\n", + "\n", + "with SystemDSContext() as sds:\n", + " d = DataManager()\n", + "\n", + " # limit the sample size\n", + " train_count = 15000\n", + " test_count = 5000\n", + "\n", + " # Get train and test datasets.\n", + " X_frame, Y_frame, Xt_frame, Yt_frame = d.get_preprocessed_dataset(sds)\n", + "\n", + " # Transformation specification\n", + " jspec_data = d.get_jspec(sds)\n", + " jspec_labels = sds.scalar(f'\"{ {\"recode\": [\"income\"]} }\"')\n", + "\n", + " # Transform frames to matrices.\n", + " X, M1 = X_frame.transform_encode(spec=jspec_data)\n", + " Xt = Xt_frame.transform_apply(spec=jspec_data, meta=M1) \n", + " Y, M2 = Y_frame.transform_encode(spec=jspec_labels)\n", + " Yt = Yt_frame.transform_apply(spec=jspec_labels, meta=M2) \n", + " \n", + " # Subsample to make training faster\n", + " X = X[0:train_count]\n", + " Y = Y[0:train_count]\n", + " Xt = Xt[0:test_count]\n", + " Yt = Yt[0:test_count]" + ] + }, + { + "cell_type": "markdown", + "id": "349c0d0b", + "metadata": {}, + "source": [ + "Here the DataManager contains the code for downloading and setting up either Pandas DataFrames or internal SystemDS Frames, for the best performance and no data transfer from pandas to SystemDS it is recommended to read directly from disk into SystemDS." + ] + }, + { + "cell_type": "markdown", + "id": "4a180e28", + "metadata": {}, + "source": [ + "# Step 2: Training\n", + "Now that we prepared the data, we can use the multiLogReg function. First, we will train the model on our training data. Afterward, we can make predictions on the test data and assess the performance of the model." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "079a61b4", + "metadata": {}, + "outputs": [], + "source": [ + "betas = multiLogReg(X, Y, verbose=False)" + ] + }, + { + "cell_type": "markdown", + "id": "d145d87a", + "metadata": {}, + "source": [ + "Note that nothing has been calculated yet. In SystemDS the calculation is executed once `compute()` is called. E.g. `betas_res = betas.compute()`.\n", + "\n", + "We can now use the trained model to make predictions on the test data." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "6b734c4d", + "metadata": {}, + "outputs": [], + "source": [ + "[_, y_pred, acc] = multiLogRegPredict(Xt, betas, Y=Yt)" + ] + }, + { + "cell_type": "markdown", + "id": "db63e4ab", + "metadata": {}, + "source": [ + "The multiLogRegPredict function has three return values:\n", + "\n", + "`m`, a matrix with the mean probability of correctly classifying each label. We do not use it further in this example.\n", + "\n", + "`y_pred`, is the predictions made using the model\n", + "\n", + "`acc`, is the accuracy achieved by the model." + ] + }, + { + "cell_type": "markdown", + "id": "bffaa936", + "metadata": {}, + "source": [ + "# Step 3: Confusion Matrix\n", + "A confusion matrix is a useful tool to analyze the performance of the model and to obtain a better understanding which classes the model has difficulties separating. The confusionMatrix function takes the predicted labels and the true labels. It then returns the confusion matrix for the predictions and the confusion matrix averages of each true class." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f3cd2e62", + "metadata": {}, + "outputs": [], + "source": [ + "confusion_matrix_abs, _ = confusionMatrix(y_pred, Yt).compute()" + ] + }, + { + "cell_type": "markdown", + "id": "37bfe729", + "metadata": {}, + "source": [ + "# Full Script" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "54190e79", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Warning: Scuro dependencies missing or wrong version installed: torch 2.4.1, torchvision 0.19.1, librosa 0.10.2, opencv-python 4.10.0.84, opt-einsum 3.3.0, h5py 3.11.0, transformers 4.46.3, nltk 3.9.1, gensim 4.3.3\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Using incubator modules: jdk.incubator.vector\n", + "26/05/06 16:39:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", + "INFO:python_e2e_l1_example:Confusion Matrix\n", + "INFO:python_e2e_l1_example:[[3580. 488.]\n", + " [ 248. 684.]]\n" + ] + } + ], + "source": [ + "from systemds.context import SystemDSContext\n", + "from systemds.examples.tutorials.adult import DataManager\n", + "from systemds.operator.algorithm import multiLogReg\n", + "from systemds.operator.algorithm import multiLogRegPredict\n", + "from systemds.operator.algorithm import confusionMatrix\n", + "from systemds.operator.algorithm import getAccuracy\n", + "\n", + "import logging\n", + "logger = logging.getLogger('python_e2e_l1_example')\n", + "logger.setLevel(logging.INFO)\n", + "\n", + "with SystemDSContext() as sds:\n", + " d = DataManager()\n", + "\n", + " # limit the sample size\n", + " train_count = 15000\n", + " test_count = 5000\n", + "\n", + " # Get train and test datasets.\n", + " X_frame, Y_frame, Xt_frame, Yt_frame = d.get_preprocessed_dataset(sds)\n", + "\n", + " # Transformation specification\n", + " jspec_data = d.get_jspec(sds)\n", + " jspec_labels = sds.scalar(f'\"{ {\"recode\": [\"income\"]} }\"')\n", + "\n", + " # Transform frames to matrices.\n", + " X, M1 = X_frame.transform_encode(spec=jspec_data)\n", + " Xt = Xt_frame.transform_apply(spec=jspec_data, meta=M1) \n", + " Y, M2 = Y_frame.transform_encode(spec=jspec_labels)\n", + " Yt = Yt_frame.transform_apply(spec=jspec_labels, meta=M2) \n", + " \n", + " # Subsample to make training faster\n", + " X = X[0:train_count]\n", + " Y = Y[0:train_count]\n", + " Xt = Xt[0:test_count]\n", + " Yt = Yt[0:test_count]\n", + "\n", + " # Train model \n", + " betas = multiLogReg(X, Y, verbose=False)\n", + "\n", + " # Apply model\n", + " [_, y_pred, acc] = multiLogRegPredict(Xt, betas, Y=Yt)\n", + "\n", + " # Confusion Matrix\n", + " confusion_matrix_abs, _ = confusionMatrix(y_pred, Yt).compute()\n", + " \n", + " # accuracy = getAccuracy(Yt, y_pred).compute()\n", + "\n", + "\n", + " logger.info(\"Confusion Matrix\")\n", + " logger.info(confusion_matrix_abs)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "c2d7049a", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAHHCAYAAAAiSltoAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAVPVJREFUeJzt3Xl8DPf/B/DXJuzm3ESQi4gjhDjraKTuLxLEVVSVEndpaImrWkdQ0p/7po4KFS3V0oo6IsQZRQhxpUVIkE0QySYh9/z+0EytWHaziZB5PT3m8ch85vOZec8+yL59jhmZIAgCiIiISLKMSjoAIiIiKllMBoiIiCSOyQAREZHEMRkgIiKSOCYDREREEsdkgIiISOKYDBAREUkckwEiIiKJYzJAREQkcUwGiF7wzz//wNPTE1ZWVpDJZNi9e3eRnv/27duQyWQIDAws0vO+y9q2bYu2bduWdBhEksVkgN5KN2/exGeffYbq1avDxMQESqUSLVq0wLJly/D06dNivbaPjw+ioqIwd+5c/Pjjj2jatGmxXu9NGjx4MGQyGZRK5Us/x3/++QcymQwymQwLFy7U+/z379+Hv78/IiMjiyBaInpTypR0AEQv2rt3Lz766CMoFAoMGjQI9erVQ1ZWFk6cOIFJkybhypUrWLduXbFc++nTpwgPD8c333yDMWPGFMs1nJ2d8fTpU5QtW7ZYzv86ZcqUwZMnT7Bnzx707dtX41hQUBBMTEyQkZFRqHPfv38fs2bNQtWqVdGoUSOd2x08eLBQ1yOiosFkgN4qMTEx6NevH5ydnXH48GE4ODiIx3x9fXHjxg3s3bu32K7/4MEDAIC1tXWxXUMmk8HExKTYzv86CoUCLVq0wE8//VQgGdi2bRu8vb3x66+/vpFYnjx5AjMzM8jl8jdyPSJ6OQ4T0Ftl/vz5SEtLw8aNGzUSgXwuLi748ssvxf2cnBzMmTMHNWrUgEKhQNWqVfH1118jMzNTo13VqlXRtWtXnDhxAu+//z5MTExQvXp1bNmyRazj7+8PZ2dnAMCkSZMgk8lQtWpVAM+61/N/fp6/vz9kMplGWUhICFq2bAlra2tYWFjA1dUVX3/9tXhc25yBw4cPo1WrVjA3N4e1tTV69OiBa9euvfR6N27cwODBg2FtbQ0rKysMGTIET5480f7BvqB///7Yt28fkpOTxbKzZ8/in3/+Qf/+/QvUT0pKwsSJE1G/fn1YWFhAqVSic+fOuHjxolgnLCwMzZo1AwAMGTJEHG7Iv8+2bduiXr16iIiIQOvWrWFmZiZ+Li/OGfDx8YGJiUmB+/fy8kK5cuVw//59ne+ViF6PyQC9Vfbs2YPq1avjgw8+0Kn+8OHDMWPGDDRu3BhLlixBmzZtEBAQgH79+hWoe+PGDfTp0wcdO3bEokWLUK5cOQwePBhXrlwBAPTq1QtLliwBAHzyySf48ccfsXTpUr3iv3LlCrp27YrMzEzMnj0bixYtQvfu3XHy5MlXtjt06BC8vLyQmJgIf39/+Pn54dSpU2jRogVu375doH7fvn2RmpqKgIAA9O3bF4GBgZg1a5bOcfbq1QsymQy//fabWLZt2zbUrl0bjRs3LlD/1q1b2L17N7p27YrFixdj0qRJiIqKQps2bcQv5jp16mD27NkAgJEjR+LHH3/Ejz/+iNatW4vnefToETp37oxGjRph6dKlaNeu3UvjW7ZsGSpWrAgfHx/k5uYCAL7//nscPHgQK1asgKOjo873SkQ6EIjeEikpKQIAoUePHjrVj4yMFAAIw4cP1yifOHGiAEA4fPiwWObs7CwAEI4dOyaWJSYmCgqFQpgwYYJYFhMTIwAQFixYoHFOHx8fwdnZuUAMM2fOFJ7/Z7RkyRIBgPDgwQOtcedfY9OmTWJZo0aNBFtbW+HRo0di2cWLFwUjIyNh0KBBBa43dOhQjXN++OGHQvny5bVe8/n7MDc3FwRBEPr06SO0b99eEARByM3NFezt7YVZs2a99DPIyMgQcnNzC9yHQqEQZs+eLZadPXu2wL3la9OmjQBAWLt27UuPtWnTRqPswIEDAgDh22+/FW7duiVYWFgIPXv2fO09EpH+2DNAbw21Wg0AsLS01Kn+n3/+CQDw8/PTKJ8wYQIAFJhb4ObmhlatWon7FStWhKurK27dulXomF+UP9fg999/R15enk5t4uPjERkZicGDB8PGxkYsb9CgATp27Cje5/NGjRqlsd+qVSs8evRI/Ax10b9/f4SFhUGlUuHw4cNQqVQvHSIAns0zMDJ69usiNzcXjx49EodAzp8/r/M1FQoFhgwZolNdT09PfPbZZ5g9ezZ69eoFExMTfP/99zpfi4h0x2SA3hpKpRIAkJqaqlP9O3fuwMjICC4uLhrl9vb2sLa2xp07dzTKq1SpUuAc5cqVw+PHjwsZcUEff/wxWrRogeHDh8POzg79+vXDjh07XpkY5Mfp6upa4FidOnXw8OFDpKena5S/eC/lypUDAL3upUuXLrC0tMT27dsRFBSEZs2aFfgs8+Xl5WHJkiWoWbMmFAoFKlSogIoVK+LSpUtISUnR+ZqVKlXSa7LgwoULYWNjg8jISCxfvhy2trY6tyUi3TEZoLeGUqmEo6MjLl++rFe7FyfwaWNsbPzSckEQCn2N/PHsfKampjh27BgOHTqEgQMH4tKlS/j444/RsWPHAnUNYci95FMoFOjVqxc2b96MXbt2ae0VAIB58+bBz88PrVu3xtatW3HgwAGEhISgbt26OveAAM8+H31cuHABiYmJAICoqCi92hKR7pgM0Fula9euuHnzJsLDw19b19nZGXl5efjnn380yhMSEpCcnCyuDCgK5cqV05h5n+/F3gcAMDIyQvv27bF48WJcvXoVc+fOxeHDh3HkyJGXnjs/zujo6ALHrl+/jgoVKsDc3NywG9Cif//+uHDhAlJTU1866TLfzp070a5dO2zcuBH9+vWDp6cnOnToUOAz0TUx00V6ejqGDBkCNzc3jBw5EvPnz8fZs2eL7PxE9B8mA/RWmTx5MszNzTF8+HAkJCQUOH7z5k0sW7YMwLNubgAFZvwvXrwYAODt7V1kcdWoUQMpKSm4dOmSWBYfH49du3Zp1EtKSirQNv/hOy8ud8zn4OCARo0aYfPmzRpfrpcvX8bBgwfF+ywO7dq1w5w5c7By5UrY29trrWdsbFyg1+GXX37BvXv3NMryk5aXJU76mjJlCmJjY7F582YsXrwYVatWhY+Pj9bPkYgKjw8dordKjRo1sG3bNnz88ceoU6eOxhMIT506hV9++QWDBw8GADRs2BA+Pj5Yt24dkpOT0aZNG5w5cwabN29Gz549tS5bK4x+/fphypQp+PDDD/HFF1/gyZMnWLNmDWrVqqUxgW727Nk4duwYvL294ezsjMTERKxevRqVK1dGy5YttZ5/wYIF6Ny5Mzw8PDBs2DA8ffoUK1asgJWVFfz9/YvsPl5kZGSEadOmvbZe165dMXv2bAwZMgQffPABoqKiEBQUhOrVq2vUq1GjBqytrbF27VpYWlrC3Nwc7u7uqFatml5xHT58GKtXr8bMmTPFpY6bNm1C27ZtMX36dMyfP1+v8xHRa5Twagail/r777+FESNGCFWrVhXkcrlgaWkptGjRQlixYoWQkZEh1svOzhZmzZolVKtWTShbtqzg5OQkTJ06VaOOIDxbWujt7V3gOi8uadO2tFAQBOHgwYNCvXr1BLlcLri6ugpbt24tsLQwNDRU6NGjh+Do6CjI5XLB0dFR+OSTT4S///67wDVeXH536NAhoUWLFoKpqamgVCqFbt26CVevXtWok3+9F5cubtq0SQAgxMTEaP1MBUFzaaE22pYWTpgwQXBwcBBMTU2FFi1aCOHh4S9dEvj7778Lbm5uQpkyZTTus02bNkLdunVfes3nz6NWqwVnZ2ehcePGQnZ2tka98ePHC0ZGRkJ4ePgr74GI9CMTBD1mHBEREVGpwzkDREREEsdkgIiISOKYDBAREUkckwEiIiKJYzJAREQkcUwGiIiIJO6dfuhQXl4e7t+/D0tLyyJ9DCoREb0ZgiAgNTUVjo6O4psxi0NGRgaysrIMPo9cLoeJiUkRRPR2eaeTgfv378PJyamkwyAiIgPFxcWhcuXKxXLujIwMmFqWB3KeGHwue3t7xMTElLqE4J1OBvLfey9384HMWPfXohK9Sy7/GVDSIRAVm9TUVDR2qyb+Pi8OWVlZQM4TKNx8AEO+K3KzoLq6GVlZWUwG3ib5QwMyYzmTASq1LJXKkg6BqNi9kaHeMiYGfVcIstI7ze6dTgaIiIh0JgNgSNJRiqemMRkgIiJpkBk92wxpX0qV3jsjIiIqQWvWrEGDBg2gVCqhVCrh4eGBffv2icfbtm0LmUymsY0aNUrjHLGxsfD29oaZmRlsbW0xadIk5OTkaNQJCwtD48aNoVAo4OLigsDAQL1jZc8AERFJg0xm4DCBfm0rV66M7777DjVr1oQgCNi8eTN69OiBCxcuoG7dugCAESNGYPbs2WIbMzMz8efc3Fx4e3vD3t4ep06dQnx8PAYNGoSyZcti3rx5AICYmBh4e3tj1KhRCAoKQmhoKIYPHw4HBwd4eXnpHCuTASIikoY3PEzQrVs3jf25c+dizZo1OH36tJgMmJmZwd7e/qXtDx48iKtXr+LQoUOws7NDo0aNMGfOHEyZMgX+/v6Qy+VYu3YtqlWrhkWLFgEA6tSpgxMnTmDJkiV6JQMcJiAiItKDWq3W2DIzM1/bJjc3Fz///DPS09Ph4eEhlgcFBaFChQqoV68epk6diidP/nsWQnh4OOrXrw87OzuxzMvLC2q1GleuXBHrdOjQQeNaXl5eCA8P1+ue2DNARETSUETDBC8+7G7mzJnw9/d/aZOoqCh4eHggIyMDFhYW2LVrF9zc3AAA/fv3h7OzMxwdHXHp0iVMmTIF0dHR+O233wAAKpVKIxEAIO6rVKpX1lGr1Xj69ClMTU11ujUmA0REJBEGDhP825keFxcH5XPP/1AoFFpbuLq6IjIyEikpKdi5cyd8fHxw9OhRuLm5YeTIkWK9+vXrw8HBAe3bt8fNmzdRo0YNA+LUH4cJiIiI9JC/OiB/e1UyIJfL4eLigiZNmiAgIAANGzbEsmXLXlrX3d0dAHDjxg0Azx59nJCQoFEnfz9/noG2OkqlUudeAYDJABERSUX+MIEhm4Hy8vK0zjGIjIwEADg4OAAAPDw8EBUVhcTERLFOSEgIlEqlONTg4eGB0NBQjfOEhIRozEvQBYcJiIhIGt7waoKpU6eic+fOqFKlClJTU7Ft2zaEhYXhwIEDuHnzJrZt24YuXbqgfPnyuHTpEsaPH4/WrVujQYMGAABPT0+4ublh4MCBmD9/PlQqFaZNmwZfX1+xN2LUqFFYuXIlJk+ejKFDh+Lw4cPYsWMH9u7dq1esTAaIiIiKQWJiIgYNGoT4+HhYWVmhQYMGOHDgADp27Ii4uDgcOnQIS5cuRXp6OpycnNC7d29MmzZNbG9sbIzg4GCMHj0aHh4eMDc3h4+Pj8ZzCapVq4a9e/di/PjxWLZsGSpXrowNGzbotawQAGSCIAhFdudvmFqthpWVFRT1R/BFRVRq3T66pKRDICo2qWo1ajpVQEpKisakvKIkfle8PwGyMtrH919HyMlE5plFxRprSWHPABERSQPfTaAVkwEiIpKGN/w44ndJ6U1ziIiISCfsGSAiImngMIFWTAaIiEgaZDIDkwEOExAREVEpxZ4BIiKSBiPZs82Q9qUUkwEiIpIGzhnQqvTeGREREemEPQNERCQNfM6AVkwGiIhIGjhMoFXpvTMiIiLSCXsGiIhIGjhMoBWTASIikgYOE2jFZICIiKSBPQNald40h4iIiHTCngEiIpIGDhNoxWSAiIikgcMEWpXeNIeIiIh0wp4BIiKSCAOHCUrx/5+ZDBARkTRwmECr0pvmEBERkU7YM0BERNIgkxm4mqD09gwwGSAiImng0kKtSu+dERERkU7YM0BERNLACYRaMRkgIiJp4DCBVkwGiIhIGtgzoFXpTXOIiIhIJ+wZICIiaeAwgVZMBoiISBo4TKBV6U1ziIiISCfsGSAiIkmQyWSQsWfgpZgMEBGRJDAZ0I7DBERERBLHngEiIpIG2b+bIe1LKSYDREQkCRwm0I7DBERERBLHngEiIpIE9gxox2SAiIgkgcmAdkwGiIhIEpgMaMc5A0RERBLHngEiIpIGLi3UiskAERFJAocJtOMwARERUTFYs2YNGjRoAKVSCaVSCQ8PD+zbt088npGRAV9fX5QvXx4WFhbo3bs3EhISNM4RGxsLb29vmJmZwdbWFpMmTUJOTo5GnbCwMDRu3BgKhQIuLi4IDAzUO1YmA0REJAnP3mAsM2DT73qVK1fGd999h4iICJw7dw7/+9//0KNHD1y5cgUAMH78eOzZswe//PILjh49ivv376NXr15i+9zcXHh7eyMrKwunTp3C5s2bERgYiBkzZoh1YmJi4O3tjXbt2iEyMhLjxo3D8OHDceDAAf0+G0EQBP1u7+2hVqthZWUFRf0RkBnLSzocomJx++iSkg6BqNikqtWo6VQBKSkpUCqVxXKN/O8K677rIZObFfo8QtYTJO8YYVCsNjY2WLBgAfr06YOKFSti27Zt6NOnDwDg+vXrqFOnDsLDw9G8eXPs27cPXbt2xf3792FnZwcAWLt2LaZMmYIHDx5ALpdjypQp2Lt3Ly5fvixeo1+/fkhOTsb+/ft1jos9A0RERHpQq9UaW2Zm5mvb5Obm4ueff0Z6ejo8PDwQERGB7OxsdOjQQaxTu3ZtVKlSBeHh4QCA8PBw1K9fX0wEAMDLywtqtVrsXQgPD9c4R36d/HPoiskAERFJgmFDBP9NPnRycoKVlZW4BQQEaL1mVFQULCwsoFAoMGrUKOzatQtubm5QqVSQy+WwtrbWqG9nZweVSgUAUKlUGolA/vH8Y6+qo1ar8fTpU50/G64mICIiaSiipYVxcXEawwQKhUJrE1dXV0RGRiIlJQU7d+6Ej48Pjh49akAQxYPJABERkR7yVwfoQi6Xw8XFBQDQpEkTnD17FsuWLcPHH3+MrKwsJCcna/QOJCQkwN7eHgBgb2+PM2fOaJwvf7XB83VeXIGQkJAApVIJU1NTne+JwwRERCQNhg4RFMFzBvLy8pCZmYkmTZqgbNmyCA0NFY9FR0cjNjYWHh4eAAAPDw9ERUUhMTFRrBMSEgKlUgk3NzexzvPnyK+Tfw5dsWeAiIgkwdCHDunbdurUqejcuTOqVKmC1NRUbNu2DWFhYThw4ACsrKwwbNgw+Pn5wcbGBkqlEmPHjoWHhweaN28OAPD09ISbmxsGDhyI+fPnQ6VSYdq0afD19RWHJkaNGoWVK1di8uTJGDp0KA4fPowdO3Zg7969esXKZICIiCThTScDiYmJGDRoEOLj42FlZYUGDRrgwIED6NixIwBgyZIlMDIyQu/evZGZmQkvLy+sXr1abG9sbIzg4GCMHj0aHh4eMDc3h4+PD2bPni3WqVatGvbu3Yvx48dj2bJlqFy5MjZs2AAvLy/97o3PGSB6u/E5A1SavcnnDJQfsAlGBjxnIC/rCR4FDSnWWEsKewaIiEga+KIirZgMEBGRJLzpYYJ3CVcTEBERSRx7BoiISBLYM6AdkwEiIpIEJgPacZiAiIhI4tgzQEREksCeAe2YDBARkTRwaaFWHCYgIiKSOPYMEBGRJHCYQDsmA0REJAlMBrRjMkBERJLAZEA7zhkgIiKSOPYMEBGRNHA1gVZMBoiISBI4TKAdhwmIiIgkjj0D77ihvVtiaO9WcHKwAQBcv6XCgo37cOjUVa1tlBammP55N3Rt1xDllGaIi3+MrxfvRMgr2hiqR/v38PUob1RxKI9bcQ/gv2K3xvWmjOiCXp6NUcmuHLKzcxF5PRbfrt6DiCt3ii0mevesDjqE+ev2Ykif1pg59sOX1snOycXqrYfw64GzUD1MQXUnW3z1WVe0da9TrLHtPRKJRT/sw11VEqpVqoivRnVFu+ZuYkwLN/yJsNPXEBv/CJbmJmjZpBamfNYVdhWsijUu+g97BrR7K3oGVq1ahapVq8LExATu7u44c+ZMSYf0zrifmIxZK39Hu0Hz8T+fBTh+7m8ELRyJ2tXtX1q/bBlj7Fo1BlUcbDB4ykY06zMHX87bhvgHKYWOoUXjmrj4+yytx99vUA0bvh2Mrb+Ho82n32Hv0YvYunAk6tRwEOvcjE3E5AW/oMUn89B5xGLE3k/CbyvHoLy1RaHjotLl4rVYbPsjHLVrOL6y3sINf2LbnnDM+rIXDm2eggHdP8Bn0zbh8t93C33t8As30OLj2VqPR1yOwRdzfsTHXdzx5/qJ8GxVDyO/+QHRt+IBAE8zsnDl77sYO6gjgtdPwNo5Q3AzLhHDv95Q6JhIfzLIxISgUFspnjRQ4snA9u3b4efnh5kzZ+L8+fNo2LAhvLy8kJiYWNKhvRP2H7+MkFNXcSvuAW7GJuLbNXuQ/iQTTetVe2n9T7t7oJzSDAMmrsNfl24hLj4Jp87fwOV/7ol1ZDIZxg/2RORuf9w/vhjHg75C9/81KnSMn/Vri9Dwa1ixNRR/307AvLV7cfF6HEZ81Eass/PAORw9E4079x7h+i0Vpi39DUoLU9St+epf/CQN6U8yMe7brfhuUl9YWZq+su6ug+fg+2kHtGvuhiqOFTCwZwu0a14HG3aEiXXy8vKwaushtPx4Dlw7TkanoQvwZ1hkoeP7YecxtHm/Nj775H9wqWqHCcO6oG6tyti86ziAZ71xWxePRtf/vYcaVWzRuG5VzP6yN6Ki7+JewuNCX5eoqJR4MrB48WKMGDECQ4YMgZubG9auXQszMzP88MMPJR3aO8fISIZeHZvAzFSOs1ExL63TuXV9nI2KwYIpHyN6/zyc+vlr+A32hJHRfxmv32BPfNzlffh9tx0e/eZi9U9H8P1sH3zQ2KVQcb1fvxrCzl7XKDt8+hqa1a/60vplyxjD58MWSEl9gst/33tpHZKW6Ut3op1HHbRs6vraulnZOVDINUdATRRlcTbqlri/OigUvx04i7kTPkLI5skY9lEbjJsbhNORNwoV34Urt9GiSS2NstbNXHH+FcNcqelPIZPJoLR4dXJDRcegXgEDhxjediU6ZyArKwsRERGYOnWqWGZkZIQOHTogPDy8BCN7t7jVcMSBHybARF4G6U8zMXDSekTHqF5a17lSebRqWgu/7D+LvuPWoLpTRSyc/DHKlDHG/A37IC9bBuOHeOJD35ViQnHn3iM0b1gDQz5siVPn9f9laVteiQePUjXKHiSlwra8UqPMq2U9bJg7BGYmZaF6qMaHY1YiKSVd7+tR6fJH6Hlc+fsefv9+vE71WzerjQ07wvB+wxpwdiyPkxH/YP+xS8jLywMAZGblYNXWQ9i6aDSa1KsKAKjiWAHnom5h2x/haN5I/6T3QVIqKpSz1CirWM4SD5PUL62fkZmN774PRvf278HS3ETv61EhcWmhViWaDDx8+BC5ubmws7PTKLezs8P169cL1M/MzERmZqa4r1a//B+a1PxzJwGtBwRAaWGKHu3fw2r/gej62bKXJgRGMiM8fJyKcfN+Ql6egIvX4+BQ0RpjB7bH/A37UN2pAsxNFfht5RiNdvKyxrgU/d+Ya9zRReLPxkYyKORlNMp+2XcWft/9rNd9HD/3N1oPCEB5awsM6vkBNs0big5DFuLh4zS9zkOlx/3Ex5i9Yhd+XDQaJoqyOrWZ+cWH+GrBdrQfGACZTAZnx/L4qPP72PHns7lId+49wNOMLAycuEajXXZ2LtxqVhL33TpNEX/OzRWQlZ2jUdazYxPMm9BX73vKzsnFGP/NEAQB3/p9pHd7ouLwTq0mCAgIwKxZ2ieqSVV2Ti5i7j4EAFy8Hof33KpgVL+2GB9Q8Ms44VEKsnNykZcniGV/31bBvoIVypYxhrmpAgDw8fg1iE9M1miblZ0j/tx6QID4c9N6VTFzTA90G7VMLEtNzxB/TnykRsXyL/yvycYSiY80k7knGVmIufsQMXcf4tzl2zj36wwM7PEBlgQe1PWjoFImKvouHj5OQ9cR/yWaubl5OHPxFrbsOoG/QxbA2FhztLO8tQXWzx2GjMxsJKvTYVfBCt99H4wqjs9W3KQ/zQIA/PDdCNi/MJNf/tzwwp8bJoo/R16LxXff78HPS33FMovn/kdf0cYSDx+/0Pv1OBUVbDR7v7JzcuE7czPuJjzGT0s+Z6/AG8bVBNqVaDJQoUIFGBsbIyEhQaM8ISEB9vYFZ8NPnToVfn5+4r5arYaTk1Oxx/muMZLJNH6pPe+vi7fQx6spZDIZBOFZQlCjii3iHzxLEqJjVMjIzIaTXblXDgnkJx8A4GhbDrm5eRplzzsTFYM2zVyx9qcwsayde22cjbr96vswkkFe9p3KV6mItWhSEwc2TdYom/TdT6hRxRaj+rcvkAg8z0RRFvYVrZGdk4v9xy7Bu20jAEDNqnaQy8vgfsLjVw4JVK1cUfw5/kEKjI2NNMqe917dqjgV8TeGPTcp9sS5v9G4rrO4n58I3L73AD8t9UU5K/NX3jsVPSYD2pXob1q5XI4mTZogNDQUPXv2BPBslm9oaCjGjBlToL5CoYBCoXjDUb7dZvh2x6FTVxCnegxLMxP06dQULZvURO+xqwEAa/wHIv5BCmav+gMA8MOvxzH8o9b4bkIfrNtxFDWcKsJvsCfWbT8KAEh7komVW0Mx1683ZEZGOB15E0oLE7g3rIHU9Az8vPcvvWP8/ucwBH8/Dr4D/oeDJ66gl2cTNKpTBePm/QQAMDORY8JQL+w7FoWEhymwsbbA8I9aw6GiNX4PPV9EnxS9iyzMTOBa3UGjzNRUDmsrc7Hcb24Q7CpaYcrIrgCAC1fvIOFhCtxcHKF6kIKlgQeQl5eHzz75n3jOkR+3w5xVvyNPENCsfnWkpj/FuagYWJiboE+n9/WOc2if1vj4i5VYv/0I2jV3w57DFxAVHYeAic+GEbJzcjF6RiCu/H0XG78bjtzcPLFnzFppxqT3DZHJnm2GtC+tSvxvoJ+fH3x8fNC0aVO8//77WLp0KdLT0zFkyJCSDu2dUKGcBdb4D4JdBSXUaRm4cuMeeo9djbAzz+ZcVLa3QZ7w35DAvYRk9PliNeaO74UT26Yi/kEyvv85DEu3hIh15q4NxsPkNIwf3BFVK32ClNSnuBgdhyWbDhQqxjOXYjBiWiC+Gd0V0z/vhltxD/DpxHW4dvPZGuzcvDzUrGqHft7uKG9tjqSUJ7hw9Q66jFyC67dePhGSKN+9xMeQPbcaJjMrGws3/InY+EcwN1WgnXsdLPlmgMaSxAnDOsPG2hyrg0IRd3/Hs2WstSrD99MOhYqhSb1qWDZ9IBZt/BML1u9F1coVsW7uUDFhUT1IwaGTlwEAXYYt1Gj701JfeLxXuJU6REVFJgjPfVOUkJUrV2LBggVQqVRo1KgRli9fDnd399e2U6vVsLKygqL+CMiM5W8gUqI37/bRJSUdAlGxSVWrUdOpAlJSUqBUKl/foBDyvyuqj90JI0Xhh2fyMtNxa0WfYo21pJR4zwAAjBkz5qXDAkREREXGwGGC0ry0sMQfOkREREQl663oGSAiIipuXE2gHZMBIiKSBK4m0I7DBERERBLHngEiIpIEIyOZxkvZ9CUY0PZtx2SAiIgkgcME2nGYgIiISOLYM0BERJLA1QTaMRkgIiJJ4DCBdkwGiIhIEtgzoB3nDBAREUkcewaIiEgS2DOgHZMBIiKSBM4Z0I7DBERERBLHngEiIpIEGQwcJijF7zBmMkBERJLAYQLtOExAREQkcUwGiIhIEvJXExiy6SMgIADNmjWDpaUlbG1t0bNnT0RHR2vUadu2bYFrjBo1SqNObGwsvL29YWZmBltbW0yaNAk5OTkadcLCwtC4cWMoFAq4uLggMDBQr1iZDBARkSTkDxMYsunj6NGj8PX1xenTpxESEoLs7Gx4enoiPT1do96IESMQHx8vbvPnzxeP5ebmwtvbG1lZWTh16hQ2b96MwMBAzJgxQ6wTExMDb29vtGvXDpGRkRg3bhyGDx+OAwcO6Bwr5wwQEREVg/3792vsBwYGwtbWFhEREWjdurVYbmZmBnt7+5ee4+DBg7h69SoOHToEOzs7NGrUCHPmzMGUKVPg7+8PuVyOtWvXolq1ali0aBEAoE6dOjhx4gSWLFkCLy8vnWJlzwAREUlCUQ0TqNVqjS0zM1On66ekpAAAbGxsNMqDgoJQoUIF1KtXD1OnTsWTJ0/EY+Hh4ahfvz7s7OzEMi8vL6jValy5ckWs06FDB41zenl5ITw8XOfPhj0DREQkCUW1msDJyUmjfObMmfD3939l27y8PIwbNw4tWrRAvXr1xPL+/fvD2dkZjo6OuHTpEqZMmYLo6Gj89ttvAACVSqWRCAAQ91Uq1SvrqNVqPH36FKampq+9NyYDREQkCUX1OOK4uDgolUqxXKFQvLatr68vLl++jBMnTmiUjxw5Uvy5fv36cHBwQPv27XHz5k3UqFGj0LHqi8MEREREelAqlRrb65KBMWPGIDg4GEeOHEHlypVfWdfd3R0AcOPGDQCAvb09EhISNOrk7+fPM9BWR6lU6tQrADAZICIiqTB0JYGenQqCIGDMmDHYtWsXDh8+jGrVqr22TWRkJADAwcEBAODh4YGoqCgkJiaKdUJCQqBUKuHm5ibWCQ0N1ThPSEgIPDw8dI6VyQAREUnCm37OgK+vL7Zu3Ypt27bB0tISKpUKKpUKT58+BQDcvHkTc+bMQUREBG7fvo0//vgDgwYNQuvWrdGgQQMAgKenJ9zc3DBw4EBcvHgRBw4cwLRp0+Dr6yv2SIwaNQq3bt3C5MmTcf36daxevRo7duzA+PHjdY6VyQAREVExWLNmDVJSUtC2bVs4ODiI2/bt2wEAcrkchw4dgqenJ2rXro0JEyagd+/e2LNnj3gOY2NjBAcHw9jYGB4eHvj0008xaNAgzJ49W6xTrVo17N27FyEhIWjYsCEWLVqEDRs26LysEOAEQiIikog3/W4CQRBeedzJyQlHjx597XmcnZ3x559/vrJO27ZtceHCBb3iex6TASIikoSiWk1QGnGYgIiISOLYM0BERJLAVxhrx2SAiIgkgcME2nGYgIiISOLYM0BERJLAngHtmAwQEZEkcM6AdkwGiIhIEtgzoB3nDBAREUkcewaIiEgSOEygHZMBIiKSBA4TaMdhAiIiIoljzwAREUmCDAYOExRZJG8fJgNERCQJRjIZjAzIBgxp+7bjMAEREZHEsWeAiIgkgasJtGMyQEREksDVBNoxGSAiIkkwkj3bDGlfWnHOABERkcSxZ4CIiKRBZmBXfynuGWAyQEREksAJhNpxmICIiEji2DNARESSIPv3jyHtSysmA0REJAlcTaAdhwmIiIgkjj0DREQkCXzokHY6JQN//PGHzifs3r17oYMhIiIqLlxNoJ1OyUDPnj11OplMJkNubq4h8RAREdEbplMykJeXV9xxEBERFSu+wlg7g+YMZGRkwMTEpKhiISIiKjYcJtBO79UEubm5mDNnDipVqgQLCwvcunULADB9+nRs3LixyAMkIiIqCvkTCA3ZSiu9k4G5c+ciMDAQ8+fPh1wuF8vr1auHDRs2FGlwREREVPz0Tga2bNmCdevWYcCAATA2NhbLGzZsiOvXrxdpcEREREUlf5jAkK200nvOwL179+Di4lKgPC8vD9nZ2UUSFBERUVHjBELt9O4ZcHNzw/HjxwuU79y5E++9916RBEVERERvjt49AzNmzICPjw/u3buHvLw8/Pbbb4iOjsaWLVsQHBxcHDESEREZTPbvZkj70krvnoEePXpgz549OHToEMzNzTFjxgxcu3YNe/bsQceOHYsjRiIiIoNxNYF2hXrOQKtWrRASElLUsRAREVEJKPRDh86dO4dr164BeDaPoEmTJkUWFBERUVHjK4y10zsZuHv3Lj755BOcPHkS1tbWAIDk5GR88MEH+Pnnn1G5cuWijpGIiMhgfGuhdnrPGRg+fDiys7Nx7do1JCUlISkpCdeuXUNeXh6GDx9eHDESERFRMdK7Z+Do0aM4deoUXF1dxTJXV1esWLECrVq1KtLgiIiIilIp/s+9QfROBpycnF76cKHc3Fw4OjoWSVBERERFjcME2uk9TLBgwQKMHTsW586dE8vOnTuHL7/8EgsXLizS4IiIiIpK/gRCQ7bSSqdkoFy5crCxsYGNjQ2GDBmCyMhIuLu7Q6FQQKFQwN3dHefPn8fQoUOLO14iIqJ3QkBAAJo1awZLS0vY2tqiZ8+eiI6O1qiTkZEBX19flC9fHhYWFujduzcSEhI06sTGxsLb2xtmZmawtbXFpEmTkJOTo1EnLCwMjRs3hkKhgIuLCwIDA/WKVadhgqVLl+p1UiIiorfNmx4mOHr0KHx9fdGsWTPk5OTg66+/hqenJ65evQpzc3MAwPjx47F371788ssvsLKywpgxY9CrVy+cPHkSwLMheG9vb9jb2+PUqVOIj4/HoEGDULZsWcybNw8AEBMTA29vb4waNQpBQUEIDQ3F8OHD4eDgAC8vL93uTRAEQa+7e4uo1WpYWVlBUX8EZMby1zcgegfdPrqkpEMgKjapajVqOlVASkoKlEplsVwj/7tiwMZTkJtZFPo8WU/SEDTsg0LH+uDBA9ja2uLo0aNo3bo1UlJSULFiRWzbtg19+vQBAFy/fh116tRBeHg4mjdvjn379qFr1664f/8+7OzsAABr167FlClT8ODBA8jlckyZMgV79+7F5cuXxWv169cPycnJ2L9/v06x6T1n4HkZGRlQq9UaGxERUWn24vdeZmamTu1SUlIAADY2NgCAiIgIZGdno0OHDmKd2rVro0qVKggPDwcAhIeHo379+mIiAABeXl5Qq9W4cuWKWOf5c+TXyT+HLvROBtLT0zFmzBjY2trC3Nwc5cqV09iIiIjeRvmvMDZkA56tqrOyshK3gICA1147Ly8P48aNQ4sWLVCvXj0AgEqlglwuFx/gl8/Ozg4qlUqs83wikH88/9ir6qjVajx9+lSnz0bvpYWTJ0/GkSNHsGbNGgwcOBCrVq3CvXv38P333+O7777T93RERERvhExm2HMG8tvGxcVpDBMoFIrXtvX19cXly5dx4sSJwgdQjPROBvbs2YMtW7agbdu2GDJkCFq1agUXFxc4OzsjKCgIAwYMKI44iYiI3gpKpVKvOQNjxoxBcHAwjh07pvHIfnt7e2RlZSE5OVmjdyAhIQH29vZinTNnzmicL3+1wfN1XlyBkJCQAKVSCVNTU51i1HuYICkpCdWrVwfw7ANJSkoCALRs2RLHjh3T93RERERvxJt+hbEgCBgzZgx27dqFw4cPo1q1ahrHmzRpgrJlyyI0NFQsi46ORmxsLDw8PAAAHh4eiIqKQmJiolgnJCQESqUSbm5uYp3nz5FfJ/8cutA7GahevTpiYmIAPJvosGPHDgDPegxeHPcgIiJ6W+QPExiy6cPX1xdbt27Ftm3bYGlpCZVKBZVKJY7jW1lZYdiwYfDz88ORI0cQERGBIUOGwMPDA82bNwcAeHp6ws3NDQMHDsTFixdx4MABTJs2Db6+vuLwxKhRo3Dr1i1MnjwZ169fx+rVq7Fjxw6MHz9e51j1TgaGDBmCixcvAgC++uorrFq1CiYmJhg/fjwmTZqk7+mIiIhKpTVr1iAlJQVt27aFg4ODuG3fvl2ss2TJEnTt2hW9e/dG69atYW9vj99++008bmxsjODgYBgbG8PDwwOffvopBg0ahNmzZ4t1qlWrhr179yIkJAQNGzbEokWLsGHDBp2fMQAUwXMG7ty5g4iICLi4uKBBgwaGnEpvfM4ASQGfM0Cl2Zt8zsDQLX8Z/JyBHwa5F2usJUXvCYQvcnZ2hrOzc1HEQkREVGyKajVBaaRTMrB8+XKdT/jFF18UOhgiIqLiwrcWaqdTMrBkiW7dlDKZjMkAERHRO0anZCB/9cDbKjZsYakbvyHKd/+xbk8QI3oXpWflvL5SETGCYc/gN+j5/W85g+cMEBERvQs4TKBdaU50iIiISAfsGSAiIkmQyQAjriZ4KSYDREQkCUYGJgOGtH3bcZiAiIhI4gqVDBw/fhyffvopPDw8cO/ePQDAjz/++Na+mpGIiOhNv6joXaJ3MvDrr7/Cy8sLpqamuHDhAjIzMwEAKSkpmDdvXpEHSEREVBTyhwkM2UorvZOBb7/9FmvXrsX69etRtmxZsbxFixY4f/58kQZHRERExU/vCYTR0dFo3bp1gXIrKyskJycXRUxERERFju8m0E7vngF7e3vcuHGjQPmJEydQvXr1IgmKiIioqBnJZAZvpZXeycCIESPw5Zdf4q+//oJMJsP9+/cRFBSEiRMnYvTo0cURIxERkcGMimArrfQeJvjqq6+Ql5eH9u3b48mTJ2jdujUUCgUmTpyIsWPHFkeMREREVIz0TgZkMhm++eYbTJo0CTdu3EBaWhrc3NxgYWFRHPEREREVCc4Z0K7QTyCUy+Vwc3MryliIiIiKjREMG/c3QunNBvROBtq1a/fKBy8cPnzYoICIiIjozdI7GWjUqJHGfnZ2NiIjI3H58mX4+PgUVVxERERFisME2umdDCxZsuSl5f7+/khLSzM4ICIiouLAFxVpV2QrJT799FP88MMPRXU6IiIiekOK7BXG4eHhMDExKarTERERFSmZDAZNIOQwwXN69eqlsS8IAuLj43Hu3DlMnz69yAIjIiIqSpwzoJ3eyYCVlZXGvpGREVxdXTF79mx4enoWWWBERET0ZuiVDOTm5mLIkCGoX78+ypUrV1wxERERFTlOINROrwmExsbG8PT05NsJiYjonSMrgj+lld6rCerVq4dbt24VRyxERETFJr9nwJCttNI7Gfj2228xceJEBAcHIz4+Hmq1WmMjIiKid4vOcwZmz56NCRMmoEuXLgCA7t27azyWWBAEyGQy5ObmFn2UREREBuKcAe10TgZmzZqFUaNG4ciRI8UZDxERUbGQyWSvfLeOLu1LK52TAUEQAABt2rQptmCIiIjozdNraWFpzoqIiKh04zCBdnolA7Vq1XptQpCUlGRQQERERMWBTyDUTq9kYNasWQWeQEhERETvNr2SgX79+sHW1ra4YiEiIio2RjKZQS8qMqTt207nZIDzBYiI6F3GOQPa6fzQofzVBERERFS66NwzkJeXV5xxEBERFS8DJxCW4lcT6P8KYyIioneREWQwMuAb3ZC2bzsmA0REJAlcWqid3i8qIiIiotKFPQNERCQJXE2gHZMBIiKSBD5nQDsOExAREUkckwEiIpKE/AmEhmz6OHbsGLp16wZHR0fIZDLs3r1b4/jgwYPF1yrnb506ddKok5SUhAEDBkCpVMLa2hrDhg1DWlqaRp1Lly6hVatWMDExgZOTE+bPn6/3Z8NkgIiIJMEIMnGooFCbnksL09PT0bBhQ6xatUprnU6dOiE+Pl7cfvrpJ43jAwYMwJUrVxASEoLg4GAcO3YMI0eOFI+r1Wp4enrC2dkZERERWLBgAfz9/bFu3Tq9YuWcASIiomLQuXNndO7c+ZV1FAoF7O3tX3rs2rVr2L9/P86ePYumTZsCAFasWIEuXbpg4cKFcHR0RFBQELKysvDDDz9ALpejbt26iIyMxOLFizWShtdhzwAREUlCUQ0TqNVqjS0zM7PQMYWFhcHW1haurq4YPXo0Hj16JB4LDw+HtbW1mAgAQIcOHWBkZIS//vpLrNO6dWvI5XKxjpeXF6Kjo/H48WOd42AyQEREkmBUBBsAODk5wcrKStwCAgIKFU+nTp2wZcsWhIaG4v/+7/9w9OhRdO7cGbm5uQAAlUpV4E3BZcqUgY2NDVQqlVjHzs5Oo07+fn4dXXCYgIiISA9xcXFQKpXivkKhKNR5+vXrJ/5cv359NGjQADVq1EBYWBjat29vcJz6YM8AERFJwosz9wuzAYBSqdTYCpsMvKh69eqoUKECbty4AQCwt7dHYmKiRp2cnBwkJSWJ8wzs7e2RkJCgUSd/X9tchJdhMkBERJIgK4KtON29exePHj2Cg4MDAMDDwwPJycmIiIgQ6xw+fBh5eXlwd3cX6xw7dgzZ2dlinZCQELi6uqJcuXI6X5vJABERSYJBywoL8fTCtLQ0REZGIjIyEgAQExODyMhIxMbGIi0tDZMmTcLp06dx+/ZthIaGokePHnBxcYGXlxcAoE6dOujUqRNGjBiBM2fO4OTJkxgzZgz69esHR0dHAED//v0hl8sxbNgwXLlyBdu3b8eyZcvg5+en32ejV20iIiLSyblz5/Dee+/hvffeAwD4+fnhvffew4wZM2BsbIxLly6he/fuqFWrFoYNG4YmTZrg+PHjGsMOQUFBqF27Ntq3b48uXbqgZcuWGs8QsLKywsGDBxETE4MmTZpgwoQJmDFjhl7LCgFOICQiIgl5k28XaNu2LQRB0Hr8wIEDrz2HjY0Ntm3b9so6DRo0wPHjx/WO73lMBoiISBIK80jhF9uXVhwmICIikjj2DBARkSQ8vzywsO1LKyYDREQkCc8/RbCw7Uur0nxvREREpAP2DBARkSRwmEA7JgNERCQJhj5FsPSmAhwmICIikjz2DBARkSRwmEA7JgNERCQJXE2gHZMBIiKSBPYMaFeaEx0iIiLSAXsGiIhIEriaQDsmA0REJAl8UZF2HCYgIiKSOPYMEBGRJBhBBiMDOvsNafu2YzJARESSwGEC7ThMQEREJHHsGSAiIkmQ/fvHkPalFZMBIiKSBA4TaMdhAiIiIoljzwAREUmCzMDVBBwmICIiesdxmEA7JgNERCQJTAa045wBIiIiiWPPABERSQKXFmrHZICIiCTBSPZsM6R9acVhAiIiIoljzwAREUkChwm0YzJARESSwNUE2nGYgIiISOLYM0BERJIgg2Fd/aW4Y4DJABERSQNXE2jHYQIiIiKJY8/AO27xpgMIPnIR/9xJgImiLN5vUB3+Y3qgZlU7rW227TkN39lbNcoU8jJQnVxarLGu33EUK7aGIvGRGvVqVsL/TfoITepWBQA8TklHwLq9OHL6Ou4mPEZ5awt4t22Ar0d1hZWFabHGRW+3hIcpWLxhL46fjUZGZhaqOFbAtxP7ol4tJ61tgkPPY+MvYYi99xAW5iZo1aw2Jo7whrXSvNjiPHPxJuZ/vwc37qhgX9Ean/Vvjw89m4nH1/90GCEnoxAT9wAm8jJo5FYVfsO7oJqTbbHFRJq4mkC7Eu0ZOHbsGLp16wZHR0fIZDLs3r27JMN5J506fwPDP2qNgz9MxG8rxyA7Jxe9xq5E+tPMV7azNDfB9X3zxO3SH7MNimPbntPo+tlSrcd/OxiBaUt3Ycrwzgj7cQrq1ayE3mNX4UFSKgAg/kEKVA9SMPvLD3Hq56+xeuanCA2/ii/mBBkUF73bUlKf4NPxq1CmjDHWzh2GP9ZPwqSRXaF8RYJ4/koMpi74Gb293sfv6yZi8bSBiLoei5lLdhY6jnuqJNT1nKT1+N34JHw+bSPeb1gDv64Zj4EftsLMxTtx4ly0WOds1E180v0D/LRsDNZ/NxI5ubkYMXU9njzNKnRcpJ/81QSGbKVVifYMpKeno2HDhhg6dCh69epVkqG8s3au8NXYXz3zU9T0nIrIa3Fo0dhFazuZTAa7CkqtxzOzsvHt6j349WAEUlKfok4NB/iP7YGWTWoVKs7V2w5jUM8PMKC7BwBg8dR+OHjyCrb+EY7xgz3h5uKILfNHiPWrVa6IaaO74bMZW5CTk4syZYwLdV16t23cEQb7itaYO/Fjsayyg80r21y8egeV7Mrh0w9bivX7ejfHxh1hGvV27vsLm3cew11VEirZlcOAni3xSfcPChXn9r3hqGRvg8mfdQMA1KhihwuXY7Dlt2No2dQVALBu3giNNnMnfoxWfWfh6j930bRB9UJdl/Qjg2GTAEtxLlCyyUDnzp3RuXPnkgyh1FGnZQAAyinNXlkv/Wkm6nebjrw8AQ1rO2H6591Rp4aDeHzy/F9wPUaFDXOHwKGiFYKPXESfL1bj5E9fo0YV/bo1s7JzEHk9DuMHe4plRkZGaPO+K85GxbzyXizNTZgISNiR8Cto0cQV4+f8iHOXbsK2ghX6dfsAH3Vx19qmoZszlm7aj2NnrqFVs9p4lJyGg8ej0KpZbbFOcOh5rNx8EN+M6Yk6NSrh2s17mLlkJ0xN5Ojp2VTvOC9evYPmjWtqlLVo6orv1vyhtU1q+rN/q1aWr/63SvQmvFNzBjIzM5GZ+V/3t1qtLsFo3j55eXmYungn3BtWh5uLo9Z6Ls62WDl9AOq6VII67SlWbA2F17BFCN/+DSrZlUOcKglBwacRtWc2HCpaAwDGDuyA0PBrCNpzGjN8u+sV16PkNOTm5qGijaVGeUUbJf65naC1zYKN++DzYeH+p0alw934JGwPDodP79YY+cn/EBUdh4DVu1G2jLHWL+3Gdath/lefYMLcIGRlZSMnNw9tm7th2tgPxTorfzyISZ91RceW9QE86z24eScBv/x5ulDJwMPHqahgrfn3u3w5C6Q9yUBGZjZMFGU1juXl5eH/1v6B9+pWRc1q9npfjwrHCDIYGdDXb1SK+wbeqWQgICAAs2bNKukw3loT5+/AtZvx2Ld+/Cvrvd+gOt5/rlvy/YbV4f7RHAT+dhLfjO6KqzfuIzc3D816a84jyMzKgY3VswlYcaokePT9VjyWk5uH7JxcVG7tJ5aNH+KFCUO89L4PddpTfDxuDVyrOeCrkd56t6fSI08QUK9WZYwb+qwHsY5LJdy4rcKOveFav7Rv3ElAwOo/MHpAB7RoWgsPklKxaH0wZi/7FXMm9MWTp1mIu/8IMxb/ojGPIDc3D5bmJuJ+9xELcT/h8bMdQQAANO3+jXi8Sb1q+H7e8ELd17crd+Gf2yr8uPjzQrWnwuEwgXbvVDIwdepU+Pn992WjVqvh5KR9RrGUTJq/AweOX8af68ahkl05vdqWLWOMBq5OuHX3AQAg/UkmjI2NcGTLFBgba84xNTdVAAAcKljhWNBUsXzPkUjsORyJdXMGi2X5QxXlrS1gbGwkThbM9yBJDdvymvMWUtMz0OeL1bAwM8HWBSNQlkMEklbRxhI1qmiujKlexRYhJ6K0ttnw82G8V7cqhvZtCwBwrQ6YmsgxyG81vhjcCbJ/F4vPGtcH9WtX0WhrbPTf3/e13w5Ddk4uACDxUQoGT1yLX9f8l2g//7/9CuUs8TBZ8+/3o8dpsDAzKdAr8O3KXTh6+ho2L/oc9v/2vBGVtHcqGVAoFFAoFCUdxltFEARMXvAL9oZdxJ61X8K5UgW9z5Gbm4erN+6jYws3AEAD18rIzc3Dg8ep+OC9l09CLFPGGNWdKor7FctZwkRRVqMsn7xsGTSq7YSjZ6Ph3bYhgGfdpMfO/o3hH7UW66nTnqLPF6sgL1sG2xZ/VuCXKEnPe3WrIubfJDXf7bsP4fiKhPdpZhbKGGsmkcb/JgACBFQsp4RteSXi4h+ha/vGWs/z/DXK/JsUa/v31dDNGcfPXNcoO3X+bzR0+y/ZEAQBc1ftRujJywhcOOq1EyGpGLBrQCs+dOgdN/H/dmDHvrNYP2cwLMxMkPBQjYSHajzN+G+50qiZWzBr5e/i/vz1+3D49DXcvvsQF6/HYeSMzYhTJWFgj2fj8y7OdvioUzOM9v8Rew5H4s69h4i4chuLNx3AgROXCxXn5/3/hy27T+Gn4NOIjlHB77vtSH+aiQHdmgN4lgj0HrsK6U+zsGL6AKSmZYj3kpubZ8AnRO+yQb1a49K1O1j3Uyju3HuI4MMXsPPP0/ik239zSZZs/BNT5/8k7rdt7oZDJ6Lw855TiIt/hPNXYjBv9e+o7+oE2/JWAADfQZ7YsP0Itu46gdt3H+DvmHjsOnAWgTuPFirOj709cDf+ERauD8at2ET89McpHDh6CYN6/ZfszlmxC8Gh5zF/an+YmSrwIEmNB0lqZGRmF/LTIX3JiuBPaVWiPQNpaWm4ceOGuB8TE4PIyEjY2NigSpUqr2hJ+X749TgAoOuoZRrlq2Z8iv7/ftHeVSVpTJpJTn2CL+duQ+KjVFhbmqJhnSo4sNEPtav/t5pg1cxPsXDjfkxbtgvxickob22OpvWqwatVvULF2cuzCR4mp2He93uR+CgV9WtVws7lvuIwwaXoOJy7fBsA0PhDzXkhF3+fhSqO5Qt1XXq31Xd1wrKZPlj6wz6s2XoIle1tMGV0D43/0T9IUiM+MVnc/9CzGZ48ycS2P05hwbpgWJqbwr1RDfgN/2/+SZ/O7jBRyLHplzAs3BAMUxM5alV1wMB/lyPqq7KDDVZ/Owz/t/YPbN19AvYVrDDLr4+4rBAAtgeHAwAGT1yr0fbbiX01Hk5EVBJkgvDvzJgSEBYWhnbt2hUo9/HxQWBg4Gvbq9VqWFlZIeFRCpRK7Wvmid5l9x8/LekQiIpNaqoajWrYIyWl+H6P539XhEbGwsKy8NdIS1WjfaMqOsd67NgxLFiwABEREYiPj8euXbvQs2dP8bggCJg5cybWr1+P5ORktGjRAmvWrEHNmv8tU01KSsLYsWOxZ88eGBkZoXfv3li2bBksLCzEOpcuXYKvry/Onj2LihUrYuzYsZg8ebJe91aiwwRt27aFIAgFNl0SASIiIn3IimDTR/6D9VatWvXS4/Pnz8fy5cuxdu1a/PXXXzA3N4eXlxcyMjLEOgMGDMCVK1cQEhKC4OBgHDt2DCNHjhSPq9VqeHp6wtnZGREREViwYAH8/f2xbt06vWJ9pyYQEhERvSte9WA9QRCwdOlSTJs2DT169AAAbNmyBXZ2dti9ezf69euHa9euYf/+/Th79iyaNn22lHbFihXo0qULFi5cCEdHRwQFBSErKws//PAD5HI56tati8jISCxevFgjaXgdTiAkIiJpKKKuAbVarbE9/zA8XcXExEClUqFDhw5imZWVFdzd3REe/mx+SXh4OKytrcVEAAA6dOgAIyMj/PXXX2Kd1q1bQy6Xi3W8vLwQHR2Nx48f6xwPkwEiIpKEolpN4OTkBCsrK3ELCAjQOxaVSgUAsLPTfI6GnZ2deEylUsHWVvPx72XKlIGNjY1GnZed4/lr6ILDBEREJAmGvnkwv21cXJzGBMLS8Pwb9gwQERHpQalUamyFSQbs7Z+9kyIhQfP9LAkJCeIxe3t7JCYmahzPyclBUlKSRp2XneP5a+iCyQAREUnCm15N8CrVqlWDvb09QkNDxTK1Wo2//voLHh7PXvXu4eGB5ORkREREiHUOHz6MvLw8uLu7i3WOHTuG7Oz/Hl4VEhICV1dXlCun+6PpmQwQEZE0vOFsIC0tDZGRkYiMjATw34P1YmNjIZPJMG7cOHz77bf4448/EBUVhUGDBsHR0VF8FkGdOnXQqVMnjBgxAmfOnMHJkycxZswY9OvXD46Oz95M279/f8jlcgwbNgxXrlzB9u3bsWzZMo33+OiCcwaIiIiKwblz5zQerJf/BZ3/YL3JkycjPT0dI0eORHJyMlq2bIn9+/fDxOS/t2cGBQVhzJgxaN++vfjQoeXLl4vHrayscPDgQfj6+qJJkyaoUKECZsyYodeyQqCEn0BoKD6BkKSATyCk0uxNPoHwWNRdg59A2Lp+5WKNtaSwZ4CIiCShqFYTlEacM0BERCRx7BkgIiJJMHRFQCnuGGAyQEREEsFsQCsOExAREUkcewaIiEgSnn+/QGHbl1ZMBoiISBK4mkA7JgNERCQJnDKgHecMEBERSRx7BoiISBrYNaAVkwEiIpIETiDUjsMEREREEseeASIikgSuJtCOyQAREUkCpwxox2ECIiIiiWPPABERSQO7BrRiMkBERJLA1QTacZiAiIhI4tgzQEREksDVBNoxGSAiIknglAHtmAwQEZE0MBvQinMGiIiIJI49A0REJAlcTaAdkwEiIpIGAycQluJcgMMEREREUseeASIikgTOH9SOyQAREUkDswGtOExAREQkcewZICIiSeBqAu2YDBARkSTwccTacZiAiIhI4tgzQEREksD5g9oxGSAiImlgNqAVkwEiIpIETiDUjnMGiIiIJI49A0REJAkyGLiaoMgiefswGSAiIknglAHtOExAREQkcewZICIiSeBDh7RjMkBERBLBgQJtOExAREQkcewZICIiSeAwgXZMBoiISBI4SKAdhwmIiIgkjskAERFJQv4wgSGbPvz9/SGTyTS22rVri8czMjLg6+uL8uXLw8LCAr1790ZCQoLGOWJjY+Ht7Q0zMzPY2tpi0qRJyMnJKYqPQwOHCYiISBJK4t0EdevWxaFDh8T9MmX++9odP3489u7di19++QVWVlYYM2YMevXqhZMnTwIAcnNz4e3tDXt7e5w6dQrx8fEYNGgQypYti3nz5hX6Pl6GyQAREUlDCUwaKFOmDOzt7QuUp6SkYOPGjdi2bRv+97//AQA2bdqEOnXq4PTp02jevDkOHjyIq1ev4tChQ7Czs0OjRo0wZ84cTJkyBf7+/pDL5QbcjCYOExARERWTf/75B46OjqhevToGDBiA2NhYAEBERASys7PRoUMHsW7t2rVRpUoVhIeHAwDCw8NRv3592NnZiXW8vLygVqtx5cqVIo2TPQNERCQJRdUxoFarNcoVCgUUCkWB+u7u7ggMDISrqyvi4+Mxa9YstGrVCpcvX4ZKpYJcLoe1tbVGGzs7O6hUKgCASqXSSATyj+cfK0pMBoiISBKK6jkDTk5OGuUzZ86Ev79/gfqdO3cWf27QoAHc3d3h7OyMHTt2wNTUtPCBFAMmA0RERHqIi4uDUqkU91/WK/Ay1tbWqFWrFm7cuIGOHTsiKysLycnJGr0DCQkJ4hwDe3t7nDlzRuMc+asNXjYPwRCcM0BERJIgK4I/AKBUKjU2XZOBtLQ03Lx5Ew4ODmjSpAnKli2L0NBQ8Xh0dDRiY2Ph4eEBAPDw8EBUVBQSExPFOiEhIVAqlXBzcyvCT4Y9A0REJBVveDXBxIkT0a1bNzg7O+P+/fuYOXMmjI2N8cknn8DKygrDhg2Dn58fbGxsoFQqMXbsWHh4eKB58+YAAE9PT7i5uWHgwIGYP38+VCoVpk2bBl9fX50TEF0xGSAiIioGd+/exSeffIJHjx6hYsWKaNmyJU6fPo2KFSsCAJYsWQIjIyP07t0bmZmZ8PLywurVq8X2xsbGCA4OxujRo+Hh4QFzc3P4+Phg9uzZRR6rTBAEocjP+oao1WpYWVkh4VGKxvgNUWly//HTkg6BqNikpqrRqIY9UlKK7/d4/nfFrXuPYGnANVLValSvVL5YYy0p7BkgIiJJ4FsLteMEQiIiIoljzwAREUmEYe8mKM0vMWYyQEREksBhAu04TEBERCRxTAaIiIgkjsMEREQkCRwm0I7JABERSYLMwAmEhk0+fLtxmICIiEji2DNARESSwGEC7ZgMEBGRJLzh9xS9UzhMQEREJHHsGSAiImlg14BWTAaIiEgSuJpAOw4TEBERSRx7BoiISBK4mkA7JgNERCQJnDKgHZMBIiKSBmYDWnHOABERkcSxZ4CIiCSBqwm0YzJARESSwAmE2r3TyYAgCACAVLW6hCMhKj6pqU9LOgSiYpOWmgrgv9/nxUlt4HeFoe3fZu90MpD6718il2pOJRwJEREZIjU1FVZWVsVybrlcDnt7e9Qsgu8Ke3t7yOXyIojq7SIT3kQ6Vkzy8vJw//59WFpaQlaa+2/eImq1Gk5OToiLi4NSqSzpcIiKFP9+v3mCICA1NRWOjo4wMiq+Oe0ZGRnIysoy+DxyuRwmJiZFENHb5Z3uGTAyMkLlypVLOgxJUiqV/GVJpRb/fr9ZxdUj8DwTE5NS+SVeVLi0kIiISOKYDBAREUkckwHSi0KhwMyZM6FQKEo6FKIix7/fJFXv9ARCIiIiMhx7BoiIiCSOyQAREZHEMRkgIiKSOCYDREREEsdkgHS2atUqVK1aFSYmJnB3d8eZM2dKOiSiInHs2DF069YNjo6OkMlk2L17d0mHRPRGMRkgnWzfvh1+fn6YOXMmzp8/j4YNG8LLywuJiYklHRqRwdLT09GwYUOsWrWqpEMhKhFcWkg6cXd3R7NmzbBy5UoAz94L4eTkhLFjx+Krr74q4eiIio5MJsOuXbvQs2fPkg6F6I1hzwC9VlZWFiIiItChQwexzMjICB06dEB4eHgJRkZEREWByQC91sOHD5Gbmws7OzuNcjs7O6hUqhKKioiIigqTASIiIoljMkCvVaFCBRgbGyMhIUGjPCEhAfb29iUUFRERFRUmA/RacrkcTZo0QWhoqFiWl5eH0NBQeHh4lGBkRERUFMqUdAD0bvDz84OPjw+aNm2K999/H0uXLkV6ejqGDBlS0qERGSwtLQ03btwQ92NiYhAZGQkbGxtUqVKlBCMjejO4tJB0tnLlSixYsAAqlQqNGjXC8uXL4e7uXtJhERksLCwM7dq1K1Du4+ODwMDANx8Q0RvGZICIiEjiOGeAiIhI4pgMEBERSRyTASIiIoljMkBERCRxTAaIiIgkjskAERGRxDEZICIikjgmA0QGGjx4MHr27Cnut23bFuPGjXvjcYSFhUEmkyE5OVlrHZlMht27d+t8Tn9/fzRq1MiguG7fvg2ZTIbIyEiDzkNExYfJAJVKgwcPhkwmg0wmg1wuh4uLC2bPno2cnJxiv/Zvv/2GOXPm6FRXly9wIqLixncTUKnVqVMnbNq0CZmZmfjzzz/h6+uLsmXLYurUqQXqZmVlQS6XF8l1bWxsiuQ8RERvCnsGqNRSKBSwt7eHs7MzRo8ejQ4dOuCPP/4A8F/X/ty5c+Ho6AhXV1cAQFxcHPr27Qtra2vY2NigR48euH37tnjO3Nxc+Pn5wdraGuXLl8fkyZPx4hO9XxwmyMzMxJQpU+Dk5ASFQgEXFxds3LgRt2/fFp+HX65cOchkMgwePBjAs7dCBgQEoFq1ajA1NUXDhg2xc+dOjev8+eefqFWrFkxNTdGuXTuNOHU1ZcoU1KpVC2ZmZqhevTqmT5+O7OzsAvW+//57ODk5wczMDH379kVKSorG8Q0bNqBOnTowMTFB7dq1sXr1ar1jIaKSw2SAJMPU1BRZWVnifmhoKKKjoxESEoLg4GBkZ2fDy8sLlpaWOH78OE6ePAkLCwt06tRJbLdo0SIEBgbihx9+wIkTJ5CUlIRdu3a98rqDBg3CTz/9hOXLl+PatWv4/vvvYWFhAScnJ/z6668AgOjoaMTHx2PZsmUAgICAAGzZsgVr167FlStXMH78eHz66ac4evQogGdJS69evdCtWzdERkZi+PDh+Oqrr/T+TCwtLREYGIirV69i2bJlWL9+PZYsWaJR58aNG9ixYwf27NmD/fv348KFC/j888/F40FBQZgxYwbmzp2La9euYd68eZg+fTo2b96sdzxEVEIEolLIx8dH6NGjhyAIgpCXlyeEhIQICoVCmDhxonjczs5OyMzMFNv8+OOPgqurq5CXlyeWZWZmCqampsKBAwcEQRAEBwcHYf78+eLx7OxsoXLlyuK1BEEQ2rRpI3z55ZeCIAhCdHS0AEAICQl5aZxHjhwRAAiPHz8WyzIyMgQzMzPh1KlTGnWHDRsmfPLJJ4IgCMLUqVMFNzc3jeNTpkwpcK4XARB27dql9fiCBQuEJk2aiPszZ84UjI2Nhbt374pl+/btE4yMjIT4+HhBEAShRo0awrZt2zTOM2fOHMHDw0MQBEGIiYkRAAgXLlzQel0iKlmcM0ClVnBwMCwsLJCdnY28vDz0798f/v7+4vH69etrzBO4ePEibty4AUtLS43zZGRk4ObNm0hJSUF8fLzGa5vLlCmDpk2bFhgqyBcZGQljY2O0adNG57hv3LiBJ0+eoGPHjhrlWVlZeO+99wAA165dK/D6aA8PD52vkW/79u1Yvnw5bt68ibS0NOTk5ECpVGrUqVKlCipVqqRxnby8PERHR8PS0hI3b97EsGHDMGLECLFOTk4OrKys9I6HiEoGkwEqtdq1a4c1a9ZALpfD0dERZcpo/nU3NzfX2E9LS0OTJk0QFBRU4FwVK1YsVAympqZ6t0lLSwMA7N27V+NLGHg2D6KohIeHY8CAAZg1axa8vLxgZWWFn3/+GYsWLdI71vXr1xdIToyNjYssViIqXkwGqNQyNzeHi4uLzvUbN26M7du3w9bWtsD/jvM5ODjgr7/+QuvWrQE8+x9wREQEGjdu/NL69evXR15eHo4ePYoOHToUOJ7fM5GbmyuWubm5QaFQIDY2VmuPQp06dcTJkPlOnz79+pt8zqlTp+Ds7IxvvvlGLLtz506BerGxsbh//z4cHR3F6xgZGcHV1RV2dnZwdHTErVu3MGDAAL2uT0RvD04gJPrXgAEDUKFCBfTo0QPHjx9HTEwMwsLC8MUXX+Du3bsAgC+//BLfffcddu/ejevXr+Pzzz9/5TMCqlatCh8fHwwdOhS7d+8Wz7ljxw4AgLOzM2QyGYKDg/HgwQOkpaXB0tISEydOxPjx47F582bcvHkT58+fx4oVK8RJeaNGjcI///yDSZMmITo6Gtu2bUNgYKBe91uzZk3Exsbi559/xs2bN7F8+fKXToY0MTGBj48PLl68iOPHj+OLL75A3759YW9vDwCYNWsWAgICsHz5cvz999+IiorCpk2bsHjxYr3iIaKSw2SA6F9mZmY4duwYqlSpgl69eqFOnToYNmwYMjIyxJ6CCRMmYODAgfDx8YGHhwcsLS3x4YcfvvK8a9asQZ8+ffD555+jdu3aGDFiBNLT0wEAlSpVwqxZs/DVV1/Bzs4OY8aMAQDMmTMH06dPR0BAAOrUqYNOnTph7969qFatGoBn4/i//vordu/ejYYNG2Lt2rWYN2+eXvfbvXt3jB8/HmPGjEGjRo1w6tQpTJ8+vUA9FxcX9OrVC126dIGnpycaNGigsXRw+PDh2LBhAzZt2oT69eujTZs2CAwMFGMlorefTNA284mIiIgkgT0DREREEsdkgIiISOKYDBAREUkckwEiIiKJYzJAREQkcUwGiIiIJI7JABERkcQxGSAiIpI4JgNEREQSx2SAiIhI4pgMEBERSRyTASIiIon7f0XNc+8a2LoEAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "from sklearn.metrics import ConfusionMatrixDisplay\n", + "\n", + "disp = ConfusionMatrixDisplay(confusion_matrix=confusion_matrix_abs)\n", + "disp.plot(cmap=plt.cm.Blues)\n", + "plt.title('Confusion Matrix')\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "a9360ea0", + "metadata": {}, + "source": [ + "The Confusion Matrix created has four different quadrants:\n", + "\n", + "- True Negative (Top-Left Quadrant)\n", + "- False Positive (Top-Right Quadrant)\n", + "- False Negative (Bottom-Left Quadrant)\n", + "- True Positive (Bottom-Right Quadrant)\n", + "\n", + "True means that the values were accurately predicted, False means that there was an error or wrong prediction." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "90b88789", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[3580., 488.],\n", + " [ 248., 684.]])" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "confusion_matrix_abs" + ] + }, + { + "cell_type": "markdown", + "id": "c42dc06f", + "metadata": {}, + "source": [ + "# Accuracy\n", + "\n", + "Accuracy measures how often the model is correct.\n", + "\n", + "(True Positive + True Negative) / Total Predictions\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d69262cd", + "metadata": {}, + "outputs": [], + "source": [ + "TN, FP, FN, TP = confusion_matrix_abs[0,0], confusion_matrix_abs[0,1], confusion_matrix_abs[1,0], confusion_matrix_abs[1,1]" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "263ca554", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "np.float64(0.8528)" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# from systemds.operator.algorithm import getAccuracy\n", + "# accuracy = getAccuracy(Yt, y_pred).compute()\n", + "\n", + "\n", + "accuracy = (TP + TN) / test_count\n", + "accuracy" + ] + }, + { + "cell_type": "markdown", + "id": "54ea1ecf", + "metadata": {}, + "source": [ + "# Precision\n", + "Of the positives predicted, what percentage is truly positive?\n", + "\n", + "True Positive / (True Positive + False Positive)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "d7daf015", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "np.float64(0.5836177474402731)" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "precision = TP / (TP + FP)\n", + "precision" + ] + }, + { + "cell_type": "markdown", + "id": "9b24ea83", + "metadata": {}, + "source": [ + "# Sensitivity (Recall)\n", + "Of all the positive cases, what percentage are predicted positive?\n", + "\n", + "Sensitivity (sometimes called Recall) measures how good the model is at predicting positives.\n", + "\n", + "This means it looks at true positives and false negatives (which are positives that have been incorrectly predicted as negative).\n", + "\n", + "True Positive / (True Positive + False Negative)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "57e04ed3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "np.float64(0.7339055793991416)" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sensitivity = TP /(TP + FN)\n", + "sensitivity" + ] + }, + { + "cell_type": "markdown", + "id": "52abc77a", + "metadata": {}, + "source": [ + "# Specificity\n", + "How well the model is at prediciting negative results?\n", + "\n", + "Specificity is similar to sensitivity, but looks at it from the persepctive of negative results.\n", + "\n", + "True Negative / (True Negative + False Positive)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "3dcfa6b8", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "np.float64(0.880039331366765)" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "specificity = TN / (TN + FP)\n", + "specificity" + ] + }, + { + "cell_type": "markdown", + "id": "127fe71d", + "metadata": {}, + "source": [ + "# F-score\n", + "F-score is the \"harmonic mean\" of precision and sensitivity.\n", + "\n", + "It considers both false positive and false negative cases and is good for imbalanced datasets.\n", + "\n", + "2 * ((Precision * Sensitivity) / (Precision + Sensitivity))" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "bceed706", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "np.float64(0.6501901140684411)" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "f_score = 2 * ((precision * sensitivity) / (precision + sensitivity))\n", + "f_score" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "python_venv (3.12.3)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/main/python/notebooks/quick_start.ipynb b/src/main/python/notebooks/quick_start.ipynb new file mode 100644 index 00000000000..5801f0882a0 --- /dev/null +++ b/src/main/python/notebooks/quick_start.ipynb @@ -0,0 +1,234 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0213111f", + "metadata": {}, + "source": [ + "# Matrix Operations" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "f3d1b80e", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Using incubator modules: jdk.incubator.vector\n", + "26/05/06 14:56:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", + "INFO:simple_example:[[13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", + " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", + " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", + " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", + " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]]\n" + ] + } + ], + "source": [ + "import logging\n", + "\n", + "from systemds.context import SystemDSContext\n", + "\n", + "logger = logging.getLogger('simple_example')\n", + "logger.setLevel(logging.INFO)\n", + "\n", + "# Create a context and if necessary (no SystemDS py4j instance running)\n", + "# it starts a subprocess which does the execution in SystemDS\n", + "with SystemDSContext() as sds:\n", + " # Full generates a matrix completely filled with one number.\n", + " # Generate a 5x10 matrix filled with 4.2\n", + " m = sds.full((5, 10), 4.20)\n", + " # multiply with scalar. Nothing is executed yet!\n", + " m_res = m * 3.1\n", + " # Do the calculation in SystemDS by calling compute().\n", + " # The returned value is an numpy array that can be directly printed.\n", + " logger.info(m_res.compute())\n", + " # context will automatically be closed and process stopped" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "cadb888c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Using incubator modules: jdk.incubator.vector\n", + "26/05/06 15:02:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", + "26/05/06 15:02:21 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", + "26/05/06 15:02:21 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", + "INFO:simple_example:[[ 50.01 125.05 160.05 32.01 204.04]\n", + " [ 37.01 81.03 268.04 110.02 6.03]\n", + " [ 39.01 244.04 430.05 140.04 368.04]\n", + " [246.03 154.02 267.03 104.04 72.02]\n", + " [ 14.02 186.02 475.05 168.02 315.05]]\n" + ] + } + ], + "source": [ + "import logging\n", + "\n", + "import numpy as np\n", + "from systemds.context import SystemDSContext\n", + "\n", + "logger = logging.getLogger('simple_example')\n", + "logger.setLevel(logging.INFO)\n", + "\n", + "# create a random array\n", + "m1 = np.array(np.random.randint(100, size=5 * 5) + 1.01, dtype=np.double)\n", + "m1 = m1.reshape(5, 5)\n", + "# create another random array\n", + "m2 = np.array(np.random.randint(5, size=5 * 5) + 1, dtype=np.double)\n", + "m2 = m2.reshape(5, 5)\n", + "\n", + "# Create a context\n", + "with SystemDSContext() as sds:\n", + " # element-wise matrix multiplication, note that nothing is executed yet!\n", + " m_res = sds.from_numpy(m1) * sds.from_numpy(m2)\n", + " # lets do the actual computation in SystemDS! The result is an numpy array\n", + " m_res_np = m_res.compute()\n", + " logger.info(m_res_np)" + ] + }, + { + "cell_type": "markdown", + "id": "87cc9e28", + "metadata": {}, + "source": [ + "# Algorithms as built-in functions" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "bf202fb1", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Using incubator modules: jdk.incubator.vector\n", + "26/05/06 15:05:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", + "26/05/06 15:05:20 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", + "26/05/06 15:05:20 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", + "INFO:simple_example:[[ 0.02033445]\n", + " [-0.00324092]\n", + " [ 0.0014692 ]\n", + " [ 0.02649209]\n", + " [-0.00616902]\n", + " [-0.0095087 ]\n", + " [ 0.01039221]\n", + " [-0.0011352 ]\n", + " [-0.01686351]\n", + " [-0.03839821]]\n" + ] + } + ], + "source": [ + "import logging\n", + "\n", + "import numpy as np\n", + "from systemds.context import SystemDSContext\n", + "from systemds.operator.algorithm import l2svm\n", + "\n", + "logger = logging.getLogger('simple_example')\n", + "logger.setLevel(logging.INFO)\n", + "\n", + "# Set a seed\n", + "np.random.seed(0)\n", + "# Generate random features and labels in numpy\n", + "# This can easily be exchanged with a data set.\n", + "features = np.array(np.random.randint(\n", + " 100, size=10 * 10) + 1.01, dtype=np.double)\n", + "features = features.reshape(10, 10)\n", + "labels = np.zeros((10, 1))\n", + "\n", + "# l2svm labels can only be 0 or 1\n", + "for i in range(10):\n", + " if np.random.random() > 0.5:\n", + " labels[i][0] = 1\n", + "\n", + "# compute our model\n", + "with SystemDSContext() as sds:\n", + " model = l2svm(sds.from_numpy(features),\n", + " sds.from_numpy(labels), verbose=False).compute()\n", + " logger.info(model)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "266e1c4b", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Using incubator modules: jdk.incubator.vector\n", + "26/05/06 15:07:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", + "INFO:simple_example:[[-0.02674835]\n", + " [-0.01684308]\n", + " [ 0.02429331]\n", + " [ 0.03332674]\n", + " [-0.00411165]\n", + " [ 0.00514915]\n", + " [ 0.01035419]\n", + " [ 0.00611115]\n", + " [-0.02550657]\n", + " [-0.0117999 ]]\n" + ] + } + ], + "source": [ + "import logging\n", + "\n", + "from systemds.context import SystemDSContext\n", + "from systemds.operator.algorithm import l2svm\n", + "\n", + "logger = logging.getLogger('simple_example')\n", + "logger.setLevel(logging.INFO)\n", + "\n", + "with SystemDSContext() as sds:\n", + " # Generate 10 by 10 matrix with values in range 0 to 100.\n", + " features2 = sds.rand(10, 10, 0, 100)\n", + " # Add value to all cells in features\n", + " features2 += 1.1\n", + " # Generate labels of all ones and zeros\n", + " labels2 = sds.rand(10, 1, 1, 1, sparsity=0.5)\n", + "\n", + " model2 = l2svm(features2, labels2, verbose=False).compute()\n", + " logger.info(model2)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "python_venv (3.12.3)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/main/python/requirements.txt b/src/main/python/requirements.txt new file mode 100644 index 00000000000..3c7af412a4c --- /dev/null +++ b/src/main/python/requirements.txt @@ -0,0 +1,9 @@ +numpy +scipy +py4j +wheel +requests +setuptools + +scikit-learn +matplotlib From 021bb7de77d8a4531e8697271c9f520fd6a42db1 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Thu, 21 May 2026 14:33:53 +0200 Subject: [PATCH 02/26] Add CLAUDE.md --- CLAUDE.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000000..d3fcc2be49c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,113 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project + +Apache SystemDS — an end-to-end ML system that compiles DML scripts (R-like syntax) into hybrid execution plans across local (CP), Apache Spark, GPU, and federated backends. + +## Build Commands + +```bash +# Full build (skips tests) +mvn clean package -DskipTests + +# Build and run default tests +mvn clean package + +# Run a single Java test class +mvn test -Dtest=FullMatrixMultiplicationTest + +# Run a test with a specific method +mvn test -Dtest=FullMatrixMultiplicationTest#testMethod + +# Run checkstyle (disabled by default) +mvn checkstyle:check -Dcheckstyle.skip=false + +# Python tests (from src/main/python/) +cd src/main/python && source python_venv/bin/activate && pip install -e . && python -m pytest tests/ + +# Run a DML script directly (after building) +./bin/systemds hello.dml +``` + +The surefire default test is `org.apache.sysds.test.usertest.**`. Override with `-Dtest=` to target specific classes. Test JVM is configured with `-Xmx3000m`. + +## Code Style + +Apply the Eclipse formatter profile at [dev/CodeStyle_eclipse.xml](dev/CodeStyle_eclipse.xml) before committing. Checkstyle rules are at [dev/checkstyle/checkstyle.xml](dev/checkstyle/checkstyle.xml). + +## Commit Tags + +All commits must be prefixed: `[SYSTEMDS-#]` for Jira issues, `[MINOR]` for small changes, `[DOC]` for docs, `[HOTFIX]` for release patches. The project uses linear history — rebase, never merge commits. + +## Compilation Pipeline + +The full compilation sequence is in [DMLScript.java:460-510](src/main/java/org/apache/sysds/api/DMLScript.java#L460): + +1. **Parse** — DML source → `DMLProgram` AST (ANTLR-based, `parser/dml/`) +2. **Live Variable Analysis + Validate** — `DMLTranslator.liveVariableAnalysis()` / `validateParseTree()` +3. **Construct HOPs** — AST → HOP DAG (`DMLTranslator.constructHops()`) +4. **Rewrite HOP DAGs** — algebraic simplifications, IPA, memory estimates, CSE (`hops/rewrite/`) +5. **Construct LOPs** — HOP DAG → LOP DAG; exec type (CP/Spark/GPU/Fed) selected here +6. **Rewrite LOP DAGs** — inject prefetch, broadcast, OOC tee operators +7. **Generate runtime program** — LOPs emit instruction strings; codegen (Spoof) fuses operators +8. **Execute** — `ProgramBlock` hierarchy interprets instructions via `ExecutionContext` + +## Architecture Patterns + +### Two-Level IR: HOPs → LOPs +`Hop` ([hops/Hop.java](src/main/java/org/apache/sysds/hops/Hop.java)) is the algebraic IR. Each concrete subclass (`BinaryOp`, `AggBinaryOp`, `UnaryOp`, etc.) implements abstract template methods: +- `constructLops()` — lowers this operator to backend-specific LOPs +- `optFindExecType()` — selects CP / Spark / GPU / Fed based on cost model +- `inferOutputCharacteristics()` — size/sparsity estimation +- `computeOutputMemEstimate()` / `computeIntermediateMemEstimate()` — memory cost + +`Lop` ([lops/Lop.java](src/main/java/org/apache/sysds/lops/Lop.java)) is the backend-specific IR. LOPs emit instruction strings consumed by the runtime. + +### DAGs, not Trees +Both HOPs and LOPs are DAGs with explicit `_input`/`_parent` lists and a `_visited` boolean. All rewrite/lowering passes do DFS traversal with the visited flag. This enables CSE — shared subcomputations appear once in the DAG. + +### Chain of Responsibility for Rewrites +`ProgramRewriter` ([hops/rewrite/ProgramRewriter.java](src/main/java/org/apache/sysds/hops/rewrite/ProgramRewriter.java)) holds an ordered list of `HopRewriteRule` subclasses and fires each over the full DAG. To add an optimization: subclass `HopRewriteRule`, implement `rewriteHopDAGs()` / `rewriteHopDAG()`, and register it. Existing rules include loop vectorization, loop-invariant hoisting, Spark checkpoint injection, OOC tee injection, and compressed reblock. + +### Parallel AST ↔ Runtime Hierarchies +The parser (`Statement`/`StatementBlock`) and runtime (`ProgramBlock`) mirror each other exactly: `ForStatement` → `ForProgramBlock`, `ParForStatement` → `ParForProgramBlock`, etc. The AST is structural only; all execution behavior lives in the runtime mirror. + +### String-Serialized Instruction Boundary +LOPs emit plain strings as instructions. At runtime, `InstructionParser.parseSingleInstruction()` reads the exec-type prefix (`CP·`, `SPARK·`, `GPU·`, `FED·`, `OOC·`) and dispatches to the appropriate backend parser. This decouples the compiler from all backend implementations. + +### Caching Layer +`CacheableData` ([runtime/controlprogram/caching/CacheableData.java](src/main/java/org/apache/sysds/runtime/controlprogram/caching/CacheableData.java)) is a generic abstract envelope for large data objects. Subclasses (`MatrixObject`, `FrameObject`, `TensorObject`) inherit full eviction/spill-to-disk lifecycle. + +## Key Source Locations + +| Area | Path | +|---|---| +| Compiler entry point | `src/main/java/org/apache/sysds/api/DMLScript.java` | +| DML → HOP translator | `src/main/java/org/apache/sysds/parser/DMLTranslator.java` | +| HOP base + optimizer | `src/main/java/org/apache/sysds/hops/` | +| HOP rewrites | `src/main/java/org/apache/sysds/hops/rewrite/` | +| LOP base | `src/main/java/org/apache/sysds/lops/` | +| Runtime control flow | `src/main/java/org/apache/sysds/runtime/controlprogram/` | +| Runtime instructions | `src/main/java/org/apache/sysds/runtime/instructions/` | +| Compressed Linear Algebra | `src/main/java/org/apache/sysds/runtime/compress/` | +| Operator fusion (Spoof) | `src/main/java/org/apache/sysds/runtime/codegen/` | +| Out-of-core execution | `src/main/java/org/apache/sysds/runtime/ooc/` | +| Built-in algorithms (DML) | `scripts/builtin/` | +| Staging area (new algos) | `scripts/staging/` | +| Python API | `src/main/python/systemds/` | + +## Testing + +Java tests extend `AutomatedTestBase`. The default exec mode is `ExecMode.HYBRID`; tests call `setExecMode()` / `resetExecMode()` to test specific backends. + +- **`src/test/java/.../test/functions/`** — end-to-end integration tests that run DML scripts through the full pipeline +- **`src/test/java/.../test/component/`** — unit tests targeting specific subsystems (compress, matrix, codegen, ooc, parfor, etc.) +- **`src/test/java/.../test/applications/`** — full ML algorithm correctness tests + +Each function test has a paired DML script under `src/test/scripts/functions/`. + +## Adding a New Built-in Algorithm + +New algorithms live in `scripts/staging/` until they have sufficient test coverage, then move to `scripts/builtin/`. The Python API wrappers in `src/main/python/systemds/operator/algorithm/builtin/` are auto-generated — see `src/main/python/generator/` for the generation scripts. From 4e9bdd10e6949fb3321211fbbe02980d642ccbe8 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Fri, 26 Jun 2026 20:20:31 +0200 Subject: [PATCH 03/26] Initial DPBuiltinCPInstruction and RDPAccountant --- .../cp/DPBuiltinCPInstruction.java | 349 +++++++++++++++++ .../runtime/privacy/dp/RDPAccountant.java | 283 ++++++++++++++ .../cp/DPBuiltinCPInstructionTest.java | 350 ++++++++++++++++++ 3 files changed, 982 insertions(+) create mode 100755 src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java create mode 100755 src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java create mode 100755 src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java new file mode 100755 index 00000000000..f75ccb13cdb --- /dev/null +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -0,0 +1,349 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.sysds.runtime.instructions.cp; + +import org.apache.sysds.runtime.DMLRuntimeException; +import org.apache.sysds.runtime.controlprogram.context.ExecutionContext; +import org.apache.sysds.runtime.functionobjects.Plus; +import org.apache.sysds.runtime.instructions.InstructionUtils; +import org.apache.sysds.runtime.matrix.data.MatrixBlock; +import org.apache.sysds.runtime.matrix.operators.BinaryOperator; +import org.apache.sysds.runtime.privacy.dp.RDPAccountant; + +import java.util.LinkedHashMap; +import java.util.concurrent.ThreadLocalRandom; + +/** + * CP instruction for differential-privacy release of an already-computed + * aggregate. + * + *

DML syntax (post-aggregate form, Option A): + * 

+ *   result = dp_laplace(aggregate, sensitivity=1.0, epsilon=0.5)
+ *   result = dp_gaussian(aggregate, sensitivity=1.0, epsilon=0.5, delta=1e-5)
+ *
+ * + *

The instruction receives a materialised matrix (the aggregate result), + * injects calibrated noise element-wise, records the release with the + * session-scoped {@link RDPAccountant}, and returns the noisy matrix. + * + *

Noise is generated in Java and added via a {@code MatrixBlock} binary + * operation so that the output allocation path is identical to every other + * CP instruction (no special memory-management required). + * + *

The {@link #sensitivityOf} method is deliberately separated from the + * noise-scale computation. In Phase 1 it returns the caller-supplied + * constant. In the future HOP-level rewrite pass (Phase 2) the body of this + * single method is replaced with a static analysis that reads the + * sensitivity bound computed by the compiler; every other line in this class + * stays unchanged. + * + *

Registration required in: + *

    + *
  • {@code org.apache.sysds.common.Builtins} – add + * {@code DP_LAPLACE("dp_laplace", false)} and + * {@code DP_GAUSSIAN("dp_gaussian", false)}
    • + *
  • {@code org.apache.sysds.runtime.instructions.CPInstructionParser} – + * add opcode-to-type mappings and a parse branch that returns a + * {@code DPBuiltinCPInstruction}
    • + *
  • {@code org.apache.sysds.runtime.controlprogram.context.ExecutionContext} + * – add {@code getRDPAccountant()} returning a session-scoped + * {@link RDPAccountant} (lazy-initialised field)
    • + *
+ */ +public class DPBuiltinCPInstruction extends ComputationCPInstruction { + + // ----------------------------------------------------------------------- + // Constants + // ----------------------------------------------------------------------- + + /** Opcode registered in Builtins and CPInstructionParser. */ + public static final String OPCODE_LAPLACE = "dp_laplace"; + public static final String OPCODE_GAUSSIAN = "dp_gaussian"; + + // ----------------------------------------------------------------------- + // Fields + // ----------------------------------------------------------------------- + + /** + * Named parameters extracted from the serialised instruction string. + * Keys: "target", "sensitivity", "epsilon", "delta" (Gaussian only). + * + * Using the same LinkedHashMap convention as + * ParameterizedBuiltinCPInstruction so that CPInstructionParser can + * call the shared constructParameterMap() helper unchanged. + */ + private final LinkedHashMap _params; + + // ----------------------------------------------------------------------- + // Constructor (private – use parseInstruction) + // ----------------------------------------------------------------------- + + private DPBuiltinCPInstruction( + CPOperand input, + CPOperand output, + String opcode, + String istr, + LinkedHashMap params) { + // input1 = the aggregate matrix; input2/3 unused at this level + // (scalars come from _params, not CPOperand fields, so that they + // can be either literals or DML variable names transparently). + super(null, input, null, null, output, opcode, istr); + _params = params; + } + + // ----------------------------------------------------------------------- + // Static factory / parser + // ----------------------------------------------------------------------- + + /** + * Reconstructs a {@code DPBuiltinCPInstruction} from its serialised + * instruction string produced by the LOP layer. + * + *

Expected format (INSTRUCTION_DELIM = '\u00b0'): + * 

+     *   dp_gaussian°target=mVar1·MATRIX·FP64°sensitivity=1.0·SCALAR·FP64·true
+     *              °epsilon=0.5·SCALAR·FP64·true°delta=1e-5·SCALAR·FP64·true
+     *              °_mVar2·MATRIX·FP64
+     *
+ * + * The first token is always the opcode; the last token is always the + * output operand; the tokens in between are key=value pairs. This matches + * the convention used by ParameterizedBuiltinCPInstruction exactly. + */ + public static DPBuiltinCPInstruction parseInstruction(String str) { + String[] parts = InstructionUtils.getInstructionPartsWithValueType(str); + InstructionUtils.checkNumFields(parts, 4, 5); // laplace=4, gaussian=5 + String opcode = parts[0]; + + // Output operand is always the last token. + CPOperand output = new CPOperand(parts[parts.length - 1]); + + // The "target" parameter holds the variable name of the input matrix. + // ParameterizedBuiltinCPInstruction.constructParameterMap strips the + // type suffixes and returns bare key=value pairs. + LinkedHashMap params = + ParameterizedBuiltinCPInstruction.constructParameterMap(parts); + + // The target CPOperand is needed by ComputationCPInstruction's + // getInputs() / getLineageItem() machinery. + CPOperand input = new CPOperand(params.get("target"), + org.apache.sysds.common.Types.ValueType.FP64, + org.apache.sysds.common.Types.DataType.MATRIX); + + // Validate required keys. + if (!params.containsKey("sensitivity")) + throw new DMLRuntimeException(opcode + ": missing 'sensitivity'"); + if (!params.containsKey("epsilon")) + throw new DMLRuntimeException(opcode + ": missing 'epsilon'"); + if (opcode.equals(OPCODE_GAUSSIAN) && !params.containsKey("delta")) + throw new DMLRuntimeException(opcode + ": missing 'delta'"); + + return new DPBuiltinCPInstruction(input, output, opcode, str, params); + } + + // ----------------------------------------------------------------------- + // Core execution + // ----------------------------------------------------------------------- + + /** + * Executes the DP release. + * + *
    + *
  1. Read the aggregate {@link MatrixBlock} from the variable table.
    2. + *
  2. Determine sensitivity via {@link #sensitivityOf} (Phase-1 stub).
    3. + *
  3. Generate a noise {@link MatrixBlock} of the same shape.
    4. + *
  4. Add noise element-wise using the existing binary-operator path.
    5. + *
  5. Record the release with the session-scoped + * {@link RDPAccountant}; throw if budget is exhausted.
    6. + *
  6. Write the noisy block back to the variable table and release + * the input pin.
    7. + *
+ */ + @Override + public void processInstruction(ExecutionContext ec) { + + // ── 1. Read aggregate input ───────────────────────────────────────── + // getMatrixInput pins the block in memory and increments the + // reference count; we must call releaseMatrixInput afterwards. + MatrixBlock inBlock = ec.getMatrixInput(_params.get("target")); + + // ── 2. Parse DP parameters ────────────────────────────────────────── + double epsilon = parsePositiveDouble("epsilon"); + double delta = instOpcode.equals(OPCODE_GAUSSIAN) + ? parsePositiveDouble("delta") : 0.0; + + // ── 3. Determine sensitivity (Phase-1: caller-supplied constant) ──── + double sensitivity = sensitivityOf(inBlock); + + // ── 4. Generate and add noise ──────────────────────────────────────── + MatrixBlock noiseBlock = generateNoise(inBlock, sensitivity, epsilon, delta); + + // Element-wise addition via the standard binary-operator path. + // binaryOperations allocates the output block internally. + BinaryOperator plusOp = new BinaryOperator(Plus.getPlusFnObject()); + MatrixBlock outBlock = new MatrixBlock(); + inBlock.binaryOperations(plusOp, noiseBlock, outBlock); + + // ── 5. Record release and enforce budget ──────────────────────────── + // getRDPAccountant() returns a lazy-initialised RDPAccountant that is + // owned by this ExecutionContext (added in a companion EC patch). + RDPAccountant accountant = ec.getRDPAccountant(); + accountant.compose(epsilon, delta, sensitivity); // throws on exhaustion + + // ── 6. Write output and release input pin ─────────────────────────── + ec.releaseMatrixInput(_params.get("target")); + ec.setMatrixOutput(output.getName(), outBlock); + } + + // ----------------------------------------------------------------------- + // Sensitivity seam (Phase-1 stub; Phase-2 replaces this body only) + // ----------------------------------------------------------------------- + + /** + * Returns the sensitivity of {@code aggregate} to a single-record change. + * + *

Phase 1 (now): returns the caller-supplied literal from the + * DML script. Sensitivity analysis is the caller's responsibility. + * + *

Phase 2 (HOP-level rewrite pass): replace this body with a + * call that inspects the HOP node that produced {@code aggregate}, reads + * the {@code sensitivityBound} field computed during compilation, and + * returns it. No other line in this class changes. + * + * @param aggregate the already-computed aggregate block (ignored in + * Phase 1; used in Phase 2 to look up lineage) + * @return caller-supplied sensitivity constant + */ + private double sensitivityOf(MatrixBlock aggregate) { + // Phase 1: unwrap the literal or variable value from the param map. + // In Phase 2, replace the body below with HOP-annotation lookup. + return parsePositiveDouble("sensitivity"); + } + + // ----------------------------------------------------------------------- + // Noise generation + // ----------------------------------------------------------------------- + + /** + * Generates a noise {@link MatrixBlock} of the same shape as + * {@code aggregate}, filled with samples from the mechanism-appropriate + * distribution calibrated to ({@code sensitivity}, {@code epsilon}, + * {@code delta}). + * + *

Both mechanisms produce a dense block. Sparsity exploitation is + * left for future work; for the aggregate outputs targeted here (e.g. + * column means, row sums) the aggregate is already dense. + */ + private MatrixBlock generateNoise( + MatrixBlock aggregate, + double sensitivity, + double epsilon, + double delta) { + + int rows = aggregate.getNumRows(); + int cols = aggregate.getNumColumns(); + MatrixBlock noise = new MatrixBlock(rows, cols, false); // dense + noise.allocateDenseBlock(); + + if (instOpcode.equals(OPCODE_LAPLACE)) { + fillLaplaceNoise(noise, sensitivity / epsilon); + } else { + // Gaussian mechanism: calibrate sigma for (epsilon, delta)-DP. + // Standard formula: sigma >= sensitivity * sqrt(2 * ln(1.25/delta)) / epsilon + double sigma = sensitivity + * Math.sqrt(2.0 * Math.log(1.25 / delta)) + / epsilon; + fillGaussianNoise(noise, sigma); + } + + noise.recomputeNonZeros(); + return noise; + } + + /** + * Fills {@code block} with i.i.d. Laplace(0, scale) samples using the + * inverse-CDF method. + * + *

For u ~ Uniform(0, 1): X = -scale * sign(u - 0.5) * ln(1 - 2|u - 0.5|) + */ + private static void fillLaplaceNoise(MatrixBlock block, double scale) { + ThreadLocalRandom rng = ThreadLocalRandom.current(); + int rows = block.getNumRows(); + int cols = block.getNumColumns(); + for (int r = 0; r < rows; r++) { + for (int c = 0; c < cols; c++) { + double u = rng.nextDouble(); // u in (0, 1) + double v = u - 0.5; + // Guard against the degenerate u == 0.5 case (ln(0) = -inf). + if (v == 0.0) v = 1e-15; + double sample = -scale * Math.signum(v) * Math.log(1.0 - 2.0 * Math.abs(v)); + block.set(r, c, sample); + } + } + } + + /** + * Fills {@code block} with i.i.d. N(0, sigma²) samples. + * + *

Uses {@link ThreadLocalRandom#nextGaussian()} which is thread-safe + * and does not require external libraries. + */ + private static void fillGaussianNoise(MatrixBlock block, double sigma) { + ThreadLocalRandom rng = ThreadLocalRandom.current(); + int rows = block.getNumRows(); + int cols = block.getNumColumns(); + for (int r = 0; r < rows; r++) { + for (int c = 0; c < cols; c++) { + block.set(r, c, sigma * rng.nextGaussian()); + } + } + } + + // ----------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------- + + /** + * Parses a parameter value as a positive {@code double}. + * + * @throws DMLRuntimeException if the key is absent, unparseable, or + * non-positive + */ + private double parsePositiveDouble(String key) { + String raw = _params.get(key); + if (raw == null) + throw new DMLRuntimeException( + instOpcode + ": parameter '" + key + "' is missing"); + double v; + try { + v = Double.parseDouble(raw); + } catch (NumberFormatException e) { + throw new DMLRuntimeException( + instOpcode + ": parameter '" + key + + "' is not a valid number: " + raw); + } + if (!(v > 0.0)) + throw new DMLRuntimeException( + instOpcode + ": parameter '" + key + + "' must be strictly positive, got " + v); + return v; + } +} diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java new file mode 100755 index 00000000000..884012e4ee6 --- /dev/null +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java @@ -0,0 +1,283 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.sysds.runtime.privacy.dp; + +import org.apache.sysds.runtime.DMLRuntimeException; + +/** + * Session-scoped Rényi Differential Privacy (RDP) budget accountant. + * + *

Purpose

+ * Tracks composition of DP releases across the lifetime of a DML script + * execution. Each call to {@link #compose} records one release and checks + * whether the cumulative privacy cost has exceeded the user-specified budget. + * + *

Why Rényi DP?

+ * Basic composition adds epsilons linearly, giving very loose bounds with + * many releases. Rényi DP divergences compose additively at the + * same order α. Converting the running Rényi sum to (ε, δ) via the standard + * conversion formula yields substantially tighter bounds — particularly for + * Gaussian mechanisms, which are common in federated learning. + * + *

Orders tracked

+ * We track a discrete set of Rényi orders α ∈ {2, 4, 8, 16, 32, 64, 128, + * 256, 512, 1024}. At query time we take the minimum converted ε across all + * orders, which is the tightest available bound. + * + *

Composition rules

+ * For the Gaussian mechanism with noise scale σ and sensitivity Δf, the + * Rényi divergence of order α between outputs on neighbouring datasets is: + * 
+ *   D_α = α · Δf² / (2σ²)
+ *
+ * where σ is back-derived from the caller's (ε, δ) parameters via the + * standard calibration formula. See {@link #rdpGaussian} for details. + * + * For the Laplace mechanism with scale b = Δf/ε, the Rényi divergence at + * order α is: + * 
+ *   D_α = (1/(α-1)) · ln( α/(2α-1) · exp((α-1)/b) + (α-1)/(2α-1) · exp(-α/b) )
+ *       (for α > 1; the limit as α → 1 is 1/b, i.e. the KL divergence)
+ *
+ * + *

Conversion: Rényi DP → (ε, δ)-DP

+ * Given accumulated Rényi divergence R[α] at order α and a target δ: + * 
+ *   ε(α) = R[α] + log(1 - 1/α) - log(δ · (α - 1)) / α
+ *
+ * The reported total cost is min_α ε(α). + * + *

Lifecycle

+ * One instance is created per {@code ExecutionContext} (lazy init). It is + * garbage-collected with the context when the script finishes; no state + * leaks between script executions or between concurrent scripts. + * + *

Thread safety

+ * Not thread-safe. A single DML script executes instructions sequentially + * on one thread, so no synchronisation is needed. + * + * @see DPBuiltinCPInstruction + */ +public class RDPAccountant { + + // ----------------------------------------------------------------------- + // Rényi orders to track + // ----------------------------------------------------------------------- + + /** + * Discrete set of Rényi orders α. All must be > 1. + * Finer grids give tighter bounds; this set is a reasonable default + * that covers the range relevant for typical ML workloads. + */ + private static final double[] ORDERS = { + 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 + }; + + // ----------------------------------------------------------------------- + // State + // ----------------------------------------------------------------------- + + /** Running accumulated Rényi divergence at each order. */ + private final double[] _rdpSum = new double[ORDERS.length]; + + /** User-specified total privacy budget (ε). */ + private final double _epsilonBudget; + + /** User-specified δ used for the RDP-to-(ε,δ) conversion. */ + private final double _delta; + + /** Number of releases recorded so far (for error messages). */ + private int _releaseCount = 0; + + // ----------------------------------------------------------------------- + // Constructor + // ----------------------------------------------------------------------- + + /** + * Creates an accountant with the given global budget. + * + *

Typical usage: the DML script sets the budget once at the top + * (future work: a {@code dp_set_budget(epsilon, delta)} built-in), + * or the accountant is created with defaults and the budget is checked + * after each release. + * + * @param epsilonBudget total ε budget for the script execution (must be > 0) + * @param delta δ used for the RDP-to-(ε,δ) conversion (must be in (0,1)) + */ + public RDPAccountant(double epsilonBudget, double delta) { + if (!(epsilonBudget > 0)) + throw new DMLRuntimeException( + "RDPAccountant: epsilonBudget must be > 0, got " + epsilonBudget); + if (!(delta > 0 && delta < 1)) + throw new DMLRuntimeException( + "RDPAccountant: delta must be in (0,1), got " + delta); + _epsilonBudget = epsilonBudget; + _delta = delta; + } + + /** + * Convenience constructor using a liberal default δ = 1e-5. + * Suitable when the calling script does not specify δ explicitly. + */ + public RDPAccountant(double epsilonBudget) { + this(epsilonBudget, 1e-5); + } + + // ----------------------------------------------------------------------- + // Core API + // ----------------------------------------------------------------------- + + /** + * Records one DP release and checks the budget. + * + *

This method must be called before the result is written to + * the variable table. If the budget is exhausted, it throws and the + * caller's result is discarded, preventing an unaccounted release. + * + *

The mechanism type (Laplace vs Gaussian) is inferred from the + * parameters: if {@code delta == 0} the release is treated as Laplace; + * otherwise it is treated as Gaussian. + * + * @param epsilon the ε parameter for this individual release (> 0) + * @param delta the δ parameter for this release (0 for Laplace) + * @param sensitivity the L2 sensitivity Δf of the released quantity (> 0) + * @throws DMLRuntimeException if the cumulative ε after this release + * would exceed the budget + */ + public void compose(double epsilon, double delta, double sensitivity) { + _releaseCount++; + + // Accumulate Rényi divergence at each order. + for (int i = 0; i < ORDERS.length; i++) { + double alpha = ORDERS[i]; + double rdp; + if (delta == 0.0) { + rdp = rdpLaplace(alpha, sensitivity, epsilon); + } else { + // Back-derive σ from the (ε, δ) calibration formula for the + // Gaussian mechanism, then compute the RDP contribution. + double sigma = gaussianSigma(sensitivity, epsilon, delta); + rdp = rdpGaussian(alpha, sensitivity, sigma); + } + _rdpSum[i] += rdp; + } + + // Convert accumulated RDP to (ε, δ) and check. + double spentEpsilon = totalEpsilonSpent(); + if (spentEpsilon > _epsilonBudget) { + throw new DMLRuntimeException(String.format( + "Privacy budget exhausted after %d release(s): " + + "spent ε ≈ %.6f exceeds budget ε = %.6f (δ = %.2e). " + + "Reduce the number of releases or widen the budget.", + _releaseCount, spentEpsilon, _epsilonBudget, _delta)); + } + } + + // ----------------------------------------------------------------------- + // Inspection + // ----------------------------------------------------------------------- + + /** + * Returns the current total privacy cost as an ε value under the + * accountant's δ, using the tightest available Rényi order. + */ + public double totalEpsilonSpent() { + double minEpsilon = Double.MAX_VALUE; + for (int i = 0; i < ORDERS.length; i++) { + double alpha = ORDERS[i]; + // Standard RDP-to-(ε,δ) conversion: + // ε(α) = R[α] + log(1 - 1/α) - log(δ·(α-1)) / α + // Reference: Mironov 2017, Proposition 3. + double eps = _rdpSum[i] + + Math.log(1.0 - 1.0 / alpha) + - Math.log(_delta * (alpha - 1.0)) / alpha; + if (eps < minEpsilon) + minEpsilon = eps; + } + return minEpsilon; + } + + /** Returns the remaining ε budget (may be negative if budget is exceeded). */ + public double remainingBudget() { + return _epsilonBudget - totalEpsilonSpent(); + } + + /** Returns the number of DP releases recorded so far. */ + public int releaseCount() { + return _releaseCount; + } + + // ----------------------------------------------------------------------- + // Mechanism-specific RDP contributions + // ----------------------------------------------------------------------- + + /** + * Rényi divergence of order α for the Laplace mechanism with scale + * b = sensitivity / epsilon. + * + *

For α > 1 and integer α, the closed form is: + * 

+     *   D_α = (1/(α-1)) · ln( α/(2α-1)·exp((α-1)/b) + (α-1)/(2α-1)·exp(-α/b) )
+     *
+ * + *

For non-integer α we use the same formula, which is the natural + * analytic continuation (see Mironov 2017, Proposition 3, example 1). + * We clamp the log argument to avoid NaN when inputs are degenerate. + */ + private static double rdpLaplace(double alpha, double sensitivity, double epsilon) { + double b = sensitivity / epsilon; // Laplace scale + double t1 = alpha / (2.0 * alpha - 1.0) * Math.exp((alpha - 1.0) / b); + double t2 = (alpha - 1.0) / (2.0 * alpha - 1.0) * Math.exp(-alpha / b); + double arg = t1 + t2; + if (arg <= 0) return 0.0; // degenerate: treat as zero cost + return Math.log(arg) / (alpha - 1.0); + } + + /** + * Rényi divergence of order α for the Gaussian mechanism with noise + * scale σ and L2 sensitivity Δf. + * + *

For α > 1: + * 

+     *   D_α = α · Δf² / (2σ²)
+     *
+ * + *

This is the standard result for the Gaussian mechanism (see + * Mironov 2017, Proposition 3, example 2). + */ + private static double rdpGaussian(double alpha, double sensitivity, double sigma) { + return alpha * (sensitivity * sensitivity) / (2.0 * sigma * sigma); + } + + /** + * Back-derives the Gaussian noise scale σ from the (ε, δ)-DP parameters + * using the standard calibration inequality: + * 

+     *   σ = Δf · sqrt(2 · ln(1.25 / δ)) / ε
+     *
+ * + *

This is the formula used by {@code DPBuiltinCPInstruction} to + * generate the actual noise, so the RDP contribution it records is + * exactly consistent with the noise injected. + */ + private static double gaussianSigma(double sensitivity, double epsilon, double delta) { + return sensitivity * Math.sqrt(2.0 * Math.log(1.25 / delta)) / epsilon; + } +} diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java new file mode 100755 index 00000000000..e9bc2c6ea9e --- /dev/null +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -0,0 +1,350 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.sysds.test.functions.privacy.dp; + +import org.apache.sysds.runtime.DMLRuntimeException; +import org.apache.sysds.runtime.privacy.dp.RDPAccountant; +import org.junit.Test; + +import static org.junit.Assert.*; + +/** + * Tests for {@code DPBuiltinCPInstruction} and {@code RDPAccountant}. + * + *

The tests are grouped into three levels: + *

    + *
  1. Unit tests on RDPAccountant — verify composition, conversion, + * and budget enforcement in isolation, with no dependency on the full + * SystemDS runtime.
    2. + *
  2. Noise distribution tests — verify that the noise blocks + * generated by the Laplace and Gaussian mechanisms have statistically + * correct means and variances (Kolmogorov-Smirnov style sanity checks).
    3. + *
  3. DML integration tests — run complete DML scripts and verify + * end-to-end correctness via the existing AutomatedTestBase machinery.
    4. + *
+ * + *

The DML integration tests require a built SystemDS jar and are separated + * into a companion class {@code DPBuiltinDMLTest} (shown at the bottom of + * this file as a skeleton). + */ +public class DPBuiltinCPInstructionTest { + + private static final double EPS = 1e-9; + + // ======================================================================= + // 1. RDPAccountant unit tests + // ======================================================================= + + @Test + public void testAccountantInitialisesAtZeroCost() { + RDPAccountant acc = new RDPAccountant(1.0, 1e-5); + // No releases yet: total cost should be a large negative number + // (conversion formula gives -∞ when rdpSum = 0 for all orders), + // so remainingBudget() should exceed the budget. + assertTrue("No releases should leave budget intact", + acc.remainingBudget() > 0); + assertEquals(0, acc.releaseCount()); + } + + @Test + public void testSingleLaplaceReleaseDoesNotExceedBudget() { + // epsilon=0.5, budget=1.0: one release should consume < budget. + RDPAccountant acc = new RDPAccountant(1.0, 1e-5); + acc.compose(0.5, 0.0, 1.0); // Laplace, sensitivity=1 + assertEquals(1, acc.releaseCount()); + assertTrue("Single release within budget", + acc.totalEpsilonSpent() <= 1.0); + } + + @Test + public void testSingleGaussianReleaseDoesNotExceedBudget() { + RDPAccountant acc = new RDPAccountant(1.0, 1e-5); + acc.compose(0.5, 1e-5, 1.0); // Gaussian + assertEquals(1, acc.releaseCount()); + assertTrue("Single Gaussian release within budget", + acc.totalEpsilonSpent() <= 1.0); + } + + @Test(expected = DMLRuntimeException.class) + public void testBudgetExhaustionThrows() { + // Budget = 0.1, but we try to make 10 releases at epsilon=0.5 each. + // After enough releases the budget must be exceeded. + RDPAccountant acc = new RDPAccountant(0.1, 1e-5); + for (int i = 0; i < 10; i++) { + acc.compose(0.5, 0.0, 1.0); // will throw before the 10th + } + } + + @Test + public void testCompositionIsMonotonicallyIncreasing() { + RDPAccountant acc = new RDPAccountant(100.0, 1e-5); // large budget + double prev = acc.totalEpsilonSpent(); + for (int i = 0; i < 5; i++) { + acc.compose(0.3, 1e-5, 1.0); + double current = acc.totalEpsilonSpent(); + assertTrue("Epsilon spent must increase with each release", + current > prev); + prev = current; + } + } + + @Test + public void testGaussianTighterThanLaplaceForSameEpsilon() { + // For the same nominal (ε, δ), Gaussian uses RDP composition which + // is tighter than Laplace with basic composition. After 5 releases: + // Laplace (basic, worst-case): 5ε + // Gaussian (RDP) : something < 5ε + double eps = 0.5; + double delta = 1e-5; + + RDPAccountant gaussian = new RDPAccountant(100.0, delta); + RDPAccountant laplace = new RDPAccountant(100.0, delta); + + for (int i = 0; i < 5; i++) { + gaussian.compose(eps, delta, 1.0); + laplace.compose(eps, 0.0, 1.0); + } + + // After 5 releases, Gaussian RDP bound should be tighter. + // (Both may be < 5*eps; the point is Gaussian <= Laplace.) + assertTrue("Gaussian RDP bound should be <= Laplace bound after 5 releases", + gaussian.totalEpsilonSpent() <= laplace.totalEpsilonSpent() + 1e-6); + } + + @Test + public void testRemainingBudgetDecreasesMonotonically() { + RDPAccountant acc = new RDPAccountant(2.0, 1e-5); + double prev = acc.remainingBudget(); + for (int i = 0; i < 3; i++) { + acc.compose(0.2, 1e-5, 1.0); + double current = acc.remainingBudget(); + assertTrue("Remaining budget must decrease", current < prev); + prev = current; + } + } + + @Test + public void testSmallerSensitivityCheaper() { + // A release with sensitivity 0.1 should consume less budget than + // one with sensitivity 1.0 at the same (ε, δ). + RDPAccountant acc1 = new RDPAccountant(100.0, 1e-5); + RDPAccountant acc2 = new RDPAccountant(100.0, 1e-5); + acc1.compose(0.5, 1e-5, 0.1); // low sensitivity + acc2.compose(0.5, 1e-5, 1.0); // high sensitivity + + assertTrue("Lower sensitivity must cost less budget", + acc1.totalEpsilonSpent() < acc2.totalEpsilonSpent()); + } + + // ======================================================================= + // 2. Noise distribution tests (statistical sanity checks) + // ======================================================================= + // These tests generate many samples and verify that the empirical mean + // is near zero and the empirical variance matches the theoretical value + // within a reasonable tolerance. + // + // Note: these tests exercise the static fill* methods indirectly by + // calling the noise-generation logic via reflection or by making the + // methods package-private. The simplest approach for a student project + // is to make fillLaplaceNoise / fillGaussianNoise package-private and + // call them directly from the test (same package). + + @Test + public void testLaplaceNoiseMeanNearZero() { + // For 10000 samples the empirical mean should be within 3σ/√n of 0. + int n = 10_000; + double scale = 2.0; + double[] samples = sampleLaplace(n, scale); + double mean = mean(samples); + double theoreticalStdErr = scale * Math.sqrt(2.0) / Math.sqrt(n); + assertTrue("Laplace mean should be near 0", + Math.abs(mean) < 5 * theoreticalStdErr); + } + + @Test + public void testLaplaceNoiseVarianceCorrect() { + // Var[Laplace(0, b)] = 2b². Allow 10% relative error for n=10000. + int n = 10_000; + double scale = 1.5; + double[] samples = sampleLaplace(n, scale); + double variance = variance(samples); + double expected = 2.0 * scale * scale; + assertEquals("Laplace variance", expected, variance, 0.1 * expected); + } + + @Test + public void testGaussianNoiseMeanNearZero() { + int n = 10_000; + double sigma = 3.0; + double[] samples = sampleGaussian(n, sigma); + double mean = mean(samples); + double theoreticalStdErr = sigma / Math.sqrt(n); + assertTrue("Gaussian mean should be near 0", + Math.abs(mean) < 5 * theoreticalStdErr); + } + + @Test + public void testGaussianNoiseVarianceCorrect() { + int n = 10_000; + double sigma = 2.0; + double[] samples = sampleGaussian(n, sigma); + double variance = variance(samples); + double expected = sigma * sigma; + assertEquals("Gaussian variance", expected, variance, 0.1 * expected); + } + + // ----------------------------------------------------------------------- + // Helpers for noise distribution tests + // ----------------------------------------------------------------------- + + /** Sample n Laplace(0, scale) values using the inverse-CDF method. */ + private static double[] sampleLaplace(int n, double scale) { + java.util.concurrent.ThreadLocalRandom rng = + java.util.concurrent.ThreadLocalRandom.current(); + double[] out = new double[n]; + for (int i = 0; i < n; i++) { + double u = rng.nextDouble(); + double v = u - 0.5; + if (v == 0.0) v = 1e-15; + out[i] = -scale * Math.signum(v) * Math.log(1.0 - 2.0 * Math.abs(v)); + } + return out; + } + + /** Sample n N(0, sigma²) values. */ + private static double[] sampleGaussian(int n, double sigma) { + java.util.concurrent.ThreadLocalRandom rng = + java.util.concurrent.ThreadLocalRandom.current(); + double[] out = new double[n]; + for (int i = 0; i < n; i++) { + out[i] = sigma * rng.nextGaussian(); + } + return out; + } + + private static double mean(double[] xs) { + double s = 0; + for (double x : xs) s += x; + return s / xs.length; + } + + private static double variance(double[] xs) { + double m = mean(xs); + double s = 0; + for (double x : xs) s += (x - m) * (x - m); + return s / (xs.length - 1); + } +} + + +// ========================================================================== +// 3. DML integration test skeleton +// ========================================================================== +// +// Full integration tests extend AutomatedTestBase and drive the DML runner. +// Each test: +// (a) Writes a DML script to a temp file. +// (b) Provides input matrices via TestUtils. +// (c) Calls runTest() and reads the output MatrixBlock. +// (d) Verifies that the noisy result differs from the clean result by a +// statistically plausible amount (not zero, not astronomically large). +// +// The test below is a skeleton that compiles but needs the full SystemDS +// test infrastructure to run. + +/* +package org.apache.sysds.test.functions.privacy.dp; + +import org.apache.sysds.common.Types; +import org.apache.sysds.runtime.matrix.data.MatrixBlock; +import org.apache.sysds.test.AutomatedTestBase; +import org.apache.sysds.test.TestConfiguration; +import org.apache.sysds.test.TestUtils; +import org.junit.Test; + +public class DPBuiltinDMLTest extends AutomatedTestBase { + + private static final String TEST_DIR = "functions/privacy/dp/"; + private static final String TEST_CLASS = TEST_DIR + DPBuiltinDMLTest.class.getSimpleName() + "/"; + private static final String DML_LAPLACE = + "X = read($1);\n" + + "result = dp_laplace(colMeans(X), sensitivity=1.0, epsilon=$2);\n" + + "write(result, $3, format=\"binary\");\n"; + private static final String DML_GAUSSIAN = + "X = read($1);\n" + + "result = dp_gaussian(colMeans(X), sensitivity=1.0, epsilon=$2, delta=1e-5);\n" + + "write(result, $3, format=\"binary\");\n"; + + @Override + public void setUp() { + addTestConfiguration("DPLaplace", new TestConfiguration(TEST_CLASS, "DPLaplace")); + addTestConfiguration("DPGaussian", new TestConfiguration(TEST_CLASS, "DPGaussian")); + } + + @Test + public void testLaplaceOutputDiffersFromCleanMean() { + runDPTest("DPLaplace", DML_LAPLACE, "0.5"); + } + + @Test + public void testGaussianOutputDiffersFromCleanMean() { + runDPTest("DPGaussian", DML_GAUSSIAN, "0.5"); + } + + @Test + public void testHighEpsilonIsCloserToTruth() { + // Higher ε → less noise → result closer to the true mean. + double noisyLow = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "0.1"); + double noisyHigh = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "4.0"); + assertTrue("ε=4 should give less noise than ε=0.1", noisyHigh < noisyLow); + } + + private void runDPTest(String testName, String dml, String epsilonStr) { + getAndLoadTestConfiguration(testName); + int rows = 100, cols = 10; + double[][] data = TestUtils.generateTestMatrix(rows, cols, 0, 1, 1.0, 42); + writeInputMatrixWithMTD("X", data, false); + writeScriptFile(testName + ".dml", dml); + programArgs = new String[]{ input("X"), epsilonStr, output("result") }; + runTest(true, false, null, -1); + MatrixBlock result = readDMLMatrixFromHDFS("result"); + // The noisy result should be a (1 × cols) row vector. + assertEquals(1, result.getNumRows()); + assertEquals(cols, result.getNumColumns()); + // Must differ from the exact mean by a non-trivial amount. + // (A single-seed exact-equality check is fragile; use range check.) + double maxNoise = maxAbsValue(result); + assertTrue("Result should not be exactly zero", maxNoise > 0); + } + + private double maxAbsDiff(String testName, String dml, String epsilonStr) { + // Omitted for brevity: run the test, compute max |noisy - clean|. + return 0; // placeholder + } + + private static double maxAbsValue(MatrixBlock m) { + double max = 0; + for (int r = 0; r < m.getNumRows(); r++) + for (int c = 0; c < m.getNumColumns(); c++) + max = Math.max(max, Math.abs(m.get(r, c))); + return max; + } +} +*/ From 19c2504b25915a1979917737f2a5a633b0341769 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Fri, 26 Jun 2026 21:01:04 +0200 Subject: [PATCH 04/26] Add dp_gaussian and dp_laplace operations --- .../org/apache/sysds/common/Builtins.java | 2 ++ .../apache/sysds/common/InstructionType.java | 1 + .../java/org/apache/sysds/common/Opcodes.java | 4 +++ .../java/org/apache/sysds/common/Types.java | 2 +- .../sysds/hops/ParameterizedBuiltinOp.java | 15 ++++++++--- .../sysds/lops/ParameterizedBuiltin.java | 14 ++++++++++- .../parser/BuiltinFunctionExpression.java | 25 +++++++++++++++++++ .../apache/sysds/parser/DMLTranslator.java | 23 +++++++++++++++++ .../context/ExecutionContext.java | 9 +++++++ .../instructions/CPInstructionParser.java | 6 ++++- .../instructions/cp/CPInstruction.java | 3 ++- .../cp/DPBuiltinCPInstruction.java | 5 +--- .../cp/DPBuiltinCPInstructionTest.java | 14 +++++++---- 13 files changed, 106 insertions(+), 17 deletions(-) diff --git a/src/main/java/org/apache/sysds/common/Builtins.java b/src/main/java/org/apache/sysds/common/Builtins.java index f5719641df7..be6ee0f33db 100644 --- a/src/main/java/org/apache/sysds/common/Builtins.java +++ b/src/main/java/org/apache/sysds/common/Builtins.java @@ -116,6 +116,8 @@ public enum Builtins { DECISIONTREEPREDICT("decisionTreePredict", true), DECOMPRESS("decompress", false), DEDUP("dedup", true), + DP_LAPLACE("dp_laplace", false), + DP_GAUSSIAN("dp_gaussian", false), DEEPWALK("deepWalk", true), DET("det", false), DETECTSCHEMA("detectSchema", false), diff --git a/src/main/java/org/apache/sysds/common/InstructionType.java b/src/main/java/org/apache/sysds/common/InstructionType.java index e0e77c46c59..ed795d038e2 100644 --- a/src/main/java/org/apache/sysds/common/InstructionType.java +++ b/src/main/java/org/apache/sysds/common/InstructionType.java @@ -63,6 +63,7 @@ public enum InstructionType { MMChain, Union, EINSUM, + DPBuiltin, //SP Types MAPMM, diff --git a/src/main/java/org/apache/sysds/common/Opcodes.java b/src/main/java/org/apache/sysds/common/Opcodes.java index 9a894dde13b..f9ed57eae06 100644 --- a/src/main/java/org/apache/sysds/common/Opcodes.java +++ b/src/main/java/org/apache/sysds/common/Opcodes.java @@ -194,6 +194,10 @@ public enum Opcodes { LIST("list", InstructionType.BuiltinNary), EINSUM("einsum", InstructionType.BuiltinNary), + //DP built-in functions + DP_LAPLACE("dp_laplace", InstructionType.DPBuiltin), + DP_GAUSSIAN("dp_gaussian", InstructionType.DPBuiltin), + //Parametrized builtin functions AUTODIFF("autoDiff", InstructionType.ParameterizedBuiltin), CONTAINS("contains", InstructionType.ParameterizedBuiltin), diff --git a/src/main/java/org/apache/sysds/common/Types.java b/src/main/java/org/apache/sysds/common/Types.java index 624c9eed3c6..250fc45a5c7 100644 --- a/src/main/java/org/apache/sysds/common/Types.java +++ b/src/main/java/org/apache/sysds/common/Types.java @@ -806,7 +806,7 @@ public static ReOrgOp valueOfByOpcode(String opcode) { /** Parameterized operations that require named variable arguments */ public enum ParamBuiltinOp { - AUTODIFF, CDF, CONTAINS, INVALID, INVCDF, GROUPEDAGG, RMEMPTY, REPLACE, REXPAND, + AUTODIFF, CDF, CONTAINS, DP_LAPLACE, DP_GAUSSIAN, INVALID, INVCDF, GROUPEDAGG, RMEMPTY, REPLACE, REXPAND, LOWER_TRI, UPPER_TRI, TRANSFORMAPPLY, TRANSFORMDECODE, TRANSFORMCOLMAP, TRANSFORMMETA, TOKENIZE, TOSTRING, LIST, PARAMSERV diff --git a/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java b/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java index 61a4b8b8f91..7761521e415 100644 --- a/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java +++ b/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java @@ -182,7 +182,7 @@ public Lop constructLops() } case CONTAINS: case CDF: - case INVCDF: + case INVCDF: case REPLACE: case LOWER_TRI: case UPPER_TRI: @@ -194,7 +194,9 @@ public Lop constructLops() case TOSTRING: case PARAMSERV: case LIST: - case AUTODIFF:{ + case AUTODIFF: + case DP_LAPLACE: + case DP_GAUSSIAN:{ ParameterizedBuiltin pbilop = new ParameterizedBuiltin( inputlops, _op, getDataType(), getValueType(), et); if( isMultiThreadedOpType() ) @@ -688,7 +690,11 @@ else if( _op == ParamBuiltinOp.TRANSFORMAPPLY ) { return new MatrixCharacteristics(dc.getRows(), dc.getCols(), -1, dc.getLength()); } } - + else if( _op == ParamBuiltinOp.DP_LAPLACE || _op == ParamBuiltinOp.DP_GAUSSIAN ) { + if( dc.dimsKnown() ) + ret = new MatrixCharacteristics(dc.getRows(), dc.getCols(), -1, dc.getLength()); + } + return ret; } @Override @@ -758,7 +764,8 @@ && getTargetHop().areDimsBelowThreshold() ) { if (_op == ParamBuiltinOp.TRANSFORMCOLMAP || _op == ParamBuiltinOp.TRANSFORMMETA || _op == ParamBuiltinOp.TOSTRING || _op == ParamBuiltinOp.LIST || _op == ParamBuiltinOp.CDF || _op == ParamBuiltinOp.INVCDF - || _op == ParamBuiltinOp.PARAMSERV) { + || _op == ParamBuiltinOp.PARAMSERV + || _op == ParamBuiltinOp.DP_LAPLACE || _op == ParamBuiltinOp.DP_GAUSSIAN) { _etype = ExecType.CP; } diff --git a/src/main/java/org/apache/sysds/lops/ParameterizedBuiltin.java b/src/main/java/org/apache/sysds/lops/ParameterizedBuiltin.java index 3604121aac8..48ecbaa72df 100644 --- a/src/main/java/org/apache/sysds/lops/ParameterizedBuiltin.java +++ b/src/main/java/org/apache/sysds/lops/ParameterizedBuiltin.java @@ -204,7 +204,19 @@ public String getInstructions(String output) compileGenericParamMap(sb, _inputParams); break; } - + case DP_LAPLACE: { + sb.append(Opcodes.DP_LAPLACE); + sb.append(OPERAND_DELIMITOR); + compileGenericParamMap(sb, _inputParams); + break; + } + case DP_GAUSSIAN: { + sb.append(Opcodes.DP_GAUSSIAN); + sb.append(OPERAND_DELIMITOR); + compileGenericParamMap(sb, _inputParams); + break; + } + default: throw new LopsException(this.printErrorLocation() + "In ParameterizedBuiltin Lop, Unknown operation: " + _operation); } diff --git a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java index ab0c7993b4e..1425a794575 100644 --- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java +++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java @@ -2005,6 +2005,31 @@ else if(this.getOpCode() == Builtins.MAX_POOL || this.getOpCode() == Builtins.AV } else raiseValidateError("Local instruction not allowed in dml script"); + case DP_LAPLACE: { + checkNumParameters(3); + checkMatrixParam(getFirstExpr()); + checkScalarParam(getSecondExpr()); + checkScalarParam(getThirdExpr()); + output.setDataType(DataType.MATRIX); + output.setValueType(ValueType.FP64); + output.setDimensions( + getFirstExpr().getOutput().getDim1(), + getFirstExpr().getOutput().getDim2()); + break; + } + case DP_GAUSSIAN: { + checkNumParameters(4); + checkMatrixParam(getFirstExpr()); + checkScalarParam(getSecondExpr()); + checkScalarParam(getThirdExpr()); + checkScalarParam(getFourthExpr()); + output.setDataType(DataType.MATRIX); + output.setValueType(ValueType.FP64); + output.setDimensions( + getFirstExpr().getOutput().getDim1(), + getFirstExpr().getOutput().getDim2()); + break; + } case COMPRESS: case DECOMPRESS: if(OptimizerUtils.ALLOW_SCRIPT_LEVEL_COMPRESS_COMMAND){ diff --git a/src/main/java/org/apache/sysds/parser/DMLTranslator.java b/src/main/java/org/apache/sysds/parser/DMLTranslator.java index a8e1667d049..7950868e0b5 100644 --- a/src/main/java/org/apache/sysds/parser/DMLTranslator.java +++ b/src/main/java/org/apache/sysds/parser/DMLTranslator.java @@ -2310,6 +2310,10 @@ private Hop processBuiltinFunctionExpression(BuiltinFunctionExpression source, D if (source.getThirdExpr() != null) { expr3 = processExpression(source.getThirdExpr(), null, hops); } + Hop expr4 = null; + if (source.getFourthExpr() != null) { + expr4 = processExpression(source.getFourthExpr(), null, hops); + } Hop currBuiltinOp = null; target = (target == null) ? createTarget(source) : target; @@ -2589,6 +2593,25 @@ else if ( sop.equalsIgnoreCase(Opcodes.NOTEQUAL.toString()) ) case DECOMPRESS: currBuiltinOp = new UnaryOp(target.getName(), target.getDataType(), ValueType.FP64, OpOp1.DECOMPRESS, expr); break; + case DP_LAPLACE: { + LinkedHashMap dpLaplaceParams = new LinkedHashMap<>(); + dpLaplaceParams.put("target", expr); + dpLaplaceParams.put("sensitivity", expr2); + dpLaplaceParams.put("epsilon", expr3); + currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, + ValueType.FP64, ParamBuiltinOp.DP_LAPLACE, dpLaplaceParams); + break; + } + case DP_GAUSSIAN: { + LinkedHashMap dpGaussianParams = new LinkedHashMap<>(); + dpGaussianParams.put("target", expr); + dpGaussianParams.put("sensitivity", expr2); + dpGaussianParams.put("epsilon", expr3); + dpGaussianParams.put("delta", expr4); + currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, + ValueType.FP64, ParamBuiltinOp.DP_GAUSSIAN, dpGaussianParams); + break; + } case QUANTIZE_COMPRESS: currBuiltinOp = new BinaryOp(target.getName(), target.getDataType(), target.getValueType(), OpOp2.valueOf(source.getOpCode().name()), expr, expr2); break; diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java index 67cda352a73..baa6e74df7d 100644 --- a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java +++ b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java @@ -62,6 +62,7 @@ import org.apache.sysds.runtime.meta.MetaData; import org.apache.sysds.runtime.meta.MetaDataFormat; import org.apache.sysds.runtime.util.HDFSTool; +import org.apache.sysds.runtime.privacy.dp.RDPAccountant; import org.apache.sysds.utils.Statistics; import java.util.ArrayList; @@ -90,6 +91,8 @@ public class ExecutionContext { protected SEALClient _seal_client; + private RDPAccountant _rdpAccountant = null; + //parfor temporary functions (created by eval) protected Set _fnNames; @@ -144,6 +147,12 @@ public void setLineage(Lineage lineage) { _lineage = lineage; } + public RDPAccountant getRDPAccountant() { + if (_rdpAccountant == null) + _rdpAccountant = new RDPAccountant(1.0, 1e-5); + return _rdpAccountant; + } + public boolean isAutoCreateVars() { return _autoCreateVars; } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java b/src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java index 92e11b425dd..ca6baf058b4 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java @@ -65,6 +65,7 @@ import org.apache.sysds.runtime.instructions.cp.UnaryCPInstruction; import org.apache.sysds.runtime.instructions.cp.VariableCPInstruction; import org.apache.sysds.runtime.instructions.cp.UnionCPInstruction; +import org.apache.sysds.runtime.instructions.cp.DPBuiltinCPInstruction; import org.apache.sysds.runtime.instructions.cp.EinsumCPInstruction; import org.apache.sysds.runtime.instructions.cpfile.MatrixIndexingCPFileInstruction; @@ -226,7 +227,10 @@ public static CPInstruction parseSingleInstruction ( InstructionType cptype, Str case EINSUM: return EinsumCPInstruction.parseInstruction(str); - + + case DPBuiltin: + return DPBuiltinCPInstruction.parseInstruction(str); + default: throw new DMLRuntimeException("Invalid CP Instruction Type: " + cptype ); } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/CPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/CPInstruction.java index b8d84ca3898..668d2f36978 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/CPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/CPInstruction.java @@ -48,7 +48,8 @@ public enum CPType { EvictLineageCache, EINSUM, NoOp, Union, - QuantizeCompression + QuantizeCompression, + DPBuiltin } protected final CPType _cptype; diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java index f75ccb13cdb..746c8471ff8 100755 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -102,10 +102,7 @@ private DPBuiltinCPInstruction( String opcode, String istr, LinkedHashMap params) { - // input1 = the aggregate matrix; input2/3 unused at this level - // (scalars come from _params, not CPOperand fields, so that they - // can be either literals or DML variable names transparently). - super(null, input, null, null, output, opcode, istr); + super(CPType.DPBuiltin, null, input, null, output, opcode, istr); _params = params; } diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index e9bc2c6ea9e..6363b925b99 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -142,14 +142,18 @@ public void testRemainingBudgetDecreasesMonotonically() { @Test public void testSmallerSensitivityCheaper() { - // A release with sensitivity 0.1 should consume less budget than - // one with sensitivity 1.0 at the same (ε, δ). + // For the Gaussian mechanism, noise scales proportionally with sensitivity + // (sigma = sensitivity * C), so sensitivity² cancels in the RDP formula + // alpha * sensitivity² / (2 * sigma²). Budget consumed is epsilon-determined. + // For Laplace, higher sensitivity means a larger noise scale b = sensitivity/epsilon, + // which yields LOWER RDP divergence (more noise → better privacy). + // Test the Laplace case: higher sensitivity at same epsilon costs less budget. RDPAccountant acc1 = new RDPAccountant(100.0, 1e-5); RDPAccountant acc2 = new RDPAccountant(100.0, 1e-5); - acc1.compose(0.5, 1e-5, 0.1); // low sensitivity - acc2.compose(0.5, 1e-5, 1.0); // high sensitivity + acc1.compose(0.5, 0.0, 1.0); // high sensitivity, Laplace + acc2.compose(0.5, 0.0, 0.1); // low sensitivity, Laplace - assertTrue("Lower sensitivity must cost less budget", + assertTrue("Higher sensitivity costs less budget (Laplace: more noise for same epsilon)", acc1.totalEpsilonSpent() < acc2.totalEpsilonSpent()); } From 7dc493c87b4e1c8a7c6505969e44f1b83651ad10 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Thu, 2 Jul 2026 14:34:36 +0200 Subject: [PATCH 05/26] CLAUDE_CODE_PLAN.md --- CLAUDE_CODE_PLAN.md | 364 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 364 insertions(+) create mode 100755 CLAUDE_CODE_PLAN.md diff --git a/CLAUDE_CODE_PLAN.md b/CLAUDE_CODE_PLAN.md new file mode 100755 index 00000000000..c08b79e80e4 --- /dev/null +++ b/CLAUDE_CODE_PLAN.md @@ -0,0 +1,364 @@ +# Claude Code Plan: Differential-Privacy Built-ins for Apache SystemDS + +## Goal +Add `dp_laplace` and `dp_gaussian` as native (non-script) DML built-in +functions backed by a session-scoped Rényi-DP budget accountant. + +This plan is written for Claude Code. Follow the steps in order. Read each +file before editing it. Do not guess at class names, field names, or method +signatures — grep to verify every assumption before writing code. + +--- + +## Step 1 — Orient: understand the instruction routing mechanism + +```bash +# 1a. Find how opcodes are mapped to CPInstruction subclasses. +# We are looking for whatever replaces (or still is) the opcode→type map. +grep -rn "parseSingleInstruction\|CPType\|CPInstructionParser" \ + src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java \ + | head -60 +``` + +```bash +# 1b. Look for how a recently-added native instruction (e.g. lstm, compress) +# is wired in. This gives us the exact pattern to copy. +grep -n "lstm\|compress\|bias_add" \ + src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java \ + | head -30 +``` + +```bash +# 1c. Read the full parseSingleInstruction method to understand the switch/map +# dispatch that leads to parseInstruction() on a specific class. +grep -n "parseSingleInstruction\|case Dnn\|case Builtin\|DnnCPInstruction\ +\|ParameterizedBuiltin\|parseInstruction" \ + src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java \ + | head -60 +``` + +> **Note for Claude Code**: the exact field/method name may differ from +> `String2CPInstructionType`. Use the output of step 1b to find the real +> registration pattern. Copy it exactly — do not invent names. + +--- + +## Step 2 — Orient: understand the Builtins enum constructor + +```bash +# 2a. Read the constructor and the first ~60 enum entries to confirm the +# exact parameter signature (name, script) vs (name, script, ReturnType). +sed -n '35,100p' \ + src/main/java/org/apache/sysds/common/Builtins.java +``` + +```bash +# 2b. Confirm no "parameterized" field exists in the constructor. +grep -n "parameterized\|Parameterized\|boolean" \ + src/main/java/org/apache/sysds/common/Builtins.java | head -20 +``` + +> Expected: constructor is `(String name, boolean script)` with `script=false` +> for native built-ins. Verify before proceeding. + +--- + +## Step 3 — Orient: understand BuiltinFunctionExpression validation + +```bash +# 3a. Find how an existing similar native built-in (e.g. colMeans, abs) +# is validated in the parser and how it maps to a HOP. +grep -n "COLMEAN\|ABS\|BuiltinFunctionExpression\|case COLMEAN\|case ABS" \ + src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java \ + | head -30 +``` + +```bash +# 3b. Find DMLTranslator to understand how BuiltinFunctionExpression +# creates a HOP node. +grep -n "createBuiltin\|BuiltinOp\|UnaryOp\|colMeans\|case COLMEAN" \ + src/main/java/org/apache/sysds/parser/DMLTranslator.java \ + | head -20 +``` + +```bash +# 3c. Find how the HOP emits a LOP that becomes a CPInstruction opcode string. +grep -n "getOpCode\|getLops\|addLop\|Lops" \ + src/main/java/org/apache/sysds/hops/UnaryOp.java \ + | head -20 +``` + +> This tells us whether we need a new HOP type, a new LOP type, or whether +> dp_laplace/dp_gaussian can reuse an existing HOP+LOP path. + +--- + +## Step 4 — Create the new files + +Create the following files. All paths are relative to the repository root. + +### 4a. RDPAccountant + +**File**: `src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java` + +Verify the package declaration matches the target path: +```bash +head -5 src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java +``` + +### 4b. DPBuiltinCPInstruction + +**File**: `src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java` + +Verify the imports compile against the actual codebase: + +```bash +# Confirm BinaryOperator and Plus exist at the expected paths. +find src/main/java -name "BinaryOperator.java" -o -name "Plus.java" | head -5 + +# Confirm MatrixBlock.binaryOperations signature. +grep -n "binaryOperations" \ + src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java \ + | head -10 +``` + +If `MatrixBlock.binaryOperations` has a different signature, update the call +in `processInstruction` to match. + +--- + +## Step 5 — Patch ExecutionContext to carry the accountant + +```bash +# 5a. Read the end of ExecutionContext to find where to add the new field. +grep -n "class ExecutionContext\|private.*Map\|private.*List\|getMatrix\ +\|releaseMatrix\|setScalar" \ + src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java \ + | tail -40 +``` + +Add the following to `ExecutionContext.java`: + +```java +// --- DP budget accountant (lazy-initialised, one per script execution) --- +private RDPAccountant _rdpAccountant = null; + +public RDPAccountant getRDPAccountant() { + if (_rdpAccountant == null) + // Default budget: ε=1.0, δ=1e-5. Future work: set via DML built-in. + _rdpAccountant = new RDPAccountant(1.0, 1e-5); + return _rdpAccountant; +} +``` + +Add the import at the top of `ExecutionContext.java`: +```java +import org.apache.sysds.runtime.privacy.dp.RDPAccountant; +``` + +--- + +## Step 6 — Register in Builtins.java + +```bash +# 6a. Find a good alphabetical insertion point between "D" entries. +grep -n "^[[:space:]]*D[A-Z_]*(" \ + src/main/java/org/apache/sysds/common/Builtins.java | head -20 +``` + +Insert after the last `D`-prefixed entry (or before the first `E` entry): + +```java +DP_LAPLACE("dp_laplace", false), +DP_GAUSSIAN("dp_gaussian", false), +``` + +Confirm `script=false` is correct by checking a nearby native built-in: +```bash +grep -A1 "DIAG\|DECOMPRESS\|DET" \ + src/main/java/org/apache/sysds/common/Builtins.java | head -10 +``` + +--- + +## Step 7 — Wire into the parser + +Use the routing pattern discovered in Step 1. The pattern will be one of: + +**Pattern A — opcode map + switch** (most likely based on commit history): +```bash +# Find the CPType enum to add a new entry if needed. +grep -n "enum CPType\|Dnn,\|BuiltinNary,\|ParameterizedBuiltin," \ + src/main/java/org/apache/sysds/runtime/instructions/cp/CPInstruction.java \ + | head -20 +``` + +If a new `CPType.DPBuiltin` is needed, add it to the `CPType` enum, then add +the map entry and switch case following the exact pattern of `CPType.Dnn`. + +**Pattern B — direct opcode string match in parseSingleInstruction**: +Add a branch: +```java +else if (opcode.equals("dp_laplace") || opcode.equals("dp_gaussian")) + return DPBuiltinCPInstruction.parseInstruction(str); +``` + +> Use whichever pattern the codebase actually uses. Do not mix patterns. + +--- + +## Step 8 — Wire into BuiltinFunctionExpression + +```bash +# 8a. Find the validate() switch to add parameter checking. +grep -n "case DIAG\|case ABS\|case COLMEAN\|checkNumParameters\|checkMatrixParam" \ + src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java \ + | head -20 +``` + +Add cases for `DP_LAPLACE` and `DP_GAUSSIAN` in the `validate()` switch: + +```java +case DP_LAPLACE: { + // dp_laplace(aggregate, sensitivity, epsilon) + checkNumParameters(3); + checkMatrixParam(getFirstExpr()); // aggregate matrix + checkScalarParam(getSecondExpr()); // sensitivity + checkScalarParam(getThirdExpr()); // epsilon + output.setDataType(DataType.MATRIX); + output.setValueType(ValueType.FP64); + // Output shape matches input shape; dimensions copied from input. + output.setDimensions( + getFirstExpr().getOutput().getDim1(), + getFirstExpr().getOutput().getDim2()); + break; +} +case DP_GAUSSIAN: { + // dp_gaussian(aggregate, sensitivity, epsilon, delta) + checkNumParameters(4); + checkMatrixParam(getFirstExpr()); + checkScalarParam(getSecondExpr()); // sensitivity + checkScalarParam(getThirdExpr()); // epsilon + checkScalarParam(getFourthExpr()); // delta + output.setDataType(DataType.MATRIX); + output.setValueType(ValueType.FP64); + output.setDimensions( + getFirstExpr().getOutput().getDim1(), + getFirstExpr().getOutput().getDim2()); + break; +} +``` + +> Verify that `checkScalarParam` and `getFourthExpr` exist: +> ```bash +> grep -n "checkScalarParam\|getFourthExpr\|getThirdExpr" \ +> src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java \ +> | head -10 +> ``` + +--- + +## Step 9 — Wire into DMLTranslator + +```bash +# 9a. Find how colMeans (COLMEAN) translates to a HOP in DMLTranslator. +grep -n "COLMEAN\|case COLMEAN\|createBuiltinOp\|UnaryOp" \ + src/main/java/org/apache/sysds/parser/DMLTranslator.java \ + | head -20 +``` + +```bash +# 9b. Find where to add new cases — likely inside a large switch on Builtins. +grep -n "case ABS\|case DIAG\|case CEIL" \ + src/main/java/org/apache/sysds/parser/DMLTranslator.java | head -10 +``` + +The simplest approach: map `DP_LAPLACE` and `DP_GAUSSIAN` to a `UnaryOp` HOP +with a custom opcode string. The LOP produced by `UnaryOp` will carry the +opcode string `"dp_laplace"` or `"dp_gaussian"`, which the `CPInstructionParser` +will then route to `DPBuiltinCPInstruction.parseInstruction`. + +Add cases following the existing unary pattern: +```java +case DP_LAPLACE: +case DP_GAUSSIAN: + // Reuse UnaryOp HOP — the opcode string routes to DPBuiltinCPInstruction. + currBuiltinOp = new UnaryOp(target.getName(), DataType.MATRIX, + ValueType.FP64, OpOp1.valueOf(bi.name()), expr); + break; +``` + +> Verify `OpOp1` has a compatible entry or whether a different HOP class is +> needed. If `OpOp1` does not work, fall back to creating a `FunctionOp`. + +--- + +## Step 10 — Build and smoke test + +```bash +# Build only the affected modules to get fast feedback. +mvn compile -pl src/main/java -am -q 2>&1 | tail -30 +``` + +Fix any compilation errors before proceeding. + +```bash +# Run the self-contained unit tests (no SystemDS runtime needed). +mvn test -pl src/test/java \ + -Dtest=DPBuiltinCPInstructionTest \ + -Dsurefire.failIfNoSpecifiedTests=false \ + 2>&1 | tail -40 +``` + +```bash +# Smoke-test end-to-end with a minimal DML script. +cat > /tmp/dp_smoke.dml << 'EOF' +X = rand(rows=100, cols=10, min=0, max=1); +mu = colMeans(X); +noisy = dp_laplace(mu, sensitivity=0.1, epsilon=1.0); +print(toString(noisy)); +EOF + +./bin/systemds /tmp/dp_smoke.dml 2>&1 | tail -20 +``` + +If the smoke test fails with an opcode-not-found error, re-check Steps 7 and 9. +If it fails with a budget error, reduce the sensitivity or widen epsilon. + +--- + +## Step 11 — Run the federated benchmark + +```bash +cat > /tmp/dp_fedavg_benchmark.dml << 'EOF' +# Federated averaging with DP release of column means. +# Sweep epsilon across {0.5, 1, 4, 8} by passing $epsilon as an arg. +X = read($1); +mu = colMeans(X); +noisy = dp_gaussian(mu, sensitivity=$sensitivity, epsilon=$epsilon, delta=1e-5); +write(noisy, $2, format="csv"); +EOF + +for eps in 0.5 1 4 8; do + echo "--- epsilon=$eps ---" + ./bin/systemds /tmp/dp_fedavg_benchmark.dml \ + -args data/adult.csv /tmp/result_eps${eps}.csv \ + -nvargs sensitivity=0.01 epsilon=$eps \ + 2>&1 | tail -5 +done +``` + +--- + +## Files modified (summary) + +| File | Change | +|---|---| +| `src/main/java/org/apache/sysds/common/Builtins.java` | Add `DP_LAPLACE`, `DP_GAUSSIAN` entries | +| `src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java` | Add validate cases | +| `src/main/java/org/apache/sysds/parser/DMLTranslator.java` | Add HOP-creation cases | +| `src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java` | Register opcodes using actual routing pattern | +| `src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java` | **New file** | +| `src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java` | **New file** | +| `src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java` | Add `getRDPAccountant()` | +| `src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinCPInstructionTest.java` | **New file** (unit tests) | From bd7a9563c8b82304ccf69be2bdb0f25db7c7c9c4 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Thu, 2 Jul 2026 14:39:38 +0200 Subject: [PATCH 06/26] Fix test package name. --- src/main/java/org/apache/sysds/common/Types.java | 3 ++- .../sysds/test/component/cp/DPBuiltinCPInstructionTest.java | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/main/java/org/apache/sysds/common/Types.java b/src/main/java/org/apache/sysds/common/Types.java index 250fc45a5c7..611d5011fbb 100644 --- a/src/main/java/org/apache/sysds/common/Types.java +++ b/src/main/java/org/apache/sysds/common/Types.java @@ -806,7 +806,8 @@ public static ReOrgOp valueOfByOpcode(String opcode) { /** Parameterized operations that require named variable arguments */ public enum ParamBuiltinOp { - AUTODIFF, CDF, CONTAINS, DP_LAPLACE, DP_GAUSSIAN, INVALID, INVCDF, GROUPEDAGG, RMEMPTY, REPLACE, REXPAND, + AUTODIFF, CDF, CONTAINS, DP_LAPLACE, DP_GAUSSIAN, INVALID, INVCDF, + GROUPEDAGG, RMEMPTY, REPLACE, REXPAND, LOWER_TRI, UPPER_TRI, TRANSFORMAPPLY, TRANSFORMDECODE, TRANSFORMCOLMAP, TRANSFORMMETA, TOKENIZE, TOSTRING, LIST, PARAMSERV diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index 6363b925b99..68b40d4d170 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -17,7 +17,7 @@ * under the License. */ -package org.apache.sysds.test.functions.privacy.dp; +package org.apache.sysds.test.component.cp; import org.apache.sysds.runtime.DMLRuntimeException; import org.apache.sysds.runtime.privacy.dp.RDPAccountant; From 5d31fdaa855e3aac46c0d1aad6b619a81f39f1df Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Mon, 6 Jul 2026 11:12:59 +0200 Subject: [PATCH 07/26] Fix Laplace accountant --- .../runtime/privacy/dp/RDPAccountant.java | 88 ++++++++++--------- .../cp/DPBuiltinCPInstructionTest.java | 18 ++-- 2 files changed, 53 insertions(+), 53 deletions(-) diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java index 884012e4ee6..98d87e3a9e4 100755 --- a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java @@ -94,9 +94,19 @@ public class RDPAccountant { // State // ----------------------------------------------------------------------- - /** Running accumulated Rényi divergence at each order. */ + /** Running accumulated Rényi divergence at each order (Gaussian releases only). */ private final double[] _rdpSum = new double[ORDERS.length]; + /** + * Running sum of pure ε from Laplace releases. + * + * Laplace gives pure ε-DP (no δ). Basic composition is exact and tighter + * than the RDP-to-(ε,δ) conversion path for Laplace, which introduces an + * unneeded δ and produces a looser bound. We accumulate Laplace cost here + * and add it directly in {@link #totalEpsilonSpent()}. + */ + private double _pureEpsilonSum = 0.0; + /** User-specified total privacy budget (ε). */ private final double _epsilonBudget; @@ -164,19 +174,20 @@ public RDPAccountant(double epsilonBudget) { public void compose(double epsilon, double delta, double sensitivity) { _releaseCount++; - // Accumulate Rényi divergence at each order. - for (int i = 0; i < ORDERS.length; i++) { - double alpha = ORDERS[i]; - double rdp; - if (delta == 0.0) { - rdp = rdpLaplace(alpha, sensitivity, epsilon); - } else { - // Back-derive σ from the (ε, δ) calibration formula for the - // Gaussian mechanism, then compute the RDP contribution. + if (delta == 0.0) { + // Laplace mechanism: pure ε-DP. Basic composition is exact and + // tighter than converting through RDP (which would introduce an + // unnecessary δ and often produce a looser ε bound). + _pureEpsilonSum += epsilon; + } else { + // Gaussian mechanism: accumulate Rényi divergence at each order. + // Back-derive σ from the (ε, δ) calibration formula, then add the + // RDP contribution. + for (int i = 0; i < ORDERS.length; i++) { + double alpha = ORDERS[i]; double sigma = gaussianSigma(sensitivity, epsilon, delta); - rdp = rdpGaussian(alpha, sensitivity, sigma); + _rdpSum[i] += rdpGaussian(alpha, sensitivity, sigma); } - _rdpSum[i] += rdp; } // Convert accumulated RDP to (ε, δ) and check. @@ -195,23 +206,36 @@ public void compose(double epsilon, double delta, double sensitivity) { // ----------------------------------------------------------------------- /** - * Returns the current total privacy cost as an ε value under the - * accountant's δ, using the tightest available Rényi order. + * Returns the current total privacy cost as an ε value. + * + *

The total cost combines two independent composition paths: + *

    + *
  • Laplace releases: pure ε-DP, accumulated via basic composition + * (exact and tighter than the RDP conversion path for Laplace).
    • + *
  • Gaussian releases: accumulated via Rényi DP, then converted to + * (ε, δ) using the tightest available order α.
    • + *
+ * + *

The combined guarantee is (ε_total, δ)-DP where ε_total is the sum + * of the two contributions (basic composition of a pure-DP mechanism with + * an approximate-DP mechanism is additive in ε). */ public double totalEpsilonSpent() { - double minEpsilon = Double.MAX_VALUE; + // Gaussian contribution via RDP → (ε, δ) conversion (Mironov 2017, Prop. 3). + double gaussianEps = Double.MAX_VALUE; for (int i = 0; i < ORDERS.length; i++) { double alpha = ORDERS[i]; - // Standard RDP-to-(ε,δ) conversion: - // ε(α) = R[α] + log(1 - 1/α) - log(δ·(α-1)) / α - // Reference: Mironov 2017, Proposition 3. double eps = _rdpSum[i] + Math.log(1.0 - 1.0 / alpha) - Math.log(_delta * (alpha - 1.0)) / alpha; - if (eps < minEpsilon) - minEpsilon = eps; + if (eps < gaussianEps) + gaussianEps = eps; } - return minEpsilon; + // If no Gaussian releases have occurred, the RDP conversion yields a + // large positive value (log-delta term dominates). Clamp to zero so + // it doesn't inflate the total when only Laplace releases are present. + if (gaussianEps < 0) gaussianEps = 0.0; + return _pureEpsilonSum + gaussianEps; } /** Returns the remaining ε budget (may be negative if budget is exceeded). */ @@ -228,28 +252,6 @@ public int releaseCount() { // Mechanism-specific RDP contributions // ----------------------------------------------------------------------- - /** - * Rényi divergence of order α for the Laplace mechanism with scale - * b = sensitivity / epsilon. - * - *

For α > 1 and integer α, the closed form is: - * 

-     *   D_α = (1/(α-1)) · ln( α/(2α-1)·exp((α-1)/b) + (α-1)/(2α-1)·exp(-α/b) )
-     *
- * - *

For non-integer α we use the same formula, which is the natural - * analytic continuation (see Mironov 2017, Proposition 3, example 1). - * We clamp the log argument to avoid NaN when inputs are degenerate. - */ - private static double rdpLaplace(double alpha, double sensitivity, double epsilon) { - double b = sensitivity / epsilon; // Laplace scale - double t1 = alpha / (2.0 * alpha - 1.0) * Math.exp((alpha - 1.0) / b); - double t2 = (alpha - 1.0) / (2.0 * alpha - 1.0) * Math.exp(-alpha / b); - double arg = t1 + t2; - if (arg <= 0) return 0.0; // degenerate: treat as zero cost - return Math.log(arg) / (alpha - 1.0); - } - /** * Rényi divergence of order α for the Gaussian mechanism with noise * scale σ and L2 sensitivity Δf. diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index 68b40d4d170..fdeac21ff1b 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -141,19 +141,17 @@ public void testRemainingBudgetDecreasesMonotonically() { } @Test - public void testSmallerSensitivityCheaper() { - // For the Gaussian mechanism, noise scales proportionally with sensitivity - // (sigma = sensitivity * C), so sensitivity² cancels in the RDP formula - // alpha * sensitivity² / (2 * sigma²). Budget consumed is epsilon-determined. - // For Laplace, higher sensitivity means a larger noise scale b = sensitivity/epsilon, - // which yields LOWER RDP divergence (more noise → better privacy). - // Test the Laplace case: higher sensitivity at same epsilon costs less budget. + public void testHigherEpsilonCostMoreForLaplace() { + // For Laplace, the accountant uses basic (pure ε-DP) composition: cost = epsilon. + // Sensitivity determines noise scale but NOT the budget consumed — that is set + // entirely by the caller's epsilon parameter. + // A release at epsilon=1.0 costs more budget than one at epsilon=0.5. RDPAccountant acc1 = new RDPAccountant(100.0, 1e-5); RDPAccountant acc2 = new RDPAccountant(100.0, 1e-5); - acc1.compose(0.5, 0.0, 1.0); // high sensitivity, Laplace - acc2.compose(0.5, 0.0, 0.1); // low sensitivity, Laplace + acc1.compose(0.5, 0.0, 1.0); // epsilon=0.5, Laplace + acc2.compose(1.0, 0.0, 1.0); // epsilon=1.0, same sensitivity - assertTrue("Higher sensitivity costs less budget (Laplace: more noise for same epsilon)", + assertTrue("Higher epsilon costs more budget (Laplace basic composition)", acc1.totalEpsilonSpent() < acc2.totalEpsilonSpent()); } From 4b58bc402b2f49fab1444140bcc4f58d6f14e9b6 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Mon, 6 Jul 2026 17:22:33 +0200 Subject: [PATCH 08/26] Add unit tests --- .../cp/DPBuiltinCPInstructionTest.java | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index fdeac21ff1b..c04a901e650 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -155,6 +155,122 @@ public void testHigherEpsilonCostMoreForLaplace() { acc1.totalEpsilonSpent() < acc2.totalEpsilonSpent()); } + // --- Item 1: constructor error paths ------------------------------------ + + @Test(expected = DMLRuntimeException.class) + public void testConstructorRejectsZeroEpsilonBudget() { + new RDPAccountant(0.0, 1e-5); + } + + @Test(expected = DMLRuntimeException.class) + public void testConstructorRejectsNegativeEpsilonBudget() { + new RDPAccountant(-0.5, 1e-5); + } + + @Test(expected = DMLRuntimeException.class) + public void testConstructorRejectsDeltaZero() { + new RDPAccountant(1.0, 0.0); + } + + @Test(expected = DMLRuntimeException.class) + public void testConstructorRejectsDeltaOne() { + new RDPAccountant(1.0, 1.0); + } + + // --- Item 2: single-argument convenience constructor ------------------- + + @Test + public void testConvenienceConstructorDefaultsDeltaTo1e5() { + // The one-arg form delegates to (epsilonBudget, 1e-5). A Gaussian + // release whose per-release delta matches that default must produce + // identical totalEpsilonSpent() from both construction paths. + RDPAccountant oneArg = new RDPAccountant(10.0); + RDPAccountant twoArg = new RDPAccountant(10.0, 1e-5); + oneArg.compose(0.5, 1e-5, 1.0); + twoArg.compose(0.5, 1e-5, 1.0); + assertEquals("Convenience constructor must default to delta=1e-5", + twoArg.totalEpsilonSpent(), oneArg.totalEpsilonSpent(), EPS); + } + + // --- Item 3: budget exhaustion via Gaussian releases ------------------- + + @Test(expected = DMLRuntimeException.class) + public void testGaussianBudgetExhaustionThrows() { + // Budget = 0.1. Each Gaussian release costs more than 0.005, so 20 + // releases must exceed the budget well before the loop ends. + RDPAccountant acc = new RDPAccountant(0.1, 1e-5); + for (int i = 0; i < 20; i++) { + acc.compose(0.3, 1e-5, 1.0); + } + } + + // --- Item 4: mixed Laplace + Gaussian composition ---------------------- + + @Test + public void testMixedCompositionExceedsEitherAlone() { + // Compose one Laplace and one Gaussian release. The total cost must + // exceed what either mechanism contributes alone, exercising the + // _pureEpsilonSum + gaussianEps addition path in totalEpsilonSpent(). + RDPAccountant mixed = new RDPAccountant(100.0, 1e-5); + RDPAccountant lapOnly = new RDPAccountant(100.0, 1e-5); + RDPAccountant gauOnly = new RDPAccountant(100.0, 1e-5); + + mixed.compose(0.5, 0.0, 1.0); // Laplace + mixed.compose(0.5, 1e-5, 1.0); // Gaussian + + lapOnly.compose(0.5, 0.0, 1.0); + gauOnly.compose(0.5, 1e-5, 1.0); + + assertTrue("Mixed cost must exceed Laplace-only cost", + mixed.totalEpsilonSpent() > lapOnly.totalEpsilonSpent()); + assertTrue("Mixed cost must exceed Gaussian-only cost", + mixed.totalEpsilonSpent() > gauOnly.totalEpsilonSpent()); + } + + // --- Item 6: release count across multiple mixed releases -------------- + + @Test + public void testReleaseCountTracksAllReleases() { + RDPAccountant acc = new RDPAccountant(100.0, 1e-5); + assertEquals(0, acc.releaseCount()); + acc.compose(0.1, 0.0, 1.0); // Laplace + assertEquals(1, acc.releaseCount()); + acc.compose(0.1, 1e-5, 1.0); // Gaussian + assertEquals(2, acc.releaseCount()); + acc.compose(0.1, 0.0, 1.0); // Laplace + acc.compose(0.1, 0.0, 1.0); // Laplace + acc.compose(0.1, 1e-5, 1.0); // Gaussian + assertEquals(5, acc.releaseCount()); + } + + // --- Item 8: edge-case inputs for rdpGaussian / gaussianSigma ---------- + + @Test + public void testGaussianSensitivityCancelsInRDP() { + // For the Gaussian mechanism: σ = Δf·sqrt(2·ln(1.25/δ))/ε, so + // D_α = α·Δf²/(2σ²) = α·ε²/(4·ln(1.25/δ)). + // Sensitivity cancels. Two accountants with the same (ε,δ) but + // different sensitivity must report identical totalEpsilonSpent(). + RDPAccountant acc1 = new RDPAccountant(100.0, 1e-5); + RDPAccountant acc2 = new RDPAccountant(100.0, 1e-5); + acc1.compose(0.5, 1e-5, 1.0); // sensitivity = 1 + acc2.compose(0.5, 1e-5, 100.0); // sensitivity = 100, same (ε,δ) + assertEquals("Gaussian RDP cost must be independent of sensitivity when (ε,δ) are fixed", + acc1.totalEpsilonSpent(), acc2.totalEpsilonSpent(), EPS); + } + + @Test + public void testGaussianLargerEpsilonCostsMoreBudget() { + // D_α ∝ ε², so a release declared at a higher ε (less noise, more + // privacy loss) must cost more budget than one at a lower ε. + RDPAccountant lowEps = new RDPAccountant(100.0, 1e-5); + RDPAccountant highEps = new RDPAccountant(100.0, 1e-5); + lowEps.compose(0.1, 1e-5, 1.0); + highEps.compose(0.5, 1e-5, 1.0); + assertTrue("Larger epsilon per Gaussian release must cost more budget", + highEps.totalEpsilonSpent() > lowEps.totalEpsilonSpent()); + } + // ======================================================================= // 2. Noise distribution tests (statistical sanity checks) // ======================================================================= From 29369700fddc63b2c30d1ef129e3c01e7643b80a Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Mon, 6 Jul 2026 17:27:44 +0200 Subject: [PATCH 09/26] Rename RDPAccountant to DPBudgetAccountant --- .../context/ExecutionContext.java | 12 +- .../cp/DPBuiltinCPInstruction.java | 14 +- .../privacy/dp/DPBudgetAccountant.java | 271 ++++++++++++++++++ .../runtime/privacy/dp/RDPAccountant.java | 264 +---------------- .../cp/DPBuiltinCPInstructionTest.java | 58 ++-- 5 files changed, 320 insertions(+), 299 deletions(-) create mode 100644 src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java index baa6e74df7d..eaf50da88c7 100644 --- a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java +++ b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java @@ -62,7 +62,7 @@ import org.apache.sysds.runtime.meta.MetaData; import org.apache.sysds.runtime.meta.MetaDataFormat; import org.apache.sysds.runtime.util.HDFSTool; -import org.apache.sysds.runtime.privacy.dp.RDPAccountant; +import org.apache.sysds.runtime.privacy.dp.DPBudgetAccountant; import org.apache.sysds.utils.Statistics; import java.util.ArrayList; @@ -91,7 +91,7 @@ public class ExecutionContext { protected SEALClient _seal_client; - private RDPAccountant _rdpAccountant = null; + private DPBudgetAccountant _dpBudgetAccountant = null; //parfor temporary functions (created by eval) protected Set _fnNames; @@ -147,10 +147,10 @@ public void setLineage(Lineage lineage) { _lineage = lineage; } - public RDPAccountant getRDPAccountant() { - if (_rdpAccountant == null) - _rdpAccountant = new RDPAccountant(1.0, 1e-5); - return _rdpAccountant; + public DPBudgetAccountant getDPBudgetAccountant() { + if (_dpBudgetAccountant == null) + _dpBudgetAccountant = new DPBudgetAccountant(1.0, 1e-5); + return _dpBudgetAccountant; } public boolean isAutoCreateVars() { diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java index 746c8471ff8..be02be1cae7 100755 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -25,7 +25,7 @@ import org.apache.sysds.runtime.instructions.InstructionUtils; import org.apache.sysds.runtime.matrix.data.MatrixBlock; import org.apache.sysds.runtime.matrix.operators.BinaryOperator; -import org.apache.sysds.runtime.privacy.dp.RDPAccountant; +import org.apache.sysds.runtime.privacy.dp.DPBudgetAccountant; import java.util.LinkedHashMap; import java.util.concurrent.ThreadLocalRandom; @@ -42,7 +42,7 @@ * *

The instruction receives a materialised matrix (the aggregate result), * injects calibrated noise element-wise, records the release with the - * session-scoped {@link RDPAccountant}, and returns the noisy matrix. + * session-scoped {@link DPBudgetAccountant}, and returns the noisy matrix. * *

Noise is generated in Java and added via a {@code MatrixBlock} binary * operation so that the output allocation path is identical to every other @@ -64,8 +64,8 @@ * add opcode-to-type mappings and a parse branch that returns a * {@code DPBuiltinCPInstruction} *

  • {@code org.apache.sysds.runtime.controlprogram.context.ExecutionContext} - * – add {@code getRDPAccountant()} returning a session-scoped - * {@link RDPAccountant} (lazy-initialised field)
    • + * – add {@code getDPBudgetAccountant()} returning a session-scoped + * {@link DPBudgetAccountant} (lazy-initialised field) * */ public class DPBuiltinCPInstruction extends ComputationCPInstruction { @@ -169,7 +169,7 @@ public static DPBuiltinCPInstruction parseInstruction(String str) { *
  • Generate a noise {@link MatrixBlock} of the same shape.
    • *
  • Add noise element-wise using the existing binary-operator path.
    • *
  • Record the release with the session-scoped - * {@link RDPAccountant}; throw if budget is exhausted.
    • + * {@link DPBudgetAccountant}; throw if budget is exhausted. *
  • Write the noisy block back to the variable table and release * the input pin.
    • * @@ -200,9 +200,9 @@ public void processInstruction(ExecutionContext ec) { inBlock.binaryOperations(plusOp, noiseBlock, outBlock); // ── 5. Record release and enforce budget ──────────────────────────── - // getRDPAccountant() returns a lazy-initialised RDPAccountant that is + // getDPBudgetAccountant() returns a lazy-initialised DPBudgetAccountant that is // owned by this ExecutionContext (added in a companion EC patch). - RDPAccountant accountant = ec.getRDPAccountant(); + DPBudgetAccountant accountant = ec.getDPBudgetAccountant(); accountant.compose(epsilon, delta, sensitivity); // throws on exhaustion // ── 6. Write output and release input pin ─────────────────────────── diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java new file mode 100644 index 00000000000..ce5afcb7850 --- /dev/null +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java @@ -0,0 +1,271 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.sysds.runtime.privacy.dp; + +import org.apache.sysds.runtime.DMLRuntimeException; + +/** + * Session-scoped differential privacy budget accountant. + * + *

    Purpose

    + * Tracks composition of DP releases across the lifetime of a DML script + * execution. Each call to {@link #compose} records one release and checks + * whether the cumulative privacy cost has exceeded the user-specified budget. + * + *

    Composition strategy

    + * The mechanism type (Laplace vs Gaussian) is inferred from the {@code delta} + * argument passed to {@link #compose}: + * + *
      + *
    • Laplace (delta == 0): pure ε-DP. The budget cost is tracked via + * basic composition — each release contributes exactly its ε to a running + * sum. This is the tightest possible bound for pure DP and avoids the + * looser estimate that results from routing Laplace through the RDP + * conversion path (which would introduce an unnecessary δ).
      • + *
    • Gaussian (delta > 0): (ε, δ)-DP via Rényi DP composition. + * Rényi divergences at a discrete set of orders α compose additively; + * the accumulated sum is converted to (ε, δ) at query time using the + * formula from Mironov 2017. This is substantially tighter than basic + * composition for repeated Gaussian releases, which is the common case + * in federated learning.
      • + *
    + * + *

    When both mechanisms are used in the same script the total cost is: + * 

    + *   ε_total = ε_Laplace_sum + ε_Gaussian_RDP
+ *
    + * This follows from basic composition of a pure-DP mechanism with an + * approximate-DP mechanism, which is additive in ε. + * + *

    Rényi orders tracked (Gaussian path)

    + * α ∈ {2, 4, 8, 16, 32, 64, 128, 256, 512, 1024}. At query time the minimum + * converted ε across all orders is taken as the tightest available bound. + * + *

    Gaussian RDP divergence

    + * For the Gaussian mechanism with noise scale σ and L2 sensitivity Δf: + * 
    + *   D_α = α · Δf² / (2σ²)
+ *
    + * σ is back-derived from the caller's (ε, δ) via the standard calibration + * formula (see {@link #gaussianSigma}). Note that sensitivity cancels in the + * final expression, so the RDP cost depends only on the (ε, δ) parameters. + * + *

    RDP → (ε, δ) conversion (Mironov 2017, Proposition 3)

    + * 
    + *   ε(α) = R[α] + log(1 − 1/α) − log(δ·(α−1)) / α
+ *
    + * + *

    Lifecycle

    + * One instance is created per {@code ExecutionContext} (lazy init). It is + * garbage-collected with the context when the script finishes; no state + * leaks between script executions or between concurrent scripts. + * + *

    Thread safety

    + * Not thread-safe. A single DML script executes instructions sequentially + * on one thread, so no synchronisation is needed. + * + * @see DPBuiltinCPInstruction + */ +public class DPBudgetAccountant { + + // ----------------------------------------------------------------------- + // Rényi orders used for Gaussian composition + // ----------------------------------------------------------------------- + + /** + * Discrete set of Rényi orders α. All must be > 1. + * Finer grids give tighter bounds; this set covers the range relevant + * for typical ML workloads. + */ + private static final double[] ORDERS = { + 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 + }; + + // ----------------------------------------------------------------------- + // State + // ----------------------------------------------------------------------- + + /** Accumulated Rényi divergence at each order (Gaussian releases only). */ + private final double[] _rdpSum = new double[ORDERS.length]; + + /** + * Running sum of pure ε from Laplace releases. + * + *

    Laplace gives pure ε-DP (no δ). Basic composition is exact and + * tighter than the RDP conversion path for Laplace (which would introduce + * an unnecessary δ and produce a looser bound). Each Laplace release adds + * its ε here; the total is added directly in {@link #totalEpsilonSpent()}. + */ + private double _pureEpsilonSum = 0.0; + + /** Total privacy budget (ε) for the script execution. */ + private final double _epsilonBudget; + + /** δ used for the Gaussian RDP-to-(ε,δ) conversion. */ + private final double _delta; + + /** Number of releases recorded so far (for error messages). */ + private int _releaseCount = 0; + + // ----------------------------------------------------------------------- + // Constructors + // ----------------------------------------------------------------------- + + /** + * Creates an accountant with the given global budget. + * + *

    Typical usage: the DML script sets the budget once at the top + * (future work: a {@code dp_set_budget(epsilon, delta)} built-in), + * or the accountant is created with defaults and the budget is checked + * after each release. + * + * @param epsilonBudget total ε budget for the script execution (must be > 0) + * @param delta δ used for the Gaussian RDP-to-(ε,δ) conversion (must be in (0,1)) + */ + public DPBudgetAccountant(double epsilonBudget, double delta) { + if (!(epsilonBudget > 0)) + throw new DMLRuntimeException( + "DPBudgetAccountant: epsilonBudget must be > 0, got " + epsilonBudget); + if (!(delta > 0 && delta < 1)) + throw new DMLRuntimeException( + "DPBudgetAccountant: delta must be in (0,1), got " + delta); + _epsilonBudget = epsilonBudget; + _delta = delta; + } + + /** + * Convenience constructor using a liberal default δ = 1e-5. + * Suitable when the calling script does not specify δ explicitly. + */ + public DPBudgetAccountant(double epsilonBudget) { + this(epsilonBudget, 1e-5); + } + + // ----------------------------------------------------------------------- + // Core API + // ----------------------------------------------------------------------- + + /** + * Records one DP release and checks the budget. + * + *

    This method must be called before the result is written to + * the variable table. If the budget is exhausted it throws and the + * caller's result is discarded, preventing an unaccounted release. + * + *

    Mechanism selection (see class-level Javadoc for details): + *

      + *
    • {@code delta == 0} → Laplace, pure ε-DP basic composition
      • + *
    • {@code delta > 0} → Gaussian, Rényi DP composition
      • + *
    + * + * @param epsilon per-release ε parameter (must be > 0) + * @param delta per-release δ parameter (0 for Laplace, >0 for Gaussian) + * @param sensitivity L2 sensitivity Δf of the released quantity (must be > 0) + * @throws DMLRuntimeException if the cumulative ε after this release + * would exceed the budget + */ + public void compose(double epsilon, double delta, double sensitivity) { + _releaseCount++; + + if (delta == 0.0) { + // Laplace: pure ε-DP, basic composition — cost is exactly epsilon. + _pureEpsilonSum += epsilon; + } else { + // Gaussian: accumulate Rényi divergence at each order, then convert. + for (int i = 0; i < ORDERS.length; i++) { + double sigma = gaussianSigma(sensitivity, epsilon, delta); + _rdpSum[i] += rdpGaussian(ORDERS[i], sensitivity, sigma); + } + } + + double spentEpsilon = totalEpsilonSpent(); + if (spentEpsilon > _epsilonBudget) { + throw new DMLRuntimeException(String.format( + "Privacy budget exhausted after %d release(s): " + + "spent ε ≈ %.6f exceeds budget ε = %.6f (δ = %.2e). " + + "Reduce the number of releases or widen the budget.", + _releaseCount, spentEpsilon, _epsilonBudget, _delta)); + } + } + + // ----------------------------------------------------------------------- + // Inspection + // ----------------------------------------------------------------------- + + /** + * Returns the current total privacy cost as an ε value. + * + *

    Total = Laplace pure-ε sum + Gaussian RDP-converted ε (clamped to + * zero when no Gaussian releases have been recorded). + */ + public double totalEpsilonSpent() { + double gaussianEps = Double.MAX_VALUE; + for (int i = 0; i < ORDERS.length; i++) { + double alpha = ORDERS[i]; + double eps = _rdpSum[i] + + Math.log(1.0 - 1.0 / alpha) + - Math.log(_delta * (alpha - 1.0)) / alpha; + if (eps < gaussianEps) + gaussianEps = eps; + } + // Clamp: with no Gaussian releases the RDP sum is 0 and the log-delta + // term alone drives gaussianEps to a small positive value; clamp to 0 + // so Laplace-only scripts are not penalised by δ they never requested. + if (gaussianEps < 0) gaussianEps = 0.0; + return _pureEpsilonSum + gaussianEps; + } + + /** Returns the remaining ε budget (negative if the budget is exceeded). */ + public double remainingBudget() { + return _epsilonBudget - totalEpsilonSpent(); + } + + /** Returns the number of DP releases recorded so far. */ + public int releaseCount() { + return _releaseCount; + } + + // ----------------------------------------------------------------------- + // Private helpers + // ----------------------------------------------------------------------- + + /** + * Rényi divergence of order α for the Gaussian mechanism (Mironov 2017, + * Proposition 3, example 2): + * 

    +     *   D_α = α · Δf² / (2σ²)
+     *
    + */ + private static double rdpGaussian(double alpha, double sensitivity, double sigma) { + return alpha * (sensitivity * sensitivity) / (2.0 * sigma * sigma); + } + + /** + * Gaussian noise scale σ calibrated to (ε, δ)-DP: + * 
    +     *   σ = Δf · sqrt(2 · ln(1.25 / δ)) / ε
+     *
    + * Must match the formula used in {@link DPBuiltinCPInstruction} so that + * the RDP cost recorded here is consistent with the noise actually injected. + */ + private static double gaussianSigma(double sensitivity, double epsilon, double delta) { + return sensitivity * Math.sqrt(2.0 * Math.log(1.25 / delta)) / epsilon; + } +} diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java index 98d87e3a9e4..0cd3785db7b 100755 --- a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java @@ -19,267 +19,17 @@ package org.apache.sysds.runtime.privacy.dp; -import org.apache.sysds.runtime.DMLRuntimeException; - /** - * Session-scoped Rényi Differential Privacy (RDP) budget accountant. - * - *

    Purpose

    - * Tracks composition of DP releases across the lifetime of a DML script - * execution. Each call to {@link #compose} records one release and checks - * whether the cumulative privacy cost has exceeded the user-specified budget. - * - *

    Why Rényi DP?

    - * Basic composition adds epsilons linearly, giving very loose bounds with - * many releases. Rényi DP divergences compose additively at the - * same order α. Converting the running Rényi sum to (ε, δ) via the standard - * conversion formula yields substantially tighter bounds — particularly for - * Gaussian mechanisms, which are common in federated learning. - * - *

    Orders tracked

    - * We track a discrete set of Rényi orders α ∈ {2, 4, 8, 16, 32, 64, 128, - * 256, 512, 1024}. At query time we take the minimum converted ε across all - * orders, which is the tightest available bound. - * - *

    Composition rules

    - * For the Gaussian mechanism with noise scale σ and sensitivity Δf, the - * Rényi divergence of order α between outputs on neighbouring datasets is: - * 
    - *   D_α = α · Δf² / (2σ²)
- *
    - * where σ is back-derived from the caller's (ε, δ) parameters via the - * standard calibration formula. See {@link #rdpGaussian} for details. - * - * For the Laplace mechanism with scale b = Δf/ε, the Rényi divergence at - * order α is: - * 
    - *   D_α = (1/(α-1)) · ln( α/(2α-1) · exp((α-1)/b) + (α-1)/(2α-1) · exp(-α/b) )
- *       (for α > 1; the limit as α → 1 is 1/b, i.e. the KL divergence)
- *
    - * - *

    Conversion: Rényi DP → (ε, δ)-DP

    - * Given accumulated Rényi divergence R[α] at order α and a target δ: - * 
    - *   ε(α) = R[α] + log(1 - 1/α) - log(δ · (α - 1)) / α
- *
    - * The reported total cost is min_α ε(α). - * - *

    Lifecycle

    - * One instance is created per {@code ExecutionContext} (lazy init). It is - * garbage-collected with the context when the script finishes; no state - * leaks between script executions or between concurrent scripts. - * - *

    Thread safety

    - * Not thread-safe. A single DML script executes instructions sequentially - * on one thread, so no synchronisation is needed. - * - * @see DPBuiltinCPInstruction + * @deprecated Use {@link DPBudgetAccountant} instead. */ -public class RDPAccountant { - - // ----------------------------------------------------------------------- - // Rényi orders to track - // ----------------------------------------------------------------------- - - /** - * Discrete set of Rényi orders α. All must be > 1. - * Finer grids give tighter bounds; this set is a reasonable default - * that covers the range relevant for typical ML workloads. - */ - private static final double[] ORDERS = { - 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 - }; - - // ----------------------------------------------------------------------- - // State - // ----------------------------------------------------------------------- - - /** Running accumulated Rényi divergence at each order (Gaussian releases only). */ - private final double[] _rdpSum = new double[ORDERS.length]; - - /** - * Running sum of pure ε from Laplace releases. - * - * Laplace gives pure ε-DP (no δ). Basic composition is exact and tighter - * than the RDP-to-(ε,δ) conversion path for Laplace, which introduces an - * unneeded δ and produces a looser bound. We accumulate Laplace cost here - * and add it directly in {@link #totalEpsilonSpent()}. - */ - private double _pureEpsilonSum = 0.0; - - /** User-specified total privacy budget (ε). */ - private final double _epsilonBudget; - - /** User-specified δ used for the RDP-to-(ε,δ) conversion. */ - private final double _delta; - - /** Number of releases recorded so far (for error messages). */ - private int _releaseCount = 0; - - // ----------------------------------------------------------------------- - // Constructor - // ----------------------------------------------------------------------- - - /** - * Creates an accountant with the given global budget. - * - *

    Typical usage: the DML script sets the budget once at the top - * (future work: a {@code dp_set_budget(epsilon, delta)} built-in), - * or the accountant is created with defaults and the budget is checked - * after each release. - * - * @param epsilonBudget total ε budget for the script execution (must be > 0) - * @param delta δ used for the RDP-to-(ε,δ) conversion (must be in (0,1)) - */ +@Deprecated +public class RDPAccountant extends DPBudgetAccountant { + @Deprecated public RDPAccountant(double epsilonBudget, double delta) { - if (!(epsilonBudget > 0)) - throw new DMLRuntimeException( - "RDPAccountant: epsilonBudget must be > 0, got " + epsilonBudget); - if (!(delta > 0 && delta < 1)) - throw new DMLRuntimeException( - "RDPAccountant: delta must be in (0,1), got " + delta); - _epsilonBudget = epsilonBudget; - _delta = delta; + super(epsilonBudget, delta); } - - /** - * Convenience constructor using a liberal default δ = 1e-5. - * Suitable when the calling script does not specify δ explicitly. - */ + @Deprecated public RDPAccountant(double epsilonBudget) { - this(epsilonBudget, 1e-5); - } - - // ----------------------------------------------------------------------- - // Core API - // ----------------------------------------------------------------------- - - /** - * Records one DP release and checks the budget. - * - *

    This method must be called before the result is written to - * the variable table. If the budget is exhausted, it throws and the - * caller's result is discarded, preventing an unaccounted release. - * - *

    The mechanism type (Laplace vs Gaussian) is inferred from the - * parameters: if {@code delta == 0} the release is treated as Laplace; - * otherwise it is treated as Gaussian. - * - * @param epsilon the ε parameter for this individual release (> 0) - * @param delta the δ parameter for this release (0 for Laplace) - * @param sensitivity the L2 sensitivity Δf of the released quantity (> 0) - * @throws DMLRuntimeException if the cumulative ε after this release - * would exceed the budget - */ - public void compose(double epsilon, double delta, double sensitivity) { - _releaseCount++; - - if (delta == 0.0) { - // Laplace mechanism: pure ε-DP. Basic composition is exact and - // tighter than converting through RDP (which would introduce an - // unnecessary δ and often produce a looser ε bound). - _pureEpsilonSum += epsilon; - } else { - // Gaussian mechanism: accumulate Rényi divergence at each order. - // Back-derive σ from the (ε, δ) calibration formula, then add the - // RDP contribution. - for (int i = 0; i < ORDERS.length; i++) { - double alpha = ORDERS[i]; - double sigma = gaussianSigma(sensitivity, epsilon, delta); - _rdpSum[i] += rdpGaussian(alpha, sensitivity, sigma); - } - } - - // Convert accumulated RDP to (ε, δ) and check. - double spentEpsilon = totalEpsilonSpent(); - if (spentEpsilon > _epsilonBudget) { - throw new DMLRuntimeException(String.format( - "Privacy budget exhausted after %d release(s): " - + "spent ε ≈ %.6f exceeds budget ε = %.6f (δ = %.2e). " - + "Reduce the number of releases or widen the budget.", - _releaseCount, spentEpsilon, _epsilonBudget, _delta)); - } - } - - // ----------------------------------------------------------------------- - // Inspection - // ----------------------------------------------------------------------- - - /** - * Returns the current total privacy cost as an ε value. - * - *

    The total cost combines two independent composition paths: - *

      - *
    • Laplace releases: pure ε-DP, accumulated via basic composition - * (exact and tighter than the RDP conversion path for Laplace).
      • - *
    • Gaussian releases: accumulated via Rényi DP, then converted to - * (ε, δ) using the tightest available order α.
      • - *
    - * - *

    The combined guarantee is (ε_total, δ)-DP where ε_total is the sum - * of the two contributions (basic composition of a pure-DP mechanism with - * an approximate-DP mechanism is additive in ε). - */ - public double totalEpsilonSpent() { - // Gaussian contribution via RDP → (ε, δ) conversion (Mironov 2017, Prop. 3). - double gaussianEps = Double.MAX_VALUE; - for (int i = 0; i < ORDERS.length; i++) { - double alpha = ORDERS[i]; - double eps = _rdpSum[i] - + Math.log(1.0 - 1.0 / alpha) - - Math.log(_delta * (alpha - 1.0)) / alpha; - if (eps < gaussianEps) - gaussianEps = eps; - } - // If no Gaussian releases have occurred, the RDP conversion yields a - // large positive value (log-delta term dominates). Clamp to zero so - // it doesn't inflate the total when only Laplace releases are present. - if (gaussianEps < 0) gaussianEps = 0.0; - return _pureEpsilonSum + gaussianEps; - } - - /** Returns the remaining ε budget (may be negative if budget is exceeded). */ - public double remainingBudget() { - return _epsilonBudget - totalEpsilonSpent(); - } - - /** Returns the number of DP releases recorded so far. */ - public int releaseCount() { - return _releaseCount; - } - - // ----------------------------------------------------------------------- - // Mechanism-specific RDP contributions - // ----------------------------------------------------------------------- - - /** - * Rényi divergence of order α for the Gaussian mechanism with noise - * scale σ and L2 sensitivity Δf. - * - *

    For α > 1: - * 

    -     *   D_α = α · Δf² / (2σ²)
-     *
    - * - *

    This is the standard result for the Gaussian mechanism (see - * Mironov 2017, Proposition 3, example 2). - */ - private static double rdpGaussian(double alpha, double sensitivity, double sigma) { - return alpha * (sensitivity * sensitivity) / (2.0 * sigma * sigma); - } - - /** - * Back-derives the Gaussian noise scale σ from the (ε, δ)-DP parameters - * using the standard calibration inequality: - * 

    -     *   σ = Δf · sqrt(2 · ln(1.25 / δ)) / ε
-     *
    - * - *

    This is the formula used by {@code DPBuiltinCPInstruction} to - * generate the actual noise, so the RDP contribution it records is - * exactly consistent with the noise injected. - */ - private static double gaussianSigma(double sensitivity, double epsilon, double delta) { - return sensitivity * Math.sqrt(2.0 * Math.log(1.25 / delta)) / epsilon; + super(epsilonBudget); } } diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index c04a901e650..ac7d4f9bd4f 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -20,17 +20,17 @@ package org.apache.sysds.test.component.cp; import org.apache.sysds.runtime.DMLRuntimeException; -import org.apache.sysds.runtime.privacy.dp.RDPAccountant; +import org.apache.sysds.runtime.privacy.dp.DPBudgetAccountant; import org.junit.Test; import static org.junit.Assert.*; /** - * Tests for {@code DPBuiltinCPInstruction} and {@code RDPAccountant}. + * Tests for {@code DPBuiltinCPInstruction} and {@code DPBudgetAccountant}. * *

    The tests are grouped into three levels: *

      - *
    1. Unit tests on RDPAccountant — verify composition, conversion, + *
    2. Unit tests on DPBudgetAccountant — verify composition, conversion, * and budget enforcement in isolation, with no dependency on the full * SystemDS runtime.
      3. *
    3. Noise distribution tests — verify that the noise blocks @@ -49,12 +49,12 @@ public class DPBuiltinCPInstructionTest { private static final double EPS = 1e-9; // ======================================================================= - // 1. RDPAccountant unit tests + // 1. DPBudgetAccountant unit tests // ======================================================================= @Test public void testAccountantInitialisesAtZeroCost() { - RDPAccountant acc = new RDPAccountant(1.0, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(1.0, 1e-5); // No releases yet: total cost should be a large negative number // (conversion formula gives -∞ when rdpSum = 0 for all orders), // so remainingBudget() should exceed the budget. @@ -66,7 +66,7 @@ public void testAccountantInitialisesAtZeroCost() { @Test public void testSingleLaplaceReleaseDoesNotExceedBudget() { // epsilon=0.5, budget=1.0: one release should consume < budget. - RDPAccountant acc = new RDPAccountant(1.0, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(1.0, 1e-5); acc.compose(0.5, 0.0, 1.0); // Laplace, sensitivity=1 assertEquals(1, acc.releaseCount()); assertTrue("Single release within budget", @@ -75,7 +75,7 @@ public void testSingleLaplaceReleaseDoesNotExceedBudget() { @Test public void testSingleGaussianReleaseDoesNotExceedBudget() { - RDPAccountant acc = new RDPAccountant(1.0, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(1.0, 1e-5); acc.compose(0.5, 1e-5, 1.0); // Gaussian assertEquals(1, acc.releaseCount()); assertTrue("Single Gaussian release within budget", @@ -86,7 +86,7 @@ public void testSingleGaussianReleaseDoesNotExceedBudget() { public void testBudgetExhaustionThrows() { // Budget = 0.1, but we try to make 10 releases at epsilon=0.5 each. // After enough releases the budget must be exceeded. - RDPAccountant acc = new RDPAccountant(0.1, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(0.1, 1e-5); for (int i = 0; i < 10; i++) { acc.compose(0.5, 0.0, 1.0); // will throw before the 10th } @@ -94,7 +94,7 @@ public void testBudgetExhaustionThrows() { @Test public void testCompositionIsMonotonicallyIncreasing() { - RDPAccountant acc = new RDPAccountant(100.0, 1e-5); // large budget + DPBudgetAccountant acc = new DPBudgetAccountant(100.0, 1e-5); // large budget double prev = acc.totalEpsilonSpent(); for (int i = 0; i < 5; i++) { acc.compose(0.3, 1e-5, 1.0); @@ -114,8 +114,8 @@ public void testGaussianTighterThanLaplaceForSameEpsilon() { double eps = 0.5; double delta = 1e-5; - RDPAccountant gaussian = new RDPAccountant(100.0, delta); - RDPAccountant laplace = new RDPAccountant(100.0, delta); + DPBudgetAccountant gaussian = new DPBudgetAccountant(100.0, delta); + DPBudgetAccountant laplace = new DPBudgetAccountant(100.0, delta); for (int i = 0; i < 5; i++) { gaussian.compose(eps, delta, 1.0); @@ -130,7 +130,7 @@ public void testGaussianTighterThanLaplaceForSameEpsilon() { @Test public void testRemainingBudgetDecreasesMonotonically() { - RDPAccountant acc = new RDPAccountant(2.0, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(2.0, 1e-5); double prev = acc.remainingBudget(); for (int i = 0; i < 3; i++) { acc.compose(0.2, 1e-5, 1.0); @@ -146,8 +146,8 @@ public void testHigherEpsilonCostMoreForLaplace() { // Sensitivity determines noise scale but NOT the budget consumed — that is set // entirely by the caller's epsilon parameter. // A release at epsilon=1.0 costs more budget than one at epsilon=0.5. - RDPAccountant acc1 = new RDPAccountant(100.0, 1e-5); - RDPAccountant acc2 = new RDPAccountant(100.0, 1e-5); + DPBudgetAccountant acc1 = new DPBudgetAccountant(100.0, 1e-5); + DPBudgetAccountant acc2 = new DPBudgetAccountant(100.0, 1e-5); acc1.compose(0.5, 0.0, 1.0); // epsilon=0.5, Laplace acc2.compose(1.0, 0.0, 1.0); // epsilon=1.0, same sensitivity @@ -159,22 +159,22 @@ public void testHigherEpsilonCostMoreForLaplace() { @Test(expected = DMLRuntimeException.class) public void testConstructorRejectsZeroEpsilonBudget() { - new RDPAccountant(0.0, 1e-5); + new DPBudgetAccountant(0.0, 1e-5); } @Test(expected = DMLRuntimeException.class) public void testConstructorRejectsNegativeEpsilonBudget() { - new RDPAccountant(-0.5, 1e-5); + new DPBudgetAccountant(-0.5, 1e-5); } @Test(expected = DMLRuntimeException.class) public void testConstructorRejectsDeltaZero() { - new RDPAccountant(1.0, 0.0); + new DPBudgetAccountant(1.0, 0.0); } @Test(expected = DMLRuntimeException.class) public void testConstructorRejectsDeltaOne() { - new RDPAccountant(1.0, 1.0); + new DPBudgetAccountant(1.0, 1.0); } // --- Item 2: single-argument convenience constructor ------------------- @@ -184,8 +184,8 @@ public void testConvenienceConstructorDefaultsDeltaTo1e5() { // The one-arg form delegates to (epsilonBudget, 1e-5). A Gaussian // release whose per-release delta matches that default must produce // identical totalEpsilonSpent() from both construction paths. - RDPAccountant oneArg = new RDPAccountant(10.0); - RDPAccountant twoArg = new RDPAccountant(10.0, 1e-5); + DPBudgetAccountant oneArg = new DPBudgetAccountant(10.0); + DPBudgetAccountant twoArg = new DPBudgetAccountant(10.0, 1e-5); oneArg.compose(0.5, 1e-5, 1.0); twoArg.compose(0.5, 1e-5, 1.0); assertEquals("Convenience constructor must default to delta=1e-5", @@ -198,7 +198,7 @@ public void testConvenienceConstructorDefaultsDeltaTo1e5() { public void testGaussianBudgetExhaustionThrows() { // Budget = 0.1. Each Gaussian release costs more than 0.005, so 20 // releases must exceed the budget well before the loop ends. - RDPAccountant acc = new RDPAccountant(0.1, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(0.1, 1e-5); for (int i = 0; i < 20; i++) { acc.compose(0.3, 1e-5, 1.0); } @@ -211,9 +211,9 @@ public void testMixedCompositionExceedsEitherAlone() { // Compose one Laplace and one Gaussian release. The total cost must // exceed what either mechanism contributes alone, exercising the // _pureEpsilonSum + gaussianEps addition path in totalEpsilonSpent(). - RDPAccountant mixed = new RDPAccountant(100.0, 1e-5); - RDPAccountant lapOnly = new RDPAccountant(100.0, 1e-5); - RDPAccountant gauOnly = new RDPAccountant(100.0, 1e-5); + DPBudgetAccountant mixed = new DPBudgetAccountant(100.0, 1e-5); + DPBudgetAccountant lapOnly = new DPBudgetAccountant(100.0, 1e-5); + DPBudgetAccountant gauOnly = new DPBudgetAccountant(100.0, 1e-5); mixed.compose(0.5, 0.0, 1.0); // Laplace mixed.compose(0.5, 1e-5, 1.0); // Gaussian @@ -231,7 +231,7 @@ public void testMixedCompositionExceedsEitherAlone() { @Test public void testReleaseCountTracksAllReleases() { - RDPAccountant acc = new RDPAccountant(100.0, 1e-5); + DPBudgetAccountant acc = new DPBudgetAccountant(100.0, 1e-5); assertEquals(0, acc.releaseCount()); acc.compose(0.1, 0.0, 1.0); // Laplace assertEquals(1, acc.releaseCount()); @@ -251,8 +251,8 @@ public void testGaussianSensitivityCancelsInRDP() { // D_α = α·Δf²/(2σ²) = α·ε²/(4·ln(1.25/δ)). // Sensitivity cancels. Two accountants with the same (ε,δ) but // different sensitivity must report identical totalEpsilonSpent(). - RDPAccountant acc1 = new RDPAccountant(100.0, 1e-5); - RDPAccountant acc2 = new RDPAccountant(100.0, 1e-5); + DPBudgetAccountant acc1 = new DPBudgetAccountant(100.0, 1e-5); + DPBudgetAccountant acc2 = new DPBudgetAccountant(100.0, 1e-5); acc1.compose(0.5, 1e-5, 1.0); // sensitivity = 1 acc2.compose(0.5, 1e-5, 100.0); // sensitivity = 100, same (ε,δ) assertEquals("Gaussian RDP cost must be independent of sensitivity when (ε,δ) are fixed", @@ -263,8 +263,8 @@ public void testGaussianSensitivityCancelsInRDP() { public void testGaussianLargerEpsilonCostsMoreBudget() { // D_α ∝ ε², so a release declared at a higher ε (less noise, more // privacy loss) must cost more budget than one at a lower ε. - RDPAccountant lowEps = new RDPAccountant(100.0, 1e-5); - RDPAccountant highEps = new RDPAccountant(100.0, 1e-5); + DPBudgetAccountant lowEps = new DPBudgetAccountant(100.0, 1e-5); + DPBudgetAccountant highEps = new DPBudgetAccountant(100.0, 1e-5); lowEps.compose(0.1, 1e-5, 1.0); highEps.compose(0.5, 1e-5, 1.0); assertTrue("Larger epsilon per Gaussian release must cost more budget", From c6b7c826ef67a0d4638fec729680bb004e4d6b94 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Mon, 6 Jul 2026 17:55:33 +0200 Subject: [PATCH 10/26] Delete RDPAccountant --- .../runtime/privacy/dp/RDPAccountant.java | 35 ------------------- 1 file changed, 35 deletions(-) delete mode 100755 src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java deleted file mode 100755 index 0cd3785db7b..00000000000 --- a/src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.apache.sysds.runtime.privacy.dp; - -/** - * @deprecated Use {@link DPBudgetAccountant} instead. - */ -@Deprecated -public class RDPAccountant extends DPBudgetAccountant { - @Deprecated - public RDPAccountant(double epsilonBudget, double delta) { - super(epsilonBudget, delta); - } - @Deprecated - public RDPAccountant(double epsilonBudget) { - super(epsilonBudget); - } -} From ef4680f565888fe3abbaf1f30d3a71f6cb5ea025 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Mon, 6 Jul 2026 23:34:11 +0200 Subject: [PATCH 11/26] Fix parseInstruction() comment --- .../sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java index be02be1cae7..a474c075a96 100755 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -114,7 +114,7 @@ private DPBuiltinCPInstruction( * Reconstructs a {@code DPBuiltinCPInstruction} from its serialised * instruction string produced by the LOP layer. * - *

      Expected format (INSTRUCTION_DELIM = '\u00b0'): + *

      Expected format (OPERAND_DELIM = '\u00b0'): * 

            *   dp_gaussian°target=mVar1·MATRIX·FP64°sensitivity=1.0·SCALAR·FP64·true
      *              °epsilon=0.5·SCALAR·FP64·true°delta=1e-5·SCALAR·FP64·true

From 1d0fac3f87e75a580564e9b046119cc5c06a4d9e Mon Sep 17 00:00:00 2001
From: Maya Anderson 
Date: Tue, 7 Jul 2026 00:18:37 +0200
Subject: [PATCH 12/26] Integration tests

---
 .../cp/DPBuiltinCPInstructionTest.java        |  99 +------------
 .../privacy/dp/DPBuiltinDMLTest.java          | 134 ++++++++++++++++++
 2 files changed, 135 insertions(+), 98 deletions(-)
 create mode 100644 src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java

diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java
index ac7d4f9bd4f..14c11567cef 100755
--- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java
+++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java
@@ -41,8 +41,7 @@
  *
    * *

    The DML integration tests require a built SystemDS jar and are separated - * into a companion class {@code DPBuiltinDMLTest} (shown at the bottom of - * this file as a skeleton). + * into a companion class {@link org.apache.sysds.test.functions.privacy.dp.DPBuiltinDMLTest}. */ public class DPBuiltinCPInstructionTest { @@ -370,99 +369,3 @@ private static double variance(double[] xs) { return s / (xs.length - 1); } } - - -// ========================================================================== -// 3. DML integration test skeleton -// ========================================================================== -// -// Full integration tests extend AutomatedTestBase and drive the DML runner. -// Each test: -// (a) Writes a DML script to a temp file. -// (b) Provides input matrices via TestUtils. -// (c) Calls runTest() and reads the output MatrixBlock. -// (d) Verifies that the noisy result differs from the clean result by a -// statistically plausible amount (not zero, not astronomically large). -// -// The test below is a skeleton that compiles but needs the full SystemDS -// test infrastructure to run. - -/* -package org.apache.sysds.test.functions.privacy.dp; - -import org.apache.sysds.common.Types; -import org.apache.sysds.runtime.matrix.data.MatrixBlock; -import org.apache.sysds.test.AutomatedTestBase; -import org.apache.sysds.test.TestConfiguration; -import org.apache.sysds.test.TestUtils; -import org.junit.Test; - -public class DPBuiltinDMLTest extends AutomatedTestBase { - - private static final String TEST_DIR = "functions/privacy/dp/"; - private static final String TEST_CLASS = TEST_DIR + DPBuiltinDMLTest.class.getSimpleName() + "/"; - private static final String DML_LAPLACE = - "X = read($1);\n" - + "result = dp_laplace(colMeans(X), sensitivity=1.0, epsilon=$2);\n" - + "write(result, $3, format=\"binary\");\n"; - private static final String DML_GAUSSIAN = - "X = read($1);\n" - + "result = dp_gaussian(colMeans(X), sensitivity=1.0, epsilon=$2, delta=1e-5);\n" - + "write(result, $3, format=\"binary\");\n"; - - @Override - public void setUp() { - addTestConfiguration("DPLaplace", new TestConfiguration(TEST_CLASS, "DPLaplace")); - addTestConfiguration("DPGaussian", new TestConfiguration(TEST_CLASS, "DPGaussian")); - } - - @Test - public void testLaplaceOutputDiffersFromCleanMean() { - runDPTest("DPLaplace", DML_LAPLACE, "0.5"); - } - - @Test - public void testGaussianOutputDiffersFromCleanMean() { - runDPTest("DPGaussian", DML_GAUSSIAN, "0.5"); - } - - @Test - public void testHighEpsilonIsCloserToTruth() { - // Higher ε → less noise → result closer to the true mean. - double noisyLow = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "0.1"); - double noisyHigh = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "4.0"); - assertTrue("ε=4 should give less noise than ε=0.1", noisyHigh < noisyLow); - } - - private void runDPTest(String testName, String dml, String epsilonStr) { - getAndLoadTestConfiguration(testName); - int rows = 100, cols = 10; - double[][] data = TestUtils.generateTestMatrix(rows, cols, 0, 1, 1.0, 42); - writeInputMatrixWithMTD("X", data, false); - writeScriptFile(testName + ".dml", dml); - programArgs = new String[]{ input("X"), epsilonStr, output("result") }; - runTest(true, false, null, -1); - MatrixBlock result = readDMLMatrixFromHDFS("result"); - // The noisy result should be a (1 × cols) row vector. - assertEquals(1, result.getNumRows()); - assertEquals(cols, result.getNumColumns()); - // Must differ from the exact mean by a non-trivial amount. - // (A single-seed exact-equality check is fragile; use range check.) - double maxNoise = maxAbsValue(result); - assertTrue("Result should not be exactly zero", maxNoise > 0); - } - - private double maxAbsDiff(String testName, String dml, String epsilonStr) { - // Omitted for brevity: run the test, compute max |noisy - clean|. - return 0; // placeholder - } - - private static double maxAbsValue(MatrixBlock m) { - double max = 0; - for (int r = 0; r < m.getNumRows(); r++) - for (int c = 0; c < m.getNumColumns(); c++) - max = Math.max(max, Math.abs(m.get(r, c))); - return max; - } -} -*/ diff --git a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java new file mode 100644 index 00000000000..df799d59c63 --- /dev/null +++ b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java @@ -0,0 +1,134 @@ +// ========================================================================== +// 3. DML integration test skeleton +// ========================================================================== +// +// Full integration tests extend AutomatedTestBase and drive the DML runner. +// Each test: +// (a) Writes a DML script to a temp file. +// (b) Provides input matrices via TestUtils. +// (c) Calls runTest() and reads the output MatrixBlock. +// (d) Verifies that the noisy result differs from the clean result by a +// statistically plausible amount (not zero, not astronomically large). +// + + +package org.apache.sysds.test.functions.privacy.dp; + +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.HashMap; + +import org.apache.sysds.runtime.matrix.data.MatrixValue.CellIndex; +import org.apache.sysds.test.AutomatedTestBase; +import org.apache.sysds.test.TestConfiguration; +import org.apache.sysds.test.TestUtils; +import org.junit.Test; + +public class DPBuiltinDMLTest extends AutomatedTestBase { + + private static final String TEST_DIR = "functions/privacy/dp/"; + private static final String TEST_CLASS = TEST_DIR + DPBuiltinDMLTest.class.getSimpleName() + "/"; + private static final int ROWS = 100; + private static final int COLS = 10; + private static final String DML_LAPLACE = + "X = read($1);\n" + + "result = dp_laplace(colMeans(X), sensitivity=1.0, epsilon=$2);\n" + + "write(result, $3, format=\"text\");\n"; + private static final String DML_GAUSSIAN = + "X = read($1);\n" + + "result = dp_gaussian(colMeans(X), sensitivity=1.0, epsilon=$2, delta=1e-5);\n" + + "write(result, $3, format=\"text\");\n"; + + @Override + public void setUp() { + addTestConfiguration("DPLaplace", new TestConfiguration(TEST_CLASS, "DPLaplace")); + addTestConfiguration("DPGaussian", new TestConfiguration(TEST_CLASS, "DPGaussian")); + } + + @Test + public void testLaplaceOutputDiffersFromCleanMean() { + runDPTest("DPLaplace", DML_LAPLACE, "0.5"); + } + + @Test + public void testGaussianOutputDiffersFromCleanMean() { + runDPTest("DPGaussian", DML_GAUSSIAN, "0.5"); + } + + @Test + public void testHighEpsilonIsCloserToTruth() { + // Higher ε → less noise → result closer to the true mean. + // NOTE: the DPBudgetAccountant caps total spend at the default budget + // (ε = 1.0) regardless of the per-release ε requested, so ε values + // here must stay well under that cap or the release is rejected. + double noisyLow = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "0.1"); + double noisyHigh = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "0.5"); + assertTrue("ε=0.5 should give less noise than ε=0.1", noisyHigh < noisyLow); + } + + private void runDPTest(String testName, String dml, String epsilonStr) { + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + HashMap result = runAndGetResult(testName, dml, epsilonStr, data); + + // The noisy result should be a (1 × cols) row vector. + int maxRow = 0, maxCol = 0; + for (CellIndex ci : result.keySet()) { + maxRow = Math.max(maxRow, ci.row); + maxCol = Math.max(maxCol, ci.column); + } + assertTrue("Result should have 1 row", maxRow == 1); + assertTrue("Result should have " + COLS + " columns", maxCol == COLS); + // Must differ from the exact mean by a non-trivial amount. + // (A single-seed exact-equality check is fragile; use range check.) + double maxNoise = maxAbsValue(result); + assertTrue("Result should not be exactly zero", maxNoise > 0); + } + + private double maxAbsDiff(String testName, String dml, String epsilonStr) { + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + HashMap result = runAndGetResult(testName, dml, epsilonStr, data); + + double maxDiff = 0; + for (int c = 0; c < COLS; c++) { + double sum = 0; + for (int r = 0; r < ROWS; r++) + sum += data[r][c]; + double cleanMean = sum / ROWS; + double noisy = result.get(new CellIndex(1, c + 1)); + maxDiff = Math.max(maxDiff, Math.abs(noisy - cleanMean)); + } + return maxDiff; + } + + private HashMap runAndGetResult(String testName, String dml, String epsilonStr, + double[][] data) + { + getAndLoadTestConfiguration(testName); + writeInputMatrixWithMTD("X", data, false); + + fullDMLScriptName = getScript(); + try { + File scriptFile = new File(fullDMLScriptName); + scriptFile.getParentFile().mkdirs(); + Files.write(scriptFile.toPath(), dml.getBytes()); + } + catch (IOException e) { + throw new RuntimeException(e); + } + + programArgs = new String[]{ "-args", input("X"), epsilonStr, output("result") }; + runTest(true, false, null, -1); + return readDMLMatrixFromOutputDir("result"); + } + + private static double maxAbsValue(HashMap m) { + double max = 0; + for (double v : m.values()) + max = Math.max(max, Math.abs(v)); + return max; + } +} + From ae2afcb91198585ab876c2be3925a188d42ea3e6 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 7 Jul 2026 00:31:26 +0200 Subject: [PATCH 13/26] Update test comments --- .../cp/DPBuiltinCPInstructionTest.java | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index 14c11567cef..750451b2991 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -154,7 +154,7 @@ public void testHigherEpsilonCostMoreForLaplace() { acc1.totalEpsilonSpent() < acc2.totalEpsilonSpent()); } - // --- Item 1: constructor error paths ------------------------------------ + // --- Constructor error paths ------------------------------------ @Test(expected = DMLRuntimeException.class) public void testConstructorRejectsZeroEpsilonBudget() { @@ -176,7 +176,7 @@ public void testConstructorRejectsDeltaOne() { new DPBudgetAccountant(1.0, 1.0); } - // --- Item 2: single-argument convenience constructor ------------------- + // --- Single-argument convenience constructor ------------------- @Test public void testConvenienceConstructorDefaultsDeltaTo1e5() { @@ -191,8 +191,6 @@ public void testConvenienceConstructorDefaultsDeltaTo1e5() { twoArg.totalEpsilonSpent(), oneArg.totalEpsilonSpent(), EPS); } - // --- Item 3: budget exhaustion via Gaussian releases ------------------- - @Test(expected = DMLRuntimeException.class) public void testGaussianBudgetExhaustionThrows() { // Budget = 0.1. Each Gaussian release costs more than 0.005, so 20 @@ -203,8 +201,6 @@ public void testGaussianBudgetExhaustionThrows() { } } - // --- Item 4: mixed Laplace + Gaussian composition ---------------------- - @Test public void testMixedCompositionExceedsEitherAlone() { // Compose one Laplace and one Gaussian release. The total cost must @@ -226,7 +222,7 @@ public void testMixedCompositionExceedsEitherAlone() { mixed.totalEpsilonSpent() > gauOnly.totalEpsilonSpent()); } - // --- Item 6: release count across multiple mixed releases -------------- + // --- Release count across multiple mixed releases -------------- @Test public void testReleaseCountTracksAllReleases() { @@ -242,7 +238,7 @@ public void testReleaseCountTracksAllReleases() { assertEquals(5, acc.releaseCount()); } - // --- Item 8: edge-case inputs for rdpGaussian / gaussianSigma ---------- + // --- Edge-case inputs for rdpGaussian / gaussianSigma ---------- @Test public void testGaussianSensitivityCancelsInRDP() { @@ -277,11 +273,6 @@ public void testGaussianLargerEpsilonCostsMoreBudget() { // is near zero and the empirical variance matches the theoretical value // within a reasonable tolerance. // - // Note: these tests exercise the static fill* methods indirectly by - // calling the noise-generation logic via reflection or by making the - // methods package-private. The simplest approach for a student project - // is to make fillLaplaceNoise / fillGaussianNoise package-private and - // call them directly from the test (same package). @Test public void testLaplaceNoiseMeanNearZero() { From 394c02105d51eb90b36eaa943a4dc67b437becd9 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 7 Jul 2026 11:18:01 +0200 Subject: [PATCH 14/26] Fix integration test to compare clean to noisy data --- .../functions/privacy/dp/DPBuiltinDMLTest.java | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java index df799d59c63..c5ad6d78b63 100644 --- a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java +++ b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java @@ -81,16 +81,19 @@ private void runDPTest(String testName, String dml, String epsilonStr) { } assertTrue("Result should have 1 row", maxRow == 1); assertTrue("Result should have " + COLS + " columns", maxCol == COLS); - // Must differ from the exact mean by a non-trivial amount. + // Must differ from the exact (clean) mean by a non-trivial amount. // (A single-seed exact-equality check is fragile; use range check.) - double maxNoise = maxAbsValue(result); - assertTrue("Result should not be exactly zero", maxNoise > 0); + double maxDiff = maxAbsDiffFromClean(data, result); + assertTrue("Result should differ from the clean mean", maxDiff > 0); } private double maxAbsDiff(String testName, String dml, String epsilonStr) { double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); HashMap result = runAndGetResult(testName, dml, epsilonStr, data); + return maxAbsDiffFromClean(data, result); + } + private static double maxAbsDiffFromClean(double[][] data, HashMap result) { double maxDiff = 0; for (int c = 0; c < COLS; c++) { double sum = 0; @@ -123,12 +126,5 @@ private HashMap runAndGetResult(String testName, String dml, runTest(true, false, null, -1); return readDMLMatrixFromOutputDir("result"); } - - private static double maxAbsValue(HashMap m) { - double max = 0; - for (double v : m.values()) - max = Math.max(max, Math.abs(v)); - return max; - } } From 67fe1b2d4afa0eb9419b86bbbebe188f2813bc8b Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 7 Jul 2026 15:56:42 +0200 Subject: [PATCH 15/26] Make integration test more readable --- .../functions/privacy/dp/DPBuiltinDMLTest.java | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java index c5ad6d78b63..048c9501e21 100644 --- a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java +++ b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java @@ -1,5 +1,5 @@ // ========================================================================== -// 3. DML integration test skeleton +// DML integration test // ========================================================================== // // Full integration tests extend AutomatedTestBase and drive the DML runner. @@ -60,12 +60,13 @@ public void testGaussianOutputDiffersFromCleanMean() { @Test public void testHighEpsilonIsCloserToTruth() { + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); // Higher ε → less noise → result closer to the true mean. // NOTE: the DPBudgetAccountant caps total spend at the default budget // (ε = 1.0) regardless of the per-release ε requested, so ε values // here must stay well under that cap or the release is rejected. - double noisyLow = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "0.1"); - double noisyHigh = maxAbsDiff("DPGaussian", DML_GAUSSIAN, "0.5"); + double noisyLow = runAndGetMaxAbsColMeansDiffFromClean(data, "DPGaussian", DML_GAUSSIAN, "0.1"); + double noisyHigh = runAndGetMaxAbsColMeansDiffFromClean(data, "DPGaussian", DML_GAUSSIAN, "0.5"); assertTrue("ε=0.5 should give less noise than ε=0.1", noisyHigh < noisyLow); } @@ -83,17 +84,16 @@ private void runDPTest(String testName, String dml, String epsilonStr) { assertTrue("Result should have " + COLS + " columns", maxCol == COLS); // Must differ from the exact (clean) mean by a non-trivial amount. // (A single-seed exact-equality check is fragile; use range check.) - double maxDiff = maxAbsDiffFromClean(data, result); + double maxDiff = maxAbsColMeansDiffFromClean(data, result); assertTrue("Result should differ from the clean mean", maxDiff > 0); } - private double maxAbsDiff(String testName, String dml, String epsilonStr) { - double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + private double runAndGetMaxAbsColMeansDiffFromClean(double[][] data, String testName, String dml, String epsilonStr) { HashMap result = runAndGetResult(testName, dml, epsilonStr, data); - return maxAbsDiffFromClean(data, result); + return maxAbsColMeansDiffFromClean(data, result); } - private static double maxAbsDiffFromClean(double[][] data, HashMap result) { + private static double maxAbsColMeansDiffFromClean(double[][] data, HashMap result) { double maxDiff = 0; for (int c = 0; c < COLS; c++) { double sum = 0; From fbe18f095b409d96999a4c3440cc64c43f070c9a Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 7 Jul 2026 18:29:12 +0200 Subject: [PATCH 16/26] Comment fix --- .../cp/DPBuiltinCPInstruction.java | 21 ++++++------------- .../privacy/dp/DPBudgetAccountant.java | 4 +++- 2 files changed, 9 insertions(+), 16 deletions(-) diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java index a474c075a96..01d744ee26c 100755 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -34,7 +34,7 @@ * CP instruction for differential-privacy release of an already-computed * aggregate. * - *

    DML syntax (post-aggregate form, Option A): + *

    DML syntax (post-aggregate form): * 

      *   result = dp_laplace(aggregate, sensitivity=1.0, epsilon=0.5)
  *   result = dp_gaussian(aggregate, sensitivity=1.0, epsilon=0.5, delta=1e-5)
@@ -54,19 +54,6 @@
  * single method is replaced with a static analysis that reads the
  * sensitivity bound computed by the compiler; every other line in this class
  * stays unchanged.
- *
- * 
    Registration required in:
- * 
      
- *   
    • {@code org.apache.sysds.common.Builtins} – add
- *       {@code DP_LAPLACE("dp_laplace", false)} and
- *       {@code DP_GAUSSIAN("dp_gaussian", false)}
      • 
- *   
    • {@code org.apache.sysds.runtime.instructions.CPInstructionParser} –
- *       add opcode-to-type mappings and a parse branch that returns a
- *       {@code DPBuiltinCPInstruction}
      • 
- *   
    • {@code org.apache.sysds.runtime.controlprogram.context.ExecutionContext}
- *       – add {@code getDPBudgetAccountant()} returning a session-scoped
- *       {@link DPBudgetAccountant} (lazy-initialised field)
      • 
- * 
    
  */
 public class DPBuiltinCPInstruction extends ComputationCPInstruction {
 
@@ -261,10 +248,14 @@ private MatrixBlock generateNoise(
         noise.allocateDenseBlock();
 
         if (instOpcode.equals(OPCODE_LAPLACE)) {
+            // Laplace mechanism
+            // For a given epsilon, noise is drawn from the Laplace distribution at
+            // scale b = sensitivity / epsilon
             fillLaplaceNoise(noise, sensitivity / epsilon);
         } else {
             // Gaussian mechanism: calibrate sigma for (epsilon, delta)-DP.
-            // Standard formula: sigma >= sensitivity * sqrt(2 * ln(1.25/delta)) / epsilon
+            // For a given epsilon, noise is drawn from the normal distribution at
+            // sigma^2 = 2 * sensitivity^2 * log(1.25/delta) / epsilon^2
             double sigma = sensitivity
                     * Math.sqrt(2.0 * Math.log(1.25 / delta))
                     / epsilon;
diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java
index ce5afcb7850..707c296906f 100644
--- a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java
+++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java
@@ -20,6 +20,7 @@
 package org.apache.sysds.runtime.privacy.dp;
 
 import org.apache.sysds.runtime.DMLRuntimeException;
+import org.apache.sysds.runtime.instructions.cp.DPBuiltinCPInstruction;
 
 /**
  * Session-scoped differential privacy budget accountant.
@@ -216,6 +217,7 @@ public void compose(double epsilon, double delta, double sensitivity) {
      * zero when no Gaussian releases have been recorded).
      */
     public double totalEpsilonSpent() {
+        // Take min_α(ε_α) as the current total privacy cost
         double gaussianEps = Double.MAX_VALUE;
         for (int i = 0; i < ORDERS.length; i++) {
             double alpha = ORDERS[i];
@@ -260,7 +262,7 @@ private static double rdpGaussian(double alpha, double sensitivity, double sigma
     /**
      * Gaussian noise scale σ calibrated to (ε, δ)-DP:
      * 
    -     *   σ = Δf · sqrt(2 · ln(1.25 / δ)) / ε
+     *   σ = Δf · sqrt(2 · log(1.25 / δ)) / ε
      * 
    
      * Must match the formula used in {@link DPBuiltinCPInstruction} so that
      * the RDP cost recorded here is consistent with the noise actually injected.

From f22c283522f677d23b76ceda81963db6b537fb42 Mon Sep 17 00:00:00 2001
From: Maya Anderson 
Date: Tue, 7 Jul 2026 18:58:50 +0200
Subject: [PATCH 17/26] Fix L1 L2 sensitivity parameter documentation

---
 .../cp/DPBuiltinCPInstruction.java            | 27 +++++++++++++++----
 .../privacy/dp/DPBudgetAccountant.java        | 14 ++++++++--
 2 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java
index 01d744ee26c..63938f21881 100755
--- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java
+++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java
@@ -40,6 +40,16 @@
  *   result = dp_gaussian(aggregate, sensitivity=1.0, epsilon=0.5, delta=1e-5)
  *
    * + *

    Sensitivity norm: {@code sensitivity} is not interchangeable + * between the two builtins. {@code dp_laplace} calibrates its noise scale + * to the L1 sensitivity of {@code aggregate} to a single-record + * change; {@code dp_gaussian} calibrates its σ to the L2 sensitivity. + * For a scalar aggregate (e.g. a single sum or mean) the two norms coincide, + * but for a vector-valued aggregate (e.g. column means of a multi-column + * matrix) they generally differ (L2 ≤ L1 ≤ √d·L2 for d entries) — the caller + * is responsible for supplying the norm matching the builtin invoked (see + * {@link #sensitivityOf}). + * *

    The instruction receives a materialised matrix (the aggregate result), * injects calibrated noise element-wise, records the release with the * session-scoped {@link DPBudgetAccountant}, and returns the noisy matrix. @@ -202,19 +212,26 @@ public void processInstruction(ExecutionContext ec) { // ----------------------------------------------------------------------- /** - * Returns the sensitivity of {@code aggregate} to a single-record change. + * Returns the sensitivity of {@code aggregate} to a single-record change, + * in the norm required by the mechanism actually invoked: L1 for + * {@code dp_laplace}, L2 for {@code dp_gaussian} (see the class + * Javadoc). The two only coincide when {@code aggregate} is scalar. * *

    Phase 1 (now): returns the caller-supplied literal from the - * DML script. Sensitivity analysis is the caller's responsibility. + * DML script as-is, with no norm conversion or validation — the DML + * author must compute the sensitivity in the correct norm for the + * builtin they call. Sensitivity analysis is the caller's responsibility. * *

    Phase 2 (HOP-level rewrite pass): replace this body with a * call that inspects the HOP node that produced {@code aggregate}, reads - * the {@code sensitivityBound} field computed during compilation, and - * returns it. No other line in this class changes. + * the {@code sensitivityBound} field computed during compilation (in the + * norm matching {@code instOpcode}), and returns it. No other line in + * this class changes. * * @param aggregate the already-computed aggregate block (ignored in * Phase 1; used in Phase 2 to look up lineage) - * @return caller-supplied sensitivity constant + * @return caller-supplied sensitivity constant, expected to already be + * in the L1 norm (Laplace) or L2 norm (Gaussian) */ private double sensitivityOf(MatrixBlock aggregate) { // Phase 1: unwrap the literal or variable value from the param map. diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java index 707c296906f..9dd011fcb1e 100644 --- a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java @@ -39,7 +39,8 @@ * basic composition — each release contributes exactly its ε to a running * sum. This is the tightest possible bound for pure DP and avoids the * looser estimate that results from routing Laplace through the RDP - * conversion path (which would introduce an unnecessary δ). + * conversion path (which would introduce an unnecessary δ). Noise scale + * is calibrated to L1 sensitivity (see {@link #compose}). *

  • Gaussian (delta > 0): (ε, δ)-DP via Rényi DP composition. * Rényi divergences at a discrete set of orders α compose additively; * the accumulated sum is converted to (ε, δ) at query time using the @@ -178,7 +179,16 @@ public DPBudgetAccountant(double epsilonBudget) { * * @param epsilon per-release ε parameter (must be > 0) * @param delta per-release δ parameter (0 for Laplace, >0 for Gaussian) - * @param sensitivity L2 sensitivity Δf of the released quantity (must be > 0) + * @param sensitivity sensitivity Δf of the released quantity (must be > 0). + * The norm depends on the mechanism selected by + * {@code delta}: callers must supply the + * L1 sensitivity ‖f(D) − f(D′)‖₁ when + * {@code delta == 0} (Laplace), and the L2 + * sensitivity ‖f(D) − f(D′)‖₂ when {@code delta > 0} + * (Gaussian). The two coincide for scalar-valued + * releases but diverge for vector-valued ones, so + * passing the wrong norm silently under- or + * over-calibrates the noise. * @throws DMLRuntimeException if the cumulative ε after this release * would exceed the budget */ From 461da60554575863f001ca8fa0cc6d308dd5f316 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 7 Jul 2026 23:30:17 +0200 Subject: [PATCH 18/26] junit output format --- pom.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/pom.xml b/pom.xml index 068bed2e8ea..724ee4b1f1d 100644 --- a/pom.xml +++ b/pom.xml @@ -408,6 +408,7 @@ maven-surefire-plugin ${maven-surefire-plugin.version} + plain ${maven.test.skip} ${test-parallel} ${test-threadCount} From b253d141cdc0489da44f320d6a7440bf5a0a194f Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 7 Jul 2026 23:41:06 +0200 Subject: [PATCH 19/26] Remove Claude files --- CLAUDE.md | 113 -------------- CLAUDE_CODE_PLAN.md | 364 -------------------------------------------- 2 files changed, 477 deletions(-) delete mode 100644 CLAUDE.md delete mode 100755 CLAUDE_CODE_PLAN.md diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index d3fcc2be49c..00000000000 --- a/CLAUDE.md +++ /dev/null @@ -1,113 +0,0 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Project - -Apache SystemDS — an end-to-end ML system that compiles DML scripts (R-like syntax) into hybrid execution plans across local (CP), Apache Spark, GPU, and federated backends. - -## Build Commands - -```bash -# Full build (skips tests) -mvn clean package -DskipTests - -# Build and run default tests -mvn clean package - -# Run a single Java test class -mvn test -Dtest=FullMatrixMultiplicationTest - -# Run a test with a specific method -mvn test -Dtest=FullMatrixMultiplicationTest#testMethod - -# Run checkstyle (disabled by default) -mvn checkstyle:check -Dcheckstyle.skip=false - -# Python tests (from src/main/python/) -cd src/main/python && source python_venv/bin/activate && pip install -e . && python -m pytest tests/ - -# Run a DML script directly (after building) -./bin/systemds hello.dml -``` - -The surefire default test is `org.apache.sysds.test.usertest.**`. Override with `-Dtest=` to target specific classes. Test JVM is configured with `-Xmx3000m`. - -## Code Style - -Apply the Eclipse formatter profile at [dev/CodeStyle_eclipse.xml](dev/CodeStyle_eclipse.xml) before committing. Checkstyle rules are at [dev/checkstyle/checkstyle.xml](dev/checkstyle/checkstyle.xml). - -## Commit Tags - -All commits must be prefixed: `[SYSTEMDS-#]` for Jira issues, `[MINOR]` for small changes, `[DOC]` for docs, `[HOTFIX]` for release patches. The project uses linear history — rebase, never merge commits. - -## Compilation Pipeline - -The full compilation sequence is in [DMLScript.java:460-510](src/main/java/org/apache/sysds/api/DMLScript.java#L460): - -1. **Parse** — DML source → `DMLProgram` AST (ANTLR-based, `parser/dml/`) -2. **Live Variable Analysis + Validate** — `DMLTranslator.liveVariableAnalysis()` / `validateParseTree()` -3. **Construct HOPs** — AST → HOP DAG (`DMLTranslator.constructHops()`) -4. **Rewrite HOP DAGs** — algebraic simplifications, IPA, memory estimates, CSE (`hops/rewrite/`) -5. **Construct LOPs** — HOP DAG → LOP DAG; exec type (CP/Spark/GPU/Fed) selected here -6. **Rewrite LOP DAGs** — inject prefetch, broadcast, OOC tee operators -7. **Generate runtime program** — LOPs emit instruction strings; codegen (Spoof) fuses operators -8. **Execute** — `ProgramBlock` hierarchy interprets instructions via `ExecutionContext` - -## Architecture Patterns - -### Two-Level IR: HOPs → LOPs -`Hop` ([hops/Hop.java](src/main/java/org/apache/sysds/hops/Hop.java)) is the algebraic IR. Each concrete subclass (`BinaryOp`, `AggBinaryOp`, `UnaryOp`, etc.) implements abstract template methods: -- `constructLops()` — lowers this operator to backend-specific LOPs -- `optFindExecType()` — selects CP / Spark / GPU / Fed based on cost model -- `inferOutputCharacteristics()` — size/sparsity estimation -- `computeOutputMemEstimate()` / `computeIntermediateMemEstimate()` — memory cost - -`Lop` ([lops/Lop.java](src/main/java/org/apache/sysds/lops/Lop.java)) is the backend-specific IR. LOPs emit instruction strings consumed by the runtime. - -### DAGs, not Trees -Both HOPs and LOPs are DAGs with explicit `_input`/`_parent` lists and a `_visited` boolean. All rewrite/lowering passes do DFS traversal with the visited flag. This enables CSE — shared subcomputations appear once in the DAG. - -### Chain of Responsibility for Rewrites -`ProgramRewriter` ([hops/rewrite/ProgramRewriter.java](src/main/java/org/apache/sysds/hops/rewrite/ProgramRewriter.java)) holds an ordered list of `HopRewriteRule` subclasses and fires each over the full DAG. To add an optimization: subclass `HopRewriteRule`, implement `rewriteHopDAGs()` / `rewriteHopDAG()`, and register it. Existing rules include loop vectorization, loop-invariant hoisting, Spark checkpoint injection, OOC tee injection, and compressed reblock. - -### Parallel AST ↔ Runtime Hierarchies -The parser (`Statement`/`StatementBlock`) and runtime (`ProgramBlock`) mirror each other exactly: `ForStatement` → `ForProgramBlock`, `ParForStatement` → `ParForProgramBlock`, etc. The AST is structural only; all execution behavior lives in the runtime mirror. - -### String-Serialized Instruction Boundary -LOPs emit plain strings as instructions. At runtime, `InstructionParser.parseSingleInstruction()` reads the exec-type prefix (`CP·`, `SPARK·`, `GPU·`, `FED·`, `OOC·`) and dispatches to the appropriate backend parser. This decouples the compiler from all backend implementations. - -### Caching Layer -`CacheableData` ([runtime/controlprogram/caching/CacheableData.java](src/main/java/org/apache/sysds/runtime/controlprogram/caching/CacheableData.java)) is a generic abstract envelope for large data objects. Subclasses (`MatrixObject`, `FrameObject`, `TensorObject`) inherit full eviction/spill-to-disk lifecycle. - -## Key Source Locations - -| Area | Path | -|---|---| -| Compiler entry point | `src/main/java/org/apache/sysds/api/DMLScript.java` | -| DML → HOP translator | `src/main/java/org/apache/sysds/parser/DMLTranslator.java` | -| HOP base + optimizer | `src/main/java/org/apache/sysds/hops/` | -| HOP rewrites | `src/main/java/org/apache/sysds/hops/rewrite/` | -| LOP base | `src/main/java/org/apache/sysds/lops/` | -| Runtime control flow | `src/main/java/org/apache/sysds/runtime/controlprogram/` | -| Runtime instructions | `src/main/java/org/apache/sysds/runtime/instructions/` | -| Compressed Linear Algebra | `src/main/java/org/apache/sysds/runtime/compress/` | -| Operator fusion (Spoof) | `src/main/java/org/apache/sysds/runtime/codegen/` | -| Out-of-core execution | `src/main/java/org/apache/sysds/runtime/ooc/` | -| Built-in algorithms (DML) | `scripts/builtin/` | -| Staging area (new algos) | `scripts/staging/` | -| Python API | `src/main/python/systemds/` | - -## Testing - -Java tests extend `AutomatedTestBase`. The default exec mode is `ExecMode.HYBRID`; tests call `setExecMode()` / `resetExecMode()` to test specific backends. - -- **`src/test/java/.../test/functions/`** — end-to-end integration tests that run DML scripts through the full pipeline -- **`src/test/java/.../test/component/`** — unit tests targeting specific subsystems (compress, matrix, codegen, ooc, parfor, etc.) -- **`src/test/java/.../test/applications/`** — full ML algorithm correctness tests - -Each function test has a paired DML script under `src/test/scripts/functions/`. - -## Adding a New Built-in Algorithm - -New algorithms live in `scripts/staging/` until they have sufficient test coverage, then move to `scripts/builtin/`. The Python API wrappers in `src/main/python/systemds/operator/algorithm/builtin/` are auto-generated — see `src/main/python/generator/` for the generation scripts. diff --git a/CLAUDE_CODE_PLAN.md b/CLAUDE_CODE_PLAN.md deleted file mode 100755 index c08b79e80e4..00000000000 --- a/CLAUDE_CODE_PLAN.md +++ /dev/null @@ -1,364 +0,0 @@ -# Claude Code Plan: Differential-Privacy Built-ins for Apache SystemDS - -## Goal -Add `dp_laplace` and `dp_gaussian` as native (non-script) DML built-in -functions backed by a session-scoped Rényi-DP budget accountant. - -This plan is written for Claude Code. Follow the steps in order. Read each -file before editing it. Do not guess at class names, field names, or method -signatures — grep to verify every assumption before writing code. - ---- - -## Step 1 — Orient: understand the instruction routing mechanism - -```bash -# 1a. Find how opcodes are mapped to CPInstruction subclasses. -# We are looking for whatever replaces (or still is) the opcode→type map. -grep -rn "parseSingleInstruction\|CPType\|CPInstructionParser" \ - src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java \ - | head -60 -``` - -```bash -# 1b. Look for how a recently-added native instruction (e.g. lstm, compress) -# is wired in. This gives us the exact pattern to copy. -grep -n "lstm\|compress\|bias_add" \ - src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java \ - | head -30 -``` - -```bash -# 1c. Read the full parseSingleInstruction method to understand the switch/map -# dispatch that leads to parseInstruction() on a specific class. -grep -n "parseSingleInstruction\|case Dnn\|case Builtin\|DnnCPInstruction\ -\|ParameterizedBuiltin\|parseInstruction" \ - src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java \ - | head -60 -``` - -> **Note for Claude Code**: the exact field/method name may differ from -> `String2CPInstructionType`. Use the output of step 1b to find the real -> registration pattern. Copy it exactly — do not invent names. - ---- - -## Step 2 — Orient: understand the Builtins enum constructor - -```bash -# 2a. Read the constructor and the first ~60 enum entries to confirm the -# exact parameter signature (name, script) vs (name, script, ReturnType). -sed -n '35,100p' \ - src/main/java/org/apache/sysds/common/Builtins.java -``` - -```bash -# 2b. Confirm no "parameterized" field exists in the constructor. -grep -n "parameterized\|Parameterized\|boolean" \ - src/main/java/org/apache/sysds/common/Builtins.java | head -20 -``` - -> Expected: constructor is `(String name, boolean script)` with `script=false` -> for native built-ins. Verify before proceeding. - ---- - -## Step 3 — Orient: understand BuiltinFunctionExpression validation - -```bash -# 3a. Find how an existing similar native built-in (e.g. colMeans, abs) -# is validated in the parser and how it maps to a HOP. -grep -n "COLMEAN\|ABS\|BuiltinFunctionExpression\|case COLMEAN\|case ABS" \ - src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java \ - | head -30 -``` - -```bash -# 3b. Find DMLTranslator to understand how BuiltinFunctionExpression -# creates a HOP node. -grep -n "createBuiltin\|BuiltinOp\|UnaryOp\|colMeans\|case COLMEAN" \ - src/main/java/org/apache/sysds/parser/DMLTranslator.java \ - | head -20 -``` - -```bash -# 3c. Find how the HOP emits a LOP that becomes a CPInstruction opcode string. -grep -n "getOpCode\|getLops\|addLop\|Lops" \ - src/main/java/org/apache/sysds/hops/UnaryOp.java \ - | head -20 -``` - -> This tells us whether we need a new HOP type, a new LOP type, or whether -> dp_laplace/dp_gaussian can reuse an existing HOP+LOP path. - ---- - -## Step 4 — Create the new files - -Create the following files. All paths are relative to the repository root. - -### 4a. RDPAccountant - -**File**: `src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java` - -Verify the package declaration matches the target path: -```bash -head -5 src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java -``` - -### 4b. DPBuiltinCPInstruction - -**File**: `src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java` - -Verify the imports compile against the actual codebase: - -```bash -# Confirm BinaryOperator and Plus exist at the expected paths. -find src/main/java -name "BinaryOperator.java" -o -name "Plus.java" | head -5 - -# Confirm MatrixBlock.binaryOperations signature. -grep -n "binaryOperations" \ - src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java \ - | head -10 -``` - -If `MatrixBlock.binaryOperations` has a different signature, update the call -in `processInstruction` to match. - ---- - -## Step 5 — Patch ExecutionContext to carry the accountant - -```bash -# 5a. Read the end of ExecutionContext to find where to add the new field. -grep -n "class ExecutionContext\|private.*Map\|private.*List\|getMatrix\ -\|releaseMatrix\|setScalar" \ - src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java \ - | tail -40 -``` - -Add the following to `ExecutionContext.java`: - -```java -// --- DP budget accountant (lazy-initialised, one per script execution) --- -private RDPAccountant _rdpAccountant = null; - -public RDPAccountant getRDPAccountant() { - if (_rdpAccountant == null) - // Default budget: ε=1.0, δ=1e-5. Future work: set via DML built-in. - _rdpAccountant = new RDPAccountant(1.0, 1e-5); - return _rdpAccountant; -} -``` - -Add the import at the top of `ExecutionContext.java`: -```java -import org.apache.sysds.runtime.privacy.dp.RDPAccountant; -``` - ---- - -## Step 6 — Register in Builtins.java - -```bash -# 6a. Find a good alphabetical insertion point between "D" entries. -grep -n "^[[:space:]]*D[A-Z_]*(" \ - src/main/java/org/apache/sysds/common/Builtins.java | head -20 -``` - -Insert after the last `D`-prefixed entry (or before the first `E` entry): - -```java -DP_LAPLACE("dp_laplace", false), -DP_GAUSSIAN("dp_gaussian", false), -``` - -Confirm `script=false` is correct by checking a nearby native built-in: -```bash -grep -A1 "DIAG\|DECOMPRESS\|DET" \ - src/main/java/org/apache/sysds/common/Builtins.java | head -10 -``` - ---- - -## Step 7 — Wire into the parser - -Use the routing pattern discovered in Step 1. The pattern will be one of: - -**Pattern A — opcode map + switch** (most likely based on commit history): -```bash -# Find the CPType enum to add a new entry if needed. -grep -n "enum CPType\|Dnn,\|BuiltinNary,\|ParameterizedBuiltin," \ - src/main/java/org/apache/sysds/runtime/instructions/cp/CPInstruction.java \ - | head -20 -``` - -If a new `CPType.DPBuiltin` is needed, add it to the `CPType` enum, then add -the map entry and switch case following the exact pattern of `CPType.Dnn`. - -**Pattern B — direct opcode string match in parseSingleInstruction**: -Add a branch: -```java -else if (opcode.equals("dp_laplace") || opcode.equals("dp_gaussian")) - return DPBuiltinCPInstruction.parseInstruction(str); -``` - -> Use whichever pattern the codebase actually uses. Do not mix patterns. - ---- - -## Step 8 — Wire into BuiltinFunctionExpression - -```bash -# 8a. Find the validate() switch to add parameter checking. -grep -n "case DIAG\|case ABS\|case COLMEAN\|checkNumParameters\|checkMatrixParam" \ - src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java \ - | head -20 -``` - -Add cases for `DP_LAPLACE` and `DP_GAUSSIAN` in the `validate()` switch: - -```java -case DP_LAPLACE: { - // dp_laplace(aggregate, sensitivity, epsilon) - checkNumParameters(3); - checkMatrixParam(getFirstExpr()); // aggregate matrix - checkScalarParam(getSecondExpr()); // sensitivity - checkScalarParam(getThirdExpr()); // epsilon - output.setDataType(DataType.MATRIX); - output.setValueType(ValueType.FP64); - // Output shape matches input shape; dimensions copied from input. - output.setDimensions( - getFirstExpr().getOutput().getDim1(), - getFirstExpr().getOutput().getDim2()); - break; -} -case DP_GAUSSIAN: { - // dp_gaussian(aggregate, sensitivity, epsilon, delta) - checkNumParameters(4); - checkMatrixParam(getFirstExpr()); - checkScalarParam(getSecondExpr()); // sensitivity - checkScalarParam(getThirdExpr()); // epsilon - checkScalarParam(getFourthExpr()); // delta - output.setDataType(DataType.MATRIX); - output.setValueType(ValueType.FP64); - output.setDimensions( - getFirstExpr().getOutput().getDim1(), - getFirstExpr().getOutput().getDim2()); - break; -} -``` - -> Verify that `checkScalarParam` and `getFourthExpr` exist: -> ```bash -> grep -n "checkScalarParam\|getFourthExpr\|getThirdExpr" \ -> src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java \ -> | head -10 -> ``` - ---- - -## Step 9 — Wire into DMLTranslator - -```bash -# 9a. Find how colMeans (COLMEAN) translates to a HOP in DMLTranslator. -grep -n "COLMEAN\|case COLMEAN\|createBuiltinOp\|UnaryOp" \ - src/main/java/org/apache/sysds/parser/DMLTranslator.java \ - | head -20 -``` - -```bash -# 9b. Find where to add new cases — likely inside a large switch on Builtins. -grep -n "case ABS\|case DIAG\|case CEIL" \ - src/main/java/org/apache/sysds/parser/DMLTranslator.java | head -10 -``` - -The simplest approach: map `DP_LAPLACE` and `DP_GAUSSIAN` to a `UnaryOp` HOP -with a custom opcode string. The LOP produced by `UnaryOp` will carry the -opcode string `"dp_laplace"` or `"dp_gaussian"`, which the `CPInstructionParser` -will then route to `DPBuiltinCPInstruction.parseInstruction`. - -Add cases following the existing unary pattern: -```java -case DP_LAPLACE: -case DP_GAUSSIAN: - // Reuse UnaryOp HOP — the opcode string routes to DPBuiltinCPInstruction. - currBuiltinOp = new UnaryOp(target.getName(), DataType.MATRIX, - ValueType.FP64, OpOp1.valueOf(bi.name()), expr); - break; -``` - -> Verify `OpOp1` has a compatible entry or whether a different HOP class is -> needed. If `OpOp1` does not work, fall back to creating a `FunctionOp`. - ---- - -## Step 10 — Build and smoke test - -```bash -# Build only the affected modules to get fast feedback. -mvn compile -pl src/main/java -am -q 2>&1 | tail -30 -``` - -Fix any compilation errors before proceeding. - -```bash -# Run the self-contained unit tests (no SystemDS runtime needed). -mvn test -pl src/test/java \ - -Dtest=DPBuiltinCPInstructionTest \ - -Dsurefire.failIfNoSpecifiedTests=false \ - 2>&1 | tail -40 -``` - -```bash -# Smoke-test end-to-end with a minimal DML script. -cat > /tmp/dp_smoke.dml << 'EOF' -X = rand(rows=100, cols=10, min=0, max=1); -mu = colMeans(X); -noisy = dp_laplace(mu, sensitivity=0.1, epsilon=1.0); -print(toString(noisy)); -EOF - -./bin/systemds /tmp/dp_smoke.dml 2>&1 | tail -20 -``` - -If the smoke test fails with an opcode-not-found error, re-check Steps 7 and 9. -If it fails with a budget error, reduce the sensitivity or widen epsilon. - ---- - -## Step 11 — Run the federated benchmark - -```bash -cat > /tmp/dp_fedavg_benchmark.dml << 'EOF' -# Federated averaging with DP release of column means. -# Sweep epsilon across {0.5, 1, 4, 8} by passing $epsilon as an arg. -X = read($1); -mu = colMeans(X); -noisy = dp_gaussian(mu, sensitivity=$sensitivity, epsilon=$epsilon, delta=1e-5); -write(noisy, $2, format="csv"); -EOF - -for eps in 0.5 1 4 8; do - echo "--- epsilon=$eps ---" - ./bin/systemds /tmp/dp_fedavg_benchmark.dml \ - -args data/adult.csv /tmp/result_eps${eps}.csv \ - -nvargs sensitivity=0.01 epsilon=$eps \ - 2>&1 | tail -5 -done -``` - ---- - -## Files modified (summary) - -| File | Change | -|---|---| -| `src/main/java/org/apache/sysds/common/Builtins.java` | Add `DP_LAPLACE`, `DP_GAUSSIAN` entries | -| `src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java` | Add validate cases | -| `src/main/java/org/apache/sysds/parser/DMLTranslator.java` | Add HOP-creation cases | -| `src/main/java/org/apache/sysds/runtime/instructions/CPInstructionParser.java` | Register opcodes using actual routing pattern | -| `src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java` | **New file** | -| `src/main/java/org/apache/sysds/runtime/privacy/dp/RDPAccountant.java` | **New file** | -| `src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java` | Add `getRDPAccountant()` | -| `src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinCPInstructionTest.java` | **New file** (unit tests) | From 379ddcd6b5158cfec26ed6cb272d169be48ec028 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Thu, 9 Jul 2026 23:44:53 +0200 Subject: [PATCH 20/26] Remove files irrelevant to the PR --- src/main/python/notebooks/python_e2e_l1.ipynb | 520 ------------------ src/main/python/notebooks/quick_start.ipynb | 234 -------- src/main/python/requirements.txt | 9 - 3 files changed, 763 deletions(-) delete mode 100644 src/main/python/notebooks/python_e2e_l1.ipynb delete mode 100644 src/main/python/notebooks/quick_start.ipynb delete mode 100644 src/main/python/requirements.txt diff --git a/src/main/python/notebooks/python_e2e_l1.ipynb b/src/main/python/notebooks/python_e2e_l1.ipynb deleted file mode 100644 index 34564cffbd6..00000000000 --- a/src/main/python/notebooks/python_e2e_l1.ipynb +++ /dev/null @@ -1,520 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "26ef529d", - "metadata": {}, - "source": [ - "This example shows how one can work the SystemDS framework. More precisely, we will make use of the built-in DataManager, Multinomial Logistic Regression function, and the Confusion Matrix function. The dataset used in this tutorial is a preprocessed version of the “UCI Adult Data Set”. If one wants to skip the explanation then the full script is available at the end of this level.\n", - "\n", - "We will train a Multinomial Logistic Regression model on the training dataset and subsequently use the test dataset to assess how well our model can predict if the income is above or below $50K/yr based on the features." - ] - }, - { - "cell_type": "markdown", - "id": "18ac25e2", - "metadata": {}, - "source": [ - "# Step 1: Load and prepare data\n", - "First, we get our training and testing data from the built-in DataManager. Since the multiLogReg function requires the labels (Y) to be > 0, we add 1 to all labels. This ensures that the smallest label is >= 1. Additionally we will only take a fraction of the training and test set into account to speed up the execution." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "40c4453f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Warning: Scuro dependencies missing or wrong version installed: torch 2.4.1, torchvision 0.19.1, librosa 0.10.2, opencv-python 4.10.0.84, opt-einsum 3.3.0, h5py 3.11.0, transformers 4.46.3, nltk 3.9.1, gensim 4.3.3\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Using incubator modules: jdk.incubator.vector\n", - "26/05/06 15:19:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" - ] - } - ], - "source": [ - "from systemds.context import SystemDSContext\n", - "from systemds.examples.tutorials.adult import DataManager\n", - "from systemds.operator.algorithm import multiLogReg\n", - "from systemds.operator.algorithm import multiLogRegPredict\n", - "from systemds.operator.algorithm import confusionMatrix\n", - "\n", - "with SystemDSContext() as sds:\n", - " d = DataManager()\n", - "\n", - " # limit the sample size\n", - " train_count = 15000\n", - " test_count = 5000\n", - "\n", - " # Get train and test datasets.\n", - " X_frame, Y_frame, Xt_frame, Yt_frame = d.get_preprocessed_dataset(sds)\n", - "\n", - " # Transformation specification\n", - " jspec_data = d.get_jspec(sds)\n", - " jspec_labels = sds.scalar(f'\"{ {\"recode\": [\"income\"]} }\"')\n", - "\n", - " # Transform frames to matrices.\n", - " X, M1 = X_frame.transform_encode(spec=jspec_data)\n", - " Xt = Xt_frame.transform_apply(spec=jspec_data, meta=M1) \n", - " Y, M2 = Y_frame.transform_encode(spec=jspec_labels)\n", - " Yt = Yt_frame.transform_apply(spec=jspec_labels, meta=M2) \n", - " \n", - " # Subsample to make training faster\n", - " X = X[0:train_count]\n", - " Y = Y[0:train_count]\n", - " Xt = Xt[0:test_count]\n", - " Yt = Yt[0:test_count]" - ] - }, - { - "cell_type": "markdown", - "id": "349c0d0b", - "metadata": {}, - "source": [ - "Here the DataManager contains the code for downloading and setting up either Pandas DataFrames or internal SystemDS Frames, for the best performance and no data transfer from pandas to SystemDS it is recommended to read directly from disk into SystemDS." - ] - }, - { - "cell_type": "markdown", - "id": "4a180e28", - "metadata": {}, - "source": [ - "# Step 2: Training\n", - "Now that we prepared the data, we can use the multiLogReg function. First, we will train the model on our training data. Afterward, we can make predictions on the test data and assess the performance of the model." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "079a61b4", - "metadata": {}, - "outputs": [], - "source": [ - "betas = multiLogReg(X, Y, verbose=False)" - ] - }, - { - "cell_type": "markdown", - "id": "d145d87a", - "metadata": {}, - "source": [ - "Note that nothing has been calculated yet. In SystemDS the calculation is executed once `compute()` is called. E.g. `betas_res = betas.compute()`.\n", - "\n", - "We can now use the trained model to make predictions on the test data." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "6b734c4d", - "metadata": {}, - "outputs": [], - "source": [ - "[_, y_pred, acc] = multiLogRegPredict(Xt, betas, Y=Yt)" - ] - }, - { - "cell_type": "markdown", - "id": "db63e4ab", - "metadata": {}, - "source": [ - "The multiLogRegPredict function has three return values:\n", - "\n", - "`m`, a matrix with the mean probability of correctly classifying each label. We do not use it further in this example.\n", - "\n", - "`y_pred`, is the predictions made using the model\n", - "\n", - "`acc`, is the accuracy achieved by the model." - ] - }, - { - "cell_type": "markdown", - "id": "bffaa936", - "metadata": {}, - "source": [ - "# Step 3: Confusion Matrix\n", - "A confusion matrix is a useful tool to analyze the performance of the model and to obtain a better understanding which classes the model has difficulties separating. The confusionMatrix function takes the predicted labels and the true labels. It then returns the confusion matrix for the predictions and the confusion matrix averages of each true class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f3cd2e62", - "metadata": {}, - "outputs": [], - "source": [ - "confusion_matrix_abs, _ = confusionMatrix(y_pred, Yt).compute()" - ] - }, - { - "cell_type": "markdown", - "id": "37bfe729", - "metadata": {}, - "source": [ - "# Full Script" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "54190e79", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Warning: Scuro dependencies missing or wrong version installed: torch 2.4.1, torchvision 0.19.1, librosa 0.10.2, opencv-python 4.10.0.84, opt-einsum 3.3.0, h5py 3.11.0, transformers 4.46.3, nltk 3.9.1, gensim 4.3.3\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Using incubator modules: jdk.incubator.vector\n", - "26/05/06 16:39:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", - "INFO:python_e2e_l1_example:Confusion Matrix\n", - "INFO:python_e2e_l1_example:[[3580. 488.]\n", - " [ 248. 684.]]\n" - ] - } - ], - "source": [ - "from systemds.context import SystemDSContext\n", - "from systemds.examples.tutorials.adult import DataManager\n", - "from systemds.operator.algorithm import multiLogReg\n", - "from systemds.operator.algorithm import multiLogRegPredict\n", - "from systemds.operator.algorithm import confusionMatrix\n", - "from systemds.operator.algorithm import getAccuracy\n", - "\n", - "import logging\n", - "logger = logging.getLogger('python_e2e_l1_example')\n", - "logger.setLevel(logging.INFO)\n", - "\n", - "with SystemDSContext() as sds:\n", - " d = DataManager()\n", - "\n", - " # limit the sample size\n", - " train_count = 15000\n", - " test_count = 5000\n", - "\n", - " # Get train and test datasets.\n", - " X_frame, Y_frame, Xt_frame, Yt_frame = d.get_preprocessed_dataset(sds)\n", - "\n", - " # Transformation specification\n", - " jspec_data = d.get_jspec(sds)\n", - " jspec_labels = sds.scalar(f'\"{ {\"recode\": [\"income\"]} }\"')\n", - "\n", - " # Transform frames to matrices.\n", - " X, M1 = X_frame.transform_encode(spec=jspec_data)\n", - " Xt = Xt_frame.transform_apply(spec=jspec_data, meta=M1) \n", - " Y, M2 = Y_frame.transform_encode(spec=jspec_labels)\n", - " Yt = Yt_frame.transform_apply(spec=jspec_labels, meta=M2) \n", - " \n", - " # Subsample to make training faster\n", - " X = X[0:train_count]\n", - " Y = Y[0:train_count]\n", - " Xt = Xt[0:test_count]\n", - " Yt = Yt[0:test_count]\n", - "\n", - " # Train model \n", - " betas = multiLogReg(X, Y, verbose=False)\n", - "\n", - " # Apply model\n", - " [_, y_pred, acc] = multiLogRegPredict(Xt, betas, Y=Yt)\n", - "\n", - " # Confusion Matrix\n", - " confusion_matrix_abs, _ = confusionMatrix(y_pred, Yt).compute()\n", - " \n", - " # accuracy = getAccuracy(Yt, y_pred).compute()\n", - "\n", - "\n", - " logger.info(\"Confusion Matrix\")\n", - " logger.info(confusion_matrix_abs)" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "c2d7049a", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAHHCAYAAAAiSltoAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAVPVJREFUeJzt3Xl8DPf/B/DXJuzm3ESQi4gjhDjraKTuLxLEVVSVEndpaImrWkdQ0p/7po4KFS3V0oo6IsQZRQhxpUVIkE0QySYh9/z+0EytWHaziZB5PT3m8ch85vOZec8+yL59jhmZIAgCiIiISLKMSjoAIiIiKllMBoiIiCSOyQAREZHEMRkgIiKSOCYDREREEsdkgIiISOKYDBAREUkckwEiIiKJYzJAREQkcUwGiF7wzz//wNPTE1ZWVpDJZNi9e3eRnv/27duQyWQIDAws0vO+y9q2bYu2bduWdBhEksVkgN5KN2/exGeffYbq1avDxMQESqUSLVq0wLJly/D06dNivbaPjw+ioqIwd+5c/Pjjj2jatGmxXu9NGjx4MGQyGZRK5Us/x3/++QcymQwymQwLFy7U+/z379+Hv78/IiMjiyBaInpTypR0AEQv2rt3Lz766CMoFAoMGjQI9erVQ1ZWFk6cOIFJkybhypUrWLduXbFc++nTpwgPD8c333yDMWPGFMs1nJ2d8fTpU5QtW7ZYzv86ZcqUwZMnT7Bnzx707dtX41hQUBBMTEyQkZFRqHPfv38fs2bNQtWqVdGoUSOd2x08eLBQ1yOiosFkgN4qMTEx6NevH5ydnXH48GE4ODiIx3x9fXHjxg3s3bu32K7/4MEDAIC1tXWxXUMmk8HExKTYzv86CoUCLVq0wE8//VQgGdi2bRu8vb3x66+/vpFYnjx5AjMzM8jl8jdyPSJ6OQ4T0Ftl/vz5SEtLw8aNGzUSgXwuLi748ssvxf2cnBzMmTMHNWrUgEKhQNWqVfH1118jMzNTo13VqlXRtWtXnDhxAu+//z5MTExQvXp1bNmyRazj7+8PZ2dnAMCkSZMgk8lQtWpVAM+61/N/fp6/vz9kMplGWUhICFq2bAlra2tYWFjA1dUVX3/9tXhc25yBw4cPo1WrVjA3N4e1tTV69OiBa9euvfR6N27cwODBg2FtbQ0rKysMGTIET5480f7BvqB///7Yt28fkpOTxbKzZ8/in3/+Qf/+/QvUT0pKwsSJE1G/fn1YWFhAqVSic+fOuHjxolgnLCwMzZo1AwAMGTJEHG7Iv8+2bduiXr16iIiIQOvWrWFmZiZ+Li/OGfDx8YGJiUmB+/fy8kK5cuVw//59ne+ViF6PyQC9Vfbs2YPq1avjgw8+0Kn+8OHDMWPGDDRu3BhLlixBmzZtEBAQgH79+hWoe+PGDfTp0wcdO3bEokWLUK5cOQwePBhXrlwBAPTq1QtLliwBAHzyySf48ccfsXTpUr3iv3LlCrp27YrMzEzMnj0bixYtQvfu3XHy5MlXtjt06BC8vLyQmJgIf39/+Pn54dSpU2jRogVu375doH7fvn2RmpqKgIAA9O3bF4GBgZg1a5bOcfbq1QsymQy//fabWLZt2zbUrl0bjRs3LlD/1q1b2L17N7p27YrFixdj0qRJiIqKQps2bcQv5jp16mD27NkAgJEjR+LHH3/Ejz/+iNatW4vnefToETp37oxGjRph6dKlaNeu3UvjW7ZsGSpWrAgfHx/k5uYCAL7//nscPHgQK1asgKOjo873SkQ6EIjeEikpKQIAoUePHjrVj4yMFAAIw4cP1yifOHGiAEA4fPiwWObs7CwAEI4dOyaWJSYmCgqFQpgwYYJYFhMTIwAQFixYoHFOHx8fwdnZuUAMM2fOFJ7/Z7RkyRIBgPDgwQOtcedfY9OmTWJZo0aNBFtbW+HRo0di2cWLFwUjIyNh0KBBBa43dOhQjXN++OGHQvny5bVe8/n7MDc3FwRBEPr06SO0b99eEARByM3NFezt7YVZs2a99DPIyMgQcnNzC9yHQqEQZs+eLZadPXu2wL3la9OmjQBAWLt27UuPtWnTRqPswIEDAgDh22+/FW7duiVYWFgIPXv2fO09EpH+2DNAbw21Wg0AsLS01Kn+n3/+CQDw8/PTKJ8wYQIAFJhb4ObmhlatWon7FStWhKurK27dulXomF+UP9fg999/R15enk5t4uPjERkZicGDB8PGxkYsb9CgATp27Cje5/NGjRqlsd+qVSs8evRI/Ax10b9/f4SFhUGlUuHw4cNQqVQvHSIAns0zMDJ69usiNzcXjx49EodAzp8/r/M1FQoFhgwZolNdT09PfPbZZ5g9ezZ69eoFExMTfP/99zpfi4h0x2SA3hpKpRIAkJqaqlP9O3fuwMjICC4uLhrl9vb2sLa2xp07dzTKq1SpUuAc5cqVw+PHjwsZcUEff/wxWrRogeHDh8POzg79+vXDjh07XpkY5Mfp6upa4FidOnXw8OFDpKena5S/eC/lypUDAL3upUuXLrC0tMT27dsRFBSEZs2aFfgs8+Xl5WHJkiWoWbMmFAoFKlSogIoVK+LSpUtISUnR+ZqVKlXSa7LgwoULYWNjg8jISCxfvhy2trY6tyUi3TEZoLeGUqmEo6MjLl++rFe7FyfwaWNsbPzSckEQCn2N/PHsfKampjh27BgOHTqEgQMH4tKlS/j444/RsWPHAnUNYci95FMoFOjVqxc2b96MXbt2ae0VAIB58+bBz88PrVu3xtatW3HgwAGEhISgbt26OveAAM8+H31cuHABiYmJAICoqCi92hKR7pgM0Fula9euuHnzJsLDw19b19nZGXl5efjnn380yhMSEpCcnCyuDCgK5cqV05h5n+/F3gcAMDIyQvv27bF48WJcvXoVc+fOxeHDh3HkyJGXnjs/zujo6ALHrl+/jgoVKsDc3NywG9Cif//+uHDhAlJTU1866TLfzp070a5dO2zcuBH9+vWDp6cnOnToUOAz0TUx00V6ejqGDBkCNzc3jBw5EvPnz8fZs2eL7PxE9B8mA/RWmTx5MszNzTF8+HAkJCQUOH7z5k0sW7YMwLNubgAFZvwvXrwYAODt7V1kcdWoUQMpKSm4dOmSWBYfH49du3Zp1EtKSirQNv/hOy8ud8zn4OCARo0aYfPmzRpfrpcvX8bBgwfF+ywO7dq1w5w5c7By5UrY29trrWdsbFyg1+GXX37BvXv3NMryk5aXJU76mjJlCmJjY7F582YsXrwYVatWhY+Pj9bPkYgKjw8dordKjRo1sG3bNnz88ceoU6eOxhMIT506hV9++QWDBw8GADRs2BA+Pj5Yt24dkpOT0aZNG5w5cwabN29Gz549tS5bK4x+/fphypQp+PDDD/HFF1/gyZMnWLNmDWrVqqUxgW727Nk4duwYvL294ezsjMTERKxevRqVK1dGy5YttZ5/wYIF6Ny5Mzw8PDBs2DA8ffoUK1asgJWVFfz9/YvsPl5kZGSEadOmvbZe165dMXv2bAwZMgQffPABoqKiEBQUhOrVq2vUq1GjBqytrbF27VpYWlrC3Nwc7u7uqFatml5xHT58GKtXr8bMmTPFpY6bNm1C27ZtMX36dMyfP1+v8xHRa5Twagail/r777+FESNGCFWrVhXkcrlgaWkptGjRQlixYoWQkZEh1svOzhZmzZolVKtWTShbtqzg5OQkTJ06VaOOIDxbWujt7V3gOi8uadO2tFAQBOHgwYNCvXr1BLlcLri6ugpbt24tsLQwNDRU6NGjh+Do6CjI5XLB0dFR+OSTT4S///67wDVeXH536NAhoUWLFoKpqamgVCqFbt26CVevXtWok3+9F5cubtq0SQAgxMTEaP1MBUFzaaE22pYWTpgwQXBwcBBMTU2FFi1aCOHh4S9dEvj7778Lbm5uQpkyZTTus02bNkLdunVfes3nz6NWqwVnZ2ehcePGQnZ2tka98ePHC0ZGRkJ4ePgr74GI9CMTBD1mHBEREVGpwzkDREREEsdkgIiISOKYDBAREUkckwEiIiKJYzJAREQkcUwGiIiIJO6dfuhQXl4e7t+/D0tLyyJ9DCoREb0ZgiAgNTUVjo6O4psxi0NGRgaysrIMPo9cLoeJiUkRRPR2eaeTgfv378PJyamkwyAiIgPFxcWhcuXKxXLujIwMmFqWB3KeGHwue3t7xMTElLqE4J1OBvLfey9384HMWPfXohK9Sy7/GVDSIRAVm9TUVDR2qyb+Pi8OWVlZQM4TKNx8AEO+K3KzoLq6GVlZWUwG3ib5QwMyYzmTASq1LJXKkg6BqNi9kaHeMiYGfVcIstI7ze6dTgaIiIh0JgNgSNJRiqemMRkgIiJpkBk92wxpX0qV3jsjIiIqQWvWrEGDBg2gVCqhVCrh4eGBffv2icfbtm0LmUymsY0aNUrjHLGxsfD29oaZmRlsbW0xadIk5OTkaNQJCwtD48aNoVAo4OLigsDAQL1jZc8AERFJg0xm4DCBfm0rV66M7777DjVr1oQgCNi8eTN69OiBCxcuoG7dugCAESNGYPbs2WIbMzMz8efc3Fx4e3vD3t4ep06dQnx8PAYNGoSyZcti3rx5AICYmBh4e3tj1KhRCAoKQmhoKIYPHw4HBwd4eXnpHCuTASIikoY3PEzQrVs3jf25c+dizZo1OH36tJgMmJmZwd7e/qXtDx48iKtXr+LQoUOws7NDo0aNMGfOHEyZMgX+/v6Qy+VYu3YtqlWrhkWLFgEA6tSpgxMnTmDJkiV6JQMcJiAiItKDWq3W2DIzM1/bJjc3Fz///DPS09Ph4eEhlgcFBaFChQqoV68epk6diidP/nsWQnh4OOrXrw87OzuxzMvLC2q1GleuXBHrdOjQQeNaXl5eCA8P1+ue2DNARETSUETDBC8+7G7mzJnw9/d/aZOoqCh4eHggIyMDFhYW2LVrF9zc3AAA/fv3h7OzMxwdHXHp0iVMmTIF0dHR+O233wAAKpVKIxEAIO6rVKpX1lGr1Xj69ClMTU11ujUmA0REJBEGDhP825keFxcH5XPP/1AoFFpbuLq6IjIyEikpKdi5cyd8fHxw9OhRuLm5YeTIkWK9+vXrw8HBAe3bt8fNmzdRo0YNA+LUH4cJiIiI9JC/OiB/e1UyIJfL4eLigiZNmiAgIAANGzbEsmXLXlrX3d0dAHDjxg0Azx59nJCQoFEnfz9/noG2OkqlUudeAYDJABERSUX+MIEhm4Hy8vK0zjGIjIwEADg4OAAAPDw8EBUVhcTERLFOSEgIlEqlONTg4eGB0NBQjfOEhIRozEvQBYcJiIhIGt7waoKpU6eic+fOqFKlClJTU7Ft2zaEhYXhwIEDuHnzJrZt24YuXbqgfPnyuHTpEsaPH4/WrVujQYMGAABPT0+4ublh4MCBmD9/PlQqFaZNmwZfX1+xN2LUqFFYuXIlJk+ejKFDh+Lw4cPYsWMH9u7dq1esTAaIiIiKQWJiIgYNGoT4+HhYWVmhQYMGOHDgADp27Ii4uDgcOnQIS5cuRXp6OpycnNC7d29MmzZNbG9sbIzg4GCMHj0aHh4eMDc3h4+Pj8ZzCapVq4a9e/di/PjxWLZsGSpXrowNGzbotawQAGSCIAhFdudvmFqthpWVFRT1R/BFRVRq3T66pKRDICo2qWo1ajpVQEpKisakvKIkfle8PwGyMtrH919HyMlE5plFxRprSWHPABERSQPfTaAVkwEiIpKGN/w44ndJ6U1ziIiISCfsGSAiImngMIFWTAaIiEgaZDIDkwEOExAREVEpxZ4BIiKSBiPZs82Q9qUUkwEiIpIGzhnQqvTeGREREemEPQNERCQNfM6AVkwGiIhIGjhMoFXpvTMiIiLSCXsGiIhIGjhMoBWTASIikgYOE2jFZICIiKSBPQNald40h4iIiHTCngEiIpIGDhNoxWSAiIikgcMEWpXeNIeIiIh0wp4BIiKSCAOHCUrx/5+ZDBARkTRwmECr0pvmEBERkU7YM0BERNIgkxm4mqD09gwwGSAiImng0kKtSu+dERERkU7YM0BERNLACYRaMRkgIiJp4DCBVkwGiIhIGtgzoFXpTXOIiIhIJ+wZICIiaeAwgVZMBoiISBo4TKBV6U1ziIiISCfsGSAiIkmQyWSQsWfgpZgMEBGRJDAZ0I7DBERERBLHngEiIpIG2b+bIe1LKSYDREQkCRwm0I7DBERERBLHngEiIpIE9gxox2SAiIgkgcmAdkwGiIhIEpgMaMc5A0RERBLHngEiIpIGLi3UiskAERFJAocJtOMwARERUTFYs2YNGjRoAKVSCaVSCQ8PD+zbt088npGRAV9fX5QvXx4WFhbo3bs3EhISNM4RGxsLb29vmJmZwdbWFpMmTUJOTo5GnbCwMDRu3BgKhQIuLi4IDAzUO1YmA0REJAnP3mAsM2DT73qVK1fGd999h4iICJw7dw7/+9//0KNHD1y5cgUAMH78eOzZswe//PILjh49ivv376NXr15i+9zcXHh7eyMrKwunTp3C5s2bERgYiBkzZoh1YmJi4O3tjXbt2iEyMhLjxo3D8OHDceDAAf0+G0EQBP1u7+2hVqthZWUFRf0RkBnLSzocomJx++iSkg6BqNikqtWo6VQBKSkpUCqVxXKN/O8K677rIZObFfo8QtYTJO8YYVCsNjY2WLBgAfr06YOKFSti27Zt6NOnDwDg+vXrqFOnDsLDw9G8eXPs27cPXbt2xf3792FnZwcAWLt2LaZMmYIHDx5ALpdjypQp2Lt3Ly5fvixeo1+/fkhOTsb+/ft1jos9A0RERHpQq9UaW2Zm5mvb5Obm4ueff0Z6ejo8PDwQERGB7OxsdOjQQaxTu3ZtVKlSBeHh4QCA8PBw1K9fX0wEAMDLywtqtVrsXQgPD9c4R36d/HPoiskAERFJgmFDBP9NPnRycoKVlZW4BQQEaL1mVFQULCwsoFAoMGrUKOzatQtubm5QqVSQy+WwtrbWqG9nZweVSgUAUKlUGolA/vH8Y6+qo1ar8fTpU50/G64mICIiaSiipYVxcXEawwQKhUJrE1dXV0RGRiIlJQU7d+6Ej48Pjh49akAQxYPJABERkR7yVwfoQi6Xw8XFBQDQpEkTnD17FsuWLcPHH3+MrKwsJCcna/QOJCQkwN7eHgBgb2+PM2fOaJwvf7XB83VeXIGQkJAApVIJU1NTne+JwwRERCQNhg4RFMFzBvLy8pCZmYkmTZqgbNmyCA0NFY9FR0cjNjYWHh4eAAAPDw9ERUUhMTFRrBMSEgKlUgk3NzexzvPnyK+Tfw5dsWeAiIgkwdCHDunbdurUqejcuTOqVKmC1NRUbNu2DWFhYThw4ACsrKwwbNgw+Pn5wcbGBkqlEmPHjoWHhweaN28OAPD09ISbmxsGDhyI+fPnQ6VSYdq0afD19RWHJkaNGoWVK1di8uTJGDp0KA4fPowdO3Zg7969esXKZICIiCThTScDiYmJGDRoEOLj42FlZYUGDRrgwIED6NixIwBgyZIlMDIyQu/evZGZmQkvLy+sXr1abG9sbIzg4GCMHj0aHh4eMDc3h4+PD2bPni3WqVatGvbu3Yvx48dj2bJlqFy5MjZs2AAvLy/97o3PGSB6u/E5A1SavcnnDJQfsAlGBjxnIC/rCR4FDSnWWEsKewaIiEga+KIirZgMEBGRJLzpYYJ3CVcTEBERSRx7BoiISBLYM6AdkwEiIpIEJgPacZiAiIhI4tgzQEREksCeAe2YDBARkTRwaaFWHCYgIiKSOPYMEBGRJHCYQDsmA0REJAlMBrRjMkBERJLAZEA7zhkgIiKSOPYMEBGRNHA1gVZMBoiISBI4TKAdhwmIiIgkjj0D77ihvVtiaO9WcHKwAQBcv6XCgo37cOjUVa1tlBammP55N3Rt1xDllGaIi3+MrxfvRMgr2hiqR/v38PUob1RxKI9bcQ/gv2K3xvWmjOiCXp6NUcmuHLKzcxF5PRbfrt6DiCt3ii0mevesDjqE+ev2Ykif1pg59sOX1snOycXqrYfw64GzUD1MQXUnW3z1WVe0da9TrLHtPRKJRT/sw11VEqpVqoivRnVFu+ZuYkwLN/yJsNPXEBv/CJbmJmjZpBamfNYVdhWsijUu+g97BrR7K3oGVq1ahapVq8LExATu7u44c+ZMSYf0zrifmIxZK39Hu0Hz8T+fBTh+7m8ELRyJ2tXtX1q/bBlj7Fo1BlUcbDB4ykY06zMHX87bhvgHKYWOoUXjmrj4+yytx99vUA0bvh2Mrb+Ho82n32Hv0YvYunAk6tRwEOvcjE3E5AW/oMUn89B5xGLE3k/CbyvHoLy1RaHjotLl4rVYbPsjHLVrOL6y3sINf2LbnnDM+rIXDm2eggHdP8Bn0zbh8t93C33t8As30OLj2VqPR1yOwRdzfsTHXdzx5/qJ8GxVDyO/+QHRt+IBAE8zsnDl77sYO6gjgtdPwNo5Q3AzLhHDv95Q6JhIfzLIxISgUFspnjRQ4snA9u3b4efnh5kzZ+L8+fNo2LAhvLy8kJiYWNKhvRP2H7+MkFNXcSvuAW7GJuLbNXuQ/iQTTetVe2n9T7t7oJzSDAMmrsNfl24hLj4Jp87fwOV/7ol1ZDIZxg/2RORuf9w/vhjHg75C9/81KnSMn/Vri9Dwa1ixNRR/307AvLV7cfF6HEZ81Eass/PAORw9E4079x7h+i0Vpi39DUoLU9St+epf/CQN6U8yMe7brfhuUl9YWZq+su6ug+fg+2kHtGvuhiqOFTCwZwu0a14HG3aEiXXy8vKwaushtPx4Dlw7TkanoQvwZ1hkoeP7YecxtHm/Nj775H9wqWqHCcO6oG6tyti86ziAZ71xWxePRtf/vYcaVWzRuG5VzP6yN6Ki7+JewuNCX5eoqJR4MrB48WKMGDECQ4YMgZubG9auXQszMzP88MMPJR3aO8fISIZeHZvAzFSOs1ExL63TuXV9nI2KwYIpHyN6/zyc+vlr+A32hJHRfxmv32BPfNzlffh9tx0e/eZi9U9H8P1sH3zQ2KVQcb1fvxrCzl7XKDt8+hqa1a/60vplyxjD58MWSEl9gst/33tpHZKW6Ut3op1HHbRs6vraulnZOVDINUdATRRlcTbqlri/OigUvx04i7kTPkLI5skY9lEbjJsbhNORNwoV34Urt9GiSS2NstbNXHH+FcNcqelPIZPJoLR4dXJDRcegXgEDhxjediU6ZyArKwsRERGYOnWqWGZkZIQOHTogPDy8BCN7t7jVcMSBHybARF4G6U8zMXDSekTHqF5a17lSebRqWgu/7D+LvuPWoLpTRSyc/DHKlDHG/A37IC9bBuOHeOJD35ViQnHn3iM0b1gDQz5siVPn9f9laVteiQePUjXKHiSlwra8UqPMq2U9bJg7BGYmZaF6qMaHY1YiKSVd7+tR6fJH6Hlc+fsefv9+vE71WzerjQ07wvB+wxpwdiyPkxH/YP+xS8jLywMAZGblYNXWQ9i6aDSa1KsKAKjiWAHnom5h2x/haN5I/6T3QVIqKpSz1CirWM4SD5PUL62fkZmN774PRvf278HS3ETv61EhcWmhViWaDDx8+BC5ubmws7PTKLezs8P169cL1M/MzERmZqa4r1a//B+a1PxzJwGtBwRAaWGKHu3fw2r/gej62bKXJgRGMiM8fJyKcfN+Ql6egIvX4+BQ0RpjB7bH/A37UN2pAsxNFfht5RiNdvKyxrgU/d+Ya9zRReLPxkYyKORlNMp+2XcWft/9rNd9HD/3N1oPCEB5awsM6vkBNs0big5DFuLh4zS9zkOlx/3Ex5i9Yhd+XDQaJoqyOrWZ+cWH+GrBdrQfGACZTAZnx/L4qPP72PHns7lId+49wNOMLAycuEajXXZ2LtxqVhL33TpNEX/OzRWQlZ2jUdazYxPMm9BX73vKzsnFGP/NEAQB3/p9pHd7ouLwTq0mCAgIwKxZ2ieqSVV2Ti5i7j4EAFy8Hof33KpgVL+2GB9Q8Ms44VEKsnNykZcniGV/31bBvoIVypYxhrmpAgDw8fg1iE9M1miblZ0j/tx6QID4c9N6VTFzTA90G7VMLEtNzxB/TnykRsXyL/yvycYSiY80k7knGVmIufsQMXcf4tzl2zj36wwM7PEBlgQe1PWjoFImKvouHj5OQ9cR/yWaubl5OHPxFrbsOoG/QxbA2FhztLO8tQXWzx2GjMxsJKvTYVfBCt99H4wqjs9W3KQ/zQIA/PDdCNi/MJNf/tzwwp8bJoo/R16LxXff78HPS33FMovn/kdf0cYSDx+/0Pv1OBUVbDR7v7JzcuE7czPuJjzGT0s+Z6/AG8bVBNqVaDJQoUIFGBsbIyEhQaM8ISEB9vYFZ8NPnToVfn5+4r5arYaTk1Oxx/muMZLJNH6pPe+vi7fQx6spZDIZBOFZQlCjii3iHzxLEqJjVMjIzIaTXblXDgnkJx8A4GhbDrm5eRplzzsTFYM2zVyx9qcwsayde22cjbr96vswkkFe9p3KV6mItWhSEwc2TdYom/TdT6hRxRaj+rcvkAg8z0RRFvYVrZGdk4v9xy7Bu20jAEDNqnaQy8vgfsLjVw4JVK1cUfw5/kEKjI2NNMqe917dqjgV8TeGPTcp9sS5v9G4rrO4n58I3L73AD8t9UU5K/NX3jsVPSYD2pXob1q5XI4mTZogNDQUPXv2BPBslm9oaCjGjBlToL5CoYBCoXjDUb7dZvh2x6FTVxCnegxLMxP06dQULZvURO+xqwEAa/wHIv5BCmav+gMA8MOvxzH8o9b4bkIfrNtxFDWcKsJvsCfWbT8KAEh7komVW0Mx1683ZEZGOB15E0oLE7g3rIHU9Az8vPcvvWP8/ucwBH8/Dr4D/oeDJ66gl2cTNKpTBePm/QQAMDORY8JQL+w7FoWEhymwsbbA8I9aw6GiNX4PPV9EnxS9iyzMTOBa3UGjzNRUDmsrc7Hcb24Q7CpaYcrIrgCAC1fvIOFhCtxcHKF6kIKlgQeQl5eHzz75n3jOkR+3w5xVvyNPENCsfnWkpj/FuagYWJiboE+n9/WOc2if1vj4i5VYv/0I2jV3w57DFxAVHYeAic+GEbJzcjF6RiCu/H0XG78bjtzcPLFnzFppxqT3DZHJnm2GtC+tSvxvoJ+fH3x8fNC0aVO8//77WLp0KdLT0zFkyJCSDu2dUKGcBdb4D4JdBSXUaRm4cuMeeo9djbAzz+ZcVLa3QZ7w35DAvYRk9PliNeaO74UT26Yi/kEyvv85DEu3hIh15q4NxsPkNIwf3BFVK32ClNSnuBgdhyWbDhQqxjOXYjBiWiC+Gd0V0z/vhltxD/DpxHW4dvPZGuzcvDzUrGqHft7uKG9tjqSUJ7hw9Q66jFyC67dePhGSKN+9xMeQPbcaJjMrGws3/InY+EcwN1WgnXsdLPlmgMaSxAnDOsPG2hyrg0IRd3/Hs2WstSrD99MOhYqhSb1qWDZ9IBZt/BML1u9F1coVsW7uUDFhUT1IwaGTlwEAXYYt1Gj701JfeLxXuJU6REVFJgjPfVOUkJUrV2LBggVQqVRo1KgRli9fDnd399e2U6vVsLKygqL+CMiM5W8gUqI37/bRJSUdAlGxSVWrUdOpAlJSUqBUKl/foBDyvyuqj90JI0Xhh2fyMtNxa0WfYo21pJR4zwAAjBkz5qXDAkREREXGwGGC0ry0sMQfOkREREQl663oGSAiIipuXE2gHZMBIiKSBK4m0I7DBERERBLHngEiIpIEIyOZxkvZ9CUY0PZtx2SAiIgkgcME2nGYgIiISOLYM0BERJLA1QTaMRkgIiJJ4DCBdkwGiIhIEtgzoB3nDBAREUkcewaIiEgS2DOgHZMBIiKSBM4Z0I7DBERERBLHngEiIpIEGQwcJijF7zBmMkBERJLAYQLtOExAREQkcUwGiIhIEvJXExiy6SMgIADNmjWDpaUlbG1t0bNnT0RHR2vUadu2bYFrjBo1SqNObGwsvL29YWZmBltbW0yaNAk5OTkadcLCwtC4cWMoFAq4uLggMDBQr1iZDBARkSTkDxMYsunj6NGj8PX1xenTpxESEoLs7Gx4enoiPT1do96IESMQHx8vbvPnzxeP5ebmwtvbG1lZWTh16hQ2b96MwMBAzJgxQ6wTExMDb29vtGvXDpGRkRg3bhyGDx+OAwcO6Bwr5wwQEREVg/3792vsBwYGwtbWFhEREWjdurVYbmZmBnt7+5ee4+DBg7h69SoOHToEOzs7NGrUCHPmzMGUKVPg7+8PuVyOtWvXolq1ali0aBEAoE6dOjhx4gSWLFkCLy8vnWJlzwAREUlCUQ0TqNVqjS0zM1On66ekpAAAbGxsNMqDgoJQoUIF1KtXD1OnTsWTJ0/EY+Hh4ahfvz7s7OzEMi8vL6jValy5ckWs06FDB41zenl5ITw8XOfPhj0DREQkCUW1msDJyUmjfObMmfD3939l27y8PIwbNw4tWrRAvXr1xPL+/fvD2dkZjo6OuHTpEqZMmYLo6Gj89ttvAACVSqWRCAAQ91Uq1SvrqNVqPH36FKampq+9NyYDREQkCUX1OOK4uDgolUqxXKFQvLatr68vLl++jBMnTmiUjxw5Uvy5fv36cHBwQPv27XHz5k3UqFGj0LHqi8MEREREelAqlRrb65KBMWPGIDg4GEeOHEHlypVfWdfd3R0AcOPGDQCAvb09EhISNOrk7+fPM9BWR6lU6tQrADAZICIiqTB0JYGenQqCIGDMmDHYtWsXDh8+jGrVqr22TWRkJADAwcEBAODh4YGoqCgkJiaKdUJCQqBUKuHm5ibWCQ0N1ThPSEgIPDw8dI6VyQAREUnCm37OgK+vL7Zu3Ypt27bB0tISKpUKKpUKT58+BQDcvHkTc+bMQUREBG7fvo0//vgDgwYNQuvWrdGgQQMAgKenJ9zc3DBw4EBcvHgRBw4cwLRp0+Dr6yv2SIwaNQq3bt3C5MmTcf36daxevRo7duzA+PHjdY6VyQAREVExWLNmDVJSUtC2bVs4ODiI2/bt2wEAcrkchw4dgqenJ2rXro0JEyagd+/e2LNnj3gOY2NjBAcHw9jYGB4eHvj0008xaNAgzJ49W6xTrVo17N27FyEhIWjYsCEWLVqEDRs26LysEOAEQiIikog3/W4CQRBeedzJyQlHjx597XmcnZ3x559/vrJO27ZtceHCBb3iex6TASIikoSiWk1QGnGYgIiISOLYM0BERJLAVxhrx2SAiIgkgcME2nGYgIiISOLYM0BERJLAngHtmAwQEZEkcM6AdkwGiIhIEtgzoB3nDBAREUkcewaIiEgSOEygHZMBIiKSBA4TaMdhAiIiIoljzwAREUmCDAYOExRZJG8fJgNERCQJRjIZjAzIBgxp+7bjMAEREZHEsWeAiIgkgasJtGMyQEREksDVBNoxGSAiIkkwkj3bDGlfWnHOABERkcSxZ4CIiKRBZmBXfynuGWAyQEREksAJhNpxmICIiEji2DNARESSIPv3jyHtSysmA0REJAlcTaAdhwmIiIgkjj0DREQkCXzokHY6JQN//PGHzifs3r17oYMhIiIqLlxNoJ1OyUDPnj11OplMJkNubq4h8RAREdEbplMykJeXV9xxEBERFSu+wlg7g+YMZGRkwMTEpKhiISIiKjYcJtBO79UEubm5mDNnDipVqgQLCwvcunULADB9+nRs3LixyAMkIiIqCvkTCA3ZSiu9k4G5c+ciMDAQ8+fPh1wuF8vr1auHDRs2FGlwREREVPz0Tga2bNmCdevWYcCAATA2NhbLGzZsiOvXrxdpcEREREUlf5jAkK200nvOwL179+Di4lKgPC8vD9nZ2UUSFBERUVHjBELt9O4ZcHNzw/HjxwuU79y5E++9916RBEVERERvjt49AzNmzICPjw/u3buHvLw8/Pbbb4iOjsaWLVsQHBxcHDESEREZTPbvZkj70krvnoEePXpgz549OHToEMzNzTFjxgxcu3YNe/bsQceOHYsjRiIiIoNxNYF2hXrOQKtWrRASElLUsRAREVEJKPRDh86dO4dr164BeDaPoEmTJkUWFBERUVHjK4y10zsZuHv3Lj755BOcPHkS1tbWAIDk5GR88MEH+Pnnn1G5cuWijpGIiMhgfGuhdnrPGRg+fDiys7Nx7do1JCUlISkpCdeuXUNeXh6GDx9eHDESERFRMdK7Z+Do0aM4deoUXF1dxTJXV1esWLECrVq1KtLgiIiIilIp/s+9QfROBpycnF76cKHc3Fw4OjoWSVBERERFjcME2uk9TLBgwQKMHTsW586dE8vOnTuHL7/8EgsXLizS4IiIiIpK/gRCQ7bSSqdkoFy5crCxsYGNjQ2GDBmCyMhIuLu7Q6FQQKFQwN3dHefPn8fQoUOLO14iIqJ3QkBAAJo1awZLS0vY2tqiZ8+eiI6O1qiTkZEBX19flC9fHhYWFujduzcSEhI06sTGxsLb2xtmZmawtbXFpEmTkJOTo1EnLCwMjRs3hkKhgIuLCwIDA/WKVadhgqVLl+p1UiIiorfNmx4mOHr0KHx9fdGsWTPk5OTg66+/hqenJ65evQpzc3MAwPjx47F371788ssvsLKywpgxY9CrVy+cPHkSwLMheG9vb9jb2+PUqVOIj4/HoEGDULZsWcybNw8AEBMTA29vb4waNQpBQUEIDQ3F8OHD4eDgAC8vL93uTRAEQa+7e4uo1WpYWVlBUX8EZMby1zcgegfdPrqkpEMgKjapajVqOlVASkoKlEplsVwj/7tiwMZTkJtZFPo8WU/SEDTsg0LH+uDBA9ja2uLo0aNo3bo1UlJSULFiRWzbtg19+vQBAFy/fh116tRBeHg4mjdvjn379qFr1664f/8+7OzsAABr167FlClT8ODBA8jlckyZMgV79+7F5cuXxWv169cPycnJ2L9/v06x6T1n4HkZGRlQq9UaGxERUWn24vdeZmamTu1SUlIAADY2NgCAiIgIZGdno0OHDmKd2rVro0qVKggPDwcAhIeHo379+mIiAABeXl5Qq9W4cuWKWOf5c+TXyT+HLvROBtLT0zFmzBjY2trC3Nwc5cqV09iIiIjeRvmvMDZkA56tqrOyshK3gICA1147Ly8P48aNQ4sWLVCvXj0AgEqlglwuFx/gl8/Ozg4qlUqs83wikH88/9ir6qjVajx9+lSnz0bvpYWTJ0/GkSNHsGbNGgwcOBCrVq3CvXv38P333+O7777T93RERERvhExm2HMG8tvGxcVpDBMoFIrXtvX19cXly5dx4sSJwgdQjPROBvbs2YMtW7agbdu2GDJkCFq1agUXFxc4OzsjKCgIAwYMKI44iYiI3gpKpVKvOQNjxoxBcHAwjh07pvHIfnt7e2RlZSE5OVmjdyAhIQH29vZinTNnzmicL3+1wfN1XlyBkJCQAKVSCVNTU51i1HuYICkpCdWrVwfw7ANJSkoCALRs2RLHjh3T93RERERvxJt+hbEgCBgzZgx27dqFw4cPo1q1ahrHmzRpgrJlyyI0NFQsi46ORmxsLDw8PAAAHh4eiIqKQmJiolgnJCQESqUSbm5uYp3nz5FfJ/8cutA7GahevTpiYmIAPJvosGPHDgDPegxeHPcgIiJ6W+QPExiy6cPX1xdbt27Ftm3bYGlpCZVKBZVKJY7jW1lZYdiwYfDz88ORI0cQERGBIUOGwMPDA82bNwcAeHp6ws3NDQMHDsTFixdx4MABTJs2Db6+vuLwxKhRo3Dr1i1MnjwZ169fx+rVq7Fjxw6MHz9e51j1TgaGDBmCixcvAgC++uorrFq1CiYmJhg/fjwmTZqk7+mIiIhKpTVr1iAlJQVt27aFg4ODuG3fvl2ss2TJEnTt2hW9e/dG69atYW9vj99++008bmxsjODgYBgbG8PDwwOffvopBg0ahNmzZ4t1qlWrhr179yIkJAQNGzbEokWLsGHDBp2fMQAUwXMG7ty5g4iICLi4uKBBgwaGnEpvfM4ASQGfM0Cl2Zt8zsDQLX8Z/JyBHwa5F2usJUXvCYQvcnZ2hrOzc1HEQkREVGyKajVBaaRTMrB8+XKdT/jFF18UOhgiIqLiwrcWaqdTMrBkiW7dlDKZjMkAERHRO0anZCB/9cDbKjZsYakbvyHKd/+xbk8QI3oXpWflvL5SETGCYc/gN+j5/W85g+cMEBERvQs4TKBdaU50iIiISAfsGSAiIkmQyQAjriZ4KSYDREQkCUYGJgOGtH3bcZiAiIhI4gqVDBw/fhyffvopPDw8cO/ePQDAjz/++Na+mpGIiOhNv6joXaJ3MvDrr7/Cy8sLpqamuHDhAjIzMwEAKSkpmDdvXpEHSEREVBTyhwkM2UorvZOBb7/9FmvXrsX69etRtmxZsbxFixY4f/58kQZHRERExU/vCYTR0dFo3bp1gXIrKyskJycXRUxERERFju8m0E7vngF7e3vcuHGjQPmJEydQvXr1IgmKiIioqBnJZAZvpZXeycCIESPw5Zdf4q+//oJMJsP9+/cRFBSEiRMnYvTo0cURIxERkcGMimArrfQeJvjqq6+Ql5eH9u3b48mTJ2jdujUUCgUmTpyIsWPHFkeMREREVIz0TgZkMhm++eYbTJo0CTdu3EBaWhrc3NxgYWFRHPEREREVCc4Z0K7QTyCUy+Vwc3MryliIiIiKjREMG/c3QunNBvROBtq1a/fKBy8cPnzYoICIiIjozdI7GWjUqJHGfnZ2NiIjI3H58mX4+PgUVVxERERFisME2umdDCxZsuSl5f7+/khLSzM4ICIiouLAFxVpV2QrJT799FP88MMPRXU6IiIiekOK7BXG4eHhMDExKarTERERFSmZDAZNIOQwwXN69eqlsS8IAuLj43Hu3DlMnz69yAIjIiIqSpwzoJ3eyYCVlZXGvpGREVxdXTF79mx4enoWWWBERET0ZuiVDOTm5mLIkCGoX78+ypUrV1wxERERFTlOINROrwmExsbG8PT05NsJiYjonSMrgj+lld6rCerVq4dbt24VRyxERETFJr9nwJCttNI7Gfj2228xceJEBAcHIz4+Hmq1WmMjIiKid4vOcwZmz56NCRMmoEuXLgCA7t27azyWWBAEyGQy5ObmFn2UREREBuKcAe10TgZmzZqFUaNG4ciRI8UZDxERUbGQyWSvfLeOLu1LK52TAUEQAABt2rQptmCIiIjozdNraWFpzoqIiKh04zCBdnolA7Vq1XptQpCUlGRQQERERMWBTyDUTq9kYNasWQWeQEhERETvNr2SgX79+sHW1ra4YiEiIio2RjKZQS8qMqTt207nZIDzBYiI6F3GOQPa6fzQofzVBERERFS66NwzkJeXV5xxEBERFS8DJxCW4lcT6P8KYyIioneREWQwMuAb3ZC2bzsmA0REJAlcWqid3i8qIiIiotKFPQNERCQJXE2gHZMBIiKSBD5nQDsOExAREUkckwEiIpKE/AmEhmz6OHbsGLp16wZHR0fIZDLs3r1b4/jgwYPF1yrnb506ddKok5SUhAEDBkCpVMLa2hrDhg1DWlqaRp1Lly6hVatWMDExgZOTE+bPn6/3Z8NkgIiIJMEIMnGooFCbnksL09PT0bBhQ6xatUprnU6dOiE+Pl7cfvrpJ43jAwYMwJUrVxASEoLg4GAcO3YMI0eOFI+r1Wp4enrC2dkZERERWLBgAfz9/bFu3Tq9YuWcASIiomLQuXNndO7c+ZV1FAoF7O3tX3rs2rVr2L9/P86ePYumTZsCAFasWIEuXbpg4cKFcHR0RFBQELKysvDDDz9ALpejbt26iIyMxOLFizWShtdhzwAREUlCUQ0TqNVqjS0zM7PQMYWFhcHW1haurq4YPXo0Hj16JB4LDw+HtbW1mAgAQIcOHWBkZIS//vpLrNO6dWvI5XKxjpeXF6Kjo/H48WOd42AyQEREkmBUBBsAODk5wcrKStwCAgIKFU+nTp2wZcsWhIaG4v/+7/9w9OhRdO7cGbm5uQAAlUpV4E3BZcqUgY2NDVQqlVjHzs5Oo07+fn4dXXCYgIiISA9xcXFQKpXivkKhKNR5+vXrJ/5cv359NGjQADVq1EBYWBjat29vcJz6YM8AERFJwosz9wuzAYBSqdTYCpsMvKh69eqoUKECbty4AQCwt7dHYmKiRp2cnBwkJSWJ8wzs7e2RkJCgUSd/X9tchJdhMkBERJIgK4KtON29exePHj2Cg4MDAMDDwwPJycmIiIgQ6xw+fBh5eXlwd3cX6xw7dgzZ2dlinZCQELi6uqJcuXI6X5vJABERSYJBywoL8fTCtLQ0REZGIjIyEgAQExODyMhIxMbGIi0tDZMmTcLp06dx+/ZthIaGokePHnBxcYGXlxcAoE6dOujUqRNGjBiBM2fO4OTJkxgzZgz69esHR0dHAED//v0hl8sxbNgwXLlyBdu3b8eyZcvg5+en32ejV20iIiLSyblz5/Dee+/hvffeAwD4+fnhvffew4wZM2BsbIxLly6he/fuqFWrFoYNG4YmTZrg+PHjGsMOQUFBqF27Ntq3b48uXbqgZcuWGs8QsLKywsGDBxETE4MmTZpgwoQJmDFjhl7LCgFOICQiIgl5k28XaNu2LQRB0Hr8wIEDrz2HjY0Ntm3b9so6DRo0wPHjx/WO73lMBoiISBIK80jhF9uXVhwmICIikjj2DBARkSQ8vzywsO1LKyYDREQkCc8/RbCw7Uur0nxvREREpAP2DBARkSRwmEA7JgNERCQJhj5FsPSmAhwmICIikjz2DBARkSRwmEA7JgNERCQJXE2gHZMBIiKSBPYMaFeaEx0iIiLSAXsGiIhIEriaQDsmA0REJAl8UZF2HCYgIiKSOPYMEBGRJBhBBiMDOvsNafu2YzJARESSwGEC7ThMQEREJHHsGSAiIkmQ/fvHkPalFZMBIiKSBA4TaMdhAiIiIoljzwAREUmCzMDVBBwmICIiesdxmEA7JgNERCQJTAa045wBIiIiiWPPABERSQKXFmrHZICIiCTBSPZsM6R9acVhAiIiIoljzwAREUkChwm0YzJARESSwNUE2nGYgIiISOLYM0BERJIgg2Fd/aW4Y4DJABERSQNXE2jHYQIiIiKJY8/AO27xpgMIPnIR/9xJgImiLN5vUB3+Y3qgZlU7rW227TkN39lbNcoU8jJQnVxarLGu33EUK7aGIvGRGvVqVsL/TfoITepWBQA8TklHwLq9OHL6Ou4mPEZ5awt4t22Ar0d1hZWFabHGRW+3hIcpWLxhL46fjUZGZhaqOFbAtxP7ol4tJ61tgkPPY+MvYYi99xAW5iZo1aw2Jo7whrXSvNjiPHPxJuZ/vwc37qhgX9Ean/Vvjw89m4nH1/90GCEnoxAT9wAm8jJo5FYVfsO7oJqTbbHFRJq4mkC7Eu0ZOHbsGLp16wZHR0fIZDLs3r27JMN5J506fwPDP2qNgz9MxG8rxyA7Jxe9xq5E+tPMV7azNDfB9X3zxO3SH7MNimPbntPo+tlSrcd/OxiBaUt3Ycrwzgj7cQrq1ayE3mNX4UFSKgAg/kEKVA9SMPvLD3Hq56+xeuanCA2/ii/mBBkUF73bUlKf4NPxq1CmjDHWzh2GP9ZPwqSRXaF8RYJ4/koMpi74Gb293sfv6yZi8bSBiLoei5lLdhY6jnuqJNT1nKT1+N34JHw+bSPeb1gDv64Zj4EftsLMxTtx4ly0WOds1E180v0D/LRsDNZ/NxI5ubkYMXU9njzNKnRcpJ/81QSGbKVVifYMpKeno2HDhhg6dCh69epVkqG8s3au8NXYXz3zU9T0nIrIa3Fo0dhFazuZTAa7CkqtxzOzsvHt6j349WAEUlKfok4NB/iP7YGWTWoVKs7V2w5jUM8PMKC7BwBg8dR+OHjyCrb+EY7xgz3h5uKILfNHiPWrVa6IaaO74bMZW5CTk4syZYwLdV16t23cEQb7itaYO/Fjsayyg80r21y8egeV7Mrh0w9bivX7ejfHxh1hGvV27vsLm3cew11VEirZlcOAni3xSfcPChXn9r3hqGRvg8mfdQMA1KhihwuXY7Dlt2No2dQVALBu3giNNnMnfoxWfWfh6j930bRB9UJdl/Qjg2GTAEtxLlCyyUDnzp3RuXPnkgyh1FGnZQAAyinNXlkv/Wkm6nebjrw8AQ1rO2H6591Rp4aDeHzy/F9wPUaFDXOHwKGiFYKPXESfL1bj5E9fo0YV/bo1s7JzEHk9DuMHe4plRkZGaPO+K85GxbzyXizNTZgISNiR8Cto0cQV4+f8iHOXbsK2ghX6dfsAH3Vx19qmoZszlm7aj2NnrqFVs9p4lJyGg8ej0KpZbbFOcOh5rNx8EN+M6Yk6NSrh2s17mLlkJ0xN5Ojp2VTvOC9evYPmjWtqlLVo6orv1vyhtU1q+rN/q1aWr/63SvQmvFNzBjIzM5GZ+V/3t1qtLsFo3j55eXmYungn3BtWh5uLo9Z6Ls62WDl9AOq6VII67SlWbA2F17BFCN/+DSrZlUOcKglBwacRtWc2HCpaAwDGDuyA0PBrCNpzGjN8u+sV16PkNOTm5qGijaVGeUUbJf65naC1zYKN++DzYeH+p0alw934JGwPDodP79YY+cn/EBUdh4DVu1G2jLHWL+3Gdath/lefYMLcIGRlZSMnNw9tm7th2tgPxTorfzyISZ91RceW9QE86z24eScBv/x5ulDJwMPHqahgrfn3u3w5C6Q9yUBGZjZMFGU1juXl5eH/1v6B9+pWRc1q9npfjwrHCDIYGdDXb1SK+wbeqWQgICAAs2bNKukw3loT5+/AtZvx2Ld+/Cvrvd+gOt5/rlvy/YbV4f7RHAT+dhLfjO6KqzfuIzc3D816a84jyMzKgY3VswlYcaokePT9VjyWk5uH7JxcVG7tJ5aNH+KFCUO89L4PddpTfDxuDVyrOeCrkd56t6fSI08QUK9WZYwb+qwHsY5LJdy4rcKOveFav7Rv3ElAwOo/MHpAB7RoWgsPklKxaH0wZi/7FXMm9MWTp1mIu/8IMxb/ojGPIDc3D5bmJuJ+9xELcT/h8bMdQQAANO3+jXi8Sb1q+H7e8ELd17crd+Gf2yr8uPjzQrWnwuEwgXbvVDIwdepU+Pn992WjVqvh5KR9RrGUTJq/AweOX8af68ahkl05vdqWLWOMBq5OuHX3AQAg/UkmjI2NcGTLFBgba84xNTdVAAAcKljhWNBUsXzPkUjsORyJdXMGi2X5QxXlrS1gbGwkThbM9yBJDdvymvMWUtMz0OeL1bAwM8HWBSNQlkMEklbRxhI1qmiujKlexRYhJ6K0ttnw82G8V7cqhvZtCwBwrQ6YmsgxyG81vhjcCbJ/F4vPGtcH9WtX0WhrbPTf3/e13w5Ddk4uACDxUQoGT1yLX9f8l2g//7/9CuUs8TBZ8+/3o8dpsDAzKdAr8O3KXTh6+ho2L/oc9v/2vBGVtHcqGVAoFFAoFCUdxltFEARMXvAL9oZdxJ61X8K5UgW9z5Gbm4erN+6jYws3AEAD18rIzc3Dg8ep+OC9l09CLFPGGNWdKor7FctZwkRRVqMsn7xsGTSq7YSjZ6Ph3bYhgGfdpMfO/o3hH7UW66nTnqLPF6sgL1sG2xZ/VuCXKEnPe3WrIubfJDXf7bsP4fiKhPdpZhbKGGsmkcb/JgACBFQsp4RteSXi4h+ha/vGWs/z/DXK/JsUa/v31dDNGcfPXNcoO3X+bzR0+y/ZEAQBc1ftRujJywhcOOq1EyGpGLBrQCs+dOgdN/H/dmDHvrNYP2cwLMxMkPBQjYSHajzN+G+50qiZWzBr5e/i/vz1+3D49DXcvvsQF6/HYeSMzYhTJWFgj2fj8y7OdvioUzOM9v8Rew5H4s69h4i4chuLNx3AgROXCxXn5/3/hy27T+Gn4NOIjlHB77vtSH+aiQHdmgN4lgj0HrsK6U+zsGL6AKSmZYj3kpubZ8AnRO+yQb1a49K1O1j3Uyju3HuI4MMXsPPP0/ik239zSZZs/BNT5/8k7rdt7oZDJ6Lw855TiIt/hPNXYjBv9e+o7+oE2/JWAADfQZ7YsP0Itu46gdt3H+DvmHjsOnAWgTuPFirOj709cDf+ERauD8at2ET89McpHDh6CYN6/ZfszlmxC8Gh5zF/an+YmSrwIEmNB0lqZGRmF/LTIX3JiuBPaVWiPQNpaWm4ceOGuB8TE4PIyEjY2NigSpUqr2hJ+X749TgAoOuoZRrlq2Z8iv7/ftHeVSVpTJpJTn2CL+duQ+KjVFhbmqJhnSo4sNEPtav/t5pg1cxPsXDjfkxbtgvxickob22OpvWqwatVvULF2cuzCR4mp2He93uR+CgV9WtVws7lvuIwwaXoOJy7fBsA0PhDzXkhF3+fhSqO5Qt1XXq31Xd1wrKZPlj6wz6s2XoIle1tMGV0D43/0T9IUiM+MVnc/9CzGZ48ycS2P05hwbpgWJqbwr1RDfgN/2/+SZ/O7jBRyLHplzAs3BAMUxM5alV1wMB/lyPqq7KDDVZ/Owz/t/YPbN19AvYVrDDLr4+4rBAAtgeHAwAGT1yr0fbbiX01Hk5EVBJkgvDvzJgSEBYWhnbt2hUo9/HxQWBg4Gvbq9VqWFlZIeFRCpRK7Wvmid5l9x8/LekQiIpNaqoajWrYIyWl+H6P539XhEbGwsKy8NdIS1WjfaMqOsd67NgxLFiwABEREYiPj8euXbvQs2dP8bggCJg5cybWr1+P5ORktGjRAmvWrEHNmv8tU01KSsLYsWOxZ88eGBkZoXfv3li2bBksLCzEOpcuXYKvry/Onj2LihUrYuzYsZg8ebJe91aiwwRt27aFIAgFNl0SASIiIn3IimDTR/6D9VatWvXS4/Pnz8fy5cuxdu1a/PXXXzA3N4eXlxcyMjLEOgMGDMCVK1cQEhKC4OBgHDt2DCNHjhSPq9VqeHp6wtnZGREREViwYAH8/f2xbt06vWJ9pyYQEhERvSte9WA9QRCwdOlSTJs2DT169AAAbNmyBXZ2dti9ezf69euHa9euYf/+/Th79iyaNn22lHbFihXo0qULFi5cCEdHRwQFBSErKws//PAD5HI56tati8jISCxevFgjaXgdTiAkIiJpKKKuAbVarbE9/zA8XcXExEClUqFDhw5imZWVFdzd3REe/mx+SXh4OKytrcVEAAA6dOgAIyMj/PXXX2Kd1q1bQy6Xi3W8vLwQHR2Nx48f6xwPkwEiIpKEolpN4OTkBCsrK3ELCAjQOxaVSgUAsLPTfI6GnZ2deEylUsHWVvPx72XKlIGNjY1GnZed4/lr6ILDBEREJAmGvnkwv21cXJzGBMLS8Pwb9gwQERHpQalUamyFSQbs7Z+9kyIhQfP9LAkJCeIxe3t7JCYmahzPyclBUlKSRp2XneP5a+iCyQAREUnCm15N8CrVqlWDvb09QkNDxTK1Wo2//voLHh7PXvXu4eGB5ORkREREiHUOHz6MvLw8uLu7i3WOHTuG7Oz/Hl4VEhICV1dXlCun+6PpmQwQEZE0vOFsIC0tDZGRkYiMjATw34P1YmNjIZPJMG7cOHz77bf4448/EBUVhUGDBsHR0VF8FkGdOnXQqVMnjBgxAmfOnMHJkycxZswY9OvXD46Oz95M279/f8jlcgwbNgxXrlzB9u3bsWzZMo33+OiCcwaIiIiKwblz5zQerJf/BZ3/YL3JkycjPT0dI0eORHJyMlq2bIn9+/fDxOS/t2cGBQVhzJgxaN++vfjQoeXLl4vHrayscPDgQfj6+qJJkyaoUKECZsyYodeyQqCEn0BoKD6BkKSATyCk0uxNPoHwWNRdg59A2Lp+5WKNtaSwZ4CIiCShqFYTlEacM0BERCRx7BkgIiJJMHRFQCnuGGAyQEREEsFsQCsOExAREUkcewaIiEgSnn+/QGHbl1ZMBoiISBK4mkA7JgNERCQJnDKgHecMEBERSRx7BoiISBrYNaAVkwEiIpIETiDUjsMEREREEseeASIikgSuJtCOyQAREUkCpwxox2ECIiIiiWPPABERSQO7BrRiMkBERJLA1QTacZiAiIhI4tgzQEREksDVBNoxGSAiIknglAHtmAwQEZE0MBvQinMGiIiIJI49A0REJAlcTaAdkwEiIpIGAycQluJcgMMEREREUseeASIikgTOH9SOyQAREUkDswGtOExAREQkcewZICIiSeBqAu2YDBARkSTwccTacZiAiIhI4tgzQEREksD5g9oxGSAiImlgNqAVkwEiIpIETiDUjnMGiIiIJI49A0REJAkyGLiaoMgiefswGSAiIknglAHtOExAREQkcewZICIiSeBDh7RjMkBERBLBgQJtOExAREQkcewZICIiSeAwgXZMBoiISBI4SKAdhwmIiIgkjskAERFJQv4wgSGbPvz9/SGTyTS22rVri8czMjLg6+uL8uXLw8LCAr1790ZCQoLGOWJjY+Ht7Q0zMzPY2tpi0qRJyMnJKYqPQwOHCYiISBJK4t0EdevWxaFDh8T9MmX++9odP3489u7di19++QVWVlYYM2YMevXqhZMnTwIAcnNz4e3tDXt7e5w6dQrx8fEYNGgQypYti3nz5hX6Pl6GyQAREUlDCUwaKFOmDOzt7QuUp6SkYOPGjdi2bRv+97//AQA2bdqEOnXq4PTp02jevDkOHjyIq1ev4tChQ7Czs0OjRo0wZ84cTJkyBf7+/pDL5QbcjCYOExARERWTf/75B46OjqhevToGDBiA2NhYAEBERASys7PRoUMHsW7t2rVRpUoVhIeHAwDCw8NRv3592NnZiXW8vLygVqtx5cqVIo2TPQNERCQJRdUxoFarNcoVCgUUCkWB+u7u7ggMDISrqyvi4+Mxa9YstGrVCpcvX4ZKpYJcLoe1tbVGGzs7O6hUKgCASqXSSATyj+cfK0pMBoiISBKK6jkDTk5OGuUzZ86Ev79/gfqdO3cWf27QoAHc3d3h7OyMHTt2wNTUtPCBFAMmA0RERHqIi4uDUqkU91/WK/Ay1tbWqFWrFm7cuIGOHTsiKysLycnJGr0DCQkJ4hwDe3t7nDlzRuMc+asNXjYPwRCcM0BERJIgK4I/AKBUKjU2XZOBtLQ03Lx5Ew4ODmjSpAnKli2L0NBQ8Xh0dDRiY2Ph4eEBAPDw8EBUVBQSExPFOiEhIVAqlXBzcyvCT4Y9A0REJBVveDXBxIkT0a1bNzg7O+P+/fuYOXMmjI2N8cknn8DKygrDhg2Dn58fbGxsoFQqMXbsWHh4eKB58+YAAE9PT7i5uWHgwIGYP38+VCoVpk2bBl9fX50TEF0xGSAiIioGd+/exSeffIJHjx6hYsWKaNmyJU6fPo2KFSsCAJYsWQIjIyP07t0bmZmZ8PLywurVq8X2xsbGCA4OxujRo+Hh4QFzc3P4+Phg9uzZRR6rTBAEocjP+oao1WpYWVkh4VGKxvgNUWly//HTkg6BqNikpqrRqIY9UlKK7/d4/nfFrXuPYGnANVLValSvVL5YYy0p7BkgIiJJ4FsLteMEQiIiIoljzwAREUmEYe8mKM0vMWYyQEREksBhAu04TEBERCRxTAaIiIgkjsMEREQkCRwm0I7JABERSYLMwAmEhk0+fLtxmICIiEji2DNARESSwGEC7ZgMEBGRJLzh9xS9UzhMQEREJHHsGSAiImlg14BWTAaIiEgSuJpAOw4TEBERSRx7BoiISBK4mkA7JgNERCQJnDKgHZMBIiKSBmYDWnHOABERkcSxZ4CIiCSBqwm0YzJARESSwAmE2r3TyYAgCACAVLW6hCMhKj6pqU9LOgSiYpOWmgrgv9/nxUlt4HeFoe3fZu90MpD6718il2pOJRwJEREZIjU1FVZWVsVybrlcDnt7e9Qsgu8Ke3t7yOXyIojq7SIT3kQ6Vkzy8vJw//59WFpaQlaa+2/eImq1Gk5OToiLi4NSqSzpcIiKFP9+v3mCICA1NRWOjo4wMiq+Oe0ZGRnIysoy+DxyuRwmJiZFENHb5Z3uGTAyMkLlypVLOgxJUiqV/GVJpRb/fr9ZxdUj8DwTE5NS+SVeVLi0kIiISOKYDBAREUkckwHSi0KhwMyZM6FQKEo6FKIix7/fJFXv9ARCIiIiMhx7BoiIiCSOyQAREZHEMRkgIiKSOCYDREREEsdkgHS2atUqVK1aFSYmJnB3d8eZM2dKOiSiInHs2DF069YNjo6OkMlk2L17d0mHRPRGMRkgnWzfvh1+fn6YOXMmzp8/j4YNG8LLywuJiYklHRqRwdLT09GwYUOsWrWqpEMhKhFcWkg6cXd3R7NmzbBy5UoAz94L4eTkhLFjx+Krr74q4eiIio5MJsOuXbvQs2fPkg6F6I1hzwC9VlZWFiIiItChQwexzMjICB06dEB4eHgJRkZEREWByQC91sOHD5Gbmws7OzuNcjs7O6hUqhKKioiIigqTASIiIoljMkCvVaFCBRgbGyMhIUGjPCEhAfb29iUUFRERFRUmA/RacrkcTZo0QWhoqFiWl5eH0NBQeHh4lGBkRERUFMqUdAD0bvDz84OPjw+aNm2K999/H0uXLkV6ejqGDBlS0qERGSwtLQ03btwQ92NiYhAZGQkbGxtUqVKlBCMjejO4tJB0tnLlSixYsAAqlQqNGjXC8uXL4e7uXtJhERksLCwM7dq1K1Du4+ODwMDANx8Q0RvGZICIiEjiOGeAiIhI4pgMEBERSRyTASIiIoljMkBERCRxTAaIiIgkjskAERGRxDEZICIikjgmA0QGGjx4MHr27Cnut23bFuPGjXvjcYSFhUEmkyE5OVlrHZlMht27d+t8Tn9/fzRq1MiguG7fvg2ZTIbIyEiDzkNExYfJAJVKgwcPhkwmg0wmg1wuh4uLC2bPno2cnJxiv/Zvv/2GOXPm6FRXly9wIqLixncTUKnVqVMnbNq0CZmZmfjzzz/h6+uLsmXLYurUqQXqZmVlQS6XF8l1bWxsiuQ8RERvCnsGqNRSKBSwt7eHs7MzRo8ejQ4dOuCPP/4A8F/X/ty5c+Ho6AhXV1cAQFxcHPr27Qtra2vY2NigR48euH37tnjO3Nxc+Pn5wdraGuXLl8fkyZPx4hO9XxwmyMzMxJQpU+Dk5ASFQgEXFxds3LgRt2/fFp+HX65cOchkMgwePBjAs7dCBgQEoFq1ajA1NUXDhg2xc+dOjev8+eefqFWrFkxNTdGuXTuNOHU1ZcoU1KpVC2ZmZqhevTqmT5+O7OzsAvW+//57ODk5wczMDH379kVKSorG8Q0bNqBOnTowMTFB7dq1sXr1ar1jIaKSw2SAJMPU1BRZWVnifmhoKKKjoxESEoLg4GBkZ2fDy8sLlpaWOH78OE6ePAkLCwt06tRJbLdo0SIEBgbihx9+wIkTJ5CUlIRdu3a98rqDBg3CTz/9hOXLl+PatWv4/vvvYWFhAScnJ/z6668AgOjoaMTHx2PZsmUAgICAAGzZsgVr167FlStXMH78eHz66ac4evQogGdJS69evdCtWzdERkZi+PDh+Oqrr/T+TCwtLREYGIirV69i2bJlWL9+PZYsWaJR58aNG9ixYwf27NmD/fv348KFC/j888/F40FBQZgxYwbmzp2La9euYd68eZg+fTo2b96sdzxEVEIEolLIx8dH6NGjhyAIgpCXlyeEhIQICoVCmDhxonjczs5OyMzMFNv8+OOPgqurq5CXlyeWZWZmCqampsKBAwcEQRAEBwcHYf78+eLx7OxsoXLlyuK1BEEQ2rRpI3z55ZeCIAhCdHS0AEAICQl5aZxHjhwRAAiPHz8WyzIyMgQzMzPh1KlTGnWHDRsmfPLJJ4IgCMLUqVMFNzc3jeNTpkwpcK4XARB27dql9fiCBQuEJk2aiPszZ84UjI2Nhbt374pl+/btE4yMjIT4+HhBEAShRo0awrZt2zTOM2fOHMHDw0MQBEGIiYkRAAgXLlzQel0iKlmcM0ClVnBwMCwsLJCdnY28vDz0798f/v7+4vH69etrzBO4ePEibty4AUtLS43zZGRk4ObNm0hJSUF8fLzGa5vLlCmDpk2bFhgqyBcZGQljY2O0adNG57hv3LiBJ0+eoGPHjhrlWVlZeO+99wAA165dK/D6aA8PD52vkW/79u1Yvnw5bt68ibS0NOTk5ECpVGrUqVKlCipVqqRxnby8PERHR8PS0hI3b97EsGHDMGLECLFOTk4OrKys9I6HiEoGkwEqtdq1a4c1a9ZALpfD0dERZcpo/nU3NzfX2E9LS0OTJk0QFBRU4FwVK1YsVAympqZ6t0lLSwMA7N27V+NLGHg2D6KohIeHY8CAAZg1axa8vLxgZWWFn3/+GYsWLdI71vXr1xdIToyNjYssViIqXkwGqNQyNzeHi4uLzvUbN26M7du3w9bWtsD/jvM5ODjgr7/+QuvWrQE8+x9wREQEGjdu/NL69evXR15eHo4ePYoOHToUOJ7fM5GbmyuWubm5QaFQIDY2VmuPQp06dcTJkPlOnz79+pt8zqlTp+Ds7IxvvvlGLLtz506BerGxsbh//z4cHR3F6xgZGcHV1RV2dnZwdHTErVu3MGDAAL2uT0RvD04gJPrXgAEDUKFCBfTo0QPHjx9HTEwMwsLC8MUXX+Du3bsAgC+//BLfffcddu/ejevXr+Pzzz9/5TMCqlatCh8fHwwdOhS7d+8Wz7ljxw4AgLOzM2QyGYKDg/HgwQOkpaXB0tISEydOxPjx47F582bcvHkT58+fx4oVK8RJeaNGjcI///yDSZMmITo6Gtu2bUNgYKBe91uzZk3Exsbi559/xs2bN7F8+fKXToY0MTGBj48PLl68iOPHj+OLL75A3759YW9vDwCYNWsWAgICsHz5cvz999+IiorCpk2bsHjxYr3iIaKSw2SA6F9mZmY4duwYqlSpgl69eqFOnToYNmwYMjIyxJ6CCRMmYODAgfDx8YGHhwcsLS3x4YcfvvK8a9asQZ8+ffD555+jdu3aGDFiBNLT0wEAlSpVwqxZs/DVV1/Bzs4OY8aMAQDMmTMH06dPR0BAAOrUqYNOnTph7969qFatGoBn4/i//vordu/ejYYNG2Lt2rWYN2+eXvfbvXt3jB8/HmPGjEGjRo1w6tQpTJ8+vUA9FxcX9OrVC126dIGnpycaNGigsXRw+PDh2LBhAzZt2oT69eujTZs2CAwMFGMlorefTNA284mIiIgkgT0DREREEsdkgIiISOKYDBAREUkckwEiIiKJYzJAREQkcUwGiIiIJI7JABERkcQxGSAiIpI4JgNEREQSx2SAiIhI4pgMEBERSRyTASIiIon7f0XNc+8a2LoEAAAAAElFTkSuQmCC", - "text/plain": [ - "
    " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "import matplotlib.pyplot as plt\n", - "from sklearn.metrics import ConfusionMatrixDisplay\n", - "\n", - "disp = ConfusionMatrixDisplay(confusion_matrix=confusion_matrix_abs)\n", - "disp.plot(cmap=plt.cm.Blues)\n", - "plt.title('Confusion Matrix')\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "a9360ea0", - "metadata": {}, - "source": [ - "The Confusion Matrix created has four different quadrants:\n", - "\n", - "- True Negative (Top-Left Quadrant)\n", - "- False Positive (Top-Right Quadrant)\n", - "- False Negative (Bottom-Left Quadrant)\n", - "- True Positive (Bottom-Right Quadrant)\n", - "\n", - "True means that the values were accurately predicted, False means that there was an error or wrong prediction." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "90b88789", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "array([[3580., 488.],\n", - " [ 248., 684.]])" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "confusion_matrix_abs" - ] - }, - { - "cell_type": "markdown", - "id": "c42dc06f", - "metadata": {}, - "source": [ - "# Accuracy\n", - "\n", - "Accuracy measures how often the model is correct.\n", - "\n", - "(True Positive + True Negative) / Total Predictions\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "d69262cd", - "metadata": {}, - "outputs": [], - "source": [ - "TN, FP, FN, TP = confusion_matrix_abs[0,0], confusion_matrix_abs[0,1], confusion_matrix_abs[1,0], confusion_matrix_abs[1,1]" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "263ca554", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "np.float64(0.8528)" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# from systemds.operator.algorithm import getAccuracy\n", - "# accuracy = getAccuracy(Yt, y_pred).compute()\n", - "\n", - "\n", - "accuracy = (TP + TN) / test_count\n", - "accuracy" - ] - }, - { - "cell_type": "markdown", - "id": "54ea1ecf", - "metadata": {}, - "source": [ - "# Precision\n", - "Of the positives predicted, what percentage is truly positive?\n", - "\n", - "True Positive / (True Positive + False Positive)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "d7daf015", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "np.float64(0.5836177474402731)" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "precision = TP / (TP + FP)\n", - "precision" - ] - }, - { - "cell_type": "markdown", - "id": "9b24ea83", - "metadata": {}, - "source": [ - "# Sensitivity (Recall)\n", - "Of all the positive cases, what percentage are predicted positive?\n", - "\n", - "Sensitivity (sometimes called Recall) measures how good the model is at predicting positives.\n", - "\n", - "This means it looks at true positives and false negatives (which are positives that have been incorrectly predicted as negative).\n", - "\n", - "True Positive / (True Positive + False Negative)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "57e04ed3", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "np.float64(0.7339055793991416)" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "sensitivity = TP /(TP + FN)\n", - "sensitivity" - ] - }, - { - "cell_type": "markdown", - "id": "52abc77a", - "metadata": {}, - "source": [ - "# Specificity\n", - "How well the model is at prediciting negative results?\n", - "\n", - "Specificity is similar to sensitivity, but looks at it from the persepctive of negative results.\n", - "\n", - "True Negative / (True Negative + False Positive)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "3dcfa6b8", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "np.float64(0.880039331366765)" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "specificity = TN / (TN + FP)\n", - "specificity" - ] - }, - { - "cell_type": "markdown", - "id": "127fe71d", - "metadata": {}, - "source": [ - "# F-score\n", - "F-score is the \"harmonic mean\" of precision and sensitivity.\n", - "\n", - "It considers both false positive and false negative cases and is good for imbalanced datasets.\n", - "\n", - "2 * ((Precision * Sensitivity) / (Precision + Sensitivity))" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "bceed706", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "np.float64(0.6501901140684411)" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "f_score = 2 * ((precision * sensitivity) / (precision + sensitivity))\n", - "f_score" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "python_venv (3.12.3)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/src/main/python/notebooks/quick_start.ipynb b/src/main/python/notebooks/quick_start.ipynb deleted file mode 100644 index 5801f0882a0..00000000000 --- a/src/main/python/notebooks/quick_start.ipynb +++ /dev/null @@ -1,234 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "0213111f", - "metadata": {}, - "source": [ - "# Matrix Operations" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "f3d1b80e", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Using incubator modules: jdk.incubator.vector\n", - "26/05/06 14:56:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", - "INFO:simple_example:[[13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", - " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", - " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", - " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]\n", - " [13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02 13.02]]\n" - ] - } - ], - "source": [ - "import logging\n", - "\n", - "from systemds.context import SystemDSContext\n", - "\n", - "logger = logging.getLogger('simple_example')\n", - "logger.setLevel(logging.INFO)\n", - "\n", - "# Create a context and if necessary (no SystemDS py4j instance running)\n", - "# it starts a subprocess which does the execution in SystemDS\n", - "with SystemDSContext() as sds:\n", - " # Full generates a matrix completely filled with one number.\n", - " # Generate a 5x10 matrix filled with 4.2\n", - " m = sds.full((5, 10), 4.20)\n", - " # multiply with scalar. Nothing is executed yet!\n", - " m_res = m * 3.1\n", - " # Do the calculation in SystemDS by calling compute().\n", - " # The returned value is an numpy array that can be directly printed.\n", - " logger.info(m_res.compute())\n", - " # context will automatically be closed and process stopped" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "cadb888c", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Using incubator modules: jdk.incubator.vector\n", - "26/05/06 15:02:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", - "26/05/06 15:02:21 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", - "26/05/06 15:02:21 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", - "INFO:simple_example:[[ 50.01 125.05 160.05 32.01 204.04]\n", - " [ 37.01 81.03 268.04 110.02 6.03]\n", - " [ 39.01 244.04 430.05 140.04 368.04]\n", - " [246.03 154.02 267.03 104.04 72.02]\n", - " [ 14.02 186.02 475.05 168.02 315.05]]\n" - ] - } - ], - "source": [ - "import logging\n", - "\n", - "import numpy as np\n", - "from systemds.context import SystemDSContext\n", - "\n", - "logger = logging.getLogger('simple_example')\n", - "logger.setLevel(logging.INFO)\n", - "\n", - "# create a random array\n", - "m1 = np.array(np.random.randint(100, size=5 * 5) + 1.01, dtype=np.double)\n", - "m1 = m1.reshape(5, 5)\n", - "# create another random array\n", - "m2 = np.array(np.random.randint(5, size=5 * 5) + 1, dtype=np.double)\n", - "m2 = m2.reshape(5, 5)\n", - "\n", - "# Create a context\n", - "with SystemDSContext() as sds:\n", - " # element-wise matrix multiplication, note that nothing is executed yet!\n", - " m_res = sds.from_numpy(m1) * sds.from_numpy(m2)\n", - " # lets do the actual computation in SystemDS! The result is an numpy array\n", - " m_res_np = m_res.compute()\n", - " logger.info(m_res_np)" - ] - }, - { - "cell_type": "markdown", - "id": "87cc9e28", - "metadata": {}, - "source": [ - "# Algorithms as built-in functions" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "bf202fb1", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Using incubator modules: jdk.incubator.vector\n", - "26/05/06 15:05:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", - "26/05/06 15:05:20 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", - "26/05/06 15:05:20 WARNING SystemDSContext: Deprecated method from_numpy. Use from_py instead.\n", - "INFO:simple_example:[[ 0.02033445]\n", - " [-0.00324092]\n", - " [ 0.0014692 ]\n", - " [ 0.02649209]\n", - " [-0.00616902]\n", - " [-0.0095087 ]\n", - " [ 0.01039221]\n", - " [-0.0011352 ]\n", - " [-0.01686351]\n", - " [-0.03839821]]\n" - ] - } - ], - "source": [ - "import logging\n", - "\n", - "import numpy as np\n", - "from systemds.context import SystemDSContext\n", - "from systemds.operator.algorithm import l2svm\n", - "\n", - "logger = logging.getLogger('simple_example')\n", - "logger.setLevel(logging.INFO)\n", - "\n", - "# Set a seed\n", - "np.random.seed(0)\n", - "# Generate random features and labels in numpy\n", - "# This can easily be exchanged with a data set.\n", - "features = np.array(np.random.randint(\n", - " 100, size=10 * 10) + 1.01, dtype=np.double)\n", - "features = features.reshape(10, 10)\n", - "labels = np.zeros((10, 1))\n", - "\n", - "# l2svm labels can only be 0 or 1\n", - "for i in range(10):\n", - " if np.random.random() > 0.5:\n", - " labels[i][0] = 1\n", - "\n", - "# compute our model\n", - "with SystemDSContext() as sds:\n", - " model = l2svm(sds.from_numpy(features),\n", - " sds.from_numpy(labels), verbose=False).compute()\n", - " logger.info(model)" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "266e1c4b", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Using incubator modules: jdk.incubator.vector\n", - "26/05/06 15:07:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", - "INFO:simple_example:[[-0.02674835]\n", - " [-0.01684308]\n", - " [ 0.02429331]\n", - " [ 0.03332674]\n", - " [-0.00411165]\n", - " [ 0.00514915]\n", - " [ 0.01035419]\n", - " [ 0.00611115]\n", - " [-0.02550657]\n", - " [-0.0117999 ]]\n" - ] - } - ], - "source": [ - "import logging\n", - "\n", - "from systemds.context import SystemDSContext\n", - "from systemds.operator.algorithm import l2svm\n", - "\n", - "logger = logging.getLogger('simple_example')\n", - "logger.setLevel(logging.INFO)\n", - "\n", - "with SystemDSContext() as sds:\n", - " # Generate 10 by 10 matrix with values in range 0 to 100.\n", - " features2 = sds.rand(10, 10, 0, 100)\n", - " # Add value to all cells in features\n", - " features2 += 1.1\n", - " # Generate labels of all ones and zeros\n", - " labels2 = sds.rand(10, 1, 1, 1, sparsity=0.5)\n", - "\n", - " model2 = l2svm(features2, labels2, verbose=False).compute()\n", - " logger.info(model2)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "python_venv (3.12.3)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/src/main/python/requirements.txt b/src/main/python/requirements.txt deleted file mode 100644 index 3c7af412a4c..00000000000 --- a/src/main/python/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -numpy -scipy -py4j -wheel -requests -setuptools - -scikit-learn -matplotlib From 4159947ef8428d717e9c7032cdd1acdb6f2f670d Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Thu, 9 Jul 2026 23:35:44 +0200 Subject: [PATCH 21/26] dp_laplace and dp_gaussian now take the original matrix and build a transformation matrix T internally, returning T %*% X with noise fused into a single matrix multiply. --- .../sysds/hops/ParameterizedBuiltinOp.java | 10 +- .../parser/BuiltinFunctionExpression.java | 50 +++- .../apache/sysds/parser/DMLTranslator.java | 16 +- .../cp/DPBuiltinCPInstruction.java | 234 ++++++++++++------ .../privacy/dp/DPBuiltinDMLTest.java | 103 ++++++-- 5 files changed, 297 insertions(+), 116 deletions(-) diff --git a/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java b/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java index 7761521e415..3997d3402c8 100644 --- a/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java +++ b/src/main/java/org/apache/sysds/hops/ParameterizedBuiltinOp.java @@ -691,8 +691,14 @@ else if( _op == ParamBuiltinOp.TRANSFORMAPPLY ) { } } else if( _op == ParamBuiltinOp.DP_LAPLACE || _op == ParamBuiltinOp.DP_GAUSSIAN ) { - if( dc.dimsKnown() ) - ret = new MatrixCharacteristics(dc.getRows(), dc.getCols(), -1, dc.getLength()); + if( dc.dimsKnown() ) { + Hop query = getParameterHop("query"); + String queryVal = (query instanceof LiteralOp) ? ((LiteralOp)query).getStringValue() : null; + if( "colMeans".equals(queryVal) || "colSums".equals(queryVal) ) + ret = new MatrixCharacteristics(1, dc.getCols(), -1, dc.getCols()); + else if( "identity".equals(queryVal) ) + ret = new MatrixCharacteristics(dc.getRows(), dc.getCols(), -1, dc.getLength()); + } } return ret; diff --git a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java index 1425a794575..a75e5ca7b8f 100644 --- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java +++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java @@ -2006,28 +2006,34 @@ else if(this.getOpCode() == Builtins.MAX_POOL || this.getOpCode() == Builtins.AV else raiseValidateError("Local instruction not allowed in dml script"); case DP_LAPLACE: { - checkNumParameters(3); + checkNumParameters(4); checkMatrixParam(getFirstExpr()); checkScalarParam(getSecondExpr()); + checkValueTypeParam(getSecondExpr(), ValueType.STRING); checkScalarParam(getThirdExpr()); + checkScalarParam(getFourthExpr()); + String dpLaplaceQuery = getDPQueryLiteral(getSecondExpr()); + long[] dpLaplaceDims = getDPOutputDims(dpLaplaceQuery, + getFirstExpr().getOutput().getDim1(), getFirstExpr().getOutput().getDim2()); output.setDataType(DataType.MATRIX); output.setValueType(ValueType.FP64); - output.setDimensions( - getFirstExpr().getOutput().getDim1(), - getFirstExpr().getOutput().getDim2()); + output.setDimensions(dpLaplaceDims[0], dpLaplaceDims[1]); break; } case DP_GAUSSIAN: { - checkNumParameters(4); + checkNumParameters(5); checkMatrixParam(getFirstExpr()); checkScalarParam(getSecondExpr()); + checkValueTypeParam(getSecondExpr(), ValueType.STRING); checkScalarParam(getThirdExpr()); checkScalarParam(getFourthExpr()); + checkScalarParam(getFifthExpr()); + String dpGaussianQuery = getDPQueryLiteral(getSecondExpr()); + long[] dpGaussianDims = getDPOutputDims(dpGaussianQuery, + getFirstExpr().getOutput().getDim1(), getFirstExpr().getOutput().getDim2()); output.setDataType(DataType.MATRIX); output.setValueType(ValueType.FP64); - output.setDimensions( - getFirstExpr().getOutput().getDim1(), - getFirstExpr().getOutput().getDim2()); + output.setDimensions(dpGaussianDims[0], dpGaussianDims[1]); break; } case COMPRESS: @@ -2136,6 +2142,34 @@ else if(this.getOpCode() == Builtins.MAX_POOL || this.getOpCode() == Builtins.AV } } + /** + * dp_laplace/dp_gaussian require the "query" parameter to be a compile-time + * string literal so that the output shape (and thus the transformation + * matrix T built at runtime) is known during validation. + */ + private String getDPQueryLiteral(Expression queryExpr) { + if (!(queryExpr instanceof StringIdentifier)) + raiseValidateError(getOpCode() + ": 'query' must be a string literal", false, + LanguageErrorCodes.INVALID_PARAMETERS); + return ((StringIdentifier) queryExpr).getValue(); + } + + /** Output dimensions of T %*% X for the given named query, X being n x d. */ + private long[] getDPOutputDims(String query, long n, long d) { + switch (query) { + case "colMeans": + case "colSums": + return new long[] {1, d}; + case "identity": + return new long[] {n, d}; + default: + raiseValidateError(getOpCode() + ": unknown query type '" + query + + "' (expected colMeans, colSums, or identity)", false, + LanguageErrorCodes.INVALID_PARAMETERS); + return null; // unreachable + } + } + private void validateEinsum(DataIdentifier output){ if(getSecondExpr() == null) raiseValidateError("Einsum: at least one input matrix required", false, diff --git a/src/main/java/org/apache/sysds/parser/DMLTranslator.java b/src/main/java/org/apache/sysds/parser/DMLTranslator.java index 7950868e0b5..06df5fec357 100644 --- a/src/main/java/org/apache/sysds/parser/DMLTranslator.java +++ b/src/main/java/org/apache/sysds/parser/DMLTranslator.java @@ -2314,6 +2314,10 @@ private Hop processBuiltinFunctionExpression(BuiltinFunctionExpression source, D if (source.getFourthExpr() != null) { expr4 = processExpression(source.getFourthExpr(), null, hops); } + Hop expr5 = null; + if (source.getFifthExpr() != null) { + expr5 = processExpression(source.getFifthExpr(), null, hops); + } Hop currBuiltinOp = null; target = (target == null) ? createTarget(source) : target; @@ -2596,8 +2600,9 @@ else if ( sop.equalsIgnoreCase(Opcodes.NOTEQUAL.toString()) ) case DP_LAPLACE: { LinkedHashMap dpLaplaceParams = new LinkedHashMap<>(); dpLaplaceParams.put("target", expr); - dpLaplaceParams.put("sensitivity", expr2); - dpLaplaceParams.put("epsilon", expr3); + dpLaplaceParams.put("query", expr2); + dpLaplaceParams.put("sensitivity", expr3); + dpLaplaceParams.put("epsilon", expr4); currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, ValueType.FP64, ParamBuiltinOp.DP_LAPLACE, dpLaplaceParams); break; @@ -2605,9 +2610,10 @@ else if ( sop.equalsIgnoreCase(Opcodes.NOTEQUAL.toString()) ) case DP_GAUSSIAN: { LinkedHashMap dpGaussianParams = new LinkedHashMap<>(); dpGaussianParams.put("target", expr); - dpGaussianParams.put("sensitivity", expr2); - dpGaussianParams.put("epsilon", expr3); - dpGaussianParams.put("delta", expr4); + dpGaussianParams.put("query", expr2); + dpGaussianParams.put("sensitivity", expr3); + dpGaussianParams.put("epsilon", expr4); + dpGaussianParams.put("delta", expr5); currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, ValueType.FP64, ParamBuiltinOp.DP_GAUSSIAN, dpGaussianParams); break; diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java index 63938f21881..9d49f959e13 100755 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -21,49 +21,49 @@ import org.apache.sysds.runtime.DMLRuntimeException; import org.apache.sysds.runtime.controlprogram.context.ExecutionContext; -import org.apache.sysds.runtime.functionobjects.Plus; import org.apache.sysds.runtime.instructions.InstructionUtils; +import org.apache.sysds.runtime.matrix.data.LibMatrixMult; +import org.apache.sysds.runtime.matrix.data.LibMatrixReorg; import org.apache.sysds.runtime.matrix.data.MatrixBlock; -import org.apache.sysds.runtime.matrix.operators.BinaryOperator; import org.apache.sysds.runtime.privacy.dp.DPBudgetAccountant; import java.util.LinkedHashMap; import java.util.concurrent.ThreadLocalRandom; /** - * CP instruction for differential-privacy release of an already-computed - * aggregate. + * CP instruction for differential-privacy release of a linear query over the + * original matrix. * - *

    DML syntax (post-aggregate form): + *

    DML syntax (raw-matrix form): * 

    - *   result = dp_laplace(aggregate, sensitivity=1.0, epsilon=0.5)
- *   result = dp_gaussian(aggregate, sensitivity=1.0, epsilon=0.5, delta=1e-5)
+ *   result = dp_laplace(X, query="colMeans", sensitivity=1.0, epsilon=0.5)
+ *   result = dp_gaussian(X, query="colMeans", sensitivity=1.0, epsilon=0.5, delta=1e-5)
  *
    * + *

    The instruction receives the original {@code n x d} matrix {@code X}, + * builds a transformation matrix {@code T} ({@code k x n}) from the named + * {@code query} (see {@link #buildTransform}), and returns a noisy release of + * {@code T %*% X}. The noise is not added as a separate elementwise + * pass over a materialised aggregate: it is injected by augmenting {@code T} + * with an identity block and {@code X} with the noise matrix, so that the + * noisy release is the result of a single {@link LibMatrixMult#matrixMult} + * call (see {@link #processInstruction} for the derivation). + * *

    Sensitivity norm: {@code sensitivity} is not interchangeable * between the two builtins. {@code dp_laplace} calibrates its noise scale - * to the L1 sensitivity of {@code aggregate} to a single-record + * to the L1 sensitivity of {@code T %*% X} to a single-record * change; {@code dp_gaussian} calibrates its σ to the L2 sensitivity. - * For a scalar aggregate (e.g. a single sum or mean) the two norms coincide, - * but for a vector-valued aggregate (e.g. column means of a multi-column - * matrix) they generally differ (L2 ≤ L1 ≤ √d·L2 for d entries) — the caller - * is responsible for supplying the norm matching the builtin invoked (see - * {@link #sensitivityOf}). - * - *

    The instruction receives a materialised matrix (the aggregate result), - * injects calibrated noise element-wise, records the release with the - * session-scoped {@link DPBudgetAccountant}, and returns the noisy matrix. - * - *

    Noise is generated in Java and added via a {@code MatrixBlock} binary - * operation so that the output allocation path is identical to every other - * CP instruction (no special memory-management required). + * For a scalar release (e.g. {@code query="colMeans"} on single-column + * {@code X}) the two norms coincide, but for a vector- or matrix-valued + * release they generally differ — the caller is responsible for supplying + * the norm matching the builtin invoked (see {@link #sensitivityOf}). * *

    The {@link #sensitivityOf} method is deliberately separated from the - * noise-scale computation. In Phase 1 it returns the caller-supplied - * constant. In the future HOP-level rewrite pass (Phase 2) the body of this - * single method is replaced with a static analysis that reads the - * sensitivity bound computed by the compiler; every other line in this class - * stays unchanged. + * noise-scale computation. It currently returns the caller-supplied + * constant. A future rewrite pass could replace the body of this single + * method with a static analysis that derives sensitivity from {@code T}'s + * column norms and a declared per-record bound on {@code X}; every other + * line in this class would stay unchanged. */ public class DPBuiltinCPInstruction extends ComputationCPInstruction { @@ -81,7 +81,7 @@ public class DPBuiltinCPInstruction extends ComputationCPInstruction { /** * Named parameters extracted from the serialised instruction string. - * Keys: "target", "sensitivity", "epsilon", "delta" (Gaussian only). + * Keys: "target", "query", "sensitivity", "epsilon", "delta" (Gaussian only). * * Using the same LinkedHashMap convention as * ParameterizedBuiltinCPInstruction so that CPInstructionParser can @@ -113,9 +113,9 @@ private DPBuiltinCPInstruction( * *

    Expected format (OPERAND_DELIM = '\u00b0'): * 

    -     *   dp_gaussian°target=mVar1·MATRIX·FP64°sensitivity=1.0·SCALAR·FP64·true
-     *              °epsilon=0.5·SCALAR·FP64·true°delta=1e-5·SCALAR·FP64·true
-     *              °_mVar2·MATRIX·FP64
+     *   dp_gaussian°target=mVar1·MATRIX·FP64°query=colMeans·SCALAR·STRING·true
+     *              °sensitivity=1.0·SCALAR·FP64·true°epsilon=0.5·SCALAR·FP64·true
+     *              °delta=1e-5·SCALAR·FP64·true°_mVar2·MATRIX·FP64
      *
    * * The first token is always the opcode; the last token is always the @@ -124,7 +124,7 @@ private DPBuiltinCPInstruction( */ public static DPBuiltinCPInstruction parseInstruction(String str) { String[] parts = InstructionUtils.getInstructionPartsWithValueType(str); - InstructionUtils.checkNumFields(parts, 4, 5); // laplace=4, gaussian=5 + InstructionUtils.checkNumFields(parts, 5, 6); // laplace=5, gaussian=6 String opcode = parts[0]; // Output operand is always the last token. @@ -143,6 +143,8 @@ public static DPBuiltinCPInstruction parseInstruction(String str) { org.apache.sysds.common.Types.DataType.MATRIX); // Validate required keys. + if (!params.containsKey("query")) + throw new DMLRuntimeException(opcode + ": missing 'query'"); if (!params.containsKey("sensitivity")) throw new DMLRuntimeException(opcode + ": missing 'sensitivity'"); if (!params.containsKey("epsilon")) @@ -161,81 +163,162 @@ public static DPBuiltinCPInstruction parseInstruction(String str) { * Executes the DP release. * *
      - *
    1. Read the aggregate {@link MatrixBlock} from the variable table.
      2. - *
    2. Determine sensitivity via {@link #sensitivityOf} (Phase-1 stub).
      3. - *
    3. Generate a noise {@link MatrixBlock} of the same shape.
      4. - *
    4. Add noise element-wise using the existing binary-operator path.
      5. + *
    5. Read the original {@link MatrixBlock} {@code X} from the variable + * table.
      6. + *
    6. Build the transformation matrix {@code T} ({@code k x n}) from + * {@code query} (see {@link #buildTransform}).
      7. + *
    7. Determine sensitivity via {@link #sensitivityOf}.
      8. + *
    8. Generate a noise {@link MatrixBlock} shaped {@code k x d}.
      9. + *
    9. Fuse {@code T %*% X + noise} into a single + * {@link LibMatrixMult#matrixMult} call (see below).
      10. *
    10. Record the release with the session-scoped * {@link DPBudgetAccountant}; throw if budget is exhausted.
      11. *
    11. Write the noisy block back to the variable table and release * the input pin.
      12. *
    + * + *

    Fusion derivation: for {@code T} ({@code k x n}), {@code X} + * ({@code n x d}) and noise {@code N} ({@code k x d}), let + * {@code T' = [T | I_k]} ({@code k x (n+k)}) and + * {@code X' = [X ; N]} ({@code (n+k) x d}). Then + * {@code T' %*% X' = T %*% X + I_k %*% N = T %*% X + N}, computed as one + * matrix multiply instead of a multiply followed by a separate + * elementwise add. */ @Override public void processInstruction(ExecutionContext ec) { - // ── 1. Read aggregate input ───────────────────────────────────────── + // ── 1. Read original input matrix X ───────────────────────────────── // getMatrixInput pins the block in memory and increments the // reference count; we must call releaseMatrixInput afterwards. - MatrixBlock inBlock = ec.getMatrixInput(_params.get("target")); + MatrixBlock X = ec.getMatrixInput(_params.get("target")); // ── 2. Parse DP parameters ────────────────────────────────────────── double epsilon = parsePositiveDouble("epsilon"); double delta = instOpcode.equals(OPCODE_GAUSSIAN) ? parsePositiveDouble("delta") : 0.0; + String query = _params.get("query"); + + // ── 3. Build the transformation matrix T (k x n) ──────────────────── + MatrixBlock T = buildTransform(query, X.getNumRows()); - // ── 3. Determine sensitivity (Phase-1: caller-supplied constant) ──── - double sensitivity = sensitivityOf(inBlock); + // ── 4. Determine sensitivity (caller-supplied constant) ───────────── + double sensitivity = sensitivityOf(T); - // ── 4. Generate and add noise ──────────────────────────────────────── - MatrixBlock noiseBlock = generateNoise(inBlock, sensitivity, epsilon, delta); + // ── 5. Generate noise shaped like the release T %*% X (k x d) ─────── + MatrixBlock noiseBlock = generateNoise(T.getNumRows(), X.getNumColumns(), + sensitivity, epsilon, delta); - // Element-wise addition via the standard binary-operator path. - // binaryOperations allocates the output block internally. - BinaryOperator plusOp = new BinaryOperator(Plus.getPlusFnObject()); - MatrixBlock outBlock = new MatrixBlock(); - inBlock.binaryOperations(plusOp, noiseBlock, outBlock); + // ── 6. Fuse T %*% X + noise into a single matrix multiply ─────────── + MatrixBlock Ik = identity(T.getNumRows()); + MatrixBlock Tp = T.append(Ik, null, true); // [T | I_k] + MatrixBlock Xp = X.append(noiseBlock, null, false); // [X ; noise] + MatrixBlock outBlock = LibMatrixMult.matrixMult(Tp, Xp); - // ── 5. Record release and enforce budget ──────────────────────────── + // ── 7. Record release and enforce budget ──────────────────────────── // getDPBudgetAccountant() returns a lazy-initialised DPBudgetAccountant that is // owned by this ExecutionContext (added in a companion EC patch). DPBudgetAccountant accountant = ec.getDPBudgetAccountant(); accountant.compose(epsilon, delta, sensitivity); // throws on exhaustion - // ── 6. Write output and release input pin ─────────────────────────── + // ── 8. Write output and release input pin ─────────────────────────── ec.releaseMatrixInput(_params.get("target")); ec.setMatrixOutput(output.getName(), outBlock); } // ----------------------------------------------------------------------- - // Sensitivity seam (Phase-1 stub; Phase-2 replaces this body only) + // Transformation matrix construction // ----------------------------------------------------------------------- /** - * Returns the sensitivity of {@code aggregate} to a single-record change, - * in the norm required by the mechanism actually invoked: L1 for - * {@code dp_laplace}, L2 for {@code dp_gaussian} (see the class - * Javadoc). The two only coincide when {@code aggregate} is scalar. + * Builds the {@code k x n} transformation matrix {@code T} for the given + * named query, to be left-multiplied against the {@code n x d} input + * {@code X} as {@code T %*% X}. * - *

    Phase 1 (now): returns the caller-supplied literal from the - * DML script as-is, with no norm conversion or validation — the DML - * author must compute the sensitivity in the correct norm for the - * builtin they call. Sensitivity analysis is the caller's responsibility. + *

      + *
    • {@code "colMeans"}: {@code T} is {@code 1 x n}, filled with + * {@code 1/n} — {@code T %*% X} is the column-mean row vector.
      • + *
    • {@code "colSums"}: {@code T} is {@code 1 x n}, filled with + * {@code 1.0} — {@code T %*% X} is the column-sum row vector.
      • + *
    • {@code "identity"}: {@code T} is the {@code n x n} identity + * (built sparsely via {@link #identity}) — {@code T %*% X} is + * {@code X} itself, i.e. a noisy release of the raw matrix.
      • + *
    + * + *

    Row-wise aggregates ({@code rowMeans}/{@code rowSums}) reduce across + * the feature axis of {@code X}, i.e. they are naturally + * {@code X %*% T'} (right-multiply), not {@code T %*% X}, so they are + * intentionally not supported here. + */ + private static MatrixBlock buildTransform(String query, int n) { + switch (query) { + case "colMeans": { + MatrixBlock T = new MatrixBlock(1, n, false); + T.allocateDenseBlock(); + double v = 1.0 / n; + for (int c = 0; c < n; c++) + T.set(0, c, v); + T.recomputeNonZeros(); + return T; + } + case "colSums": { + MatrixBlock T = new MatrixBlock(1, n, false); + T.allocateDenseBlock(); + for (int c = 0; c < n; c++) + T.set(0, c, 1.0); + T.recomputeNonZeros(); + return T; + } + case "identity": + return identity(n); + default: + throw new DMLRuntimeException( + "dp_laplace/dp_gaussian: unknown query type '" + query + + "' (expected colMeans, colSums, or identity)"); + } + } + + /** + * Builds a {@code k x k} identity matrix, sparsely, by reusing the + * existing {@link LibMatrixReorg#diag} reorg operator (the same runtime + * path DML's {@code diag()} builtin uses to expand a vector into a + * diagonal matrix). Keeps memory {@code O(k)} rather than {@code O(k^2)}, + * which matters for the {@code query="identity"} case where {@code k} + * equals the number of rows of {@code X}. + */ + private static MatrixBlock identity(int k) { + MatrixBlock ones = new MatrixBlock(k, 1, false); + ones.allocateDenseBlock(); + for (int i = 0; i < k; i++) + ones.set(i, 0, 1.0); + ones.recomputeNonZeros(); + return LibMatrixReorg.diag(ones, new MatrixBlock(k, k, true)); + } + + // ----------------------------------------------------------------------- + // Sensitivity seam + // ----------------------------------------------------------------------- + + /** + * Returns the sensitivity of the release {@code T %*% X} to a + * single-record change, in the norm required by the mechanism actually + * invoked: L1 for {@code dp_laplace}, L2 for + * {@code dp_gaussian} (see the class Javadoc). The two only coincide + * when the release is scalar. * - *

    Phase 2 (HOP-level rewrite pass): replace this body with a - * call that inspects the HOP node that produced {@code aggregate}, reads - * the {@code sensitivityBound} field computed during compilation (in the - * norm matching {@code instOpcode}), and returns it. No other line in - * this class changes. + *

    Returns the caller-supplied literal from the DML script as-is, with + * no norm conversion or validation — the DML author must compute the + * sensitivity in the correct norm for the builtin they call. A future + * rewrite pass could replace this body with an analysis that derives + * sensitivity from {@code T}'s column norms and a declared per-record + * bound on {@code X}; no other line in this class would need to change. * - * @param aggregate the already-computed aggregate block (ignored in - * Phase 1; used in Phase 2 to look up lineage) + * @param T the transformation matrix (unused for now; kept as the seam + * for a future sensitivity-derivation pass) * @return caller-supplied sensitivity constant, expected to already be * in the L1 norm (Laplace) or L2 norm (Gaussian) */ - private double sensitivityOf(MatrixBlock aggregate) { - // Phase 1: unwrap the literal or variable value from the param map. - // In Phase 2, replace the body below with HOP-annotation lookup. + private double sensitivityOf(MatrixBlock T) { return parsePositiveDouble("sensitivity"); } @@ -244,23 +327,22 @@ private double sensitivityOf(MatrixBlock aggregate) { // ----------------------------------------------------------------------- /** - * Generates a noise {@link MatrixBlock} of the same shape as - * {@code aggregate}, filled with samples from the mechanism-appropriate - * distribution calibrated to ({@code sensitivity}, {@code epsilon}, - * {@code delta}). + * Generates a {@code rows x cols} noise {@link MatrixBlock} — matching + * the shape of the release {@code T %*% X} — filled with samples from the + * mechanism-appropriate distribution calibrated to ({@code sensitivity}, + * {@code epsilon}, {@code delta}). * *

    Both mechanisms produce a dense block. Sparsity exploitation is - * left for future work; for the aggregate outputs targeted here (e.g. - * column means, row sums) the aggregate is already dense. + * left for future work; for the releases targeted here (e.g. column + * means, column sums) the noise is dense regardless. */ private MatrixBlock generateNoise( - MatrixBlock aggregate, + int rows, + int cols, double sensitivity, double epsilon, double delta) { - int rows = aggregate.getNumRows(); - int cols = aggregate.getNumColumns(); MatrixBlock noise = new MatrixBlock(rows, cols, false); // dense noise.allocateDenseBlock(); diff --git a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java index 048c9501e21..32a02045753 100644 --- a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java +++ b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java @@ -14,6 +14,7 @@ package org.apache.sysds.test.functions.privacy.dp; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import java.io.File; @@ -33,15 +34,19 @@ public class DPBuiltinDMLTest extends AutomatedTestBase { private static final String TEST_CLASS = TEST_DIR + DPBuiltinDMLTest.class.getSimpleName() + "/"; private static final int ROWS = 100; private static final int COLS = 10; - private static final String DML_LAPLACE = + + private static final String DML_LAPLACE_TEMPLATE = "X = read($1);\n" - + "result = dp_laplace(colMeans(X), sensitivity=1.0, epsilon=$2);\n" + + "result = dp_laplace(X, query=\"%s\", sensitivity=1.0, epsilon=$2);\n" + "write(result, $3, format=\"text\");\n"; - private static final String DML_GAUSSIAN = + private static final String DML_GAUSSIAN_TEMPLATE = "X = read($1);\n" - + "result = dp_gaussian(colMeans(X), sensitivity=1.0, epsilon=$2, delta=1e-5);\n" + + "result = dp_gaussian(X, query=\"%s\", sensitivity=1.0, epsilon=$2, delta=1e-5);\n" + "write(result, $3, format=\"text\");\n"; + private static final String DML_LAPLACE = String.format(DML_LAPLACE_TEMPLATE, "colMeans"); + private static final String DML_GAUSSIAN = String.format(DML_GAUSSIAN_TEMPLATE, "colMeans"); + @Override public void setUp() { addTestConfiguration("DPLaplace", new TestConfiguration(TEST_CLASS, "DPLaplace")); @@ -50,17 +55,46 @@ public void setUp() { @Test public void testLaplaceOutputDiffersFromCleanMean() { - runDPTest("DPLaplace", DML_LAPLACE, "0.5"); + runColMeansDPTest("DPLaplace", DML_LAPLACE, "0.5"); } @Test public void testGaussianOutputDiffersFromCleanMean() { - runDPTest("DPGaussian", DML_GAUSSIAN, "0.5"); + runColMeansDPTest("DPGaussian", DML_GAUSSIAN, "0.5"); + } + + @Test + public void testLaplaceColSums() { + // query="colSums": T is 1 x n filled with 1.0, output is the noisy column-sum row vector. + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + HashMap result = runAndGetResult("DPLaplace", + String.format(DML_LAPLACE_TEMPLATE, "colSums"), "0.5", data); + assertShape(result, 1, COLS); + double maxDiff = maxAbsDiffFromClean(data, result, DPBuiltinDMLTest::colSum); + assertTrue("Result should differ from the clean column sums", maxDiff > 0); + } + + @Test + public void testGaussianIdentity() { + // query="identity": T is the n x n identity, output is a noisy release of X itself. + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + HashMap result = runAndGetResult("DPGaussian", + String.format(DML_GAUSSIAN_TEMPLATE, "identity"), "0.5", data); + assertShape(result, ROWS, COLS); + // identity releases X row-by-row, so compare cell-by-cell rather than via a per-column reduction. + double maxCellDiff = 0; + for (int r = 0; r < ROWS; r++) { + for (int c = 0; c < COLS; c++) { + double noisy = result.get(new CellIndex(r + 1, c + 1)); + maxCellDiff = Math.max(maxCellDiff, Math.abs(noisy - data[r][c])); + } + } + assertTrue("Result should differ from the clean matrix", maxCellDiff > 0); } @Test public void testHighEpsilonIsCloserToTruth() { - double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); // Higher ε → less noise → result closer to the true mean. // NOTE: the DPBudgetAccountant caps total spend at the default budget // (ε = 1.0) regardless of the per-release ε requested, so ε values @@ -70,42 +104,62 @@ public void testHighEpsilonIsCloserToTruth() { assertTrue("ε=0.5 should give less noise than ε=0.1", noisyHigh < noisyLow); } - private void runDPTest(String testName, String dml, String epsilonStr) { + private void runColMeansDPTest(String testName, String dml, String epsilonStr) { double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); HashMap result = runAndGetResult(testName, dml, epsilonStr, data); + assertShape(result, 1, COLS); + // Must differ from the exact (clean) mean by a non-trivial amount. + // (A single-seed exact-equality check is fragile; use range check.) + double maxDiff = maxAbsDiffFromClean(data, result, DPBuiltinDMLTest::colMean); + assertTrue("Result should differ from the clean mean", maxDiff > 0); + } + + private double runAndGetMaxAbsColMeansDiffFromClean(double[][] data, String testName, String dml, String epsilonStr) { + HashMap result = runAndGetResult(testName, dml, epsilonStr, data); + return maxAbsDiffFromClean(data, result, DPBuiltinDMLTest::colMean); + } - // The noisy result should be a (1 × cols) row vector. + private static void assertShape(HashMap result, int expectedRows, int expectedCols) { int maxRow = 0, maxCol = 0; for (CellIndex ci : result.keySet()) { maxRow = Math.max(maxRow, ci.row); maxCol = Math.max(maxCol, ci.column); } - assertTrue("Result should have 1 row", maxRow == 1); - assertTrue("Result should have " + COLS + " columns", maxCol == COLS); - // Must differ from the exact (clean) mean by a non-trivial amount. - // (A single-seed exact-equality check is fragile; use range check.) - double maxDiff = maxAbsColMeansDiffFromClean(data, result); - assertTrue("Result should differ from the clean mean", maxDiff > 0); + assertEquals("Result should have " + expectedRows + " row(s)", expectedRows, maxRow); + assertEquals("Result should have " + expectedCols + " column(s)", expectedCols, maxCol); } - private double runAndGetMaxAbsColMeansDiffFromClean(double[][] data, String testName, String dml, String epsilonStr) { - HashMap result = runAndGetResult(testName, dml, epsilonStr, data); - return maxAbsColMeansDiffFromClean(data, result); + @FunctionalInterface + private interface CleanColumnFn { + double apply(double[][] data, int col); } - private static double maxAbsColMeansDiffFromClean(double[][] data, HashMap result) { + /** Computes max|noisy(1,c) - clean(data,c)| across the (1 x COLS) row-vector releases. */ + private static double maxAbsDiffFromClean(double[][] data, HashMap result, + CleanColumnFn cleanFn) { double maxDiff = 0; for (int c = 0; c < COLS; c++) { - double sum = 0; - for (int r = 0; r < ROWS; r++) - sum += data[r][c]; - double cleanMean = sum / ROWS; + double clean = cleanFn.apply(data, c); double noisy = result.get(new CellIndex(1, c + 1)); - maxDiff = Math.max(maxDiff, Math.abs(noisy - cleanMean)); + maxDiff = Math.max(maxDiff, Math.abs(noisy - clean)); } return maxDiff; } + private static double colMean(double[][] data, int c) { + double sum = 0; + for (int r = 0; r < ROWS; r++) + sum += data[r][c]; + return sum / ROWS; + } + + private static double colSum(double[][] data, int c) { + double sum = 0; + for (int r = 0; r < ROWS; r++) + sum += data[r][c]; + return sum; + } + private HashMap runAndGetResult(String testName, String dml, String epsilonStr, double[][] data) { @@ -127,4 +181,3 @@ private HashMap runAndGetResult(String testName, String dml, return readDMLMatrixFromOutputDir("result"); } } - From d8f0f451fc3181520047ec3d09adf563524a761f Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Mon, 13 Jul 2026 14:48:48 +0200 Subject: [PATCH 22/26] Remove 4th and 5th parameters and instead get them in a loop, according to David's suggestion --- .../apache/sysds/parser/DMLTranslator.java | 41 +++++++++---------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/src/main/java/org/apache/sysds/parser/DMLTranslator.java b/src/main/java/org/apache/sysds/parser/DMLTranslator.java index 06df5fec357..6b51c75b4d5 100644 --- a/src/main/java/org/apache/sysds/parser/DMLTranslator.java +++ b/src/main/java/org/apache/sysds/parser/DMLTranslator.java @@ -2310,14 +2310,6 @@ private Hop processBuiltinFunctionExpression(BuiltinFunctionExpression source, D if (source.getThirdExpr() != null) { expr3 = processExpression(source.getThirdExpr(), null, hops); } - Hop expr4 = null; - if (source.getFourthExpr() != null) { - expr4 = processExpression(source.getFourthExpr(), null, hops); - } - Hop expr5 = null; - if (source.getFifthExpr() != null) { - expr5 = processExpression(source.getFifthExpr(), null, hops); - } Hop currBuiltinOp = null; target = (target == null) ? createTarget(source) : target; @@ -2598,24 +2590,31 @@ else if ( sop.equalsIgnoreCase(Opcodes.NOTEQUAL.toString()) ) currBuiltinOp = new UnaryOp(target.getName(), target.getDataType(), ValueType.FP64, OpOp1.DECOMPRESS, expr); break; case DP_LAPLACE: { + String[] dpLaplaceParamNames = {"target", "query", "sensitivity", "epsilon"}; LinkedHashMap dpLaplaceParams = new LinkedHashMap<>(); - dpLaplaceParams.put("target", expr); - dpLaplaceParams.put("query", expr2); - dpLaplaceParams.put("sensitivity", expr3); - dpLaplaceParams.put("epsilon", expr4); - currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, - ValueType.FP64, ParamBuiltinOp.DP_LAPLACE, dpLaplaceParams); + dpLaplaceParams.put(dpLaplaceParamNames[0], expr); + dpLaplaceParams.put(dpLaplaceParamNames[1], expr2); + dpLaplaceParams.put(dpLaplaceParamNames[2], expr3); + for (int i = 3; i < dpLaplaceParamNames.length; i++) { + dpLaplaceParams.put(dpLaplaceParamNames[i], + source.getExpr(i) != null ? processExpression(source.getExpr(i), null, hops) : null); + } + currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, ValueType.FP64, + ParamBuiltinOp.DP_LAPLACE, dpLaplaceParams); break; } case DP_GAUSSIAN: { + String[] dpGaussianParamNames = {"target", "query", "sensitivity", "epsilon", "delta"}; LinkedHashMap dpGaussianParams = new LinkedHashMap<>(); - dpGaussianParams.put("target", expr); - dpGaussianParams.put("query", expr2); - dpGaussianParams.put("sensitivity", expr3); - dpGaussianParams.put("epsilon", expr4); - dpGaussianParams.put("delta", expr5); - currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, - ValueType.FP64, ParamBuiltinOp.DP_GAUSSIAN, dpGaussianParams); + dpGaussianParams.put(dpGaussianParamNames[0], expr); + dpGaussianParams.put(dpGaussianParamNames[1], expr2); + dpGaussianParams.put(dpGaussianParamNames[2], expr3); + for (int i = 3; i < dpGaussianParamNames.length; i++) { + dpGaussianParams.put(dpGaussianParamNames[i], + source.getExpr(i) != null ? processExpression(source.getExpr(i), null, hops) : null); + } + currBuiltinOp = new ParameterizedBuiltinOp(target.getName(), DataType.MATRIX, ValueType.FP64, + ParamBuiltinOp.DP_GAUSSIAN, dpGaussianParams); break; } case QUANTIZE_COMPRESS: From 124ca53c1b9dc26339678fd6ce87ebb805c578b1 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 14 Jul 2026 14:20:33 +0200 Subject: [PATCH 23/26] Add dp_set_budget(epsilon, delta) built-in Lets a DML script declare its session-wide differential-privacy budget once at the top, instead of always falling back to the hardcoded default. Resolved entirely at compile time: epsilon/delta must be literals, validated in BuiltinFunctionExpression and stored on DMLProgram during HOP construction, then read by ExecutionContext.getDPBudgetAccountant(). --- .../org/apache/sysds/common/Builtins.java | 1 + .../parser/BuiltinFunctionExpression.java | 22 +++++ .../org/apache/sysds/parser/DMLProgram.java | 33 +++++++- .../apache/sysds/parser/DMLTranslator.java | 19 +++++ .../context/ExecutionContext.java | 16 +++- .../cp/DPBuiltinCPInstructionTest.java | 58 +++++++++++++ .../privacy/dp/DPBuiltinDMLTest.java | 81 ++++++++++++++++++- 7 files changed, 225 insertions(+), 5 deletions(-) diff --git a/src/main/java/org/apache/sysds/common/Builtins.java b/src/main/java/org/apache/sysds/common/Builtins.java index be6ee0f33db..6dce5fb30eb 100644 --- a/src/main/java/org/apache/sysds/common/Builtins.java +++ b/src/main/java/org/apache/sysds/common/Builtins.java @@ -118,6 +118,7 @@ public enum Builtins { DEDUP("dedup", true), DP_LAPLACE("dp_laplace", false), DP_GAUSSIAN("dp_gaussian", false), + DP_SET_BUDGET("dp_set_budget", false), DEEPWALK("deepWalk", true), DET("det", false), DETECTSCHEMA("detectSchema", false), diff --git a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java index a75e5ca7b8f..4fc43e0c89b 100644 --- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java +++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java @@ -2036,6 +2036,28 @@ else if(this.getOpCode() == Builtins.MAX_POOL || this.getOpCode() == Builtins.AV output.setDimensions(dpGaussianDims[0], dpGaussianDims[1]); break; } + case DP_SET_BUDGET: { + checkNumParameters(2); + checkScalarParam(getFirstExpr()); + checkScalarParam(getSecondExpr()); + // resolved entirely at compile time (see DMLTranslator), so both + // arguments must be known before the HOP DAG is built. + if (!isConstant(getFirstExpr()) || !isConstant(getSecondExpr())) + raiseValidateError(getOpCode() + ": 'epsilon' and 'delta' must be compile-time numeric literals", + false, LanguageErrorCodes.INVALID_PARAMETERS); + double dpSetBudgetEpsilon = getDoubleValue(getFirstExpr()); + double dpSetBudgetDelta = getDoubleValue(getSecondExpr()); + if (!(dpSetBudgetEpsilon > 0)) + raiseValidateError(getOpCode() + ": epsilon must be > 0, got " + dpSetBudgetEpsilon, + false, LanguageErrorCodes.INVALID_PARAMETERS); + if (!(dpSetBudgetDelta > 0 && dpSetBudgetDelta < 1)) + raiseValidateError(getOpCode() + ": delta must be in (0,1), got " + dpSetBudgetDelta, + false, LanguageErrorCodes.INVALID_PARAMETERS); + output.setDataType(DataType.SCALAR); + output.setValueType(ValueType.FP64); + output.setDimensions(0, 0); + break; + } case COMPRESS: case DECOMPRESS: if(OptimizerUtils.ALLOW_SCRIPT_LEVEL_COMPRESS_COMMAND){ diff --git a/src/main/java/org/apache/sysds/parser/DMLProgram.java b/src/main/java/org/apache/sysds/parser/DMLProgram.java index 2f69cb7ea0d..d2c0b9ae7d3 100644 --- a/src/main/java/org/apache/sysds/parser/DMLProgram.java +++ b/src/main/java/org/apache/sysds/parser/DMLProgram.java @@ -36,7 +36,21 @@ public class DMLProgram private ArrayList _blocks; private Map> _namespaces; private boolean _containsRemoteParfor; - + + /** + * Session-wide differential privacy budget resolved at compile time from a + * {@code dp_set_budget(epsilon, delta)} call (its arguments must be numeric + * literals — see {@code BuiltinFunctionExpression}). Null until such a call + * is encountered during HOP construction; consulted by + * {@code ExecutionContext#getDPBudgetAccountant()} in place of its hardcoded + * default. This is deliberately a plain field (not a Hop/Lop/Instruction): + * since {@code Program} holds a reference back to this {@code DMLProgram} + * (see {@code Program#getDMLProg()}), the value survives from compile time + * through to runtime without needing a runtime instruction at all. + */ + private Double _dpBudgetEpsilon; + private Double _dpBudgetDelta; + public DMLProgram(){ _blocks = new ArrayList<>(); _namespaces = new HashMap<>(); @@ -67,6 +81,23 @@ public void setContainsRemoteParfor(boolean flag) { public boolean containsRemoteParfor() { return _containsRemoteParfor; } + + public void setDPBudget(double epsilon, double delta) { + _dpBudgetEpsilon = epsilon; + _dpBudgetDelta = delta; + } + + public boolean hasDPBudget() { + return _dpBudgetEpsilon != null; + } + + public double getDPBudgetEpsilon() { + return _dpBudgetEpsilon; + } + + public double getDPBudgetDelta() { + return _dpBudgetDelta; + } public static boolean isInternalNamespace(String namespace) { return DEFAULT_NAMESPACE.equals(namespace) diff --git a/src/main/java/org/apache/sysds/parser/DMLTranslator.java b/src/main/java/org/apache/sysds/parser/DMLTranslator.java index 6b51c75b4d5..a9d5279c26b 100644 --- a/src/main/java/org/apache/sysds/parser/DMLTranslator.java +++ b/src/main/java/org/apache/sysds/parser/DMLTranslator.java @@ -2617,6 +2617,25 @@ else if ( sop.equalsIgnoreCase(Opcodes.NOTEQUAL.toString()) ) ParamBuiltinOp.DP_GAUSSIAN, dpGaussianParams); break; } + case DP_SET_BUDGET: { + // Resolved entirely at compile time: BuiltinFunctionExpression.validateExpression + // already enforced that both arguments are numeric literals, so 'expr'/'expr2' are + // guaranteed LiteralOps here. There is deliberately no runtime Hop/Lop/Instruction for + // this call — the budget is applied directly to the DMLProgram (reachable later from + // ExecutionContext via Program.getDMLProg(), see ExecutionContext.getDPBudgetAccountant()) + // before any instruction executes, so there is nothing for the DAG linearizer to reorder + // or drop as dead code. + if (_dmlProg.hasDPBudget()) + throw new LanguageException(source.getOpCode() + ": dp_set_budget may only be called once per " + + "script (already set to epsilon=" + _dmlProg.getDPBudgetEpsilon() + + ", delta=" + _dmlProg.getDPBudgetDelta() + ")"); + if (!(expr instanceof LiteralOp) || !(expr2 instanceof LiteralOp)) + throw new LanguageException(source.getOpCode() + + ": epsilon and delta must be compile-time numeric literals"); + _dmlProg.setDPBudget(((LiteralOp) expr).getDoubleValue(), ((LiteralOp) expr2).getDoubleValue()); + currBuiltinOp = expr; // echo epsilon back as confirmation + break; + } case QUANTIZE_COMPRESS: currBuiltinOp = new BinaryOp(target.getName(), target.getDataType(), target.getValueType(), OpOp2.valueOf(source.getOpCode().name()), expr, expr2); break; diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java index eaf50da88c7..bce0f2f679a 100644 --- a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java +++ b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java @@ -28,6 +28,7 @@ import org.apache.sysds.conf.ConfigurationManager; import org.apache.sysds.hops.OptimizerUtils; import org.apache.sysds.hops.fedplanner.FTypes.FType; +import org.apache.sysds.parser.DMLProgram; import org.apache.sysds.runtime.DMLRuntimeException; import org.apache.sysds.runtime.controlprogram.LocalVariableMap; import org.apache.sysds.runtime.controlprogram.Program; @@ -147,9 +148,20 @@ public void setLineage(Lineage lineage) { _lineage = lineage; } + /** + * Returns the session-scoped {@link DPBudgetAccountant}, lazily initialised on + * first use. If the DML script called {@code dp_set_budget(epsilon, delta)} + * with compile-time literal arguments, that value (resolved onto the + * {@code DMLProgram} during HOP construction — see {@code DMLTranslator}'s + * {@code DP_SET_BUDGET} case) is used instead of the hardcoded default. + */ public DPBudgetAccountant getDPBudgetAccountant() { - if (_dpBudgetAccountant == null) - _dpBudgetAccountant = new DPBudgetAccountant(1.0, 1e-5); + if (_dpBudgetAccountant == null) { + DMLProgram dmlProg = (_prog != null) ? _prog.getDMLProg() : null; + _dpBudgetAccountant = (dmlProg != null && dmlProg.hasDPBudget()) + ? new DPBudgetAccountant(dmlProg.getDPBudgetEpsilon(), dmlProg.getDPBudgetDelta()) + : new DPBudgetAccountant(1.0, 1e-5); + } return _dpBudgetAccountant; } diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index 750451b2991..132ce516088 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -19,7 +19,11 @@ package org.apache.sysds.test.component.cp; +import org.apache.sysds.parser.DMLProgram; import org.apache.sysds.runtime.DMLRuntimeException; +import org.apache.sysds.runtime.controlprogram.Program; +import org.apache.sysds.runtime.controlprogram.context.ExecutionContext; +import org.apache.sysds.runtime.controlprogram.context.ExecutionContextFactory; import org.apache.sysds.runtime.privacy.dp.DPBudgetAccountant; import org.junit.Test; @@ -176,6 +180,60 @@ public void testConstructorRejectsDeltaOne() { new DPBudgetAccountant(1.0, 1.0); } + // ======================================================================= + // 1b. DMLProgram / ExecutionContext.getDPBudgetAccountant() (dp_set_budget) + // ======================================================================= + // + // dp_set_budget(epsilon, delta) is resolved entirely at compile time onto + // DMLProgram (DMLTranslator's DP_SET_BUDGET case) rather than through a + // runtime instruction; ExecutionContext.getDPBudgetAccountant() consults + // Program.getDMLProg() on first (lazy) access. These tests exercise that + // plumbing directly, without going through the DML compiler. + + @Test + public void testDMLProgramHasDPBudgetTracksSetState() { + DMLProgram dmlProg = new DMLProgram(); + assertFalse("No dp_set_budget call yet", dmlProg.hasDPBudget()); + dmlProg.setDPBudget(2.0, 1e-6); + assertTrue("dp_set_budget was called", dmlProg.hasDPBudget()); + assertEquals(2.0, dmlProg.getDPBudgetEpsilon(), EPS); + assertEquals(1e-6, dmlProg.getDPBudgetDelta(), EPS); + } + + @Test + public void testGetDPBudgetAccountantUsesCompileTimeResolvedBudget() { + DMLProgram dmlProg = new DMLProgram(); + dmlProg.setDPBudget(5.0, 1e-6); + ExecutionContext ec = ExecutionContextFactory.createContext(new Program(dmlProg)); + + DPBudgetAccountant acc = ec.getDPBudgetAccountant(); + acc.compose(2.0, 0.0, 1.0); // would exceed the hardcoded default budget of 1.0 + assertTrue("Compile-time-resolved budget should be used instead of the hardcoded default", + acc.remainingBudget() > 0); + } + + @Test(expected = DMLRuntimeException.class) + public void testGetDPBudgetAccountantFallsBackToDefaultWithoutDPSetBudget() { + // No dp_set_budget call: the hardcoded default budget of epsilon=1.0 applies, + // so a release at epsilon=1.5 must be rejected. + DMLProgram dmlProg = new DMLProgram(); + ExecutionContext ec = ExecutionContextFactory.createContext(new Program(dmlProg)); + ec.getDPBudgetAccountant().compose(1.5, 0.0, 1.0); + } + + @Test + public void testGetDPBudgetAccountantIsLazyAndCachedPerContext() { + // The accountant must be created once and reused across calls on the + // same ExecutionContext, not rebuilt (which would reset releaseCount()). + DMLProgram dmlProg = new DMLProgram(); + dmlProg.setDPBudget(10.0, 1e-6); + ExecutionContext ec = ExecutionContextFactory.createContext(new Program(dmlProg)); + + ec.getDPBudgetAccountant().compose(1.0, 0.0, 1.0); + assertEquals("Same accountant instance must be reused across calls", + 1, ec.getDPBudgetAccountant().releaseCount()); + } + // --- Single-argument convenience constructor ------------------- @Test diff --git a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java index 32a02045753..9b2ef30e0af 100644 --- a/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java +++ b/src/test/java/org/apache/sysds/test/functions/privacy/dp/DPBuiltinDMLTest.java @@ -22,6 +22,9 @@ import java.nio.file.Files; import java.util.HashMap; +import org.apache.sysds.parser.LanguageException; +import org.apache.sysds.parser.ParseException; +import org.apache.sysds.runtime.DMLRuntimeException; import org.apache.sysds.runtime.matrix.data.MatrixValue.CellIndex; import org.apache.sysds.test.AutomatedTestBase; import org.apache.sysds.test.TestConfiguration; @@ -47,10 +50,38 @@ public class DPBuiltinDMLTest extends AutomatedTestBase { private static final String DML_LAPLACE = String.format(DML_LAPLACE_TEMPLATE, "colMeans"); private static final String DML_GAUSSIAN = String.format(DML_GAUSSIAN_TEMPLATE, "colMeans"); + // dp_set_budget(epsilon, delta) is resolved entirely at compile time (its arguments + // must be literals), called via a dummy assignment, then a single dp_laplace release + // at $2 records a cost of exactly $2 (Laplace basic composition). + private static final String DML_SET_BUDGET_TEMPLATE = + "eps = dp_set_budget(%s, 1e-6);\n" + + "X = read($1);\n" + + "result = dp_laplace(X, query=\"colMeans\", sensitivity=1.0, epsilon=$2);\n" + + "write(result, $3, format=\"text\");\n"; + + // Two dp_set_budget calls in the same script; DMLTranslator must reject this at + // compile time (DMLProgram.hasDPBudget()). + private static final String DML_SET_BUDGET_TWICE = + "eps = dp_set_budget(3.0, 1e-6);\n" + + "eps2 = dp_set_budget(5.0, 1e-6);\n" + + "X = read($1);\n" + + "result = dp_laplace(X, query=\"colMeans\", sensitivity=1.0, epsilon=$2);\n" + + "write(result, $3, format=\"text\");\n"; + + // A budget argument computed at runtime (not a literal); must be rejected at + // compile time by BuiltinFunctionExpression's isConstant() check. + private static final String DML_SET_BUDGET_NON_LITERAL = + "X = read($1);\n" + + "computed = sum(X) / nrow(X);\n" + + "eps = dp_set_budget(computed, 1e-6);\n" + + "result = dp_laplace(X, query=\"colMeans\", sensitivity=1.0, epsilon=$2);\n" + + "write(result, $3, format=\"text\");\n"; + @Override public void setUp() { addTestConfiguration("DPLaplace", new TestConfiguration(TEST_CLASS, "DPLaplace")); addTestConfiguration("DPGaussian", new TestConfiguration(TEST_CLASS, "DPGaussian")); + addTestConfiguration("DPSetBudget", new TestConfiguration(TEST_CLASS, "DPSetBudget")); } @Test @@ -104,6 +135,41 @@ public void testHighEpsilonIsCloserToTruth() { assertTrue("ε=0.5 should give less noise than ε=0.1", noisyHigh < noisyLow); } + @Test + public void testSetBudgetLiteralAllowsExceedingDefaultBudget() { + // Default budget is epsilon=1.0; a single release at epsilon=1.5 would be + // rejected unless dp_set_budget(3.0, ...) widens it first. + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + HashMap result = runAndGetResult("DPSetBudget", + String.format(DML_SET_BUDGET_TEMPLATE, "3.0"), "1.5", data); + assertShape(result, 1, COLS); + } + + @Test + public void testSetBudgetNarrowBudgetStillEnforced() { + // An explicit narrow budget must still be enforced: epsilon=0.8 exceeds + // the explicit budget of 0.5. + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + runExpectingException("DPSetBudget", String.format(DML_SET_BUDGET_TEMPLATE, "0.5"), "0.8", data, + DMLRuntimeException.class); + } + + @Test + public void testSetBudgetCalledTwiceFailsAtCompileTime() { + // Thrown from DMLTranslator.processBuiltinFunctionExpression (HOP construction), + // which wraps all case-block exceptions in ParseException (see processExpression's + // catch-all) — unlike the non-literal check below, which runs during validation + // and so surfaces as a bare LanguageException. + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + runExpectingException("DPSetBudget", DML_SET_BUDGET_TWICE, "0.5", data, ParseException.class); + } + + @Test + public void testSetBudgetRejectsNonLiteralArgs() { + double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); + runExpectingException("DPSetBudget", DML_SET_BUDGET_NON_LITERAL, "0.5", data, LanguageException.class); + } + private void runColMeansDPTest(String testName, String dml, String epsilonStr) { double[][] data = TestUtils.generateTestMatrix(ROWS, COLS, 0, 1, 1.0, 42); HashMap result = runAndGetResult(testName, dml, epsilonStr, data); @@ -163,6 +229,19 @@ private static double colSum(double[][] data, int c) { private HashMap runAndGetResult(String testName, String dml, String epsilonStr, double[][] data) { + prepareScript(testName, dml, epsilonStr, data); + runTest(true, false, null, -1); + return readDMLMatrixFromOutputDir("result"); + } + + private void runExpectingException(String testName, String dml, String epsilonStr, double[][] data, + Class expectedException) + { + prepareScript(testName, dml, epsilonStr, data); + runTest(true, true, expectedException, -1); + } + + private void prepareScript(String testName, String dml, String epsilonStr, double[][] data) { getAndLoadTestConfiguration(testName); writeInputMatrixWithMTD("X", data, false); @@ -177,7 +256,5 @@ private HashMap runAndGetResult(String testName, String dml, } programArgs = new String[]{ "-args", input("X"), epsilonStr, output("result") }; - runTest(true, false, null, -1); - return readDMLMatrixFromOutputDir("result"); } } From 025f9bbdf5a9096083a9faaaa20fc1fffff2c757 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 14 Jul 2026 14:31:21 +0200 Subject: [PATCH 24/26] Change DPBudgetAccountant defaults to be constants --- .../sysds/parser/BuiltinFunctionExpression.java | 4 ++-- .../controlprogram/context/ExecutionContext.java | 4 ++-- .../sysds/runtime/privacy/dp/DPBudgetAccountant.java | 12 ++++++++++++ 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java index 4fc43e0c89b..1c2c6ba8110 100644 --- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java +++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java @@ -2047,10 +2047,10 @@ else if(this.getOpCode() == Builtins.MAX_POOL || this.getOpCode() == Builtins.AV false, LanguageErrorCodes.INVALID_PARAMETERS); double dpSetBudgetEpsilon = getDoubleValue(getFirstExpr()); double dpSetBudgetDelta = getDoubleValue(getSecondExpr()); - if (!(dpSetBudgetEpsilon > 0)) + if (dpSetBudgetEpsilon <= 0) raiseValidateError(getOpCode() + ": epsilon must be > 0, got " + dpSetBudgetEpsilon, false, LanguageErrorCodes.INVALID_PARAMETERS); - if (!(dpSetBudgetDelta > 0 && dpSetBudgetDelta < 1)) + if ((dpSetBudgetDelta <= 0) || (dpSetBudgetDelta >= 1)) raiseValidateError(getOpCode() + ": delta must be in (0,1), got " + dpSetBudgetDelta, false, LanguageErrorCodes.INVALID_PARAMETERS); output.setDataType(DataType.SCALAR); diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java index bce0f2f679a..7f25b2364fe 100644 --- a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java +++ b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java @@ -153,14 +153,14 @@ public void setLineage(Lineage lineage) { * first use. If the DML script called {@code dp_set_budget(epsilon, delta)} * with compile-time literal arguments, that value (resolved onto the * {@code DMLProgram} during HOP construction — see {@code DMLTranslator}'s - * {@code DP_SET_BUDGET} case) is used instead of the hardcoded default. + * {@code DP_SET_BUDGET} case) is used instead of the hardcoded defaults. */ public DPBudgetAccountant getDPBudgetAccountant() { if (_dpBudgetAccountant == null) { DMLProgram dmlProg = (_prog != null) ? _prog.getDMLProg() : null; _dpBudgetAccountant = (dmlProg != null && dmlProg.hasDPBudget()) ? new DPBudgetAccountant(dmlProg.getDPBudgetEpsilon(), dmlProg.getDPBudgetDelta()) - : new DPBudgetAccountant(1.0, 1e-5); + : new DPBudgetAccountant(); } return _dpBudgetAccountant; } diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java index 9dd011fcb1e..51cd48dee4d 100644 --- a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java @@ -91,6 +91,10 @@ public class DPBudgetAccountant { // Rényi orders used for Gaussian composition // ----------------------------------------------------------------------- + private static final double DEFAULT_EPSILON_BUDGET = 1.0; + + private static final double DEFAULT_DELTA = 1e-5; + /** * Discrete set of Rényi orders α. All must be > 1. * Finer grids give tighter bounds; this set covers the range relevant @@ -160,6 +164,14 @@ public DPBudgetAccountant(double epsilonBudget) { this(epsilonBudget, 1e-5); } + /** + * Default constructor using defaults. + * Suitable when the calling script does not specify ε, δ explicitly. + */ + public DPBudgetAccountant() { + this(DEFAULT_EPSILON_BUDGET, DEFAULT_DELTA); + } + // ----------------------------------------------------------------------- // Core API // ----------------------------------------------------------------------- From f29cd3e438763f06dc6c40a3cb15432e8300bb35 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 14 Jul 2026 23:54:43 +0200 Subject: [PATCH 25/26] Cleanup comments --- .../cp/DPBuiltinCPInstruction.java | 74 +++++++++---------- .../privacy/dp/DPBudgetAccountant.java | 62 ++++++---------- .../cp/DPBuiltinCPInstructionTest.java | 18 ++--- 3 files changed, 63 insertions(+), 91 deletions(-) diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java index 9d49f959e13..1ca7d75bf86 100755 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/DPBuiltinCPInstruction.java @@ -34,31 +34,29 @@ * CP instruction for differential-privacy release of a linear query over the * original matrix. * - *

    DML syntax (raw-matrix form): - * 

    + * DML syntax (raw-matrix form):
  *   result = dp_laplace(X, query="colMeans", sensitivity=1.0, epsilon=0.5)
  *   result = dp_gaussian(X, query="colMeans", sensitivity=1.0, epsilon=0.5, delta=1e-5)
- *
    * - *

    The instruction receives the original {@code n x d} matrix {@code X}, + * The instruction receives the original {@code n x d} matrix {@code X}, * builds a transformation matrix {@code T} ({@code k x n}) from the named * {@code query} (see {@link #buildTransform}), and returns a noisy release of - * {@code T %*% X}. The noise is not added as a separate elementwise + * {@code T %*% X}. The noise is not added as a separate elementwise * pass over a materialised aggregate: it is injected by augmenting {@code T} * with an identity block and {@code X} with the noise matrix, so that the * noisy release is the result of a single {@link LibMatrixMult#matrixMult} * call (see {@link #processInstruction} for the derivation). * - *

    Sensitivity norm: {@code sensitivity} is not interchangeable + * Sensitivity norm: {@code sensitivity} is not interchangeable * between the two builtins. {@code dp_laplace} calibrates its noise scale - * to the L1 sensitivity of {@code T %*% X} to a single-record - * change; {@code dp_gaussian} calibrates its σ to the L2 sensitivity. + * to the L1 sensitivity of {@code T %*% X} to a single-record + * change; {@code dp_gaussian} calibrates its σ to the L2 sensitivity. * For a scalar release (e.g. {@code query="colMeans"} on single-column * {@code X}) the two norms coincide, but for a vector- or matrix-valued * release they generally differ — the caller is responsible for supplying * the norm matching the builtin invoked (see {@link #sensitivityOf}). * - *

    The {@link #sensitivityOf} method is deliberately separated from the + * The {@link #sensitivityOf} method is deliberately separated from the * noise-scale computation. It currently returns the caller-supplied * constant. A future rewrite pass could replace the body of this single * method with a static analysis that derives sensitivity from {@code T}'s @@ -111,12 +109,10 @@ private DPBuiltinCPInstruction( * Reconstructs a {@code DPBuiltinCPInstruction} from its serialised * instruction string produced by the LOP layer. * - *

    Expected format (OPERAND_DELIM = '\u00b0'): - * 

    +     * Expected format (OPERAND_DELIM = '\u00b0'):
      *   dp_gaussian°target=mVar1·MATRIX·FP64°query=colMeans·SCALAR·STRING·true
      *              °sensitivity=1.0·SCALAR·FP64·true°epsilon=0.5·SCALAR·FP64·true
      *              °delta=1e-5·SCALAR·FP64·true°_mVar2·MATRIX·FP64
-     *
    * * The first token is always the opcode; the last token is always the * output operand; the tokens in between are key=value pairs. This matches @@ -162,22 +158,20 @@ public static DPBuiltinCPInstruction parseInstruction(String str) { /** * Executes the DP release. * - *
      - *
    1. Read the original {@link MatrixBlock} {@code X} from the variable - * table.
      2. - *
    2. Build the transformation matrix {@code T} ({@code k x n}) from - * {@code query} (see {@link #buildTransform}).
      3. - *
    3. Determine sensitivity via {@link #sensitivityOf}.
      4. - *
    4. Generate a noise {@link MatrixBlock} shaped {@code k x d}.
      5. - *
    5. Fuse {@code T %*% X + noise} into a single - * {@link LibMatrixMult#matrixMult} call (see below).
      6. - *
    6. Record the release with the session-scoped - * {@link DPBudgetAccountant}; throw if budget is exhausted.
      7. - *
    7. Write the noisy block back to the variable table and release - * the input pin.
      8. - *
    + * - Read the original {@link MatrixBlock} {@code X} from the variable + * table. + * - Build the transformation matrix {@code T} ({@code k x n}) from + * {@code query} (see {@link #buildTransform}). + * - Determine sensitivity via {@link #sensitivityOf}. + * - Generate a noise {@link MatrixBlock} shaped {@code k x d}. + * - Fuse {@code T %*% X + noise} into a single + * {@link LibMatrixMult#matrixMult} call (see below). + * - Record the release with the session-scoped + * {@link DPBudgetAccountant}; throw if budget is exhausted. + * - Write the noisy block back to the variable table and release + * the input pin. * - *

    Fusion derivation: for {@code T} ({@code k x n}), {@code X} + * Fusion derivation: for {@code T} ({@code k x n}), {@code X} * ({@code n x d}) and noise {@code N} ({@code k x d}), let * {@code T' = [T | I_k]} ({@code k x (n+k)}) and * {@code X' = [X ; N]} ({@code (n+k) x d}). Then @@ -235,17 +229,15 @@ public void processInstruction(ExecutionContext ec) { * named query, to be left-multiplied against the {@code n x d} input * {@code X} as {@code T %*% X}. * - *

      - *
    • {@code "colMeans"}: {@code T} is {@code 1 x n}, filled with - * {@code 1/n} — {@code T %*% X} is the column-mean row vector.
      • - *
    • {@code "colSums"}: {@code T} is {@code 1 x n}, filled with - * {@code 1.0} — {@code T %*% X} is the column-sum row vector.
      • - *
    • {@code "identity"}: {@code T} is the {@code n x n} identity + * - {@code "colMeans"}: {@code T} is {@code 1 x n}, filled with + * {@code 1/n} — {@code T %*% X} is the column-mean row vector. + * - {@code "colSums"}: {@code T} is {@code 1 x n}, filled with + * {@code 1.0} — {@code T %*% X} is the column-sum row vector. + * - {@code "identity"}: {@code T} is the {@code n x n} identity * (built sparsely via {@link #identity}) — {@code T %*% X} is - * {@code X} itself, i.e. a noisy release of the raw matrix.
      • - *
    + * {@code X} itself, i.e. a noisy release of the raw matrix. * - *

    Row-wise aggregates ({@code rowMeans}/{@code rowSums}) reduce across + * Row-wise aggregates ({@code rowMeans}/{@code rowSums}) reduce across * the feature axis of {@code X}, i.e. they are naturally * {@code X %*% T'} (right-multiply), not {@code T %*% X}, so they are * intentionally not supported here. @@ -302,11 +294,11 @@ private static MatrixBlock identity(int k) { /** * Returns the sensitivity of the release {@code T %*% X} to a * single-record change, in the norm required by the mechanism actually - * invoked: L1 for {@code dp_laplace}, L2 for + * invoked: L1 for {@code dp_laplace}, L2 for * {@code dp_gaussian} (see the class Javadoc). The two only coincide * when the release is scalar. * - *

    Returns the caller-supplied literal from the DML script as-is, with + * Returns the caller-supplied literal from the DML script as-is, with * no norm conversion or validation — the DML author must compute the * sensitivity in the correct norm for the builtin they call. A future * rewrite pass could replace this body with an analysis that derives @@ -332,7 +324,7 @@ private double sensitivityOf(MatrixBlock T) { * mechanism-appropriate distribution calibrated to ({@code sensitivity}, * {@code epsilon}, {@code delta}). * - *

    Both mechanisms produce a dense block. Sparsity exploitation is + * Both mechanisms produce a dense block. Sparsity exploitation is * left for future work; for the releases targeted here (e.g. column * means, column sums) the noise is dense regardless. */ @@ -369,7 +361,7 @@ private MatrixBlock generateNoise( * Fills {@code block} with i.i.d. Laplace(0, scale) samples using the * inverse-CDF method. * - *

    For u ~ Uniform(0, 1): X = -scale * sign(u - 0.5) * ln(1 - 2|u - 0.5|) + * For u ~ Uniform(0, 1): X = -scale * sign(u - 0.5) * ln(1 - 2|u - 0.5|) */ private static void fillLaplaceNoise(MatrixBlock block, double scale) { ThreadLocalRandom rng = ThreadLocalRandom.current(); @@ -390,7 +382,7 @@ private static void fillLaplaceNoise(MatrixBlock block, double scale) { /** * Fills {@code block} with i.i.d. N(0, sigma²) samples. * - *

    Uses {@link ThreadLocalRandom#nextGaussian()} which is thread-safe + * Uses {@link ThreadLocalRandom#nextGaussian()} which is thread-safe * and does not require external libraries. */ private static void fillGaussianNoise(MatrixBlock block, double sigma) { diff --git a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java index 51cd48dee4d..11ddc3ac79f 100644 --- a/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java +++ b/src/main/java/org/apache/sysds/runtime/privacy/dp/DPBudgetAccountant.java @@ -25,61 +25,49 @@ /** * Session-scoped differential privacy budget accountant. * - *

    Purpose

    * Tracks composition of DP releases across the lifetime of a DML script * execution. Each call to {@link #compose} records one release and checks * whether the cumulative privacy cost has exceeded the user-specified budget. * - *

    Composition strategy

    * The mechanism type (Laplace vs Gaussian) is inferred from the {@code delta} * argument passed to {@link #compose}: * - *
      - *
    • Laplace (delta == 0): pure ε-DP. The budget cost is tracked via + * - Laplace (delta == 0): pure ε-DP. The budget cost is tracked via * basic composition — each release contributes exactly its ε to a running * sum. This is the tightest possible bound for pure DP and avoids the * looser estimate that results from routing Laplace through the RDP * conversion path (which would introduce an unnecessary δ). Noise scale - * is calibrated to L1 sensitivity (see {@link #compose}).
      • - *
    • Gaussian (delta > 0): (ε, δ)-DP via Rényi DP composition. + * is calibrated to L1 sensitivity (see {@link #compose}). + * - Gaussian (delta > 0): (ε, δ)-DP via Rényi DP composition. * Rényi divergences at a discrete set of orders α compose additively; * the accumulated sum is converted to (ε, δ) at query time using the * formula from Mironov 2017. This is substantially tighter than basic * composition for repeated Gaussian releases, which is the common case - * in federated learning.
      • - *
    + * in federated learning. * - *

    When both mechanisms are used in the same script the total cost is: - * 

    + * When both mechanisms are used in the same script the total cost is:
  *   ε_total = ε_Laplace_sum + ε_Gaussian_RDP
- *
    * This follows from basic composition of a pure-DP mechanism with an * approximate-DP mechanism, which is additive in ε. * - *

    Rényi orders tracked (Gaussian path)

    + * Rényi orders tracked (Gaussian path) * α ∈ {2, 4, 8, 16, 32, 64, 128, 256, 512, 1024}. At query time the minimum * converted ε across all orders is taken as the tightest available bound. * - *

    Gaussian RDP divergence

    + * Gaussian RDP divergence * For the Gaussian mechanism with noise scale σ and L2 sensitivity Δf: - * 
      *   D_α = α · Δf² / (2σ²)
- *
    * σ is back-derived from the caller's (ε, δ) via the standard calibration * formula (see {@link #gaussianSigma}). Note that sensitivity cancels in the * final expression, so the RDP cost depends only on the (ε, δ) parameters. * - *

    RDP → (ε, δ) conversion (Mironov 2017, Proposition 3)

    - * 
    + * RDP → (ε, δ) conversion (Mironov 2017, Proposition 3)
  *   ε(α) = R[α] + log(1 − 1/α) − log(δ·(α−1)) / α
- *
    * - *

    Lifecycle

    * One instance is created per {@code ExecutionContext} (lazy init). It is * garbage-collected with the context when the script finishes; no state * leaks between script executions or between concurrent scripts. * - *

    Thread safety

    * Not thread-safe. A single DML script executes instructions sequentially * on one thread, so no synchronisation is needed. * @@ -96,7 +84,7 @@ public class DPBudgetAccountant { private static final double DEFAULT_DELTA = 1e-5; /** - * Discrete set of Rényi orders α. All must be > 1. + * Discrete set of Rényi orders α. All must be > 1. * Finer grids give tighter bounds; this set covers the range relevant * for typical ML workloads. */ @@ -114,7 +102,7 @@ public class DPBudgetAccountant { /** * Running sum of pure ε from Laplace releases. * - *

    Laplace gives pure ε-DP (no δ). Basic composition is exact and + * Laplace gives pure ε-DP (no δ). Basic composition is exact and * tighter than the RDP conversion path for Laplace (which would introduce * an unnecessary δ and produce a looser bound). Each Laplace release adds * its ε here; the total is added directly in {@link #totalEpsilonSpent()}. @@ -137,12 +125,12 @@ public class DPBudgetAccountant { /** * Creates an accountant with the given global budget. * - *

    Typical usage: the DML script sets the budget once at the top + * Typical usage: the DML script sets the budget once at the top * (future work: a {@code dp_set_budget(epsilon, delta)} built-in), * or the accountant is created with defaults and the budget is checked * after each release. * - * @param epsilonBudget total ε budget for the script execution (must be > 0) + * @param epsilonBudget total ε budget for the script execution (must be > 0) * @param delta δ used for the Gaussian RDP-to-(ε,δ) conversion (must be in (0,1)) */ public DPBudgetAccountant(double epsilonBudget, double delta) { @@ -179,23 +167,21 @@ public DPBudgetAccountant() { /** * Records one DP release and checks the budget. * - *

    This method must be called before the result is written to + * This method must be called before the result is written to * the variable table. If the budget is exhausted it throws and the * caller's result is discarded, preventing an unaccounted release. * - *

    Mechanism selection (see class-level Javadoc for details): - *

      - *
    • {@code delta == 0} → Laplace, pure ε-DP basic composition
      • - *
    • {@code delta > 0} → Gaussian, Rényi DP composition
      • - *
    + * Mechanism selection (see class-level Javadoc for details): + * - {@code delta == 0} → Laplace, pure ε-DP basic composition + * - {@code delta > 0} → Gaussian, Rényi DP composition * * @param epsilon per-release ε parameter (must be > 0) * @param delta per-release δ parameter (0 for Laplace, >0 for Gaussian) - * @param sensitivity sensitivity Δf of the released quantity (must be > 0). - * The norm depends on the mechanism selected by - * {@code delta}: callers must supply the - * L1 sensitivity ‖f(D) − f(D′)‖₁ when - * {@code delta == 0} (Laplace), and the L2 + * @param sensitivity sensitivity Δf of the released quantity (must be > 0). + * The norm depends on the mechanism selected by + * {@code delta}: callers must supply the + * L1 sensitivity ‖f(D) − f(D′)‖₁ when + * {@code delta == 0} (Laplace), and the L2 * sensitivity ‖f(D) − f(D′)‖₂ when {@code delta > 0} * (Gaussian). The two coincide for scalar-valued * releases but diverge for vector-valued ones, so @@ -235,7 +221,7 @@ public void compose(double epsilon, double delta, double sensitivity) { /** * Returns the current total privacy cost as an ε value. * - *

    Total = Laplace pure-ε sum + Gaussian RDP-converted ε (clamped to + * Total = Laplace pure-ε sum + Gaussian RDP-converted ε (clamped to * zero when no Gaussian releases have been recorded). */ public double totalEpsilonSpent() { @@ -273,9 +259,7 @@ public int releaseCount() { /** * Rényi divergence of order α for the Gaussian mechanism (Mironov 2017, * Proposition 3, example 2): - * 

          *   D_α = α · Δf² / (2σ²)
-     *
    */ private static double rdpGaussian(double alpha, double sensitivity, double sigma) { return alpha * (sensitivity * sensitivity) / (2.0 * sigma * sigma); @@ -283,9 +267,7 @@ private static double rdpGaussian(double alpha, double sensitivity, double sigma /** * Gaussian noise scale σ calibrated to (ε, δ)-DP: - * 
          *   σ = Δf · sqrt(2 · log(1.25 / δ)) / ε
-     *
    * Must match the formula used in {@link DPBuiltinCPInstruction} so that * the RDP cost recorded here is consistent with the noise actually injected. */ diff --git a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java index 132ce516088..c10dd065fd8 100755 --- a/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java +++ b/src/test/java/org/apache/sysds/test/component/cp/DPBuiltinCPInstructionTest.java @@ -32,19 +32,17 @@ /** * Tests for {@code DPBuiltinCPInstruction} and {@code DPBudgetAccountant}. * - *

    The tests are grouped into three levels: - *

      - *
    1. Unit tests on DPBudgetAccountant — verify composition, conversion, + * The tests are grouped into three levels: + * - Unit tests on DPBudgetAccountant — verify composition, conversion, * and budget enforcement in isolation, with no dependency on the full - * SystemDS runtime.
      2. - *
    2. Noise distribution tests — verify that the noise blocks + * SystemDS runtime. + * - Noise distribution tests — verify that the noise blocks * generated by the Laplace and Gaussian mechanisms have statistically - * correct means and variances (Kolmogorov-Smirnov style sanity checks).
      3. - *
    3. DML integration tests — run complete DML scripts and verify - * end-to-end correctness via the existing AutomatedTestBase machinery.
      4. - *
    + * correct means and variances (Kolmogorov-Smirnov style sanity checks). + * - DML integration tests — run complete DML scripts and verify + * end-to-end correctness via the existing AutomatedTestBase machinery. * - *

    The DML integration tests require a built SystemDS jar and are separated + * The DML integration tests require a built SystemDS jar and are separated * into a companion class {@link org.apache.sysds.test.functions.privacy.dp.DPBuiltinDMLTest}. */ public class DPBuiltinCPInstructionTest { From d8912b1fd5c7b14f2a89c62d2ba1e7decee42da6 Mon Sep 17 00:00:00 2001 From: Maya Anderson Date: Tue, 14 Jul 2026 16:44:14 +0200 Subject: [PATCH 26/26] Differential Privacy Benchmark MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four federated workers simulated on localhost, a logistic regression FedAvg loop in DML where the coordinator applies dp_gaussian to the aggregated gradient, a sweep over ε ∈ {0.5, 1, 4, 8} plus a non-private baseline, and a matplotlib accuracy-vs-ε plot saved as a PNG. Add clip_norm (default 4.0) as a script parameter. Inside the private == 1 branch, each row's gradient contribution is clipped to L2-norm less than clip_norm. --- benchmark/scripts/collect_results.py | 41 +++++++++ benchmark/scripts/eval.dml | 22 +++++ benchmark/scripts/fedavg_dp.dml | 130 +++++++++++++++++++++++++++ benchmark/scripts/plot.py | 114 +++++++++++++++++++++++ benchmark/scripts/prepare_data.py | 123 +++++++++++++++++++++++++ benchmark/scripts/run_benchmark.sh | 15 ++++ benchmark/scripts/run_sweep.sh | 69 ++++++++++++++ benchmark/scripts/start_workers.sh | 24 +++++ benchmark/scripts/stop_workers.sh | 11 +++ src/main/python/requirements.txt | 31 +++++++ 10 files changed, 580 insertions(+) create mode 100644 benchmark/scripts/collect_results.py create mode 100644 benchmark/scripts/eval.dml create mode 100644 benchmark/scripts/fedavg_dp.dml create mode 100644 benchmark/scripts/plot.py create mode 100644 benchmark/scripts/prepare_data.py create mode 100755 benchmark/scripts/run_benchmark.sh create mode 100755 benchmark/scripts/run_sweep.sh create mode 100755 benchmark/scripts/start_workers.sh create mode 100755 benchmark/scripts/stop_workers.sh create mode 100644 src/main/python/requirements.txt diff --git a/benchmark/scripts/collect_results.py b/benchmark/scripts/collect_results.py new file mode 100644 index 00000000000..c15ce63aa84 --- /dev/null +++ b/benchmark/scripts/collect_results.py @@ -0,0 +1,41 @@ +""" +Parse per-run accuracy files into a single results.csv. + +Output columns: label, epsilon, private, accuracy +""" +import pathlib, csv, re + +RESULTS = pathlib.Path("benchmark/results") + +rows = [] + +def parse_acc(path: pathlib.Path) -> float: + txt = path.read_text().strip() + # SystemDS writes a bare float. + return float(txt) + +# Non-private baseline. +baseline_path = RESULTS / "acc_baseline.txt" +if baseline_path.exists(): + rows.append(dict(label="baseline", epsilon="inf", + private=0, accuracy=parse_acc(baseline_path))) + +# DP runs. +for eps in [0.5, 1, 4, 8]: + p = RESULTS / f"acc_eps_{eps}.txt" + if p.exists(): + rows.append(dict(label=f"ε={eps}", epsilon=eps, + private=1, accuracy=parse_acc(p))) + else: + print(f"Warning: {p} not found — skipping") + +out = RESULTS / "results.csv" +with open(out, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=["label","epsilon","private","accuracy"]) + w.writeheader() + w.writerows(rows) + +print(f"Wrote {out}") +for r in rows: + print(f" {r['label']:12s} acc={r['accuracy']:.4f}") + diff --git a/benchmark/scripts/eval.dml b/benchmark/scripts/eval.dml new file mode 100644 index 00000000000..7f0c89fbaa4 --- /dev/null +++ b/benchmark/scripts/eval.dml @@ -0,0 +1,22 @@ +# eval.dml — compute binary classification accuracy on held-out test set. +# Arguments: data_dir, model_path, out_acc +data_dir = $data_dir; +model_path = $model_path; +out_acc = $out_acc; + +X_test = read(data_dir + "/X_test.csv", + data_type="matrix", value_type="double", format="csv"); +y_test = read(data_dir + "/y_test.csv", + data_type="matrix", value_type="double", format="csv"); +w = read(model_path, + data_type="matrix", value_type="double", format="csv"); + +scores = X_test %*% w; +preds = (scores > 0.0); # threshold at 0 (log-odds) +correct = sum(preds == y_test); +n_test = nrow(y_test); +accuracy = correct / n_test; + +print("Accuracy: " + accuracy); +write(accuracy, out_acc, format="csv"); + diff --git a/benchmark/scripts/fedavg_dp.dml b/benchmark/scripts/fedavg_dp.dml new file mode 100644 index 00000000000..d4722fdb2a6 --- /dev/null +++ b/benchmark/scripts/fedavg_dp.dml @@ -0,0 +1,130 @@ +# ── fedavg_dp.dml ──────────────────────────────────────────────────────────── +# Arguments (passed via -nvargs): +# data_dir : path to benchmark/data/ +# epsilon : DP privacy budget ε (use 9999 for non-private baseline) +# delta : DP delta (1e-5 can be used) +# clip_norm : per-example gradient L2-norm clip bound (default 4.0) — +# sensitivity = clip_norm / n follows from this, since +# clipping each record's gradient contribution to clip_norm +# is what makes that bound actually hold. +# n_rounds : number of FedAvg rounds (default 50) +# lr : learning rate (default 0.1) +# w1_rows : rows in worker 1 shard +# w2_rows : rows in worker 2 shard +# w3_rows : rows in worker 3 shard +# w4_rows : rows in worker 4 shard +# n_features : number of features +# out : path to write final weights +# private : 1 = apply DP noise (default), 0 = non-private baseline + +data_dir = $data_dir; +epsilon = $epsilon; +delta = $delta; +clip_norm = ifdef($clip_norm, 4.0); +n_rounds = ifdef($n_rounds, 50); +lr = ifdef($lr, 0.1); +private = ifdef($private, 1); +out_path = $out; + +# Per-round epsilon: dp_gaussian is called once per round, and +# DPBudgetAccountant composes the cost of every release against the single +# total budget set by dp_set_budget(). Spending the full epsilon on every +# round would exhaust the budget almost immediately, so split it evenly +# across rounds instead. +round_epsilon = epsilon / n_rounds; + +w1r = $w1_rows; +w2r = $w2_rows; +w3r = $w3_rows; +w4r = $w4_rows; +d = $n_features; +n = w1r + w2r + w3r + w4r; + +# Sensitivity of the released (mean) gradient to a single record changing: +# with each record's contribution clipped to L2-norm <= clip_norm below, +# the sum can move by at most clip_norm, so the average moves by clip_norm/n. +sensitivity = clip_norm / n; + +# ── Build federated matrix from 4 local workers ─────────────────────────── +# Row ranges are 0-based [begin, end) for each partition. +r1s=0; r1e=w1r; +r2s=w1r; r2e=w1r+w2r; +r3s=w1r+w2r; r3e=w1r+w2r+w3r; +r4s=w1r+w2r+w3r; r4e=n; + +X = federated( + addresses=list( + "localhost:8301/" + data_dir + "/worker1/X_train.csv", + "localhost:8302/" + data_dir + "/worker2/X_train.csv", + "localhost:8303/" + data_dir + "/worker3/X_train.csv", + "localhost:8304/" + data_dir + "/worker4/X_train.csv"), + ranges=list( + list(r1s, 0), list(r1e, d), + list(r2s, 0), list(r2e, d), + list(r3s, 0), list(r3e, d), + list(r4s, 0), list(r4e, d))); + +y = federated( + addresses=list( + "localhost:8301/" + data_dir + "/worker1/y_train.csv", + "localhost:8302/" + data_dir + "/worker2/y_train.csv", + "localhost:8303/" + data_dir + "/worker3/y_train.csv", + "localhost:8304/" + data_dir + "/worker4/y_train.csv"), + ranges=list( + list(r1s, 0), list(r1e, 1), + list(r2s, 0), list(r2e, 1), + list(r3s, 0), list(r3e, 1), + list(r4s, 0), list(r4e, 1))); + +# ── Initialise weights ──────────────────────────────────────────────────── +w = matrix(0, rows=d, cols=1); + +if (private == 1) { + eps = dp_set_budget($epsilon, $delta); +} + +# ── Training rounds ─────────────────────────────────────────────────────── +for (round in 1:n_rounds) { + + # Forward pass — executes on federated workers. + scores = X %*% w; # (n × 1), federated + probs = 1.0 / (1.0 + exp(-scores)); # (n × 1), federated + + residuals = probs - y; # (n × 1), federated + + # DP noise injection — only when private=1. + if (private == 1) { + # Per-example L2-norm clipping: each record's raw gradient + # contribution is X[i,:] * residual_i, with norm + # ||X[i,:]||_2 * |residual_i|. Scaling it down to clip_norm + # whenever it exceeds that bound is what makes + # sensitivity = clip_norm / n (set above) an actual, provable + # bound instead of an arbitrary constant. + row_norms = sqrt(rowSums(X^2)); # (n × 1) ||X[i,:]||_2 + contrib_norms = abs(residuals) * row_norms; # (n × 1) ||X[i,:]*residual_i||_2 + clip_scale = clip_norm / pmax(contrib_norms, clip_norm); # (n × 1), in (0,1] + clipped_residuals = residuals * clip_scale; # (n × 1) + + # Gradient aggregation — t(X) %*% clipped_residuals is a (d × 1) + # local sum that the coordinator collects in one federated + # aggregate instruction. + grad = t(X) %*% clipped_residuals / n; # (d × 1), LOCAL after agg, clipped + + noisy_grad = dp_gaussian(grad, + "identity", + sensitivity=sensitivity, + epsilon=round_epsilon, + delta=delta); + w = w - lr * noisy_grad; + } else { + # Gradient aggregation — t(X) %*% residuals is an (d × 1) local sum + # that the coordinator collects in one federated aggregate instruction. + grad = t(X) %*% residuals / n; # (d × 1), LOCAL after agg + w = w - lr * grad; + } +} + +# ── Write model weights ─────────────────────────────────────────────────── +write(w, out_path, format="csv"); +print("FedAvg done. epsilon=" + epsilon + " rounds=" + n_rounds); + diff --git a/benchmark/scripts/plot.py b/benchmark/scripts/plot.py new file mode 100644 index 00000000000..ac0a82850e3 --- /dev/null +++ b/benchmark/scripts/plot.py @@ -0,0 +1,114 @@ +""" +Read results.csv and produce two figures: + +1. accuracy_vs_epsilon.png + Line plot: x = ε, y = accuracy. + Horizontal dashed line = non-private baseline. + Points labelled with accuracy values. + +2. privacy_cost.png + Bar chart showing accuracy loss relative to baseline (utility cost of DP). +""" +import pathlib +import csv +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import matplotlib.ticker as ticker + +RESULTS = pathlib.Path("benchmark/results") + +# ── Load ────────────────────────────────────────────────────────────────── +rows = [] +with open(RESULTS / "results.csv") as f: + for r in csv.DictReader(f): + rows.append({ + "label": r["label"], + "epsilon": float(r["epsilon"]) if r["epsilon"] != "inf" else None, + "private": int(r["private"]), + "accuracy": float(r["accuracy"]), + }) + +baseline = next(r for r in rows if r["private"] == 0) +dp_rows = sorted([r for r in rows if r["private"] == 1], + key=lambda r: r["epsilon"]) + +eps_vals = [r["epsilon"] for r in dp_rows] +acc_vals = [r["accuracy"] for r in dp_rows] +baseline_acc = baseline["accuracy"] + +# ── Figure 1: Accuracy vs ε ─────────────────────────────────────────────── +fig, ax = plt.subplots(figsize=(7, 4.5)) + +ax.plot(eps_vals, acc_vals, marker="o", linewidth=2, + color="#028090", label="DP-FedAvg (Gaussian)") +ax.axhline(baseline_acc, linestyle="--", color="#1C3A5E", + linewidth=1.5, label=f"Non-private baseline ({baseline_acc:.3f})") + +# Annotate each DP point. +for eps, acc in zip(eps_vals, acc_vals): + ax.annotate(f"{acc:.3f}", xy=(eps, acc), + xytext=(0, 8), textcoords="offset points", + ha="center", fontsize=9, color="#028090") + +ax.set_xscale("log") +ax.set_xticks(eps_vals) +ax.get_xaxis().set_major_formatter(ticker.ScalarFormatter()) +ax.set_xlabel("Privacy budget ε (smaller = stronger privacy)", fontsize=11) +ax.set_ylabel("Test accuracy", fontsize=11) +ax.set_title("Accuracy vs. Privacy Budget — DP-FedAvg on Adult (4 workers)", + fontsize=12) +ax.legend(fontsize=9) +ax.set_ylim(max(0, min(acc_vals) - 0.05), min(1.0, baseline_acc + 0.05)) +ax.grid(True, which="both", linestyle=":", alpha=0.5) + +plt.tight_layout() +out1 = RESULTS / "accuracy_vs_epsilon.png" +fig.savefig(out1, dpi=150) +print(f"Saved {out1}") +plt.close() + +# ── Figure 2: Utility cost (accuracy drop) ──────────────────────────────── +fig, ax = plt.subplots(figsize=(6, 4)) + +drops = [baseline_acc - acc for acc in acc_vals] +colors = ["#B91C1C" if d > 0.02 else "#028090" for d in drops] +# Position bars at their true ε value on a log-scaled x-axis (rather than +# evenly-spaced categorical slots) so the visual spacing between 0.5→1 and +# 4→8 reflects the same 2x ratio. Bar widths scale with x so they stay a +# constant fraction of their slot in log space instead of shrinking/growing. +widths = [e * 0.4 for e in eps_vals] +bars = ax.bar(eps_vals, drops, color=colors, width=widths, + edgecolor="white") +ax.set_xscale("log") +ax.set_xticks(eps_vals) +ax.get_xaxis().set_major_formatter(ticker.ScalarFormatter()) + +for bar, drop in zip(bars, drops): + ax.text(bar.get_x() + bar.get_width() / 2, + bar.get_height() + 0.001, + f"{drop:.3f}", ha="center", va="bottom", fontsize=9) + +ax.legend(["drop > 0.02", "drop <= 0.02"]) + +ax.axhline(0, color="black", linewidth=0.8) +ax.set_xlabel("Privacy budget ε", fontsize=11) +ax.set_ylabel("Accuracy drop vs. baseline", fontsize=11) +ax.set_title("Utility Cost of Differential Privacy — DP-FedAvg on Adult", + fontsize=11) +ax.grid(True, axis="y", linestyle=":", alpha=0.5) + +plt.tight_layout() +out2 = RESULTS / "privacy_cost.png" +fig.savefig(out2, dpi=150) +print(f"Saved {out2}") +plt.close() + +# ── Console summary table ───────────────────────────────────────────────── +print() +print(f"{'ε':>8} {'accuracy':>10} {'drop':>8}") +print("-" * 34) +print(f"{'baseline':>8} {baseline_acc:10.4f} {'—':>8}") +for eps, acc, drop in zip(eps_vals, acc_vals, drops): + print(f"{eps:>8.1f} {acc:10.4f} {drop:8.4f}") + diff --git a/benchmark/scripts/prepare_data.py b/benchmark/scripts/prepare_data.py new file mode 100644 index 00000000000..dce8e8e8006 --- /dev/null +++ b/benchmark/scripts/prepare_data.py @@ -0,0 +1,123 @@ +""" +Download the UCI Adult dataset, binarise labels, standardise features, +split into 4 equal horizontal partitions for federated workers, and write +SystemDS .mtd metadata files alongside each CSV shard. + +Outputs +------- +benchmark/data/worker{1..4}/X_train.csv + X_train.csv.mtd +benchmark/data/worker{1..4}/y_train.csv + y_train.csv.mtd +benchmark/data/X_test.csv + X_test.csv.mtd +benchmark/data/y_test.csv + y_test.csv.mtd +benchmark/data/meta.txt # n_train, n_test, n_features +""" + +import json, os, pathlib +import numpy as np +import pandas as pd +from sklearn.model_selection import train_test_split +from sklearn.preprocessing import StandardScaler + +ADULT_TRAIN_URL = ( + "https://archive.ics.uci.edu/ml/machine-learning-databases" + "/adult/adult.data" +) +ADULT_TEST_URL = ( + "https://archive.ics.uci.edu/ml/machine-learning-databases" + "/adult/adult.test" +) + +COLS = [ + "age","workclass","fnlwgt","education","education_num","marital_status", + "occupation","relationship","race","sex","capital_gain","capital_loss", + "hours_per_week","native_country","label", +] +NUMERIC = ["age","fnlwgt","education_num","capital_gain", + "capital_loss","hours_per_week"] + +DATA_DIR = pathlib.Path("benchmark/data") +N_WORKERS = 4 + +def download(url, dest): + import urllib.request + if not dest.exists(): + print(f"Downloading {url}") + urllib.request.urlretrieve(url, dest) + +def load_adult(path, skip_rows=0): + df = pd.read_csv(path, names=COLS, skipinitialspace=True, + skiprows=skip_rows, na_values="?").dropna() + # binarise label: >50K → 1, else 0 + df["label"] = (df["label"].str.strip().str.rstrip(".") == ">50K").astype(float) + # one-hot encode categoricals + cats = [c for c in COLS[:-1] if c not in NUMERIC] + df = pd.get_dummies(df, columns=cats, drop_first=True) + return df + +def write_csv_and_mtd(arr: np.ndarray, path: pathlib.Path, description: str): + """Write a CSV and a matching SystemDS .mtd metadata file.""" + path.parent.mkdir(parents=True, exist_ok=True) + np.savetxt(path, arr, delimiter=",", fmt="%.8f") + rows, cols = arr.shape + mtd = { + "data_type": "matrix", + "value_type": "double", + "rows": rows, + "cols": cols, + "format": "csv", + "header": False, + "description": description, + } + with open(str(path) + ".mtd", "w") as f: + json.dump(mtd, f, indent=2) + print(f" {path} ({rows} × {cols})") + +# ── Download ───────────────────────────────────────────────────────────────── +download(ADULT_TRAIN_URL, DATA_DIR / "raw" / "adult.data") +download(ADULT_TEST_URL, DATA_DIR / "raw" / "adult.test") + +train_df = load_adult(DATA_DIR / "raw" / "adult.data") +test_df = load_adult(DATA_DIR / "raw" / "adult.test", skip_rows=1) + +# Align columns (test may have different dummies after get_dummies). +train_df, test_df = train_df.align(test_df, join="left", axis=1, fill_value=0) + +# ── Feature / label split ──────────────────────────────────────────────────── +feature_cols = [c for c in train_df.columns if c != "label"] +X_train = train_df[feature_cols].values.astype(float) +y_train = train_df["label"].values.reshape(-1, 1).astype(float) +X_test = test_df[feature_cols].values.astype(float) +y_test = test_df["label"].values.reshape(-1, 1).astype(float) + +# ── Standardise (fit on train only) ───────────────────────────────────────── +scaler = StandardScaler() +X_train = scaler.fit_transform(X_train) +X_test = scaler.transform(X_test) + +n_train, n_features = X_train.shape +n_test = X_test.shape[0] +print(f"Train: {n_train} × {n_features} | Test: {n_test} × {n_features}") + +# ── Partition across workers (equal horizontal splits) ────────────────────── +indices = np.array_split(np.arange(n_train), N_WORKERS) +for i, idx in enumerate(indices, start=1): + wdir = DATA_DIR / f"worker{i}" + write_csv_and_mtd(X_train[idx], wdir / "X_train.csv", + f"Adult features, worker {i}") + write_csv_and_mtd(y_train[idx], wdir / "y_train.csv", + f"Adult labels, worker {i}") + +# ── Test set (coordinator-local) ──────────────────────────────────────────── +write_csv_and_mtd(X_test, DATA_DIR / "X_test.csv", "Adult test features") +write_csv_and_mtd(y_test, DATA_DIR / "y_test.csv", "Adult test labels") + +# ── Metadata for DML scripts ───────────────────────────────────────────────── +worker_rows = [len(idx) for idx in indices] +with open(DATA_DIR / "meta.txt", "w") as f: + f.write(f"n_train={n_train}\n") + f.write(f"n_test={n_test}\n") + f.write(f"n_features={n_features}\n") + for i, r in enumerate(worker_rows, start=1): + f.write(f"worker{i}_rows={r}\n") +print("Wrote benchmark/data/meta.txt") + diff --git a/benchmark/scripts/run_benchmark.sh b/benchmark/scripts/run_benchmark.sh new file mode 100755 index 00000000000..70966fa30ad --- /dev/null +++ b/benchmark/scripts/run_benchmark.sh @@ -0,0 +1,15 @@ +# 1. Prepare data (once). +python benchmark/scripts/prepare_data.py + +# 2. Run the sweep (starts workers, trains, evaluates, stops workers). +bash benchmark/scripts/run_sweep.sh + +# 3. Collect results into CSV. +python benchmark/scripts/collect_results.py + +# 4. Generate plots. +python benchmark/scripts/plot.py + +# 5. Confirm outputs exist. +ls -lh benchmark/results/accuracy_vs_epsilon.png benchmark/results/privacy_cost.png + diff --git a/benchmark/scripts/run_sweep.sh b/benchmark/scripts/run_sweep.sh new file mode 100755 index 00000000000..4696ea5240a --- /dev/null +++ b/benchmark/scripts/run_sweep.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Runs FedAvg for each epsilon value and the non-private baseline, +# then evaluates accuracy. Results are appended to results/results.csv. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +JAR="$REPO_ROOT/target/SystemDS.jar" +SCRIPTS="$REPO_ROOT/benchmark/scripts" +DATA="$REPO_ROOT/benchmark/data" +RESULTS="$REPO_ROOT/benchmark/results" +mkdir -p "$RESULTS" + +# Read dataset metadata written by prepare_data.py. +source <(grep -E '^(n_train|n_test|n_features|worker[1-4]_rows)=' \ + "$DATA/meta.txt" | sed 's/=/="/;s/$/"/') + +COMMON_ARGS="\ + data_dir=$DATA \ + n_features=$n_features \ + w1_rows=$worker1_rows \ + w2_rows=$worker2_rows \ + w3_rows=$worker3_rows \ + w4_rows=$worker4_rows \ + n_rounds=300 \ + lr=1.0 \ + clip_norm=4.0 \ + delta=1e-5" + +# ── Start workers ───────────────────────────────────────────────────────── +echo "=== Starting federated workers ===" +bash "$SCRIPTS/start_workers.sh" + +run_one() { + local label="$1" # e.g. "eps_0.5" or "baseline" + local extra="$2" # extra -nvargs for this run + local model="$RESULTS/model_${label}.csv" + local acc_file="$RESULTS/acc_${label}.txt" + + echo "" + echo "--- Training: $label ---" + java --add-modules=jdk.incubator.vector -jar "$JAR" \ + -f "$SCRIPTS/fedavg_dp.dml" \ + -nvargs $COMMON_ARGS $extra out="$model" \ + 2>&1 | tee "$RESULTS/train_${label}.log" | grep -E "FedAvg|error|Error" || true + + echo " Evaluating …" + java --add-modules=jdk.incubator.vector -jar "$JAR" \ + -f "$SCRIPTS/eval.dml" \ + -nvargs data_dir="$DATA" model_path="$model" out_acc="$acc_file" \ + 2>&1 | grep "Accuracy:" +} + +# ── Non-private baseline ────────────────────────────────────────────────── +run_one "baseline" "private=0 epsilon=9999" + +# ── DP runs ─────────────────────────────────────────────────────────────── +for EPS in 0.5 1 4 8; do + LABEL="eps_${EPS}" + run_one "$LABEL" "private=1 epsilon=${EPS}" +done + +# ── Stop workers ────────────────────────────────────────────────────────── +echo "" +echo "=== Stopping federated workers ===" +bash "$SCRIPTS/stop_workers.sh" + +echo "" +echo "All runs complete. Logs and model files in $RESULTS/" + diff --git a/benchmark/scripts/start_workers.sh b/benchmark/scripts/start_workers.sh new file mode 100755 index 00000000000..23b5634230c --- /dev/null +++ b/benchmark/scripts/start_workers.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Start 4 local SystemDS federated workers on ports 8301-8304. +# Each worker is given the absolute path to its data shard directory. +set -e +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +JAR="$REPO_ROOT/target/SystemDS.jar" +DATA_DIR="$REPO_ROOT/benchmark/data" +LOG_DIR="$REPO_ROOT/benchmark/results" +mkdir -p "$LOG_DIR" + +for i in 1 2 3 4; do + PORT=$((8300 + i)) + echo "Starting worker $i on port $PORT …" + java --add-modules=jdk.incubator.vector -jar "$JAR" \ + -w "$PORT" \ + > "$LOG_DIR/worker${i}.log" 2>&1 & + echo $! > "$LOG_DIR/worker${i}.pid" +done + +# Give workers time to bind their ports. +sleep 3 +echo "Workers running. PIDs:" +for i in 1 2 3 4; do cat "$LOG_DIR/worker${i}.pid"; done + diff --git a/benchmark/scripts/stop_workers.sh b/benchmark/scripts/stop_workers.sh new file mode 100755 index 00000000000..d1ff3cfde41 --- /dev/null +++ b/benchmark/scripts/stop_workers.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +LOG_DIR="$(cd "$(dirname "$0")/../../benchmark/results" && pwd)" +for i in 1 2 3 4; do + PID_FILE="$LOG_DIR/worker${i}.pid" + if [ -f "$PID_FILE" ]; then + PID=$(cat "$PID_FILE") + kill "$PID" 2>/dev/null && echo "Stopped worker $i (PID $PID)" || true + rm -f "$PID_FILE" + fi +done + diff --git a/src/main/python/requirements.txt b/src/main/python/requirements.txt new file mode 100644 index 00000000000..5d17a102c29 --- /dev/null +++ b/src/main/python/requirements.txt @@ -0,0 +1,31 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +numpy +pandas +scipy +py4j +wheel +requests +setuptools + +scikit-learn +matplotlib