Refactor DAV client, add missing operations

.github/scripts/dav/README.md

@@ -0,0 +1,81 @@
+# DAV Integration Test Scripts
+
+Scripts for setting up and running WebDAV integration tests locally and in CI.
+
+## Scripts
+
+### `setup.sh`
+Builds and starts a WebDAV test server using Docker.
+
+**Usage:**
+```bash
+./.github/scripts/dav/setup.sh
+```
+
+This will:
+- Clean up any existing WebDAV containers
+- Build the `webdav-test` Docker image from `dav/integration/testdata/`
+- Start the server on `https://localhost:8443`
+- Verify the server is ready
+
+### `run-int.sh`
+Runs the DAV integration tests.
+
+**Required environment variables:**
+- `DAV_ENDPOINT` - WebDAV server URL (e.g., `https://localhost:8443`)
+- `DAV_USER` - Username for authentication (e.g., `testuser`)
+- `DAV_PASSWORD` - Password for authentication (e.g., `testpass`)
+- `DAV_CA_CERT` - CA certificate content (optional, for HTTPS with custom CA)
+- `DAV_SECRET` - Secret for signed URLs (optional, for signed URL tests)
+
+**Usage:**
+```bash
+export DAV_ENDPOINT="https://localhost:8443"
+export DAV_USER="testuser"
+export DAV_PASSWORD="testpass"
+export DAV_CA_CERT="$(cat dav/integration/testdata/certs/server.crt)"
+export DAV_SECRET="test-secret-key"
+
+./.github/scripts/dav/run-int.sh
+```
+
+### `teardown.sh`
+Cleans up the WebDAV test environment.
+
+**Usage:**
+```bash
+./.github/scripts/dav/teardown.sh
+```
+
+This will:
+- Stop and remove the WebDAV container
+- Remove the WebDAV test Docker image
+
+### `utils.sh`
+Utility functions used by other scripts. Contains:
+- `cleanup_webdav_container` - Stops and removes the WebDAV container
+- `cleanup_webdav_image` - Removes the WebDAV test image
+
+## Running Locally
+
+To run the full integration test suite locally:
+
+```bash
+# From the repository root
+./.github/scripts/dav/setup.sh
+
+export DAV_ENDPOINT="https://localhost:8443"
+export DAV_USER="testuser"
+export DAV_PASSWORD="testpass"
+export DAV_CA_CERT="$(cat dav/integration/testdata/certs/server.crt)"
+export DAV_SECRET="test-secret-key"
+
+./.github/scripts/dav/run-int.sh
+
+# Cleanup
+./.github/scripts/dav/teardown.sh
+```
+
+## CI Usage
+
+These scripts are used by the GitHub Actions workflow in `.github/workflows/dav-integration.yml`.

.github/scripts/dav/run-int.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Get the directory where this script is located
+script_dir="$( cd "$(dirname "${0}")" && pwd )"
+repo_root="$(cd "${script_dir}/../../.." && pwd)"
+
+: "${DAV_ENDPOINT:?DAV_ENDPOINT environment variable must be set}"
+: "${DAV_USER:?DAV_USER environment variable must be set}"
+: "${DAV_PASSWORD:?DAV_PASSWORD environment variable must be set}"
+
+echo "Running DAV integration tests..."
+echo "  Endpoint: ${DAV_ENDPOINT}"
+echo "  User: ${DAV_USER}"
+
+pushd "${repo_root}/dav" > /dev/null
+  echo -e "\nRunning tests with $(go version)..."
+  ginkgo -v ./integration
+popd > /dev/null

.github/scripts/dav/setup.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Get the directory where this script is located
+script_dir="$( cd "$(dirname "${0}")" && pwd )"
+repo_root="$(cd "${script_dir}/../../.." && pwd)"
+
+source "${script_dir}/utils.sh"
+
+# Cleanup any existing containers first
+cleanup_webdav_container
+
+echo "Building WebDAV test server Docker image..."
+cd "${repo_root}/dav/integration/testdata"
+docker build -t webdav-test .
+
+echo "Starting WebDAV test server..."
+docker run -d --name webdav -p 8443:443 webdav-test
+
+# Wait for Apache to be ready
+echo "Waiting for Apache to start..."
+sleep 5
+
+# Verify htpasswd file in container
+echo "Verifying htpasswd file in container..."
+docker exec webdav cat /usr/local/apache2/htpasswd
+
+# Test connection
+echo "Testing WebDAV server connection..."
+if curl -k -u testuser:testpass -v https://localhost:8443/ 2>&1 | grep -q "200 OK\|301\|Authorization"; then
+    echo "✓ WebDAV server is ready"
+else
+    echo "⚠ WebDAV server might not be fully ready yet"
+fi

.github/scripts/dav/teardown.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+script_dir="$( cd "$(dirname "${0}")" && pwd )"
+
+source "${script_dir}/utils.sh"
+
+echo "Tearing down WebDAV test environment..."
+cleanup_webdav_container
+cleanup_webdav_image
+
+echo "✓ Teardown complete"

.github/scripts/dav/utils.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+# Cleanup Docker container and image
+function cleanup_webdav_container {
+    echo "Stopping and removing WebDAV container..."
+    docker stop webdav 2>/dev/null || true
+    docker rm webdav 2>/dev/null || true
+}
+
+function cleanup_webdav_image {
+    echo "Removing WebDAV test image..."
+    docker rmi webdav-test 2>/dev/null || true
+}

.github/workflows/dav-integration.yml

@@ -0,0 +1,49 @@
+name: DAV Integration Tests
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+    - ".github/workflows/dav-integration.yml"
+    - "dav/**"
+    - "go.mod"
+    - "go.sum"
+  push:
+    branches:
+      - main
+
+concurrency:
+  group: dav-integration
+  cancel-in-progress: false
+
+jobs:
+  dav-integration:
+    name: DAV Integration Tests
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+
+      - name: Install Ginkgo
+        run: go install github.com/onsi/ginkgo/v2/ginkgo@latest
+
+      - name: Setup WebDAV test server
+        run: ./.github/scripts/dav/setup.sh
+
+      - name: Run Integration Tests
+        env:
+          DAV_ENDPOINT: "https://localhost:8443"
+          DAV_USER: "testuser"
+          DAV_PASSWORD: "testpass"
+          DAV_SECRET: "test-secret-key"
+          DAV_CA_CERT_FILE: "dav/integration/testdata/certs/server.crt"
+        run: |
+          export DAV_CA_CERT="$(cat ${DAV_CA_CERT_FILE})"
+          ./.github/scripts/dav/run-int.sh
+

.github/workflows/unit-test.yml

@@ -42,7 +42,7 @@ jobs:
               run: |
                     export CGO_ENABLED=0
                     go version
-                    go test -v ./dav/...
+                    go run github.com/onsi/ginkgo/v2/ginkgo --skip-package=integration ./dav/...
             
             - name: gcs unit tests
               run: |

dav/README.md

@@ -8,42 +8,129 @@ For general usage and build instructions, see the [main README](../README.md).
 
 ## DAV-Specific Configuration
 
-The DAV client requires a JSON configuration file with WebDAV endpoint details and credentials.
+The DAV client requires a JSON configuration file with the following structure:
+
+``` json
+{
+  "endpoint":        "<string> (required)",
+  "user":            "<string> (optional)",
+  "password":        "<string> (optional)",
+  "retry_attempts":  <uint> (optional - default: 3),
+  "tls": {
+    "cert": {
+      "ca":          "<string> (optional - PEM-encoded CA certificate)"
+    }
+  },
+  "secret":          "<string> (optional - required for pre-signed URLs)",
+  "signing_method":  "<string> (optional - 'sha256' (default) or 'md5')"
+}
+```
 
 **Usage examples:**
 ```bash
-# Upload an object
-storage-cli -s dav -c dav-config.json put local-file.txt remote-object
+# Upload a blob
+storage-cli -s dav -c dav-config.json put local-file.txt remote-blob
+
+# Fetch a blob (destination file will be overwritten if exists)
+storage-cli -s dav -c dav-config.json get remote-blob local-file.txt
+
+# Delete a blob
+storage-cli -s dav -c dav-config.json delete remote-blob
+
+# Check if blob exists
+storage-cli -s dav -c dav-config.json exists remote-blob
 
-# Fetch an object
-storage-cli -s dav -c dav-config.json get remote-object local-file.txt
+# List all blobs
+storage-cli -s dav -c dav-config.json list
 
-# Delete an object
-storage-cli -s dav -c dav-config.json delete remote-object
+# List blobs with prefix
+storage-cli -s dav -c dav-config.json list my-prefix
 
-# Check if an object exists
-storage-cli -s dav -c dav-config.json exists remote-object
+# Copy a blob
+storage-cli -s dav -c dav-config.json copy source-blob destination-blob
 
-# Generate a signed URL (e.g., GET for 1 hour)
-storage-cli -s dav -c dav-config.json sign remote-object get 60s
+# Delete blobs by prefix
+storage-cli -s dav -c dav-config.json delete-recursive my-prefix-
+
+# Get blob properties (outputs JSON with ContentLength, ETag, LastModified)
+storage-cli -s dav -c dav-config.json properties remote-blob
+
+# Ensure storage exists (initialize WebDAV storage)
+storage-cli -s dav -c dav-config.json ensure-storage-exists
+
+# Generate a pre-signed URL (e.g., GET for 3600 seconds)
+storage-cli -s dav -c dav-config.json sign remote-blob get 3600s
+```
+
+### Using Signed URLs with curl
+
+```bash
+# Downloading a blob:
+curl -X GET <signed-url>
+
+# Uploading a blob:
+curl -X PUT -T path/to/file <signed-url>
 ```
 
 ## Pre-signed URLs
 
 The `sign` command generates a pre-signed URL for a specific object, action, and duration.
 
-The request is signed using HMAC-SHA256 with a secret provided in the configuration.
+The request is signed using the algorithm selected by `signing_method` configuration parameter with a secret provided in the configuration.
+
+**Supported signing methods:**
+- **`sha256`** (default): HMAC-SHA256 signature
+- **`md5`**: MD5-based signature
+
+The exact signature format depends on the selected signing method and signer implementation.
+
+The generated URL format varies based on signing method:
+- **SHA256**: `/signed/{blob-path}?st={hmac-sha256}&ts={timestamp}&e={expires}`
+- **MD5**: `/read/{blob-path}?md5={md5-hash}&expires={timestamp}` or `/write/{blob-path}?md5={md5-hash}&expires={timestamp}`
 
-The HMAC format is:
-`<HTTP Verb><Object ID><Unix timestamp of the signature time><Unix timestamp of the expiration time>`
+**Note:** Pre-signed URLs require the WebDAV server to have signature verification middleware. Standard WebDAV servers don't support this - it's a Cloud Foundry extension.
 
-The generated URL format:
-`https://blobstore.url/signed/object-id?st=HMACSignatureHash&ts=GenerationTimestamp&e=ExpirationTimestamp`
+## Object Path Handling
+
+The DAV client treats object IDs as the final storage paths and uses them exactly as provided by the caller. The client does not apply any path transformations, partitioning, or prefixing - the caller is responsible for providing the complete object path including any directory structure.
+
+For example:
+- Simple paths: `my-blob-id`
+- Partitioned paths: `ab/cd/my-blob-id`
+- Nested paths: `folder/subfolder/my-blob-id`
+
+All are stored exactly as specified. If your use case requires a specific directory layout (e.g., partitioning by hash prefix), implement this in the caller before invoking storage-cli.
+
+## Features
+
+### Automatic Retry Logic
+All operations automatically retry on transient errors with 1-second delays between attempts. Default is 3 retry attempts, configurable via `retry_attempts` in config.
+
+### TLS/HTTPS Support
+Supports HTTPS connections with custom CA certificates for internal or self-signed certificates.
 
 ## Testing
 
 ### Unit Tests
 Run unit tests from the repository root:
+
 ```bash
-ginkgo --cover -v -r ./dav/...
+ginkgo --cover -v -r ./dav/client
 ```
+
+Or using go test:
+```bash
+go test ./dav/client/...
+```
+
+### Integration Tests
+
+The DAV implementation includes Go-based integration tests that run against a real WebDAV server. These tests require a WebDAV server to be available and the following environment variables to be set:
+
+- `DAV_ENDPOINT` - WebDAV server URL
+- `DAV_USER` - Username for authentication
+- `DAV_PASSWORD` - Password for authentication
+- `DAV_CA_CERT` - CA certificate (optional, for HTTPS with custom CA)
+- `DAV_SECRET` - Secret for signed URLs (optional, for signed URL tests)
+
+If these environment variables are not set, the integration tests will be skipped.