Datalevin Documentation Site

Prerequisites

• Java 21+
• Clojure CLI 1.12+
• Node.js / npm (for Tailwind CSS)

Setup

# Clone with submodules
git clone --recurse-submodules <repo-url>

# Or if already cloned, init submodules
git submodule update --init --recursive

# Install npm dependencies (Tailwind CSS)
npm install

# Create .env from example
cp .env.example .env

Edit .env as needed. The defaults work for local development; GitHub OAuth credentials are optional.

Development

Start the nREPL:

clojure -M:dev

Then in the REPL:

(require 'dev)
(dev/start)

This starts:

• Web server on http://localhost:3000
• Tailwind CSS watcher (recompiles on class changes)
• Live reload via file watcher — editing .clj, .css, or .md files triggers an automatic browser refresh

Other REPL commands:

(dev/stop)      ; stop everything
(dev/restart)   ; full restart
(dev/reload!)   ; manually trigger a browser refresh

Multi-Language Code Examples

For Datalevin API examples that apply across supported host languages, wrap the parallel code fences in a multi-lang block and keep this order: Clojure, Java, Python, JavaScript.

<div class="multi-lang">

```clojure
(d/q '[:find ?e :where [?e :user/name "Alice"]] db)
```

```java
Datalevin.q("[:find ?e :where [?e :user/name \"Alice\"]]", db);
```

```python
d.q('[:find ?e :where [?e :user/name "Alice"]]', db)
```

```javascript
d.q(`[:find ?e :where [?e :user/name "Alice"]]`, db);
```

</div>

Use ordinary single-language fences for shell commands, SQL, EDN/Datalog reference snippets, Babashka pods, server REPL tasks, and Clojure-specific APIs.

Pre-Publication Example Verification

The docs include a manifest-driven verifier for Clojure examples:

# Inventory Clojure fences and current manifest status counts
clojure -X:verify-examples :mode :inventory

# Add/update manifest skeleton entries after large doc edits
clojure -X:verify-examples :mode :write-skeleton

# Run examples marked :runnable in test/doc_example_manifest.edn
clojure -X:verify-examples

# Run the same runnable Clojure examples through Babashka pod mode
clojure -X:verify-examples :runtime :pod :pod-binary '"dtlv"'

# Release-gate mode: fail if any Clojure fence is still unclassified
clojure -X:verify-examples :fail-on-unclassified true

Classify each executable-language code fence in test/doc_example_manifest.edn as one of:

• runnable: self-contained or runnable with a small chapter fixture
• fragment: illustrative code that depends on omitted surrounding setup
• external: requires a server, network service, credentials, native model, or destructive/operator action
• api-sketch: intentionally not a tested public API example

The release-blocking verification pass should mark and run every runnable Clojure example, then run a representative Java/Python/JavaScript matrix for each API family covered by multi-language examples. Do not claim in the manuscript that examples were executed until the manifest records the commands, Datalevin version, host runtimes, and pass/fail results.

Skeleton generation preserves existing classifications for current snippet IDs, updates source/preview metadata, adds new snippets, and drops entries for removed snippets.

Manifest entries may use :setup and :teardown strings for local fixtures. For repeated fixtures, define a top-level :fixtures map and reference entries with :fixture.

Build CSS (production)

npm run css:build

Build uberjar

Build a standalone jar for small VMs:

npm run css:build
clojure -T:build uber

This produces:

target/datalevin-docs-standalone.jar

The uberjar avoids resolving the production classpath on every boot and is the preferred deployment target on a 1 GB VM.

Build final PDF

The final PDF is generated from the Markdown chapters with a print-oriented preprocessing step. The generated PDF uses a page-numbered table of contents, adds an index from resources/docs/index-terms.edn and generated function entries, and keeps Clojure fenced code blocks only; Java, Python, JavaScript, shell, JSON, SQL, and other non-Clojure fenced blocks remain in the web book but are omitted from the print PDF source.

Prerequisites:

• pandoc
• XeLaTeX, usually from TeX Live
• makeindex, usually included with TeX Live
• rsvg-convert for SVG diagrams
• The PDF type stack configured in build.clj: Charter for body text, Avenir Next for headings, and Menlo for code. These are available on macOS; adjust the font metadata in build.clj if building on another platform.

Build it with:

clojure -T:build final-pdf

This produces:

target/pdf/datalevin-definitive-guide.md
target/pdf/datalevin-definitive-guide.tex
target/pdf/datalevin-definitive-guide.pdf

Set DATALEVIN_VERSION to override the default version used for {{datalevin-version}} substitution.

Build the Kindle EPUB

The Kindle edition is a reflowable EPUB 3 built directly from the Markdown chapters. It keeps the part, chapter, and section hierarchy in the eBook navigation; retains Clojure and command-line examples; preserves working links; and uses responsive styling for code, tables, callouts, and diagrams. This build is independent of the print PDF and does not modify the print artifacts. The cover is intentionally not embedded in the manuscript because KDP adds the separately uploaded marketing cover to the Kindle book.

Prerequisites:

• pandoc
• rsvg-convert for the eBook cover image

Build it with:

clojure -T:build epub

This produces:

target/epub/datalevin-definitive-guide.md
target/epub/datalevin-definitive-guide.epub
target/epub/datalevin-definitive-guide-cover.jpg

Upload the .epub file as the manuscript and the .jpg file as the separate Kindle eBook cover. Use KDP's online previewer to check a phone, tablet, and e-reader before publishing.

Production

Run the release gate and build the self-contained JAR:

npm ci
npm run css:build
clojure -X:prod:test
clojure -T:build uber

The website build removes only its old classes and JAR, so print, eBook, and Ingram artifacts elsewhere under target/ are preserved. It validates resources/docs/manifest.edn and produces:

target/datalevin-docs-standalone.jar

The JAR includes the Markdown chapters and ensures the documentation search index is current before accepting traffic. Deploy the JAR alone; a source checkout is not required at runtime.

scripts/start-prod.sh requires Java 21+, sets ENV=prod, validates the JAR, and starts it with bounded heap/metaspace settings. Application startup fails before opening the database when production configuration is unsafe, including:

• a non-HTTPS or non-origin BASE_URL;
• a missing, default, or shorter-than-32-character SESSION_SECRET;
• a missing DATALEVIN_VERSION;
• incomplete OAuth or SMTP credential pairs;
• conflicting SMTP SSL/TLS settings or SMTP debug logging;
• a configured REINDEX_SECRET shorter than 32 characters.

For a manual smoke run, copy .env.example, set real production values, export them, and run:

scripts/start-prod.sh

That wrapper runs the same jar-based startup pattern as the systemd unit:

java -Xms256m -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication -XX:MaxMetaspaceSize=128m -jar target/datalevin-docs-standalone.jar

The production defaults target the 16 GB Netcup host while leaving ample room for Datalevin's mmap usage, MariaDB, phpBB, and the operating-system page cache. The wrapper defaults to G1 with string deduplication and a 128 MB metaspace cap. You can override the defaults for a smaller host if needed:

JAVA_XMS=256m JAVA_XMX=384m scripts/start-prod.sh

If you want a lower-overhead collector on a very small VM, switch the wrapper to Serial GC:

JAVA_GC=serial scripts/start-prod.sh

Override APP_JAR if the jar lives outside the repo checkout:

APP_JAR=/home/hyang/apps/datalevin/datalevin-docs-standalone.jar scripts/start-prod.sh

Production topology

Bind the application to loopback and terminate TLS at a reverse proxy:

Internet -> HTTPS reverse proxy -> 127.0.0.1:3000 -> Datalevin Docs

Set BIND_HOST=127.0.0.1. Set TRUST_PROXY=true only when the application cannot be reached directly and the proxy overwrites X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Host; otherwise leave it false. The canonical public origin still comes from BASE_URL.

For GitHub OAuth, register this production callback URL when enabled:

https://datalevin.org/auth/github/callback

The reverse proxy or load balancer should use GET /health/ready. Available health endpoints are:

• GET /health/live: process liveness;
• GET /health/ready and GET /health: readiness, including a live Datalevin connection.

systemd

An example unit file is included at deploy/systemd/datalevin-docs.service. It uses a 1 GB JVM heap, keeps MemoryHigh=3G and OOMPolicy=stop, and relies on Datalevin's own mapsize instead of a hard MemoryMax cap. The JAR path is configured separately so it can remain in the deployer's existing application directory.

Typical install steps:

sudo useradd --system --home /nonexistent --shell /usr/sbin/nologin datalevin
mkdir -p "$HOME/apps/datalevin"
install -m 0644 target/datalevin-docs-standalone.jar "$HOME/apps/datalevin/datalevin-docs-standalone.jar"
sudo setfacl -m u:datalevin:--x "$HOME" "$HOME/apps" "$HOME/apps/datalevin"
sudo install -d -o root -g datalevin -m 0750 /etc/datalevin-docs
sudo install -m 0644 deploy/systemd/datalevin-docs.service /etc/systemd/system/datalevin-docs.service
sudo install -o root -g datalevin -m 0640 deploy/systemd/datalevin-docs.env.example /etc/datalevin-docs/datalevin-docs.env

Edit /etc/datalevin-docs/datalevin-docs.env. Do not put ENV in this file; the unit pins it to prod. At minimum, configure:

APP_JAR=/home/hyang/apps/datalevin/datalevin-docs-standalone.jar
PORT=3000
BIND_HOST=127.0.0.1
BASE_URL=https://datalevin.org
DB_PATH=/var/lib/datalevin-docs/data
DATALEVIN_VERSION=1.0.0
SESSION_SECRET=replace-with-at-least-32-random-characters
TRUST_PROXY=true

Systemd does not expand ~ in environment files, so the JAR uses the absolute path under /home/hyang. The unit exposes home directories read-only inside its mount namespace, and the ACL grants the datalevin service account only path traversal; the JAR itself remains world-readable but writable only by the deployer.

Generate a session secret with a cryptographically secure tool, for example openssl rand -base64 48. Keep the environment file out of source control and rotate credentials if they are ever exposed. Without SMTP config, the site can start, but registration verification and password-reset delivery fail safely; development continues to log those messages for local testing.

To enable email delivery later, add the IONOS SMTP settings:

MAIL_FROM="Datalevin Docs <your-mailbox@datalevin.org>"
SMTP_HOST=smtp.ionos.com
SMTP_PORT=587
SMTP_USER=your-mailbox@datalevin.org
SMTP_PASS=your-mailbox-password
SMTP_SSL=false
SMTP_TLS=true
SMTP_DEBUG=false

Then enable the service:

sudo systemctl daemon-reload
sudo systemctl enable --now datalevin-docs
curl --fail --silent http://127.0.0.1:3000/health/ready

The unit creates /var/lib/datalevin-docs with the correct service ownership, uses graceful shutdown, restarts on failure, and limits filesystem writes to the Datalevin state directory and private temporary storage.

Deploy, backup, and rollback

Before replacing a running release, make an online Datalevin copy. The dtlv copy command is safe while the database is in use:

sudo install -d -o datalevin -g datalevin -m 0750 /var/backups/datalevin-docs
sudo -u datalevin dtlv -d /var/lib/datalevin-docs/data -c copy /var/backups/datalevin-docs/data-$(date -u +%Y%m%dT%H%M%SZ)

Deploy and verify:

rsync --chmod=F644 target/datalevin-docs-standalone.jar hyang@SERVER:/home/hyang/apps/datalevin/datalevin-docs-standalone.jar.new
ssh hyang@SERVER
mv "$HOME/apps/datalevin/datalevin-docs-standalone.jar.new" "$HOME/apps/datalevin/datalevin-docs-standalone.jar"
sudo systemctl restart datalevin-docs
sudo systemctl --no-pager --full status datalevin-docs
curl --fail --silent http://127.0.0.1:3000/health/ready

Keep the previous JAR until the new release passes readiness and a public HTTPS smoke check. To roll back application code, reinstall that previous JAR and restart the service. If data restoration is required, stop the service first, preserve the failed database directory for investigation, copy the selected backup back to /var/lib/datalevin-docs/data, restore ownership to datalevin:datalevin, and then start the service. Test backup restoration on a non-production host periodically; an untested backup is not a recovery plan.