diff --git a/docs/developer-guide/adding-widgets-on-insights.md b/docs/developer-guide/adding-widgets-on-insights.md index 78f7e179..593640b4 100644 --- a/docs/developer-guide/adding-widgets-on-insights.md +++ b/docs/developer-guide/adding-widgets-on-insights.md @@ -1,5 +1,5 @@ --- -sidebar_position: 10 +sidebar_position: 35 --- # Adding Widgets on Insights diff --git a/docs/developer-guide/agent-protocol-spi.md b/docs/developer-guide/agent-protocol-spi.md new file mode 100644 index 00000000..f1fce3f2 --- /dev/null +++ b/docs/developer-guide/agent-protocol-spi.md @@ -0,0 +1,16 @@ +--- +sidebar_position: 13 +--- + +# Agent and protocol SPI + +See the `HTTPProtocol.java` as an example as this will give you a good guide of what to do to build protocol: + +`agent/src/main/java/org/openremote/agent/protocol/http/HTTPProtocol.java` + +Here’s some helpful info: + +- Protocols are one instance per Agent; you’ll see that each Protocol has a corresponding Agent class see here for example. +- Each Asset type has a concrete class and Agents are a sub type of Asset so they therefore have their own classes also (this gives us better type safety) +- An Agent's configuration is stored in individual attributes; these attributes are defined in the Agent class. +- agentLink MetaItem now contains all the configuration needed to connect an attribute to a specific Agent diff --git a/docs/developer-guide/building-and-testing.md b/docs/developer-guide/building-and-testing.md deleted file mode 100644 index 234909c6..00000000 --- a/docs/developer-guide/building-and-testing.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -sidebar_position: 4 ---- - -# Building and testing - -## Building - -Ensure you have prepared your environment as described [here](preparing-the-environment.md). - -Our build tooling is `gradle` and building the code is just a matter of executing the following gradle tasks: - -```shell -./gradlew clean installDist -``` - -## Testing - -Our test suite is mostly integration tests and they require the `keycloak` and `postgres` services to be running on `localhost`, you can start these using the `/profile/dev-testing.yml` Docker Compose profile. - -If you want to run the tests, execute: - -```shell -./gradlew test -``` diff --git a/docs/developer-guide/connecting-protocol-adaptors-with-agents.md b/docs/developer-guide/connecting-protocol-adaptors-with-agents.md deleted file mode 100644 index 7b5e9d2f..00000000 --- a/docs/developer-guide/connecting-protocol-adaptors-with-agents.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -sidebar_position: 12 ---- - -# Connecting Protocol adaptors with Agents - -The Agent asset is special: you create it to connect some services or devices to the OpenRemote context, so you can manage them as assets, too. It's the glue in OpenRemote that lets you connect external services to the asset model, you connect sensors and actuators. - -An agent may be running locally or remotely on a gateway that is closer to the devices you want to connect. The asset details in the OpenRemote context representing your devices and services reflect the latest known state of the things you have deployed in the field. - -Protocols are a main extension point of OpenRemote, they translate the messages from and to external systems into reads and writes of the assets and attribute values used by OpenRemote. - -When creating an agent asset, you can create protocol configurations, which are a special type of attribute. Each agent attribute that is a protocol configuration then automatically gets its own instance of the protocol you have selected. - -Imagine you want to plug several USB adapters into your IoT gateway and have each represented by its own protocol configuration attribute on an agent, for example as a KNX and ZWave configuration. You then also store and update the agent asset location to where the gateway box is located, and customize other attributes that help you manage devices in the field. - -You can link any asset attribute in your context to an agent protocol configuration. When an attribute is linked to an agent's protocol configuration, all value change and write operations of the attribute are delegated to the agent and ultimately the protocol implementation. - -Whenever a client sends a message to write an attribute value, when that attribute is linked to an agent the message is passed through to the agent and no rules, flows, database and therefore context update is made. The protocol may decide to update the attribute's state in the context, the same way it would perform a regular update from a sensor read. This is how OpenRemote maps attribute value changes to actuator events. - -## Agent model and protocol SPI ## - -See the KNXProtocol.java as an example as this will give you a good guide of what to do to build protocol: - -`agent/src/main/java/org/openremote/agent/protocol/knx/KNXProtocol.java` - -Here’s some helpful info: - -- Protocols are one instance per Agent; you’ll see that each Protocol has a corresponding Agent class see here for example. -- Each Asset type has a concrete class and Agents are a sub type of Asset so they therefore have their own classes also (this gives us better type safety) -- An Agent's configuration is stored in individual attributes; these attributes are defined in the Agent class. -- agentLink MetaItem now contains all the configuration needed to connect an attribute to a specific Agent diff --git a/docs/developer-guide/creating-a-custom-project.md b/docs/developer-guide/creating-a-custom-project.md index f651e92e..ab3fe318 100644 --- a/docs/developer-guide/creating-a-custom-project.md +++ b/docs/developer-guide/creating-a-custom-project.md @@ -1,5 +1,5 @@ --- -sidebar_position: 8 +sidebar_position: 12 --- # Creating a custom project diff --git a/docs/developer-guide/data-migration.md b/docs/developer-guide/data-migration.md deleted file mode 100644 index fc5c5bf2..00000000 --- a/docs/developer-guide/data-migration.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -sidebar_position: 15 ---- - -# Data migration - -Sometimes it is desirable to bulk edit existing assets (add/remove attributes and/or configuration items) rather than wiping the DB and starting again. - -DB migration scripts can be used to perform these migrations but also raw SQL can be used to make such alterations. - -There are several DB functions included in the system to help with this task: - -## DB Functions -The DB functions and their arguments can be found in the code at: - -https://github.com/openremote/openremote/blob/master/manager/src/main/resources/org/openremote/manager/setup/database - -## Examples - -### Add/Update meta items to specific attribute of asset(s) -```sql -SELECT a.id, ADD_UPDATE_META(a, 'newAttribute1', - jsonb_build_object('meta1', false, 'meta2', 456, 'meta3', 'Some text')) -FROM asset a WHERE...; -``` - -### Remove meta items from specific attribute of asset(s) -```sql -SELECT a.id, REMOVE_META(a, 'newAttribute1', 'meta2', 'meta3') FROM asset a WHERE...; -``` - -### Remove attributes from specific asset(s) -```sql -SELECT a.id, REMOVE_ATTRIBUTES(a, 'oldAttribute1', 'oldAttribute2') FROM asset a WHERE...; -``` - -### Add attribute to specific asset(s) (with meta items) -**Warning**: this will override any existing attribute data! -```sql -SELECT a.id, ADD_ATTRIBUTE(a, 'newAttribute2', 'GEO_JSONPoint', '1'::jsonb, now(), - jsonb_build_object('meta1', true, 'meta2', 123)) -FROM asset a WHERE...; -``` - -### Add attribute to specific asset(s) (without meta items) -**Warning**: this will override any existing attribute data! -```sql -SELECT a.id, ADD_ATTRIBUTE(a, 'newAttribute1', 'GEO_JSONPoint', '1'::jsonb, now(), null) -FROM asset a WHERE...; -``` diff --git a/docs/developer-guide/docker-compose-profiles.md b/docs/developer-guide/docker-compose-profiles.md index 18ef8705..28fa9738 100644 --- a/docs/developer-guide/docker-compose-profiles.md +++ b/docs/developer-guide/docker-compose-profiles.md @@ -1,5 +1,5 @@ --- -sidebar_position: 7 +sidebar_position: 11 --- # Docker Compose profiles diff --git a/docs/developer-guide/gateway-tunnelling-setup.md b/docs/developer-guide/gateway-tunnelling-setup.md index b7d73659..1d616caf 100644 --- a/docs/developer-guide/gateway-tunnelling-setup.md +++ b/docs/developer-guide/gateway-tunnelling-setup.md @@ -17,7 +17,7 @@ This guide describes the steps necessary to setup the gateway tunnelling functio * `ssh-keygen -t ed25519 -b 4096 -f server_key` * `mv server_key deployment/sish/keys` -### Docker envrionment variables +### Docker environment variables * Set Keycloak container environment variables: * `KEYCLOAK_ISSUER_BASE_URI: https://${OR_HOSTNAME}/auth` diff --git a/docs/developer-guide/installing-and-using-docker.md b/docs/developer-guide/installing-and-using-docker.md deleted file mode 100644 index 0a59efd6..00000000 --- a/docs/developer-guide/installing-and-using-docker.md +++ /dev/null @@ -1,230 +0,0 @@ ---- -sidebar_position: 2 ---- - -# Installing and using Docker - -All OpenRemote projects are delivered as Docker images that support `linux/aarch64` and `linux/amd64`, you'll need a Docker host to run containers and service stacks. - -## Local Engine - -For a local engine (developer workstation setup) simply installing [Docker Desktop](https://www.docker.com/products/docker-desktop) is enough (uses WSL2 backend on Windows). - -Ensure the following commands work: - -```shell -docker -v -docker-compose -v -``` - -**If running Docker Toolbox Edition make sure the code you are working on is in your 'HOME' directory**, and please ensure the following command lists the output of your `HOME` directory (host volume mapping seems unreliable in versions <= 19.03.01 which use VirtualBox 5.x): -```shell -docker run --rm -v ~:/data alpine ls -al /data -``` - -***If you see your `HOME` directory files listed then please skip the following steps otherwise please follow and try the above command again*** - -1. Open Docker Quickstart Terminal -2. Type `docker-machine rm default` -3. Download and install VirtualBox >= 6.x -4. Close and reopen the Docker Quickstart Terminal -5. Try the above `docker run` command again - - -Assuming all the above is working and correct then if you are using Docker Toolbox Edition (Virtual Box); please follow the following steps to add required port mappings: - - 1. Open VirtualBox and select the `docker` VM (called `default` unless specified otherwise) then go to Settings -> Network -> Adapter 1 -> Advanced -> Port Forwarding - 2. Add the following rules: - - | Name | Protocol | Host IP | Host Port | Guest IP | Guest Port | - | --- | --- | --- | --- | --- | --- | - |postgresql|TCP||5432||5432| - | keycloak | TCP | | 8081 | | 8081 | - | map | TCP | | 8082 | | 8082 | - | proxy http | TCP | | 80 | | 80 | - | proxy https | TCP | | 443 | | 443 | - - -## ARM SBC (RPi, ODROID, etc.) installation steps - -The following steps are relevant for an `ODROID C2` but should be similar for other SBCs (note that armbian don't support the RPi at the time of writing so use the Raspbian 64bit OS or similar), the important thing is that you have a 64bit OS which is required to use our Docker images because the OpenJDK at the time of writing doesn't have a JIT compiler 32bit JDK for ARM - one is in progress though): - -1. Download Armbian (https://www.armbian.com/odroid-c2/) and flash to SD card using Etcher -1. Power on and SSH into ODROID then follow Armbian prompts to change root password and Ctrl-C to skip/abort creating a new user -1. Install curl `apt-get install curl` -1. Install docker using convenience script: - 1. `curl -fsSL https://get.docker.com -o get-docker.sh` - 2. `sh get-docker.sh` -1. Check install was successful with `docker --version` -1. Install docker-compose (this is starting to be included in docker package directly now as `docker compose` check if that's the case, if not then follow the next steps): - 1. `curl -L "https://github.com/ubiquiti/docker-compose-aarch64/releases/download/1.22.0/docker-compose-Linux-aarch64" -o /usr/local/bin/docker-compose` - 1. `chmod +x /usr/local/bin/docker-compose` -1. Check install was successful with `docker compose -h` or `docker-compose --version` - - - -## Remote Engine - -### Installing a remote engine - -To install a remote engine (hosted deployment), you need SSH public key access to a Linux host, and then Docker Machine can do the rest and install all required packages on the host and configure secure access: - -```shell -docker-machine create --driver generic \ - --generic-ip-address= \ - --generic-ssh-port= \ - -``` - -Follow the instructions [here](https://docs.docker.com/machine/drivers/generic/). - -After you let Docker Machine install the Docker daemon on the remote host, you must fix the generated Docker client credentials configuration files on your local machine. - -Move `~/.docker/machine/certs/*` into `~/.docker/machine/machines//` and fix the paths in `~/.docker/machine/machines//config.json`. - -When a remote engine is first installed the client credentials should be zipped and made available in a private and secure location. The client credentials can be found at `~/.docker/machine/machines//`. - -### Using a remote engine - -You do not need SSH access on the remote host to simply use an already installed Docker engine. - -Manually copy the client credentials of the remote engine from the secure location to your local machine by unzipping the credentials into `~/.docker/machine/machines//` and then you will need to fix the paths in `~/.docker/machine/machines//config.json`. - -***For Windows you will have to use escaped backslashes e.g. `C:\\Users\Me\\.docker\\machine\\`.*** - -Once the remote engine is installed ensure that `docker-machine ls` shows the new engine and that the State is `Running`: - -```shell -docker-machine ls -eval $(docker-machine env or-host123) -docker ps -``` - -This will show running containers on Docker Machine `or-host123`. - -## Using Docker Compose - -Our services are configured as a stack of containers with Docker Compose. - -Service images will be built automatically when they do not exist in the Docker engine image cache or when the `Dockerfile` for the service changes. **Docker Compose does not track changes to the files used in a service so when code changes are made you will need to manually force a build of the service**. - -Here are a few useful Docker Compose commands: - -* `docker-compose -f pull ...` - Force pull requested services from Docker Hub, if no services specified then all services will be pulled (e.g. docker-compose -f profile/manager.yml pull to pull all services) - -* `docker-compose -f build ...` - Build/Rebuild requested services, if no services specified then all services will be built - -* `docker-compose -f up ...` - Creates and starts requested services, if no services specified then all services will be created and started (also auto attaches to console output from each service use `-d` to not attach to the console output) - -* `docker-compose -f up --build ...` - Creates and starts requested services but also forces building the services first (useful if the source code has changed as docker-compose will not be aware of this change and you would otherwise end up deploying 'stale' services) - -* `docker-compose -f down ...` - Stops and removes requested services, if no services specified then all services will be stopped and removed. Sometimes the shutdown doesn't work properly and you have to run `down` again to completely remove all containers and networks. - -When deploying profiles you can provide a project name to prefix the container names (by default Docker Compose will use the configuration profile's folder name); the project name can be specified with the -p argument using the CLI: - -```shell -docker-compose -p -f -f up -d ... -``` - - - -## Building images - -Run the following command to build the images with the proper tags: -```shell -DATE_TAG= docker-compose -f profile/deploy.yml build - -or - -DATE_TAG= docker-compose -p my_custom_project \ - -f profile/production.yml \ - build -``` -When the environment variable `DATE_TAG` is omitted, then the tag `latest` is used for the image. You should consider exporting the variable before executing multiple commands in a shell. - -## Labelling images - -To make it easy to track which version of the code was used to build the images, the `GIT_COMMIT` label should be supplied when executing the Docker Compose build command e.g.: -```shell -docker-compose -f profile/deploy.yml build --build-arg GIT_COMMIT= - -or - -docker-compose -p my_custom_project \ - -f profile/production.yml \ - build --build-arg GIT_COMMIT= -``` -This information can then be viewed using the `docker inspect` command: -```shell -docker inspect -``` -Or to just output the GIT_COMMIT value: -```shell -docker inspect -f '{{index .ContainerConfig.Labels "git-commit"}}' -``` -For this to work the following lines must be included in the `Dockerfile` of each image: -``` -ARG GIT_COMMIT=unknown -LABEL git-commit=$GIT_COMMIT -``` - -## Publishing images -Push images to [Docker Hub](https://hub.docker.com/u/openremote): - -```shell -docker login - -docker push openremote/proxy:latest -docker push openremote/postgresql:latest -docker push openremote/keycloak:latest -docker push openremote/manager:latest -docker push openremote/tileserver-gl:latest -docker push openremote/deployment:latest - -or - -docker push openremote/proxy:$DATE_TAG -docker push openremote/postgresql:$DATE_TAG -docker push openremote/keycloak:$DATE_TAG -docker push openremote/manager:$DATE_TAG -docker push openremote/tileserver-gl:$DATE_TAG -docker push openremote/deployment:$DATE_TAG - -``` - -## Exporting and importing images -The Docker images created from the docker-compose files can be exported and sent to another machine to import them. - -## Buildx (Cross platform build tool) -Buildx allows cross platform image compilation (i.e. build ARM64 image on an AMD64 machine): - -* Create buildx instance (if not already done) - `docker buildx create --name builder` -* Select buildx instance - `docker buildx use builder` - -To build for example the manager image: - -`docker buildx build --load --platform linux/arm64 -t openremote/manager:latest openremote/manager/build/install/manager/` diff --git a/docs/developer-guide/licensing-guidelines-for-contributors.md b/docs/developer-guide/licensing-guidelines-for-contributors.md index dcfaeaa0..7cb5f48f 100644 --- a/docs/developer-guide/licensing-guidelines-for-contributors.md +++ b/docs/developer-guide/licensing-guidelines-for-contributors.md @@ -13,7 +13,7 @@ Tip: use `./gradlew dependencies` on projects that use gradle as build tool. In all source files (except where not appropriate such as property files or test fixtures), include a copyright header: ``` -Copyright 2021, OpenRemote Inc. +Copyright ${YEAR}, OpenRemote Inc. See the CONTRIBUTORS.txt file in the distribution for a full listing of individual contributors. diff --git a/docs/developer-guide/maintaining-an-installation.md b/docs/developer-guide/maintaining-an-installation.md deleted file mode 100644 index b57aaeb1..00000000 --- a/docs/developer-guide/maintaining-an-installation.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -sidebar_position: 9 ---- - -# Maintaining an installation - -## Monitoring - -Tail log output of all containers with `docker-compose -p openremote -f profile/demo.yml logs -f`. - -Use `docker stats` to show CPU, memory, network read/writes, and total disk read/writes for running containers. - -## Diagnosing database problems - -Access the database: - -```shell -docker exec -it openremote_postgresql_1 psql -U postgres -``` - -Get statistics for all tables and indices (note that these are collected statistics, not live data): - -```sql -SELECT - t.tablename, - indexname, - c.reltuples AS approximate_row_count, - pg_size_pretty(pg_relation_size(quote_ident(t.tablename)::text)) AS table_size, - pg_size_pretty(pg_relation_size(quote_ident(indexrelname)::text)) AS index_size, - CASE WHEN indisunique THEN 'Y' - ELSE 'N' - END AS UNIQUE, - idx_scan AS number_of_scans, - idx_tup_read AS tuples_read, - idx_tup_fetch AS tuples_fetched -FROM pg_tables t -LEFT OUTER JOIN pg_class c ON t.tablename=c.relname -LEFT OUTER JOIN - ( SELECT c.relname AS ctablename, ipg.relname AS indexname, x.indnatts AS number_of_columns, idx_scan, idx_tup_read, idx_tup_fetch, indexrelname, indisunique FROM pg_index x - JOIN pg_class c ON c.oid = x.indrelid - JOIN pg_class ipg ON ipg.oid = x.indexrelid - JOIN pg_stat_all_indexes psai ON x.indexrelid = psai.indexrelid ) - AS foo - ON t.tablename = foo.ctablename -WHERE t.schemaname='openremote' -ORDER BY 1,2; -``` - -Find transactions/queries waiting for more than 30 seconds: - -```sql -SELECT pid, client_addr, now() - query_start AS duration, query, state - FROM pg_stat_activity - WHERE now() - query_start > interval '30 seconds'; -``` - -Kill them with: - -```sql -select pg_cancel_backend(pid); -``` - -Pay attention to shared memory setting of postgres container; Docker sets this very low by default and this will cause problems for any reasonable size DB, see here for info: - -* https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB. - -More useful queries and maintenance operations can be found here: - -* https://wiki.postgresql.org/wiki/Index_Maintenance -* https://github.com/ioguix/pgsql-bloat-estimation - -You can log all queries taking longer than 2 seconds: - -```sql -alter system set log_min_duration_statement=2000; -select pg_reload_conf(); -show log_min_duration_statement; -``` - -To disable, set the duration to `-1`. - -Use `explain analyze ` to obtain the query plan and display it in https://explain.depesz.com/. - -## Diagnosing JVM memory problems - -If the JVM was started with `-XX:NativeMemoryTracking=summary`, use this to get an overview (see [here](https://trustmeiamadeveloper.com/2016/03/18/where-is-my-memory-java/) for more details): - -```shell -docker exec -it or-manager-1 /usr/bin/jcmd 1 VM.native_memory summary -``` - -Otherwise, use [`jstat`](https://docs.oracle.com/en/java/javase/21/docs/specs/man/jstat.html) to monitor a running system/JVM. - -Get current memory configuration: - -```shell -docker exec -it or-manager-1 /usr/bin/jstat -gccapacity 1 -``` - -Output contains: - -``` -NGCMN: Minimum new generation capacity (kB). -NGCMX: Maximum new generation capacity (kB). -NGC: Current new generation capacity (kB). -S0C: Current survivor space 0 capacity (kB). -S1C: Current survivor space 1 capacity (kB). -EC: Current eden space capacity (kB). -OGCMN: Minimum old generation capacity (kB). -OGCMX: Maximum old generation capacity (kB). -OGC: Current old generation capacity (kB). -OC: Current old space capacity (kB). -MCMN: Minimum metaspace capacity (kB). -MCMX: Maximum metaspace capacity (kB). -MC: Metaspace capacity (kB). -CCSMN: Compressed class space minimum capacity (kB). -CCSMX: Compressed class space maximum capacity (kB). -CCSC: Compressed class space capacity (kB). -YGC: Number of young generation GC events. -FGC: Number of full GC events. -``` - -The following command will connect to a Manager and print garbage collection statistics (polling 1000 times, waiting for 2.5 seconds): - -```shell -docker exec -it or-manager-1 /usr/bin/jstat -gcutil 1 2500 1000 -``` - -The output contains: - -``` -S0: Survivor space 0 utilization as a percentage of the space's current capacity. -S1: Survivor space 1 utilization as a percentage of the space's current capacity. -E: Eden space utilization as a percentage of the space's current capacity. -O: Old space utilization as a percentage of the space's current capacity. -M: Metaspace utilization as a percentage of the space's current capacity. -CCS: Compressed class space utilization as a percentage. -YGC: Number of young generation GC events. -YGCT: Young generation garbage collection time. -FGC: Number of full GC events. -FGCT: Full garbage collection time. -GCT: Total garbage collection time. -``` - -Force garbage collection with: - -```shell -docker exec -it openremote_manager_1 /usr/bin/jcmd 1 GC.run -``` - -## Install the JDK -By default the manager Docker image only contains a JRE; many java profiling tools are available in the JDK so to install within a running manager container: -```shell -docker exec or-manager-1 /bin/bash -c 'microdnf --setopt=install_weak_deps=0 --setopt=tsflags=nodocs install -y java-21-openjdk-devel && microdnf clean all && rpm -q java-21-openjdk-devel' -``` - - - -## Performing a heap dump (`jmap`) - -The [`jmap`](https://docs.oracle.com/en/java/javase/21/docs/specs/man/jmap.html) tool within the JDK can be used to create a heap dump of a running JVM. - -### Create heap dump -```shell -docker exec or-manager-1 /bin/bash -c 'jmap -dump:live,format=b,file=/dump.hprof 1' -``` - -### Copy to Docker host -```shell -docker cp or-manager-1:/dump.hprof ~ -``` - -### Use scp to copy from Docker host to local machine -```shell -scp {HOST}:~/dump.hprof ~ -``` - -You can then explore the heap dump with an IDE or other tool. - -## Performing a thread dump (`jstack`) -The [`jstack`](https://docs.oracle.com/en/java/javase/21/docs/specs/man/jstack.html) tool within the JDK can be used to create a thread dump of a running JVM. - -### Create thread dump -```shell -docker exec or-manager-1 /bin/bash -c 'jstack -l 1 > /threads.txt' -``` - -### Copy to Docker host -```shell -docker cp or-manager-1:/threads.txt ~ -``` - -### Use scp to copy from Docker host to local machine -```shell -scp {HOST}:~/threads.txt ~ -``` - -You can then explore the thread dump with an IDE or other tool. diff --git a/docs/developer-guide/preparing-the-environment.md b/docs/developer-guide/preparing-the-environment.md index 6a486f83..c88efdc0 100644 --- a/docs/developer-guide/preparing-the-environment.md +++ b/docs/developer-guide/preparing-the-environment.md @@ -1,5 +1,5 @@ --- -sidebar_position: 1 +sidebar_position: 00 --- # Preparing the environment @@ -9,24 +9,15 @@ To build OpenRemote projects, you have to first prepare the environment on your Ensure you have installed and configured the following tools: ## Runtime tooling -To run docker images from Docker Hub you need to install the following tooling: -* Docker see [here](installing-and-using-docker.md#local-engine) - -If you want to manage remote Docker engines then you will also need to install `docker-machine` separately (since docker 2.2.x): - -* [Docker machine](https://docs.docker.com/machine/install-machine/) +OpenRemote is primarily packaged as Docker images and deployed using docker compose, you will need a docker compatible runtime: +* [Docker](https://docs.docker.com/engine/install/) +* [Docker compose](https://docs.docker.com/compose/install/) Ensure the following commands execute successfully: ```shell -docker ps -docker-compose version -``` - -If you installed Docker machine then make sure the following command executes successfully: - -```shell -docker-machine version +docker -v +docker-compose -v ``` ## Development tooling @@ -51,6 +42,5 @@ Ensure that you have the `JAVA_HOME` environment variable set to the path of JDK ## See also -- [Installing and using Docker](installing-and-using-docker.md) - [Next 'Get Started' step: Build the code and run the manager](https://github.com/openremote/openremote/blob/master/README.md) - [Get Started](https://openremote.io/get-started-iot-platform/) diff --git a/docs/developer-guide/setting-up-an-ide.md b/docs/developer-guide/setting-up-an-ide.md index 2b61f1a6..0e996887 100644 --- a/docs/developer-guide/setting-up-an-ide.md +++ b/docs/developer-guide/setting-up-an-ide.md @@ -1,5 +1,5 @@ --- -sidebar_position: 3 +sidebar_position: 10 --- # Setting up an IDE @@ -8,13 +8,7 @@ This guide helps you set up an environment with an IDE when you are done [Prepar This is not necessary if you prefer [Working on the UI](working-on-ui-and-apps.md) only, any file manager and text editor will suffice. -## Download custom-project - -Using Git, clone the [custom-project](https://github.com/openremote/custom-project) repository. - -## Run Docker container - -If you have successfully downloaded your custom-project, you can build the Docker container by running one of the following commands from your custom-project directory: +If you have successfully cloned the OpenRemote repo or a custom project repo, you can run the Docker containers needed for development by running one of the following commands: ### Without SSL and proxy @@ -28,18 +22,6 @@ docker-compose -p openremote -f profile/dev-testing.yml up --build -d docker-compose -p openremote -f profile/dev-proxy.yml up --build -d ``` -:::note - -You will need to add the following environment variables within your IDE for the manager to work behind the proxy with SSL: - -```shell -WEBSERVER_LISTEN_HOST=0.0.0.0 -IDENTITY_NETWORK_WEBSERVER_PORT=443 -IDENTITY_NETWORK_SECURE=true -``` - -::: - ## Importing a project in an IDE ### IntelliJ IDEA @@ -136,7 +118,7 @@ If you are using the custom project repository as starting point the run configu - Main class: `org.openremote.manager.Main` - Any environment variables that customise deployment (usually custom projects have some) -## Accessing the Manager UI +### Accessing the Manager UI The manager UI web application isn't compiled until build time. \ To run the manager app run `npm run serve` from the `/openremote/ui/app/manager` directory.\ @@ -152,13 +134,23 @@ The web server binds to only localhost interface (i.e. `127.0.0.1`). You can ove Go to [Working on the UI](working-on-ui-and-apps.md#working-on-an-app-eg-manager-ui) for more information. -## VisualVM -To inspect the threads, analyzing CPU and memory allocation you should running a VisualVM. +## Building + +Our build tooling is `gradle` and building the code is just a matter of executing the following gradle tasks: + +```shell +./gradlew clean installDist +``` -- Download and install VisualVM from the [VisualVM](https://visualvm.github.io/) website. -- Install the [VisualVM Launcher](https://plugins.jetbrains.com/plugin/7115-visualvm-launcher) for IntelliJ IDEA +## Testing -## Executing tests +Our test suite is mostly integration tests located in the `/test` module and they require the `keycloak` and `postgres` services to be running on `localhost`, you can start these using the `/profile/dev-testing.yml` Docker Compose profile. -Any JUnit test can be directly executed, you can create a Run Configuration for the `test` module and run all tests in the IDE. Ensure that the `profile/dev-testing.yml` background service stack is running, as described above. +Unit tests are located within each related module and can just be directly executed. + +If you want to run the tests, execute: + +```shell +./gradlew test +``` diff --git a/docs/developer-guide/system-administration.md b/docs/developer-guide/system-administration.md new file mode 100644 index 00000000..e7645ff3 --- /dev/null +++ b/docs/developer-guide/system-administration.md @@ -0,0 +1,548 @@ +--- +sidebar_position: 20 +--- + +# System Administration + +## Monitoring + +Use `docker stats` to show CPU, memory, network read/writes, and total disk read/writes for running containers. +Also use prometheus metrics to monitor individual container health (see: [Metrics](../user-guide/metrics.md)). + +## JVM + +### Diagnosing JVM memory problems + +If the JVM was started with `-XX:NativeMemoryTracking=summary`, use this to get an overview (see [here](https://trustmeiamadeveloper.com/2016/03/18/where-is-my-memory-java/) for more details): + +```shell +docker exec -it or-manager-1 /usr/bin/jcmd 1 VM.native_memory summary +``` + +Otherwise, use [`jstat`](https://docs.oracle.com/en/java/javase/21/docs/specs/man/jstat.html) to monitor a running system/JVM. + +Get current memory configuration: + +```shell +docker exec -it or-manager-1 /usr/bin/jstat -gccapacity 1 +``` + +Output contains: + +``` +NGCMN: Minimum new generation capacity (kB). +NGCMX: Maximum new generation capacity (kB). +NGC: Current new generation capacity (kB). +S0C: Current survivor space 0 capacity (kB). +S1C: Current survivor space 1 capacity (kB). +EC: Current eden space capacity (kB). +OGCMN: Minimum old generation capacity (kB). +OGCMX: Maximum old generation capacity (kB). +OGC: Current old generation capacity (kB). +OC: Current old space capacity (kB). +MCMN: Minimum metaspace capacity (kB). +MCMX: Maximum metaspace capacity (kB). +MC: Metaspace capacity (kB). +CCSMN: Compressed class space minimum capacity (kB). +CCSMX: Compressed class space maximum capacity (kB). +CCSC: Compressed class space capacity (kB). +YGC: Number of young generation GC events. +FGC: Number of full GC events. +``` + +The following command will connect to a Manager and print garbage collection statistics (polling 1000 times, waiting for 2.5 seconds): + +```shell +docker exec -it or-manager-1 /usr/bin/jstat -gcutil 1 2500 1000 +``` + +The output contains: + +``` +S0: Survivor space 0 utilization as a percentage of the space's current capacity. +S1: Survivor space 1 utilization as a percentage of the space's current capacity. +E: Eden space utilization as a percentage of the space's current capacity. +O: Old space utilization as a percentage of the space's current capacity. +M: Metaspace utilization as a percentage of the space's current capacity. +CCS: Compressed class space utilization as a percentage. +YGC: Number of young generation GC events. +YGCT: Young generation garbage collection time. +FGC: Number of full GC events. +FGCT: Full garbage collection time. +GCT: Total garbage collection time. +``` + +Force garbage collection with: + +```shell +docker exec -it openremote_manager_1 /usr/bin/jcmd 1 GC.run +``` + +### Install the JDK +By default the manager Docker image only contains a JRE; many java profiling tools are available in the JDK so to install within a running manager container: +```shell +docker exec or-manager-1 /bin/bash -c 'microdnf --setopt=install_weak_deps=0 --setopt=tsflags=nodocs install -y java-21-openjdk-devel && microdnf clean all && rpm -q java-21-openjdk-devel' +``` + +### Performing a heap dump (`jmap`) + +The [`jmap`](https://docs.oracle.com/en/java/javase/21/docs/specs/man/jmap.html) tool within the JDK can be used to create a heap dump of a running JVM. + +#### Create heap dump +```shell +docker exec or-manager-1 /bin/bash -c 'jmap -dump:live,format=b,file=/dump.hprof 1' +``` + +#### Copy to Docker host +```shell +docker cp or-manager-1:/dump.hprof ~ +``` + +#### Use scp to copy from Docker host to local machine +```shell +scp {HOST}:~/dump.hprof ~ +``` + +You can then explore the heap dump with an IDE or other tool. + +### Performing a thread dump (`jstack`) +The [`jstack`](https://docs.oracle.com/en/java/javase/21/docs/specs/man/jstack.html) tool within the JDK can be used to create a thread dump of a running JVM. + +#### Create thread dump +```shell +docker exec or-manager-1 /bin/bash -c 'jstack -l 1 > /threads.txt' +``` + +#### Copy to Docker host +```shell +docker cp or-manager-1:/threads.txt ~ +``` + +#### Use scp to copy from Docker host to local machine +```shell +scp {HOST}:~/threads.txt ~ +``` + +You can then explore the thread dump with an IDE or other tool. + +--- + + +## Database +This document summarises the database performance, storage footprint, and system efficiency of the OpenRemote instance before and after enabling **TimescaleDB Hypercore** compression features. It serves as a reference for System Administrators to understand the expected baselines, trade-offs, and critical maintenance thresholds. + +--- + +Use the following docker command to access the database on the docker host: +```shell +docker exec -it openremote_postgresql_1 psql -U postgres +``` + +### Storage & compression +Enabling Hypercore compression drastically reduced the database storage footprint by converting uncompressed historical row-data into highly compressed columnar structures. + +| Metric | Before Hypercore | After Hypercore | Improvement | +| :--- | :--- | :--- | :--- | +| **Total DB Disk Size** | 225.5 GB | 16.1 GB | **~93% Reduction** | +| **Datapoint Table Size** | 219.6 GB | 10.0 GB | **~95% Reduction** | +| **Index Size** | 130.8 GB | 3.4 GB | **~97% Reduction** | +| **Compression Ratio** | N/A | ~61.8x | - | + +**SysAdmin Note:** The massive reduction in index size occurs because TimescaleDB discards traditional B-Tree indexes on compressed chunks, relying instead on lightweight columnar metadata. + +--- + +### Memory & cache efficiency +Prior to compression, the database footprint vastly exceeded the allocated Docker memory, resulting in severe disk thrashing and Out-Of-Memory (OOM) risks. Post-compression, the active data easily fits within PostgreSQL's `shared_buffers`. + +| Metric | Before Hypercore | After Hypercore | +| :--- | :--- | :--- | +| **Overall Cache Hit Ratio** | 5.6% | **95.3%** | +| **Uncompressed Cache Hit Ratio** | 0.5% | **91.9%** | + +**SysAdmin Note:** A >95% cache hit ratio means PostgreSQL is serving almost all queries instantly from RAM. To maintain this efficiency, ensure that the Docker `shm_size` (Shared Memory) is appropriately configured to be slightly larger (e.g., +10-20%) than the PostgreSQL `shared_buffers` setting to prevent startup crashes. + +--- + +### Query performance benchmarks +Query execution times were fundamentally transformed. Heavy historical aggregations that previously choked the database are now highly optimised, with a minor architectural trade-off for ultra-fast, single-point queries. + +| Query Type | Example | Before (ms) | After (ms) | Change | +| :--- | :--- | :--- | :--- | :--- | +| **Overall Average** | All Queries | ~4,359 ms | ~436 ms | **~90% Faster** | +| **Heavy Aggregation** | `4hOF-kwMax_Daily_MinMaxAvg_30d` | 44,442.0 ms | 583.9 ms | **98.6% Faster** | +| **Heavy Aggregation** | `2pPI-power_Daily_MinMaxAvg_30d` | 9,951.1 ms | 284.2 ms | **97.1% Faster** | +| **Heavy Aggregation** | `4jp8-power_Daily_MinMaxAvg_30d` | 7,815.3 ms | 158.4 ms | **97.9% Faster** | +| **Light Micro-Query** | `4hOF-kwMin_Hourly_Avg_7d` | 4.4 ms | 157.1 ms | *(Expected Trade-off)* | + +**SysAdmin Note:** +* **The Win:** Complex, multi-day historical aggregations improved by 97-98%, eliminating 40+ second query timeouts. +* **The Trade-off:** Micro-queries taking \<10ms previously will now take ~50-150ms. This is expected behavior; querying compressed chunks requires on-the-fly decompression CPU overhead to unpack the columnar data back into rows. + +--- + +### Critical maintenance guidelines + +Here's a summary of critical PostgreSQL memory settings: + +### PostgreSQL & Docker Memory Summary + +| Setting / Limit | Scope | Primary Purpose | How it relates to Hypercore & Docker | +| :--- | :--- | :--- | :--- | +| **Total Container RAM** | Docker Limit | The absolute physical ceiling. | If the combined database footprint exceeds this, the Linux OOM killer crashes the container. | +| **`shm_size`** | Docker Limit | Sets the maximum allowed size for the shared memory file system (`/dev/shm`). | Must be set ~10-20% higher than `shared_buffers` so the database has permission to allocate its global cache. | +| **`shared_buffers`** | Global Postgres Cache | The main memory pool where PostgreSQL caches active data. | Holds the highly compressed Hypercore columnar blocks. Needs to fit entirely inside Docker's `shm_size`. | +| **`maintenance_work_mem`** | Local Postgres Memory | A dedicated workspace for heavy background maintenance tasks. | **Critical for compression.** Used to sort raw data before compressing it into chunks. If set too high, it causes an OOM crash. | +| **`work_mem`** | Local Postgres Memory | Workspace for everyday query operations (sorting, joining, grouping). | **Multiplies rapidly.** It is allocated *per operation*. Used to temporarily decompress columnar chunks back into rows for user queries. | +| **`effective_cache_size`** | Query Planner Hint | Estimates how much RAM the Linux OS has left over for caching files. | **Does not allocate RAM.** Helps the query planner decide the fastest way to scan compressed Hypercore blocks. | + +**The Golden Rule:** `shared_buffers` (Fixed) + `maintenance_work_mem` (Spiky) + `work_mem` (Highly Spiky) must **never** exceed the **Total Container RAM**. + +To keep the instance healthy and prevent Docker crashes, you can follow these guidelines: + +1. Calculate largest uncompressed chunk size N (see `pg_datapoint_largest_uncompressed_chunk` metric or query or use approximate calculation ~1 million datapoints / week = 300MB) +2. Calculate minimum RAM size P = (4 x N) = 1.2GB (Round up to 2GB here) +3. Set `shared_buffers` at 25% RAM (0.25 x P) = 512MB +4. Set `shm_size` (`shared_buffers` + ~20%) = ~614MB (Round up to a clean number 1GB) +5. Set `maintenance_work_mem` (0.125 x P) = 250MB (Round up to a clean number 256MB) +6. Set `work_mem` (1% x P) = 16MB (or larger if there are few concurrent connections) +7. Set `effective_cache_size` (P - `shared_buffers`) = 1.5GB +8. Actively monitor `pg_hot_update_percent` metric to attain a >90% HOT update percentage; tweak the asset table `fillfactor` to achieve this (default is set to: 90% as asset table is not significantly large) + +Apply this config in docker compose yaml as follows: +```yaml +services: + postgresql: + image: openremote/postgresql:latest + shm_size: 1gb + deploy: + resources: + limits: + memory: 2G + command: + - postgres + - -c + - shared_buffers=512MB + - -c + - effective_cache_size=1536MB + - -c + - maintenance_work_mem=256MB + - -c + - work_mem=16MB +``` + +#### Some simple guidelines +* Docker's `shm_size` must be sized appropriately. A simple rule of thumb is to take PostgreSQL `shared_buffers` and add a margin (10% or a flat 256MB); the `shared_buffers` must fit inside this size (**NOTE: Docker sets this very low by default and this will cause problems for any reasonable size DB**) +* **Do not** run `timescaledb-tune` automatically on container startup. It may miscalculate limits if Docker's cgroup namespaces are restricted. +* **Best Practice:** Run `timescaledb-tune` with explicit CPU and memory values to avoid known issues with the script not being able to figure out docker aware values itself `timescaledb-tune --memory="2GB" --cpus="2" --dry-run`, the `dry-run` setting ensures no changes are made to the configuration file and instead hardcode the tuned `shared_buffers`, `effective_cache_size`, `maintenance_work_mem` and `work_mem` directly into the `docker-compose.yml` command args. +* **The 25% Rule:** The largest uncompressed chunk should never exceed 25% of the total Docker container RAM limit. +* For a 2GB container limit, the `pg_datapoint_largest_uncompressed_chunk` metric must stay below **512 MB**. +* If chunks grow too large, the background compression worker will exhaust RAM during the sorting phase and trigger the Linux OOM killer. If this occurs, reduce the hypertable's `chunk_time_interval` via `set_chunk_time_interval()`. +* The asset table is constantly being updated (whenever an attribute event is written to the attributes JSONB column) this can cause excessive autovacuum executions and can be a significant bottleneck for large asset tables due to the way row updates are handled by PostgreSQL; HOT updates is an optimisation PostgreSQL uses to minimise this overhead and the asset table `fillfactor` can be tweaked to optimise this (default: 90%) + +### Backup/Restore +* Create backup: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak` +* Optional: Exclude datapoint records from the backup using the following command: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak --exclude-table-data='_timescaledb_internal._hyper_*'` +* Copy to the Docker host: `docker cp or-postgresql-1:/tmp/db.bak ~/` +* Remove the backup from within the container: `docker exec or-postgresql-1 rm /tmp/db.bak` +* SCP the backup off the source server onto the destination server: e.g. `scp :~/db.bak .` +* On the destination server stop the manager and Keycloak containers and any project specific containers that are using the DB: `docker stop or-manager-1 or-keycloak-1` +* Copy backup into the postgresql container: `docker cp db.bak or-postgresql-1:/tmp/` +* Drop existing DB: `docker exec or-postgresql-1 dropdb openremote` +* Create new DB: `docker exec or-postgresql-1 createdb openremote` +* Add POSTGIS extension: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "CREATE EXTENSION IF NOT EXISTS postgis;"` +* Add Timescale DB extension: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "CREATE EXTENSION IF NOT EXISTS timescaledb;"` +* Run timescale DB pre restore command: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "SELECT timescaledb_pre_restore();"` +* Restore the backup: `docker exec or-postgresql-1 pg_restore -Fc --verbose -U postgres -d openremote /tmp/db.bak` +* Run timescale DB restore command: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "SELECT timescaledb_post_restore();"` +* Start the stopped containers: `docker start or-keycloak-1 or-manager-1` + +For more details related to TimescaleDB backup/restore see [here](https://docs.timescale.com/self-hosted/latest/backup-and-restore/pg-dump-and-restore/). + +### Recover failed update +The `openremote/postgresql` container should handle automatic updates to the latest TimescaleDB version in the image and also auto migration +to the major version of PostgreSQL in the image (note: you must incrementally go through each major version of the `openremote/postgresql` images i.e. you cannot go from 14.x to 17.x but must go `14.x` -> `15.x` -> `17.x` \[there is no `16.x` image so this can be skipped\]). + +The PostgreSQL auto migration process will move `PGDATA` to a sub directory called `old` and create a `new` directory for the initialisation of the new major version DB, if the upgrade fails then you will need to manually handle the issue and move files back to the `PGDATA` folder like so: +```bash +docker exec --rm -it -v or_postgresql-data:/var/lib/postgresql/data --entrypoint=bash openremote/postgresql:latest-slim +mv -v data/old/* $PGDATA +rm -r data/new data/old +``` + +### Useful resources +- [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB) +- [Index maintenance](https://wiki.postgresql.org/wiki/Index_Maintenance) +- [Bloat estimation](https://github.com/ioguix/pgsql-bloat-estimation) +- [Query Exporter Documentation](https://github.com/albertodonato/query-exporter) +- [Configuration Format](https://github.com/albertodonato/query-exporter/blob/main/docs/configuration.rst) +- [PostgreSQL Statistics Views](https://www.postgresql.org/docs/current/monitoring-stats.html) +- [PostgreSQL Bloat Detection](https://wiki.postgresql.org/wiki/Show_database_bloat) + +### Useful queries +Refer to the [Query Exporter configuration file](https://github.com/openremote/openremote/blob/master/deployment/query-exporter/config.yaml) for useful DB monitoring queries. + +#### Adjust asset table fillfactor +```sql +ALTER TABLE asset SET (fillfactor = 80); +``` + +#### Clear syslogs older than 1 day +`docker exec or-postgresql-1 psql -U postgres -d openremote -c "delete from syslog_event where timestamp < now() - INTERVAL '1day';"` + +#### Export all asset data points +```sql +copy asset_datapoint to '/var/lib/postgresql/datapoints.csv' delimiter ',' CSV; +``` + +or with asset names instead of IDs and a header row in the export: +```sql +copy (select ad.timestamp, a.name, ad.attribute_name, ad.value from asset_datapoint ad, asset a where ad.entity_id = a.id) to '/var/lib/postgresql/datapoints.csv' delimiter ',' CSV HEADER; +``` + +#### Export a subset of asset data points (asset and direct child assets) +```sql +copy (select ad.timestamp, a.name, ad.attribute_name, value from asset_datapoint ad, asset a where ad.entity_id = a.id and (ad.entity_id = 'ID1' or a.parent_id = 'ID1')) to '/var/lib/postgresql/datapoints.csv' delimiter ',' CSV HEADER; +``` + +#### Delete all asset datapoints older than N days +```sql +delete from asset_datapoint where timestamp < now() - INTERVAL '7 days'; +``` + +#### Accessing the exported CSV file +```shell +docker cp _postgresql_1:/deployment/datapoints.csv ./ +``` + +#### Importing asset data points +To import data points stored in multiple exports (with potential duplicates) then first create a temp table: +```sql +CREATE TEMP TABLE tmp_table AS SELECT * FROM asset_datapoint WITH NO DATA; +``` + +Import the CSV files into this temp table: +```shell +COPY tmp_table FROM '/var/lib/postgresql/datapoints.csv' DELIMITER ',' CSV; +``` + +Remove values for assets that don't exist in the running system: +```sql +DELETE FROM tmp_table D WHERE NOT (D.entity_id IN (SELECT DISTINCT ID from ASSET)); +``` + +Insert unique values into `ASSET_DATAPOINT` table: +```sql +INSERT INTO asset_datapoint SELECT * FROM tmp_table ON CONFLICT DO NOTHING; +``` + +Drop temp table: +```sql +DROP TABLE tmp_table; +``` + +#### Remove consoles that haven't registered for more than N days + +:::note + +**Following will require manager to be restarted** + +::: + +```sql +DELETE FROM asset a +WHERE a.asset_type = 'urn:openremote:asset:console' AND + to_timestamp((a.attributes#>>'{consoleName, valueTimestamp}')::bigint /1000) < (current_timestamp - interval '30 days'); +``` + +#### Count notifications with specific title sent to consoles (Android/iOS) in past N days + +```sql +SELECT count(*) +FROM notification n +JOIN asset a ON a.id = n.target_id +WHERE n.message ->> 'title' = 'NOTIFICATION TITLE' AND + n.sent_on > (current_timestamp - interval '7 days') AND + a.attributes #>> '{consolePlatform, value}' LIKE 'Android%' AND + n.acknowledged_on IS NOT null; +``` + +#### Count notifications with specific title sent to consoles (Android/iOS) and acknowledged by: + +##### Closing/Dismissing +```sql +SELECT count(*) +FROM notification n +JOIN asset a ON a.id = n.target_id +WHERE n.message ->> 'title' = 'Kijk mee naar de proefbestrating op de Demer' AND + n.acknowledged_on IS NOT null AND + n.acknowledgement = 'CLOSED'; +``` + +##### Clicking the notification +```sql +SELECT count(*) +FROM notification n +JOIN asset a ON a.id = n.target_id +WHERE n.message ->> 'title' = 'NOTIFICATION TITLE' AND + n.acknowledged_on IS NOT null AND + n.acknowledgement IS null; +``` + +##### Clicking a specific action button +```sql +select count(*) +FROM notification n +JOIN asset a ON a.id = n.target_id +WHERE n.message ->> 'title' = 'Kijk mee naar de proefbestrating op de Demer' AND + n.acknowledged_on IS NOT null AND + n.acknowledgement LIKE '%BUTTON TITLE%'; +``` + + + +--- + +### Data migration + +Sometimes it is desirable to bulk edit existing assets (add/remove attributes and/or configuration items) rather than wiping the DB and starting again. + +DB migration scripts can be used to perform these migrations but also raw SQL can be used to make such alterations. + +There are several DB functions included in the system to help with this task: + +#### DB Functions +The DB functions and their arguments can be found in the code at: + +https://github.com/openremote/openremote/blob/master/manager/src/main/resources/org/openremote/manager/setup/database + +#### Examples + +##### Add/Update meta items to specific attribute of asset(s) +```sql +SELECT a.id, ADD_UPDATE_META(a, 'newAttribute1', + jsonb_build_object('meta1', false, 'meta2', 456, 'meta3', 'Some text')) +FROM asset a WHERE...; +``` + +##### Remove meta items from specific attribute of asset(s) +```sql +SELECT a.id, REMOVE_META(a, 'newAttribute1', 'meta2', 'meta3') FROM asset a WHERE...; +``` + +##### Remove attributes from specific asset(s) +```sql +SELECT a.id, REMOVE_ATTRIBUTES(a, 'oldAttribute1', 'oldAttribute2') FROM asset a WHERE...; +``` + +##### Add attribute to specific asset(s) (with meta items) +**Warning**: this will override any existing attribute data! +```sql +SELECT a.id, ADD_ATTRIBUTE(a, 'newAttribute2', 'GEO_JSONPoint', '1'::jsonb, now(), + jsonb_build_object('meta1', true, 'meta2', 123)) +FROM asset a WHERE...; +``` + +##### Add attribute to specific asset(s) (without meta items) +**Warning**: this will override any existing attribute data! +```sql +SELECT a.id, ADD_ATTRIBUTE(a, 'newAttribute1', 'GEO_JSONPoint', '1'::jsonb, now(), null) +FROM asset a WHERE...; +``` + +--- + +## Proxy +### SSH tunnel for proxy stats +HAProxy stats web page is only accessible on localhost in our default config, this can be tunnelled to your local machine to allow access at http://localhost:8404/stats: +```shell +ssh -L 8404:localhost:8404 +``` + +--- + +## OpenRemote Manager +### Insert a different manager_config.json file +```shell +docker exec _manager_1 mkdir -p /deployment/manager/app +docker cp ./manager_config.json _manager_1:/deployment/manager/app/manager_config.json + +# Make sure to restart the containers for applying the changes +``` + +### Insert an .mbtiles file for applying a different map +```shell +docker exec _manager_1 mkdir -p /deployment/map +docker cp ./.mbtiles _manager_1:/deployment/map/mapdata.mbtiles + +# Make sure to restart the containers for applying the changes +``` + +--- + +## Docker +### Cleaning up Docker images, containers, and volumes +Working with Docker might leave exited containers and untagged images. cleanup using (use with caution): +```shell +docker volume prune -a +docker image prune -a +``` + +### Restart exited containers (without using `docker-compose up`) +If the containers are exited then they can be restarted using the `docker` command, the startup order is important: + +* docker start `_postgresql_1` +* docker start `_keycloak_1` +* docker start `_map_1` (only if there is a map container) +* docker start `_manager_1` +* docker start `_proxy_1` + + +### Using Docker Compose + +Our services are configured as a stack of containers with Docker Compose. + +Service images will be built automatically when they do not exist in the Docker engine image cache or when the `Dockerfile` for the service changes. **Docker Compose does not track changes to the files used in a service so when code changes are made you will need to manually force a build of the service**. + +Here are a few useful Docker Compose commands: + +* `docker-compose -f pull ...` - Force pull requested services from Docker Hub, if no services specified then all services will be pulled (e.g. docker-compose -f profile/manager.yml pull to pull all services) +* `docker-compose -f build ...` - Build/Rebuild requested services, if no services specified then all services will be built +* `docker-compose -f up ...` - Creates and starts requested services, if no services specified then all services will be created and started (also auto attaches to console output from each service use `-d` to not attach to the console output) +* `docker-compose -f up --build ...` - Creates and starts requested services but also forces building the services first (useful if the source code has changed as docker-compose will not be aware of this change and you would otherwise end up deploying 'stale' services) +* `docker-compose -f down ...` - Stops and removes requested services, if no services specified then all services will be stopped and removed. Sometimes the shutdown doesn't work properly and you have to run `down` again to completely remove all containers and networks. + +* When deploying profiles you can provide a project name to prefix the container names (by default Docker Compose will use the configuration profile's folder name); the project name can be specified with the -p argument using the CLI: + +```shell +docker-compose -p -f -f up -d ... +``` + + +--- + +## Shell +### Enabling bash auto-completion + +You might want to install bash auto-completion for Docker commands. On OS X, install: + +```shell +brew install bash-completion +``` + +Then add this to your `$HOME/.profile`: + +```bash +if [ -f $(brew --prefix)/etc/bash_completion ]; then +. $(brew --prefix)/etc/bash_completion +fi +``` + +And link the completion-scripts from your local Docker install: + +```shell +find /Applications/Docker.app \ +-type f -name "*.bash-completion" \ +-exec ln -s "{}" "$(brew --prefix)/etc/bash_completion.d/" \; +``` +Start a new shell or source your profile to enable auto-completion. \ No newline at end of file diff --git a/docs/developer-guide/useful-commands-and-queries.md b/docs/developer-guide/useful-commands-and-queries.md deleted file mode 100644 index e6c7ded2..00000000 --- a/docs/developer-guide/useful-commands-and-queries.md +++ /dev/null @@ -1,287 +0,0 @@ ---- -sidebar_position: 14 ---- - -# Useful commands and queries - -## Docker -Replace `` with value used when creating the container with `docker-compose up` (`docker ps` will show the actual names). -### Restart manager Docker container - -```shell -docker restart _manager_1 -``` - -### Start interactive `psql` shell in postgresql container -```shell -docker exec -it _postgresql_1 psql -U postgres -``` - -### Insert a different manager_config.json file -```shell -docker exec _manager_1 mkdir -p /deployment/manager/app -docker cp ./manager_config.json _manager_1:/deployment/manager/app/manager_config.json - -# Make sure to restart the containers for applying the changes -``` - -### Insert an .mbtiles file for applying a different map -```shell -docker exec _manager_1 mkdir -p /deployment/map -docker cp ./.mbtiles _manager_1:/deployment/map/mapdata.mbtiles - -# Make sure to restart the containers for applying the changes -``` - -### Copy exported data point file to local machine (exported using query below) -```shell -docker cp _postgresql_1:/deployment/datapoints.csv ./ -``` - -### Backup/Restore OpenRemote DB - -* Create backup: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak` -* Optional: Exclude datapoint records from the backup using the following command: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak --exclude-table-data='_timescaledb_internal._hyper_*'` -* Copy to the Docker host: `docker cp or-postgresql-1:/tmp/db.bak ~/` -* Remove the backup from within the container: `docker exec or-postgresql-1 rm /tmp/db.bak` -* SCP the backup off the source server onto the destination server: e.g. `scp :~/db.bak .` -* On the destination server stop the manager and Keycloak containers and any project specific containers that are using the DB: `docker stop or-manager-1 or-keycloak-1` -* Copy backup into the postgresql container: `docker cp db.bak or-postgresql-1:/tmp/` -* Drop existing DB: `docker exec or-postgresql-1 dropdb openremote` -* Create new DB: `docker exec or-postgresql-1 createdb openremote` -* Add POSTGIS extension: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "CREATE EXTENSION IF NOT EXISTS postgis;"` -* Add Timescale DB extension: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "CREATE EXTENSION IF NOT EXISTS timescaledb;"` -* Run timescale DB pre restore command: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "SELECT timescaledb_pre_restore();"` -* Restore the backup: `docker exec or-postgresql-1 pg_restore -Fc --verbose -U postgres -d openremote /tmp/db.bak` -* Run timescale DB restore command: `docker exec or-postgresql-1 psql -U postgres -d openremote -c "SELECT timescaledb_post_restore();"` -* Start the stopped containers: `docker start or-keycloak-1 or-manager-1` - -For more details related to TimescaleDB backup/restore see [here](https://docs.timescale.com/self-hosted/latest/backup-and-restore/pg-dump-and-restore/). - -### Clear Syslogs older than 1 day -`docker exec or-postgresql-1 psql -U postgres -d openremote -c "delete from syslog_event where timestamp < now() - INTERVAL '1day';"` - -### Restart exited containers (without using `docker-compose up`) -If the containers are exited then they can be restarted using the `docker` command, the startup order is important: - -* docker start `_postgresql_1` -* docker start `_keycloak_1` -* docker start `_map_1` (only if there is a map container) -* docker start `_manager_1` -* docker start `_proxy_1` - -:::note - -**On Docker v18.02 there is a bug which means you might see the message `Error response from daemon: container "": already exists` to resolve this simply enter the following command (you can ignore any error messages) and try starting again** - -::: - -`docker-containerd-ctr --address /run/docker/containerd/docker-containerd.sock --namespace moby c rm $(docker ps -aq --no-trunc)` - -### Running demo deployment - -```shell -eval $(docker-machine env or-host1) - -OR_ADMIN_PASSWORD=******** OR_SETUP_RUN_ON_RESTART=true OR_HOSTNAME=demo.openremote.io \ -OR_EMAIL_ADMIN=support@openremote.io docker-compose -p demo up --build -d -``` - -To find out the password: - -`docker exec demo_manager_1 env | awk -F= '/ADMIN_PASSWORD/ {print $2}'` - - -### Enabling bash auto-completion - -You might want to install bash auto-completion for Docker commands. On OS X, install: - -```shell -brew install bash-completion -``` - -Then add this to your `$HOME/.profile`: - -```bash -if [ -f $(brew --prefix)/etc/bash_completion ]; then -. $(brew --prefix)/etc/bash_completion -fi -``` - -And link the completion-scripts from your local Docker install: - -```shell -find /Applications/Docker.app \ --type f -name "*.bash-completion" \ --exec ln -s "{}" "$(brew --prefix)/etc/bash_completion.d/" \; -``` -Start a new shell or source your profile to enable auto-completion. - -### Cleaning up images, containers, and volumes - -Working with Docker might leave exited containers and untagged images. If you build a new image with the same tag as an existing image, the old image will not be deleted but simply untagged. If you stop a container, it will not be automatically removed. The following bash function can be used to clean up untagged images and stopped containers, add it to your `$HOME/.profile`: - -```bash -function dcleanup(){ - docker rm -v $(docker ps --filter status=exited -q 2>/dev/null) 2>/dev/null - docker rmi $(docker images --filter dangling=true -q 2>/dev/null) 2>/dev/null -} -``` - -To remove data volumes no longer referenced by a container (deleting ALL persistent data!), use: - -```shell -docker volume prune -``` - -### Revert failed PostgreSQL container auto upgrade -```shell -docker exec --rm -it -v or_postgresql-data:/var/lib/postgresql/data --entrypoint=bash openremote/postgresql -mv -v data/old/* $PGDATA -rm -r data/new data/old -``` - -## Bash -### SSH tunnel for proxy stats -HAProxy stats web page is only accessible on localhost in our default config, this can be tunnelled to your local machine to allow access at http://localhost:8404/stats: -```shell -ssh -L 8404:localhost:8404 -``` - -## Queries - -### Get DB table size info -```sql -SELECT - schema_name, - relname, - pg_size_pretty(table_size) AS size, - table_size - -FROM ( - SELECT - pg_catalog.pg_namespace.nspname AS schema_name, - relname, - pg_relation_size(pg_catalog.pg_class.oid) AS table_size - - FROM pg_catalog.pg_class - JOIN pg_catalog.pg_namespace ON relnamespace = pg_catalog.pg_namespace.oid - ) t -WHERE schema_name NOT LIKE 'pg_%' -ORDER BY table_size DESC; -``` - -### Data points -#### Export all asset data points -```sql -copy asset_datapoint to '/var/lib/postgresql/datapoints.csv' delimiter ',' CSV; -``` - -or with asset names instead of IDs and a header row in the export: -```sql -copy (select ad.timestamp, a.name, ad.attribute_name, ad.value from asset_datapoint ad, asset a where ad.entity_id = a.id) to '/var/lib/postgresql/datapoints.csv' delimiter ',' CSV HEADER; -``` - -#### Export a subset of asset data points (asset and direct child assets) -```sql -copy (select ad.timestamp, a.name, ad.attribute_name, value from asset_datapoint ad, asset a where ad.entity_id = a.id and (ad.entity_id = 'ID1' or a.parent_id = 'ID1')) to '/var/lib/postgresql/datapoints.csv' delimiter ',' CSV HEADER; -``` - -#### Delete all asset datapoints older than N days -```sql -delete from asset_datapoint where timestamp < now() - INTERVAL '7 days'; -``` - -#### Importing asset data points -To import data points stored in multiple exports (with potential duplicates) then first create a temp table: -```sql -CREATE TEMP TABLE tmp_table AS SELECT * FROM asset_datapoint WITH NO DATA; -``` - -Import the CSV files into this temp table: -```shell -COPY tmp_table FROM '/var/lib/postgresql/datapoints.csv' DELIMITER ',' CSV; -``` - -Remove values for assets that don't exist in the running system: -```sql -DELETE FROM tmp_table D WHERE NOT (D.entity_id IN (SELECT DISTINCT ID from ASSET)); -``` - -Insert unique values into `ASSET_DATAPOINT` table: -```sql -INSERT INTO asset_datapoint SELECT * FROM tmp_table ON CONFLICT DO NOTHING; -``` - -Drop temp table: -```sql -DROP TABLE tmp_table; -``` - -### Selfsigned SSL -To add the OpenRemote selfsigned certificate to the default java keystore `cacerts`: - -1. Convert to `p12`: `openssl pkcs12 -export -in proxy/selfsigned/localhost.pem -out or_selfsigned.p12` (use default password `changeit` -2. Import into default java keystore: `openssl pkcs12 -export -in openremote/proxy/selfsigned/localhost.pem -out or_selfsigned.p12` (Windows will need to execute this in Command Prompt with Admin permissions) - -### Consoles - -#### Remove consoles that haven't registered for more than N days - -:::note - -**Following will require manager to be restarted** - -::: - -```sql -DELETE FROM asset a -WHERE a.asset_type = 'urn:openremote:asset:console' AND - to_timestamp((a.attributes#>>'{consoleName, valueTimestamp}')::bigint /1000) < (current_timestamp - interval '30 days'); -``` - -### Notifications - -#### Count notifications with specific title sent to consoles (Android/iOS) in past N days - -```sql -SELECT count(*) -FROM notification n -JOIN asset a ON a.id = n.target_id -WHERE n.message ->> 'title' = 'NOTIFICATION TITLE' AND - n.sent_on > (current_timestamp - interval '7 days') AND - a.attributes #>> '{consolePlatform, value}' LIKE 'Android%' AND - n.acknowledged_on IS NOT null; -``` - -#### Count notifications with specific title sent to consoles (Android/iOS) and acknowledged by: - -##### Closing/Dismissing -```sql -SELECT count(*) -FROM notification n -JOIN asset a ON a.id = n.target_id -WHERE n.message ->> 'title' = 'Kijk mee naar de proefbestrating op de Demer' AND - n.acknowledged_on IS NOT null AND - n.acknowledgement = 'CLOSED'; -``` - -##### Clicking the notification -```sql -SELECT count(*) -FROM notification n -JOIN asset a ON a.id = n.target_id -WHERE n.message ->> 'title' = 'NOTIFICATION TITLE' AND - n.acknowledged_on IS NOT null AND - n.acknowledgement IS null; -``` - -##### Clicking a specific action button -```sql -select count(*) -FROM notification n -JOIN asset a ON a.id = n.target_id -WHERE n.message ->> 'title' = 'Kijk mee naar de proefbestrating op de Demer' AND - n.acknowledged_on IS NOT null AND - n.acknowledgement LIKE '%BUTTON TITLE%'; -``` diff --git a/docs/developer-guide/working-on-maps.md b/docs/developer-guide/working-on-maps.md index bf909934..3cae5391 100644 --- a/docs/developer-guide/working-on-maps.md +++ b/docs/developer-guide/working-on-maps.md @@ -1,5 +1,5 @@ --- -sidebar_position: 6 +sidebar_position: 40 --- # Working on maps diff --git a/docs/developer-guide/working-on-the-mobile-consoles.md b/docs/developer-guide/working-on-the-mobile-consoles.md index 25fbb1d4..08a77441 100644 --- a/docs/developer-guide/working-on-the-mobile-consoles.md +++ b/docs/developer-guide/working-on-the-mobile-consoles.md @@ -1,5 +1,5 @@ --- -sidebar_position: 11 +sidebar_position: 50 --- # Working on the mobile consoles diff --git a/docs/developer-guide/working-on-ui-and-apps.md b/docs/developer-guide/working-on-ui-and-apps.md index 6467e978..2342c29b 100644 --- a/docs/developer-guide/working-on-ui-and-apps.md +++ b/docs/developer-guide/working-on-ui-and-apps.md @@ -1,5 +1,5 @@ --- -sidebar_position: 5 +sidebar_position: 30 --- # Working on UI and apps diff --git a/docs/quick-start.md b/docs/quick-start.md index 1edaca86..e13e8623 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -58,7 +58,7 @@ Try creating assets, agents, rules, users, realms, etc. using the Manager UI, so ## Where's the data stored? Persistent data is stored in a PostgreSQL DB which is stored in the `openremote_postgresql-data` Docker volume which is durably stored independently of the running containers (see all with `docker volume ls`). Note that historical attribute data is purged daily based on value of `OR_DATA_POINTS_MAX_AGE_DAYS`; this value can also be overridden for individual attributes by using the `dataPointsMaxAgeDays` configuration item. -See the [Developer Guide](developer-guide/useful-commands-and-queries.md#backuprestore-openremote-db) for details on making backups of the database. +See the [Developer Guide](developer-guide/system-administration#backuprestore) for details on making backups of the database. ## Contributing to OpenRemote diff --git a/docs/user-guide/deploying/custom-deployment.md b/docs/user-guide/deploying/custom-deployment.md index 486b2300..a49a95de 100644 --- a/docs/user-guide/deploying/custom-deployment.md +++ b/docs/user-guide/deploying/custom-deployment.md @@ -45,7 +45,7 @@ Most of the changes made in the manager_config.json will not be visible to the d Create your own asset type that fits your product. In the asset type you define its name, icon, and colour, and set its attributes with configuration items (called meta items in the code). If you need some inspiration, you can look at OpenRemote's [default asset types](https://github.com/openremote/openremote/tree/master/model/src/main/java/org/openremote/model/asset/impl). ### Agents & Protocols (/agent) -Protocols are a main extension point of OpenRemote, they translate the messages from and to external systems into reads and writes of the assets and attribute values used by OpenRemote. When creating an [agent asset](../../developer-guide/connecting-protocol-adaptors-with-agents.md), you can create protocol configurations, which are a special type of attribute. Each agent attribute that is a protocol configuration then automatically gets its own instance of the protocol you have selected. +Protocols are a main extension point of OpenRemote, they translate the messages from and to external systems into reads and writes of the assets and attribute values used by OpenRemote. When creating an [agent asset](../../developer-guide/agent-protocol-spi.md), you can create protocol configurations, which are a special type of attribute. Each agent attribute that is a protocol configuration then automatically gets its own instance of the protocol you have selected. ### Setup code (/setup) Define which assets and rules should be present when you deploy your project. You can set attribute values and their configuration items, add realms and users, and create a structure of assets.\ @@ -71,8 +71,4 @@ If you want to deploy the OpenRemote stack on a custom domain then all that is n - `443` HTTPS - `8883` MQTT -The `proxy` service uses Let's Encrypt to auto generate the SSL certificate for the domain and it will also auto renew the certificates; if you already have an SSL certificate for the domain then this can be volume mapped into the `proxy` service. - -## See Also - -- [Installing Docker (incl. SBCs as Raspberry Pi or Odroid)](../../developer-guide/installing-and-using-docker.md) +The `proxy` service uses Let's Encrypt to auto generate the SSL certificate for the domain and it will also auto renew the certificates; if you already have an SSL certificate for the domain then this can be volume mapped into the `proxy` service. \ No newline at end of file diff --git a/docs/user-guide/deploying/release-management.md b/docs/user-guide/deploying/release-management.md index 43e1c854..2c96b49e 100644 --- a/docs/user-guide/deploying/release-management.md +++ b/docs/user-guide/deploying/release-management.md @@ -38,7 +38,7 @@ When updating a custom project to a new OpenRemote release, you can follow the s 1. Read the [release notes](https://github.com/openremote/openremote/releases) to get familiar with the changes 2. Update the code to use the new version: - 1. Docker images: Update the `openremote/manager` image tag in the `docker-compose.yml` file (or environment variable) + 1. Docker images: Update the `openremote/manager` image tag in the `docker-compose.yml` file (or environment variable) [NOTE: The OpenRemote CI/CD will auto set `MANAGER_VERSION` env variable based on what is found for `openremoteVersion` in `gradle.properties` so this may not be needed] 2. Java code: Update the `openremoteVersion` in the `gradle.properties` file 3. TypeScript code: Update the openremote package dependencies, e.g. using `yarn up -E "@openremote/*@^1.2.0"` 3. Check that the code in your custom project still builds correctly using: `./gradlew clean installDist` diff --git a/docs/user-guide/metrics.md b/docs/user-guide/metrics.md index de314480..92a719b5 100644 --- a/docs/user-guide/metrics.md +++ b/docs/user-guide/metrics.md @@ -27,16 +27,17 @@ graph LR subgraph Docker [Docker Containers] direction TB - Manager["Manager
http://localhost:8404/metrics
- Micrometer with Prometheus Registry
- Runs on own embedded web server port 8404
- OR_METRICS_ENABLED: true/false"]:::greenStyle HAProxy["HA Proxy
http://localhost:8404/metrics
- Uses prometheus-exporter
- Runs on own embedded web server port 8404
- Configured via haproxy.cfg"]:::greenStyle + Manager["Manager
http://localhost:8405/metrics
- Micrometer with Prometheus Registry
- Runs on own embedded web server port 8404
- OR_METRICS_ENABLED: true/false"]:::greenStyle Keycloak["Keycloak
http://localhost:8080/metrics
- Built in prometheus metrics support
- KC_METRICS_ENABLED: true/false
- Do not publicly expose"]:::orangeStyle - PostgreSQL["PostgreSQL
- No metrics at present could use postgresql-exporter"]:::redStyle + PostgreSQL["PostgreSQL
http://localhost:8406/metrics
- Uses separate query-exporter docker container and config"]:::redStyle end end %% Connections PromScrape --> Manager PromScrape --> HAProxy + PromScape --> PostgreSQL CWAgent --> CW CW --> DB @@ -429,3 +430,267 @@ Refer to the website of each container app for details of metrics exposed and th + +## PostgreSQL (via Query Exporter) + +The following metrics are exposed by the Query Exporter, which connects directly to the OpenRemote PostgreSQL database to monitor TimescaleDB performance, connection limits, and general database health. The +following is based on the default configuration found in `/deployment/query-exporter/config.yaml`. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Metric nameTypeLabelsDescription
pg_collation_mismatch_countgauge(none)Number of text indexes with collation version mismatches requiring a REINDEX
pg_cache_hit_percentagegauge(none)What percentage of data is being served instantly from RAM versus being slowly read from disk. You want this as high as possible
pg_connections_limitgauge(none)Count of connections max limit
pg_connections_usedgauge(none)Count of connections in use
pg_connections_freegauge(none)Count of connections available
pg_connections_stuckgauge(none)Count of connections with state of idle in transaction
pg_hot_update_percentgaugetable_nameTable percentage of updates that are HOT updates indicates good fillfactor
pg_dead_tuple_percentgaugetable_nameTable ratio of dead tuples to live ones a ratio > 10-20% indicates not aggressive enough autovacuum
pg_last_autovacuum_hoursgaugetable_nameTable hours since last auto vacuum run successfully
pg_last_autoanalyze_hoursgaugetable_nameTable hours since last auto analyze run successfully
pg_db_disk_sizegauge(none)DB size in MB
pg_datapoint_raw_data_sizegauge(none)Asset datapoint table raw uncompressed size in MB
pg_datapoint_indexes_sizegauge(none)Asset datapoint table indexes size in MB
pg_datapoint_toast_sizegauge(none)Asset datapoint TOAST table size in MB
pg_datapoint_disk_sizegauge(none)Asset datapoint table size in MB
pg_datapoint_chunk_countgauge(none)Asset datapoint table hypertable chunk count
pg_datapoint_uncompressed_chunk_countgauge(none)Asset datapoint table hypertable uncompressed chunk count
pg_datapoint_chunks_needing_compressiongauge(none)Asset datapoint table hypertable chunks needing compression count
pg_datapoint_chunk_start_weeksgauge(none)Asset datapoint table oldest hypertable chunk in weeks
pg_datapoint_chunk_end_weeksgauge(none)Asset datapoint table newest hypertable chunk in weeks
pg_datapoint_chunks_not_analyzedgauge(none)Asset datapoint table hypertable chunks not yet analyzed
pg_datapoint_largest_uncompressed_chunkgauge(none)Asset datapoint table largest uncompressed hypertable chunk in MB
pg_datapoint_uncompressed_cache_hit_ratiogauge(none)Asset datapoint table cache hit ratio for uncompressed chunks (Aim for 99%+)
pg_datapoint_uncompressed_blks_read_totalcounter(none)Asset datapoint total physical disk blocks read for uncompressed chunks (Monitor rate with spikes indicate RAM spillover)
pg_datapoint_compression_ratiogauge(none)Asset datapoint table compression ratio
pg_datapoint_querygauge(none)Dummy metric to get typical query time metric
pg_background_errorscounter(none)Count of errors in background worker processes
pg_timescale_job_total_runscounterjob_id | proc_nameTimescaleDB job total runs by job
pg_timescale_job_total_failurescounterjob_id | proc_nameTimescaleDB job total failures by job
pg_timescale_job_last_run_duration_secondsgaugejob_id | proc_nameTimescaleDB job last run duration in seconds
pg_timescale_job_next_start_secondsgaugejob_id | proc_nameSeconds until next scheduled run for each TimescaleDB job
pg_timescale_job_last_run_statusgaugejob_id | proc_name | last_run_statusTimescaleDB job last run status marker
pg_wal_totalcounter(none)Total WAL written since statistics reset in MB
pg_bgwriter_checkpoints_timed_totalcounter(none)Scheduled checkpoints executed
pg_bgwriter_checkpoints_req_totalcounter(none)Requested checkpoints executed
pg_bgwriter_checkpoint_write_time_seconds_totalcounter(none)Total time spent writing checkpoints in seconds
pg_bgwriter_checkpoint_sync_time_seconds_totalcounter(none)Total time spent syncing checkpoints in seconds
pg_table_bloat_countgauge(none)Number of tables where dead tuples > 30% of live rows
pg_index_bloat_countgauge(none)Number of indexes that are larger than 150% of table size
pg_long_running_queries_countgauge(none)Number of queries that have been running for longer than 30 seconds
pg_longest_query_duration_secondsgauge(none)Duration of the longest running query in seconds