docs: restructure how-to and tutorial pages

docs/explanations/extension-code.md

@@ -11,7 +11,7 @@ Plans and devices can be in any pip-installable package, such as:
 
 * A package on pypi
 * A GitHub repository
-* A local directory via the [scratch area](../how-to/edit-live.md).
+* A local directory via the [scratch area](../how-to/1.5%20edit-live.md).
 
 The easiest place to put the code is a repository created with the [`python-copier-template`](https://diamondlightsource.github.io/python-copier-template/main/index.html). Which can then become any of the above. [Example for the I22 beamline](https://github.com/DiamondLightSource/i22-bluesky).
 

docs/how-to/configure-app.md → docs/how-to/1. configure-app.md

@@ -1,4 +1,4 @@
-# Configure the Application
+# 1. Configure the Application
 
 Blueapi's default configuration can be overridden
 by defining a yaml file which can be passed to the `blueapi` command.
@@ -16,3 +16,12 @@ avoid it having to be passed to every time the cli is used:
 export BLUEAPI_CONFIG=/path/to/file.yaml
 blueapi <subcommand>
 ```
+
+The next few pages detail various aspects that can be configured.
+
+1.1 [Add plans and devices](./1.1%20add-plans-and-devices.md)
+1.2 [Authentication/Authorization](./1.2%20authn-authz.md)
+1.3 [Logging](./1.3%20configure-logging.md)
+1.4 [Numtracker](./1.4%20integrate-with-numtracker.md)
+1.5 [Install editable Python packages](./1.5%20edit-live.md)
+1.6 [Tiled](./1.6%20configure-tiled.md)

docs/how-to/add-plans-and-devices.md → docs/how-to/1.1 add-plans-and-devices.md

@@ -1,16 +1,16 @@
-# Add Plans and Devices to your Blueapi Environment
+# 1.1 Add Plans and Devices to your Blueapi Environment
 
 :::{seealso}
 [The bluesky documentation](https://blueskyproject.io/bluesky/main/index.html) for an introduction to the nature of plans and devices and why you would want to customize them for your experimental needs.
 :::
 
-Blueapi can be configured to load custom code at startup that defines plans and devices. The code must be in your Python environment (via `pip install <package>`) or your [scratch area](./edit-live.md).
+Blueapi can be configured to load custom code at startup that defines plans and devices. The code must be in your Python environment (via `pip install <package>`) or your [scratch area](./1.5%20edit-live.md).
 
 
 ## Configuration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](./1.%20configure-app.md)
 :::
 
 First determine the import path of your code. If you were going to import it in a Python file, what would you put?
@@ -25,6 +25,8 @@ To add plans, you would add the following into your configuration file:
 :language: yaml
 ```
 
+For BlueAPI to detect plans from repositories and be registered, they must be added to the `__init__.py` file with `__all__` = [<plan_name>, ...].
+
 
 Devices are added similarly, using `deviceManager` as the `kind`, like so:
 ```{literalinclude} ../../tests/unit_tests/valid_example_config/plans_and_devices.yaml

docs/how-to/authenticate.md → docs/how-to/1.2 authn-authz.md

@@ -2,16 +2,12 @@
 > If you are using `oauth2-proxy` to secure the Swagger UI documentation page, you can log out by visiting the `/logout` URL. For this to work correctly, ensure that the blueapi server is configured with 
 > `oidc.logout_redirect_endpoint` set to `/oauth2/sign_out`, which is required for `oauth2-proxy`.
 
-# Authenticate to BlueAPI-Cli
+# 1.2 Authenticate/Authorise to BlueAPI-Cli
 
-## Introduction
+## Authentication
 BlueAPI provides a secure and efficient way to interact with its services. This guide walks you through the steps to log in and log out using BlueAPI with OpenID Connect (OIDC) authentication.
 
-## Configuration
-
-:::{seealso}
-[Configure the Application](./configure-app.md)
-:::
+### Configuration
 
 Here is an example configuration for authenticating to p46-blueapi:
 ```{literalinclude} ../../tests/unit_tests/valid_example_config/client_auth.yaml
@@ -22,7 +18,7 @@ Here is an example configuration for authenticating to p46-blueapi:
 
 ---
 
-## Log In
+### Log In
 
 1. Execute the login command:
 
@@ -43,7 +39,7 @@ Here is an example configuration for authenticating to p46-blueapi:
 
 ---
 
-## Log Out
+### Log Out
 
 To log out and securely remove the cached access token, follow these steps:
 
@@ -63,3 +59,16 @@ To log out and securely remove the cached access token, follow these steps:
    ```
    Logged out
    ```
+
+## Authorization 
+
+The authorization provider is opa.diamond.ac.uk
+
+### Configuration 
+
+An example configuration to to customise authorization is shown in this `config.yaml` file. 
+
+```{literalinclude} ../../tests/system_tests/config.yaml
+:language: yaml
+```
+Else, the default values in `OpaConfig` class defined in `config.py` will be used. 

docs/how-to/configure-logging.md → docs/how-to/1.3 configure-logging.md

@@ -1,5 +1,5 @@
 
-# Configure Logging
+# 1.3 Configure Logging
 
 By default BlueAPI will log to stdout at the [INFO level](https://docs.python.org/3/library/logging.html#logging-levels), but can be reconfigured to log at any level, and to output to Graylog.
 
@@ -8,7 +8,7 @@ When logging to [Graylog](https://graylog.org) is enabled, BlueAPI will also con
 ## BlueAPI Cofiguration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](./1.%20configure-app.md)
 :::
 
 An example logging config is shown below:

docs/how-to/integrate-with-numtracker.md → docs/how-to/1.4 integrate-with-numtracker.md

@@ -1,12 +1,12 @@
-# Integrate With Numtracker
+# 1.4 Integrate With Numtracker
 
 [Numtracker](https://github.com/DiamondLightSource/numtracker.git) is a DLS API that can be used to coordinate where detectors write their data.
 
 ## Prequisities
 
 You will need to [get numtracker itself configured for your instrument](https://github.com/DiamondLightSource/numtracker/wiki/new_beamline). 
 
-Blueapi needs [valid authentication configured](./authenticate.md) to communicate with numtracker. It will propogate its auth token so both blueapi and numtracker should be aware of who the user is and that they have permission to be on the instrument sessions (visits) that are intended for use.
+Blueapi needs [valid authentication configured](../how-to/1.2%20authn-authz.md) to communicate with numtracker. It will propogate its auth token so blueapi is aware of who the user is and that they have permission to be on the instrument sessions (visits) that are intended for use. Numtracker will not be aware of the user. 
 
 ## Configuration
 

docs/how-to/edit-live.md → docs/how-to/1.5 edit-live.md

@@ -1,4 +1,4 @@
-# Edit Plans and Device Live
+# 1.5 Edit Plans and Device Live
 
 You may want to tweak/edit your plans and devices live, i.e. without having to make a new release of a Python module, `pip install` it and restart blueapi. Blueapi can be configured to use a special directory called the "scratch area" where source code can be checked out and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).
 

docs/how-to/1.6 configure-tiled.md

@@ -0,0 +1,28 @@
+# 1.6 Tiled
+
+Tiled is a server and data management system used to store and organise experimental data through a web API. It is run as a (mostly) centralised service. The RunEngine inserts bluesky documents directly into Tiled. At the moment, NeXus files are written from the messages published on the message bus. Tiled should contain the same information that is in the NeXus files but in due course, Tiled should be the one source of truth. 
+
+## Configuration
+
+An example configuration to enable tiled is shown in this `config.yaml` file. 
+
+```{literalinclude} ../../tests/system_tests/config.yaml
+:language: yaml
+```
+
+The options to configure Tiled are in the `TiledConfig` model which can be found in the [config.py file](../../src/blueapi/config.py). Apart from the url, the main aspect is authentication. This is either an ApiKey:
+
+```
+authentication: secret-key 
+```
+
+or service account details:
+
+```
+authentication:
+    client_id: "client-id"
+    client_secret: "secret"
+    token_url: "url"
+```
+
+The values in the `config.yaml` are the default ones use if the user doesn't provide them. 

docs/how-to/deploy-new-blueapi-version.md

@@ -0,0 +1,8 @@
+# Deploying a new version of BlueAPI
+
+Follow these [instructions](https://gitlab.diamond.ac.uk/sscc-docs/developer-guide/-/blob/4f28b0c4b17e34ae19faf698024bceb6e20992c9/topics/kubernetes/docs/tutorials/local_helm_on_argocd.md) for changing/testing new Blueapi deployments. 
+
+
+Links to the relevant services and deployment repositories:
+- Testing (pxx) beamlines: [epics-containers](https://github.com/epics-containers)
+- Other beamlines: [GitLab folder](https://gitlab.diamond.ac.uk/controls/containers/beamline)

docs/tutorials/local-auth.md → docs/how-to/local-auth.md

@@ -1,6 +1,6 @@
-# Run Auth Locally
+# Run BlueAPI and connect to services locally
 
-BlueAPI can be secured using OIDC authenticaiton. For development it can be useful to run a containerised version of the OIDC stack, to serve a local instance of BlueAPI.
+For development purposes, it can be useful to run BlueAPI and adjacent services (Numtracker, Tiled, OPA etc.) locally. To do this, a containerised version of the stack should be run. This includes launching an instance of NumTracker, RabbitMQ, Keycloak, Tiled, OPA and a number of IOCs, in detached mode. Running BlueAPI and connected services locally as opposed to connecting to the cluster can be useful for learning more about the stack, running system tests, checking if changes during development propagate as expected etc. 
 
 To run the stack:
 
@@ -18,6 +18,8 @@ To run the stack:
 > alias docker=podman
 > ```
 
+There are also docs in the [Developer Guide](https://dev-guide.diamond.ac.uk/epics-containers/reference/setup/#docker-compose) to set up docker-compose on DLS workstations.
+
 3. Run `source tests/system_tests/.env` which will set required EPICS environmental variables
 4. Run `blueapi -c tests/system_tests/config.yaml serve` to launch BlueAPI configured to use the launched stack. This may take a while, as BlueAPI will attempt to connect to a number of devices via Channel Access
 

docs/how-to/run-container.md

@@ -19,3 +19,12 @@ podman run ghcr.io/diamondlightsource/blueapi:latest
 ```
 
 To get a released version, use a numbered release instead of `latest`.
+
+## Starting the message bus
+
+The worker requires a running instance of the message bus, which in this case is RabbitMQ. The easiest way to start it is
+ to `compose` the services in `tests/system_tests/compose.yaml`
+
+```sh
+docker compose -f tests/system_tests/compose.yaml run rabbitmq --detach
+```

docs/tutorials/run-with-dev.md → docs/how-to/run-with-dev.md

@@ -10,7 +10,7 @@ Ensure you are inside your virtual environment:
 source venv/bin/activate
 ```
 
-You will need to follow the instructions for setting up RabbitMQ as in [instructions for setting up RabbitMQ](run-bus.md).
+You will need to follow the instructions for setting up RabbitMQ as in [instructions for setting up RabbitMQ](../tutorials/5.%20config-add-rabbitmq-and-run-plan-cli.md).
 
 The worker will be available from the command line (`blueapi serve`), but can be started from vscode with additional
 debugging capabilities.
@@ -22,5 +22,5 @@ debugging capabilities.
 [debug in vscode](../images/debug-vscode.png)
 
 :::{seealso}
-[Scratch Area](../how-to/edit-live.md) for in-the-loop development of plans and devices
+[Scratch Area](../how-to/1.5%20edit-live.md) for in-the-loop development of plans and devices
 :::

docs/how-to/use-persistent-volumes.md

@@ -5,7 +5,7 @@ BlueAPI can use persistent volumes as a scratch area. This allows the user to re
 ## Configuration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](../how-to/1.%20configure-app.md)
 :::
 
 The relevant configuration is below:

docs/how-to/write-devices.md

@@ -23,5 +23,5 @@ The return type annotation `-> MyTypeOfDetector` is required as blueapi uses it
 The device is created via a function rather than a global to preserve side-effect-free imports. Each device must have its own factory function.
 
 :::{seealso}
-[Numtracker integration](./integrate-with-numtracker.md) for how to configure where detectors write files.
+[Numtracker integration](../how-to/1.4%20integrate-with-numtracker.md) for how to configure where detectors write files.
 :::

docs/reference/dev-commands.md

@@ -0,0 +1,14 @@
+# Commands for dev tooling
+
+This page outlines useful commands for dev tooling. More information can be found in the [python-copier-template docs](https://github.com/DiamondLightSource/python-copier-template/tree/main/docs). 
+
+The table below outlines useful commands with instructions for download on Diamond workstations.
+
+| Command           | Installation  | Example of use|
+| -------------     | ------------- | ------------- |
+| `podman`          | [Dev-Guide link](https://dev-guide.diamond.ac.uk/containers/tutorials/podman/)  | `podman run ghcr.io/diamondlightsource/blueapi:latest` to pull the blueapi container from the GitHub container registry |
+| `docker-compose`  | [Dev-Guide link](https://dev-guide.diamond.ac.uk/epics-containers/reference/setup/#docker-compose)  | `docker compose -f tests/system_tests/compose.yaml up -d` to spin up dummy versions of blueapi associated services |
+| `uv`              | `module load uv`  | `uv run --with blueapi path/to/script.py` to run a standalone script |
+| `prek`            | Run `uv run prek install` the first time the repo is cloned to set up pre-commit checks  | `uv run prek` or `uv run prek --all-files`  |
+| `tox`             |  `uv tool install tox` | `tox -e tests -- tests/unit_tests/test_config.py::test_config_yaml_parsed` is an example of running a specific test and `tox -e tests` will run all unit tests|
+| `just`            | `uv tool rust-just`  | `just lint` will run all pre-commit checks|

docs/resources/enabled_plugins

@@ -0,0 +1 @@
+[rabbitmq_stomp].

docs/tutorials/installation.md → docs/tutorials/1. installation.md

@@ -1,4 +1,4 @@
-# Installation
+# 1. Installation
 
 ## Check your version of python
 
@@ -8,6 +8,10 @@ typing into a terminal:
 python3 --version
 ```
 
+## Clone the blueapi repository locally 
+
+Clone the blueapi repository to your local scratch directory and `cd` into it.
+
 ## Create a virtual environment
 
 It is recommended that you install into a “virtual environment” so this

docs/tutorials/2. quickstart.md

@@ -0,0 +1,39 @@
+# 2. Quickstart guide
+
+Blueapi acts as a worker that can run Bluesky plans against devices for a specific laboratory setup. It can control devices to collect data and export events to tell downstream services about the data it has collected.
+
+You should be in the blueapi directory that you have cloned. From here, if you open VSCode, you will probably be prompted to 'Reopen in Container' which you can do and then run the following commands to start the worker. 
+
+If you not, you can still run the following from the venv created in page 1 but you may need to run the command below to have the dependencies required for this tutorial.
+
+```
+pip install .[demo]
+```
+
+## Start Worker
+
+To start the worker:
+```
+blueapi serve
+```
+This will use all the defaults in `src/blueapi/config.py`
+
+## Basic Introspection
+
+Blueapi comes with a minimal CLI client for basic control/debugging. We can use this to see which plans and devices are available. In a new terminal window, run the following commands. 
+
+If working in a venv, you may need to reactivate it. To do this, assuming you are still in the blueapi repo in your terminal, use:
+```
+source .venv/bin/activate
+```
+
+Now, run the following to see what plans and devices are available:
+
+``` 
+blueapi controller plans
+blueapi controller devices
+```
+
+We should now have an empty context.
+
+See also [Full CLI reference](../reference/cli.md)

docs/tutorials/3. config-add-devices.md

@@ -0,0 +1,28 @@
+# 3. Config add devices
+
+Blueapi is highly configurable. The config details aspects that impact how Blueapi works and it is customisable. 
+
+As a first step, create a `config.yaml` file. This would be best placed in the `docs/resources` folder. 
+
+Initially, add devices using the following code:
+
+```yaml
+env:
+  metadata:
+    instrument: demo
+  sources:
+    - kind: deviceManager
+      module: blueapi.tutorial.devices
+```
+
+The `env` section of the `config.yaml` file lets you create and set environment variables. The `demo` points to optional dependencies in the [pyproject.toml file](../../pyproject.toml). The module above points to the devices you will use for this tutorial which can be found [here](../../src/blueapi/tutorial/devices.py).
+
+Save the file and restart the worker to use this new `config.yaml` file you have created. To do this, navigate back to the terminal window where the worker is running, CTRL+C to close it then run the following command: 
+``` 
+blueapi --config docs/resources/config.yaml serve 
+```
+
+The above command assumes you created the `config.yaml` file in `docs/resources`. Else, replace `docs/resources/config.yaml` with whichever file path your `config.yaml` has.
+
+Now run 
+```blueapi controller devices``` in the same terminal window as you did when previously running this command and you should see `det` and `stage`.

docs/tutorials/4. config-add-plans.md

@@ -0,0 +1,16 @@
+# 4. Config add plans
+
+As well as adding devices to the `config.yaml`, you can add plans. Under the code you already have, add the following:
+
+``` yaml
+    - kind: planFunctions
+      module: dodal.plans
+    - kind: planFunctions
+      module: dodal.plan_stubs.wrapped
+```
+These plans that will be used in this tutorial can be found in the [dodal repository](https://github.com/DiamondLightSource/dodal/tree/b4b38f1e0cf480faeb29d67695440c45a148f169). Specifically, in `dodal/src/dodal/plans/wrapped.py` and `dodal/src/dodal/plan_stubs/wrapped.py`.
+
+Save the changes you've made to the `config.yaml` file. Navigate back to the terminal where the blueapi worker is running. Restart the server to use this updated `config.yaml` file.
+
+Now run 
+```blueapi controller plans``` in the same terminal window used previously for this command and you should see a list of plans that can be run. For example, `count`, `list_grid_scan`, `num_scan` etc.