Celery message broker

Canopy runs background jobs, such as report generation and portal synchronisation, through a message broker. Valkey 7.2 and newer is the preferred broker. RabbitMQ 3.x/4.x remains supported, and is the broker to use on Ubuntu 22.04 and EL 9, where Valkey is not packaged.

A single-machine installation needs one step, canopy-setup standalone, which configures the broker along with PostgreSQL and nginx and generates a broker password for you. canopy-setup valkey and canopy-setup rabbitmq configure the broker on its own. Nothing else below is needed for a single-server installation.

Broker configuration

Set CELERY_BROKER in /etc/canopy/canopy.ini. Valkey URLs use the redis scheme:

CELERY_BROKER=redis://:PASSWORD@localhost:6379/0

An existing RabbitMQ deployment can continue to use an AMQP URL:

CELERY_BROKER=amqp://guest:guest@localhost:5672//

Canopy owns /etc/valkey/canopy.conf and rewrites it whenever canopy-setup valkey runs, so any edit you make there is lost. Put your own Valkey settings in /etc/valkey/canopy-local.conf, which Canopy creates once and never rewrites. Both files are included at the end of /etc/valkey/valkey.conf, with yours last, so your settings override Canopy’s.

To configure the broker, or to repair its configuration, as root:

canopy-setup valkey

The Celery broker health check warns when Valkey approaches its memory limit. The warning does not mark the deployment unhealthy. Raise the limit, as root:

canopy-setup valkey -e valkey_maxmemory_mb=1024

Changing the broker password

The generated password is stored in requirepass in /etc/valkey/canopy.conf and in the CELERY_BROKER URL in /etc/canopy/canopy.ini. Both files are readable by root and by their service group only. An upgrade changes neither.

These steps apply when Canopy authenticates as the default account. Under an ACL, change the password in the account line instead, as described in Restricting Canopy with an ACL.

To set a password of your own, as root:

  1. Stop Canopy:

    systemctl stop canopy canopy-celery

  2. Set the new value in the CELERY_BROKER URL in /etc/canopy/canopy.ini. The password must not contain characters that need percent-encoding in a URL.

  3. Copy the value into requirepass in /etc/valkey/canopy.conf, or run canopy-setup valkey, which copies it from the URL for you.

  4. Restart the broker, then start Canopy. The unit is valkey-server on Ubuntu and valkey on EL 10:

    systemctl restart valkey-server
systemctl start canopy canopy-celery

Restricting Canopy with an ACL

A password alone gives Canopy the default account, which can run every Valkey command. An ACL account restricts it to what Canopy uses. Use one when the broker is reached over a network.

Add these two lines to /etc/valkey/canopy-local.conf, each on one line, replacing PASSWORD:

user canopy on >PASSWORD ~* &* +@read +@write +@list +@hash +@set +@sortedset +@string +@pubsub +@transaction +@scripting +@connection -@admin -flushall -flushdb -swapdb -keys +info
user default off

Then put the account name in the URL, in place of the empty field used for a password on its own:

CELERY_BROKER=redis://canopy:PASSWORD@localhost:6379/0

Run canopy-setup valkey afterwards. It reads the account name from the URL and stops copying that password into requirepass, so the default account keeps a password of its own and cannot be used to bypass the ACL.

Use the account line exactly as given. Canopy needs every grant listed, and each exclusion removes a command that would otherwise let a compromised or faulty client reconfigure the server or empty it in one step. -flushall and -flushdb are both needed, because +@write grants them.

Warning

Risk: the account can reconfigure or shut down the server if the exclusions are removed, or moved before the grants above them.

Warning

Risk: user default off locks out every client that authenticates with requirepass alone, including valkey-cli -a. Use valkey-cli --user canopy --pass PASSWORD instead.

Transport encryption

Valkey listens in plaintext on port 6379 by default, and Canopy’s default configuration matches that. A localhost-only broker needs no TLS.

Enable TLS when the broker is on another host. Configure tls-port and the certificate settings in Valkey, then use the rediss scheme in CELERY_BROKER.

Warning

Risk: a rediss:// URL on its own does not verify the broker’s certificate, which leaves the connection open to an active machine-in-the-middle attack. Always set CELERY_BROKER_USE_SSL together with a rediss:// URL.

Set both values in /etc/canopy/canopy.ini:

CELERY_BROKER=rediss://:PASSWORD@valkey.example.com:6379/0
CELERY_BROKER_USE_SSL={"ssl_cert_reqs": "required", "ssl_ca_certs": "/etc/ssl/certs/ca-certificates.crt"}

ssl_cert_reqs must be required. ssl_ca_certs must point at the certificate authority that signed the broker’s certificate. Add ssl_certfile and ssl_keyfile when the broker requires a client certificate.

Message timeout

Valkey and Redis have no acknowledgement mechanism, so Celery emulates one. It re-queues any message a worker has held for longer than CELERY_BROKER_VISIBILITY_TIMEOUT. The default is 43200 seconds, which is 12 hours. Set it in /etc/canopy/canopy.ini:

CELERY_BROKER_VISIBILITY_TIMEOUT=43200

The value must be a whole number of seconds, and at least 1. Canopy applies it to a Valkey or Redis broker only. RabbitMQ and SQS have acknowledgement mechanisms of their own and ignore it.

A message ages from the moment a worker receives it, not from the moment the task starts. Set the value above the longest time a message can wait behind other work, not just above the longest a task runs.

Warning

Risk: a job runs twice and repeats its effects, such as sending a notification email again or regenerating a report, if you lower CELERY_BROKER_VISIBILITY_TIMEOUT or change CELERY_WORKER_DISABLE_PREFETCH. Canopy sets both for a Valkey broker. Contact support before changing either.

A visibility_timeout entry in CELERY_BROKER_TRANSPORT_OPTIONS still overrides CELERY_BROKER_VISIBILITY_TIMEOUT. Use the dedicated setting instead; the entry remains only for deployments that already carry one.

Configuring RabbitMQ

canopy-setup rabbitmq runs playbooks/rabbitmq.yml. The play configures a RabbitMQ on the Canopy machine. It never configures or contacts another host.

Use the command in these cases:

  • Your platform does not package Valkey. Ubuntu 22.04 and EL 9 need RabbitMQ.

  • You keep a RabbitMQ deployment that you already run.

  • You run the steps one at a time instead of canopy-setup standalone. canopy-setup postgresql configures the database, canopy-setup nginx configures the reverse proxy, and this command configures the broker. See The standalone setup command.

  • You restore a broker that stopped. The command starts RabbitMQ and enables it at boot.

Valkey is the preferred broker on a platform that packages it: Ubuntu 24.04/26.04 and EL 10. Use canopy-setup valkey there. RabbitMQ stays supported on those platforms, and this command configures it.

Do not use the command when your broker runs on another host. Set CELERY_BROKER by hand instead.

canopy-setup standalone runs this play for you on Ubuntu 22.04 and EL 9. You do not need both commands.

What it does

  1. On Ubuntu, installs rabbitmq-server from the distribution archive.

  2. On EL, checks that rabbitmq-server is installed. It stops with an error when the package is absent. No EL repository carries it, so you install it first. See Platform installation guides.

  3. Starts the service and enables it at boot.

  4. Sets CELERY_BROKER=amqp://guest:guest@localhost:5672//, then restarts canopy and canopy-celery.

Canopy connects as guest, the account RabbitMQ ships with. RabbitMQ accepts that account over the loopback address only.

Warning

Risk: a broker on another host refuses the guest account. Give that broker an account of its own, and use TLS.

What it leaves alone

The command claims CELERY_BROKER only while that setting names a local service and carries no credentials. It keeps a URL that names another host, and it keeps a URL that carries a user name or a password.

A Valkey URL carries the generated password, so this command does not replace it. To go back to RabbitMQ from Valkey, edit CELERY_BROKER by hand. See Migrating from RabbitMQ.

The guest URL above carries credentials too, so a second run does not rewrite it. Correct a wrong broker URL by hand.

RabbitMQ 4.3.0 and later

RabbitMQ 4.3.0 disables its deprecated features by default. One of them, transient_nonexcl_queues, covers queues that are neither durable nor exclusive. Celery declares its control and event queues that way, so the broker rejects them.

RabbitMQ 4.2.x and earlier permit the feature and need no change.

Warning

Risk: Canopy’s background jobs stop on RabbitMQ 4.3.0 and later. The broker refuses the queue declaration with (541) INTERNAL_ERROR.

To permit the feature, add this key to rabbitmq.conf on every node in the cluster:

deprecated_features.permit.transient_nonexcl_queues = true

Then restart RabbitMQ.

Warning

Risk: the key has no effect when only some nodes carry it. Add it to every node, and add it before you upgrade the cluster to 4.3.0.

RabbitMQ removes the feature in a later release. Move to Valkey where your platform packages it. See Migrating from RabbitMQ.

Migrating from RabbitMQ

Migration does not transfer queued or running jobs. Schedule a maintenance window and stop Canopy before changing the broker. Any jobs left in RabbitMQ will not be available through Valkey.

As root:

  1. Stop the application and worker:

    systemctl stop canopy canopy-celery

  2. Install and configure Valkey. This also generates a password and sets CELERY_BROKER to the matching redis:// URL:

    canopy-setup valkey

  3. Verify Valkey is available, using the password from /etc/valkey/canopy.conf:

    valkey-cli -a PASSWORD ping

    The response must be PONG.

  4. For a broker on another host, change CELERY_BROKER in /etc/canopy/canopy.ini to the appropriate redis:// or rediss:// URL, and set CELERY_BROKER_USE_SSL for rediss.

  5. Start Canopy and inspect the service logs:

    systemctl start canopy canopy-celery
journalctl -e -u canopy -u canopy-celery

  6. Keep RabbitMQ installed until the Valkey-backed deployment has been accepted. To roll back, stop Canopy, restore the previous AMQP URL, ensure RabbitMQ is running, and start Canopy again.

Jobs submitted to either broker after a switch remain only in that broker. Canopy never stops or removes RabbitMQ during an upgrade.