S3-Compatible Object Storage¶

By default, Canopy uses the local filesystem for user data file storage. This includes user-uploaded files, such as images, documents, and other data such as generated reports. This is suitable for most installations, especially those on a single server.

Object storage is an alternative to the local filesystem for storing user data. It provides a detached and scalable storage solution that can be accessed over the network. This is useful for deployments using cloud providers or distributed systems.

For these use cases, you can configure Canopy to use S3-compatible object storage.

Some common S3-compatible object storage providers include:

The configuration values for object storage are set in the canopy.ini file.

Basic Configuration¶

The following example shows the minimum set of values required to configure Canopy to use S3-compatible object storage:

OBJECT_STORAGE_ENABLE=true
OBJECT_STORAGE_ENDPOINT_URL=https://s3.eu-west-1.amazonaws.com
OBJECT_STORAGE_BUCKET_NAME=canopy
OBJECT_STORAGE_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
OBJECT_STORAGE_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

The configuration options presented in the following table are the most commonly required for object storage configuration:

Configuration	Description	Example Value(s)	Required
OBJECT_STORAGE_ENABLE	Enable object storage	true \| false	True
OBJECT_STORAGE_ENDPOINT_URL	S3 Endpoint URL for object storage	https://s3.eu-west-1.amazonaws.com	True
OBJECT_STORAGE_BUCKET_NAME	S3 bucket name for object storage	canopy	True
OBJECT_STORAGE_ACCESS_KEY	S3 Access Key	AKIAIOSFODNN7EXAMPLE	True
OBJECT_STORAGE_SECRET_KEY	S3 Secret Key	wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY	True
OBJECT_STORAGE_REGION_NAME	AWS region for the bucket	eu-west-1	False
OBJECT_STORAGE_LOCATION	Path prefix within the bucket	canopy-media	False

Note

The OBJECT_STORAGE_LOCATION setting allows you to set a path prefix within the bucket.

This is only required if you want to store files in a specific subdirectory within the bucket.

The IAM user or role associated with the access key and secret key must have the necessary permissions to access the specified bucket and perform operations such as uploading, downloading, and deleting files.

The following example IAM policy grants the necessary permissions to access the specified bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CanopyObjectStorageAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME/*",
                "arn:aws:s3:::BUCKET_NAME"
            ]
        }
    ]
}

Note

This policy should be attached to the IAM user or role associated with the access key and secret key.

Ensure that the bucket exists and the region is correctly specified in your configuration.

Replace BUCKET_NAME with your actual bucket name.

Additional Configuration¶

The following table provides some additional configuration options that are available and may be useful for specific use cases:

Configuration	Description	Example Value(s)
OBJECT_STORAGE_VERIFY	Certificate verification for S3 requests	false \| certs/cert.pem
OBJECT_STORAGE_CLIENT_CONFIG	Custom boto3 client configuration	connect_timeout=120,max_pool_connections=20
OBJECT_STORAGE_EXTRA_OPTIONS	Additional S3 storage options	querystring_expire=3600,querystring_auth=true

Note

The OBJECT_STORAGE_CLIENT_CONFIG setting allows you to specify additional configurations to the S3 storage client.

This is a comma-separated list of key-value pairs, such as option1=value1,option2=value2.

The compatible options are listed in the botocore documentation.

Note

The OBJECT_STORAGE_EXTRA_OPTIONS setting allows you to specify additional configurations to the storage backend.

This is a comma-separated list of key-value pairs, such as option1=value1,option2=value2.

The compatible options are listed in lowercase format in the django-storages documentation.

Data Migration¶

In order to migrate existing data to/from object storage, you can use the canopy-manage syncstorage command.

Warning

Ensure that you have configured the object storage settings in the canopy.ini file before running this command.

This command will copy all user data from the local filesystem to the configured object storage bucket, or vice versa.

To view a list of available options, run the command with the --help flag.

Warning

canopy-manage syncstorage doesn’t preserve file creation timestamps.

The rclone tool can also be used to migrate data between the local filesystem and object storage and is an alternative that preserves file metadata.

Logging¶

By default, underlying interactions with the object storage are not logged.

To debug issues with object storage, you can enable logging by setting the LOG_LEVEL_STORAGES option to DEBUG in the canopy.ini file and by viewing the Canopy logs using journalctl -u canopy -e.