--- description: Store base64Binary element values in external blob storage instead of PostgreSQL and serve resources with the data inlined on read. --- > For the complete documentation index, see [llms.txt](https://staging.health-samurai.io/docs/aidbox/llms.txt). > Use it to discover all available pages before guessing URLs. --- # Offload base64Binary data to external storage {% hint style="info" %} This functionality is available starting from Aidbox version **2607**. {% endhint %} FHIR resources carry binary payloads in `base64Binary` elements. `Binary.data`, `DocumentReference.content.attachment.data`, and `Patient.photo.data` are common examples, many other resource types have such elements, and extensions with a `valueBase64Binary` value can appear on any element. Stored inline, these payloads grow the resource tables and everything built on top of them: history, backups, replication. Data offload moves the payloads out of the database. On create and update, Aidbox uploads the decoded bytes to external blob storage and stores the resource with a pointer to the blob in place of the data. On read, Aidbox downloads the bytes and returns the resource with the data inlined, so API clients work with the resource as if nothing was offloaded. Offload is a property of an [API](README.md#apis). You configure it with the `dataOffloadToExternalStorage` parameter of [`$create-api`](README.md#usdcreate-api) or [`$configure-api`](README.md#usdconfigure-api). [Azure Blob Storage](#azure-blob-storage), [AWS S3](#aws-s3), and [GCP Cloud Storage](#gcp-cloud-storage) are the storage providers supported today. ## How it works ``` ╭────────╮ ╭────────╮ ╭──────────────────╮ │ │ │ │ │ │ │ Client ├create─or─update─►│ Aidbox │ ├────resource─with─hash─and─location────►│ PostgreSQL │ │ │ │ │ │ │ ╰────────╯ ╰────┬───╯ ╰──────────────────╯ │ │ │ │ │ │ ╭──────────────────╮ │ │ │ └───────────────────decoded─bytes──────────────────►│ External storage │ │ │ ╰──────────────────╯ ``` On create (`POST`) and update (`PUT`), Aidbox finds the configured `base64Binary` elements in the incoming resource. For each element that has a value, Aidbox: 1. Decodes the base64 value and uploads the bytes to the configured storage as a blob named by a random UUID, placed under `blobNamePrefix` when the configuration sets one. 2. Removes the element value from the resource. 3. Adds an extension on the element (a primitive extension, in the `_data` form) that records the blob location and the hash of the data. The blob holds the raw decoded bytes, not the base64 string. Downloading it with the provider's own tooling yields the original file. Aidbox commits the blob after the resource write succeeds. If the write fails, Aidbox abandons the upload and no blob appears in the storage. The stored extension looks like this: ```json { "resourceType": "Binary", "id": "b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02", "contentType": "application/octet-stream", "_data": { "extension": [ { "url": "http://health-samurai.io/fhir/core/data-offloaded-to-external-storage", "extension": [ { "url": "hash", "valueString": "Kq5sNclPz7QV2+lfQIuc6R7oRu0=" }, { "url": "location", "valueString": "azure://my-container/mystorageaccount/4f1f61a2-9e3b-4b0e-bb1d-6a1a1c2f7e58" } ] } ] } } ``` | Sub-extension | Description | |---------------|--------------------------------------------------------------------------------------------------------------------| | `hash` | Base64-encoded SHA-1 digest of the decoded data, the same convention as [`Attachment.hash`](https://build.fhir.org/datatypes-definitions.html#Attachment.hash). | | `location` | Blob address in a provider-specific form. See [Storage providers](#storage-providers). | On instance read (`GET /fhir/{resourceType}/{id}`), Aidbox downloads the blob, restores the element value, and removes the offload extension from the response. Other extensions on the same element stay in place, both in storage and in responses. ## Configuration Pass the `dataOffloadToExternalStorage` parameter to `$create-api` or `$configure-api`. Its parts: | Part | Type | Required | Description | |---------------------------------|-----------------------------|-------------------------------------|-----------------------------------------------------------------------------| | `fhirpathToBase64BinaryElement` | string | yes, repeatable | Path to a `base64Binary` element to offload. See the expression rules below. | | `storageProvider` | code | yes | `azure`, `aws`, or `gcp`. See [Storage providers](#storage-providers). | | `blobNamePrefix` | string | no | Common path for the blob names, such as `env-1/tenant-a`. Aidbox names each blob `{prefix}/{uuid}`. | | `azureContainer` | Reference(`AzureContainer`) | when `storageProvider` is `azure` | Container that receives the blobs. | | `awsAccount` | Reference(`AwsAccount`) | when `storageProvider` is `aws` | Account with the credentials and region for S3 access. | | `awsBucket` | string | when `storageProvider` is `aws` | Bucket that receives the objects. | | `gcpServiceAccount` | Reference(`GcpServiceAccount`) | when `storageProvider` is `gcp` | Service account with the credentials for Cloud Storage access. | | `gcpBucket` | string | when `storageProvider` is `gcp` | Bucket that receives the objects. | ### Element path expressions `fhirpathToBase64BinaryElement` accepts a restricted FHIRPath subset: element names separated by dots, with an optional `[n]` index on array elements. Functions and filters are not supported. | Expression | Effect | |---------------------------|-----------------------------------------------------------------------------| | `data` | `Binary.data`. | | `photo.data` | `data` of every `Patient.photo` item. | | `photo[0].data` | `data` of the first `Patient.photo` item. | | `content.attachment.data` | `data` of every `DocumentReference.content` item. | | `extension.valueBase64Binary` | `base64Binary` values of top-level extensions. | When a path segment names an array and carries no index, the expression matches every item. Missing and empty elements are skipped. ## Storage providers `storageProvider` selects where the blobs go, and each provider brings its own configuration parts and prerequisites. ### Azure Blob Storage Set `storageProvider` to `azure` and reference an [AzureContainer](../../reference/system-resources-reference/core-module-resources.md#azurecontainer) resource in the `azureContainer` part. Blobs land in that container, and the `location` sub-extension records them as `azure://{container-name}/{storage-account}/{blob-name}`. Offload uses the same [AzureAccount](../../reference/system-resources-reference/core-module-resources.md#azureaccount) and [AzureContainer](../../reference/system-resources-reference/core-module-resources.md#azurecontainer) resources as the [Azure Blob Storage](../../file-storage/azure-blob-storage.md) file storage integration. Create an `AzureAccount` with one of the supported credential sets: * `key`: a storage account access key. * `tenantId`, `clientId`, and `clientSecret`: Azure AD application credentials. * No credentials: Aidbox falls back to [DefaultAzureCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.identity.defaultazurecredential), which picks up workload identity in environments configured for it. For the Azure AD methods, the identity needs a role that grants blob read and write access on the container, such as `Storage Blob Data Contributor`. ```http PUT /fhir/AzureAccount/my-account Content-Type: application/json { "key": "" } ``` Create an `AzureContainer` that points to the storage account and container: ```http PUT /fhir/AzureContainer/my-container Content-Type: application/json { "account": { "id": "my-account", "resourceType": "AzureAccount" }, "storage": "mystorageaccount", "container": "my-container" } ``` ### AWS S3 Set `storageProvider` to `aws`, reference an [AwsAccount](../../reference/system-resources-reference/core-module-resources.md#awsaccount) resource in the `awsAccount` part, and name the bucket in `awsBucket`. Objects land in that bucket, and the `location` sub-extension records them as `s3://{bucket}/{object-name}`. Offload uses the same `AwsAccount` resource as the [AWS S3](../../file-storage/aws-s3.md) file storage integration. The `region` field is required. Pass `access-key-id` and `secret-access-key` for explicit credentials, or omit them so Aidbox uses the [default credentials provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) (environment variables, instance profile, or pod identity). ```http PUT /fhir/AwsAccount/my-aws-account Content-Type: application/json { "region": "us-east-1", "access-key-id": "", "secret-access-key": "" } ``` The examples below configure Azure. For AWS, the offload parameter carries `awsAccount` and `awsBucket` instead of `azureContainer`: ```json { "name": "dataOffloadToExternalStorage", "part": [ { "name": "fhirpathToBase64BinaryElement", "valueString": "data" }, { "name": "storageProvider", "valueCode": "aws" }, { "name": "awsAccount", "valueReference": { "reference": "AwsAccount/my-aws-account" } }, { "name": "awsBucket", "valueString": "my-bucket" }, { "name": "blobNamePrefix", "valueString": "env-1/tenant-a" } ] } ``` ### GCP Cloud Storage Set `storageProvider` to `gcp`, reference a [GcpServiceAccount](../../reference/system-resources-reference/core-module-resources.md#gcpserviceaccount) resource in the `gcpServiceAccount` part, and name the bucket in `gcpBucket`. Objects land in that bucket, and the `location` sub-extension records them as `gs://{bucket}/{object-name}`. Offload uses the same `GcpServiceAccount` resource as the [GCP Cloud Storage](../../file-storage/gcp-cloud-storage.md) file storage integration. Pass `service-account-email` and `private-key` for explicit credentials, or create the resource without them so Aidbox falls back to [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials), which picks up Workload Identity in GKE and Cloud Run. ```http PUT /fhir/GcpServiceAccount/my-gcp-account Content-Type: application/json { "service-account-email": "storage-access@my-project.iam.gserviceaccount.com", "private-key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n" } ``` The service account needs read and write access to the objects in the bucket, including multipart uploads. `roles/storage.objectAdmin` on the bucket covers all of them. With explicit credentials, Aidbox takes the GCP project from `service-account-email`: the part between `@` and the first dot. With Application Default Credentials, the project comes from the runtime environment. For GCP, the offload parameter carries `gcpServiceAccount` and `gcpBucket` instead of `azureContainer`: ```json { "name": "dataOffloadToExternalStorage", "part": [ { "name": "fhirpathToBase64BinaryElement", "valueString": "data" }, { "name": "storageProvider", "valueCode": "gcp" }, { "name": "gcpServiceAccount", "valueReference": { "reference": "GcpServiceAccount/my-gcp-account" } }, { "name": "gcpBucket", "valueString": "my-bucket" } ] } ``` ## Example: offload Binary.data Create a storage for `Binary` and connect it to an API with offload enabled. See [Storages](README.md#storages) for the `$create-storage` step; the example below uses the `storageId` it returned. {% tabs %} {% tab title="Request" %} ```http POST /fhir/$create-api Content-Type: application/json { "resourceType": "Parameters", "parameter": [ { "name": "resourceType", "valueString": "Binary" }, { "name": "storageId", "valueString": "2791c25a-c28d-47ea-ab96-3e13162a5b58" }, { "name": "apiTemplate", "valueString": "pre-2604" }, { "name": "dataOffloadToExternalStorage", "part": [ { "name": "fhirpathToBase64BinaryElement", "valueString": "data" }, { "name": "storageProvider", "valueCode": "azure" }, { "name": "azureContainer", "valueReference": { "reference": "AzureContainer/my-container" } } ] } ] } ``` {% endtab %} {% tab title="Response" %} ```json { "resourceType": "Parameters", "parameter": [ { "name": "apiId", "valueString": "39473529-a37e-4b98-afc2-bea014bbe68e" }, { "name": "apiTemplate", "valueString": "pre-2604" }, { "name": "resourceType", "valueString": "Binary" }, { "name": "storageId", "valueString": "2791c25a-c28d-47ea-ab96-3e13162a5b58" } ] } ``` {% endtab %} {% endtabs %} Create a Binary with raw content. The response returns the resource as stored: `data` is absent and the offload extension holds the hash and the blob location. {% tabs %} {% tab title="Request" %} ```http POST /fhir/Binary Content-Type: application/octet-stream hello world ``` {% endtab %} {% tab title="Response" %} ```json { "resourceType": "Binary", "id": "b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02", "contentType": "application/octet-stream", "_data": { "extension": [ { "url": "http://health-samurai.io/fhir/core/data-offloaded-to-external-storage", "extension": [ { "url": "hash", "valueString": "Kq5sNclPz7QV2+lfQIuc6R7oRu0=" }, { "url": "location", "valueString": "azure://my-container/mystorageaccount/4f1f61a2-9e3b-4b0e-bb1d-6a1a1c2f7e58" } ] } ] } } ``` {% endtab %} {% endtabs %} Read the resource. Aidbox fetches the blob and returns the data inlined, without the offload extension: {% tabs %} {% tab title="Request" %} ```http GET /fhir/Binary/b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02 Accept: application/fhir+json ``` {% endtab %} {% tab title="Response" %} ```json { "resourceType": "Binary", "id": "b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02", "contentType": "application/octet-stream", "data": "aGVsbG8gd29ybGQ=" } ``` {% endtab %} {% endtabs %} Raw reads work the same way. When the `Accept` header contains the type stored in `Binary.contentType`, Aidbox serves the decoded bytes: {% tabs %} {% tab title="Request" %} ```http GET /fhir/Binary/b9f7a86e-16a5-45f5-8b1c-3e2a90c31c02 Accept: application/octet-stream ``` {% endtab %} {% tab title="Response" %} ``` hello world ``` {% endtab %} {% endtabs %} Starting from version 2608, Aidbox streams raw Binary uploads and raw reads between the client and the storage, so files larger than the memory available to the instance go through. The [`web.max-body`](../../reference/all-settings.md#web.max-body) setting still caps the upload size. See [Streaming large files](../../api/rest-api/other/binary.md#streaming-large-files). ## Example: offload Patient.photo.data Offload works for any resource type and any `base64Binary` element. This configuration offloads every `Patient.photo.data` value: ```json { "name": "dataOffloadToExternalStorage", "part": [ { "name": "fhirpathToBase64BinaryElement", "valueString": "photo.data" }, { "name": "storageProvider", "valueCode": "azure" }, { "name": "azureContainer", "valueReference": { "reference": "AzureContainer/my-container" } } ] } ``` Create a Patient with a photo. The response returns each photo's `data` replaced by the extension: {% tabs %} {% tab title="Request" %} ```http POST /fhir/Patient Content-Type: application/json { "resourceType": "Patient", "name": [{ "given": ["Amy"] }], "photo": [ { "contentType": "image/png", "data": "iVBORw0KGgoAAAANSUhEUgAA..." } ] } ``` {% endtab %} {% tab title="Response" %} ```json { "resourceType": "Patient", "id": "5f0c7e2a-8d31-4f5e-9b1a-2c3d4e5f6a7b", "name": [{ "given": ["Amy"] }], "photo": [ { "contentType": "image/png", "_data": { "extension": [ { "url": "http://health-samurai.io/fhir/core/data-offloaded-to-external-storage", "extension": [ { "url": "hash", "valueString": "L4pJTPTGwsyBb1TgAAmVGmPKmoc=" }, { "url": "location", "valueString": "azure://my-container/mystorageaccount/9d2c5a11-7b4f-4e0a-8f26-3c1d9e0b5a44" } ] } ] } } ] } ``` {% endtab %} {% endtabs %} `GET /fhir/Patient/5f0c7e2a-8d31-4f5e-9b1a-2c3d4e5f6a7b` returns the Patient with `photo[0].data` restored. ## Behavior and limitations * Offload runs on the FHIR REST create (`POST`) and update (`PUT`) interactions, and on the entries of transaction bundles. When a transaction fails, its blobs never become visible in the storage. Conditional update and PATCH store the data inline. * Aidbox restores data on instance read and version read. Search and history responses return the stored form: the element is absent and the extension holds the location. * When the upload to the external storage fails, the request fails with a `500` `OperationOutcome` and the resource is not written. When the download fails on read, the request fails with a `500` `OperationOutcome` as well. * A configured element whose value is not valid base64 fails the write with `422`. * Aidbox does not delete blobs. Deleting a resource leaves its blob in the storage, and updating a resource uploads a new blob while the old one stays. * On read, Aidbox resolves the storage from the current API configuration, and the blob name comes from the stored `location`. Repointing the API at a different container or bucket makes data offloaded through the old one unreadable. ## See also * [Storage and API configuration](README.md) * [Binary resource](../../api/rest-api/other/binary.md) * [Azure Blob Storage](../../file-storage/azure-blob-storage.md) * [AWS S3](../../file-storage/aws-s3.md) * [GCP Cloud Storage](../../file-storage/gcp-cloud-storage.md)