Storage Backends#

Storage Backend classes are responsible for managing and interacting with the system that handles actual file storage, be it a local file system or a remote, 3rd party cloud based storage.

Storage Adapters can implement one or more of several interfaces, which defines the capabilities provided by the backend, and which transfer adapters the backend can be used with.

Types of Storage Backends#

Each storage backend adapter can implement one or more of the following interfaces:

  • StreamingStorage - provides APIs for streaming object upload / download through the Giftless HTTP server. Works with the basic_streaming transfer adapter.

  • ExternalStorage - provides APIs for referring clients to upload / download objects using an external HTTP server. Works with the basic_external transfer adapter. Typically, these backends interact with Cloud Storage providers.

  • MultipartStorage - provides APIs supporting the special multipart-basic transfer mode. Typically, these backends interact with Cloud Storage providers.

  • VerifiableStorage - provides API for verifying that an object was uploaded properly. Most concrete storage adapters implement this interface.

Configuring the Storage Backend#

Storage backend configuration is provided as part of the configuration of each Transfer Adapter. For example:

TRANSFER_ADAPTERS:
  basic:  # <- the name of the transfer mode, you can have more than one
    factory: giftless.transfer.basic_external:factory
    options:
      storage_class: giftless.storage.google_cloud:GoogleCloudStorage
      storage_options:
      
    # add an example here

Built-In Storage Backends#

Microsoft Azure Blob Storage#

giftless.storage.azure:AzureBlobStorage#

Modify your giftless.yaml file according to the following config:

    $ cat giftless.yaml

    TRANSFER_ADAPTERS:
      basic:
        factory: giftless.transfer.basic_external:factory
        options:
          storage_class: ..storage.azure:AzureBlobsStorage
          storage_options:
            connection_string: GetYourAzureConnectionStringAndPutItHere==
            container_name: lfs-storage
            path_prefix: large-files

Google Cloud Storage#

giftless.storage.google_cloud:GoogleCloudStorage#

To use Google Cloud Storage as a backend, you’ll first need:

  • A Google Cloud Storage bucket to store objects in

  • an account key JSON file (see here).

The key must be associated with either a user or a service account, and should have read / write permissions on objects in the bucket.

If you plan to access objects from a browser, your bucket needs to have CORS enabled.

You can deploy the account key JSON file and provide the path to it as the account_key_file storage option:

TRANSFER_ADAPTERS:
  basic:
    factory: giftless.transfer.basic_streaming:factory
    options:
      storage_class: giftless.storage.google_cloud:GoogleCloudStorage
      storage_options:
        project_name: my-gcp-project
        bucket_name: git-lfs
        account_key_file: /path/to/credentials.json

Alternatively, you can base64-encode the contents of the JSON file and provide it inline as account_key_base64:

TRANSFER_ADAPTERS:
  basic:
    factory: giftless.transfer.basic_streaming:factory
    options:
      storage_class: giftless.storage.google_cloud:GoogleCloudStorage
      storage_options:
        project_name: my-gcp-project
        bucket_name: git-lfs
        account_key_base64: S0m3B4se64RandomStuff.....ThatI5Redac7edHeReF0rRead4b1lity==

Amazon S3 Storage#

giftless.storage.amazon_s3:AmazonS3Storage#

Modify your giftless.yaml file according to the following config:

    $ cat giftless.yaml

    TRANSFER_ADAPTERS:
      basic:
        factory: giftless.transfer.basic_external:factory
        options:
          storage_class: giftless.storage.amazon_s3:AmazonS3Storage
          storage_options:
            bucket_name: bucket-name
            path_prefix: optional_prefix

boto3 authentication#

AwsS3Storage supports 3 ways of authentication defined in more detail in docs:

  1. Environment variables

  2. Shared credential file (~/.aws/credentials)

  3. AWS config file (~/.aws/config)

  4. Instance metadata service on an Amazon EC2 instance that has an IAM role configured (usually used in production).

Running updated yaml config with uWSGI#

After configuring your giftless.yaml file, export it:

$ export GIFTLESS_CONFIG_FILE=giftless.yaml

You will need uWSGI running. Install it with your preferred package manager. Here is an example of how to run it:

    # Run uWSGI in HTTP mode on port 8080
    $ uwsgi -M -T --threads 2 -p 2 --manage-script-name \
        --module giftless.wsgi_entrypoint --callable app --http 127.0.0.1:8080

Notes#

  • If you plan to access objects directly from a browser (e.g. using a JavaScript based Git LFS client library), your GCS bucket needs to be CORS enabled.

Local Filesystem Storage#

giftless.storage.local:LocalStorage#

TBD