8 Configure Remote Storage

Xiangxuan Qu edited this page 2023-05-24 23:25:51 +09:00

Table of Contents

• Supported Vendors
• Configure Remote Storage

Supported Vendors

Here are a list of vendors supported. For other S3 compatible vendors, you can use just s3, but usually you may need to tweak these parameters:

remote.configure -name=cloud1 -type=s3 -s3.access_key=xxx -s3.secret_key=yyy -s3.endpoint=zzzzz -s3.force_path_style=true -s3.v4_signature=false

Name	Type	Note	Reference
AWS S3	s3
Google Cloud Storage	gcs		https://cloud.google.com/docs/authentication/getting-started
Azure Blob Storage	azure		https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage
BackBlaze	b2	Endpoint format: s3.[region]. backblazeb2.com	https://help.backblaze.com/hc/en-us/articles/360047425453-Getting-Started-with-the-S3-Compatible-API
Wasabi	wasabi	Endpoint format: s3.[region].wasabisys.com	https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-storage-regions-
Storj	storj
Filebase	filebase
Aliyun 对象存储 OSS	aliyun	Endpoint format: oss-[region]-aliyuncs.com	https://help.aliyun.com/document_detail/31837.htm
Tencent 对象存储 COS	tencent	Endpoint format: cos.[region].myqcloud.com	https://cloud.tencent.com/document/product/436/6224
Baidu 对象存储 BOS	baidu	Endpoint format: s3.[region].bcebos.com	https://cloud.baidu.com/doc/BOS/s/ojwvyq973

Configure Remote Storage

Remote Storage should be configured first before being used.

Run remote.configure in weed shell:

> help remote.configure
  remote.configure	# remote storage configuration

	# see the current configurations
	remote.configure

	# set or update a configuration
        # storage class default value is "STANDARD_IA"
	remote.configure -name=cloud1 -type=s3 -s3.access_key=xxx -s3.secret_key=yyy -s3.storage_class="STANDARD"
	remote.configure -name=cloud2 -type=gcs -gcs.appCredentialsFile=~/service-account-file.json
	remote.configure -name=cloud3 -type=azure -azure.account_name=xxx -azure.account_key=yyy

	# delete one configuration
	remote.configure -delete -name=cloud1

In the above example, a remote storage is configured to name cloud1, which is the name to reference files stored on this account.

The access key and secret key can be changed on the fly.

Introduction

• Components
• Getting Started
• Production Setup
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• Server Startup Setup
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture

FUSE Mount

• FIO benchmark

WebDAV

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 API Audit log
• S3 Nginx Proxy

AWS IAM

• Amazon IAM API
• AWS IAM CLI

Machine Learning

• TensorFlow with SeaweedFS

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• run Presto on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Messaging

• Filer Metadata Events

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube

Advanced

• Large File Handling
• Optimization
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery

Security

• Security Overview
• Security Configuration
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware