gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
Page: Filer Metadata Events
Pages
AWS CLI with SeaweedFS AWS IAM CLI Actual Users Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Components Configure Remote Storage Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Directories and Files Docker Image Registry with SeaweedFS Environment Variables Erasure Coding for warm storage FAQ FIO benchmark FUSE Mount Failover Master Server Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Home Independent Benchmarks Kubernetes Backups and Recovery with K8up Large File Handling Load Command Line Options from a file Master Server API Migrate to Filer Store Mount Remote Storage Optimization Path Specific Configuration Path Specific Filer Store Production Setup Replication Run Blob Storage on Public Internet Run Presto on SeaweedFS S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Quota S3 Nginx Proxy SRV Service Discovery SeaweedFS Java Client SeaweedFS in Docker Swarm Security Configuration Security Overview Server Startup Setup Store file with a Time To Live Super Large Directories System Metrics TensorFlow with SeaweedFS Tiered Storage UrBackup with SeaweedFS Use Cases Volume Management Volume Server API WebDAV Words from SeaweedFS Users nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS weed shell
4 Filer Metadata Events
chrislu edited this page 2022-07-29 01:40:03 -07:00
Table of Contents

On filer, there is a /topics/.system/log folder, it stores all filer metadata change events.

Metadata Event Format

The events are stored in files organized by timestamp, yyyy-MM-dd/hh-mm.segment.

The events are encoded by protobuf, defined in https://github.com/seaweedfs/seaweedfs/blob/master/weed/pb/filer.proto . The related sections are:

service SeaweedFiler {
    rpc SubscribeMetadata (SubscribeMetadataRequest) returns (stream SubscribeMetadataResponse) {
    }
}

message SubscribeMetadataRequest {
    string client_name = 1;
    string path_prefix = 2;
    int64 since_ns = 3;
}
message SubscribeMetadataResponse {
    string directory = 1;
    EventNotification event_notification = 2;
    int64 ts_ns = 3;
}
message EventNotification {
    Entry old_entry = 1;
    Entry new_entry = 2;
    bool delete_chunks = 3;
    string new_parent_path = 4;
}
message LogEntry {
    int64 ts_ns = 1;
    int32 partition_key_hash = 2;
    bytes data = 3;
}

The ondisk file is a repeated bytes of the following format:

  1. 4 bytes of LogEntry size
  2. serialized LogEntry

The LogEntry.data stores serialized SubscribeMetadataResponse

Read Metadata Events

The events can be read by any program as files. One example is here: https://github.com/seaweedfs/seaweedfs/blob/master/unmaintained/see_log_entry/see_log_entry.go

Subscribe to Metadata

The protobuf definition also defined a RPC call, where you can subscribe to all metadata changes asynchronously. This is how the weed mount can asynchronously get metadata updates from the filer, and avoid cross-wire metadata read operations to dramatically cut down operation latency.

service SeaweedFiler {
    rpc SubscribeMetadata (SubscribeMetadataRequest) returns (stream SubscribeMetadataResponse) {
    }
}

message SubscribeMetadataRequest {
    string client_name = 1;
    string path_prefix = 2;
    int64 since_ns = 3;
}
message SubscribeMetadataResponse {
    string directory = 1;
    EventNotification event_notification = 2;
    int64 ts_ns = 3;
}

This is similar to inotify in normal file system, but actually much more powerful. Usually inotify can only monitor one directory and its direct children, but SeaweedFS metadata subscription can monitor one directory and all its sub-directories and files.

This API also allows you to read from any point of time and replay all metadata events.

This API could be useful to build powerful event driven data processing pipelines. Please let me know if you have some great ideas or accomplishment!

What kind of events it contains?

The SubscribeMetadataResponse contains EventNotification, which contains old_entry and new_entry. The following events can be derived:

  • Create Event: new_entry is not null, and old_entry is null.
  • Delete Event: old_entry is not null, and new_entry is null.
  • Update Event: old_entry is not null, and new_entry is not null.
  • Rename Event: similar to Update Event, but also need SubscribeMetadataResponse.directory and EventNotification.new_parent_path.

Purging Metadata Logs

It is ok to delete the metadata logs. However, many features rely on these metadata events, such as:

  • Async filer replication
  • Sync metadata with weed mount
  • Async filer metadata backup
  • Filer backup

So better not to mess with the metadata logs, which should not take too much space. Just leave them there.

Introduction

API

Configuration

Filer

Filer Stores

Advanced Filer Configurations

FUSE Mount

WebDAV

Cloud Drive

AWS S3 API

AWS IAM

Machine Learning

HDFS

Replication and Backup

Messaging

Use Cases

Operations

Advanced

Security

Misc Use Case Examples