FastFS Upload Format Documentation

FastFS is a decentralized file storage system built on the NEAR blockchain that allows users to upload and serve files through a standardized protocol. Files are stored on-chain and served through a web interface with predictable URLs.

Related Repositories

• FastFS Server: https://github.com/fastnear/fastfs-server
• FastData Indexer: https://github.com/fastnear/fastdata-indexer
• Drag & Drop Interface: https://github.com/fastnear/fastdata-drag-and-drop

Overview

FastFS uses NEAR blockchain transactions to store file data. When a file is uploaded, it becomes accessible via a URL that combines the transaction's predecessor ID, receiver ID, and the file's relative path.

URL Structure

Files uploaded to FastFS are accessible via URLs with the following format:

https://{predecessor_id}.fastfs.io/{receiver_id}/{relative_path}

Example:

• Transaction: https://nearblocks.io/txns/5vUqtNhYchWyrifviVZ9F6k4QprgfW9fjSAtcGMdQyRw
• Resulting URL: https://mob.near.fastfs.io/fastfs.near/fastnear.png
  • mob.near = predecessor_id (uploader)
  • fastfs.near = receiver_id (can be any account)
  • fastnear.png = relative_path (filename)

Transaction Requirements

Method Name

All FastFS uploads must use the function call method:

__fastdata_fastfs

Data Format

Arguments must be serialized using the Borsh format with the following schema structure:

Borsh Schema

// Rust representation
#[derive(Clone, BorshSerialize, BorshDeserialize)]
pub enum FastfsData {
    Simple(Box<SimpleFastfs>),
}

#[derive(Clone, BorshSerialize, BorshDeserialize)]
pub struct SimpleFastfs {
    pub relative_path: String,
    pub content: Option<FastfsFileContent>,
}

#[derive(Debug, Clone, BorshSerialize, BorshDeserialize)]
pub struct FastfsFileContent {
    pub mime_type: String,
    pub content: Vec<u8>,
}

JavaScript Schema

const FastfsSchema = {
  FastfsFileContent: {
    struct: {
      mimeType: "string",
      content: { array: { type: "u8" } },
    },
  },
  SimpleFastfs: {
    struct: {
      relativePath: "string",
      content: { option: "FastfsFileContent" },
    },
  },
  FastfsData: {
    enum: [{ struct: { simple: "SimpleFastfs" } }],
  },
};

Upload Process

1. Prepare File Data

Create a FastFS data structure:

const fastfsData = {
  simple: {
    relativePath: "path/to/your/file.png",
    content: {
      mimeType: "image/png",
      content: new Uint8Array(fileBuffer) // Your file as bytes
    }
  }
};

2. Serialize Data

Use Borsh serialization to encode the data:

import { serialize as borshSerialize } from "borsh";

function encodeFfs(ffs) {
  return near.utils.bytesToBase64(borshSerialize(FastfsSchema.FastfsData, ffs));
}

const serializedData = encodeFfs(fastfsData);

3. Submit Transaction

Send a NEAR transaction with:

• Method: __fastdata_fastfs
• Arguments: The serialized data from step 2
• Receiver: The contract account that will serve as the namespace

Validation Rules

The FastFS system enforces the following validation rules:

Path Validation

• Maximum relative path length: 1,024 characters
• Path should be a valid relative file path

Content Validation

• MIME type: Required and cannot be empty
• Content: Binary file data as byte array
• Content field: Optional (can be null for file deletion)

Validation Function

impl SimpleFastfs {
    pub fn is_valid(&self) -> bool {
        if self.relative_path.len() > MAX_RELATIVE_PATH_LENGTH {
            return false;
        }
        if let Some(content) = &self.content {
            if content.mime_type.is_empty() {
                return false;
            }
        }
        true
    }
}

Example Implementation

Here's a complete example of uploading a file to FastFS:

import { serialize as borshSerialize } from "borsh";

// Define the schema
const FastfsSchema = {
  FastfsFileContent: {
    struct: {
      mimeType: "string",
      content: { array: { type: "u8" } },
    },
  },
  SimpleFastfs: {
    struct: {
      relativePath: "string",
      content: { option: "FastfsFileContent" },
    },
  },
  FastfsData: {
    enum: [{ struct: { simple: "SimpleFastfs" } }],
  },
};

// Prepare the file data
const fileData = {
  simple: {
    relativePath: "images/logo.png",
    content: {
      mimeType: "image/png",
      content: new Uint8Array(pngFileBuffer)
    }
  }
};

// Serialize and upload
const serializedData = borshSerialize(FastfsSchema.FastfsData, fileData);
const base64Data = near.utils.bytesToBase64(serializedData);

// Submit transaction
await near.functionCall({
  contractId: "fastfs.near",
  methodName: "__fastdata_fastfs",
  args: base64Data,
  gas: "300000000000000", // 300 TGas
  attachedDeposit: "0"
});

Storage and Indexing

The FastFS indexer processes transactions and stores the following data in ScyllaDB:

• Transaction metadata: Receipt ID, block height, timestamp
• Account information: Predecessor ID, receiver ID, signer ID
• File data: Relative path, MIME type, content bytes
• Indexing information: Shard ID, receipt index, action index

Access Patterns

Direct File Access

GET https://{predecessor_id}.fastfs.io/{receiver_id}/{relative_path}

File Deletion

Files can be removed by uploading without content by setting the content field to null:

const deleteFile = {
  simple: {
    relativePath: "path/to/file.json",
    content: null
  }
};

This will remove the file from the FastFS system.

Error Handling

Common validation failures:

• Path too long: Relative path exceeds 1,024 characters
• Empty MIME type: MIME type is required when content is provided
• Invalid Borsh serialization: Data doesn't match the expected schema
• Transaction failure: Insufficient gas or other NEAR transaction errors

Best Practices

1. File Organization: Use meaningful directory structures in relative paths
2. MIME Types: Always specify accurate MIME types for proper content serving
3. File Size: Be mindful of NEAR transaction size limits
4. Gas Estimation: Larger files require more gas for transaction processing
5. Batch Uploads: For multiple files, consider separate transactions to avoid gas limits

Integration Examples

Web Application Upload

async function uploadToFastFS(file, relativePath) {
  const fileBuffer = await file.arrayBuffer();
  const fastfsData = {
    simple: {
      relativePath: relativePath,
      content: {
        mimeType: file.type,
        content: new Uint8Array(fileBuffer)
      }
    }
  };
  
  const serialized = borshSerialize(FastfsSchema.FastfsData, fastfsData);
  // Submit transaction...
}

Drag and Drop Interface

For easy file uploads, you can use the web-based drag and drop interface:

• Repository: https://github.com/fastnear/fastdata-drag-and-drop

This documentation provides a complete guide for implementing FastFS file uploads. The system provides a decentralized, blockchain-based file storage solution with predictable URL patterns and robust validation.