In CouchDB mem3_sync is the mechanism that keeps shards of the same range synchronised across multiple cluster nodes. It does so by implementing the CouchDB replication protocol over Erlang distributed messages. In slightly more detail this means it reads the by-seq index of a shard and copies out all live content into the new shard.

During cluster management operations like increasing the number of nodes it can become necessary to move an existing shard from one node (A) to another (B) where there is no such shard on node B already, because it just freshly joined the cluster and no database shard map points to it.

The easiest way to move a shard to node B is to edit the shard map for a respective database and change the entries for a particular shard range or ranges to point to node B instead of node A. mem3 will recognise the change in the shard map and notice the missing shard on node B and then will initiate mem3_sync to fill up the shard to match the copies on the other cluster nodes.

Unfortunately, implementing CouchDB replication means that data transfer does not happen at network speed because following the protocol correctly means multiple request/response cycles during each batch of operation.

However, when replicating into an empty shard, none of these extra roundtrips are strictly necessary. They are necessary later when a shard might be slightly behind other nodes so that the shard can be caught up correctly, but for the initial sync, mem3_sync does more work than strictly needed.

A common recommendation for speeding up this process, especially for large databases, is to first scp or rsync the .couch shard file to the new host and then let mem3_sync top it up.

It would be a lot nicer if CouchDB chose a faster binary-copy on its own when it knows it is replicating into an empty shard file.

Proposal

We propose the following algorithm to speed up the process:

• when mem3_sync is invoked: check if the target shard exists on node B.
• if yes: use the remaining functionality of mem3_sync nothing changes.
• if no:
  • create a hard link of the source shard:
    • dbname.timestamp.couch.hardlink.file-length, e.g. db.1234567890.couch.hardlink.7865432
    • ensure that compaction cannot occur between recording the file length and calling ln. it should be in the couch_file gen_server loop therefore.
    • if the hardlink operation fails, fall back to normal mem3_sync functionality, log a warning about using an odd file system
• create a target shard to copy data into with the name of the shard file plus a .initial suffix.
  • if the suffixed file already exists on node B, create a sha256 checksum of it and the hard link up to the file length recorded in the hard link name. If the checksum matches, continue copying bytes from length(target) to length(source) (as recorded at the beginning of this), do not read until EOF.
  • if not: start copying bytes from position 0 in the source file on node A until the length reached that is specified in the hard link name
• when done
  • remove hard link on node A
  • optional: do another sha256 of the final file and compare against source
  • continue with regular mem3_sync functionality to top up the shard file
  • then rename the file to remove the .initial suffix

TODO:

• decide when/how to clean up stale .hardlink and .initial files.