Pin API

Local Pinning

All commands under ipfs pin local

`add <ipfs-path> [--name=<name>] [--meta=<json>] [--type={recursive, direct}]`

Name is restricted to UTF-8 and <= 255 characters
- multiple pins can have the same name
Meta field can be implemented later
- ~~Should error if name is set in meta~~
Defaults to recursive pin

Returns

type PinOutput struct {
ID    string
Name  string
Cid   string
Type  string
Meta  string -- future iteration
}

Should we make add non-blocking by default and return some Status in all PinOutput objects?
- Status could be an integer for Progress as in the current ipfs pin add or could have more fields eg. {Status : {queued | pinning | pinned | failed}, BlocksDownloaded : int}

`ls [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--status=<status>] [--meta=<json match expr>] [--type={recursive | direct | recursive, direct}]`

Returns PinOutput for each pin listed
Must match all criteria
–name-match="contains" (i.e. case-insensitive, partial or full match) by default, other options for now are "exact"
- should exact also be case insensitive and do we need to add another option if so
  - Initial reaction – case insensitive all the way to minimize surprises with remote API
Meta field can be implemented later
- need some syntax for the JSON matching – open to recommendations
Type option
- Default to recursive, direct

`rm [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--meta=<json match expr>]`

Returns PinOutput for each pin removed
Must match all criteria
Meta field can be implemented later
May delete multiple pins at once
- Any need for –multiple (or –force) to allow deleting multiple pins at once?
  - Yes, require --force to delete more than one pin
  - Failure to pass this would result in an error like "tried to delete 33 pins, to delete them all repeat the command with –multiple", or actually listing all the pins to be deleted
- Any need to specify at least one criteria?
  - Yes, but it can be overriden with --force
  - ~~If no to the above then are we ok with ipfs pin local rm deleting all pins? Seems like an easy mistake for a user to make.~~
Should we allow --type={recursive, direct} here to allow removing all recursive or direct pins?
~~Note: there is no way to unpin CID X as well as all CIDs decended from X.~~
- ~~Is this really something people want? Seems pretty niche and counter to what we're trying to get people to do with pins.~~

Bonus `ipfs add`

Add --pin-name which errors if --pin=false
Modify AddEvent to have an extra field PinID string

Remote Pinning

`add <ipfs-path> [--name=<name>] [--meta=<json>] --service=<service>`

Name is restricted to UTF-8 and <= 255 characters
- multiple pins can have the same name
Meta field can be implemented later
- Should error if name is set in meta

Returns

type RemotePinOutput struct {
ID        string
Name      string
Cid       string
Status    string
Meta      string -- future iteration
Delegates []string
}

Add non-blocking by default
- Should we add a flag to make it block until it's done or failed?
Should delegates actually be returned to the user?
- If so should they be a separate field or just part of the Meta?

`ls [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--meta=<json match expr>] [--status=<status>] --service=<service>`

Returns RemotePinOutput for each pin listed
Must match all criteria
Meta field can be implemented later

`rm [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--meta=<json match expr>] [--status=<status>] --service=<service>`

Returns RemotePinOutput for each pin removed
Must match all criteria
Meta field can be implemented later
May delete multiple pins at once
- Any need for –multiple (or –force) to allow deleting multiple pins at once?
  - Failure to pass this would result in an error like "tried to delete 33 pins, to delete them all repeat the command with –multiple", or actually listing all the pins to be deleted
- Any need to specify at least one criteria?
- If no to the above then are we ok with ipfs pin local rm deleting all pins? Seems like an easy mistake for a user to make.

–format

All of the above commands could also take a flag --format=<format> so that only selective parts of the PinOutput are returned (e.g. only the name or only the CID).

This doesn't seem required for the first iteration.

Differences

The differences between local + remote in the two proposed APIs is:

remote always has --service
- if unifying APIs could reserve --service=local for local
remote only has recursive pins
- if unifying APIs could error when type is unsupported
remote only has name matching type "contains" (is also case-insensitive)
- we may want to select local behavior to plan towards unification (e.g. no case-sensitive matching)
remote returns delegates
- not sure if it should
- even so could potentially be unified under Meta.Delegates
  - Maybe we should update the pinning service API spec to forbid names like Name or Delegates as keys in Meta?
remote only matches meta exactly
- Question: in the Pinning Service API example does querying for meta={"app_id":"99986338-1113-4706-8302-4420da6158aa"} work when meta has multiple fields? i.e. is the query syntax here to do an exact match on set fields?

lidel

2020/10/12 10:39:50

- Defaults to recursive pin -

I don't think we should do any validation of data in meta. If someone wants to have a key named "name", let them (Edited)

Jacob Heun

2020/10/12 12:18:57

I agree

aschmahmann

2020/10/12 17:26:14

OK, I was concerned it might be confusing but fine by me.

2020/10/12 10:46:06

Thoughts on renaming "ID" to "PinID"? I feel would help to disambiguate it from "CID" (Pinning Services API has similar problem, it now uses "requestid" to avoid confusion with CID and to informa about what the "id" is) (Edited)

2020/10/12 15:25:31

PinID SGTM

Guest Steele2020/10/12 18:21:20

I actually like "ID" as that is that typical name of a unique identifier for _this_ data structure, and it avoids stutter (pin.PinID).

2020/10/12 10:57:56

make `add` non-blocking by default

Bit torn here. Non-blocking mode is useful for pinning over a remote HTTP API, but a chore on CLI or in general, when working on localhost. I'd say if we do this, there should be a flag that controls the blocking/non-blocking mode. Perhaps CLI could be blocking by default, and just do polling for pinning progress reporting etc. (Edited)

2020/10/12 11:00:54

should exact also be case insensitive

my assumption would be that "exact" is 1:1 same bytes, so "full match, case sensitive" (Edited)

2020/10/12 17:32:26

Fine by me, main concern is that remote pinning API doesn't support it so if someone uses a case-sensitive name they might run into trouble when using the remote API

2020/10/13 14:52:26

Do we actually need to add --name-match right now? If we leave it out they can unify on "contains" by default, and we can reevaluate "exact" in the future

2020/10/13 23:11:27

No I don't think we need to, although we'll need to document that `--name` means contains. @lidel WDYT? (Edited)

2020/10/12 11:06:54

need some syntax for the JSON matching -- open to recommendations

We could go with "no syntax". Just pass JSON object as a query, and API returns all pins that have meta objects that match passed key+value pairs (if value is undefined then return pins that have key match) Alternative is to implement graphQL-like queries with arguments https://graphql.org/learn/queries/, but i like the idea of just passing key+value more, because you don't need any special syntax. (Edited)

2020/10/12 11:52:00

I'd avoid complex syntax because that enables people to abuse pin storage and use it as a relational database. For example, Pinata's legacy "PinList" API supports queries like "?metadata[keyvalues]={"exampleKey":{"value":"2018-01-01 00:00:00.000+00","secondValue":"2018-02-01 00:00:00.000+00","op":"between"}}" which is an overkill IMO, makes scaling problematic and brings performance issues over time. (Edited)

2020/10/12 11:08:46

### `rm [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipf

I'd remove "all", just adds unnecessary noise. (Edited)

2020/10/12 12:47:40

2020/10/12 17:36:31

Especially since we're removing indirect it's totally unnecessary, just do recursive, direct.

2020/10/12 11:10:21

### `rm [--id=<pinid>] [--name=

I would remove it. Just like you said, if someone wants "indirectly pinned blocks" they can just "refs -r" If we only have "direct" and "recursive" types, it is much easier to reason about. (Edited)

2020/10/12 12:47:25

2020/10/12 17:35:49

+1 SGTM

Andrew Gillis

2020/10/12 18:39:50

2020/10/12 11:14:15

Any need to specify at least one criteria?

rm -rf / is impossible on Linux, one needs to pass --no-preserve-root flag. We should have similar failsafe (--force) (Edited)

2020/10/12 17:52:44

Anything more special e.g. `--allow-delete-all` or is using the same `--force` we allow for deleting multiple pins enough?

2020/10/12 20:24:50

I think --force is sufficient and has same meaning.

2020/10/13 14:58:09

Agreed, +1 to just using --force

2020/10/12 11:15:25

--force

I think asking user to add "--force" if they really know what they are doing sounds good. This convention is present in a lot of unix tools. (Edited)

2020/10/12 12:51:17

Agreed, I think `--force` would be good to have

2020/10/12 11:17:43

Should we allow `--type={recursive, direct}` here to allow removing all recursive or direct pins?

does not sound like very useful, can always be implemented on top of "pin local ls --type" and "pin local rm" (Edited)

2020/10/12 12:53:27

+1, I'd leave the type out for now

2020/10/12 11:18:21

y to unpin CID `X` as well as all CIDs decended from `X`.

IMO not a problem: one can build one-liner that does this with "refs -r" and then calls "pin rm" for each block (Edited)

2020/10/12 12:54:07

agreed

2020/10/12 11:21:59

Should we add a flag to make it block until it's done or failed?

I think it would be really useful. It could even be the default mode in CLI running in interactive mode. The caveat here is to come up with behavior that also works in "local" namespace. Would be nice if both have nearly identical ergonomics. (Edited)

2020/10/12 13:33:16

To keep this consistent with the local API having the default be blocking would be ideal I think, although this would not be great for IPFS internals having to poll the remote API.

2020/10/15 16:39:49

I think it's not too bad on the internals, and seems like a sensible default.

2020/10/12 11:29:22

Should delegates actually be returned to the user? - If so should they be a separate field or just part of the Meta?

I believe we should keep Delegates as a separate field, avoid mixing with meta because if someone reads specs of both APIs it will fuse their brain :)) The knowledge of who is providing pinned data will be useful hint and inspiration for app builders and we should include it in the MVP. (Edited)

2020/10/12 11:30:08

(see my comment in local section) (Edited)

2020/10/12 11:34:06

Counterpoint: if we include "--format" in the very first version of "ls" commands, people who don't like the default behaviour will be less vocal because they can always customize the output :^) (Edited)

2020/10/12 11:40:06

even so could potentially be unified under Meta.Delegates - Maybe we should update the pinning service API spec to forbid names like Name or Delegates as keys in Meta?

I tried going this way with pinning service api, but it balloons the spec with rules and additional error handling that needs to be documented for consistent behavior. keeping Meta separate, to be "just a dictionary for additional user-provided metadata" is way safer (Edited)

2020/10/12 11:44:08

is the query syntax here to do an exact match on set fields?

Yes. I believe the idea is to pass {key:value} for exact key+value match or something like {key: null} to match only by key (Edited)

2020/10/12 11:46:12

And this should work when matched object has more keys, so so query {key1:val1} will match {key1:val1, key2,val2} etc

2020/10/12 12:28:48

I think it should block by default and we could add a flag to make it non-blocking. This would keep the default behavior consistent with `ipfs add`. If work needs to happen for me to get the pinned content, I'd expect it to show me that progress by default, and if I don't care about it running in the background I can pass it a flag. (Edited)

2020/10/12 12:46:59

recursive

Default to recursive, I think this is the more common use case. (Edited)

2020/10/12 18:39:16

I would expect `ipfs pin ls` to list recursive and direct pins, if I had any direct. As long as it does not list indirect pins. The common use case is "recursive" because direct pins are rare.

2020/10/13 14:55:20

Yes, sorry, I read this wrong the first time. ls should return recursive and direct pins by default.

2020/10/15 17:24:34

+1 for listing recursive+direct

2020/10/12 13:36:29

Exposing the delegates to the users doesn't really give them anything does it? IPFS should internally be connecting to those nodes. I don't think there's any harm in showing it to the user, but it's really not meaningful to them. (Edited)

2020/10/12 13:37:52

Yeah, definitely not required but could save us a lot of back and forth on issues/discuss/etc. (Edited)

2020/10/12 15:32:50

Would love to put this in too, just don't want us to necessarily block on it. (Edited)

2020/10/12 17:28:39

Sounds good to do blocking by default. Two questions though. 1) Flag name. `--blocking=true`, `--background=true`, something else? (Edited)

2020/10/12 17:31:18

2) Status object. Right now `ipfs pin --progress` utilizes `Progress int` which is really just BlocksDownloaded. The remote pinning API doesn't really give any info other than "in progress" does just throwing both into the same object seem reasonable (i.e. Status + BlocksDownloaded + any other stat we come up with). Would it be reasonable for users to care about the metric for downloading? e.g. if for UnixFS we estimated percentage of file downloaded, but for IPLD just went with blocks. (Edited)

2020/10/12 17:51:30

"tried to delete 33 pins, to delete them all repeat the command with --multiple", or actually listing all the pins to be deleted

Thoughts on what we output here? Perhaps "tried to delete X pins, including ..." where we list some number of those pins (e.g. 10) (Edited)

2020/10/15 17:32:24

Would --id support passing more than value here? "This operation would remove X pins and requires additional confirmation. To proceed either pass multiple pin IDs in --id=1,2,3 or repeat the command with --force" (Edited)

2020/10/12 20:19:32

Must match all criteria

Not entirely clear here: This sounds like the criteria are logically AND'ed. Does that mean there is no point in providing any other criteria if `--id` or `--cid` is specified? Or, can I specify the ID of one pin and select a different pin using a name? (Edited)

2020/10/13 23:16:02

Yes, the criteria is AND. If you're using `--id` there's probably no need for anything else (maybe we should even error if `--id` + anything else is set? (WDYT @lidel ) There may be multiple pins for a given CID so `--cid` is only one field to select on.

2020/10/15 17:28:50

Even tho "ID" will be globally unique, I think passing "--id foo --name bar" is still a valid query that we should support (easy way to check if this specific pin has specific name, or in the future, metadata)

2020/10/12 20:22:33

To delete multiple pins, specify individual pin IDs or use --force to delete all pins matching specification (Edited)

2020/10/12 20:26:19

Agree. If I already know how to identify the pin, then I probably already know its type. (Edited)

2020/10/12 20:28:31

Seems like that leads to wanting a `rm -r`. Maybe I am wrong? (Edited)

2020/10/12 20:36:42

Add `--pin-name` which errors if `--pin=false

Is the purpose of this to name an existing ping that does not yet have a name? Can it be used to change the name of an existing pin? What does --pin=true/false do? (Edited)

2020/10/13 16:00:17

ipfs add doesn't pin by default, so you can pass the --pin flag to do so.

2020/10/13 16:03:47

That's an interesting question. While I don't think the recommended use case would be to do `ipfs add cid1 --pin` and then `ipfs add cid1 --pin-name coolname --pin` for the same cid, it should probably update the pins name. This could have some implications to ref counting on pins.

2020/10/15 17:39:32

Right now `ipfs add` DOES pin by default (implicit `--pin=true`) so only `--pin-name` would be passed by end user. I believe this implicit pin during `ipfs add` should always create a new Pin, what we add here is option to specify `name`, that is all. It should not do anything fancy, like modifying existing pins. (Edited)

2020/10/13 14:48:13

1) hmmm, maybe --background. If the output isn't needed people could just dump it to /dev/null, but that gets challenging if we need the Pin but also want to not block (Edited)

2020/10/13 14:50:20

2) I think the whole status object is fine. In the future maybe remote APIs will give some of this data back in the meta, but for now we should probably document that progress on remote services is not going to be meaningful outside of the status. (Edited)

2020/10/13 23:22:12

I'm ok leaving it out, but figured we may want to make the `rm` and `ls` options the same. Technically we can only have `rm --id=<pinid>` if we wanted. Note, doing `ipfs pin local --type=recursive | ipfs pin rm` might not be so bad locally but is many HTTP API calls which from what I recall from @lidel is a problem for browsers (Edited)

2020/10/15 17:34:50

Ack, but we can always add it later, for now I'd keep API pretty basic.

2020/10/13 23:22:28

Sold. If users start complaining we can always add in something else (Edited)

2020/10/15 16:38:22

Talked with @lidel about this a bit and I think we do need `--name-match` now because "contains" is O(N) on the datastore which might not be great for users with lots of pins. We'd still leave the default as "contains" because this seems to be the user preference expressed to pinning services such as Pinata in the past (Edited)

2020/10/15 17:23:00

When it comes to ergonomics, we could go with "--name-match contains|exact" "contains" is partial, case-insensitive match (used by default to maximize UX). alternative names to consider: "fuzzy", "relaxed", "partial" "exact" is full, case-sensitive match (when 1:1 match and speed are required)

2020/10/15 16:43:35

Adding it in means having some discrepancy between the local and remote returned outputs. It's not the end of the world, but might be nice to avoid if possible. I'm also not sure it's guaranteed that the delegates will actually keep the data (e.g. a pinning service could have a set of nodes that downloads a ton of data and then redistributes it internally to appropriate nodes). Perhaps I'm missing something but it doesn't seem super useful to expose the delegates. (Edited)

2020/10/15 17:46:29

My thinking was that information about Delegates could be used in apps for opportunistically pre-connecting to a know Provider when requested CID does not exist in local datastore (so its not just helping with transfer during pinning, but also when reading data back later). That being said, agree that we can always add it later if needed, so it can be dropped from MVP for the sake of simplicity and keeping the same struct in both local and remote.

Pin API

Local Pinning

add <ipfs-path> [--name=<name>] [--meta=<json>] [--type={recursive, direct}]

ls [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--status=<status>] [--meta=<json match expr>] [--type={recursive | direct | recursive, direct}]

rm [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--meta=<json match expr>]

Bonus ipfs add

Remote Pinning

add <ipfs-path> [--name=<name>] [--meta=<json>] --service=<service>

ls [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--meta=<json match expr>] [--status=<status>] --service=<service>

rm [--id=<pinid>] [--name=<name> --name-match=<match-type>] [--cid=<ipfs-path>] [--meta=<json match expr>] [--status=<status>] --service=<service>

–format

Differences

Read more

Docker IPFS Proxy

IPLD and Compatiblity

WASM IPLD Codecs and ADLs

BitTorrent as IPLD