Future-Proofing the Source Hydrator API

Multi-Source Support

Proposal 1: sourceHydrator.drySources

NOTE: This is my current recommendation.

Pros:

• Parallel to source/sources UX
• Parallel to source/sources implementation

Cons:

• Two fields, gotta validate that one and only one is present (similar to regular multi-source)

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      targetRevision: HEAD
      path: helm-guestbook
    drySources:
    - repoURL: https://github.com/argoproj/argocd-example-apps
      targetRevision: HEAD
      path: helm-guestbook
    - repoURL: https://github.com/argoproj/argocd-example-apps
      targetRevision: HEAD
      path: external-secrets

Proposal 2: sourceHydrator.drySource.additionalSources

Pros:

• No need for drySource/drySources mutual exclusivity validation

Cons:

• Awkward apparent primary/secondary relationship between top-level drySource and nested additionalSources
• Awkward lack of parallelism between additionalSources and source modifiers like helm and kustomize
• Awkward apparent relationship between top level modifiers (like drySource.helm) and nested modifiers (drySource.additionalSources[0].helm); unclear whether the former config applies to the latter sources

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      targetRevision: HEAD
      path: helm-guestbook
      additionalSources:
      - repoURL: https://github.com/argoproj/argocd-example-apps
        targetRevision: HEAD
        path: external-secrets

Proposal 3: source.hydrator/sources[].hydrator with Nested Dry Source

Pros:

• No new field or validation/fallback requirements de-conflicting with source and sources
• Somewhat easier adoption, because you can add to and update an existing field instead of using a completely new one

Cons:

• The fact that the top-level config is the "hydrated config" is not explicit in the API field names; only the "dry" config is explicitly named as such
• User has to move the top level targetRevision and path into the drySource
• targetRevision field for hydrated source is misleading because it actually must be a branch
• Existence of chart, helm, and kustomize fields under sources[0] is misleading, because none of those is actually valid when using the hydrator
• The presence of the directory field breaks rendered-manifests guarantees: you can filter after hydration, which means the hydrated state isn't necessarily the synced state.
• The presence of the directory field is weird, because there isn't really a valid use case
• The lack of parallelism in the source fields' behavior is awkward: helm, kustomize, and directory are hydration modifiers, but hydrator introduces completely out-of-band behavior
• It's unclear how the hydrator should behave when two sources' configs match vs. differ
  • If the hydration destination branch is the same, do we concatenate the output of all sources into one file?
  • If the hydration destination branches differ, is that an error? Or do we just asynchronously hydrate to the different destinations?
• If differently-destined are allowed, there's a risk of races when syncing: if we've hydrated and pushed source 1, we might sync that before we finish hydrating and pushing source 2. To prevent this race, we'd need a locking mechanism shared by the synchronization and hydration code.
• In general, this mixes hydration and synchronization concerns. We currently use the ApplicationSource field to determine whether the configured source has changed since the last sync. If we embed hydration config, the synchronization code will need special comparison logic to avoid considering hydration fields when comparing the last sync to the current config.

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  sources:
  - repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: environments/e2e
    path: guestbook
    hydrator:
      drySource:
        targetRevision: HEAD
        path: helm-guestbook
      hydrateTo:
        targetBranch: environments/e2e-next
  - repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: environments/e2e
    path: guestbook
    hydrator:
      drySource:
        targetRevision: HEAD
        path: external-secrets
      hydrateTo:
        targetBranch: environments/e2e-next

Proposal 4: source.hydrator/sources[].hydrator with Nested Hydrated Source

Pros:

• No new field or validation/fallback requirements de-conflicting with source and sources
• Somewhat easier adoption, because you can add to an existing field instead of using a completely new one
  • It's even easier than Proposal 3, because you don't have to change the top-level (dry) source config

Cons:

• The fact that the top-level config is the "dry config" is not explicit in the API field names; only the "sync" (hydrated) config is explicitly named as such
• targetRevision field for hydrated source is misleading because it actually must be a branch
• Existence of chart, helm, kustomize, and directory fields under sources[0] is misleading, those fields will not be supported by the hydrator in the initial implementation
• The lack of parallelism in the source fields' behavior is awkward: helm, kustomize, and directory are hydration modifiers, but hydrator introduces completely out-of-band behavior
• It's unclear how the hydrator should behave when two sources' configs match vs. differ
  • If the hydration destination branch is the same, do we concatenate the output of all sources into one file?
  • If the hydration destination branches differ, is that an error? Or do we just asynchronously hydrate to the different destinations?
• If differently-destined are allowed, there's a risk of races when syncing: if we've hydrated and pushed source 1, we might sync that before we finish hydrating and pushing source 2. To prevent this race, we'd need a locking mechanism shared by the synchronization and hydration code.
• In general, this mixes hydration and synchronization concerns. We currently use the ApplicationSource field to determine whether the configured source has changed since the last sync. If we embed hydration config, the synchronization code will need special comparison logic to avoid considering hydration fields when comparing the last sync to the current config.

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  sources:
  - repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD
    path: helm-guestbook
    hydrator:
      syncSource:
        targetRevision: environments/e2e
        path: guestbook
      hydrateTo:
        targetBranch: environments/e2e-next
  - repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD
    path: external-secrets
    hydrator:
      drySource:
        targetRevision: environments/e2e
        path: guestbook
      hydrateTo:
        targetBranch: environments/e2e-next