FlyteAdmin Notifications (Webhook)

Issues Discussion

Motivation: Why do you think this is important?

Currently flyteadmin notifications are delivered using the PagerDuty, Github and Slack email APIs. On AWS deployments FlyteAdmin uses SES to trigger emails, for all others the only alternative email implementation is SendGrid integration.

Setting up SES or SendGrid can be somewhat complicated. Furthermore, asking your Flyte users to configure the aforementioned services with email integrations adds even more overhead. It would be simpler as an alternative to provide webhook integration for notification so that users only have to configure existing API keys for PagerDuty/Github/Slack.

Flyte currently only allows sending notifications by email and requires users to explicitly define notification rules in their launchplans.

FlyteAdmin Webhook

Add A webhook in flyteAdmin, triggered by an event in a source system and sent to a destination system (Slack, Gmail, Discord), with a payload of data (task, node, workflow envent / cloud event).

  • Webhook service sends a HTTP request to destination systems.
  • Configuring system-wide alerts (no need to explicitly enable notifications on a per-launchplan basis)
  • Limiting alerts to a subset of available states, i.e. receive notification for all workflow failures and aborts.
  • Limit for certain domains/projects/workflows
  • Message templating for Slack
  • No need to explicitly enable notifications on a per-launchplan basis.

  1. Admin publisher send the message to sns
  2. Message published to an SNS topic is distributed to a number of SQS queues in parallel
  3. Webhook service consumes the message in the specific queue.
  4. Webhook service pushes the message to the external service.

Add a Webhook should require simple configuration

System wide config

  1. Create a slack incoming webhook
  2. Add webhook URL to the flyteadmin configMap (UI could support updating webhook)
  3. Set up what events webhook should listen to
  4. Set up when and which notification should be sent























type WebHookConfig struct {
	// Type of webhook service to use. Currently only "slack" is supported.
	Name                         string                       `json:"name"`
	URL                          string                       `json:"url"`
	Payload                      string                       `json:"payload"`
	SecretName                   string                       `json:"secret"`
	NotificationsProcessorConfig NotificationsProcessorConfig `json:"processor"`
}

// WebhooksNotificationConfig defines the configuration for the webhook service.
type WebhooksNotificationConfig struct {
	// Defines the cloud provider that backs the scheduler. In the absence of a specification the no-op, 'local'
	// scheme is used.
	Type                         string                       `json:"type"`
	AWSConfig                    AWSConfig                    `json:"aws"`
	GCPConfig                    GCPConfig                    `json:"gcp"`
	NotificationsPublisherConfig NotificationsPublisherConfig `json:"publisher"`
	WebhooksConfig               []WebHookConfig              `json:"webhooks"`
	// Number of times to attempt recreating a notifications processor client should there be any disruptions.
	ReconnectAttempts int `json:"reconnectAttempts"`
	// Specifies the time interval to wait before attempting to reconnect the notifications processor client.
	ReconnectDelaySeconds int `json:"reconnectDelaySeconds"`
}



















# FlyteAdmin ConfigMap
webhook:
- name: Slack
  url: https://hooks.slack.com/services/T00000000
  secret: <secret_name>
  trigger:
    - when: workflow.status.phase not in ['Succeeded']
      send: [slack-notification-1]
    - when: workflow.status.phase in ['Succeeded']
      send: [slack-notification-2]
  template:
    // TBD: should we support fine-grain notification?
    - slack-notification-1
      message: |
      Workflow {{.workflow.name}} status is {{.workflow.status}}.
    - slack-notification-2
      message: |
      Workflow {{.workflow.name}} Succeeded

Flytekit

Use Launch Plan to override default webhook system config.













int_doubler_wf_lp = LaunchPlan.get_or_create(
    name="int_doubler_wf",
    workflow=int_doubler_wf,
    default_inputs={"a": 4},
    notifications=[
        Webhook(
            phases=[WorkflowExecutionPhase.SUCCEEDED],
            url=["https://slack/..."],
            message="Workflow {{.workflow.name}} status is {{.workflow.status}}."
        )
    ],
)

FlyteConsole

  1. Navigate to Integrations -> Generic Webhooks.
  2. Click New Webhook and then perform the following:
    • Webhook URL: Enter the webhook URL.
    • Description (optional): Enter an optional description.
    • Event Subscription: Select the event types that you would like to subscribe to.
    • Custom Header (optional): You may optionally create a custom header by entering a Name and Value.
    • Click Add Webhook.

Matchable Resource

Add wehbook config to MatchableResource, people can defines a webhook config that can be configured by customizable Project-, ProjectDomain.

Sandbox

We can add a sandbox publisher in the flyteadmin, this publisher directly sends the event or message to the exteranl service (slack) without setting up sns/sqs. We utilize a message queue because it ensures the delivery of events to external services. we can sacrifice the guaranteed delivery in the sandbox.

Question

  • How to add secret?
  • How to use webhook in on-prem clusters. (Kafka?)

Reference

Select a repo