flyte-community

@flyte-community

Joined on Feb 24, 2023

  • HacmKD documents Logistics Goal: Discuss RFCs, features, integrations and major changes with Maintainers and Technical Steering Committee members When: Every 2 weeks on Thursdays, alternating times:7:00 am PT (Timezone converter) 11:00 am PT (Timezone converter) iCal File Join the mailing list to receive the invites/updates Where: Zoom bridge
     Like  Bookmark
  • 1.15 minor releases highlights Heads up: To prevent a cognitive name collision with LLM Agents, Flyte Agents are being renamed to Connectors. This is a WIP and no user-level changes are expected. We'll keep you posted. Webhook Connector: easily send workflow notifications using webhooks. No more email queues! https://github.com/flyteorg/flytekit/pull/3058 Simplified interface for declaring Resources, including support for tuple/list and stricter validations https://github.com/flyteorg/flytekit/pull/3177 Customize the Spark Driver and/or Executor Pod specs using PodTemplates! https://github.com/flyteorg/flytekit/pull/3016 Both the FileSensor and BaseSensor now support timeout, preventing the Connector from waiting indefinitely for a condition to be met https://github.com/flyteorg/flytekit/pull/2745
     Like  Bookmark
  • HacmKD documents Logistics Goal: Share success/failure stories with other Flyte community members, get updates on what's new in the project and learn how other organizations/projects are using/integrating Flyte When: First Tuesday of every month at 9:00AM Pacific Time (convert to your timezone)Import the iCalendar Where: Zoom bridge How:
     Like 1 Bookmark
  • @dylanspag-lmco thanks so much for your effort so far. I've been talking to other maintainers about this change and we think the best path forward is to no accept it as it would break RawContainer functionality, the only instance where the copilot container is used in Flyte. Flyte needs a way to determine that the user code finished execution and to capture the outcome (Success/Failure) and three mechanisms were envisioned to achieve that goal as described in the original flytecopilot design Poll the kube-api: it would require less process on the user code but a deeper investigation/benchmarking to determine the sane defaults to configure the kube-client and avoid overloading the kube-api server. Recent testing showed that this won't work, and it would require to spend time updating the code. Implement a file-based protocol. With this option, the user code had to include logic to drop the outputs to a certain directory and make the User code use a FileWatcher to determine the result of its own execution. Depending on the outcome (SUCCESS/FAILURE) it would need to write a different file, all of that before returning. Recent testing shows that it works butthere are potential performance implications going this route.Also, there is now wat to inspect that user code ran, just waiting for a file to drop. Notice that for the two options above there is no way to activate them or select them, no flag. Share process namespace between copilot and the user code container (current default). With this mechanism, the copilot container downloads the inputs, watches the user code container for a result (using the shared process namespace and, hence, requiring the SYS_PTRACE capability), and copies the outputs, or updates flyteadmin, depending on whether the task succeeded or failed, respectively.
     Like  Bookmark
  • Outline Why Plugins in Flyte? How flyteplugins work? How flyteplugins agent work? How to setup Agent Service in Development Mode? How to Develop your agent efficiently? Why plugins in Flyte? Flyte backend plugins enable a more streamlined worflow lifecycle where Tasks that invoke specific APIs/modules served by a plugin, won't spin up a new Pod but will execute the task in the external system, potentially reducing the overall workflow execution time and saving compute resources.
     Like  Bookmark