Try   HackMD

Submarine Swagger

import dependencies










from __future__ import print_function
import time
import swagger_client
from swagger_client.rest import ApiException
from pprint import pprint

from swagger_client import JobLibrarySpec
from swagger_client import JobTaskSpec
from swagger_client import JobSpec

Swagger client set host and port

https://stackoverflow.com/questions/49142551/python-swagger-client-set-host-and-port





configuration = swagger_client.Configuration()
configuration.host = 'http://submarine:8080/api'
api_client = swagger_client.ApiClient(configuration=configuration)
api_instance = swagger_client.JobsApi(api_client=api_client)

Create Experiment

https://github.com/apache/submarine/tree/master/docs/submarine-server#job-spec













jobLibrarySpec = JobLibrarySpec(name='tensorflow', version='2.1.0',image='gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0', cmd='python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150', env_vars={'ENV1': 'ENV1'})

worker = JobTaskSpec(name='tensorlfow', image=None, cmd=None, env_vars=None, resources='cpu=4,memory=2048M', replicas=1, cpu=None, gpu=None, memory=None)

body = swagger_client.JobSpec(name='mnist', namespace='submarine', library_spec=jobLibrarySpec, task_specs={"Worker": worker})

try:
    # Create a job
    api_response = api_instance.create_job(body=body)
    pprint(api_response)
except ApiException as e:
    print("Exception when calling JobsApi->create_job: %s\n" % e)

Output
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More → 
































{'attributes': {},
 'code': 200,
 'result': {'acceptedTime': '2020-05-19T07:02:30.000+08:00',
            'createdTime': None,
            'finishedTime': None,
            'jobId': 'job_1589842062823_0001',
            'name': 'mnist',
            'runningTime': None,
            'spec': {'librarySpec': {'cmd': 'python '
                                            '/var/tf_mnist/mnist_with_summaries.py '
                                            '--log_dir=/train/log '
                                            '--learning_rate=0.01 '
                                            '--batch_size=150',
                                     'envVars': {'ENV1': 'ENV1'},
                                     'image': 'gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0',
                                     'name': 'tensorflow',
                                     'version': '2.1.0'},
                     'name': '22mnist',
                     'namespace': 'submarine',
                     'projects': None,
                     'taskSpecs': {'Worker': {'cmd': None,
                                              'envVars': None,
                                              'image': None,
                                              'name': 'tensorlfow',
                                              'replicas': 1,
                                              'resourceMap': {'cpu': '4',
                                                              'memory': '2048M'},
                                              'resources': 'cpu=4,memory=2048M'}}},
            'status': 'Accepted',
            'uid': '037b9053-3d7e-4dcf-a622-6217653fb399'},
 'success': True}

Get Experiment








id = 'job_1589842062823_0001'
try:
    # Find job by id
    api_response = api_instance.get_job(id)
    pprint(api_response)
except ApiException as e:
    print("Exception when calling JobsApi->get_job: %s\n" % e)

List Experiment

api_response = api_instance.list_job(id)

Output
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More → 





























































{'attributes': {},
 'code': 200,
 'result': [{'acceptedTime': '2020-05-19T07:02:30.000+08:00',
             'createdTime': None,
             'finishedTime': None,
             'jobId': 'job_1589842062823_0001',
             'name': 'mnist',
             'runningTime': None,
             'spec': {'librarySpec': {'cmd': 'python '
                                             '/var/tf_mnist/mnist_with_summaries.py '
                                             '--log_dir=/train/log '
                                             '--learning_rate=0.01 '
                                             '--batch_size=150',
                                      'envVars': {'ENV1': 'ENV1'},
                                      'image': 'gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0',
                                      'name': 'tensorflow',
                                      'version': '2.1.0'},
                      'name': 'mnist',
                      'namespace': 'submarine',
                      'projects': None,
                      'taskSpecs': {'Worker': {'cmd': None,
                                               'envVars': None,
                                               'image': None,
                                               'name': 'tensorlfow',
                                               'replicas': 1,
                                               'resourceMap': {'cpu': '4',
                                                               'memory': '2048M'},
                                               'resources': 'cpu=4,memory=2048M'}}},
             'status': 'Accepted',
             'uid': '037b9053-3d7e-4dcf-a622-6217653fb399'},
            {'acceptedTime': '2020-05-21T11:54:29.000+08:00',
             'createdTime': None,
             'finishedTime': None,
             'jobId': 'job_1589842062823_0003',
             'name': '111mnist',
             'runningTime': None,
             'spec': {'librarySpec': {'cmd': 'python '
                                             '/var/tf_mnist/mnist_with_summaries.py '
                                             '--log_dir=/train/log '
                                             '--learning_rate=0.01 '
                                             '--batch_size=150',
                                      'envVars': {'ENV1': 'ENV1'},
                                      'image': 'gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0',
                                      'name': 'tensorflow',
                                      'version': '2.1.0'},
                      'name': 'mnist',
                      'namespace': 'submarine',
                      'projects': None,
                      'taskSpecs': {'Worker': {'cmd': None,
                                               'envVars': None,
                                               'image': None,
                                               'name': 'tensorlfow',
                                               'replicas': 1,
                                               'resourceMap': {'cpu': '4',
                                                               'memory': '2048M'},
                                               'resources': 'cpu=4,memory=2048M'}}},
             'status': 'Accepted',
             'uid': '7ad8205a-b3d0-485f-ad74-5496b038b382'}],
 'success': True}

Log Experiment




id = 'job_1589780188747_0001'
api_response = api_instance.get_log(id)
pprint(api_response)

Output
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More → 




















{'attributes': {},
 'code': 200,
 'result': {'jobId': 'job_1589842062823_0001',
            'logContent': [{'podLog': 'WARNING:tensorflow:From '
                                      '/var/tf_mnist/mnist_with_summaries.py:39: '
                                      'read_data_sets (from '
                                      'tensorflow.contrib.learn.python.learn.datasets.mnist) '
                                      'is deprecated and will be removed in a '
                                      'future version.\n'
                                      'Instructions for updating:\n'
                                      'Please use alternatives such as '
                                      'official/mnist/dataset.py from '
                                      'tensorflow/models.\n'
                                      'WARNING:tensorflow:From '
                                      '/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: '
                                      'maybe_download (from '
                                      'tensorflow.contrib.learn.python.learn.datasets.base) '
                                      'is deprecated and will be removed in a '
                                      .........

Delete Experiment



api_response = api_instance.delete_job(id)
pprint(api_response)

Ping Experiment



api_response = api_instance.ping()
pprint(api_response)

Output
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More → 



("{'status': 'OK', 'code': 200, 'success': True, 'message': None, 'result': "
 "'Pong', 'attributes': {}}")
tags: Submarine Swagger
Last changed by 
 
kevin su
0
213

Read more

SkyPilot Agent

SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, highest GPU availability, and managed execution.

Apr 17, 2024
Out of Core Plugin [RFC]

Motivation Currently it is hard to implement backend plugins, especially for data-scientists & MLE’s who do not have working knowledge of Golang. Also, performance requirements, maintenance and development is cumbersome. The document here proposes a path to make it possible to write plugins rapidly, while decoupling them from the core flytepropeller engine. Goals Plugins should be easy to author - no need of code generation, using tools that MLEs and Data Scientists are not accustomed to using. Most important plugins for Flyte today are plugins that communicate with external services. It should be possible to test these plugins independently and also deploy them privately. It should be possible for users to use backend plugins for local development, especially in flytekit and unionML

Dec 28, 2023
Airflow Agent

PR: https://github.com/flyteorg/flytekit/pull/1782

Sep 8, 2023
FlyteAdmin Notifications (Webhook)

Issues Discussion Motivation: Why do you think this is important? Currently flyteadmin notifications are delivered using the PagerDuty, Github and Slack email APIs. On AWS deployments FlyteAdmin uses SES to trigger emails, for all others the only alternative email implementation is SendGrid integration. Setting up SES or SendGrid can be somewhat complicated. Furthermore, asking your Flyte users to configure the aforementioned services with email integrations adds even more overhead. It would be simpler as an alternative to provide webhook integration for notification so that users only have to configure existing API keys for PagerDuty/Github/Slack. Flyte currently only allows sending notifications by email and requires users to explicitly define notification rules in their launchplans. FlyteAdmin Webhook

Jun 5, 2023
Read more from kevin su
Published on HackMD