gigl.orchestration.kubeflow.KfpOrchestrator#

class gigl.orchestration.kubeflow.kfp_orchestrator.KfpOrchestrator#

Bases: object

Orchestration of Kubeflow Pipelines for GiGL. Methods:

compile: Compiles the Kubeflow pipeline. run: Runs the Kubeflow pipeline. upload: Uploads the pipeline to KFP. wait_for_completion: Waits for the pipeline run to complete.

Methods

__init__

compile

Compiles the GiGL Kubeflow pipeline. Args: cuda_container_image (str): Container image for CUDA (see: containers/Dockerfile.cuda). cpu_container_image (str): Container image for CPU. dataflow_container_image (str): Container image for Dataflow. dst_compiled_pipeline_path (Uri): Destination path for where to store the compiled pipeline yaml. additional_job_args: Optional additional arguments to be passed into components, by component. tag: Optional tag, which is provided will be used to tag the pipeline description.

run

Runs the GiGL Kubeflow pipeline. Args: applied_task_identifier (AppliedTaskIdentifier): Identifier for the task. task_config_uri (Uri): URI for the task config. resource_config_uri (Uri): URI for the resource config. start_at (str): Component to start at. stop_after (str): Component to stop after. compiled_pipeline_path (Uri): Path to the compiled pipeline. If compile is False, this should be provided and is directly used to run the pipeline and skip compilation. If compile is True, this flag is optional and if provided, is used as the destination path for where to store the compiled pipeline yaml. additional_job_args: Optional additional arguements to be passed into components, by component.

wait_for_completion

__weakref__#: list of weak references to the object (if defined)

classmethod compile(cuda_container_image: str, cpu_container_image: str, dataflow_container_image: str, dst_compiled_pipeline_path: ~gigl.common.types.uri.uri.Uri = /home/runner/work/GiGL/GiGL/build/gigl_pipeline_gnn.yaml, additional_job_args: dict[~gigl.src.common.constants.components.GiGLComponents, dict[str, str]] | None = None, tag: str | None = None) → Uri#: Compiles the GiGL Kubeflow pipeline. Args:

cuda_container_image (str): Container image for CUDA (see: containers/Dockerfile.cuda). cpu_container_image (str): Container image for CPU. dataflow_container_image (str): Container image for Dataflow. dst_compiled_pipeline_path (Uri): Destination path for where to store the compiled pipeline yaml. additional_job_args: Optional additional arguments to be passed into components, by component. tag: Optional tag, which is provided will be used to tag the pipeline description.

run(applied_task_identifier: AppliedTaskIdentifier, task_config_uri: ~gigl.common.types.uri.uri.Uri, resource_config_uri: ~gigl.common.types.uri.uri.Uri, start_at: str = 'config_populator', stop_after: str | None = None, compiled_pipeline_path: ~gigl.common.types.uri.uri.Uri = /home/runner/work/GiGL/GiGL/build/gigl_pipeline_gnn.yaml) → PipelineJob#

Runs the GiGL Kubeflow pipeline. Args:

applied_task_identifier (AppliedTaskIdentifier): Identifier for the task. task_config_uri (Uri): URI for the task config. resource_config_uri (Uri): URI for the resource config. start_at (str): Component to start at. stop_after (str): Component to stop after. compiled_pipeline_path (Uri): Path to the compiled pipeline.

If compile is False, this should be provided and is directly used to run the pipeline and skip compilation. If compile is True, this flag is optional and if provided, is used as the destination path for where to store the compiled pipeline yaml.

additional_job_args: Optional additional arguements to be passed into components, by component.

Returns:: aiplatform.PipelineJob: The job that was created.

`__init__`
`compile`	Compiles the GiGL Kubeflow pipeline. Args: cuda_container_image (str): Container image for CUDA (see: containers/Dockerfile.cuda). cpu_container_image (str): Container image for CPU. dataflow_container_image (str): Container image for Dataflow. dst_compiled_pipeline_path (Uri): Destination path for where to store the compiled pipeline yaml. additional_job_args: Optional additional arguments to be passed into components, by component. tag: Optional tag, which is provided will be used to tag the pipeline description.
`run`	Runs the GiGL Kubeflow pipeline. Args: applied_task_identifier (AppliedTaskIdentifier): Identifier for the task. task_config_uri (Uri): URI for the task config. resource_config_uri (Uri): URI for the resource config. start_at (str): Component to start at. stop_after (str): Component to stop after. compiled_pipeline_path (Uri): Path to the compiled pipeline. If compile is False, this should be provided and is directly used to run the pipeline and skip compilation. If compile is True, this flag is optional and if provided, is used as the destination path for where to store the compiled pipeline yaml. additional_job_args: Optional additional arguements to be passed into components, by component.
`wait_for_completion`