gigl.src.common.utils.init_beam_pipeline_options

gigl.src.common.utils.dataflow.init_beam_pipeline_options(applied_task_identifier: AppliedTaskIdentifier, job_name_suffix: str, component: GiGLComponents | None = None, custom_worker_image_uri: str | None = None, **kwargs: Any) → PipelineOptions

Can pass in any options i.e. init_beam_pipeline_options(num_workers=1, max_num_workers=32, …) The options passed in will override default options if we define them. For example, you can override the job_name by passing in job_name=”something”

Args:

applied_task_identifier (AppliedTaskIdentifier) job_name_suffix (str): Unique identifier for the dataflow job in relation to this task (applied_task_identifier)

i.e. job_name_suffix = “inference”

Returns:

PipelineOptions: options you can use to generate the pipeline