CamberEngine

Engine is the main class to run scientific jobs on the Camber platform. It has a preconfigured and optimized compute power that Camber users can instantiate at will.

As the preceding diagram shows, users primarily operate on individual notebook file systems. Engines are compute nodes that run specific computational job.

Each engine is designed to run specific codes or frameworks. They each have dependencies and are optimized differently depending on the target workload. Since the engine is mounted on the user file system, any outputs from the jobs also write to this file system.

You can run Camber compute a few lines of code:

from camber.athena import AthenaEngine

# instantiate the desired engine
engine = AthenaEngine(
	engine_size="XSMALL",
	with_gpu=False,
	mount_dir=".",  # default mount on current working directory
)

# create the job/workload
job = engine.create_job(command="env | sort")

# check the output of the job
job.read_logs(tail_lines=50)

# clean up the job
job.clear()

# reconfigure the engine and run a different job
job = engine.use_size("LARGE").create_job(command="sleep 10")

Description

CamberEngine is the foundation of all engines offered by Camber. Each engine subclasses comes from the CamberEngine class, which carries several universal methods for workflows.

Attributes

engine_type(str): the type of the engine
engine_size(str): the size of the engine
num_engines(int): the number of engines to use (for MPI)
with_gpu(bool): whether GPU is attached to the engine
mount_dir(str): the primary mount location of the engine on the notebook file system
env_vars(str): environment variables attached to the engine
version(str): the version of the engine, applicable to select engines
history(CamberJobCollection): all jobs created by the engine object so far

Methods

`use_size()`

use_size(self, engine_size: str, with_gpu: bool = False)

Configure the engine to be a specific size and, if needed, attach GPU capabilities.

Parameters

engine_size (str): Specifies the size of the engine. Valid options are XMICRO, MICRO, XXSMALL, XSMALL, SMALL, MEDIUM, and LARGE for CPU workloads, and XSMALL, MEDIUM, LARGE, XLARGE, and XXLARGE for GPU workloads.
with_gpu (bool): Indicates whether to attach GPU capabilities to the engine. Default is False.

Returns

self: Returns the newly configured engine using the new size.

`use_count()`

use_count(self, num_engines: int = 1)

Configure a given number of engines to use for the calculation (MPI).

Parameters

num_engines(int): Specifies the number of engines. This is used primarily for MPI calculations. For optimal multi-engine performance, it is highly recommended that you use an engine size of LARGE or greater when using more than one engine .

Returns

self: Returns the class commanding the specified number of engines

`use_mount()`

use_mount(self, mount_dir: str)

Set the directory on the local file system that the engine should mount to for running jobs.

Parameters

mount_dir (str): The directory path on the local file system to be used as the mount point. If you must set a custom mount directory, it is highly recommended to use the relative path in your current working directory. Otherwise, provide the full absolute path.

Returns

self: Returns the engine instance with the updated mount directory.

`use_env_vars()`

use_env_vars(self, env_vars: Dict[str, str])

Set environment variables that can be accessed as the job executes. All subsequent jobs that run on this engine also can use these environment variables.

Parameters

env_vars (Dict[str, str]): A dictionary of environment variable names and their corresponding values to be set.

Returns

self: Returns the engine instance with the updated environment variables.

`use_version()`

use_version(self, version: str)

Specify the version of the engine to use for job execution.

Parameters

version (str): The version of the engine to be used.

Returns

self: Returns the engine instance configured to use the specified version.

`create_job()`

create_job(self, command: str, extra_env_vars: Optional[Dict[str, str]] = None, extra_file_dirs: Optional[Dict[str, str]] = None, engine_version: Optiona[str] = None, tags: Union[str, List[str]] = None)

Creates a job on the Camber compute platform that runs a provided command.

Parameters

command (str): The shell command to be executed by the job.
extra_env_vars (Optional[Dict[str, str]]): Additional environment variables to add to the engine environment.
extra_file_dirs (Optional[Dict[str, str]]): Additional directories to be mounted for this job. Keys are the paths on the local file system, and values are the paths within the job environment. Defaults to None.
engine_version (Optional[str]): Uses a specific version of engine. Defaults to None.
tags (Union[str, List[str]]): Tags to be associated with this job. Can be a single string or a list of strings. Defaults to None.

Returns

CamberJob: An instance of the CamberJob class representing the created job.

`get_job()`

get_job(self, job_id: str)

Retrieve details of an existing job using its job ID.

Parameters

job_id (str): The unique identifier of the job to retrieve.

Returns

CamberJob: An instance of the CamberJob class representing the retrieved job.

`list_jobs()`

list_jobs(self, tags: Optional[Union[str, List[str]]] = None)

List all jobs created by this engine, optionally filtered by tags.

Parameters

tags (Optional[Union[str, List[str]]]): Tags to filter the jobs. Can be a single string or a list of strings. Defaults to None.

Returns

List[CamberJob]: A list of CamberJob instances that use the same engine type and fit the tag filters, if any.

`clear_job()`

clear_job(self, job_id: str)

Clear the specified job by its job ID, effectively cleaning up any resources associated with it.

Parameters

job_id (str): The unique identifier of the job to be cleared.

Returns

None: This method does not return any value.

`clear_history()`

clear_history(self)

Clear the job history for the engine, removing all records of previously created jobs.

Parameters

None

Returns

This method returns no value.

`create_scatter_job()`

create_scatter_job(self, params_grid: Dict[str, List[str]])

Creates a set of scatter jobs with the specified dictionary parameter grid. The keys of the dictionary should be the parameter names to create_job, and the values should be lists of values to scatter over. The Cartesian product of all values from all parameters is used to create individual jobs.

For example:

params_grid = {
    "engine_size": ["XSMALL", "SMALL"],
    "engine_type": ["ATHENA", "MESA"],
    "command": ["echo test"],
}
jobs = CamberJob.general_scatter(params_grid)

# the above is equivalent to creating the following 4 jobs
job1 = CamberJob.create(
		engine_size="XSMALL",
		engine_type="ATHENA",
		command="echo test"
)
job2 = CamberJob.create(
		engine_size="XSMALL",
		engine_type="MESA",
		command="echo test"
)
job3 = CamberJob.create(
		engine_size="SMALL",
		engine_type="ATHENA",
		command="echo test"
)
job4 = CamberJob.create(
		engine_size="SMALL",
		engine_type="MESA",
		command="echo test"
)

Parameters

params_grid (Dict[str, List[str]]): A dictionary where keys are the parameter names and values are lists of possible values for those parameters.

Returns

List[CamberJob]: A list of CamberJob instances representing the scatter jobs created.

Science Engines

As mentioned, the CamberEngine class is the basis for all science engines on the Camber platform. Below are specific engines implemented on and managed by the Camber platform.

CamberJob camber.athena