Client API

Gateway

class dask_gateway.Gateway(address=None, proxy_address=None, auth=None, asynchronous=False, loop=None)

A client for a Dask Gateway Server.

Parameters
  • address (str, optional) – The address to the gateway server.

  • proxy_address (str, int, optional) – The address of the scheduler proxy server. If an int, it’s used as the port, with the host/ip taken from address. Provide a full address if a different host/ip should be used.

  • auth (GatewayAuth, optional) – The authentication method to use.

  • asynchronous (bool, optional) – If true, starts the client in asynchronous mode, where it can be used in other async code.

  • loop (IOLoop, optional) – The IOLoop instance to use. Defaults to the current loop in asynchronous mode, otherwise a background loop is started.

adapt_cluster(cluster_name, minimum=None, maximum=None, active=True, **kwargs)

Configure adaptive scaling for a cluster.

Parameters
  • cluster_name (str) – The cluster name.

  • minimum (int, optional) – The minimum number of workers to scale to. Defaults to 0.

  • maximum (int, optional) – The maximum number of workers to scale to. Defaults to infinity.

  • active (bool, optional) – If True (default), adaptive scaling is activated. Set to False to deactivate adaptive scaling.

close()

Close the gateway client

cluster_options(use_local_defaults=True, **kwargs)

Get the available cluster configuration options.

Parameters

use_local_defaults (bool, optional) – Whether to use any default options from the local configuration. Default is True, set to False to use only the server-side defaults.

Returns

cluster_options – A dict of cluster options.

Return type

dask_gateway.options.Options

connect(cluster_name, shutdown_on_close=False)

Connect to a submitted cluster.

Parameters
  • cluster_name (str) – The cluster to connect to.

  • shutdown_on_close (bool, optional) – If True, the cluster will be automatically shutdown on close. Default is False.

Returns

cluster

Return type

GatewayCluster

get_cluster(cluster_name, **kwargs)

Get information about a specific cluster.

Parameters

cluster_name (str) – The cluster name.

Returns

report

Return type

ClusterReport

list_clusters(status=None, **kwargs)

List clusters for this user.

Parameters

status (ClusterStatus, str, or list, optional) – The cluster status (or statuses) to select. Valid options are ‘starting’, ‘started’, ‘running’, ‘stopping’, ‘stopped’, ‘failed’. By default selects active clusters (‘starting’, ‘started’, ‘running’).

Returns

clusters

Return type

list of ClusterReport

new_cluster(cluster_options=None, shutdown_on_close=True, **kwargs)

Submit a new cluster to the gateway, and wait for it to be started.

Same as calling submit and connect in one go.

Parameters
  • cluster_options (dask_gateway.options.Options, optional) – An Options object describing the desired cluster configuration.

  • shutdown_on_close (bool, optional) – If True (default), the cluster will be automatically shutdown on close. Set to False to have cluster persist until explicitly shutdown.

  • **kwargs – Additional cluster configuration options. If cluster_options is provided, these are applied afterwards as overrides. Available options are specific to each deployment of dask-gateway, see cluster_options for more information.

Returns

cluster

Return type

GatewayCluster

scale_cluster(cluster_name, n, **kwargs)

Scale a cluster to n workers.

Parameters
  • cluster_name (str) – The cluster name.

  • n (int) – The number of workers to scale to.

stop_cluster(cluster_name, **kwargs)

Stop a cluster.

Parameters

cluster_name (str) – The cluster name.

submit(cluster_options=None, **kwargs)

Submit a new cluster to be started.

This returns quickly with a cluster_name, which can later be used to connect to the cluster.

Parameters
  • cluster_options (dask_gateway.options.Options, optional) – An Options object describing the desired cluster configuration.

  • **kwargs – Additional cluster configuration options. If cluster_options is provided, these are applied afterwards as overrides. Available options are specific to each deployment of dask-gateway, see cluster_options for more information.

Returns

cluster_name – The cluster name.

Return type

str

GatewayCluster

class dask_gateway.GatewayCluster(address=None, proxy_address=None, auth=None, cluster_options=None, shutdown_on_close=True, asynchronous=False, loop=None, **kwargs)

A dask-gateway cluster.

Parameters
  • address (str, optional) – The address to the gateway server.

  • proxy_address (str, int, optional) – The address of the scheduler proxy server. If an int, it’s used as the port, with the host/ip taken from address. Provide a full address if a different host/ip should be used.

  • auth (GatewayAuth, optional) – The authentication method to use.

  • cluster_options (mapping, optional) – A mapping of cluster options to use to start the cluster.

  • shutdown_on_close (bool, optional) – If True (default), the cluster will be automatically shutdown on close.

  • asynchronous (bool, optional) – If true, starts the cluster in asynchronous mode, where it can be used in other async code.

  • loop (IOLoop, optional) – The IOLoop instance to use. Defaults to the current loop in asynchronous mode, otherwise a background loop is started.

  • **kwargs – Additional cluster configuration options. If cluster_options is provided, these are applied afterwards as overrides. Available options are specific to each deployment of dask-gateway, see Gateway.cluster_options for more information.

adapt(minimum=None, maximum=None, active=True, **kwargs)

Configure adaptive scaling for the cluster.

Parameters
  • minimum (int, optional) – The minimum number of workers to scale to. Defaults to 0.

  • maximum (int, optional) – The maximum number of workers to scale to. Defaults to infinity.

  • active (bool, optional) – If True (default), adaptive scaling is activated. Set to False to deactivate adaptive scaling.

close(shutdown=None)

Close the cluster object.

Parameters

shutdown (bool, optional) – Whether to shutdown the cluster. If True, the cluster will be shutdown, if False only the local state will be cleaned up. Defaults to the value of shutdown_on_close, set to True or False to override.

classmethod from_name(cluster_name, shutdown_on_close=False, address=None, proxy_address=None, auth=None, asynchronous=False, loop=None)

Connect to a submitted cluster.

Parameters
  • cluster_name (str) – The cluster to connect to.

  • shutdown_on_close (bool, optional) – If True, the cluster will be automatically shutdown on close. Default is False.

  • **kwargs – Additional parameters to pass to the GatewayCluster constructor. See the docstring for more information.

Returns

cluster

Return type

GatewayCluster

get_client(set_as_default=True)

Get a Client for this cluster.

Returns

client

Return type

dask.distributed.Client

scale(n, **kwargs)

Scale the cluster to n workers.

Parameters

n (int) – The number of workers to scale to.

shutdown()

Shutdown this cluster. Alias for close(shutdown=True).

Options

class dask_gateway.options.Options(*fields)

Options to submit to dask-gateway.

A mutable-mapping that describes all cluster options available for user’s to set. Options can be modified programatically, or via a widget when using a web interface (e.g. Jupyter Notebooks) with ipywidgets installed.

Examples

Options objects are normally created via the Gateway.cluster_options method:

>>> options = gateway.cluster_options()  

Available options can then be accessed or set via attribute or key:

>>> options.worker_cores  
1
>>> options["worker_cores"]  
1
>>> options.worker_cores = 2  
>>> options.worker_cores  
2

Accessing invalid options error appropriately:

>>> options.not_a_valid_option = 'myvalue'  
Traceback (most recent call last):
    ...
AttributeError: No option 'not_a_valid_option' available

Errors are also raised if values being set are invalid (e.g. wrong type, out-of-bounds, etc…).

Exceptions

class dask_gateway.GatewayClusterError

Exception related to starting/stopping/scaling of a gateway cluster

class dask_gateway.GatewayServerError

Exception related to the operation of the gateway server.

Indicates an internal error in the gateway server.