Install on a Hadoop Cluster

Here we provide instructions for installing and configuring dask-gateway-server on a Hadoop Cluster.

Create a user account

Before installing anything, you’ll need to create the user account that will be used to run dask-gateway-server. The name of the user doesn’t matter, only the permissions they have. Here we’ll use dask:

$ adduser dask

Enable proxy-user permissions

Dask-Gateway makes full use of Hadoop’s security model, and will start Dask workers in containers with the requesting user’s permissions (e.g. if alice creates a cluster, their dask workers will be running as user alice). To accomplish this, the gateway server needs proxy-user permissions. This allows the Dask-Gateway server to perform actions impersonating another user.

For dask-gateway-server to work properly, you’ll need to enable proxy-user permissions for the dask user account. The users dask has permission to impersonate can be restricted to certain groups, and requests to impersonate may be restricted to certain hosts. At a minimum, the dask user will require permission to impersonate anyone using the gateway, with requests allowed from at least the host running the dask-gateway-server.

<property>
  <name>hadoop.proxyuser.dask.hosts</name>
  <value>host-where-dask-gateway-is-running</value>
</property>
<property>
  <name>hadoop.proxyuser.dask.groups</name>
  <value>group1,group2</value>
</property>

If looser restrictions are acceptable, you may also use the wildcard * to allow impersonation of any user or from any host.

<property>
  <name>hadoop.proxyuser.dask.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.dask.groups</name>
  <value>*</value>
</property>

See the proxy-user documentation for more information.

Create install directories

A dask-gateway-server installation has three types of files which will need their own directories created before installation:

  • Software files. This includes a Python environment and all required libraries. Here we use /opt/dask-gateway.

  • Configuration files. Here we use /etc/dask-gateway.

  • Runtime files. Here we use /var/dask-gateway.

# Software files
$ mkdir -p /opt/dask-gateway

# Configuration files
$ mkdir /etc/dask-gateway

# Runtime files
$ mkdir /var/dask-gateway
$ chown dask /var/dask-gateway

Install a python environment

To avoid interactions between the system python installation and dask-gateway-server, we’ll install a full Python environment into the software directory. The easiest way to do this is to use miniconda, but this isn’t a strict requirement.

$ curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh
$ bash /tmp/miniconda.sh -b -p /opt/dask-gateway/miniconda
$ rm /tmp/miniconda.sh

We also recommend adding miniconda to the root user’s path to ease further commands.

$ echo 'export PATH="/opt/dask-gateway/miniconda/bin:$PATH"' >> /root/.bashrc
$ source /root/.bashrc

Install dask-gateway-server

Now we can install dask-gateway-server and its dependencies.

$ conda install -y -c conda-forge dask-gateway-server-yarn

If you want to use Kerberos for user-facing authentication, you’ll also want to install dask-gateway-server-kerberos:

$ conda install -y -c conda-forge dask-gateway-server-kerberos

Configure dask-gateway-server

Now we’re ready to configure our dask-gateway-server installation. Configuration is written as a Python file (typically /etc/dask-gateway/dask_gateway_config.py). Options are assigned to a config object c, which is then loaded by the gateway on startup. You are free to use any python syntax/libraries in this file that you want, the only things that matter to dask-gateway-server are the values set on the c config object.

Here we’ll walk through a few common configuration options you may want to set.

Enable YARN integration

First you’ll want to enable YARN as the cluster manager backend.

# Configure the gateway to use YARN as the cluster manager
c.DaskGateway.cluster_manager_class = (
    "dask_gateway_server.managers.yarn.YarnClusterManager"
)

Enable kerberos security (optional)

If your cluster has Kerberos enabled, you’ll also need to create a principal and keytab for the dask user. You’ll also need to create a HTTP service principal for the host running dask-gateway-server (if one doesn’t already exist). Keytabs can be created on the command-line as:

# Create the dask principal
$ kadmin -q "addprinc -randkey [email protected]_REALM.COM"

# Create the HTTP principal (if not already created)
$ kadmin -q "addprinc -randkey HTTP/FQDN"

# Create a keytab
$ kadmin -q "xst -norandkey -k /etc/dask-gateway/dask.keytab dask HTTP/FQDN"

where FQDN is the fully qualified domain name of the host running dask-gateway-server.

Store the keytab file wherever you see fit (we recommend storing it along with the other configuration in /etc/dask-gateway/, as above). You’ll also want to make sure that dask.keytab is only readable by the dask user.

$ chown dask /etc/dask-gateway/dask.keytab
$ chmod 400 /etc/dask-gateway/dask.keytab

To configure dask-gateway-server to use this keytab file, you’ll need to add the following line to your dask_gateway_config.py:

# Specify the location of the keytab file, and the principal name
c.YarnClusterManager.keytab = "/etc/dask-gateway/dask.keytab"
c.YarnClusterManager.principal = "dask"

# Enable Kerberos for user-facing authentication
c.DaskGateway.authenticator_class = "dask_gateway_server.auth.KerberosAuthenticator"
c.KerberosAuthenticator.keytab = "/etc/dask-gateway/dask.keytab"

Configure the server addresses (optional)

By default, dask-gateway-server will listen at the following addresses:

  • http://:8000: the public facing URL of the whole application. Set by c.DaskGateway.public_url.

  • tls://:8786: the scheduler proxy address, set by c.DaskGateway.gateway_url.

If you’d rather serve at different ports, you can configure these yourself:

c.DaskGateway.public_url = 'http://:<PORT-1>'
c.DaskGateway.gateway_url = 'http://:<PORT-2>'

Specify user python environments

Since the Dask workers/schedulers will be running in their own YARN containers, you’ll need to provide a way for Python environments to be available to these containers. You have a few options here:

  • Install identical Python environments on every node

  • Archive environments to be distributed to the container at runtime (recommended)

In either case, the Python environment requires the at least the dask-gateway package be installed.

Using a local environment

If you’ve installed identical Python environments on every node, you only need to configure dask-gateway-server to use the provided Python. This could be done a few different ways:

# Configure the paths to the dask-gateway-scheduler/dask-gateway-worker CLIs
c.YarnClusterManager.scheduler_cmd = "/path/to/dask-gateway-scheduler"
c.YarnClusterManager.worker_cmd = "/path/to/dask-gateway-worker"

# OR
# Activate a local conda environment before startup
c.YarnClusterManager.scheduler_setup = 'source /path/to/miniconda/bin/activate /path/to/environment'
c.YarnClusterManager.worker_setup = 'source /path/to/miniconda/bin/activate /path/to/environment'

# OR
# Activate a virtual environment before startup
c.YarnClusterManager.scheduler_setup = 'source /path/to/your/environment/bin/activate'
c.YarnClusterManager.worker_setup = 'source /path/to/your/environment/bin/activate'

Using an archived environment

YARN also provides mechanisms to “localize” files/archives to a container before starting the application. This can be used to distribute Python environments at runtime. This approach is appealing in that it doesn’t require installing anything throughout the cluster, and allows for centrally managing user’s Python environments.

Packaging environments for distribution is usually accomplished using

Both are tools for taking an environment and creating an archive of it in a way that (most) absolute paths in any libraries or scripts are altered to be relocatable. This archive then can be distributed with your application, and will be automatically extracted during YARN resource localization

Below we demonstrate creating and packaging a Conda environment containing dask-gateway, as well as pandas and scikit-learn. Additional packages could be added as needed.

Packaging a conda environment with conda-pack

# Make a folder for storing the conda environments locally
$ mkdir /opt/dask-gateway/envs

# Create a new conda environment
$ conda create -c conda-forge -y -p /opt/dask-gateway/envs/example
...

# Activate the environment
$ conda activate /opt/dask-gateway/envs/example

# Install dask-gateway, along with any other packages
$ conda install -c conda-forge -y dask-gateway pandas scikit-learn conda-pack

# Package the environment into example.tar.gz
$ conda pack -o example.tar.gz
Collecting packages...
Packing environment at '/opt/dask-gateway/envs/example' to 'example.tar.gz'
[########################################] | 100% Completed | 17.9s

Using the packaged environment

It is recommended to upload the environments to some directory on HDFS beforehand, to avoid repeating the upload cost for every user. This directory should be readable by all users, but writable only by the admin user managing Python environments (here we’ll use the dask user, and create a /dask-gateway directory).

$ hdfs dfs -mkdir -p /dask-gateway
$ hdfs dfs -chown dask /dask-gateway
$ hdfs dfs -chmod 755 /dask-gateway

Uploading our already packaged environment to hdfs:

$ hdfs dfs -put /opt/dask-gateway/envs/example.tar.gz /dask-gateway/example.tar.gz

To use the packaged environment with dask-gateway-server, you need to include the archive in YarnClusterManager.localize_files, and activate the environment in YarnClusterManager.scheduler_setup/YarnClusterManager.worker_setup.

c.YarnClusterManager.localize_files = {
    'environment': {
        'source': 'hdfs:///dask-gateway/example.tar.gz',
        'visibility': 'public'
    }
}
c.YarnClusterManager.scheduler_setup = 'source environment/bin/activate'
c.YarnClusterManager.worker_setup = 'source environment/bin/activate'

Note that we set visibility to public for the environment, so that multiple users can all share the same localized environment (reducing the cost of moving the environments around).

For more information, see the Skein documentation on distributing files.

Additional configuration options

dask-gateway-server has several additional configuration fields. See the Configuration Reference docs (specifically the yarn configuration docs) for more information on all available options. At a minimum you’ll probably want to configure the worker resource limits, as well as which YARN queue to use.

# The resource limits for a worker
c.YarnClusterManager.worker_memory = '4 G'
c.YarnClusterManager.worker_cores = 2

# The YARN queue to use
c.YarnClusterManager.queue = '...'

If your cluster is under high load (and jobs may be slow to start), you may also want to increase the cluster/worker timeout values:

# Increase startup timeouts to 5 min (600 seconds) each
c.YarnClusterManager.cluster_start_timeout = 600
c.YarnClusterManager.worker_start_timeout = 600

Example

In summary, an example dask_gateway_config.py configuration might look like:

# Configure the gateway to use YARN as the cluster manager
c.DaskGateway.cluster_manager_class = (
    "dask_gateway_server.managers.yarn.YarnClusterManager"
)

# Specify the location of the keytab file, and the principal name
c.YarnClusterManager.keytab = "/etc/dask-gateway/dask.keytab"
c.YarnClusterManager.principal = "dask"

# Enable Kerberos for user-facing authentication
c.DaskGateway.authenticator_class = "dask_gateway_server.auth.KerberosAuthenticator"
c.KerberosAuthenticator.keytab = "/etc/dask-gateway/dask.keytab"

# Specify location of the archived Python environment
c.YarnClusterManager.localize_files = {
    'environment': {
        'source': 'hdfs:///dask-gateway/example.tar.gz',
        'visibility': 'public'
    }
}
c.YarnClusterManager.scheduler_setup = 'source environment/bin/activate'
c.YarnClusterManager.worker_setup = 'source environment/bin/activate'

# Limit resources for a single worker
c.YarnClusterManager.worker_memory = '4 G'
c.YarnClusterManager.worker_cores = 2

# Specify the YARN queue to use
c.YarnClusterManager.queue = 'dask'

Open relevant ports

For users to access the gateway server, they’ll need access to the ports set in Configure the server addresses (optional) above (by default these are 8000 and 8786). How to open up these ports is system specific and is left as an exercise for the reader.

Start dask-gateway-server

At this point you should be able to start the gateway server as the dask user using your created configuration file. The dask-gateway-server process will be a long running process - how you intend to manage it (supervisord, etc…) is system specific. The requirements are:

  • Start with dask as the user

  • Start with /var/dask-gateway as the working directory

  • Add /opt/dask-gateway/miniconda/bin to path

  • Specify the configuration file location with -f /etc/dask-gateway/dask_gateway_config.py

For ease, we recommend creating a small bash script stored at /opt/dask-gateway/start-dask-gateway to set this up:

#!/usr/bin/env bash

export PATH="/opt/dask-gateway/miniconda/bin:$PATH"
cd /var/dask-gateway
dask-gateway-server -f /etc/dask-gateway/dask_gateway_config.py

For testing here’s how you might start dask-gateway-server manually:

$ cd /var/dask-gateway
$ sudo -iu dask /opt/dask-gateway/start-dask-gateway

Validate things are working

If the server started with no errors, you’ll want to check that things are working properly. The easiest way to do this is to try connecting as a user.

A user’s environment requires the dask-gateway library be installed. If your cluster is secured with kerberos, you’ll also need to install dask-gateway-kerberos.

# Install the dask-gateway client library
$ conda create -n dask-gateway -c conda-forge dask-gateway

# If kerberos is enabled, also install dask-gateway-kerberos
$ conda create -n dask-gateway -c conda-forge dask-gateway-kerberos

You can connect to the gateway by creating a dask_gateway.Gateway object, specifying the public address (if you changed the port for proxy server via gateway_url, you’ll also need to specify the proxy_address).

>>> from dask_gateway import Gateway

# When running without kerberos
>>> gateway = Gateway("http://public-address")

# OR, if kerberos is enabled, you'll need to kinit and then do
>>> gateway = Gateway("http://public-address", auth="kerberos")

You should now be able to make API calls. Try dask_gateway.Gateway.list_clusters(), this should return an empty list.

>>> gateway.list_clusters()
[]

Next, see if you can create a cluster. This may take a few minutes.

>>> cluster = gateway.new_cluster()

The last thing you’ll want to check is if you can successfully connect to your newly created cluster.

>>> client = cluster.get_client()

If everything worked properly, you can shutdown your cluster with dask_gateway.GatewayCluster.shutdown().

>>> cluster.shutdown()