Development
Contents
Development¶
This page provides information on how to build, test, and develop
dask-gateway.
Building Dask-Gateway¶
Clone Repository¶
Clone the Dask-Gateway git repository:
$ git clone https://github.com/dask/dask-gateway.git
Install Dependencies (Conda)¶
We recommend using the Conda package manager to setup your development environment. Here we setup a conda environment to contain the build dependencies.
# Create a new conda environment
$ conda create -n dask-gateway
# Activate environment
$ conda activate dask-gateway
# Install dependencies
$ conda install -c conda-forge \
cryptography \
tornado \
traitlets \
sqlalchemy \
dask
Besides the above dependencies, you’ll also need a Go compiler. You can install Go using your system package manager, the Go website, or use Conda:
$ conda install -c conda-forge go
Install Dependencies (Pip)¶
You can also setup the development environment using pip.
$ pip install \
cryptography \
tornado \
traitlets \
sqlalchemy \
dask
Besides the above dependencies, you’ll also need a Go compiler. You can install Go using your system package manager or the Go website.
(Optional) Install Dependencies via Conda Environment File¶
You can easily install all dependencies needed for local development with the dev-environment.yaml
environment file:
$ conda env create --file dev-environment.yaml
Build and Install Dask-Gateway¶
Dask-Gateway is composed of two packages, both contained in the same repository:
dask-gateway-server: the gateway server, located in thedask-gateway-serversubdirectory.dask-gateway: the gateway client, located in thedask-gatewaysubdirectory.
The install directions below are written assuming you’re in the top directory of the repository.
Building dask-gateway-server
# Build and install dask-gateway-server as an editable package
$ pip install -e ./dask-gateway-server
# or, build and install as a regular package
$ pip install ./dask-gateway-server
Building dask-gateway
# Build and install dask-gateway as an editable package
$ pip install -e ./dask-gateway
# or, build and install as a regular package
$ pip install ./dask-gateway
Running the Tests¶
The tests are located in the tests subdirectory, and test both packages. To
run the tests you also need to install pytest:
# Install pytest with conda
$ conda install -c conda-forge pytest
# Or install with pip
$ pip install pytest
The tests can then be run as:
# Run the test suite
$ pytest -v tests
In addition to the main tests, additional tests for the various backends are
run in docker (or in minikube for kubernetes). The scripts for setting up
these test environments are located in the continuous_integration
subdirectory:
Hadoop Tests:
continuous_integration/docker/hadoopPBS Tests:
continuous_integration/docker/pbsSlurm Tests:
continuous_integration/docker/slurmKubernetes Tests:
continuous_integration/kubernetes
The particularities of each setup differ, please see the
.github/workflows/test.yaml file for the specifics.
Building the Documentation¶
Dask-Gateway uses Sphinx for documentation. The source files are located in
dask-gateway/docs/source. To build the documentation locally, first install
the documentation build requirements. Note that autodoc-traits is used to
inspect the Python source code, which makes us also need the packages listed
after autodoc-traits to generate the full configuration reference.
# Install docs dependencies with conda
$ conda install -c conda-forge sphinx dask-sphinx-theme sphinx-autobuild autodoc-traits kubernetes_asyncio skein sqlalchemy
# Or install with pip
$ pip install sphinx dask-sphinx-theme sphinx-autobuild autodoc-traits kubernetes_asyncio skein sqlalchemy
Then build the documentation with make
# Running from the dask-gateway/docs folder
$ make html
The resulting HTML files end up in the _build/html directory.
Then consider using another make command to rebuild the documentation and
refresh a browser viewing the built documentation.
# Running from the dask-gateway/docs folder
$ make devenv