Contributor’s Guide

Here are some useful notes about DeepView development.

Development Installation

The current development branch is develop. Direct pushes to this branch are not allowed. For all contributions, branch from and send pull requests to this branch.

Clone the Code

Clone the code from the main repository:

git clone git@github.com:satishlokkoju/deepview.git

Install DeepView from Local Code

1. DeepView currently supports Python version 3.10 or greater for macOS or Linux. Python 3.10 is recommended.

MacOS: The Python installer package can be downloaded from the`` Python.org website. During installation, deselecting GUI applications, UNIX command-line tools, and Python documentation will reduce the size of what is installed.

Ubuntu:

sudo apt install -y python3.10-dev python3.10-venv python3.10-tk
sudo apt-get install -y libsm6 libxext6 libxrender-dev libgl1-mesa-glx

2. It’s recommended to use a conda environment to manage all dependencies:

conda create -n deepview python=3.10
conda activate deepview

  1. Checkout the branch to install from (like our development branch, develop).

  2. Install DeepView and other requirements for development:

    make install

This will install deepview[notebook,test,doc,tensorflow,dataset-report,image,dimreduction,duplicates].

  1. Start using DeepView! For example:

    # import DeepView Components like this:
from deepview.introspectors import PFA

To uninstall:

make uninstall

(Optional) Install Git Large File Storage (LFS)

To track large files and binaries, DeepView uses Git LFS, which replaces the actual file and history with a text pointer, and stores the file contents on GitHub.

Follow the installation instructions (download binary, or install via Homebrew), and then set up with:

git lfs install

Currently, tracked via .gitattributes, the DeepView LFS files are:

  • .png

  • .jpg

  • .jpeg

  • .gif

These files are used in the docs (e.g., this page!), as well as the notebook examples. To add another large file (e.g. .mp4, small model), please track them with git lfs via:

# all files with extension
git lfs track "*.<extension>"

Or:

# One specific file
git lfs track "<specific_file>"

And make sure to commit any changes to the .gitattributes.

Writing Code

Updating & Building the Docs

The DeepView docs are built with Sphinx.

Update dev environment

1. If DeepView was installed with make install, the developer’s installation, all Sphinx dependencies for building the docs will already be installed. If DeepView was installed via pypi & pip, install documentation requirements via:

pip install "deepview[doc]"

  1. Install pandoc.

Doc Structure

All code for docs is stored in the docs/ folder:

  • api/: all API docs

  • dev/

    • contributing: Developer’s guide for installing and contributing back to DeepView

  • general/: intro / start pages

    • installation: full DeepView installation

    • support: how to get support on DeepView

    • example_notebooks: quick links to all example notebooks

  • how_to/: guides on some the fundamental DeepView concepts

    • connect_data: how to connect data into DeepView

    • connect_model: how to load model into a DeepView pipeline

    • introspect: understand DeepView “introspect”

  • img/: graphics (.png, .gif, .jpg, .jpeg currently tracked with git lfs)

  • introspectors/: algorithm pages for the various DeepView introspectors

  • reference/:

    • changelog: link to CHANGELOG.md

    • how_to_cite: information for citing DeepView + its various algorithms

  • utils/: API reference for certain DeepView components

    • data_producers: all built-in producers

    • pipeline_stages: useful pipeline stages (including processors)

  • conf.py: Sphinx configuration file, with Sphinx extensions used (like Napoleon)

  • index.rst: main landing page

  • Makefile: Sphinx build docs

Editing the Docs

All docs use .rst format. A nice cheat sheet can be found here.

Messing with the table of contents and side bar can be tricky, beware. When making modifications for the sidebar, use a clean build and remove the _build directory first. Warning: a clean build will also re-run all the notebooks from scratch, which can be quite time consuming.

Build docs locally

From the base deepview directory run:

make doc

Open docs/_build/html/index.html.

Alternatively, inside the docs folder, the following can be run directly:

make html

Tests

Writing Tests

Follow the existing examples in the codebase to add new tests. For help with pytest, check out this tutorial.

Running Tests

This project uses pytest and pytest extensions as follows:

Tool

Purpose

Configuration File

pytest

Unit testing.

pytest.ini

mypy (via pytest-mypy)

Typed static code analysis.

mypy.ini

flake8 (via pytest-flake8)

PEP8 compliance testing.

part of pytest.ini

coverage (via pytest-cov)

Code coverage report generation.

Run all tests:

make test

Run tests on wheels:

make test-wheel

Run static type check on notebooks:

make test-notebooks

Remove all generated files:

make clean

Contributing to Canvas Components

To add more functionality to canvas_ux, canvas_viz, or any component, first set up a dev environment. Then, packages can be updated and deployed. If your change could be useful to other users, please consider making a pull request.

Canvas Development Installation

1. Install JavaScript package managers.

Install Node and Yarn globally on your machine.

2. Create a Python environment.

With conda, using canvas as the environment name:

conda create -n canvas python=3.10
conda activate canvas

Or, instead using venv:

virtualenv --python /usr/local/opt/python/bin/python3 venv
source venv/bin/activate

3. Install Python dependencies.

Install Python dependences by running:

pip install -r requirements.txt

4. Install Canvas packages.

First, install the canvas_ux Python package.

scripts/dev-install.sh

Optionally, install all the widgets:

scripts/dev-install-widgets.sh

5. Build and watch for changes.

For the main canvas_ux package:

yarn dev

For canvas_viz:

cd canvas_viz
yarn watch

Optionally, for the widgets:

scripts/dev-watch-widgets.sh

Canvas Deployment Note

As all packages depend on canvas_viz, whenever canvas_viz is updated, all packages need to follow. To do that, you need to manually bump all _version.py files for all widgets and for the main Canvas package. Then, you can use:

scripts/dev-watch-widgets.sh

Submitting a Pull Request

Submit a new request.

A new pull request requires checking off the following list:

  • I’ve searched through existing Pull Requests and can confirm my PR has not been previously submitted.

  • I’ve written new tests for my core changes, as applicable.

  • I’ve tested all tests (including my new additions) with make test.

  • I’ve updated documentation as necessary and verified that the docs build and look nice.

  • My PR is of reasonable size for someone to review. (You may be asked to break it up into smaller pieces if it is not.)