Contributor’s Guide

Here are some useful notes about DeepView development.

Development Installation

Development Branch

The current development branch is develop. Direct pushes to this branch are not allowed. For all contributions, branch from and send pull requests to this branch.

Clone the Code

Clone the code from the main repository:

git clone git@github.com:satishlokkoju/deepview.git

Install DeepView from Local Code

1. Install python version 3.9 (recommended). To run TensorFlow1, install Python 3.7.

MacOS: The Python installer package can be downloaded from the Python.org website. During installation, deselecting GUI applications, UNIX command-line tools, and Python documentation will reduce the size of what is installed.

Ubuntu:

sudo apt install -y python3.9-dev python3.9-venv python3.9-tk
sudo apt-get install -y libsm6 libxext6 libxrender-dev libgl1-mesa-glx

2. Create and activate a python 3.9 virtual environment, then update pip and wheel:

   python3.9 -m venv .venv39
   source .venv39/bin/activate
   pip install --upgrade pip wheel
   

3. Checkout the branch to install from (like our development branch, develop).

4. Install DeepView and other requirements for development:

   make install
   

This will install deepview[notebook,test,doc,tensorflow,dataset-report,image,dimreduction,duplicates].

To install TF1-compatible code, try:

make install-tf1

Or for TF1 GPU:

make install-tf1-gpu

5. Start using DeepView! For example:

   # import DeepView Components like this:
   from deepview.introspectors import PFA
   

To uninstall:

make uninstall

(Optional) Install Git Large File Storage (LFS)

To track large files and binaries, DeepView uses Git LFS, which replaces the actual file and history with a text pointer, and stores the file contents on GitHub.

Follow the installation instructions (download binary, or install via Homebrew), and then set up with:

git lfs install

Currently, tracked via .gitattributes, the DeepView LFS files are:

• .png

• .jpg

• .jpeg

• .gif

These files are used in the docs (e.g., this page!), as well as the notebook examples. To add another large file (e.g. .mp4, small model), please track them with git lfs via:

# all files with extension
git lfs track "*.<extension>"

Or:

# One specific file
git lfs track "<specific_file>"

And make sure to commit any changes to the .gitattributes.

Writing Code

Updating & Building the Docs

The DeepView docs are built with Sphinx.

Update dev environment

1. If DeepView was installed with make install, the developer’s installation, all Sphinx dependencies for building the docs will already be installed. If DeepView was installed via pypi & pip, install documentation requirements via:

pip install "deepview[doc]"

2. Install pandoc.

Doc Structure

All code for docs is stored in the docs/ folder:

• api/: all API docs

• dev/

  • contributing: Developer’s guide for installing and contributing back to DeepView

• general/: intro / start pages

  • installation: full DeepView installation

  • support: how to get support on DeepView

  • example_notebooks: quick links to all example notebooks

• how_to/: guides on some the fundamental DeepView concepts

  • connect_data: how to connect data into DeepView

  • connect_model: how to load model into a DeepView pipeline

  • introspect: understand DeepView “introspect”

• img/: graphics (.png, .gif, .jpg, .jpeg currently tracked with git lfs)

• introspectors/: algorithm pages for the various DeepView introspectors

• reference/:

  • changelog: link to CHANGELOG.md

  • how_to_cite: information for citing DeepView + its various algorithms

• utils/: API reference for certain DeepView components

  • data_producers: all built-in producers

  • pipeline_stages: useful pipeline stages (including processors)

• conf.py: Sphinx configuration file, with Sphinx extensions used (like Napoleon)

• index.rst: main landing page

• Makefile: Sphinx build docs

Editing the Docs

All docs use .rst format. A nice cheat sheet can be found here.

Messing with the table of contents and side bar can be tricky, beware. When making modifications for the sidebar, use a clean build and remove the _build directory first. Warning: a clean build will also re-run all the notebooks from scratch, which can be quite time consuming.

Build docs locally

From the base deepview directory run:

make doc

Open docs/_build/html/index.html.

Alternatively, inside the docs folder, the following can be run directly:

make html

Tests

Writing Tests

Follow the existing examples in the codebase to add new tests. For help with pytest, check out this tutorial.

Running Tests

This project uses pytest and pytest extensions as follows:

Tool	Purpose	Configuration File
pytest	Unit testing.	pytest.ini
mypy (via pytest-mypy)	Typed static code analysis.	mypy.ini
flake8 (via pytest-flake8)	PEP8 compliance testing.	part of pytest.ini
coverage (via pytest-cov)	Code coverage report generation.

Run all tests:

make test

Run tests on wheels:

make test-wheel

Run static type check on notebooks:

make test-notebooks

Remove all generated files:

make clean

Submitting a Pull Request

Submit a new request.

A new pull request requires checking off the following list:

• I’ve searched through existing Pull Requests and can confirm my PR has not been previously submitted.

• I’ve written new tests for my core changes, as applicable.

• I’ve tested all tests (including my new additions) with make test.

• I’ve updated documentation as necessary and verified that the docs build and look nice.

• My PR is of reasonable size for someone to review. (You may be asked to break it up into smaller pieces if it is not.)