Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

BNFO 262 notebook environments

This repository houses the source code for the compute environments used in BNFO 262 at UC San Diego. Each directory in this repository corresponds with a different Docker image. There is one image for each weekly module of the course.

GitHub Actions automatically build and push updates to each environment for any commits made to the master branch or to a PR that will be merged into master. Images from PRs are tagged by their PR number.

Our Docker images

Our Docker images are pushed to GitHub's container registry, not Docker's. You can find a list of our Docker images in the "Packages" section in the right sidebar of this repository.

The following container registry URLs can be provided to the DataHub team at the beginning of every academic quarter:

ghcr.io/biom262/chipseq:master

ghcr.io/biom262/gwas:master

ghcr.io/biom262/imgproc:master

ghcr.io/biom262/networks:master

ghcr.io/biom262/popgen:master

ghcr.io/biom262/programming-R:master

ghcr.io/biom262/rna-seq:master

ghcr.io/biom262/scrna-seq:master

ghcr.io/biom262/spatial-tx:master

ghcr.io/biom262/stats:master

ghcr.io/biom262/variant_calling:master

Modifying an environment

  1. First, click here to create a GitHub codespace:

    https://github.com/codespaces/new/biom262/cmm262-notebook?skip_quickstart=true&machine=standardLinux32gb

    A GitHub codespace is a pre-configured VSCode development environment that will have all of the necessary tools pre-installed. Codespaces are free with the student developer pack.

    Instead of a codespace, you can also just clone the repository locally. In that case, you should also make sure to install conda and conda-lock.

    conda create -yn lock 'conda-forge::conda-lock'
    
  2. Create a new branch off of the master branch. Give it an informative name.

  3. Add the new software to the conda environment used by that module. Make sure to follow best practices (see the section below)!

    Note: Never use one conda environment.yml file for more than one module. Each module should have its own .yml file. Mixing modules into the same environment will make it difficult for future TAs to maintain the environment, since they won't be able to tell which packages to add or remove as the notebooks change.

  4. Check that the conda environment can still be solved by doing a dry-run

    conda env create --dry-run --file environment.yml
    
  5. Make sure to update the conda-linux-64.lock file

    If you added or modified a package, you should update just that package:

    conda activate lock
    conda-lock --kind explicit --platform linux-64 --file environment.yml --update PACKAGENAME
    

    Otherwise, you can just regenerate the entire .lock file from scratch:

    conda activate lock
    conda-lock --kind explicit --platform linux-64 --file environment.yml
    
  6. Commit and push your changes

  7. Once you're ready, create a pull request to merge it back into the master branch

  8. Wait up to 30 minutes for the images to be built and for the checks to pass

  9. You should see a green check mark if all of the checks pass. If not, click on the red X and then "Details" to view the error message. Add additional commit(s) to fix the issue.

  10. Test your changes (see section below) and add any commits as needed

  11. Once all checks and tests pass, merge your pull request!

Testing a new environment

Once you create a pull request within our notebook repository, GitHub Actions will automatically build an updated set of Docker images. The images will be tagged by the number assigned to your pull request.

  1. If off-campus, connect to the UCSD VPN first. Then log into DataHub via ssh from your laptop's terminal. You will need to enter your UCSD username/password.

    ssh USERNAME@dsmlp-login.ucsd.edu
    
  2. Run your container on DataHub

    launch-scipy-ml.sh -W BNFO262_WIXX_A00 -P Always -i ghcr.io/biom262/MODULENAME:pr-#
    

    Within the above command, you should replace the following:

    • XX should be replaced with the last two digits of the current year (ex: 24 for 2024)
    • MODULENAME should be replaced with the module name (ex: chipseq)
    • # should be replaced with the ID number of the pull request (ex: 11 for this pull request)
  3. Executing that command will generate a URL to a DataHub environment that uses your updated changes. Open the URL in your browser, and use that notebook environment to test if your changes work as expected. You should rerun your notebooks one more time here -- there's a possibility that they don't work here, even if they worked earlier!

    If the URL isn’t working, make sure you connect to the UCSD VPN.

    Note: If DataHub gives you "Error: ImagePullBackOff", then it probably means that your container image has yet to be pushed to the image repository. You can check the list of available images that have been pushed to the image repository here. If the pr-# tag does not appear there, then you will probably need to wait a bit and check back later.

Best practices for conda environments

reproducible_conda_envs

  1. Write your environment file manually. Don't create an environment and then export it to a .yml file using conda env export. This will inevitably create .yml files that cannot be easily updated in future years. Also, the .yml file will be unlikely to work with other environments besides your own (or other base Docker images besides the one from which you exported it) because of the inclusion of system-level packages.

    You can try to avoid system-level packages by providing the --from-history --no-builds flags but then you're likely to get dependency conflicts. And even then, the file will probably still need lots of manual fixing to follow the rest of the best practices. There is simply no replacement to a manually written file.

  2. You should only use packages from community-driven, open-source channels like conda-forge and bioconda. Other channels (like anaconda, r, and defaults) have been known to eventually purge old packages, breaking existing .yml files.

  3. Always specify conda-forge before bioconda in the channels list. Note that conda-forge is needed whenever bioconda is needed, but the opposite is not true.

  4. You should always specify nodefaults as a channel in the channels list, since the defaults channel conflicts with conda-forge.

  5. You should specify exact package versions and channels to reduce the amount of time it takes for conda to find the correct versions and channels (aka "solve the environment"). This also makes the .yml file much more reproducible and less likely to break in the future.

    Here's an example where we specify the channel name (conda-forge), the package name (r-base), and the package version (3.6.3):

    dependencies:
      - conda-forge::r-base==3.6.3
    

    To pin to exact package versions, use a double equals == instead of a single equals = sign. If you aren't sure which versions to specify, you can do a dry run without versions and then use whichever versions are chosen by the dry run.

  6. If a package can be installed via conda, do not specify it as a pip dependency in your environment file. Avoid pip dependencies if possible, since they may change your environment in unexpected ways.

  7. Do not include dependencies of any packages already listed in your environment file unless you import or use those dependencies in your own code.

    For example, if you use scanpy and it imports pytables, you shouldn't add pytables to your .yml file unless you directly import and use pytables in your code. This rule helps to ensure .yml files can be easily updated in future years.

  8. When checking whether your environment file will solve, make sure the --strict-channel-priority setting is turned on. See here for more info.

  9. If you are creating an environment that should be used from an R notebook, you must also specify the r-irkernel package as a dependency. This allows the environment to be detected by nb_conda_kernels.

    dependencies:
      - conda-forge::r-irkernel==1.3.1
    

    Otherwise, if you are creating an environment that should be used from a Python notebook, specify the ipykernel package. In either case, be sure to specify a version. I would recommend using the most recent one unless that gives you conflicts because the other packages in your environment are too old.

    dependencies:
      - conda-forge::ipykernel==6.20.1
    

References

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages