This repository houses the source code for the compute environments used in BNFO 262 at UC San Diego. Each directory in this repository corresponds with a different Docker image. There is one image for each weekly module of the course.
GitHub Actions automatically build and push updates to each environment for any commits made to the master branch or to a PR that will be merged into master. Images from PRs are tagged by their PR number.
Our Docker images are pushed to GitHub's container registry, not Docker's. You can find a list of our Docker images in the "Packages" section in the right sidebar of this repository.
The following container registry URLs can be provided to the DataHub team at the beginning of every academic quarter:
ghcr.io/biom262/chipseq:master
ghcr.io/biom262/gwas:master
ghcr.io/biom262/imgproc:master
ghcr.io/biom262/networks:master
ghcr.io/biom262/popgen:master
ghcr.io/biom262/programming-R:master
ghcr.io/biom262/rna-seq:master
ghcr.io/biom262/scrna-seq:master
ghcr.io/biom262/spatial-tx:master
ghcr.io/biom262/stats:master
ghcr.io/biom262/variant_calling:master
-
First, click here to create a GitHub codespace:
A GitHub codespace is a pre-configured VSCode development environment that will have all of the necessary tools pre-installed. Codespaces are free with the student developer pack.
Instead of a codespace, you can also just clone the repository locally. In that case, you should also make sure to install conda and
conda-lock.conda create -yn lock 'conda-forge::conda-lock' -
Create a new branch off of the
masterbranch. Give it an informative name. -
Add the new software to the conda environment used by that module. Make sure to follow best practices (see the section below)!
Note: Never use one conda
environment.ymlfile for more than one module. Each module should have its own.ymlfile. Mixing modules into the same environment will make it difficult for future TAs to maintain the environment, since they won't be able to tell which packages to add or remove as the notebooks change. -
Check that the conda environment can still be solved by doing a dry-run
conda env create --dry-run --file environment.yml -
Make sure to update the
conda-linux-64.lockfileIf you added or modified a package, you should update just that package:
conda activate lock conda-lock --kind explicit --platform linux-64 --file environment.yml --update PACKAGENAMEOtherwise, you can just regenerate the entire
.lockfile from scratch:conda activate lock conda-lock --kind explicit --platform linux-64 --file environment.yml -
Commit and push your changes
-
Once you're ready, create a pull request to merge it back into the
masterbranch -
Wait up to 30 minutes for the images to be built and for the checks to pass
-
You should see a green check mark if all of the checks pass. If not, click on the red X and then "Details" to view the error message. Add additional commit(s) to fix the issue.
-
Test your changes (see section below) and add any commits as needed
-
Once all checks and tests pass, merge your pull request!
Once you create a pull request within our notebook repository, GitHub Actions will automatically build an updated set of Docker images. The images will be tagged by the number assigned to your pull request.
-
If off-campus, connect to the UCSD VPN first. Then log into DataHub via
sshfrom your laptop's terminal. You will need to enter your UCSD username/password.ssh USERNAME@dsmlp-login.ucsd.edu -
Run your container on DataHub
launch-scipy-ml.sh -W BNFO262_WIXX_A00 -P Always -i ghcr.io/biom262/MODULENAME:pr-#Within the above command, you should replace the following:
XXshould be replaced with the last two digits of the current year (ex:24for 2024)MODULENAMEshould be replaced with the module name (ex:chipseq)#should be replaced with the ID number of the pull request (ex:11for this pull request)
-
Executing that command will generate a URL to a DataHub environment that uses your updated changes. Open the URL in your browser, and use that notebook environment to test if your changes work as expected. You should rerun your notebooks one more time here -- there's a possibility that they don't work here, even if they worked earlier!
If the URL isn’t working, make sure you connect to the UCSD VPN.
Note: If DataHub gives you "Error: ImagePullBackOff", then it probably means that your container image has yet to be pushed to the image repository. You can check the list of available images that have been pushed to the image repository here. If the
pr-#tag does not appear there, then you will probably need to wait a bit and check back later.
-
Write your environment file manually. Don't create an environment and then export it to a
.ymlfile usingconda env export. This will inevitably create.ymlfiles that cannot be easily updated in future years. Also, the.ymlfile will be unlikely to work with other environments besides your own (or other base Docker images besides the one from which you exported it) because of the inclusion of system-level packages.You can try to avoid system-level packages by providing the
--from-history --no-buildsflags but then you're likely to get dependency conflicts. And even then, the file will probably still need lots of manual fixing to follow the rest of the best practices. There is simply no replacement to a manually written file. -
You should only use packages from community-driven, open-source channels like
conda-forgeandbioconda. Other channels (likeanaconda,r, anddefaults) have been known to eventually purge old packages, breaking existing.ymlfiles. -
Always specify
conda-forgebeforebiocondain the channels list. Note thatconda-forgeis needed wheneverbiocondais needed, but the opposite is not true. -
You should always specify
nodefaultsas a channel in the channels list, since thedefaultschannel conflicts withconda-forge. -
You should specify exact package versions and channels to reduce the amount of time it takes for conda to find the correct versions and channels (aka "solve the environment"). This also makes the
.ymlfile much more reproducible and less likely to break in the future.Here's an example where we specify the channel name (conda-forge), the package name (r-base), and the package version (3.6.3):
dependencies: - conda-forge::r-base==3.6.3To pin to exact package versions, use a double equals
==instead of a single equals=sign. If you aren't sure which versions to specify, you can do a dry run without versions and then use whichever versions are chosen by the dry run. -
If a package can be installed via conda, do not specify it as a pip dependency in your environment file. Avoid pip dependencies if possible, since they may change your environment in unexpected ways.
-
Do not include dependencies of any packages already listed in your environment file unless you import or use those dependencies in your own code.
For example, if you use
scanpyand it importspytables, you shouldn't addpytablesto your.ymlfile unless you directly import and usepytablesin your code. This rule helps to ensure.ymlfiles can be easily updated in future years. -
When checking whether your environment file will solve, make sure the --strict-channel-priority setting is turned on. See here for more info.
-
If you are creating an environment that should be used from an R notebook, you must also specify the r-irkernel package as a dependency. This allows the environment to be detected by nb_conda_kernels.
dependencies: - conda-forge::r-irkernel==1.3.1Otherwise, if you are creating an environment that should be used from a Python notebook, specify the ipykernel package. In either case, be sure to specify a version. I would recommend using the most recent one unless that gives you conflicts because the other packages in your environment are too old.
dependencies: - conda-forge::ipykernel==6.20.1
- The steps to test images were taken from DataHub's documentation: https://github.com/ucsd-ets/datahub-example-notebook
- This repository is modeled after https://github.com/aryarm/demo-docker-action. Please post issues and questions on that repository.