Setup computing and development environments with conda

To study problems related to turbulence with computers, one needs to compute and to develop code. It is good to now enough about how to install software and libraries to be relatively independant with the computers that you are going to use. Here, we will learn some methods to install programs that will be useful in general and in particular for this course.

We will use Python 3 (with Miniconda3), a good Python IDE (either Spyder or VSCode), Jupyter and a version-control tool (Mercurial, or Git if you know it and really like it).

In the following, we give indications about how to install these tools and how to get the repository of this training locally on your computer. Please, if you encounter problems, fill an issue here to explain what you did and what are the errors (please copy / paste the error log). It is also a good exercice to learn how to open an issue in an issue tracker and how to interact to find a solution to your problem.

Moreover, let’s add that the best OS for HPC (and HPC with Python) is Linux/GNU. Windows (at least without WLS) and even macOS are less adapted for this particular application. Python is a cross-platform language but nevertheless, you will get a better experience for HPC with Python on Linux. Moreover, all large HPC clusters are running on Linux.

Note for Windows users

You can now install Ubuntu inside Windows. It is called the Windows Subsystem for Linux (WSL). There are some specific instructions for WSL in the file Notes on setting up WSL (Windows Subsystem for Linux).

Install Mambaforge and Python

The first step is to install Mambaforge, which is a modified version of Miniconda with the new and fast cross-platform package manager mamba and using by default the community driven conda-forge channel.

wget "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh" -O Mambaforge_installer.sh
bash Mambaforge_installer.sh -b
$HOME/mambaforge/bin/mamba init bash

Command wget not found?

You might be more lucky with the equivalent command with curl:

curl -L https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh -o Mambaforge_installer.sh

When it’s done, try to open a new terminal (click on ctrl-alt-t) and check that the line in the new terminal starts with (base). If yes, you can close the old terminal (with ctrl-d). The indication (base) means that you use the base “environment”.

conda and mamba are 2 commandline tools to manage software installations and create “environments”.

conda and mamba: differences?

Conda is an open source package management system and environment management system that runs on Windows, macOS, Linux and z/OS.

Mamba is a new implementation of conda, which is faster for some operations and has better logging.

Definition: environment

A environment is a set of programs and libraries with particular versions. An environment is defined by the software installed on your computer and by environment variables, in particular the variable $PATH, which contains all the paths where your shell looks for programs (you can read the output of echo $PATH).

It is very useful to be able to create different environments for different tasks. It is usually better to keep the base environment only for the conda software and to use different environments for other tasks. Since this is for a Master 2 courses, we will use this strategy here. We will have

• 1 environment for some basic libraries used in this course (called main),

• 1 environment with Fluidsim (called env_fluidsim)

• 1 environment with the Spyder editor (automatically created with the tool conda-app)

• 1 environment with Mercurial (automatically created with the tool conda-app)

conda takes the programs that it installs from “channels”. We are going to use the largest open-source community driven channel called conda-forge. With Mambaforge, conda-forge is by default the main channel.

We can start by creating the main environment with the commands:

# download a requirement file
wget https://gricad-gitlab.univ-grenoble-alpes.fr/augierpi/coursem1_pa_instabilities_turbulence/-/raw/main/main_environment.yml
# install with conda
mamba env create -f main_environment.yml

While mamba creates the environment, we can study the most important conda commands:

• conda activate main

• conda deactivate

• mamba search fluidsim

• mamba create -n name_env numpy pandas

• mamba env list

Install and setup Mercurial

To install Mercurial, you can do:

pip install conda-app
conda-app install mercurial

An error during the pip install command?

If you encounter a problem with the command pip install conda-app, you can try to run it again with the option --no-cache-dir.

Note

The conda-app install command can take some time, especially if many students run it at the same time in the same computer room.

Close the terminal (ctrl-d) and reopen a new terminal (ctrl-alt-t)! Then, you can open the Mercurial configuration file with:

gedit ~/.hgrc &

Why the & at the end of this command?

The & makes the command run in the background, so that you can continue to use this terminal while gedit is running.

From man bash:

If a command is terminated by the control operator &, the shell executes the command in the background in a subshell. The shell does not wait for the command to finish, and the return status is 0.

You have to modify it to get something like this:

[ui]
username=myusername <email@adress.org>
editor=nano
tweakdefaults = True

[extensions]
hgext.extdiff =
# only to use Mercurial with GitHub and Gitlab
hggit =

[extdiff]
cmd.meld =

Warning

Mercurial has to be correctly setup! Since we will use the Gitlab instance of UGA, the Mercurial extension hg-git has to be activated so the line hggit = in the configuration file is mandatory.

You can check that everything is right with hg version -v.

Clone this repository

Once everything is installed and setup, you should be able to clone the repository of the training with:

hg clone https://gricad-gitlab.univ-grenoble-alpes.fr/augierpi/coursem1_pa_instabilities_turbulence.git

Please tell us if it does not work.

Once the repository is cloned you should have a new directory scientific-computing-m2-efm (launch the command ls to see it) and you should be able to enter into it with cd scientific-computing-m2-efm.

Build the website of this course

The source of the website of this course are in the repository and you can build to website to read it offline. Just run make from the root directory (with the main environment).

Install the Python IDE Spyder

Spyder is a free and open source scientific environment written in Python, for Python, and designed by and for scientists, engineers and data analysts.

To install Spyder, you can do:

conda-app install spyder

After opening a new terminal, you should be able to launch Spyder by running spyder &. Configure the conda environment you want to work with. For now, it should be the main environment and the path towards the Python executable should be $HOME/mambaforge/envs/main/bin/python.

Install Fluidsim

During this course, we are going to use Fluidsim. It can be installed with:

mamba create -n env_fluidsim -y fluidsim \
  ipython matplotlib ipympl ipykernel spyder-kernels \
  "pyfftw=0.13.0=py39h51d1ae8_0"

About this command

• We add ipython to the environment to be sure that the ipython command uses this environment.

• We add matplotlib and ipympl to have full support for plotting (with Qt and in JupyterLab)

• pyfftw=0.13.0=py39h51d1ae8_0 is here because of a bug in a pyfftw build in conda-forge

Bonus: vscode or vscodium

You can also install VSCode (see here), which is a generalist, open-source and lightweight IDE.