How I manage Python for every day usage

Ohh XKCD, always on point :)

The more you do with Python, the more complicated your setup can get. For example, you might want a virtualenv for per project. For some projects, you might want to have a full sand box, and for others you might want to be able to use the system wide python environment. There might be other projects where you might want to test with, or use, multiple Python versions. Some might be Anaconda projects. You might have your own scripts that depend on system-wide packages, but you want those to be accessible from anywhere. You may also want a variety of python based tools installed in sandboxes. Fortunately, there’s a few great tools out there that can help maintain a sane configuration for most needs!

I’ll split this into 2 posts:

• Global python setup
• Emacs based development environment.

Global Config

Configuring Python based tools

First off, I use a lot of python based tools for a variety of purposes. Since most of these are not usually available in package managers, it’s best to pip install these. However, since there can be complex dependency requirements, I like to keep each of these installed in their own sandbox. For this, I use pipx. I set this up using the system python 3 and install it to the user wide python dir:

PYENV_VERSION=system pip3 install --user pipx

Then I install my tools from a requirements.txt file that I keep in my dotfiles (link here if interested):

while read pkg; do    
  $HOME/.local/bin/pipx install $pkg
done < requirements_pipx.txt

Pipx installs packages to ~/.local/bin by default, so I make sure this takes precedence over my pyenv install (see next section). To do so, just make sure your PATH is set up correctly in your init files. For example:

export PATH=~/.local/bin:~/.pyenv/shims:~/.pyenv/bin:$PATH

Install pyenv

In order to install and maintain different Python version/virtualenv combos, I use an amazing package called pyenv. I install it with all the default plugins with:

git_installers=(
    # other tools excluded for post...
    pyenv/pyenv-installer/master/bin/pyenv-installer # pyenv
)
for installer in "${git_installers[@]}"; do
    curl -sL https://raw.githubusercontent.com/$installer | bash
done

(I install a few other things off git in a similar manner which is why I’m using a loop, didn’t feel like changing the code for the post)

Make sure necessary packages are installed in every version

There’s a variety of packages I like to make sure are installed in every version and environment I set up. These are mainly development related and are required for using Emacs as my IDE.

To do this, I use the pyenv-default-packages plugin (I’ll talk about the other plugins in another section):

pyenv_plugins=(
    aiguofer/pyenv-version-alias
    jawshooah/pyenv-default-packages
    aiguofer/pyenv-jupyter-kernel
)
for plugin in "${pyenv_plugins[@]}"; do
    plugin_name=$(echo $plugin | cut -d '/' -f2)
    git clone https://github.com/$plugin $(pyenv root)/plugins/$plugin_name
done

Then I create a file ~/.pyenv/default-packages with:

ipdb
elpy
jedi
epc
importmagic
flake8

Setting up Jupyter kernels

Since I use a global jupyter-console install through pipx and each project has its own version, I make sure that a Jupyter kernel is installed for each version. For this, I wrote the pyenv-jupyter-kernel plugin. I showed you how I install this above. Now, whenever a new version/virtualenv is installed, a matching Jupyter kernel is created. I leverage this in Emacs to always use the corresponding kernel for whatever I’m working on.

Setting up default python environment

I like to have a default virtualenv, which inherits system packages, that I can use for general purpose Python stuff like writing scripts I use on a day to day, spin up a default REPL to try some stuff, etc. The reason I like using a virtualenv is that I don’t have to worry about using the --user flag, and the reason I inherit system packages is so that hard-to-build packages (QT and GTK come to mind) are available through the package manager. I also like to have one for both Python 2 and 3 for the occasion when I need Python 2. I set it up like this:

pyenv virtualenv --system-site-packages -p /usr/bin/python2 default2
pyenv virtualenv --system-site-packages -p /usr/bin/python3 default3
pyenv global default3 default2
pip2 install -r requirements.txt # install required deps for scripts
pip3 install -r requirements.txt

Setting up project versions

When setting up python environments for different projects, there can be a variety of requirements. For example, you might need a specific python version. You can use pyenv to install specific versions like this:

pyenv install 3.5.7

Then you might want to set up a virtualenv using that specifc version:

pyenv virtualenv 3.5.7 my_project

Now, you probably want to automatically use that environment when you’re inside the project directory, so you could set that up:

cd /path/to/my_project
pyenv local my_project

However, you might want to also have multiple python versions set up for TOX, so you could first use pyenv to install the versions you want to test against then set it up like this:

pyenv local my_project 3.7.1 3.6.3 3.5.7 2.7.10

This will make my_project the default for installing packages/etc, but the python executables for each version will also be avialable to tox.

Or you might need multiple virtualenvs for different components of a bigger project, while having all the installed executables available in the same “project”

cd /path/to/my_project
# create a dir for each component and set up virtualenvs
pyenv local component1 component2 component3...

Playing nice with terminal prompt

I love the flexibility that this set up provides, but I disliked how tools that display your pyenv version worked when using multiple versions. For example, my global version would display as: default3:default2 and some other projects this was even worse. In order to work around this, I created the pyenv-version-alias plugin, which allows me to set an alias for different pyenv environments. For example, the global version just displays as global instead.

Since the plugin just creates a command pyenv version-alias, you need to set up other tools to use it. I use Powerlevel10k for my prompt, so I configure it by overriding the pyenv prompt using:

# override powerlevel10k
prompt_pyenv() {
    local v=$(pyenv version-alias)
# only display pyenv if we're not using 'global'
if [[ "${v}" != "global" || "${POWERLEVEL9K_PYENV_PROMPT_ALWAYS_SHOW}" == "true" ]]; then
        _p9k_prompt_segment "$0" "blue" "$_p9k_color1" 'PYTHON_ICON' 0 '' "${v//\%/%%}"
    fi
}

And this is what it ends up looking like (showing my most complex environment here):