Alexander Riccio

Upgrading, faster

Pip, the Python package management system, still lacks an easy way to update all installed packages. The “upgrade-all” ability has been in the works for nearly 4 years now.

In the meantime, many simple hacks have evolved to meet the demand. They’re all simple, and quite slow.

About six months ago I wrote a fast Python script to upgrade all local pip packages.

The idea is simple.

First:

import pip
import queue

Then, query pip for the list of installed packages:

def buildQueueOfInstalledPackages():
    distQueue = queue.Queue()
    for dist in pip.get_installed_distributions():
        distQueue.put(dist)
    return distQueue

Here is where my script gets interesting:

        queueOfPackagesToUpdate = buildQueueOfInstalledPackages()
        queueOfFailedPackages   = queue.Queue()
        if sys.version_info.major == 3 and sys.version_info.minor > 3:
            numCPUs = os.cpu_count()
        else:
            print("you're using an out-of-date version of python! Please upgrade. I'll set the number of threads to 8, as I can't query the cpu_count")
            numCPUs = 8
        [threads.append(installerThread(queueOfPackagesToUpdate, queueOfFailedPackages, "install -U")) for _ in range(numCPUs)]
        for thread in threads:
            thread.start()

I’m doing it in parallel.

The problem with upgrading pip packages is twofold. Firstly, it doesn’t have an --upgrade-all option (hence my script); Secondly, the process of upgrading each package is not just slow, but a single-threaded, cpu-bound operation.

This script is actually pretty straightforward. For each item in the queue, it spawns an instance of pip with “pip install -U package_name”, where  package_name is the name of the package. Each process will use 100% of a single core.  On versions of Python below 3.4,  it defaults to 8 threads – if you’re having problems, then you may need to tweak numCPUs yourself.

There are many easy optimizations remaining, but this is only intended as a stopgap, until pip gets the--upgrade-all option that it needs.

Migration, simpler

My script also makes it much easier to migrate a set of locally installed packages to a newer Python installation.

Run it in the source Python installation with the argument --writeToFile, and --readFromFile in the destination Python installation.

The script will write the list of packages to a file in the working directory, and then read the same file. It’s not exactly elegant, but it works.

Go grab it on GitHub:

https://github.com/ariccio/update-pip-packages

I welcome feedback, and enjoy pull requests!

~ by Alexander Riccio on December 16, 2014.

Posted in Computing, Featured Posts
Tags: pip, Python