io.zwets.it

Bioinformatician without Borders

Wed, 01 Mar 2023 00:00:00 +0300

After ten years at Kilimanjaro Clinical Research Institute, I am excited to embark on a new venture: Zwets IT takes off!

As of today I am available for hire as an independent contractor in bioinformatics, based in The Netherlands, but open for bioinformatics work around the world.

Despite some colourful suggestions for company names by friends and colleagues, I’ve gone with pragmatic: Zwets IT (https://zwets.it).

It was a close call for The Flying Bioinformatician, Nomadic Bioinformatics, and, especially, «Bio-informaticien sans Frontières».

More about me here, in my CV, on LinkedIn, or reach out at Zwets IT.

GTDB and MinHash - a little revolution

Sat, 14 Dec 2019 00:00:00 +0300

Being a bioinformatician at heart, I can’t help but feel that a little revolution is happening at the confluence of two recent developments in microbial genomics and bioinformatics.

The first is the advent of the GTDB, a phylogenetically consistent bacterial and archaeal taxonomy. Boldy going where we didn’t go before, Parks c.s. throw off the shackles of pre-genomic microbiology in favour of taxonomic assignment based on degrees of genomic similarity.

Update I just pushed gtdb-taxo, a command-line search tool and browser for the GTDB taxonomy, to GitHub.

The second development is the rise of MinHash, a computational approach designed 20 years ago for massive scale document comparison, and introduced to genomics by Ondov c.s. in the Mash tool. Mash and its various descendants (see the recent Dashing paper for an overview) have enabled the efficient many-to-many comparison of large genome collections.

It’s exciting to see the possibilities opened up by these developments, and an absolute joy to work in a field where these little revolutions still happen.

Introducing kcst

Thu, 11 Oct 2018 00:00:00 +0300

Finally got around to getting kcst in publishable shape.

kcst predicts species and MLST for bacterial genome assemblies, and does it fast: it takes less than a second for the average assembly. kcst comes with khc (k-mer hit counter), a fast and accurate tool to compute sequence similarity between a query sequence and a set of subject sequences.

Homepage is here: https://io.zwets.it/kcst, and here is the code.

Evidence-based daughter

Fri, 30 Dec 2016 00:00:00 +0300

Tonight I carried my youngest daughter (Vicky, 5 yo) from the car to our house. As I shut the car she said “Careful! I’m always worried that I get my fingers caught between the door.”

It actually happened twice over the past two weeks. The usual accident: a sibling closing the door while she was still getting out. Three fingers stuck at the hinge side … owww!

We were lucky. Land Rover hinges have plenty of play, so it was painful but she still has all her fingers.

Vicky continued: “I’m worried because, you know, it hurts to get your fingers stuck. Three fingers hurts a lot. One finger doesn’t hurt so much. Two fingers I don’t know. I haven’t tried.”

Architectural Decision Making

Fri, 28 Oct 2016 00:00:00 +0300

When working as an enterprise architect, I used to profess that optimal decisions in IT do not exist. There are as many independent valuations of a solution as there are stakeholders in the decision. Stakeholders take viewpoints, and each viewpoint defines a different set of priorities in terms of which alternatives are evaluated.

But this doesn’t at all mean that architectural considerations become a matter of taste, or must be the subject of endless debate¹, as often happens. I define “architectural decisions” loosely as: decisions in IT which have a large impact over a wide scope. They take place at every level from IT strategy down to developer teams.

In my experience, a very productive approach to shared architectural decision making is to split the discussion into two decoupled levels: evidence and utility. I have successfully used this in workshops to achieve “the best” architectural decisions, even in environments where opinions were strongly divided.

Evidence

On the evidence level, the goal is to establish agreed-upon objective conclusions based on evidence. E.g.: “Alternative A takes X less time to build, whereas B is Y times more reliable”. The quantities X and Y needn’t be exact, but must be supported by evidence and be uncontested by any party.

Any argument which cannot (in principle) be substantiated by facts is disallowed in this phase. For instance, a statement “alternative C is better because it follows service-oriented principles” is tabled until “better” is quantified, and “service-oriented principles” are qualified.

This phase turns out to be both valuable and enjoyable for participants. It is all about engineering, and everyone is working together toward a shared goal, namely to establish measurable, objective truths about the competing alternatives. Almost as a side effect, a lot of insight is gained that will be useful for the eventual design of the solution.

Utility

On the “utility” level, the goal is to elicit viewpoints potentially held by stakeholders, and the probable valuation of the alternatives given their frames of reference (objectives, constraints, priorities). In decision theory this would be called determining the utility functions. Note to self: in very complex decision processes, probabilistic graphical modeling a useful tool?

Again, the discussion is not about deciding the optimal alternative, or what is the superior viewpoint. The goal is to make all valid viewpoints explicit, so as to understand how alternatives are rated differently by different people. Participants are expected to not argue for their own view, but also to come up with the reasoning of other stakeholders.

It is especially during this phase that differences between factions are resolved. They might not sway their opinion just yet, but gaining an understanding of where other people “are coming from” as well as an awareness of one’s own biases are the much more important benefits. In fact, even when the goal of this phase is not to obtain consensus, it tends to bring parties much closer together.

Decision

It would seem that the above activities, however pleasant they may have been, haven’t brought us further. At the end of the analysis we have collected objective facts and a shared understanding of the different valuations, but what is the decision going to be?

The final decision will be made by the person with the responsibility and authority to do this. That person’s viewpoint will be known at this point (or else a major stakeholder was overlooked in the previous phase!), but this does not mean that his/her viewpoint necessarily determines the outcome. It behooves the decision taker, now that all viewpoints are known, to consider these in making ‘the right decision’.

What is interesting is that during the process, the single decision maker becomes less relevant. From the shared understanding of the knowledge that has been gained in the process, and from the (intuitive) knowledge that people have about the distribution of influence within the organisation, most people present in the second workshop will already know what the outcome must be. Not only that, they will also better understand the outcome, even if it was not their (a priori) preference.

In that sense, there ís an optimum outcome of architectural decisions: it is the one which is maximally evidence-based, and creates the most shared understanding of the outcome.

Credits

The above is not just theory. I was positively surprised by its results (and the fun had and insights gained by all participants) when I first trialed it at a client where I was consulting. I later repeated it and refined the way the workshops worked, but then left architecture consulting for genomics. I was recently reminded that I should document this somewhere. Hence this post.

The seeds for these ideas came from an obscure but well-researched and thought-out Ph.D. thesis that I stumbled upon when searching for a tool for documenting architectural decisions. The thesis [citation needed] had been written by [citation needed] during an internship at IBM (Vienna?). The tool itself never made it to deployment, as it was a 700MB download which included a full unreleased version of WebSphere Server which I couldn’t get to work. :)

I don’t remember who coined it, but I love the proposed collective noun (as in: “a pride of lions” and “a flock of seagulls”) for architects: an argument of architects. ↩

OpenCL on a dual graphics Dell M4800

Thu, 18 Aug 2016 00:00:00 +0300

This post documents how I got OpenCL working on Ubuntu 16.04 on my dual (hybrid) graphics Dell M4800 workstation. The machine has an integrated Intel HD Graphics 4600 GPU, and a discrete AMD/ATI FirePro M5100.

Background

Caffe can transparently switch between CPU and GPU for its computations. Caffe’s master branch is bound to NVidia. Its OpenCL branch is in development and should support all vendors through the OpenCL standard.

I prefer to keep my Ubuntu 16.04 plain vanilla, that is stick with packages available in the distro. I avoid PPAs, sudo make install and certainly sudo ./run-installer.sh.¹ My worry was that installing OpenCL would involve a lot of the latter, but it turned out that apt-get install goes a long way.

The tl;dr version: sudo apt-get install ocl-icd-libopencl1 beignet-opencl-icd mesa-opencl-icd makes both GPUs available as OpenCL devices. The CPU can be added quite easily. Below is my install log. YMMV.

Exploring the GPUs

My Dell M4800 has dual Intel/AMD graphics:

$ lspci -nn | grep VGA
00:02.0 VGA compatible controller [0300]: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller [8086:0416] (rev 06)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Venus XT [Radeon HD 8870M / R9 M270X/M370X] [1002:6821] (rev ff)

The AMD in the Dell M4800 is a FirePro M5100:

$ # 1002 and 6821 from lspci -nn are vendor and device id; 1028 is subvendor Dell (pciutils)
$ sed -ne '/^1002/,/^[0-9]/p' /usr/share/misc/pci.ids | sed -ne '/^\t6821/,/^\t[0-9]/p' | grep 1028
      1028 05cc  FirePro M5100
      1028 15cc  FirePro M5100

Ubuntu suggests no proprietary drivers for my GPUs:

$ ubuntu-drivers list
intel-microcode		# Driver for the CPU, not the GPU

The DRM driver modules are i915 and radeon which come with the kernel:

$ lsmod | grep drm_kms_helper
drm_kms_helper        147456  2 i915,radeon
$ dpkg -S i915.ko radeon.ko | grep $(uname -r)
linux-image-extra-4.4.0-34-generic: /lib/modules/4.4.0-34-generic/kernel/drivers/gpu/drm/i915/i915.ko
linux-image-extra-4.4.0-34-generic: /lib/modules/4.4.0-34-generic/kernel/drivers/gpu/drm/radeon/radeon.ko

The Intel integrated GPU (IGD) is powered on, the AMD discrete GPU (DIS) is idle:

$ sudo cat /sys/kernel/debug/vgaswitcheroo/switch
DIS: :DynOff:0000:01:00.0
IGD:+:Pwr:0000:00:02.0
DIS-Audio: :Off:0000:01:00.1

(Note: vga_switcheroo is the legacy kludge for hybrid graphics systems exposed by the radeon module.)

Installing OpenCL

Installing OpenCL involves three layers:

The API library and implementation loader (ICD Loader)
One or more vendor-specific platform implementations (ICDs, installable client drivers)
Any DRM/DRI hardware device drivers required by the ICDs

The top layer is provided by Ubuntu package ocl-icd-libopencl1. This package contains libOpenCL, the API library against which OpenCL applications are linked. It also provides the ICD Loader, the machinery that dynamically loads ICDs. ICDs (installable client drivers) implement the vendor-specific interaction with the hardware.

It is convenient to also install the clinfo utility which queries the API library about available platforms:

$ sudo apt-get install clinfo ocl-icd-libopencl1
$ clinfo
Number of platforms   0   # We haven't installed ICDs yet

Each ICD is defined by a file with extension .icd in directory /etc/OpenCL/vendors. The first line in the ICD file specifies the path to a shared library. The ICD Loader dlopens this library when the platform is requested. Environment variables control the order of the platforms, the default platform, and the debugging level. For instance, to run an application on a specific platform, set OCL_ICD_VENDORS=<icd-file-name>. See man libOpenCL.

Installing ICDs

Ubuntu provides ICDs for Intel and Nvidia GPUs. AMD is supported via the Mesa ICD, which implements OpenCL atop the open source DRM/DRI drivers:

$ apt-cache search '^opencl-icd$'  # virtual package provided by all ICDs
beignet-opencl-icd - OpenCL library for Intel GPUs
mesa-opencl-icd - free implementation of the OpenCL API -- ICD runtime
nvidia-opencl-icd-304 - NVIDIA OpenCL ICD
... more nvidia versions ...

$ apt-cache show beignet-opencl-icd
... supports the integrated GPUs of Ivy Bridge, Bay Trail, Haswell and Broadwell processors

$ apt-cache depends mesa-opencl-icd  # cards supported by the mesa ICD
... r600, amdgcn, amdgpu1, radeon1, nouveau2 ...

Install the Intel and Mesa ICDs to obtain the respective platforms:

$ sudo apt-get install beignet-opencl-icd mesa-opencl-icd
$ clinfo -l
Platform 0: Intel Gen OCL Driver
 `-- Device 0: Intel(R) HD Graphics Haswell GT2 Mobile
Platform 1: Clover
 `-- Device 0: AMD CAPE VERDE (DRM 2.43.0, LLVM 3.8.0)

It is that simple. Two OpenCL devices ready to rock. Run clinfo without -l for a detailed overview of the platforms and the features of each device.

Smoke Testing

Andreas Klöckner’s wiki has an OpenCL HOWTO and lots more on OpenCL, CUDA, PyOpenCL, and the like. His cl-demo application is perfect for a smoke test:

$ git clone 'https://github.com/hpc12/tools' hpc12-tools
$ cd hpc12-tools
$ make
$ ./cl-demo 1000000 10
Choose platform:
[0] Intel
[1] Mesa
Enter choice: 0
Choose device:
[0] Intel(R) HD Graphics Haswell GT2 Mobile
Enter choice: 0
... output ...
0.000824 s
14.566110 GB/s
GOOD

Good. Repeat for the Mesa/Clover platform:

$ ./cl-demo 1000000 10
... omitted for brevity ...
0.001086 s
11.053539 GB/s
GOOD

Good! We have a working OpenCL system supporting both graphics cards, and needed only apt-get install with no out-of-distro repositories. The “nice to have” to add is OpenCL CPU support.

Installing CPU support

Unfortunately there isn’t currently in Ubuntu an OpenCL ICD for CPUs. One used to come with fglrx, supporting AMD and Intel CPUs, but that package was dropped in Ubuntu 16.04. Installing one is easy though. There are two options.

Intel’s CPU driver

Intel’s “OpenCL™ Runtime for Intel® Core™ and Intel® Xeon® Processors” can be downloaded from their OpenCL™ Drivers and Runtimes for Intel® Architecture page.

Here is a way to turn the installer into a .deb (thus reducing the risk of overwriting files from other packages). Also, note the comment at the bottom of that page mentioning that install.sh or install_GUI.sh should just work now that Intel officially supports the driver on Ubuntu. I haven’t tested this.

AMD’s CPU driver

AMD’s CPU driver, which supports both AMD and Intel CPUs, comes ‘for free’ with their GPU driver. If you want to install just the CPU driver (and stick to mesa over radeon on the GPU), then this works:

Download and unpack the AMDGPU-Pro, beta driver (more on this below) and install only the ICD package:

$ wget 'https://www2.ati.com/drivers/linux/ubuntu/amdgpu-pro-16.60-379184.tar.xz'
  # note: link was current on 2017-03-01, check the AMDGPU-Pro page for latest
$ tar Jxvf amdgpu-pro_16.60-379184.tar.xz
$ cd amdgpu-pro-driver
$ sudo dpkg -i amdgpu-pro-opencl-icd_*_amd64.deb

This installs:

# The OpenCL ICD definition file for the AMD GPU platform
/etc/OpenCL/vendors/amdocl64.icd

# The libraries implementing the platform and its CPU and GPU devices.
# The GPU won't show up unless we install the DRI/DRM libraries, driver and firmware.
/usr/lib/x86_64-linux-gnu/amdgpu-pro/libamdocl12cl64.so
/usr/lib/x86_64-linux-gnu/amdgpu-pro/libamdocl64.so

We need to manually create one file to add the installed libraries to the ld path:

$ cat <<EOF | sudo sh -c 'cat > /etc/ld.so.conf.d/local-amdgpu.conf' && sudo ldconfig
# Added manually; can be removed when package amdgpu-pro-opencl-icd is uninstalled
/usr/lib/x86_64-linux-gnu/amdgpu-pro
EOF

This would have been done by amdgpu-pro-core but we don’t want to install that package because it also adds /etc/modprobe.d/amdgpu-blacklist-radeon.conf which does what its name says, and we don’t want to blacklist the radeon module.

And here we are:

$ clinfo -l
Platform 0: Intel Gen OCL Driver
 `-- Device 0: Intel(R) HD Graphics Haswell GT2 Mobile
Platform 1: Clover
 `-- Device 0: AMD CAPE VERDE (DRM 2.43.0, LLVM 3.8.0)
Platform 2: AMD Accelerated Parallel Processing
 `-- Device 0: Intel(R) Core(TM) i7-4910MQ CPU @ 2.90GHz

All processing devices on my workstation made available using only debs (plus a mostly harmless manual configuration change).

Installing proprietary GPU support

From here on it’s optimisation only. We have a working OpenCL system which makes available both GPUs and the CPU. All software used is open source (AMDGPU-PRO will be in Ubuntu once it leaves beta). Inquisitive minds may wonder if the proprietary drivers have more to offer, certainly for the AMD GPU which currently operates via the generic Mesa layer.

Intel’s GPU driver

Intel’s “OpenCL™ 2.0 Driver+Runtime for Intel® HD, Iris™, and Iris™ Pro Graphics for Linux” is available from their OpenCL™ Drivers and Runtimes for Intel® Architecture page. I haven’t tested this and stick with the open source driver. It allegedly [citation needed] is at least as good, and Intel devs are putting lots of effort into it. Kudos!

AMD’s AMDGPU-PRO driver

AMDGPU-PRO will supersede Catalyst (fglrx), which was dropped from Ubuntu 16.04. It will have a GPL kernel module (but proprietary firmware) and be supported on Ubuntu. It is currently available in beta, but works only for a handful of models. More on it here, and on AMDGPU-PRO’s official page, which has installation instructions and model compatibility lists.

Installing the beta packages requires care. Though allegedly tailored for Ubuntu 16.04, the AMDGPU-PRO packages have multiple packaging errors, such as failing to declare conflicts with existing packages, leaving you with a stuck APT. Details in Appendix I below.

AMD’s Catalyst (fglrx) driver

As AMDGPU-PRO doesn’t (yet) support my FirePro M5100, I tried the Catalyst Pro (workstation) driver. The Catalyst driver can be built “headless”, meaning that you can build and install just the OpenCL part, without going for the whole graphics stack:

$ # Omitting installation steps for the build-dependencies (devscripts, dh-modaliases, ...)
$ unzip 15.302.2301-linux-retail_end_user.zip  # This is the "Pro" version for my FirePro, YMMV
$ cd fglrx-15.302.2301
$ sudo ./amd-driver-installer-15.302.2301-x86.x86_64.run --buildpkg Ubuntu/xenial --NoXServer

This builds fglrx-core_15.302-0ubuntu1_amd64.deb, but not without a fair share of issues: the kernel module fails to build against current kernels; clinfo runs once and segfaults ever after (an old bug resurfacing due to AMD forgetting to add /etc/ati/amdpcsdb.default to the package); and, like AMDGPU-PRO, the package conflicts with the Ubuntu 16.04 OpenCL packages. Sigh.

Still, with some patching documented here I managed to install the fglrx driver and got full OpenCL support:

$ clinfo -l
Platform #0: AMD Accelerated Parallel Processing
 +-- Device #0: Capeverde
 `-- Device #1: Intel(R) Core(TM) i7-4910MQ CPU @ 2.90GHz
Platform #1: Intel Gen OCL Driver
 `-- Device #0: Intel(R) HD Graphics Haswell GT2 Mobile
Platform #2: Clover

My joy however lasted until the moment I unplugged my laptop from AC. Uptime on battery plummeted from the usual 4 to 5 hours to 1:30, while I’m not even using the card. There’s probably a way to switch it off, but I’m not inclined to spend hours again to find the next workaround for AMD’s failure to get things right.

Update I have since installed TLP, which works fabulously and gets me 6 to 7 hours on battery. Haven’t yet checked though if it also manages to kill the power to fglrx, as I somehow guess I already know the answer.

Oibaf’s drivers

An answer on AskUbuntu suggests Oibaf’s drivers as a promising alternative for fglrx. I haven’t looked at these yet.

Installing the Intel and AMD SDKs

AMD distributes the AMD Accelerated Parallel Processing (APP) SDK, and Intel has its Intel SDK for OpenCL Applications.

I’m not using these as I’m currently only interested in OpenCL as a user, not as a developer. Maybe more on the SDKs in the future.

More to Explore

StarPU
erlang-cl, pyopencl
ArrayFire
…

Appendix I: AMDGPU-PRO install notes

The organisation of the AMDGPU-PRO debian packages (downloadable from the official page) is promising: there is a main amdgpu-pro package which depends on amdgpu-pro-computing and amdgpu-pro-graphics, separating the headless from the gamers.

However AMD need to fix the packaging before AMDGPU-PRO can leave beta. Currently (release 16.30.3-315407) the packages conflict (without declaring ‘Conflicts’) with packages in Ubuntu and so fail halfway installation, leaving you with a stuck APT.

These things need fixing in the AMDGPU-PRO (Beta) Debian packages²:

amdgpu-pro-computing depends on amdgpu-pro-clinfo, which conflicts with clinfo as it replaces the clinfo tool (by one that has lots less functionality). Solution:
- amdgpu-pro-computing should Depends: clinfo | amdgpu-pro-clinfo (or better: Recommends, unless the clinfo utility is indispensable for the proper functioning of the package).
- amdgpu-pro-clinfo must Conflicts: clinfo.
amdgpu-pro-clinfo depends on amdgpu-pro-libopencl1, which conflicts with ocl-icd-libopencl1 as it replaces the libOpenCL library. Solution:
- amdgpu-pro-clinfo should Depends: ocl-icd-libopencl1 | amdgpu-pro-libopencl1. In fact, is amdgpu-pro-libopencl1 needed at all? The libOpenCL library is part of the common OpenCL infrastructure (the top layer described above) and needn’t be provided by vendor ICDs.
- amdgpu-pro-libopencl1 must Conflicts: ocl-icd-libopencl1
amdgpu-pro-computing depends on amdgpu-pro-libopencl-dev, which conflicts with ocl-icd-opencl-dev as it replaces the libOpenCL.so symlink. It shouldn’t do this as that symlink file is part of the common OpenCL infrastructure. Solution:
- amdgpu-pro-computing should Depends: ocl-icd-opencl-dev | amdgpu-pro-libopencl-dev, or leave out the amdgpu-pro-libopencl-dev altogether (for same reasons as explained above).
- Moreover, amdgpu-pro-computing probably shouldn’t depend on the -dev library but rather on the runtime library ocl-icl-libopencl1, as the -dev package is a build dependency.
amdgpu-pro-opencl-icd is missing the /etc/ld.so.conf.d/amdgpu.conf file. This file is provided by amdgpu-pro-core, which however also blacklists the radeon driver (as described above), thus preventing a ‘CPU-only’ install. Solution:
- Move the ld.so.conf.d/amdgpu.conf to package amdgpu-pro-opencl-icd as it should logically be installed together with the libraries in that package. The package provides the CPU device.
- Create a new amdgpu-pro-opencl-gpu-icd package to provide the GPU device, and make that package (as well as amdgpu-pro-graphics) depend on the core package which has the driver and the radeon blacklist.

Appendix II: graphics switching

This section is about OpenGL rather than OpenCL. I include it to document the simplicity of switching to the discrete GPU for rendering. All it requires is setting the environment variable DRI_PRIME.

Note: moved to the appendix because it is off-topic. Also, after a number of apt-get updates the Gallium rendering gives a black window and its framerate is actually lower (11000) than that for Mesa (12900).

$ xrandr --listproviders
Providers: number : 3
Provider 0: id: 0x72 cap: 0x9, Source Output, Sink Offload crtcs: 4 outputs: 5 associated providers: 2 name:Intel
Provider 1: id: 0x49 cap: 0x6, Sink Output, Source Offload crtcs: 6 outputs: 4 associated providers: 2 name:VERDE @ pci:0000:01:00.0
Provider 2: id: 0x49 cap: 0x6, Sink Output, Source Offload crtcs: 6 outputs: 4 associated providers: 2 name:VERDE @ pci:0000:01:00.0

$ DRI_PRIME=0 glxgears -info | grep -E '(GL_RENDERER|FPS)'
GL_RENDERER   = Mesa DRI Intel(R) Haswell Mobile        # Intel renders
305 frames in 5.0 seconds = 60.871 FPS                  # at rate synced with monitor

$ DRI_PRIME=1 glxgears -info | grep -E '(GL_RENDERER|FPS)'
GL_RENDERER   = Gallium 0.4 on AMD CAPE VERDE (DRM 2.43.0, LLVM 3.8.0)
70041 frames in 5.0 seconds = 14008.121 FPS             # AMD renders ... faster

$ DRI_PRIME=2 glxgears -info | grep -E '(GL_RENDERER|FPS)'
GL_RENDERER   = Mesa DRI Intel(R) Haswell Mobile 	# Surprise: Intel renders
53971 frames in 5.0 seconds = 10793.093 FPS		# without the syncing

When DRI_PRIME is 0 or unset, the Intel is selected. Its rendering frequency is 60Hz, synchronised with the monitor refresh rate. DRI_PRIME=1 selects the AMD discrete card. DRI_PRIME=2 (for some reason) selects the Intel, but without synchronised rendering.

I was surprised to see the high frame rate on the IGD. In fact, with current package versions, the IGD actually wins. Just because it is an integrated GPU, doesn’t mean that it can’t be used for computing.

Note: According to this page it is required to do xrandr --setprovideroffloadsink 0x49 0x72 (where the hex numbers are the card IDs from the xrandr --listproviders output) to make DRI_PRIME work, but I get the exact same results when I don’t. (May be related to DRI3?)

Footnotes

For out-of-distro software or cutting-edge versions I use GNU Guix which perfectly isolates these without being the kludge that containers are. ↩
AMD: if you’re reading this, I’m happy to work with you to sort this out. ↩

Total Perspective Vortex

Mon, 11 Jul 2016 00:00:00 +0300

As a child I read about the Milky Way but never saw it. I didn’t realise that living at latitude 52N in a country where the sky is always lit, meant that I couldn’t easily see it. Last week, looking up at the pitch-black midnight sky from a beach on the desolate Northeastern Tanzanian coast, I was suddenly aware that that long and narrow cloud stretching out over the sky, wasn’t a cloud after all.

It was the Milky Way, and it’s actually milky. It’s a diffuse cloud of uncountably many tiny lights! I used to think that I had a good sense of how incredibly many stars there are out there, but this just blew the lid off my perspective. Here I was on a mostly harmless little planet orbiting its star, looking at the other 100,000,000,000 stars in our galaxy.

Once you’ve grasped that perspective on Earth’s sheer tininess and somewhat come to terms with our galactical insignificance, let’s put that perspective in perspective: the 100,000,000,000 stars you see are only our galaxy. There are 200,000,000,000 galaxies just like it.

Having seen Earth in its place relative to the universe as a whole, take a look at the place of us humans vis-a-vis all other life on Earth. A number of web sites catalogue the so-called Tree of Life, the evolutionary tree branching out into all known (and many more unknown) species. OneZoom does this in a very pretty way.

Spend some time navigating the tree, then try to find us, homo sapiens. You probably won’t, it’s a long way down. Welcome to the Evolutionary Total Perspective Vortex.

The Automator

Wed, 03 Feb 2016 00:00:00 +0300

In the early days of commercial IT in the Netherlands, before English became the standard vocabulary in the field that was later to be called “IT”, the Dutch language had its own word for IT. IT was called automation (automatisering) and people working in automation were called automators (automatiseerders).

The words now sound old-fashioned and have gotten into disuse to the extent that my children would conjure up an image of Schwarzenegger or Doofenschmirz if I would tell them I am an automator.

I like the term “automation” for our field of work. It stresses the purpose of what we do rather than the means we use. I am very much an automator.

The first time around explore, the second time recognise repetition, the third time automate.

Much of what I keep in my public repositories are shell scripts wrapping tedious, error-prone, or repetitious operations. I initially wrote these for my own use, but I’ve tried to make them useful for others¹. They are operationalised documentation. When you need to keep notes anyway², it pays to put in the extra effort to condense all NTK into an executable script. It’s like having unit tests for your notes.

Footnotes

taxo to search and browse the NCBI taxonomy, unfasta for natural handling of FASTA in command line pipes and filters, and a collection of scripts built on top of BLAST in blast-galley to e.g. cut genes out of an unannotated assembly or do simple in-silico PCR. ↩
You don’t? I’d hold off the cheers until you cross the age of forty. ↩

Introducing Unfasta

Fri, 22 Jan 2016 00:00:00 +0300

Unfasta is a suite of command-line utilities for working with sequence data.

The rationale behind unfasta is to have the ability to process genomic sequence data using simple standard utilities like grep, cut and head, in the common pipes and filters style of Unix and GNU.

For instance,

# Compute the GC content of all sequences in a FASTA file
uf 'file.fa' | sed -n '2~2p' | tr -dc 'GC' | wc -c

In that pipeline,

uf reads a FASTA file and outputs it in ‘unfasta’ format, collapsing sequence data to single lines;
sed -n 2~2p filters every second line from its input, thus dropping the header lines;
tr -dc GC drops from its input all characters except G and C;
and wc -c counts the number of characters it reads, then writes this to standard output.

Pipelines are a simple and powerful way to process large streams of data, but the FASTA format is the party pooper. By allowing sequences to span multiple lines, FASTA defies processing by line-oriented standard tools. Even a seemingly obvious fgrep -q 'GAATCATCTTTA' fails with a false negative in 10-15% of cases. Unfasta originated from frustration over this missed opportunity.

Unfasta resolves the issue by converting FASTA format to ‘unfasta format’ when it enters the pipeline. The unfasta format is FASTA without line breaks in the sequence data. Note that unfasta files are still valid FASTA files.

Some examples to illustrate my case:

# Extract all deflines using sed or awk
$ sed -n 1~2p
$ awk 'NR%2==1'

# Extract the data for the 3rd and 12th sequence
$ sed -n 7,25p

# Extract the header and sequence data for identifier 'gi|22888'
$ sed -n '/>gi|22888[^0-9]/,+1p'

# Extract the bases at positions 952-1238 in the first sequence
$ sed -n 2p | cut -b 952-1238

# Extract a 500 base fragment at position 135
$ tail -c +135 | head -c 500	# or: cut -b 135-$((134+500))

# How long are the Borrelia sequences?
$ awk '/Borrelia/ { getline; print length; }'

# Does any sequence contain fragment 'ACGTATAGCGGC'? 
$ fgrep -q 'ACGTATAGCGGC' && echo "Yes" || echo "No"

Obviously the law of conservation of misery applies as you now need to memorise nutty sed commands. But that can be mitigated by wrapping the complexity in little self-explanatory shell scripts, which is what I did in unfasta.

Some examples (all assume FASTA arriving on stdin):

# The starting point is 'uf', which turns line-broken FASTA into unbroken FASTA
$ uf --help
  ... concise but complete description ...

# All uf-utils are shell scripts, so there's always the source
$ cat `which uf`
  ... skip 60 lines of boilerplate to see a ...
  ... handful of simple lines do the actual work ...

# How many sequences?
$ zcat seqs.fsa.gz | uf | uf-headers | wc -l

# Is the input good, and what are the base counts?
$ uf | uf-valid --dna -s | uf-freqs  -t
  >NODE_1_length_388420_cov_15.7201
  388420 A=117181 C=82016 G=72519 T=116704
  ...
  >TOTALS
  4101446 A=1255608 C=811259 G=785749 T=1248830

# Select only the first sequence, or the one whose defline matches a regex
$ uf | uf-select 1
$ uf | uf-select 'Acinetobacter'

# Various ways of cutting from a sequence
$ uf | uf-cut 1200/1000   # Select 1kbp starting at position 1200
$ uf | uf-cut 5:-5        # Trim four elements off of both ends
$ uf | uf-circut -50:50   # Select 100b around the start of a circular sequence
$ uf | uf-circut 100:99   # Rotate to start at base 100 (= wrappingly select all)

# Process just the content of each sequence, temporarily removing the FASTA deflines
$ uf file.fna | uf-bare | ..processing.. | uf-dress -r <(uf file.fna | uf-headers)

# More at
$ git clone 'https://github.com/zwets/unfasta'

Unfasta won’t work for everyone. It does for me because I work in bash most of the time, and have used the GNU toolset for ages. Over the years I have written software in at least a dozen ‘proper’ programming languages, but when it comes to string processing nothing beats piping together a one-liner in bash.

If you recognise this, then you know already. If not, spend some time with the Art of the Command Line and find enlightenment.

Find Unfasta on GitHub.

Static sites the next big thing

Wed, 06 Jan 2016 00:00:00 +0300

It says so in the email I just received.

From: Simon St. Laurent <oreilly@post.oreilly.com>
Date: Wed,  6 Jan 2016 03:06:03 -0800
Subject: Free ebook: why static sites are the next big thing

Hi Marco,

Given the choice between static and dynamic websites, why wouldn't you choose
to serve custom, dynamically generated content to each visitor to your site?
Yet to the surprise of many, static websites are coming back. Having been
ridiculed for years, they now fulfill a vital need for bloggers and others
who simply want to disseminate information.

Static site generators — the new breed of tools for creating static sites — 
simplify the once-tedious process of building and deploying these basic sites.
Get the lowdown on these tools and see what's behind the return of the static
site with Static Site Generators, a new report by Brian Rinaldi.

Get the free copy.

Simon St. Laurent
Strategic Content Director

Right when I’m getting serious with Jekyll. There, skipped HTML 2 through 5 but now I’m back ahead of the pack. For completeness: the free copy referred to can be got here.