Posts on Hannes Hauswedell

Owning and non-owning C++ Ranges

Sat, 17 May 2025 16:00:00 +0200

This is the first article in a series discussing some of the underlying properties of C++ ranges and in particular range adaptors. At the same time, I introduce the design of an experimental library which aims to solve some of the problems discussed here.

Preface

C++ Ranges are one of the major features of C++20. See my previous post for a detailed introduction. I am a big fan of C++ Ranges and have used them extensively in multiple projects (mostly bioinformatics, data science, …), but I have had difficulty convincing other people of the benefits and have seen many programmers struggle with even basic usage. Furthermore, there have been significant conceptual changes to the Ranges part of ISO standard after initial publication of C++20, showing the volatility of the design and making it even more difficult for non-expert users to understand what’s going on. Even now, there are still controversial discussions within the ISO C++ committee regarding vary basic aspects of C++ Ranges and views.

I see multiple challenges with the current design—that I will each write a blog post on:

Owning and non-owning ranges.
tbd
…

I will try give just enough detail to keep the read light, while providing enough information to understand what all the fuss is about! Finally, I will introduce my own experimental library design to address some of the challenges discovered. This is the first article in the series.

DISCLAIMER: I have been using Ranges for a while, but I am very aware that other people did the heavy lifting of bringing them into the standard. It is quite possible that there are things I haven’t considered. I also think it was right to try and improve the design of views. In fact, I have reversed my own opinion on several problems several times, and I am sure I wouldn’t have come to the conclusions drawn here without the previous iterations of the design. Or to use a wiser person’s words:
“Why do you go away? So that you can come back. So that you can see the place you came from with new eyes and extra colors. […] Coming back to where you started is not the same as never leaving.” — Terry Pratchett, A Hat Full of Sky

C++17 Standard Library

We will begin by having a look at ranges from the standard library prior to C++20, since this is what people are most used to. Note that although the ranges themselves are from C++17, I will use some terminology/concepts/algorithms introduced later to explain how they relate to each other. Remember that to count as a range in C++, a type needs to have just begin() and end(). Everything else is bonus 😉.

Multi-pass ranges | Containers (owning)

Containers are the ranges everybody already used before Ranges were a thing. They own their elements, i.e. the storage of the elements is managed by the container and the elements disappear when the container does. Containers are multi-pass ranges, i.e. you can iterate over them multiple times and will always observe the same elements.¹

A string of characters as the example for a container.

This rendering shows how you can imagine a “container”: a bunch of elements (letters) and some control data structures (in this case a cord and pegs). There is a clear begin (first letter) and end (end of cord). Importantly, the elements are attached to the control data structures. When you move the whole thing, the elements and the container move together. Take down the cord, and the letters disappear.

/* construction and copying */
std::string s  = "foobar";      // copies characters from static storage into string [O(n)]
std::string s2 = s;             // copies all characters from s into s2              [O(n)]

/* find the smallest character in a string */
char c = *std::ranges::min_element(s);
assert(c == 'a');

/* the same thing with a temporary */
// char d = *std::ranges::min_element(std::string{"foobar"});
// assert(d == 'a');

Examples illustrating “container semantics”.

Although the "foobar" string literal itself has static storage, a new string is allocated² and the characters are copied into it. Now the container manages the memory of the elements. Copying the container itself, copies the elements (O(n) complexity).¹

We can subsequently call std::ranges::min_element on it to get the iterator pointing to the lexicographically smallest character. The same does not work for a temporary of a string, because its elements go out-of-scope together with the container when min_element returns; the iterator would be dangling.³

Multi-pass ranges | Non-owning

If containers are owning ranges, what are non-owning ranges? C++17 introduced a first example: std::string_view, a range that consists just of a begin and end pointer⁴ into another range’s elements.

In general, non-owning ranges are the subset of ranges that do not manage the memory of their elements. std::string_view is such a range, and it even fulfils the stricter requirements of a borrowed range: A range that stores nothing beside the iterators, and where the iterators may not refer back to the range object. This has the important implication that the iterators of a borrowed range remain valid when the borrowed range itself goes out-of-scope. For a string_view, this design seems natural, but we will later encounter ranges where a trade-off has to be made.

If the string of letters is a container, you can imagine a string_view as the shadow cast by a string.

The above image illustrates the string_view as the shadow of a string: You can read the same letters in the shadow and observe very similar properties to the string (sans colour in my example 😅). But the lifetime of the shadow fundamentally depends on the object casting the shadow. If the latter is moved or destroyed, the shadow disappears, as well, and the string_view becomes dangling.

/* construction and copying */
std::string_view s  = "foobar";      // "binds" s to the static storage   [O(1)]
std::string_view s2 = s;             // "binds" s2 to the static storage  [O(1)]

/* find the smallest character in a string */
char c = *std::ranges::min_element(s);
assert(c == 'a');

/* the same thing with a temporary */
char d = *std::ranges::min_element(std::string_view{"foobar"});
assert(d == 'a');

Examples illustrating “borrowed range semantics”.

The std::string_view s1 binds to the static storage of the string literal; no allocation or character copies take place. This is also true for s2 which refers to the same static storage (and not s!). Thus, copying string_views is in O(1). While it is particularly useful to wrap string literals in std::string_view, you can also have a std::string_view refer to a std::string, a substring of a std::string or even a std::vector<char>.

Finding the smallest character works in the same way as for the std::string, however, we can now also search a temporary, because iterators into that string_view are not dependent on the string_views’s lifetime (and the literal has static storage).

Single-pass ranges

A single-pass range is one that does not offer the multi-pass guarantee, i.e. you may not be able to iterate over it multiple times, or might encounter different elements when you do. Iterating over the range, changes the range. It is not even guaranteed that you can call .begin() more than once.

Think of a single-pass range of characters like a printer printing pages with letters on them.

Conceptionally, a single-pass ranges is more “a machine that produces elements” than “a collection of elements”. This is illustrated by the printer in the image above. You can use it to generate a series of pages, but the pages aren’t part of the printer, and the printer is in a different state after each page printed. Printing could go on indefinitely, but the printer might also just stop working. Some printers may be restarted, others not 😁

No dedicated single-pass range exists in pre C++20 times, and few single-pass iterators do, e.g. std::istream_iterator which exposes the characters of an input stream.

The relationship of single-pass ranges to the question of “owning vs non-owning” is complicated:

They are often move-only types that cannot be copied (neither O(1) nor O(n)).
It is difficult to argue if the elements are “owned” at all, since they are usually “produced” and “consumed” immediately; iterators to previous elements become invalid.
It doesn’t make sense to borrow from single-pass ranges, because you can only have one valid iterator anyway.⁵

Thus I suggest keeping single-pass ranges as their own separate category, and will not cover them further in this blog post.¹

C++2x Standard Library

C++20 is when ranges actually became a thing, and the entire machinery including concepts and algorithms became part of the standard library. But the significant new features that were introduced are range adaptors. Range adaptors are ranges that are created on top of other ranges,⁶ typically exposing a subset of the underlying range or a transformation thereof. See my previous post for an introduction.

std::vector vec{1, 2, 3, 4};
auto v = vec | std::views::VIEW1 | std::views::VIEW2; // non-owning multi-pass range adaptor

This is what a typical “pipeline” looks like. It creates v which is a range adaptor on vec, applying the transformations implied by VIEW1 and VIEW2. While std::string_view was mainly used for string literals and sub-strings, the operations we can now do are much more powerful and include

filtering (e.g. “only non-space characters”)
transformations (e.g. “to upper case”)
splitting and joining (e.g. “split into words”, “join with different delimiter”)

Many of these adaptor have state, e.g. the filtering adaptor needs to store the predicate for filtering, and the joining adaptor needs to store the delimiter for joining; other adaptors have a cache. This state can either be stored in the adaptor, or in the iterator(s) of the adaptor. The former can save space in the iterator, but prevents the adaptor from being a borrowed range (the iterator needs to store a pointer to the adaptor to access the state).

Views in range-v3 and original C++20

The standard library ranges design is based on the range-v3 library by Eric Niebler. This library popularised the term view.

(Range-v3-)Views are defined as ranges that are copyable in O(1). This definition for “non-owning” is broader than for borrowed ranges; views can store state within the range, as long as it is in O(1). Borrowed ranges are thus a subset of views.⁷

I will discuss below whether this additional flexibility is worth it, but it should be noted that—so far—the definition of view is still clear and primarily relates to ownership: it means “copyable in O(1)”! Note however that already at this point, “view” did not mean “range adaptor”. Both range-v3 and early C++20 already contain standalone ranges that are also called views, e.g. std::ranges::iota_view and std::ranges::empty_view (which are also copyable in constant time).

Views post P2415 (current state)

Just before C++20 landed, the committee accepted P1456 which allowed views to be move-only¹, and then, after C++20 was released, P2325 was applied as a defect report to C++20 which allows views to not be default-constructible. I will discuss both of these in the next blog post.

However, the most important (and controversial) change came by way of P2415, which allowed views to become owning ranges. It was also applied to C++20 as a defect report, although it was quite a significant design change.

auto to_upper = [] (unsigned char c) { return std::toupper(c); };
std::string s = "foobar";

/* this has always been possible; v_indiri depends on s */
auto v_indiri = s | std::views::transform(to_upper);

/* this is "new"; v_owning is standalone */
auto v_owning = std::move(s) | std::views::transform(to_upper);

This is a useful feature, however, it resulted in the std::ranges::view concept being changed to where it no longer means “non-owning range”. While the concept still provides some value in combination with other concepts,¹ I don’t see it being used outside the standard library’s own view machinery. In particular, I don’t think anybody uses it to constrain their algorithms, which generally is the whole point of a concept.⁸

Beyond these (im-)practical implications, there is a severe lack of clarity/teachability. Since the previous definition was so widespread (and memorable), I still encounter experienced C++ programmers and even committee members who are surprised that view does not imply “non-owning range”. Considering that “views” are one of the biggest selling points of C++ Ranges, not being able to explain what “view” means is a serious problem.

But since it’s too late to “fix” the definition (and changing it again would likely just increase confusion), I suggest avoiding the term “view” altogether.

Use “range adaptor” to describe range types that operate on (or wrap) other ranges.
Use “range adaptor object” to describe the objects that can be chained in pipelines to create “range adaptors”.⁹
Use “non-owning range” to denote a range that is copyable in O(1).
Use “borrowed range” to denote a range that is copyable in O(1) and where the iterators can outlive the range.

The RADR library

📡

https://github.com/h-2/radr

I have just made public my own experimental ranges library, called “ℝange 𝔸𝕕aptors ℝeimagined” (radr).

It builds on C++20 concepts and algorithms but replaces std::views::*—while offering very similar usage patterns.

Please check it out and provide feedback! I will only cover the design choices regarding ownership in this post, but it addresses numerous other issues, and more blog posts will follow. There is extensive documentation in the docs-folder of the github repo.

Non-owning range adaptors

All non-owning (multi-pass) range adaptors in radr are borrowed ranges. This means that all functionality is implemented in terms of the iterator-sentinel-pair, and that the ranges store no further state. The cost of this is an increase in iterator size, but it dramatically reduces overall complexity and provides a much simpler mental model to users:

We have one concept of “non-owning range”, not two. The view concept is entirely irrelevant in our library.
The “borrowed range” definition is both the strongest and the most useful “non-owning range” definition.
If you put a borrowed range into our range pipelines, you always get a borrowed range out (this is not true for std::).

While I do believe that “borrowed range” is superior to even the old/range-v3 view concept, being able to avoid the new/confusing view concept was a major motivation for making radr.

/* Finding the smallest non-negative number in a vector */
std::vector vec{-1, 2, -3, 1, 7};
auto non_neg = [] (int i) { return i >= 0; };

auto it1 = std::ranges::min_element(vec | std::views::filter(non_neg));
// assert(*it == 1);   // broken, because filter_view is never borrowed

auto it2 = std::ranges::min_element(std::ref(vec) | radr::filter(non_neg));
assert(*it == 1);      // works, because all radr adaptors on lvalues are

This example highlights one of the practical advantages of the design choice.

The syntax for creating an indirection in radr:: is explicit (it requires using std::ref(); more on this in a future blog post), but otherwise the usage patterns (pipes etc) are identical to std::.

The important difference is that the returned range always models “borrowed range”, so you can reliably re-use the iterators. This is similar to how Boost range-v2 designed their adaptors. In the standard library, some adaptors return borrowed ranges, but many important ones like filter do not. Whether or not a specific adaptor does, is hard to predict for users.¹⁰

In general, RADR tries to follow the example of std::string_view for non-owning range adaptors. It has been around since C++17; people understand it; people like it.

Owning range adaptors

While P2415 is often criticised for introducing “owning views”, I don’t think having owning range adaptors is the problem (and can indeed be quite useful!). The problem is that the definition of “view” was changed to also include “owning range adaptors”; thereby removing the basic, core part of its definition, i.e. being a non-owning range.¹¹

By all prior art, a range that manages the memory of its elements is a container. A range adaptor that wraps an rvalue of a container, thereby managing its memory including the elements… is also a container… because it contains the elements! No concept is necessary to express this. The range concept by default implies that a range can have arbitrary ownership; the only things that needs to be modelled explicitly, are, whether a range is non-owning and if it is borrowed.

std::

/* create adaptor on temporary */
auto vue0 = std::vector{1, 2, 3} | std::views::take(2);

/* move existing */
std::vector vec{1, 2, 3};
auto vue1 = std::move(vec) | std::views::take(2);

radr::

/* create adaptor on temporary */
auto rad0 = std::vector{1, 2, 3} | radr::take(2);

/* move existing */
std::vector vec{1, 2, 3};
auto rad1 = std::move(vec) | radr::take(2);

With respect to wrapping rvalues of containers, you can do the same things with radr as with the standard library, and the syntax is identical. But RADR considers the returned range a container, and you can copy it in O(n).¹

Summary

There are owning and non-owning ranges, and borrowed ranges are a stricter version of the latter. “View” used to be synonymous with “non-owning range”, but the view concept was redefined multiple times and is not useful by itself anymore. It is best to use other, unambiguous terms to describe ranges. On the other hand, we have established range adaptor as a term for ranges doing something on top of other ranges. This is orthogonal to ownership, i.e. the range adaptor can point to the underlying range—or wrap it.

Two important design questions can be distilled from this:

How are non-owning range adaptors implemented?
- Are all/some/none of them borrowed?
Do we want owning range adaptors, and if so, how do we model those?

The standard library, like range-v3, chooses to have some non-owning range adaptors that are borrowed, but most are not. RADR, like Boost range-v2, chooses to make all non-owning range adaptors borrowed ranges.

The standard library and RADR offer owning range adaptors while Boost range-v2 and range-v3 do not. Owning range adaptors in the standard library are a specific kind of view; in RADR they are simply containers.

I did my best to break this complicated topic down, but I hope that I also explained that “views” being hard to explain is part of the problem. The radr library offers very similar features to std::views but tries to avoid some of the complexity.

We will explore this in more depth in the next blog post. ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
Most containers store their elements on the heap, but std::array and C++26’s std::inplace_vector store on the stack. std::string is a special case, because small strings are stored on the stack and bigger ones on the heap. However, both cases lead to undefined behaviour. ↩︎
Actually, std::ranges::min_element is smart enough to know that the iterator would dangle and returns std::ranges::dangling instead. This prevents undefined behaviour, but it also results in us not getting the smallest character. ↩︎
Or a begin + size, which is equivalent. ↩︎
Subsequent blog posts will explain that you cannot have a “read-only” borrow on a single-pass range, because single-pass ranges are not const-iterable. In addition, having a mutable borrow on a single-pass range provides no benefits over just passing the range itself; you cannot use the range anymore once you have borrowed from it. ↩︎
There is some ambiguity as to whether “range adaptor” refers to the returned type (e.g. specialisations of std::ranges::reverse_view<>) or the thing that creates the type (e.g. std::views::reverse) or both. I use “range adaptor” to refer to the type and “range adaptor object” to refer to the object used in a pipeline. The same ambiguity exists for “view”, as will be discussed later. ↩︎
If one follows the standard very literally, borrowed ranges are not required to not have any state. However, since a borrowed forward range’s iterators are copyable in O(1) and they are completely independent of the range, one could simply take the iterators and make a new range out of them (that would then be copyable in O(1)). ↩︎
Eric Niebler, the father of C++ Ranges, once said: »Generic Programming pro tip: Although Concepts are constraints on types, you don’t find them by looking at the types in your system. You find them by studying the algorithms.« People not using the concept, suggests that it is not useful. It is also noteworthy that this change was not back-ported into range-v3. ↩︎
Boost ranges calls these “range adaptor generator”, but the term “generator” is now associated with single-pass ranges. ↩︎
Recently, papers such as P3117 have suggested more range adaptors should have been made borrowed. ↩︎
To be fair, the options were quite limited since P2415 was introduced so late (after C++20 was released). ↩︎

A fast fileserver with FreeBSD, NVMEs, ZFS and NFS

Fri, 26 Apr 2024 16:30:00 +0200

I have a small server running in my flat that serves files locally via NFS and remotely via Nextcloud. This post documents the slightly overpowered upgrade of the hardware and subsequent performance / efficiency optimisations.

TL;DR

I can fully saturate a 10Gbit LAN connection, achieving more than 1100 MiB/s throughput.
I can perform a zpool scrub with 11 GiB/s, completing a 6.8TiB scrub in 11min.
Idle power usage can be brought down to 34W.

Old setup and requirements

What the server does:

Serve files via NFS to
- my workstation (high traffic)
- a couple of Laptops (low traffic)
- the TV running Kodi (medium traffic)
Host a Nextcloud which provides file storage, PIM etc. for a handful of people

Not a lot of compute is necessary, and I have tried to keep power usage low. The old hardware served me well really long:

AMD 630 CPU
16GiB RAM
2+1 * 4TiB spinning disk RAIDZ1 with SSD ZIL (“write-cache”)

The main pain point was slow disk access resulting in poor performance when large files were read by the Nextcloud. Browsing through my photo collection via NFS was also very slow, because thumbnail generation needed to pull all the images. Furthermore, low speed meant that I was not doing as much on the remote storage as I would have liked (e.g. storing games), resulting in my workstation’s storage always running out. And I was just reaching the limits of my ZFS pool anyway, so it was time for an upgrade!

New setup

To get better I/O, I thought about switching from HDD to SSD, but then realised that SSD performance is very low compared to NVME performance, although the price difference is not that much. Also, NFS+ZFS leads to quite a bit of I/O, typically requiring the use of faster caching devices, further complicating the setup. Consequently, I decided to go for a pure NVME setup. Of course, the new server would also need 10GBit networking, so that I can use all that speed in the LAN!

This is the new hardware! I will discuss the details below.

Mainboard, CPU and RAM

The main requirement for the mainboard is to offer connectivity for four NVME disks. And to be prepared for the future, I would actually like 1-2 extra NVME slots. There are two ways to attach NVMEs to a motherboard:

directly (“natively”)
via an extension card that is plugged into a PCIexpress slot

Initially, I had assumed no mainboard would offer sufficient native slots, so I did a lot of research on option 2. The summery: it is quite messy. If you want to use a single extension card that hosts multiple NVMEs (which is required in this case), you need so called “bifurcation support” on the mainboard. This lets you e.g. put a PCIe x8 card with two NVME 4x disks into a PCIe 8x slot on the mainboard. However, this feature is really poorly documented,¹ and and varies between mainboard AND CPU whether they support no bifurcation, only 8x → 4x4x or also 16x → 4x4x4x4x. The different PCIe versions and speeds, and the difference between the actually supported speed and the electrical interface add further complications.

In the end, I decided to not do any experiments and look for a board that natively supports a high number of NVME slots. For some reasons, this feature is very rare on AMD mainboards, so I switched to Intel (although actually I am a bit of an AMD fanboy). I probably could have gone with a board that has 5 slots, but I use hardware for a long time and wanted to be safe, so I took board that has 6 NVME slots (2 free slots):

ASRock Z790 NOVA WiFi

None of the available boards had a proper² 10GBit network adaptor, so having a usable PCIe slot for a dedicated card was also a requirement. It is important to check whether PCIe slots can still be used when all NVME slots are occupied; sometimes they internally share the bandwidth. But for the above board this is not the case.

Important: To be able to boot FreeBSD on this board, you need to add the following to /boot/device.hints:

hint.uart.0.disabled="1"
hint.uart.1.disabled="1"

For the CPU, I just went with something on the low TDP end of the current Intel CPU range, the Intel Core i3-12100T. Four cores + four threads was exactly what I was looking for, and 35W TDP sounded good. I paired that with some off-the-shelf 32GiB RAM kit.

Case, power supply & cooling

Strictly speaking a 2U case would have been sufficient, but I thought a 3U case might offer better air circulation. I ended up with the Gembird 19CC-3U-01. For unknown reasons, I chose a 2U horizontal CPU fan, instead of a 3U one. The latter would definitely have provided better airflow, but since the fan barely runs at all, it doesn’t make much of a difference.

I was unsuccessful in finding a good PSU that is super efficient in the average case of around 40W power usage but also covers spikes well above 100W, so I just chose the cheapest 300W one I could get :)

The case with everything in place.

The built in fans are very noisy. I chose to replace one of the intake fans with a spare one I had lying around and only connect one of the rear outtake fans. But I added an extra fan where the extension slots are to divert some airflow around the NIC—which otherwise gets quite warm. This should also blow some air over the NVME heatsinks! All fans can be regulated and fine-tuned from the BIOS of the mainboard which I totally recommend you do. At the current temperatures and average workloads the whole setup is almost silent.

Storage

Now, the fun begins. Since I needed more space than before, I clearly want a 3+1 x 4TiB RAIDZ1.

My goal was to be able to saturate a 10GBit connection (so get around 1GiB/s throughput) and still have the server be able to serve the Nextcloud without slowing down significantly. Currently the WAN upload is quite slow, but I hope to have fibre in the future. In any case, I thought that any modern NVME should be fast enough, because they all advertise speeds of multiple GiB/s.

Choice of disks

Anyway, I got two Crucial P3 Plus 4TB (which were on sale at Amazon for ~190€), as well as two Lexar NM790 4TB (which were also a lot cheaper than they are now). My assumption that that they were very comparable, was very wrong:

Disk	IOPS rand-read	IOPS read	IOPS write	MB/s read	MB/s write	“cat speed” MB/s
Crucial	53,500	794,000	455,000	2,600	4,983	~700
Lexar	53,700	796,000	456,000	4,578	5,737	~2,700

I used this fellow’s fio-script to generate all columns except the last. The last column was generated by simply cat’ing a 10GiB file of random numbers to /dev/null which roughly corresponds to the read portion of copying a 4k movie file. Since I had two disks each, I actually took the time to test all of them in different mainboard slots, but the results were very consistent: in real-life tasks, the Crucial disk underperformed significantly, while the Lexar disks were super fast. I decided to return the Crucial disks and get two more by Lexar 😎

Disk encryption

I always store my data encrypted at rest. FreeBSD offers GELI block-level encryption (similar to LUKS on Linux). But OpenZFS also provides a dataset/filesystem-level encryption since a while. I previously used GELI, but I wanted to switch to ZFS native encryption, because it provides some advantages:

Flexibility: I can choose later which datasets to encrypt; I can encrypt different datasets with different keys.
Zero-knowledge backups: I can send incremental backups off-site that are received and fully integrated into the target pool without that server ever getting the decryption keys.
Forward-compatibility: I can upgrade to better encryption algorithms later.
Linux-compatibility: I can import the existing pool in a Linux environment for debugging or benchmarking.

However, I had also heard that ZFS native encryption was slower, so I decided to do some benchmarks:

Disk	IOPS rand-read	IOPS read	IOPS write	MB/s read	MB/s write	“cat speed” MB/s
no encryption	54,700	809,000	453,000	4,796	5,868	2,732
geli-aes-256-xts	40,000	793,000	446,000	3,332	3,334	952
zfs-enc-aes-256-gcm	26,100	513,000	285,000	3,871	4,648	2,638
zfs-enc-aes-128-gcm	29,300	532,000	353,000	3,971	4,794	2,631

Interestingly, GELI³ performs much better on the IOPS, but much worse on throughput, especially on our real-life test case. Maybe some smart person knows the reason for this, but I took this benchmark as an assurance that going with native encryption was the right choice.⁴ One reason for the good performance of the native encryption seems to be that it makes use of the CPU’s avx2 extensions.

At this point, I feel like I do need to warn people about some ZFS encryption related issues that I learned about later. Please read this. I have had no problems to date, but make up your own mind.

RaidZ1

recordsize	compr.	encrypt	IOPS rand-read	IOPS read	IOPS write	MB/s read	MB/s write	“cat speed” MB/s
128 KiB	off	off	50,000	869,000	418,000	3,964	5,745	2,019
128 KiB	on	off	49,800	877,000	458,000	3,929	4,654	1,448
128 KiB	off	aes128	26,300	484,000	230,000	3,589	5,331	2,142
128 KiB	on	aes128	27,400	501,000	228,000	3,510	3,927	2,120

These are the numbers after creation of the RAIDZ1 based pool. They are quite similar to the numbers measured before. The impact of encryption on IOPS is clearly visible, less so on sequential read/write throughput. Compression seems to impact write throughput but not read throughput which is expected for zstd. It is unclear why “cat speed” is lower here.

recordsize	compr.	encrypt	IOPS rand-read	IOPS read	IOPS write	MB/s read	MB/s write	“cat speed” MB/s
1 MiB	off	off	7,235	730,000	404,000	3,686	3,548	2,142
1 MiB	on	off	7,112	800,000	470,000	3,624	3,447	2,064
1 MiB	off	aes128	3,259	497,000	258,000	3,029	3,422	2,227
1 MiB	on	aes128	3,697	506,000	249,000	3,137	3,361	2,237

Many optimisation guides suggest setting the zfs recordsize to 1 MiB for most use-cases, especially storage of media files. But this seems to drastically penalise random read IOPS while providing little to no benefit in the sequential read/write scenarios. This is actually a bit surprising and I will need to investigate this more. Is it perhaps because NVMEs are good at parallel access and therefor suffer less from fragmentation anyway?

In any case, the main take away message is that overall read and write throughputs are over 3,000 MiB/s in the synthetic case and over 2,000 MiB/s in the manual case, which is great.

Other disk performance metrics

Operation	Speed [MiB/s]
Copying 382 GiB between two datasets (both enc+comp)	1,564
Copying 505 GiB between two datasets (both enc+comp)	800
`zfs scrub` of the full pool	11,000

These numbers further illustrate some real world use-cases. It’s interesting to see the difference between the first two, but it’s also important to keep in mind that this is reading and writing at the same time. Maybe some internal caches are exhausted after a while? I didn’t debug these numbers further, but I think the speed is quite good after such a long read/write.

More interesting is the speed for scrubbing, and, yes, I have checked this a couple of times. A scrub of 6.84TiB happens in 10m - 11m, which is pretty amazing, I think, considering that it is reading the data and calculating checksums. I am assuming that sequential read is just very fast and that access to the different disks happens in parallel. The checksum implementation is apparently also avx2 optimised.

LAN

Network adapter

Based on recommendations, I decided to buy an Intel card. Cheaper 10GBit network cards are available from Marvell/Aquantia, but the driver support in FreeBSD is poor, and the performance is supposedly also not close to that of Intel.

Many people suggested I go for SFP+ (fibre) instead of 10GBase-T (copper), but I already have CAT7 cables in my flat. While I could have used fibre purely for connecting the server to the switch (and this would likely save some power), I would have had to buy a new switch and the options were just not economical—I already have a switch with two 10GBase-T ports which I had bought for exactly this setup.

The cheapest Intel 10GBase-T card out there is the X540 which is quite old and available on Amazon for around 80€. I bought two of those (one for the server and one for the workstation). More modern cards are supposedly more energy efficient, but also a lot more expensive.⁵

NFS Performance

On server and client, I set:

kern.ipc.maxsockbuf=4737024 in /etc/sysctl.conf
mtu 9000 media 10gbase-t in the /etc/rc.conf (ifconfig)

Only on the server:

nfs_server_maxio="1048576" in /etc/rc.conf

Only on the client:

nfsv4,nconnect=8,readahead=8 as the mount options for the nfs mount.
vfs.maxbcachebuf=1048576 in /boot/loader.conf (not sure any more if this makes a difference).

These settings allow larger buffers and increase the amount of readahead. This favours large sequential reads/writes over small random reads/writes.

The full options on the client end up being:

# nfsstat -m
X.Y.Z.W:/ on /mnt/server
nfsv4,minorversion=2,tcp,resvport,nconnect=8,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=8,wcommitsize=16777216,timeout=120,retrans=2147483647

I use NFS4 for my workstation and NFS3 for everyone else. I have performed no benchmarks on NFS3, but I see no reason why it would be slower.

IOPS rand-read	IOPS read	IOPS write	MB/s read	MB/s write	“cat speed” MB/s
283	292,000	33,200	1,156	594	1,164

This benchmark was performed on a dataset with 1M recordsize, encryption, but no compression. Random read IOPS are pretty bad, and I see a strong correlation here to the rsize (if I halve it, I double the IOPS; not shown in table). It’s possible that every 4KiB read actually triggers a 1MiB read in NFS which would explain this. On the other hand, the sequential read and write performance is pretty good with synthetic and real world read speeds being very close to the theoretical maximum of the 10GBit connection.

One thing to keep in mind: The blocksize when reading has a very strong impact on the performance. This can be seen when using dd with different bs arguments. Of course, 1MiB is optimal if that is also used by NFS, and cat seems to do this. However, cp does not which results in a much slower performance than if using dd if=.. of=.. bs=1M.

I have done measurements with plain nc over the wire (also reaching 1,160 MiB/s) and iperf3 which achieves 1,233 MiB/s just below the 1,250 MiB/s equivalent of 10Gbit.

Power consumption and thermals

For a computer running 24/7 in my flat, power consumption is of course important. I bought a device to measure power consumption at the outlet to get an accurate picture.

idle

Because the computer is idle most of the time, optimising idle power usage is most important.

Change	W/h
default	50
`*_cx_lowest="Cmax"`	45
disable WiFi and BT	42
`media 10gbase-t`	45
`machdep.hwpstate_pkg_ctrl=0`	41
turn on chassis fans	42
ASPM modes to L0s+L1 / enabled	34

I assume that the same setup on Linux would be slightly more efficient, but 34W in idle is acceptable.

Clearly, the most impactful changes were:

Activating ASPM for the PCIe devices in the BIOS.
Adding performance_cx_lowest="Cmax" and economy_cx_lowest="Cmax" to /etc/rc.conf.
Adding machdep.hwpstate_pkg_ctrl=0 to /boot/loader.conf.

You can find online resources on what these options do. You might need to update the BIOS to be able to disable WiFi and Bluetooth devices completely. You can also use hints in the /boot/device.hints, but this doesn’t save as much power.

Using 10GBase-T speed on the network device (instead of 1000Base-T) unfortunately increases power usage notably, but there is nothing I could find to mitigate this.

Things that are often recommended but that did not help me (at least not in idle):

NVME power states (more on this below)
lower values for sysctl dev.hwpstate_intel.*.epp (more on this below)
hw.pci.do_power_nodriver=3

idle temperatures	°C
CPU	37-40
NVMEs	52-55

The latter was particularly interesting, because I had heard that newer NVMEs, and especially those by Lexar get very warm. It should be noted though, that the mainboard comes with a large heatsink that covers all NVMEs.

under load

The only “load test” that I performed was a scrub of the pool. Since this puts stress on the NVMEs and also the CPUs, it should be at least indicative of how things are going.

during `zpool scrub`	°C
CPU	55-59
NVMEs	69-75

The power usage fluctuates between 85W and 98W. I think all of these values are acceptable.

NVME power state hint	scrub speed GiB/s	W/h
0 (default)	11	< 100
1	8	< 93
2	4	< 70

You can use nvmecontrol to tell the NVME disks to save energy. More information on this here and here. I was surprised that all of this works reliably on FreeBSD, but it does! The man-page is not great though. Simply call nvmecontrol power -p X nvmeYns1 to set the hint to X on device Y, if desired. Note that this needs to be repeated after every reboot.

`dev.hwpstate_intel.*.epp`	scrub speed GiB/s	W/h
50 (default)	11.0	< 100
100	3.3	< 60

You can use the dev.hwpstate_intel.*.epp sysctls for you cores to tune the eagerness of that core to scale up with higher number meaning less eagerness.

In the end, I decided not to apply any of these “under load optimisations”. It is just very difficult, because, as shown, all optimisations that reduce watts per time also increase time. I am not certain of any good ways to quantify this, but it feels like keeping the system at 70W for 30min instead of 100W for 10min, is not really worth it. And I kind of also want the system to be fast, that’s why I spent so much money on it 🙃

The CPU does have a cTDP mode that can be activated via the BIOS and which is “worth it”, according to some articles I have read. I might give this a try in the future.

Final remarks

What a ride! I spent a lot of time optimising and benchmarking this and I am quite happy with the outcome. I am able to exhaust the 10GBit LAN connection completely, and still have resources left on the server :)

Thanks to the people at www.bsdforen.de who had quite a few helpful suggestions.

If you see anything that I missed, or have suggestions on how to improve this setup, let me know in the comments!

Footnotes

With ASUS being the only exception. ↩︎
Proper in this context means well-supported by FreeBSD and with a good performance. Usually, that means an Intel NIC. Unfortunately all the modern boards come Marvell/Aquantia AQtion adaptors which are not well-supported by FreeBSD. ↩︎
The geli device was created with: geli init -b -s4096 -l256 ↩︎
I wanted to perform all these tests with Linux as well, but I ran out of time 🙈 ↩︎
I did try a slightly more more modern adapter with Intel 82599EN chip. This is a SFP+ chip, but I found an adaptor with built-in 10GBase-T for around 150€. It ended up having some driver issues (you needed to plug and unplug the CAT cable for the device to go UP), and it used more energy than the X540, so I sent it back. ↩︎

🌓 Commandline dark-mode switching for Qt, GTK and websites 🌓

Sun, 10 Dec 2023 16:30:00 +0000

This post documents how to toggle your entire desktop between light and dark themes, including your apps and the websites in your browser.

Motivation

Like many other people, I use my computer(s) with varying degrees of ambient light. When there is lots of light, I want a bright theme, but in the evenings, I prefer a dark theme. Switching this for Firefox and several toolkits manually almost drove me crazy, so I will document here how I automated the entire process.

I use the Sway window manager which makes things a bit more difficult, because neither the UI unification mechanisms of GNOME nor KDE automatically kick in. I use Firefox as a browser, and I also want the websites to switch themes. And of course, I want the theme switch to be applied immediately and not just on restartet apps.

Demo

Your browser does not support the video tag.

This is what it looks like when it’s done. Dolphin (KDE5), Firefox, the website inside Firefox, and GEdit (GTK) all switch together.

Primary script

#!/bin/sh

current=$(gsettings get org.gnome.desktop.interface color-scheme)

if [ "${current}" != "'prefer-dark'" ]; then #default

    echo "Switching to dark."
    gsettings set org.gnome.desktop.interface color-scheme prefer-dark
    gsettings set org.gnome.desktop.interface gtk-theme Adwaita-dark
    gsettings set org.gnome.desktop.interface icon-theme breeze-dark

else # already dark

    echo "Switching to light."
    gsettings set org.gnome.desktop.interface color-scheme default
    gsettings set org.gnome.desktop.interface gtk-theme Adwaita
    gsettings set org.gnome.desktop.interface icon-theme breeze

fi

This is the primary script. It works by manipulating the gsettings, so we will have to make everything else follow these settings. The script operates in a toggle-mode, i.e. running it repeatedly switches between light and dark. I had hoped that the color-scheme preference would be the only thing needing change, but the gtk-theme needs to also be switched explicitly. I am not aware of any theme other than Adwaita that works on all toolkits.

Switching the icon-theme is not necessary, but recommended. To get a list of installed icon themes, ls /usr/share/icons.

Packages

This is the list of packages I installed on Ubuntu 23.10. Note that if you miss certain packages, things will not work without telling you why. I started this install with a Kubuntu ISO, so depending on your setup, you might need to install more packages, e.g. libglib2.0-bin provides the gsettings binary.

Package list:

libadwaita                  # GTK3 theme (auto-installed)
adwaita-qt                  # Qt5 theme
adwaita-qt6                 # Qt6 theme
gnome-themes-extra          # GTK2 theme
gnome-themes-extra-data     # GTK2 theme and GTK3 dark theme support
qgnomeplatform-qt5          # Needed to tell Qt5 and KDE to use gsettings
#qgnomeplatform-qt6         # If your distro has it

I am not exactly sure where the GTK4 theme comes from, and I have no app to test that. If you want to use the breeze icon theme, also install breeze-icon-theme.

Configuration

GTK apps

You should already be able to switch GTK apps by running the script. Give it a try!

Firefox app

Firefox should also switch its own theme after invoking the script. If it does not, check the following:

Your XDG session is treated by Firefox as being GNOME or something similiar.¹
Go to about:addons, then “Themes” and make sure you have selected “System-Theme (automatic)”. ²
Go to about:support and look for “Windows Protocol”. It should list wayland. If it does not, restart your Firefox with MOZ_ENABLE_WAYLAND=1 set in the environment.
Go to about:support and look for “Operating System theme”. It should list Adwaita / Adwaita. If it does not, you are likely missing some crucial packages.
Double-check the gnome-themes-extra or similar packages on your distro. I didn’t have these initially and it prevented Firefox from picking up the theme.

I haven’t tried any of this with Chromium, but I might at some point in the future.

Firefox (websites)

Next are the websites inside Firefox. Make sure your Firefox propagates its own theme settings to its websites:

Websites like https://google.com should now respect your system’s theme. However, running our script will not affect open tabs; you need to reload the tab or open a new tabe to see the effects.

Many other sites do not have a dark theme, though, or do not apply it automatically. To change these sites, install the great dark reader firefox plugin!

Configure the plugin for automatic bahaviour based on the system colours (as shown above). Now is the time to test the script again! Websites controlled by Dark Reader should update immediately without a refresh. This is one reason to prefer Dark Reader’s handling over native switching (like that of https://google.com ).³ If this is the behaviour you want, make sure that websites are not disabled-by-default in Dark Reader (configurable through the small ⚙ under the website URL in the plugin-popup); this is the case for e.g. https://mastodon.social.

An option you might want to play with is found under “more → Change Browser Theme”. This makes the plugin control the Firefox application theme. This is a bit of a logic loop (script changes Firefox theme → triggers Plugin → triggers update of theme), but it often works well and usually gives a slightly different “dark theme look” for the application.

Qt and KDE apps

There are multiple ways to make Qt5 and KDE apps look like GTK apps:

Select the Adwaita / Adwaita-Dark theme as the native Qt theme (QT_QPA_PLATFORMTHEME=Adwaita / QT_QPA_PLATFORMTHEME=Adwaita-dark)
Select “gtk2” as the native Qt theme (QT_QPA_PLATFORMTHEME=gtk2)
Select “gnome” as the native Qt theme (QT_QPA_PLATFORMTHEME=gnome)

All of these work to a certain degree, and I would have liked to use first option. But for neither 1. nor 2., I was able to achieve “live switching” of already open applications upon invocation of the script. In theory, one should also be able to use KDE’s lookandfeeltool to switch between the native Adwaita and Adwaita-dark themes (or any other pair of themes), but I was not able to make this work reliably.³

Note that for Qt6 applications to switch theme with the rest, qgnomeplatform-qt6 needs to be installed, which is not available on Ubuntu. Other platforms (like Arch’s AUR) seemed to have it, though.

Note also that in Sway you need to make sure that QT_QPA_PLATFORMTHEME is defined in the context where your applications are started. This is typically not the case within the sway config, so I do the following:

bindsym $mod+space exec /home/hannes/bin/preload_profile krunner

Where preload_profile executes all arguments given to it, but imports ~/.profile before.

Possible extensions

Screen brightness

Before I managed to setup theme-switching correctly, I used a script to control brightness. Now that theme switching works, I don’t do this anymore, but in case you want this additionally:

You can use brightnessctl to adjust the brightness of the built-in screen of your Laptop.
You can use ddcutil to adjust the brightness of an external monitor (this affects actual display brightness not Gamma).

Automation

If desired, you could automate theme switching with cron or map hotkeys to the script.

Closing remarks

I am really happy I got this far; the only thing that does not update live is the icon theme in KDE applications. If anyone has advice on that, I would be grateful!

I have used the method of having everything behave like being on GNOME here. In theory, it should also be possible to set XDG portal to kde and use lookandfeeltool instead of gsettings, but I did not yet manage to make that work.If you have, please let me know!

I have verified that an XDG desktop portal of wlr or gtk works, and also that a value of kde does not work; so this won’t work within a KDE session. ↩︎
If you get spurious flashes of white between website loading or tab-switches, you can later switch this to the “Dark” theme and it should still turn bright when in global bright mode. ↩︎
Native dark themes may or may not look better than whatever Dark Reader is doing. I often prefer Dark Reader, because it allows backgrounds that are not fully black. ↩︎ ↩︎

Configuring algorithms in Modern C++

Thu, 30 Mar 2023 22:00:00 +0000

When designing library code, one often wonders: “Are these all the parameters this function will ever need?” and “How can a user conveniently change one parameter without specifying the rest?” This post introduces some Modern C++ techniques you can use to make passing configuration options easy for your users while allowing you to add more options later on.

Prerequisites

Most people who have programmed in C++ before should have no problems understanding this article, although you will likely appreciate it more, if you are a library developer or have worried about the forward-compatibility of your code.

Some of the features introduced in this post do not yet work with Clang. Most should work with MSVC, but I only double-checked the code with GCC12 (any version >= 10 should work).

Motivation

Let’s say you are writing an algorithm with the following signature:

auto algo(auto data, size_t threads = 4ull);

It takes some kind of data input, does lots of magic computation on it and returns some other data. An actual algorithm should of course clearly state what kind of input data it expects (specific type or constrained template parameter), but we want to focus on the other parameters in this post. The only configuration option that you want to expose is the number of threads it shall use. It defaults to 4, because you know that the algorithm scales well at four threads and also you assume that most of your users have at least four cores on their system.

Now, a bit later, you have added an optimisation to the algorithm called heuristic42. It improves the results in almost all cases, but there are a few corner cases where users might want to switch it off. The interface now looks like this:

auto algo(auto data, size_t threads = 4ull, bool heuristic42 = true);

This is not too bad, you might think, but there are already two ugly things about this:

To overwrite the second “config option”, the user needs to also specify the first, i.e. algo("data", 4, false);. This means that they need to look up (and enter correctly) the first config option’s default value. Also, if you change that default in a future release of the code, this change will not be reflected in the user’s invocation who unknowingly enforces the old default.
Since passing arguments to the function does not involve the parameter’s name, it is very easy to confuse the order of the parameters. Implicit conversions make this problem even worse, so invoking the above interface with algo("data", false, 4); instead of algo("data", 4, false); generates no warning, even with -Wall -Wextra -pedantic!

Wow, what a usability nightmare, and we only have two config options! Imagine adding a few more…

Dedicated config object

As previously mentioned, the parameter name cannot be used when passing arguments. However, C++20 did add designated initialisers for certain class types. So you can use the name of a member variable when initialising an object. We can use that!

struct algo_config
{
    bool heuristic42 = true;
    size_t threads   = 4ull;
};

auto algo(auto data, algo_config const & cfg)
{
    /* implementation */
}

int main()
{
    /* create the config object beforehand (e.g. after argument parsing) */
    algo_config cfg{.heuristic42 = false, .threads = 8};
    algo("data", cfg);

    /* create the config object ad-hoc */
    algo("data", algo_config{.heuristic42 = false, .threads = 8});   // set both paramaters
    algo("data", algo_config{.threads = 8});                         // set only one parameter

    /* providing the config type's name is optional */
    algo("data", {.threads = 8});

}

Compile and edit online (via godbolt)

As you can see, this solves both of the problems mentioned previously! We refer to the config elements by name to avoid mixups, and we can choose to overwrite only those parameters that we actually want to change; other configuration elements will be whatever they are set to by default. Conveniently, this allows changing the default later on, and all invocations that don’t overwrite it will pick up the new default.

Another great feature is that the API maintainer of the algorithm can easily add more members to the configuration object without invalidating any existing invocations. This allows users of the API to gradually adopt new opt-in features.

As the name of the config type can even be omitted (see last invocation), the syntactic overhead for the “ad-hoc” initialisation is very low, almost like providing the arguments directly to the function-call.

There is an important catch: The order in which the designated initialisers are given has to correspond to the order in which the respective members are defined in the type of the config. It is okay to omit initialisers at the beginning, middle or end (as long as defaults are given), but the relative order of all the initialisers that you do provide has to be correct. This might sound like a nuisance, but in contrast to the problem discussed initially (mixed up order of function arguments), you will actually get a compiler-error that tells you that you got the order wrong; so the problem is easily detected and fixed. And there is a nice rule that you can follow for such config objects: always sort the members alphabetically! That way users intuitively know the order and don’t have to look it up 💡

Types as config elements

Now, sometimes you want to pass a type as kind of parameter to an algorithm. Imagine that the algorithm internally juggles a lot of integers. Maybe it even does SIMD with them. In those cases, the size of the integers could affect performance noticeably.

Some algorithms might be able to infer from the data input’s type which integers to use for computation, but in other cases you want the user to be able to override this. Thus we need the ability to pass the desired type to the algorithm. The canonical way of doing this is via template arguments:

template <typename int_t>
auto algo(auto data, size_t threads = 4ull);

But this is has the same problems that we discussed initially: as soon as multiple types are passed, it is possible confuse the order (and not be notified); to set a later parameters, you need to also set previous ones; et cetera. There might also be weird interactions with the type of the data parameter, in case that is a template parameter.

Let’s add the “type parameter” to our config object instead:

/* We define a "type tag" so we can pass types as values */
template <typename T>
inline constinit std::type_identity<T> ttag{};

/* The config now becomes a template */
template <typename Tint_type = decltype(ttag<uint64_t>)>
struct algo_config
{
    bool heuristic42    = true;
    Tint_type int_type  = ttag<uint64_t>;
    size_t threads      = 4ull;
};

/* And also the algorithm */
template <typename ...Ts>
auto algo(auto data, algo_config<Ts...> const & cfg)
{
    /* implementation */
}

int main()
{
    /* Setting just "value parameters" still works with and without "algo_config" */
    algo("data", algo_config{.heuristic42 = false, .threads = 8});
    algo("data",            {.heuristic42 = false, .threads = 8});

    /* When setting a "type parameter", we need to add "algo_config" */
    algo("data", algo_config{.int_type = ttag<uint32_t>, .threads = 8});
}

Compile and edit online (via godbolt)

There are a few things happening here. In the beginning, we use variable templates to define an object that “stores” a type. This can later be used to initialise members of our config object.

Next, we need to make algo_config a template. Unfortunately, we need to default the template parameter as well as giving the member a default value. Finally, algo() needs template parameters for the config, as well. It is handy to just use a parameter pack here, because it means we don’t need to change it if we add more template parameters the config type. This is all a bit more verbose than before, after all we are still writing C++ 😅 But most of this will be hidden from the user anyway.

The invocation of the algorithm is almost unchanged from before, we just use ttag<uint32_t> to initialise the “type parameter” of the config. There is one caveat: when passing such “type parameters”, it is now necessary to add algo_config, although, fortunately, you do not need to spell out the template arguments. In general, this may be a bit surprising, so I recommend always including the config-name in examples to teach your users a single syntax.

Constants as config elements

Using a similar technique to the one above, we can also pass compile-time constants to the config object. This allows the algorithm to conveniently use if constexpr to choose between different codepaths, e.g. between a SIMD-based codepath and a regular one.

/* We define a "value tag" type so we can pass values as types...*/
template <auto v>
struct vtag_t
{
    static constexpr auto value = v;
};

/* ...and then we define a variable template to pass the type as value again! */
template <auto v>
inline constinit vtag_t<v> vtag{};

/* The config is a template */
template <typename Tuse_simd = vtag_t<false>>
struct algo_config
{
    bool heuristic42    = true;
    size_t threads      = 4ull;
    Tuse_simd use_simd  = vtag<false>;
};

/* The algorithm */
template <typename ...Ts>
auto algo(auto data, algo_config<Ts...> const & cfg)
{
    /* implementation */
}

int main()
{
    /* Setting just "value parameters" still works with and without "algo_config" */
    algo("data", algo_config{.heuristic42 = false, .threads = 8});
    algo("data",            {.heuristic42 = false, .threads = 8});

    /* When setting a "constant parameter", we need to add "algo_config" */
    algo("data", algo_config{.threads = 8, .use_simd = vtag<true>});
}

Compile and edit online (via godbolt)

As you can see, this is very similar to the previous example. The only difference is, that we need another initial step to encode the value as a type. It is even possible to have parameters that are (run-time) values by default, but can be configured as (compile-time) constants in the way shown above. And, of course, all kinds of config options can be combined.

Note that the definitions of the “tagging” features would happen in your utility code. Users only need to know that they can pass constants via vtag<42> and types via ttag<int32_t>.

Post scriptum

I hope this post was helpful to some of you. I think this is a big step forward for usability, and I hope Clang catches up with the required features as soon as possible!

There are two things here that could be improved:

If a template parameter can be deduced from member initialisers, it should be. This would allow us to omit the default template arguments for algo_config, i.e. = decltype(ttag<uint64_t>) and = vtag_t<false>.
When a brace-enclosed initialiser list is passed to a function template to initialise a parameter of deduced type, consider the contents of that initialiser list. This would allow us to omit align_config also when passing “type parameters” or constants.

I have the feeling that 1. might not be too difficult and also not too controversial. But I suspect that 2. would be more complicated as it interacts with function overloading and I can imagine situations were this change would break existing code.

But I’d love to here other people’s opinion on the matter!

References

The ISO WG21 papers that added these features to C++:

P0329 Designated Initialisers, C++20
P1816 and P2082 Class template argument deduction for aggregates, C++20, not yet in Clang

👾 Game-streaming without the "cloud" 🌩️

Mon, 20 Apr 2020 19:10:00 +0200

With increasing bandwidths, live-streaming of video games is becoming more and more popular – and might further accelerate the demise of the desktop computer. Most options are “cloud-gaming” services based on subscriptions where you don’t own the games and are likely to be tracked and monetised for your data. In this blog post I present the solution I built at home to replace my “living room computer”.

TL; DR

I describe how to stream a native Windows game (Divinity Original Sin 2 – installed DRM-free from GOG) via Steam Remote Play from my Desktop (running Devuan GNU/Linux) in 4K resolution and maximum quality directly to my TV (running Android). On the TV I then play the game with my flatmate using two controllers: one Xbox Controller connected via USB, one Steam-Controller connected via Bluetooth. The process of getting there is not trivial or flawless, but gaming itself works perfectly without artefacts or lag.

(The computer in the above photo is not used any longer.)

Motivation

The big “cloud” streaming services for audio and video are incredibly convenient. In my opinion they address the following problems very well:

They make content available on hardware that wouldn’t have the capability to store the content.
This includes all devices of the user, not just a single (desktop/laptop) computer.
They provide access to a lot of content at a fixed monthly price (especially useful for people who haven’t had the possibility to build up their “library”).

On the other hand, they usually force you to use software that is non-free and/or not trustworthy (if they support your OS at all). You risk losing access to all content when the subscription is terminated. And – often overlooked – they monitor you very closely: what you watch/listen to, when and where you do it, when you press pause and for how long etc. I suspect that at some point this data will be more valuable than the income generated from subscription fees, but this is not important here.

It was only a matter of time before video-gaming would also become part of the streaming business and there are now multiple contenders: Google Stadia and Playstation Now are typical “streaming services” like Netflix for video; the games are included in the monthly fee. They have all the benefits and problems discussed above. Other services like GeForce Now and Blade Shadow provide computation and streaming infrastructure but leave it to you to provide the games (manually or via gaming platforms). This has slightly different implications, but I dont’ want to discuss these further, because I haven’t used any of them and likely won’t in the future.

In any case, I am of course not a big fan of being tracked and anything I can set up with my own infrastructure at home, I try to do! [I actually don’t really spend that much time gaming nowadays, but setting this up was still fun]. In the past I have had an extra (older) computer beside the TV that I used for casual gaming. This has become to old (and noisy) to play current games, so instead I want stream the game from my own desktop computer in a different room.

Disclaimer: It should be noted that this “only” provides feature 2. above, i.e. convergence of different devices. Also this setup includes using various non-free software components, i.e. it also does not solve all of the problems mentioned above.

Setting up the host

The “host” is the computer that renders the game, in my case the desktop computer. Before attempting to stream anything, we need to make sure that everything works on the host, i.e. we need to be able to play the game, use the controllers etc.

Hardware

CPU	RAM	GPU
AMD Ryzen 6c/12t @ 3.6Ghz	32GB	GeForce RTX 2060S

Since the host needs to render the game, it needs decent hardware. Note that it needs to be able to encode the video-stream at the same time as rendering the game which means the requirements are even higher than usual. But of course all of this also depends on the exact games that you are playing and which resolution you want to stream. My specs are shown above.

The game

I chose Divinity Original Sin 2 for this article because that’s what I am playing right now and because I wanted to demonstrate that this even works with games that are not native to Linux (although I usually don’t buy games that don’t have native versions). If you buy it, I recommend getting it on GOG, because games are DRM-free there (they work without internet connection and stay working even if GOG goes bankrupt). The important thing here is that the game does not need to be a Steam game even though we will use Steam Remote Play for streaming. Buying it on Steam will make the process a little easier though.

Install the game using wine (wine in Devuan/Debian repos). I assume that you have installed it to /games/DOS2 and that you have setup a “windows disk G:" for /games. If not, adjust paths accordingly.

Steam

If you haven’t done so already, install Steam. It is available in Debian/Devuan non-free repositories as steam but will install and self-update itself in a hidden subfolder of your home-directory upon first start. You need a steam-account (free) and you need to be logged in for everything to work. I really dislike this and it means that Steam quite likely does gather data about you. I suspect that using non-Steam games makes it more difficult to track you, but I have not done any research on this. See the end of this post for possible alternatives.

The first important thing to know about Steam on Linux is that it ships many system libraries, but it doesn’t ship everything that it needs and it gives you no diagnostic about missing stuff nor are UI elements correctly disabled when the respective feature is not available. This includes support for hardware video encoding and for playing windows games.

Hardware-video encoding is a feature you really want because it reduces latency and load on your CPU. For reasons I don’t understand, Steam uses neither libva (the standard interface for video acceleration on free operating systems) nor vdpau (NVIDIA-specific but also free/open). Instead it uses the proprietary NVENC interface. On Debian / Devuan his has been patched out of all the libraries and applications, so you need to make sure that you get your libraries and apps like ffmpeg from the Debian multimedia project which has working versions. I am not entirely sure which set of libraries/apps is required, for me it was sufficient to install libnvidia-encode1 and do an apt upgrade after adding the debian multimedia repo. Note that only installing libnvidia-encode1 from the official repo was not sufficient. See the troubleshooting section on how to diagnose problems with hardware video encoding.

To play Windows games with Steam, you can use Steam’s builtin windows emulator called Proton (here’s a current articel on it). It’s a fork of Wine with additional patches (most of which are upstreamed to official wine later). Unfortunately it is not installed after a fresh install of Steam on Linux and I found no explicit way of installing it (the interface still suggests it’s there, though!). To get it, select “Enable Steam Play for all titles” in the “Advanced” “Steam Play Settings” in the settings. This activates usage of Proton for Windows games not officially supported. Then install the free Windows game 1982 from inside Steam. This will automatically install Proton which is then listed as an installed application and updated automatically in the future. You can try the game to make sure Proton works as expected. Alternatively, buy the actual game on Steam and skip the next paragraph.

Now go to “Games” → “Add a Non-Steam Game to my Library…”, then go to “BROWSE”, show all extensions and find the executable file of the game. In our case this is /games/DOS2/DefEd/bin/EoCApp.exe or G:\DOS2\DefEd\bin\EoCApp.exe (yes, it’s not in the top-level directory of the game). If your path contains spaces, it will break (without diagnostics). To fix this, simply edit the shortcut created for the game again and make sure the path is right and “set launch options” is empty (part of your path may have ended up there). In this dialog you can also explicitly state that you wish to use Proton to run the game (confusingly it will show multiple versions even if none of those are installed). You can also give the game a nicer name or icon if desired.

Test the game

You are now ready to test the game. Simply click on the respective button. Now is also a good time for testing the controller(s), because if they don’t work now, they likely won’t later on. There are many tutorials on using the (Steam) controller on Linux and there are also some notes in the troubleshooting section.

This is also a good point in time to update the firmware of the steam controller to the newest version; we will need that later on.

Setting up the client

The client is my TV, because that runs Android TV and there is a SteamLink application for Android. If your TV has a different OS, you could probably use a small set-top-box built around a cheap embedded board and use Android (or Linux) as the client OS from that. I suppose all of this would be possible on an Android Tablet, as well. I recommend connecting the TV via wired network to the host, but WiFi works at lower resolutions, too.

Controllers and Android

The Xbox controller is just plugged into the USB-port of the TV and required no further configuration. The Steam controller was a little more tricky, and I had no luck getting it to work via USB. However, it comes with Bluetooth support (only after upgrading the firmware!).

Establishing the initial Bluetooth connection between the TV and the controller was surprisingly difficult. Start the Steam-controller with Steam+Y pressed or alternatively with Steam+B pressed and only do so after initiating device search on the TV. If it does not work immediately, keep trying! After a few attempts, the TV should state that it found the device; it calls it SteamController but recognises it as a Keyboard+Mouse. That’s ok.

After the connection has been established once, the controller will auto-connect when turned off and on again (do not press Y or B during start!). The controller’s mouse feature is surprisingly useful in the Android TV when you use Android apps that are not optimised for the TV.

SteamLink / Steam Remote Play

You need to install the SteamLink application from GooglePlay or through another channel like the Aurora Store (my Android TV is not associated with a Google account so I cannot use GooglePlay). After opening the app, Android will ask whether it should allow the app to access the Xbox controller which you have to agree to everytime (the “do this in the future” checkbox has no effect). Confusingly the TV then notifies you that the controller has been disconnected. This just means that the app controls it, I think. The app takes control of the Steam controller without asking and switches it from Keyboard+Mouse mode into Controller mode, so you can use the Joystick to navigate the buttons in the app.

It should automatically detect Steam running on the host and offer to make a connection. Press A on the controller or select “Start Playing”. The connection has to be verified once on the host for obvious reasons, but if everything works well you should now see the “Steam Big Picture Mode” interface on you TV. Your Desktop has switched to this at the same time (in general, the TV will now mirror your desktop). Maybe first try a native Steam game like the aforementioned “1982” to see if everything works.

Next try to start the Game we setup above via its regular entry in your Steam Library. Note that upon starting, the screen will initially flash black and return to Steam; the game is starting in the background, do not start it again, it just needs a second!

Everything should work now! If you hear audio but your screen stays black and shows a “red antenna symbol”, see the troubleshooting section below.

You can press the Steam-Button to return to Steam (although this sometimes breaks for non-native Steam games). You can also long-press the “back/select”-button on the controller to get a SteamLink specific menu provided by the Client. It can be used to launch an on-screen keyboard and force-quit the connection to the host and return to the SteamLink interface.

Video resolutions and codecs

If everything worked so far you are likely playing in 1080p. To change this to 4k resolution, go to the SteamLink app’s settings (wheel symbol) and then to “streaming settings” and then to “advanced”. You can increase the resolution limit there and also enable “HEVC Video” which improves video quality / reduces bitrate. If your desktop does not support streaming HEVC, SteamLink will establish no connection at all. See the troubleshooting section below if you get a black screen and the “red antenna symbol”.

Alternatives

The only viable alternative to Steam’s Remote Play for streaming your own games from your own hardware is NVIDIA GameStream – not to be confused with NVIDIA GeForce Now, the “cloud gaming” service. The advantages of GameStream over Steam Remote Play seem to be the following:

The protocol is documented and there are good Free and Open Source client implementations.
It claims better performance by tying closer into the host’s drivers.
No online-connection or sign-up required like with Steam.

Also NVIDIA is a hardware company and even if the software is proprietary, it might be less likely to spy on you. The disadvantages are:

The host software is only available for Windows.
Only works with NVIDIA GPUs on the host machine, not AMD.

I could not try this, because I don’t have a Windows host and I wanted to stream from GNU/Linux.

Post scriptum

As you have seen the process really still has some rough edges, but I am honestly quite surprised that I did manage to set this up and that it works with really good quality in the end. Although I don’t like Steam because of its DRM, I have to admit that its impressive how much work they put into improving the driver situation on GNU/Linux and supporting such setups as discussed here. Consider that I didn’t even buy a game from Steam!

I would still love to see a host implementation of NVIDIA GameStream that runs on GNU/Linux. Even better would of course be a fully Free and Open Source solution.

On a sidenote: the setup I showed here can be used to stream any kind of application, even just the regular desktop if you want to (check out the advanced client options!).

Hopefully Steam Remote Play (and NVIDIA GameStream) can delay the full transition to “cloud gaming” a little.

Troubleshooting

No hardware video encoding

To see if Steam is actually using your hardware encoding, look in the following log-file:

~/.steam/debian-installation/logs/streaming_log.txt

You should have something like:

"CaptureDescriptionID"  "Desktop OpenGL NV12 + NVENC HEVC"

The NVENC part is important. If you instead have the following:

"CaptureDescriptionID"  "Desktop OpenGL NV12 + libx264 main (4 threads)"

It means you have software encoding. Play around with ffmpeg to verify that NVENC works on your system. The following two commands should work:

% ffmpeg -h encoder=h264_nvenc
% ffmpeg -h encoder=hevc_nvenc

The second one is for the HEVC codec. If you are told that the selected encoder is not available, something is broken. Check to see if you correctly upgraded to the libraries from the Debian multimedia project.

Controller not working at all on host

Likely your user account does not have permissions to access the device. It worked for me after making sure that my user was in the following groups:

audio dip video plugdev games scanner netdev input

Controller working in Steam but not in game

When you start Steam but leave BigPicture mode (Alt+Tab or minimise), Steam usually switches the controller config back to “Desktop mode” which means Keyboard+Mouse emulation. If a game is started then, it will not detect any controller. The same thing seams to happen for certain non-steam games started through Steam. A workaround is going into the Steam controller settings and selecting the “Gamepad” configuration also as default for “Desktop mode”.

Black screen and "red antenna symbol"

This happens when there is a resolution mismatch somewhere. Note that we have multiple places and layers where resolution can be set and that steam doesn’t always manage to sync these: the host (operating system, Steam, In-Game), the client (operating system, SteamLink). Additionally, Steam apparently can stream in a lower resolution than is set on either host or target.

I initially had this problem when streaming from 4k-host onto a SteamLink app that was configured to only accept 1080p. Changing the host resolution manually to 1080p before starting Steam solved this problem.

Now that everything (Host, In-Game, SteamLink on client) are configured to 4k, I still get this problem, because apparently Steam still attempts to reduce transmission resolution when starting the game. For obscure reasons the following workaround is possible: After starting the game, long-press “back/select” on the Controller to get to the SteamLink menu and select “Stop Game” there. This fixes the Black Screen and the regular game screen appears at 4K resolution. I suspect that “Stop game” terminates Steam’s wrapper around the game’s process that screws up the resolution. The devil knows why this does not terminate the game.

SteamLink produces black screen and nothing else

When switching in out of SteamLink via Android (e.g. TV remote), SteamLink sometimes doesn’t recover. Probably something goes wrong with putting the app to standby, but it’s also not something you typically would. Just quit the app correctly.

In any case, only a full reboot of the TV seems to fix this and make SteamLink usable again.

A beginner's guide to C++ Ranges and Views.

Sat, 30 Nov 2019 18:10:00 +0200

C++ Ranges are one of the major new things in C++20 and “views” are a big part of ranges. This article is a short introduction for programmers that are new to C++ Ranges.

Preface

You don’t need to have any prior knowledge of C++ Ranges, but you should have basic knowledge of C++ iterators and you should have heard of C++ Concepts before. There are various resources on C++ Concepts, e.g. Good Concepts, Wikipedia (although both contain slightly outdated syntax).

This article is based on library documentation that I wrote for the SeqAn3 library. The original is available here. There is also beginner’s documentation on C++ Concepts over there.

Since none of the large standard libraries ship C++ Ranges right now, you need to use the range-v3 library if you want to try any of this. If you do, you need to replace the std::ranges:: prefixes with just ranges:: and any std::views:: prefixes with ranges::views::.

Motivation

Traditionally most generic algorithms in the C++ standard library, like std::sort, take a pair of iterators (e.g. the object returned by begin()). If you want to sort a std::vector v, you have to call std::sort(v.begin(), v.end()) and not std::sort(v). Why was this design with iterators chosen? It is more flexible, because it allows e.g.:

sorting only all elements after the fifth one:

std::sort(v.begin() + 5, v.end())

using non-standard iterators like reverse iterators (sorts in reverse order):

std::sort(v.rbegin(), v.rend());

combine both (sorts all elements except the last 5 in reverse order):

std::sort(v.rbegin() + 5, v.rend());

But this interface is less intuitive than just calling std::sort on the entity that you wish to sort and it allows for more mistakes, e.g. mixing two incompatible iterators. C++20 introduces the notion of ranges and provides algorithms that accept such in the namespace std::ranges::, e.g. std::ranges::sort(v) now works if v is range – and vectors are ranges!

What about the examples that suggest superiority of the iterator-based approach? In C++20 you can do the following:

sorting only all elements after the fifth one:

std::ranges::sort(std::views::drop(v, 5));

sorting in reverse order:

std::ranges::sort(std::views::reverse(v));

combine both:

std::ranges::sort(std::views::drop(std::views::reverse(v), 5));

We will discuss later what std::views::reverse(v) does, for now it is enough to understand that it returns something that appears like a container and that std::ranges::sort can sort it. Later you will see that this approach offers even more flexibility than working with iterators.

Ranges

Ranges are an abstraction of “a collection of items”, or “something iterable”. The most basic definition requires only the existence of begin() and end() on the range.

Range concepts

There are different ways to classify ranges, the most important one is by the capabilities of its iterator.

Ranges are typically input ranges (they can be read from), output ranges (they can be written to) or both. E.g. a std::vector<int> is both, but a std::vector<int> const would only be an input range.

Input ranges have different strengths that are realised through more refined concepts (i.e. types that model a stronger concept, always also model the weaker one):

Concept	Description
`std::ranges::input_range`	can be iterated from beginning to end at least once
`std::ranges::forward_range`	can be iterated from beginning to end multiple times
`std::ranges::bidirectional_range`	iterator can also move backwards with `--`
`std::ranges::random_access_range`	you can jump to elements in constant-time `[]`
`std::ranges::contiguous_range`	elements are always stored consecutively in memory

These concepts are derived directly from the respective concepts on the iterators, i.e. if the iterator of a range models std::forward_iterator, than the range is a std::ranges::forward_range.

For the well-known containers from the standard library this matrix shows which concepts they model:

	`std::forward_list`	`std::list`	`std::deque`	`std::array`	`std::vector`
`std::ranges::input_range`	✅	✅	✅	✅	✅
`std::ranges::forward_range`	✅	✅	✅	✅	✅
`std::ranges::bidirectional_range`		✅	✅	✅	✅
`std::ranges::random_access_range`			✅	✅	✅
`std::ranges::contiguous_range`				✅	✅

There are also range concepts that are independent of input or output or one of the above concepts, e.g. std::ranges::sized_range which requires that the size of a range is retrievable by std::ranges::size() (in constant time).

Storage behaviour

Containers are the ranges most well known, they own their elements. The standard library already provides many containers, see above.

Views are ranges that are usually defined on another range and transform the underlying range via some algorithm or operation. ~~Views do not own any data beyond their algorithm and the time it takes to construct, destruct or copy them should not depend on the number of elements they represent.~~

Edit on 2025-05-17:
There was an important change after C++20 was released that resulted in “view” no longer meaning “non-owning range”. I discuss this in detail in a new blog post.

The algorithm is required to be lazy-evaluated so it is feasible to combine multiple views. More on this below.

The storage behaviour is orthogonal to the range concepts defined by the iterators mentioned above, i.e. you can have a container that satisfies std::ranges::random_access_range (e.g. std::vector does, but std::list does not) and you can have views that do so or don’t.

Views

Lazy-evaluation

A key feature of views is that whatever transformation they apply, they do so at the moment you request an element, not when the view is created.

std::vector vec{1, 2, 3, 4, 5, 6};
auto v = std::views::reverse(vec);

Here v is a view; creating it neither changes vec, nor does v store any elements. The time it takes to construct v and its size in memory is independent of the size of vec.

std::vector vec{1, 2, 3, 4, 5, 6};
auto v = std::views::reverse(vec);
std::cout << *v.begin() << '\n';

This will print “6”, but the important thing is that resolving the first element of v to the last element of vec happens on-demand. This guarantees that views can be used as flexibly as iterators, but it also means that if the view performs an expensive transformation, it will have to do so repeatedly if the same element is requested multiple times.

Combinability

You may have wondered why I wrote

auto v = std::views::reverse(vec);

and not

std::views::reverse v{vec};

That’s because std::views::reverse is not the view itself, it’s an adaptor that takes the underlying range (in our case the vector) and returns a view object over the vector. The exact type of this view is hidden behind the auto statement. This has the advantage, that we don’t need to worry about the template arguments of the view type, but more importantly the adaptor has an additional feature: it can be chained with other adaptors!

std::vector vec{1, 2, 3, 4, 5, 6};
auto v = vec | std::views::reverse | std::views::drop(2);

std::cout << *v.begin() << '\n';

What will this print?

Here is the solution

It will print "4", because "4" is the 0-th element of the reversed string after dropping the first two.

In the above example the vector is “piped” (similar to the unix command line) into the reverse adaptor and then into the drop adaptor and a combined view object is returned. The pipe is just a different notation that improves readability, i.e. vec | foo | bar(3) | baz(7) is equivalent to baz(bar(foo(vec), 3), 7). Note that accessing the 0th element of the view is still lazy, determining which element it maps to happens at the time of access.

Exercise

Create a view on std::vector vec{1, 2, 3, 4, 5, 6}; that filters out all uneven numbers and squares the remaining (even) values, i.e.

std::vector vec{1, 2, 3, 4, 5, 6};
auto v = vec | // ...?

std::cout << *v.begin() << '\n'; // should print 4

To solve this you can use std::views::transform and std::views::filter. Both take a invocable as argument, e.g. a lambda expression. std::views::transform applies the lambda on each element in the underlying range and std::views::filter “removes” those elements that its lambda function evaluates to false for.

Here is the solution

std::vector vec{1, 2, 3, 4, 5, 6};
auto v = vec
       | std::views::filter(   [] (auto const i) { return i % 2 == 0; })
       | std::views::transform([] (auto const i) { return i*i; });

std::cout << *v.begin() << '\n'; // prints 4

View concepts

Views are a specific kind of range that is formalised in the std::ranges::view concept. Every view returned by a view adaptor models this concept, but which other range concepts are modeled by a view?

It depends on the underlying range and also the view itself. With few exceptions, views don’t model more/stronger range concepts than their underlying range (except that they are always a std::ranges::view) and they try to preserve as much of the underlying range’s concepts as possible. For instance the view returned by std::views::reverse models std::ranges::random_access_range (and weaker concepts) iff the underlying range also models the respective concept. It never models std::ranges::contiguous_range, because the third element of the view is not located immediately after the second in memory (but instead before the second).

Perhaps surprising to some, many views also model std::ranges::output_range if the underlying range does, i.e. views are not read-only:

std::vector vec{1, 2, 3, 4, 5, 6};
auto v = vec | std::views::reverse | std::views::drop(2);

*v.begin() = 42; // now vec == {1, 2, 3, 42, 5, 6 } !!

Exercise

Have a look at the solution to the previous exercise (filter+transform). Which of the following concepts do you think v models?

Concept	yes/no?
`std::ranges::input_range`
`std::ranges::forward_range`
`std::ranges::bidirectional_range`
`std::ranges::random_access_range`
`std::ranges::contiguous_range`

`std::ranges::view`
`std::ranges::sized_range`
`std::ranges::output_range`

Here is the solution

Concept	yes/no?
`std::ranges::input_range`	✅
`std::ranges::forward_range`	✅
`std::ranges::bidirectional_range`	✅
`std::ranges::random_access_range`
`std::ranges::contiguous_range`

`std::ranges::view`	✅
`std::ranges::sized_range`
`std::ranges::output_range`

The filter does not preserve random-access and therefore not contiguity, because it doesn’t “know” which element of the underlying range is the i-th one in constant time. It cannot “jump” there, it needs to move through the underlying range element-by-element. This also means we don’t know the size.

The transform view would be able to jump, because it always performs the same operation on every element independently of each other; and it would also preserve sized-ness because the size remains the same. In any case, both properties are lost due to the filter. On the other hand the transform view produces a new element on every access (the result of the multiplication), therefore v is not an output range, you cannot assign values to its elements. This would have prevented modelling contiguous-range as well – if it hadn’t been already by the filter – because values are created on-demand and are not stored in memory at all.

Understanding which range concepts “survive” which particular view needs some practice. For the SeqAn3 library we try to document this in detail, I hope we will see something similar on cppreference.com.

Post scriptum

I am quite busy currently with my PhD thesis, but I plan to publish some smaller articles on ranges and views before the holiday season. Most will be based on text pieces I have already written but that never found their way to this blog (library documentation, WG21 papers, snippets from the thesis, …).

Thanks for reading, I hope this article was helpful! If you have any questions, please comment here or on twitter/mastodon.

A good firewall for a small network

Mon, 24 Jun 2019 00:15:00 +0200

In this article I will outline the setup of my (not so) new firewall at home. I explain how I decided which hardware to get and which software to choose, and I cover the entire process of assembling the machine and installing the operating system. Hopefully this will be helpful to poeple in similar situations.

Introduction

While the ability of firewalls to protect against all the evils of the internets is certainly exaggerated, there are some important use cases for them: you want to prevent certain inbound traffic and manipulate certain outbound traffic e.g. route it through a VPN.

For a long time I used my home server (whose main purpose is network attached storage) to also do some basic routing and VPN, but this had a couple of important drawbacks:

Just one NIC on the server meant traffic to/from the internet wasn’t physically required to go through the server.
Less reliable due to more complex setup → longer downtimes during upgrades, higher chance of failure due to hard drives.
I wouldn’t give someone else the root password to my data storage, but I did want my flat-mates to be able to reset and configure basic network components that they depend on (Router/Port-forwarding and WiFi).
I wanted to isolate the ISP-provided router more strongly from the LAN as they have a history of security vulnerabilities.

The different off-the-shelf routers I had used over the years had also worked only so-so (even those that were customisable) so I decided I needed a proper router. Since WiFi access was already out-sourced to dedicated devices I really only needed a filtering and routing device.

Hardware

Board & CPU

The central requirements for the device were:

low energy consumption
enough CPU power to route traffic at Gbit-speed, run Tor and OpenVPN (we don’t have Gbit/s internet in Berlin, yet, but I still have hopes for the future)
hardware crypto support to unburden the CPU for crypto tasks
two NICs, one for the LAN and one for the WAN

I briefly thought about getting an ARM-based embedded board, but most reviews suggested that the performance wouldn’t be enough to satisfy my requirements and also the *BSD support was mixed at best and I didn’t want to rule out running OpenBSD or FreeBSD.

Back to x86-land: I had previously used PC Engines ALIX boards as routers and was really happy with them at the time. Their new APU boards promised better performance, but thanks to the valuable feedback and some benchmarking done by the community over at BSDForen.de, I came to the conclusion that they wouldn’t be able to push more than 200Mbit/s through an OpenVPN tunnel.

In the end I decided on the Gigabyte J3455N-D3H displayed at the top. It sports a rather atypical Intel CPU (Celeron J3455) with

four physical cores @ 1.5Ghz
AESNI support
10W TDP

Having four actual cores (instead of 2 cores + hyper threading) is pretty cool now that many security-minded operating systems have started deactivating hyper threading to mitigate CPU bugs [OpenBSD] [HardenedBSD]. And the power consumption is also quite low.

I would have liked for the two NICs on the mainboard to be from Intel, but I couldn’t find a mainboard at the time that offered this (other than super-expensive SuperMicro boards). At least the driver support on modern Realteks is quite good.

Storage & Memory

The board has two memory slots and supports a maximum of 4GiB each. I decided 4GiB are enough for now and gave it one module to allow for future extensions (I know that’s suboptimal for speed).

Storage-wise I originally planned on putting a left-over SATA-SSD into the case, but in the end, I decided a tiny USB3-Stick would provide sufficient performance and be much easier to replace/debug/…

Case & Power

Since I installed a real 19" wrack in my new flat, of course the case for the firewall would have to fit nicely into that. I had a surprisingly difficult time finding a good case, because I wanted one were the board’s ports would be front-facing. That seems to be quite a rare requirement, although I really don’t understand why. Obviously having the network ports, serial ports and USB-Ports to the front makes changing the setup and debugging so much easier ¯\_(ツ)_/¯

I also couldn’t find a good power supply for such a low-power device, but I still had a 60W PicoPSU supply lying around.

Even though it came with an overpowered PSU and a proprietary IO-Shield (more on that below), I decided on the SuperMicro SC505-203B. It really does look quite good, I have to say!

Assembly

Mounting the mainboard in the case is pretty straight-forward. The biggest issue was the aforementioned proprietary I/O-Shield that came with the SuperMicro case (and was designed only for SuperMicro-boards). It was possible to remove it, however, the resulting open space did not conform to ATX spec so it wasn’t possible to just fit the Gigabyte board’s shield into it.

I quickly took the measurements and starting cutting away on the shield to make it fit. This worked ok-ish in the end, but is more dangerous than it looks (be smarter than me, wear gloves ☝ ). In retrospect I also recommend that you do not remove the bottom fold on the shield, only left, right and top; that will make it hold a lot better in the case opening.

The board can be fit into the case using standard screws in the designated places. As mentioned above, I removed the original (actively cooled) power supply unit and used the 60W PicoPSU that I had lying around from before. Since it doesn’t have the 4-pin CPU cable I had improvise. There are adaptors for this, but if you have a left-over power supply, you can also tape together something. I also put the transformer into the case (duck-tape, yeah!) so that one can plug in the power cord from the back of the case as usual.

Software

Choice

There are many operating systems I could have chosen since I decided to use an x86 platform. My criteria were:

free software (obviously)
intuitive web user interface to do at least the basic things
possibility to login via SSH if things don’t go as planned
OpenVPN client

I feel better with operating systems based on FreeBSD or OpenBSD, mainly because I have more experience with them than with GNU/Linux distributions nowadays. In previous flats I had also used OpenWRT and dd-wrt based routers, but whenever I needed to tweak something beyond what the web interface offered, it got really painful. In general the whole IPtables based stack on Linux seems overly complicated, but maybe that’s just me.

In any case, there are no OpenBSD-based router operating systems with web interfaces (that I am aware of) so I had the choice between

pfsense (FreeBSD-based)
OPNSense, fork of pfsense, based on HardenedBSD / FreeBSD

There seem to be historic tensions between the people involved in both operating systems and I couldn’t find out if there were actual distinctions in the goals of the projects. In the end, I asked other people for recommendations and found the interface and feature list of OPNSense more convincing. Also, being based on HardenedBSD sounds good (although I am not sure if HardenedBSD-specifica will really ever play out on the router).

Initially I had some issues with the install and OPNSense people were super-friendly and responded immediately. Also the interface was a lot better than I expected so I am quite sure I made the right decision.

Install

Setup is very easy:

Go to https://opnsense.org/download/, select amd64 and nano and download the image.
Unzip the image (easy to forget this).
Write the image to the USB-stick with dd (as always with dd: be careful about the target device!)
Optionally plug a serial cable into the top serial port (the mainboard has two) and connect your Laptop/Desktop with baud rate 115200
Plug the USB-stick into the firewall and boot it up.

There will be some beeping when you start the firewall. Some of this is due to the mainboard complaining that no keyboard is attached (can be ignored) and also OPNSense will play a melody when it is booted. If you are attached to the serial console you can select which interface will be WAN and which will be LAN (and their IP addresses). Otherwise you might need to plug around the LAN cables a bit to find out which is configured as which.

When I built this last year there were some more issues, but all of them have been resolved by the OPNSense people so it really is “plug’n’play”; I verified by doing a re-install!

Post-install

Go to the configured IP-address (192.168.1.1 by default) and login (root: opnsense by default). If the web-interface comes up everything has worked fine and you can disconnect serial console and do the rest via the web-interface.

After login, I would to the following:

change the password
activate SSH on the LAN interface
configure internet access and DHCP
setup any of the other services you want

For me setting up the internet meant doing a “double-NAT” with the ISP-provided router, because I need its modem and nowadays it seems impossible to get a stand-alone VDSL modem. If you do something similar just configure internet as being over DHCP.

If you want hardware accelerated SSL (also OpenVPN), go to System → Firmware → Setting and change the firmware flavour to OpenSSL (instead of LibreSSL). After that check for updates and upgrade. In the OpenVPN profile, under Hardware Crypto, you can now select Intel RDRAND engine - RAND.

Take your time to look through the interface! I found some pretty cool things like automatic backup of the configuration to a nextcloud server! The entire config of the firewall rests in one file so it’s really easy to setup a clean system from scratch.

All-in-all I am very happy with the system. Even though my setup is non-trivial, with only selected outgoing traffic going through the VPN (based on rules), I never had to get my hands dirty on the command line – everything can be done through the Web-UI.

Tutorial: Writing your first view from scratch (C++20 / P0789)

Wed, 11 Apr 2018 13:45:00 +0200

C++17 was officially released last year and the work on C++20 quickly took off. A subset of the Concepts TS was merged and the first part of the Ranges TS has been accepted, too. Currently the next part of the Ranges TS is under review: “Range Adaptors and Utilities”. It brings the much-hyped “Views” to C++, but maybe you have been using them already via the Range-V3 library? In any case you might have wondered what you need to do to actually write your own view. This is the first in a series of blog posts describing complete view implementations (not just adaptations of existing ones).

DISCLAIMER 2023-02-27

This article does not reflect the final state of ranges and views (although nothing about them is really final as they keep changing even within C++20). It likely does not work with current ranges from the standard libary!

I might write a new post at some point, but the current one is outdated.

Introduction (skip this if you have used views before)

Ranges are an abstraction of “a collection of items”, or “something iterable”. The most basic definition requires only the existence of begin() and end(), their comparability and begin being incrementable, but more refined range concepts add more requirements.

The ranges most commonly known are containers, e.g. std::vector. Containers are types of ranges that own the elements in the collection, but in this blog post we are more interested views.

What are views?

Views are ranges that usually (but not always!) performs an operation on another range. They are lazy-evaluated stateful algorithms on ranges that present their result again as a range. And they can be chained to combine different algorithms which can be done via the | operator like on the UNIX command line.

Ok, sounds cool, what does this mean in practice?

Well, you can, e.g. take a vector of ints, apply a view that computes the square of every element, and then apply a view that drops the first two elements:

 std::vector<int> vec{1, 5, 6, 8, 5};
 auto v = vec | view::transform([] (int const i) { return i*i; }) | view::drop(2);
 std::cout << *std::begin(v) << '\n'; // prints '36'

And the point here is that only one “squaring-operation” actually happens and that it happens when we dereference the iterator, not before (because of lazy evaluation!).

What type is v? It is some implementation defined type that is guaranteed to satisfy certain range concepts: the View concept and the InputRange concept. The view concept has some important requirements, among them that the type is “light-weight”, i.e. copy’able in constant time. So while views appear like containers, they behave more like iterators.

If you are lost already, I recommend you check out some of the following resources

Introduction to the C++ Ranges Library on FluentCpp (Blog Post and Video)
Ranges for the Standard Library – CppCon2015 talk by EricNiebler
Range-V3 Documentation

Prerequisites

The following sections assume you have a basic understanding of what a view does and have at least tried some of the toy examples yourself.

DISCLAIMER: Although I have been working with views and range-v3 for a while now, I am surprised by things again and again. If you think I missed something important in this article I would really appreciate feedback!

In general this post is aimed at interested intermediate C++ programmers, I try to be verbose with explanations and also provide many links for further reading.

You should have a fairly modern compiler to test the code, I test with GCC7 and Clang5 and compile with -std=c++17 -Wall -Wextra.

I refer to constraints and concepts in some of the examples. These are not crucial for the implementation so if they are entirely unfamiliar to you, just skip over them. If you use GCC on the other hand, you can uncomment the respective sections and add -fconcepts to your compiler call to activate them.

While the views we are implementing are self-contained and independent of the range-v3 library, you should get it now as some of our checks and examples require it.

And you should be curious of how to make your own view, of course 😄

Adapting existing views

Our task in this post is to write a view that works on input ranges of uint64_t and always adds the number 42, i.e. we want the following to work:

 int main()
 {
     std::vector<uint64_t> in{1, 4, 6, 89, 56, 45, 7};

     for (auto && i : in | view::add_constant)
         std::cout << i << ' ';
     std::cout << '\n'; // should print: 43 46 48 131 98 87 49

     // combine it with other views:
     for (auto && i : in | view::add_constant | ranges::view::take(3))
         std::cout << i << ' ';
     std::cout << '\n'; // should print: 43 46 48
 }

Most of the time it will be sufficient to adapt an existing view and whenever this is feasible it is of course recommended. So the recommended solution to the task is to just re-use ranges::view::transform:

 #include <iostream>
 #include <range/v3/view/transform.hpp>
 #include <range/v3/view/take.hpp>

 namespace view
 {
 auto const add_constant = ranges::view::transform([] (uint64_t const in)
                                                   {
                                                      return in + 42;
                                                   });
 }

 int main()
 {
     std::vector<uint64_t> in{1, 4, 6, 89, 56, 45, 7};

     for (auto && i : in | view::add_constant)
         std::cout << i << ' ';
     std::cout << '\n'; // should print: 43 47 64 131 98 87 49

     // combine it with other views:
     for (auto && i : in | view::add_constant | ranges::view::take(3))
         std::cout << i << ' ';
     std::cout << '\n'; // should print: 43 47 64
 }

As you can see, it’s very easy to adapt existing views!

But it’s not always possible to re-use existing views and the task was to get our hands dirty with writing our own view. The official manual has some notes on this, but while abstractions are great for code-reuse in a large library and make the code easier to understand for those that know what lies behind them, I would argue that they can also obscure the actual implementation for developers new to the code base who need to puzzle together the different levels of inheritance and template specialisation typical for C++ abstractions.

So in this post we will develop a view that does not depend on range-v3, especially not the internals.

The components of a view

What is commonly referred to as a view usually consists of multiple entities:

the actual class (template) that meets the requirements of the View concept and at least also InputRange concept; by convention of the range-v3 library it is called view_foo for the hypothetical view “foo”.
an adaptor type which overloads the () and | operators that facilitate the “piping” capabilities and return an instance of 1.; by convention of range-v3 it is called foo_fn.
an instance of the adaptor class that is the only user-facing part of the view; by convention of range-v3 it is called foo, in the namespace view, i.e. view::foo.

If the view you are creating is just a combination of existing views, you may not need to implement 1. or even 2., but we will go through all parts now.

The actual implementation

preface

#include <range/v3/all.hpp>
#include <iostream>

template <typename t>
using iterator_t = decltype(std::begin(std::declval<t &>()));

template <typename t>
using range_reference_t = decltype(*std::begin(std::declval<t &>()));

As mentionend previously, including range-v3 is optional, we only use it for concept checks – and in production code you will want to select concrete headers and not “all”.
The iterator_t metafunction retrieves the iterator type from a range by checking the return type of begin().
The range_reference_t metafunction retrieves the reference type of a range which is what you get when dereferencing the iterator. It is only needed in the concept checks. ¹

Both of these functions are defined in the range-v3 library, as well, but I have given minimal definitions here to show that we are not relying on any sophisticated magic somewhere else.

`view_add_constant`

We start with the first real part of the implementation:

template <typename urng_t>
//     requires (bool)ranges::InputRange<urng_t>() &&
//              (bool)ranges::CommonReference<range_reference_t<urng_t>, uint64_t>()
class view_add_constant : public ranges::view_base
{

view_add_constant is a class template, because it needs to hold a reference to the underlying range it operates on; that range’s type urng_t is passed in a as template parameter.
If you use GCC, you can add -fconcepts and uncomment the requires-block. It enforces certain constraints on urng_t, the most basic constraint being that it is an InputRange. The second constraint is that the underlying range is actually a range over uint64_t (possibly with reference or const).
Please note that these constraints are specific to the view we are just creating. Other views will have different requirements on the reference type or even the range itself (e.g. it could be required to satisfy RandomAccessRange).
We inherit from view_base which is an empty base class, because being derived from it signals to some library checks that this class is really trying to be a view (which is otherwise difficult to detect sometimes); in our example we could also omit it.

private:
    /* data members == "the state" */
    struct data_members_t
    {
        urng_t urange;
    };
    std::shared_ptr<data_members_t> data_members;

The only data member we have is (the reference to) the original range. It may look like we are saving a value here, but depending on the actual specialisation of the class template, urng_t may also contain & or const &.
Why do we put the member variables inside an extra data structure stored in a smart pointer? A requirement of views is that they be copy-able in constant time, e.g. there should be no expensive operations like allocations during copying. An easy and good way to achieve implicit sharing of the data members is to put them inside a shared_ptr. Thereby all copies share the data_members and they get deleted with the last copy. ²

In cases where we only hold a reference, this is not strictly required, but in those cases we still benefit from the fact that storing the reference inside the smart pointer makes our view default-constructible. This is another requirement of views – and having a top-level reference member prevents this. [Of course you can use a top-level pointer instead of a reference, but we don’t like raw pointers anymore!]
Other more complex views have more variables or “state” that they might be saving in data_members.

    /* the iterator type */
    struct iterator_type : iterator_t<urng_t>
    {
        using base = iterator_t<urng_t>;
        using reference = uint64_t;

        iterator_type() = default;
        iterator_type(base const & b) : base{b} {}

        iterator_type operator++(int)
        {
            return static_cast<base&>(*this)++;
        }

        iterator_type & operator++()
        {
            ++static_cast<base&>(*this);
            return (*this);
        }

        reference operator*() const
        {
            return *static_cast<base>(*this) + 42;
        }
    };

Next we define an iterator type. Since view_add_constant needs to satisfy basic range requirements, you need to be able to iterate over it. In our case we can stay close to the original and inherit from the original iterator.
For the iterator to satisfy the InputIterator concept we need to overload the increment operators so that their return type is of our class and not the base class. The important overload is that of the dereference operation, i.e. actually getting the value. This is the place where we interject and call the base class’s dereference, but then add the constant 42. Note that this changes the return type of the operation (::reference); it used to be uint64_t & (possibly uint64_t const &), now it’s uint64_t → A new value is always generated as the result of adding 42.
Note that more complex views might require drastically more complex iterators and it might make sense to define those externally. In general iterators involve a lot of boilerplate code, depending on the scope of your project it might make sense to add your own iterator base classes. Using CRTP also helps re-use code and reduce “non-functional” overloads.

We continue with the public interface:

public:
    /* member type definitions */
    using reference         = uint64_t;
    using const_reference   = uint64_t;
    using value_type        = uint64_t;

    using iterator          = iterator_type;
    using const_iterator    = iterator_type;

First we define the member types that are common for input ranges. Of course our value type is uint64_t as we only operate on ranges over uint64_t and we are just adding a number. As we mentioned above, our iterator will always generate new values when dereferenced so the reference types are also value types.
Note: Other view implementation might be agnostic of the actual value type, e.g. a view that reverses the elements can do so independent of the type. AND views might also satisfy OutputRange, i.e. they allow writing to the underlying range by passing through the reference. To achieve this behaviour you would write using reference = range_reference_t<urng_t>;. The value type would then be the reference type with any references stripped (using value_type = std::remove_cv_t<std::remove_reference_t<reference>>;).
The iterator type is just the type we defined above.
In general views are not required to be const-iterable, but if they are the const_iterator is the same as the iterator and const_reference is the same as reference. ³

    /* constructors and deconstructors */
    view_add_constant() = default;
    constexpr view_add_constant(view_add_constant const & rhs) = default;
    constexpr view_add_constant(view_add_constant && rhs) = default;
    constexpr view_add_constant & operator=(view_add_constant const & rhs) = default;
    constexpr view_add_constant & operator=(view_add_constant && rhs) = default;
    ~view_add_constant() = default;

    view_add_constant(urng_t && urange)
        : data_members{new data_members_t{std::forward<urng_t>(urange)}}
    {}

The constructors are pretty much standard. We have an extra constructor that initialises our urange from the value passed in. Note that this constructor covers all cases of input types (&, const &, &&), because more attributes can be stuck in the actual urng_t and because of reference collapsing.

    /* begin and end */
    iterator begin() const
    {
        return std::begin(data_members->urange);
    }
    iterator cbegin() const
    {
        return begin();
    }

    auto end() const
    {
        return std::end(data_members->urange);
    }

    auto cend() const
    {
        return end();
    }
};

Finally we add begin() and end(). Since we added a constructor for this above, we can create our view’s iterator from the underlying range’s iterator implicitly when returning from begin().
For some ranges the sentinel type (the type returned by end()) is not the same as the type returned by begin(), this is only true for BoundedRanges; the only requirement is that the types are comparable with == and !=. We need to take this into account here, that’s why the end function returns auto and not the iterator (the underlying sentinel is still comparable with our new iterator, because it inherits from the underlying range’s iterator).
As noted above, some views may not be const-iterable, in that case you can omit cbegin() and cend() and not mark begin() and end() as const.
Note that if you want your view to be stronger that an InputRange, e.g. also be a SizedRange or even a RandomAccessRange, you might want to define additional member types (size_type, difference_type) and additional member functions (size(), operator[]…). Although strictly speaking the range “traits” are now deduced completely from the range’s iterator so you don’t need additional member functions on the range.

template <typename urng_t>
//     requires (bool)ranges::InputRange<urng_t>() &&
//              (bool)ranges::CommonReference<range_reference_t<urng_t>, uint64_t>()
view_add_constant(urng_t &&) -> view_add_constant<urng_t>;

We add a user-defined type deduction guide for our view.
Class template argument deduction enables people to use your class template without having to manually specify the template parameter.
In C++17 there is automatic deduction, as well, but we need user defined deduction here, if we want to cover both cases of urng_t (value tpye and reference type) and don’t want to add more complex constructors.

static_assert((bool)ranges::InputRange<view_add_constant<std::vector<uint64_t>>>());
static_assert((bool)ranges::View<view_add_constant<std::vector<uint64_t>>>());

Now is a good time to check whether your class satisfies the concepts it needs to meet, this also works on Clang without the Concepts TS or C++20. We have picked std::vector<uint64_t> as an underlying type, but others would work, too.
If the checks fail, you have done something wrong somewhere. The compilers don’t yet tell you why certain concept checks fail (especially when using the range library’s hacked concept implementation) so you need to add more basic concept checks and try which ones succeed and which break to get hints on which requirements you are failing. A likely candidate is your iterator not meeting the InputIterator concept (old, but complete documentation).

`add_constant_fn`

Off to our second type definition, the functor/adaptor type:

struct add_constant_fn
{
    template <typename urng_t>
//         requires (bool)ranges::InputRange<urng_t>() &&
//                  (bool)ranges::CommonReference<range_reference_t<urng_t>, uint64_t>()
    auto operator()(urng_t && urange) const
    {
        return view_add_constant{std::forward<urng_t>(urange)};
    }

    template <typename urng_t>
//         requires (bool)ranges::InputRange<urng_t>() &&
//                  (bool)ranges::CommonReference<range_reference_t<urng_t>, uint64_t>()
    friend auto operator|(urng_t && urange, add_constant_fn const &)
    {
        return view_add_constant{std::forward<urng_t>(urange)};
    }

};

The first operator facilitates something similar to the constructor, it enables traditional usage of the view in the so called function-style: auto v = view::add_constant(other_range);.
The second operator enables the pipe notation: auto v = other_range | view::add_constant;. It needs to be friend or a free function and takes two arguments (both sides of the operation).
Both operators simply delegate to the constructor of view_add_constant.

`view::add_constant`

Finally we add an instance of the adaptor to namespace view:

namespace view
{

add_constant_fn constexpr add_constant;

}

Since the adapter has no state (in contrast to the view it generates), we can make it constexpr. You can now use the adaptor in the above example.

We are done 😊

Here is the full code: view_add_constant.cpp

Post scriptum

I will follow up on this with a second tutorial, it will cover writing a view that takes arguments, i.e.

std::vector<uint64_t> in{1, 4, 6, 89, 56, 45, 7};
auto v = in | view::add_number(42);
// decide this at run-time     ^

If you found mistakes (of which I am sure there are some) or if you have questions, please comment below via GitHub, Gitea, Twitter or Mastodon!

If you are confused that we are dealing with the “reference type” and not the “value type”, remember that member functions like at() and operator[] on plain old containers also always return the ::reference type. ↩︎
This is slightly different than in range-v3 where views only accept temporaries of other views, not of e.g. containers (containers can only be given as lvalue-references). This enables constant time copying of the view even without implicit sharing of the underlying range, but it mandates a rather complicated set of techniques to tell apart views from other ranges (the time complexity of a function is not encoded in the language so tricks like inheriting ranges::view are used). I find the design used here more flexible and robust. ↩︎
This might be confusing to wrap your head around, but remember that the const_iterator of a container is like an iterator over the const version of that container. The same is true for views, except that since the view does not own the elements its own const-ness does not “protect” the elements from being written to. Ranges behave similar to iterators in this regard, an iterator const on a vector can also be used to write to the value it points to. More on this in this range-v3 issue. ↩︎

The - surprisingly limited - usefulness of function multiversioning in GCC

Sat, 09 Dec 2017 18:00:00 +0200

Modern CPUs have quite a few features that generic amd64/intel64 code cannot make use of, simply because they are not available everywhere and including them would break the code on unsupporting platforms. The solution is to not use these features, or ship different specialised binaries for different target CPUs. The problem with the first approach is that you miss out on possible optimisations and the problem with the second approach is that most users don’t know which features their CPUs support, possibly picking a wrong executable (which won’t run → bad user experience) or a less optimised one (which is again problem 1). But there is an elegant GCC-specific alternative: Function multiversioning!

But does it really solve our problems? Let’s have a closer look!

Prerequisites

basic understanding of C++ and compiler optimisations (you should have heard of “inlining” before, but you don’t need to know assembler, in fact I am not an assembly expert either)
Most code snippets are demonstrated via Compiler Explorer, but the benchmarks require you to have GCC ≥ version 7 locally installed.
You might want to open a second tab or window to display the Compiler Explorer along side this post (two screens work best 😎).

Population counts

Many of the CPU features used in machine-optimised code relate to SIMD, but for our example, I will use a more simple operation: population count or short popcount.

The popcount of an integral number is the number of bits that are set to 1 in its bit-representation. [More details on Wikipedia if you are interested.]

Popcounts are used in many algorithms, and are important in bioinformatics (one of the reasons I am writing this post). You could implement a naive popcount by iterating over the bits, but GCC already has us covered with a “builtin”, called __builtin_popcountll (the “ll” in the end is for long long, i.e. 64bit integers). Here’s an example:

  __builtin_popcountll(6ull) // == 2, because 6ull's bit repr. is `...00000110`

To get a feeling for how slow/fast this function is, we are going to call it a billion times. The golden rule of optimisation is to always measure and not make wild assumptions about what you think the compiler or the CPU is/isn’t doing!

  #include <cstdint>

  uint64_t pc(uint64_t const v)
  {
      return __builtin_popcountll(v);
  }

  int main()
  {
      for (uint64_t i = 0; i < 1'000'000'000; ++i)
          volatile uint64_t ret = pc(i);
  }

view in compiler explorer

This code should should be fairly easy to understand, the volatile modifier is only used to make sure that this code is always generated (if not used the compiler will see that the return values are not actually used and optimise all the code away!). In any case, before you compile this locally, click on the link and check out the compiler explorer. Using the colour code you can easily see that our call to __builtin_popcountll in line 5 is translated to another function __popcountdi2 internally. Before you continue, add -03 to the compiler arguments in compiler explorer, this will add machine-independent optimisations. The assembly code should change, but you will still be able to find __popcountdi2.

This is a generic function that works on all amd64/intel64 platforms and counts the set bits. What does it actually do? You can search the net and find explanations that say it does some bit-shifting and table-lookups, but the important part is that it performs multiple operations to compute the popcount in a generic way.

Modern CPUs, however, have a feature that does popcount in hardware (or close). Again, we don’t need to know exactly how this works, but we would expect that this single operation function is better than anything we can do (for very large bit-vectors this is not true entirely, but that’s a different issue).

How do we use this magic builtin? Just go back to the compiler explorer, and add -mpopcnt to the compiler flags, this tells GCC to expect this feature from the hardware and optimise for it. Voila, the assembly code generated now resolves to popcnt rsi, rsi instead of call __popcountdi2 (GCC is smart and it’s builtin resolves to whatever is best on the architecture we are targeting).

But how much better is this actually? Compile both versions locally and measure, e.g. with the time command.

compiler flags	time on my pc
`-O3`	3.1s
`-O3 -mpopcnt`	0.6s

A speed-up of 5x, nice!

But what happens when the binary is run on a CPU that doesn’t have builtin popcnt? The program crashes with “Illegal hardware instruction” 💀

Function multiversioning

This is where function multiversioning (“FMV”) comes to the rescue. It is a GCC specific feature, that inserts a branching point in place of our original function and then dispatches to one of the available “clones” at run-time. You can specify how many of these “clones” you want and for which features or architectures each are built, then the dispatching function chooses the most highly optimised automatically. You can even manually write different function bodies for the different clones, but we will focus on the simpler kind of FMV where you just compile the same function body with different optimisation strategies.

Enough of the talking, here is our adapted example from above:

  #include <cstdint>

  __attribute__((target_clones("default", "popcnt")))
  uint64_t pc(uint64_t const v)
  {
      return __builtin_popcountll(v);
  }

  int main()
  {
      for (uint64_t i = 0; i < 1'000'000'000; ++i)
          volatile uint64_t ret = pc(i);
  }

view in compiler explorer

The only difference is that line 3 was inserted. The syntax is quite straight-forward insofar as anything in the C++ world is straight-forward 😉 :

We are telling GCC that we want two clones for the targets “default” and “popcnt”.
Everything else gets taken care of.

Follow the link to compiler explorer and check the assembly code (please make sure that you are not specifying -mpopcnt!). It is a little longer, but we immediately see via the colour code of __builtin_popcountll(v) that two functions are generated, one with the generic version and one with optimised version, similar to what we had above, but now in one program. The “function signatures” in the assembly code also tell us that one of them is “the original” and one is “popcnt clone”. Some further analysis will reveal a third function, the “clone .resolver” which is the dispatching function. Even without knowing any assembly you might be able to pick out the statement that looks up the CPU feature and calls the correct clone.

Great! So we have a single binary that is as fast as possible and works on older hardware. But is it really as fast as possible? Compile and run it!

version	compiler flags	time on my pc
original	`-O3`	3.1s
original	`-O3 -mpopcnt`	0.6s
FMV	`-O3`	2.2s

Ok, we are faster than the original generic code so we are probably using the optimised popcount call, but we are nowhere near our 5x speed-up. What’s going on?

Nested function calls

We have replaced the core of our computation, the function pc() with a dispatcher that chooses the best implementation. As noted above this decision happens at run-time (it has to, because we can’t know beforehand if the target CPU will support native popcount, it’s the whole point of the exercise), but now this happens one billion times!

Wow, this check seems to be more expensive than the actual popcount call. If you write a lot of optimised code, this won’t be a surprise, decision making at run-time just is very expensive.

What can we do about it? Well, we could decide between generic VS optimised before running our algorithm, instead of deciding in our algorithm on every iteration:

  #include <cstdint>

  uint64_t pc(uint64_t const v)
  {
      return __builtin_popcountll(v);
  }

  __attribute__((target_clones("default", "popcnt")))
  void loop()
  {
      for (uint64_t i = 0; i < 1'000'000'000; ++i)
          volatile uint64_t ret = pc(i);
  }

  int main()
  {
      loop();
  }

view in compiler explorer

The assembly of this gets a little more messy, but you can follow around the jmp instructions or just scan the assembly for our above mentioned instructions and you will see that we still have the two versions (although the actual pc() function is not called, because it was inlined and is moved around a bit).

Compile the code and measure the time:

version	compiler flags	time on my pc
original	`-O3`	3.1s
original	`-O3 -mpopcnt`	0.6s
FMVed `pc()`	`-O3`	2.2s
FMVed loop	`-O3`	0.6s

Hurray, we are back to our original speed-up!

If you expected this, than you likely have dealt with strongly templated code before and also heard of tag-dispatching, a technique that can be used to translate arbitrary run-time decisions to different code-paths beneath which you can treat your run-time decision as a compile-time one.

Our simplified callgraph for the above cases looks like this (the dotted line is where the dispatching takes place):

In real world code the graph is of course bigger, but it should become obvious that by moving the decision making further to the left, the code becomes faster – because we have to decide less often –, but also the size of the generated executable becomes larger – because more functions are actually compiled. [There are corner cases where the executable being bigger actually results in certain things becoming slower, but lets not get into that now.]

Anyway, I thought that FMV would be like dispatching a tag down the call-graph, but it’s not! In fact we just got lucky in our above example, because the pc() call was inlined. Inlining means that the function itself is optimised away entirely and its code is inserted at the place in the calling function where the function call would have been otherwise. Only because pc() is inlined, do we actually get the opimisation!

How do you know? Well you can force GCC to not inline pc():

  #include <cstdint>

  __attribute__((noinline))
  uint64_t pc(uint64_t const v)
  {
      return __builtin_popcountll(v);
  }

  __attribute__((target_clones("default", "popcnt")))
  void loop()
  {
      for (uint64_t i = 0; i < 1'000'000'000; ++i)
          volatile uint64_t ret = pc(i);
  }

  int main()
  {
      loop();
  }

view in compiler explorer

Just add the third line to your previous Compiler Explorer window, or open the above link. You can see that the optimised popcnt call has disappeared from the assembly and pc() only appears once. So in fact our callgraph is (no optimised pc() contained):

But how serious is this, you may ask? Didn’t the compiler inline automatically? Well, the problem about inlining is, that it is entirely up to the compiler whether it inlines a function or not (prefixing the function with inline does in fact not force it to). The deeper the call-graph gets, the more likely it is for the compiler not to inline all the way from the FMV invocation point.

Trying to save FMV for our use case

click to see some more complex but futile attempts

It’s possible to force the compiler to use inlining, but it’s also non-standard and it obviously doesn’t work if the called functions are not customisable by us (e.g. stable interfaces or external code / a library). Furthermore it might not even be desirable to force inline every function / function template, because they might be used in other places or with differently typed arguments resulting in an even higher increase of executable size.

An alternative to inlining would be to use the original form of FMV where you actually have different function bodies and in those add a custom layer of (tag-)dispatching yourself:

  #include <cstdint>

  template <bool is_optimised>
  __attribute__((noinline))
  uint64_t pc(uint64_t const v)
  {
      return __builtin_popcountll(v);
  }

  __attribute__((target("default")))
  void loop()
  {
      for (uint64_t i = 0; i < 1'000'000'000; ++i)
          volatile uint64_t ret = pc<false>(i);
  }

  __attribute__((target("popcnt")))
  void loop()
  {
      for (uint64_t i = 0; i < 1'000'000'000; ++i)
          volatile uint64_t ret = pc<true>(i);
  }

  int main()
  {
      loop();
  }

view in compiler explorer

In this code example we have turned pc() into a function template, customisable by a bool variable. This means that two versions of this function can be instantiated. We then also implement the loops separately and make each pass a different bool value to pc() as a template argument. If you look at the assembly in compiler explorer you can see that two functions are created for pc(), but unfortunately they both contain the unoptimised popcount call¹. This is due to the compiler not knowing/assuming that one of the functions is only called in an optimised context. → This method won’t solve our problem.

And while it is of course possible to add C++17’s if constexpr to pc() and start hacking custom code into the the function depending on the template parameter, it does further complicate the solution moving us further and further away from our original goal of a thin dispatching layer.

¹ Since the resulting function bodies are the same they are actually merged into a single one at optimisation levels > 1 (but this is independent of our problem).

Summary

Function multiversioning is a good thing, because it aims to solve an actual problem: delivering optimised binary code to users that can’t or don’t want to build themselves.
Unfortunately it does not multiversion the functions called by a versioned function, forcing developers to move FMV very close to the intended function call.
This has the drawback of invoking the dispatch much more often than theoretically needed, possibly incurring a penalty in run-time that might exceed the gain from more highly optimised code.
It would be great if GCC developers could address this by adding a version of FMV that recursively clones the indirectly invoked functions (without further branching), as well as providing the machine-aware context to these clones, i.e. the presumed CPU features.

Using Gitea and/or Github to host blog comments

Sat, 21 Oct 2017 18:00:00 +0200

After having moved from FSFE’s wordpress instance I thought long about whether I still want to have comments on the new blog. And how I would be able to do it with a statically generated site. I think I have found/created a pretty good solution that I document below.

How it was before

To be honest, Wordpress was a spam nightmare! I ended up excluding all non-(FSFE-)members, because it was just too difficult to get right. On the other hand I value feedback to posts so what to do?

This blog is now statically generated so it is not designed for comments anyway. The most common solution seems to be Disqus which seems to work well, but be a privacy nightmare. It hosts your comments on their server and integrates with all sorts of authentication services, of course sharing data with them et cetera. Not exposing my site visitors to tracking is very important to me and I also don’t want to advertise using your Facebook login or some such nonsense.

A good idea

However, I had vague memories of having read this article a while ago so I read up on it again:

http://donw.io/post/github-comments/

The idea is to host your comments in a GitHub bug tracker and load them dynamically via Javascript and the GitHub-API. It integrates with GoHugo, the site-generator I am also using, so I thought I’d give it a try. Please read the linked article to get a clearer picture of the idea.

Privacy improvements and other changes

It all worked rather well, but there were a few things I was unhappy with so I changed the following:

In addition to GitHub, it now works with Gitea, a Free Software alternative, too; this includes dynamically generating Markdown from the comments via ShowdownJS, because Gitea’s API is less powerful than GitHub’s.
The comments are not loaded automatically, but on-demand (so visitors don’t automatically make requests to other servers).
It is possible to have multiple instances of the script running, with different server types, target domains and/or repos.
Gracefully degrade and offer external links if no Javascript is available.
Some visual changes to fit with my custom theme.

You can see the results below. I am quite happy with the solution as many of my previous readers from FSFE can still use FSFE’s infrastructure to reply (in this case FSFE’s gitea instance). I expect many other visitors to have a GitHub account so they don’t need to sign up for another service. I am aware this still relies on third parties and that GitHub may at some point commodify the use of its API, but right now it is much better than to store and share the data with a company whose business model this already is. And it is optional.

And of course the blog itself will remain entirely free of Javascript!

The important files are available in this blog’s repo:

What do you think? Feel free to adapt this for your blog and thanks to Don Williamson for the original implementation!

New blog

Mon, 18 Sep 2017 16:00:00 +0200

I have moved my blog from https://blogs.fsfe.org/h2 to my own web-space. On the way I have switched from Wordpress to GoHugo. I hope to publish more often and will increasingly cover programming topics.

Retroactively replacing git subtree with git submodule

Wed, 27 Jul 2016 13:38:26 +0000

Combining multiple git repositories today is rather common, although the means of doing so are far from perfect. Usually people use git submodule or git subtree. If you have used neither or are happy with either method this post is completely irrelevant to you.

But maybe you decided to use git subtree and like me are rather unhappy with the choice. I will not discuss the upsides and downsides of either approach, I assume you want to change this, and you wish you could go back in time and make it right for all subsequent development. So this is what we are going to do :)

This was an exercise in git for me so I am sure more experienced people might know shortcuts, but since it took me quite some time to figure out right, I hope it is going to be helpful to others. It is however strongly recommended to read the shell scripts and possibly adapt them to your situation. I will not be held responsible for any problems!

It should be noted that by definition any changes in your git history will break current clones and forks of your repository so only do this if you are very sure what you are doing. And be nice and inform your users about this ASAP.

Pre conditions

you have a git subtree, e.g. of a library, inside your repo and you want to replace it with git submodule
going back to a commit in your history you should still get the same version of the library that you had previously included with git subtree
i.e. every subtree update should be replaced with a submodule update of the same contents and also timestamp (because this is timetravel, right?)
all tags should be preserved / rewritten to their corresponding new IDs
when updating your git subtree previously you used the “squash” feature (if you didn’t do this, it is going to be a lot harder)
you have a mostly straight branch history and are ok with loosing merge commits; all branches other than master will be rebased on master

When I say “preserve timestamp” it is important to clarify what this means: git has two notions of “timestamp”, the “author date” and the “committer date”. The author date is the date where the commit was actually created. It is apparently entirely without relevance for git branches and their structure. The committer date on the other hand is the time where the commit was included in the branch. This is often the same, but when doing operations on a branch like cherry-picking or rebasing then it is overwritten with the current date. Unfortunately git interfaces like github don’t use either date consistently so if you want a consistent appearance you need to also preserve the committer dates. Editing these however can have adverse effects potentially breaking branches or tags, because the correct chronological order of committer dates is important. We will take extra precautions below, but if you branches somehow get mangeled, or a commit appears out of place with gitg than you should double check the committer dates.

Preperation

Make a local clone of the repository and after that remove all your “remotes” with git remote remove . This ensures that you don’t accidentally push any changes…

I assume that you have checked out the master branch and that you have exported the following environment variables:

CLONE="~/devel/myapp"                      # the directory of your clone
SUBDIR="include/mylib"                     # the subdir of the subtree/submodule relative to CLONE 
SUBNAME="mylib"                            # name of the submodule (can be anything)
SUBREPO="git://github.com/mylib/mylib.git" # submodules' repo

ATTENTION: don’t screw up any of the above paths since we might be calling rm -rf on them.

If you are uncomfortable with doing search and replace operations on the command line, you can set your editor to something easy like kwrite: export EDITOR=kwrite

Replacing the subtree and subtree updates

Since we are going to delete all the merge commits and also the commits that represent changes to the subtree, we need to remember at which places we later re-insert commits. To do this, run the following:

git log --format='%at:::%an:::%ae:::%s' --no-merges | awk -F ':::' '
(PRINT == 1) && !($4 ~ /^Squashed/) {
    PRINT=0
    printf    $1 ":::" $2    ":::" $3    ":::" $4   ":::"  # commit that we work on
    printf tTIME ":::" tNAME ":::" tMAIL ":::" tREF "\n"   # commit that we insert
};
(PRINT != 1) && ($4 ~ /^Squashed/) {
    PRINT=1
    tTIME=$1
    tNAME=$2
    tMAIL=$3
    tREF=substr($0, length($0) - 6, length($0)) # cut commit id from subj
}; ' > /tmp/refinserts

What it does is find for every commit that starts with “Squashed” the subsequent commit’s subject and associate with it time of the subtree update and the subtree’s commitID, seperated with “:::”. We are paring this information with the subject line and not the commit ID, because the commit IDs in our branch are going to change! It also collapses subsequent updates to one. NOTE that if you have other commits that start wit “Squashed” in there subject line but don’t belong to the subtree, instead filter for Squashed '${SUBDIR}' (beware of the quotes!!).

Now create a little helper script, e.g. as /tmp/rebasehelper:

#!/bin/sh

reset()
{
    TIME_NAME_MAIL_SUBJ=$(git log --format='%at:::%an:::%ae:::%s:::' -1)
    # next commit time is last commit time if not overwritten
    export GIT_AUTHOR_DATE=$(git log --format='%at' -1)
}

cp /tmp/refinserts /tmp/refinserts.actual
reset

while $(grep -q -F "${TIME_NAME_MAIL_SUBJ}" /tmp/refinserts.actual); do

    LINE=$(grep -F "${TIME_NAME_MAIL_SUBJ}" /tmp/refinserts.actual)
    export  GIT_AUTHOR_DATE=$(echo $LINE | awk -F ':::' '{ print $5 }')
    export  GIT_AUTHOR_NAME=$(echo $LINE | awk -F ':::' '{ print $6 }')
    export GIT_AUTHOR_EMAIL=$(echo $LINE | awk -F ':::' '{ print $7 }')
                        REF=$(echo $LINE | awk -F ':::' '{ print $8 }')

    if [ ! -d "${SUBDIR}" ]; then
        echo "** First commit with submodule, initializing..."
        git submodule --quiet add --force --name ${SUBNAME} ${SUBREPO} ${SUBDIR} > /tmp/rebasehelper.log
        [ $? -ne 0 ] && echo "** failed:" && cat /tmp/rebasehelper.log && break

        echo "** done."
    fi

    echo "** Updating submodule..."
    cd "${SUBDIR}"
    git checkout --quiet $REF > /tmp/rebasehelper.log
    [ $? -ne 0 ] && echo "** failed:" && cat /tmp/rebasehelper.log && break

    echo "** Committing changes..."
    cd ${CLONE}
    git commit --quiet -am "[${SUBNAME}] update to $REF" > /tmp/rebasehelper.log
    [ $? -ne 0 ] && echo "** failed:" && cat /tmp/rebasehelper.log && break

    echo "** Continuing rebase."
    rm /tmp/rebasehelper.log
    grep -v -F "${TIME_NAME_MAIL_SUBJ}" /tmp/refinserts.actual > /tmp/refinserts.new
    mv /tmp/refinserts.new /tmp/refinserts.actual
    git rebase --continue
    reset
done

if [ -d "${CLONE}/.git/rebase-merge" ] || [ -d "${CLONE}/.git/rebase-apply" ]; then
    echo "The current rebase step is not related to subtree-submodule operation or needs manual resolution."
    echo "Try 'git mergetool', followed by 'git rebase --continue' or just the latter."
fi

We will call this later.

Filter-Tree

Now we actually remove references to the subtree from our history so that future operations create no conflicts:

git filter-branch --tree-filter 'rm -rf '"${SUBDIR}" HEAD

This may take some time. Other sources recommend --index-filter but that will not work because the file-references in the subtree are not relative to our repository, but to the SUBDIR. If this command doesn’t actually remove the directory, make sure to run rm -rf "${SUBDIR}".

Rebase

Now we do the rebase:

git rebase --interactive $(git log --format='%H' | tail -n 1)

Which will open your commit history in the EDITOR that you configured. NOTE that this is in chronological order, not reverse chronological like the git log command.

In the editor you now want to remove all lines that contain “Squashed” and mark the previous commit for being edited. This is multiline-regex with substitution, but in kwrite this is very straightforward:

Search:  pick(.*\n).*Squashed.*\n
Replace: e \1

This will only miss subsequent “double” updates which can safely be removed:

Search:  .*Squashed.*\n
Replace:

Save and close the editor, you will now be at the first commit that you are editing. This is the original point in time where you added your subtree, source the helper script with . /tmp/rebasehelper. If there were never any conflicts in your tree the script should run through completely and you are done. It is important to source the script with a leading . because it sets some environment variables also for your manual commits.

However, if you did have conflicts, you will be interrupted to resolve these manually, usually by git mergetool and then git rebase --continue. Whenever the rebase tells you “Stopped at…”, just call . /tmp/rebasehelper again and keep repeating the last steps, until the rebase is finished. If you are doing the whole thing on multiple branches you might want to run git rerere before every git mergetool, it might save you some merge steps.

Unfortunately the rebase will set all commit dates to the current date (although the “author date” is preserved). There is a an option that prevents exactly this, but which cannot be used together with interactivity, because the interactivity introduces commits with a newer author date (which is okay in itself but committer dates need to be chronological for rebase). We write this little scriplet to /tmp/redate to help us out:

#!/bin/sh
git filter-branch --force --env-filter '
    LAST_DATE=$(cat /tmp/redate_old);
    GIT_COMMITTER_DATE=$( (echo $LAST_DATE; echo $GIT_AUTHOR_DATE) | awk '"'"'substr($1, 2) > MAX { MAX = substr($1, 2); LINE=$0}; END { print LINE }'"'"');
    echo $GIT_COMMITTER_DATE > /tmp/redate_old;
    export GIT_COMMITTER_DATE' $@

What it does is set the COMMITTER_DATE to the AUTHOR_DATE unless the COMMITTER_DATE would be older than the last COMMITTER_DATE (which is illegal) – in that case it uses the same COMMITTER_DATE as the previous commit. This ensures correct chronology while still having sensible dates in all cases.

And then we call the script with some pre and post commands:

echo '@0 +0000' > /tmp/redate_old
/tmp/redate
git log --format='%ad' --date=raw -1 | awk '{ print "@" $1+1 " " $2 }' > /tmp/redate_master

(initialization and saving master’s final timestamp+1 for later)

Multiple Branches

For all other affected branches it is now much simpler. First checkout the branch. If your subtree/submodule is large, you might have to delete ${SUBDIR} before that with rm -rf.

Then filter the tree as described above, you might need to add force to overwrite some backups:

git filter-branch -f --tree-filter 'rm -rf '"${SUBDIR}" HEAD

Followed by a rebase on the already-fixed master branch:

git rebase --interactive master

You should now see all diverging commits, plus all the “Squashed*” commits. If you did make subtree updates on other branches and you want to retain them, then apply the same mechanism as for the master branch. If subtree changes in the other branch are nor important, you can just remove all of them from the file.

Now the committer dates in the part of the branch that are identical to master are correct (because we fixed those earlier), but those in the new part have updated commit times. We can just use our previous scriptlet, but this time we initialize with the master’s last timestamp and we only operate on the commits that are actually new:

cp /tmp/redate_master /tmp/redate_old
/tmp/redate_master master..HEAD

voila, and repeat for the other branches!

Rewiring the tags

Currently all your tags should still be there and also still be valid. Print them to a tmpfile and open them:

git tag -l > /tmp/oldtags
$EDITOR /tmp/oldtags

Review the list of tags and remove all tags that belonged to the submodule or that you don’t want to keep in the new repository from the file.

Now checkout master again, then run this nifty script (reed the note below first!):

#!/bin/sh

OLDIFS=$IFS
IFS='
'

TAGS=$(cat /tmp/oldtags)

for TAG_NAME in $TAGS; do
    IFS=$OLDIFS
    TAG_COMMIT=$(git for-each-ref --format="%(objectname)" refs/tags/${TAG_NAME})
    TAG_MESSAGE=$(git for-each-ref --format="%(contents)" refs/tags/${TAG_NAME})

    # the commit that is referenced by the annotated tag in the original branch
    ORIGINAL_COMMIT=$(git log --format='%H' --no-walk ${TAG_COMMIT})
    ORIGINAL_COMMIT_DATE=$(git log --format='%at' --no-walk ${ORIGINAL_COMMIT})
    ORIGINAL_COMMIT_SUBJECT=$(git log --format='%s' --no-walk ${ORIGINAL_COMMIT})
    # the same commit in our rewritten branch
    NEW_COMMIT=$(git log --format='%H:::%at:::%s' | awk -F ':::' '
    $2 == "'"$ORIGINAL_COMMIT_DATE"'" && $3 == "'"$ORIGINAL_COMMIT_SUBJECT"'" { print $1 }')

    # overwrite git environment variables directly:
    export GIT_COMMITTER_DATE=$(git for-each-ref --format="%(taggerdate:rfc)" refs/tags/${TAG_NAME})
    export GIT_COMMITTER_NAME=$(git for-each-ref --format="%(taggername)" refs/tags/${TAG_NAME})
    export GIT_COMMITTER_EMAIL=$(git for-each-ref --format="%(taggeremail)" refs/tags/${TAG_NAME})
    export GIT_AUTHOR_DATE=$GIT_COMMITTER_DATE
    export GIT_AUTHOR_NAME=$GIT_COMMITTER_NAME
    export GIT_AUTHOR_EMAIL=$GIT_COMMITTER_EMAIL

     # add the new tag
     git -c user.name="${GIT_COMMITTER_NAME}" -c user.email="${GIT_COMMITTER_EMAIL}" tag -f -a "${TAG_NAME}" -m "${TAG_MESSAGE}" $NEW_COMMIT

    IFS='
'
done

NOTE that for this to work, all tags must be contained in the master branch (or whichever branch you are currently on. If this is not the case, you need to create individual files with the tags of each branch and run this script repeatedly on the specific file while being on the corresponding branch.

NOTE2: This does assume that the COMMITTER_DATE of the tag will fit in at the place it is being added. I am not sure if there are edge cases where one would have to double-check the committer date similarly to how we do it above…

Since your tags are all fixed, now would be a good time to checkout some of them and verify that your software builds, passes its unit checks et cetera. Remember that in contrast to git subtree you need to manually reset your submodule via submodule update after you checkout the tag if you actually want the submodule’s revision that belonged to the tag (which is the whole point of our exercise).

Cleanup

Ok, so the new branches and tags are in place, but the folder is even bigger than before :o Now we get to clean up.

First make sure that absolutely nothing references the old stuff, i.e. delete all tags that you did not change in the previous step with git tag -d. Also make sure that you have no remotes set. If in doubt, open a git-gui like gitk or gitg with the --all parameter and confirm that only your new trees are listed.

Then perform the actual clean-up:

git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --prune=now --aggressive

To double-check call something like du -c . in your directory and look at the output. You should now see that the big directories are all related to the submodule.

Pushing the changes

Finally we will publish the changes. This is the part that is irreversible. You can take extra pre-cautions by forking your upstream and pushing to the fork first, e.g. if your repository is at https://github.com/alice/myapp make a fork to https://github.com/alicia/myapp or something like that.

To work on alicia:

git remote add alicia git@github.com:alicia/myapp.git

In any case you need to delete all tags that are not in your local repository anymore. If you decided to get rid of some branches locally, also remove them on the remote. You can do this via the command line or github’s interface (or gitlab or whatever). The command line for removing a remote tag is:

git push alicia :tagname

Also backup your releases, i.e. save the release messages and any downloads you added somewhere (I don’t have an automatic way for that).

Then force push, including the updated tags:

git push --force --tags alicia master

Repeat the last step for every branch and look at your results! Do a fresh clone of the remote somewhere to verify that everything is right. Github should have rewired your releases, to the updated tags, but if something went wrong, you can fix it through the web interface.

That was easy, right? :-)

If you have any ideas how to simplify this, please feel free to comment (FSFE account required) or reply on Twitter.

Why Privacy is more than Crypto

Tue, 31 May 2016 15:57:02 +0000

During the last year hell seems to have frozen over: our corporate ~~overlords~~ neighbours at Apple, Google and Facebook have all pushed for crypto in one way or another. For Facebook (WhatsApp) and Google (Allo) the messenger crypto has even been implemented by none less than the famous, endorsed-by-Edward-Snowden anarchist and hacker Moxie Marlinspike! So all is well on the privacy front! … but is it really?

EDIT: A French version of this post is available here. I can’t verify its correctness, but I trust the translators to have done their best and thank them for the effort!

Contents

Encryption
Software Freedom and Device Integrity
(De-)centrality, vendor control and metadata
Summary (go here if you are lazy)

I have argued some points on mobile messaging security before already and I have also spoken about this in a podcast (in German), but I fealt I needed to write about it again since there is a lot of confusion about what privacy and security actually mean (in general, but especially in the context of messaging) and recent developments give a false sense of security in my opinion.

I will discuss WhatsApp and the Facebook Messenger (both owned by Facebook), Skype (owned by Microsoft), Telegram (?), Signal (Open Whisper Systems), Threema (owned by Threema GmbH), Allo (owned by Google) and some XMPP clients, as well as briefly touching on ToX and Briar. I will not discuss “features” even if privacy related like “message has been read”-notifiers which are obviously bad. I will also not discuss anonymity which is a related subject, but from my point of view less important if dealing with “SMS-replacement apps” as you actually do know your peers anyway.

Encryption

When most people speak of privacy and security of their communication in regard to messaging it is usually about encryption or more precisely the encryption of data in motion, or the protection of your chat message while it is traveling to your peer.

There are basically three ways of doing this:

no encryption: everyone on your local WiFi or some random sysadmin on the internet backbone can read it
**transport encryption: **connections to and from service provider e.g. WhatsApp server and in between service providers are safe, but service provider can read the message
end-to-end encryption: message is only readable by your peer, but time of communication and participants still known to service provider

Also there is something called perfect forward secrecy which (counter-intuitively) means that past communications cannot be decrypted even if your long-term key is revealed/stolen.

Back in the day most apps including WhatsApp where actually of category 1, but today I expect almost all apps to be at least of category 2. This reduces the chance of large-scale unnoticed eaves-dropping (that is still possible with e-mail for example), but it is obviously not enough, as service providers ~~could be~~ are evil or can be forced to cooperate with potentially evil governments or spy agencies lacking democratic control.

Therefore you really want your messenger to do end-to-end and right now the following all do (sorted by estimated size): WhatsApp, Signal, Threema, XMMP-Clients with GPG/OTR/Omemo (ChatSecure, Conversations, Kontalk).

Messengers that have a special operating mode (“secret chat” or “incognito mode”) providing (3) are Telegram and Google Allo. It is very unfortunate that it is not turned on by default so I wouldn’t recommend them. If you are forced to use one of these, always make sure to select the private mode. It should be noted that Telegram’s end-to-end encryption is viewed as less robust by experts although most experts agree that actual text-recovery attacks are not feasible.

Other popular programs like the Facebook messenger or Skype do not provide end-to-end encryption and should definitely be avoided. It is actually proven that Skype parses your messages so I won’t discuss these two any further.

Software Freedom and device integrity

Ok, so now the data is safe while traveling from you to your friend, but what about before and after it is sent? Can’t you also try to eavesdrop on the phone of the sender or the recipient before it is send / after it is received? Yes, you can and in Germany the government has already actively used “Quellen-Telekommunikationsüberwachung” (communication surveillance at the source) precisely so they can circumvent encryption.

Let us revisit the distinction of (2.) and (3.) above. The main difference between transport encryption and end-to-end is that you don’t have to trust the service provider anymore… WRONG: In almost all cases the entity running the server is the same as the entity providing you with the program so of course you must trust the program to actually do what it claims it does. Or more precisely, there must be social and technical means that provide you with sufficient certainty that the program is trustworthy. Otherwise there is little gained from end-to-end encryption.

Software Freedom

This is where Software Freedom comes into the picture. If the source code is public there are going to be lot’s of hackers and volunteers that check whether the program actually encrypts the content. While even this public scrutiny cannot give you 100% security it is widely recognized as the best process to ensure that a program is generally secure and security problems become known (and then also fixed). Software Freedom also enables unofficial or competing implementations of the Messenger app that are still compatible; so if there are certain things that you don’t like or mistrust about the official app you can chose another one and still chat with your friends.

Some companies like Threema that don’t provide you their source of course claim that it is not required for trust. They say that they had their source code audited by some other company (which they usually paid to do this), but if you don’t trust the original company, why would you trust someone contracted by them? More importantly, how do you know the version checked by the third party actually is the same as the version installed on your phone? [you get updates quite often or not?]

This is also true for governmental or public entities that do these kind of audits. Depending on your threat model or your assumptions about society you might be inclined to trust public institutions more than private institutions (or the other way around), but if you look at e.g. Germany, with the TÜV there is actually one organisation that checks both the trust-worthiness of messenger apps and whether cars produce the correct amount of pollution. And we all know how well that went!

Trust

So when deciding on trusting a party, you need to consider:

benevolence: the party doesn’t want to compromise your privacy and/or is itself affected
competence: the party is technically capable of protecting your privacy and identifying/fixing issues
integrity: the party cannot be bought, bribed or infiltrated by secret services or other malicious third parties

After the Snowden revelations it should be very obvious that the public is the only party that can collectively fulfill these requirements so the public availability of the source code is absolutely crucial. This rules out WhatsApp, Google Allo and Threema.

“Wait a minute… but are there no other ways to check that the data in motion is actually encrypted?” Ah, of course, there are, as Threema will point out, or other people for WhatsApp. But the important part is that the service provider controls the app on your device, so they can listen in before encryption/after decryption or just “steal” your decryption keys. “I don’t believe X would do such a thing” Please keep in mind that even if you trust Facebook or Google (which you shouldn’t), can you trust them to not comply with court orders? If yes, why did you want end-to-end encryption in the first place? _“Wouldn’t someone notice?” _Hard to say; if they always did this, you might be able to recognize it from analyzing the app. But maybe they do this:

if (suspectList.contains(userID))
    sendSecretKeyToServer();

So not everyone is affected and the behaviour is never exhibited in “lab conditions”. Or the generation of your key is manipulated so that it is less random, follows a pattern that is more easily cracked. There are multiple angles to this, most of whom could easily be deployed in a later update or hidden within other features. Note also that being “on the list” is quite easy, current NSA regulations make sure that more than 25,000 people can be added for each “seed” suspect.

In light of this it is very bad that Open Whisper Systems and Moxie Marlinspike (the aforementioned famous author of Signal) publicly praise Facebook and Google, thereby increasing trust in their apps [although it is not bad per se that they helped add the crypto of course]. I am fairly sure they cannot rule out any of the above, because they have not seen the full source code to the apps, nor do they know what future updates will contain – nor would we want to have to rely on them for that matter!

The Signal Messenger

“Ok, I got it. I will use Free and Open Source Software. Like the original Signal.” Now it becomes tricky. While the source code of the Signal client software is free/open/libre it requires other non-free/closed/proprietary components to run. These pieces are not essential to the functionality, but they (a) leak some meta data to Google (more on metadata later) and (b) compromise the integrity of your device.

The last part means that even if a small part of your application is not trustworthy, then the rest isn’t either. This is even more severe for components running with system privileges as they can basically do anything at all with your phone. And it is “especially impossible” to trust non-free components that regularly send/receive data to/from other computers like these google services. Now it is true that these components are already included in most of the Android phones used in the world and it is also true that there are very few devices that actually run entirely free of non-free components, so from my point of view it is not problematic per se to make use of them when available. But to mandate their use means to exclude people who require a higher level of security (even if available!); who use alternative more secure versions of Android like CopperheadOS; or who just happen to have a phone without these Google Services (especially common in developing countries). Ultimately Signal creates a “network effect” that discourages improving the overall trustworthiness of the mobile device, because it punishes users who do so. This undermines many of the promises its authors gives.

To make matters worse: OpenWhisperSystems not only don’t support fully free systems, but have threatened to take legal and technical action to prevent independent developers from offering a modified version of the Signal client app which would work without the non-free Google components and could still interact with other Signal users ([1] [2] [3]). Because of this independent projects like LibreSignal are now stalled. Very much in contrast to their Free Software license, they oppose any clients to the Signal network not distributed by them. In this regard the Signal app is less usable and less trustworthy than e.g. Telegram which encourages independent clients to their servers and has fully free versions.

Just so that there is no wrong impression here: I don’t believe in some kind of conspiracy between Google and Moxie Marlinspike, and I thank him for making their positions clear in a friendly manner (at least in the last statements), but I do think that the aggressive protection of their brand and their insistence on controlling all client software to their network are damaging the overall struggle for trustworthy communication.

(De-)centrality, vendor control and metadata

An important part of a communication network is its topology, i.e. they way the network is structured. As can be seen in the picture above there are different approaches that are (more or less) widely used. So while the last section dealt with what is happening on your phone, this one will discuss what is happening on the servers and which role they play. It is important to note that even in centralized networks some communication might still be peer-to-peer (not going through the center), but the distinction is that they require central servers to operate.

Centralized networks

Centralized networks are the most common, i.e. all of the aforementioned apps (WhatsApp, Telegram, Signal, Threema, Allo) are based on centralized networks. While a lot internet services used to be decentral, like E-Mail or the World Wide Web, the last years have seen many centralized services appear. One could for example say that Facebook is a centralized service built on the originally decentralized WWW structure.

Usually centralized networks are part of a bigger brand or product that is marketed together as one solution (in our case to the issue of texting/SMS). For companies selling/offering these solutions it has the advantage that it has full control over the entire system and can change it rather quickly, pushing new functionality to/on all users.

Even if we assume that the service has end-to-end encryption and even if there is a client app that is Free Software, the following problems remain:

metadata: your messages’ content is encrypted, but the who-when-where information is still readable by the service provider
denial of service: you may be blocked from using the service by either the service provider or by your government

There is also the more general problem that a privately run centralized service can decide which features to add independently of whether its users actually consider them features or maybe “anti-features”, e.g. telling other users whether you are “online” or not. Some of these could be removed from the app on your phone if it actually is Free Software, but some are tied to centralized structure. I might write more on this in a separate article some time.

Metadata

As explained above, metadata is all data, that is not the content of your message. You might think that this data is unimportant data, but recent studies show that the opposite is true. Metadata includes: when you are “online” / if your phone has internet; the time of your messages and who you are texting with; a rough estimate on the length of the messages; your IP-address which can reveal rather accurately where you currently are (at work, or at home, out of town et cetera); possibly also security related information about your device (which operating system, which phone model…). This information has a lot of privacy threatening value and the US secret services actually use it to justify targeted killings (see above)!!

The amount of metadata a centralized service sees depends on the exact implementation, e.g. the “group chat” feature in Signal and supposedly also Threema is client-based so in theory the server knows nothing about the groups. On the other hand the server has timestamps from your communication and can likely correlate these. Again it is important to note that while your service provider may not log this information by default (some information must be retained, some could be deleted immediately), it might be forced to log more data by secret agencies. Signal (as mentioned before) only works in conjunction with some non-free components from Google or Apple who then always get some of your metadata, including your IP-address (and thus physical position) and the time you receive messages.

More information on metadata here and here.

Denial of service

Another major drawback of centralized services, is that they can decide not to serve you at all if they don’t want to or are obliged not to by law. Since many of the services require your phone number to register and they operate from the US, they might deny you service if you are a Cuban for example. This is especially important since we are dealing with encryption that is highly regulated in the US.

As part of Anti-Terrorism measures Germany has just introduced a new law that requires registering your ID when getting a SIM card, even prepaid. While I don’t think that it is likely, it does open the possibility for black-listing people and pressuring companies to exclude them from service.

Instead of working with the companies, a malicious government can of course also target the service directly. Operating from a few central servers makes the infrastructure much more vulnerable to being blocked nationwide. This has been reported for Signal and Telegram in China.

Disconnected Networks

When the server source code of a service provider is Free and Open Source software, you can setup your own service if you distrust the provider. This seems like a big advantage and is argued by Moxie Marlinspike as such:

Where as before you could switch hosts, or even decide to run your own server, now users are simply switching entire networks. [...] If a centralized provider with an open source infrastructure ever makes horrible changes, those that disagree have the software they need to run their own alternative instead.

And of course this is better than not having the possibility to roll your own, but the inherent value of a “social” network comes from the people who use it and it is not easy to switch if you loose the connection to your friends. This is why alternatives to Facebook have such a hard time. Even if they were better in every aspect, they just don’t have your friends.

Yes, it is easier for mobile apps that identify people via phone number, because it means you at least quickly find your friends on a new network, but for every non-technical person it is really confusing to keep 5 different apps around just so that they can keep in touch with most of their friends so switching networks should always be the ultima ratio.

Note that while OpenWhisperSystems claim that they are of this category, in reality they only publish parts of the Signal server source code so you are not able setup a server that has the same functionality (more precisely the telephoning part is missing).

Federation

Federation is a concept which solves the aforementioned problem by having the service providers speak with each other, as well. So you can change the provider and possibly the app you are using, but you will still be able to communicate with people registered on the old server. E-Mail is a typical example of a federated system: it doesn’t matter whether you are tom@gmail.com or jane@yahoo.com or even linda@server-in-my-basement.com, all people are able to reach all other people. Imagine how ridiculous it would be, if you could only reach people on your own provider!?

The drawback from a developer’s and/or company’s perspective is that you have to publicly define the communication protocols and that because the standardization process can be complicated and lengthy you are less flexible in changing the whole system. I do concur that it makes it more difficult for good features to quickly be available for most people, but as mentioned previously, I think that from a privacy and security point of view it is clearly a feature, because it involves more people and weakens the possibility of the provider pushing unwanted features on the users; and most importantly because there is no more lock-in-effect. As a bonus these kind of networks quickly produce different software implementations, both for the software running at the end-user and for the software running on the servers. This makes the system more robust against attacks and ensures that weaknesses/bugs in one piece of software don’t effect the entire system.

And, of course, as previously mentioned, the metadata is spread between different providers (which makes it harder to track all users at once) and you get to choose which of them gets yours or whether you want to operate your own. Plus, it becomes very difficult to block all providers, and you could switch in case one discriminates against you (see “Denial of Service” above).

As a sidenote: It should be mentioned that federation does imply that some metadata will be seen by, both, your service provider and your peer’s service provider. In the case of e-mail this is quite a lot, but this is not required by federation per se, i.e. a well designed federated system could avoid sharing almost all metadata in-between two service providers – other than the fact that there is a user account with a certain ID on that server.

So, is there such a system for instant messaging / texting? Yes, there is, it is called XMPP. While originally not containing strong encryption, there is now encryption that provides the same level as security as the Signal Protocol. There are also great mobile apps for Android (“Conversations”) and iOS (“ChatSecure”) and every other platform in the world, as well.

The drawback? Like e-mail, you need to setup an account somewhere and there is no automatic association with telephone numbers so you need not only convince your friends to use this fancy new program, but also manually find out which provider and username they have chosen. The independence of the phone number system might be seen as a feature by some, but as a replacement for SMS this seems unfit.

The solution: Kontalk, a messenger based on XMPP that still does automatic contact discovery via phone numbers from your address book. Unfortunately it is not yet as mature as other mentioned applications, i.e. it currently still lacks group chats and there is no support for iOS. But Kontalk does prove that it is viable to have the same features built on XMPP that you have come to expect from applications like WhatsApp or Telegram. So from my point of view it is only a matter of time until these federated solutions are on feature parity and similar in usability. Some agree with this point of view, some don’t.

Peer-to-Peer networks

Peer-to-peer networks completely eliminate the server and thereby all metadata at centralized locations. This kind of network is unbeatable from a privacy and freedom perspective and it is also almost impossible to block by an authority. An example of a peer-to-peer application is ToX, another one is Ricochet (edit: not for mobile) and there is still under development Briar which also adds anonymity so even your peer doesn’t know your IP address. Unfortunately there are principle issues on mobile devices making it hard to maintain the many connections required for these networks. Additionally it seems impossible right now to do a phone-number to user mapping so there can be no automatic contact discovery.

While I don’t currently see the possibility of these kind of apps stealing market share from WhatsApp, there are use cases, especially when you are being actively targeted by surveillance and/or you have group of people who collectively decide to move to such an app for their communication, e.g. political organisations.

Summary

Privacy is getting more and more attention and people are actively looking to protect themselves better.
It can be considered positive that major software providers feel they have to react to this, adding encryption to their software; and who knows, maybe it does make life for the NSA a little bit harder.
However there is no reason that we should trust them anymore than we have previously, as there is no way for us to know what their apps actually do, and there remain many ways they can spy on us.
If you are currently using WhatsApp, Skype, Threema or Allo and expect a similar experience you might consider switching to Telegram or Signal. They are better than the previously mentioned (in different ways), but they are **far from perfect, **as I have shown. We need federation in the medium to long-term.
Even if they seem to be nice people and very skilled hackers, we cannot trust OpenWhisperSystems to deliver us from surveillance as they are blind to certain issues and not very open to cooperation with the community.
Some cool things are cooking in XMPP-land, keep an eye out for Conversations, ChatSecure and Kontalk. If you can, support them with coding skills, donations or friendly e-mails.
If you want a zero-metadata approach and/or anonymity, try out ToX or wait for Briar.

OpenBSD on the Thinkpad X250

Fri, 15 Jan 2016 16:17:30 +0000

Since posts like these often helped me setup some device, I thought I’d write one as well this time. Also it is advertising for a good free operating system.

TL;DR

Everything works almost perfectly, good hardware, good software! You should give it a try :)

Hardware

I have got a fairly high-end config:

Intel® Core™ i7-5600U
16GB RAM
1920x1080 Display, non-glare, non-touch
512GB Samsung SSD
Intel LAN + WLAN
Pro-Dock

Installing OpenBSD

You need a current OpenBSD snapshot (or 5.9 once it is released).

get the install58.fs from a server near you, dd it to a USB-Stick and boot from it
follow the instructions on the screen to install 5.8
get the bsd.rd from a current snapshot and place it in / (maybe backup the existing one)
reboot into it (enter boot bsd.rd at the bootloader prompt)
follow the intructions on the screen to update to the 5.9-snapshot
after you have rebooted into 5.9, run sysmerge

Et voila, you are on a current branch of OpenBSD. You should have seen the virtual console resize after the update to the new version.

BIOS

The BIOS has some useful settings, you might want to make, e.g. switching Fn and lCtrl buttons and switching the meaning of the F*-keys to be F*-keys by default and not “media keys”. Unfortunately deactivating the keypad from the BIOS does not work ( it can however be deactivated by software, see below).

Xorg and periphals

No xorg.conf nor any configuration like that is needed to bring up X since the new Intel driver takes care of everything. 3D acceleration works, however the direct rendering devices come with write access only given to root. It’s probably a security “feature”, but it prevents 3D for regular users so I created /etc/rc.local (this is run on startup) with:

#!/bin/sh
/bin/chmod g+rw /dev/drm*

You should have setup the keyboard to your local layout during install, but if you haven’t done so or want to deactivate dead keys you can edit /etc/kbdtype, e.g. to de.nodead. This affects both the console and Xorg. Inside Xorg the “media buttons” for display brightness and sound work automatically.

To deactivate the mouse touchpad and make scrolling working with the pointer in combination with the middle mouse button, I have added the following to my .xinitrc/.xsession:

# deactivate touchpad
synclient TouchpadOff=1

# activate scroll wheel button
xinput set-prop "/dev/wsmouse" "WS Pointer Wheel Emulation" 1
xinput set-prop "/dev/wsmouse" "WS Pointer Wheel Emulation Axes" 6 7 4 5
xinput set-prop "/dev/wsmouse" "WS Pointer Wheel Emulation Button" 2
xinput set-prop "/dev/wsmouse" "WS Pointer Wheel Emulation Timeout" 50
xinput set-prop "/dev/wsmouse" "WS Pointer Wheel Emulation Inertia" 3

# increase pointer speed
xinput set-prop "/dev/wsmouse" "Device Accel Constant Deceleration" 0.4

I have to note though that the Pointer Wheel Emulation does not work satisfactory, there is often some jitter and then scrolling stops working. Also the increased sensitivity produces very slight ghost movements of the mouse (very small, but noticable movements of the mouse in the bottom left direction). I still need to debug this, maybe its also a hardware issue.

UPDATE: I was experiencing more problems with the keyboard, including the space bar not working reliably, so I had the hardware checked resulting in a replacement for the keyboard. After this, all problems related to the mouse disappeared, as well, so I am pretty sure it was actually a hardware issue. But I also updated to a more recent snapshot so maybe the mouse problems were software related. In any case they are gone now.

The dock seems to work (charging and on/off), although I have not yet tested hooking up external input devices or screens.

Networking

Both lan and wlan devices come with working drivers and the firmware for the wifi card is automatically downloaded upon install. It should be noted that OpenBSD does not have a service for managing networks and that it implements WPA[2] PSK inside its regular networking infrastructure (it doesn’t need wpa_supplicant). For enterprise WPA (like eduroam) however, wpa_supplicant is needed.

I have fiddled around with some solutions and ended up adapting a script. The original is wifinwid, but I changed it so that it restarts wpa_supplicant before trying every network and also only starts dhclient after it has attached to a network. This solution requires a file at /usr/local/etc/nwids.iwm0 with your saved networks in preferred order (every line is the arguments line for the ifconfig call), e.g.:

nwid MYSSID -wpa wpa wpaakms psk wpakey SECRETYO up
nwid eduroam -wpa wpa wpaakms 802.1x up

For the “enterprise” WPA networks like eduroam you have to additionally edit /etc/wpa_supplicant.conf (but not for other WPA networks!). The exact values are the same as for other operating systems, see e.g. this site. You do however need to add a wpa_supplicant_flags=-c /etc/wpa_supplicant.conf -D openbsd -i iwm0 line /etc/rc.conf.local.

Ultimately you have make the script be started on boottime and also initially make it not connect to some random unencrypted network via /etc/hostname.iwm0:

nwid NONEXISTANT
rtsol
!/usr/local/sbin/wifinwid \$if &

Now you have the behaviour that it only ever connects to known networks and that you can roam between different known networks, encrypted or unencrypted, PSK or with personal authentication.

Powermanagement

You need to activate the APM-daemon in your rc.conf.local:

apmd_flags="-A"
apmd_enable="YES"

This enables speed-stepping and also suspend and resume. I have tested the zzz command (suspend to RAM), but not yet ZZZ (suspend to disk). There seem to be no problems at all with X and the wifi also automatically reconnects if setup like above.

Brightness control works through keyboard shortcuts and the battery time seems to be good, although I haven’t done long-term tests, yet. With the large battery official number is “up to 20 hours”, if I get 12 hours that would be enough for me. A nice thing is that the notebook has an internal battery so you can change the external one without rebooting. The estimation of the remaining time via apm or sysctl seems to be accurate and my windows manager i3’s infobar supports these sensors automatically.

Summary

Altogether I am very happy about the hardware support, with the only annoyance being the mouse (which is not crucial, because I work more via keyboard anyway). Thanks to all the involved developers! I will probably update this post in the near future, once I have more things tested!

Secure Texting Part II

Mon, 23 Feb 2015 21:33:47 +0000

Last summer I blogged about secure messaging and why FSFE cares about it (and why you should, too!). Since then a few things have changed, and I want to give you an update on the situation.

The conclusion of my last article was:

TextSecure and Kontalk are both good apps in our eyes, however, TextSecure has a much larger adoption and its protocol has gone through more reviews. The protool is integrated into CyanogenMod, recommended by leading security experts and the project just recently gained lots of media attention and $400.000 funding. So we believe if we are to have a chance at migrating people away from WhatsApp than TextSecure is the way to go.

We knew that TextSecure depended on Google Play Services last year, but we were hoping that this was a temporary problem, as virtually every other messaging app in existence has a fallback mode for delivery that does not require proprietary (Google) components. Unfortunately we were wrong: nearly a year later the development of a websocket based version of TextSecure has stalled. Lead developers at WhisperSystems have stated repeatedly that it is not important to them, and the many requests, tests and code contributions from external people did not result in the situation now being any better than it was a year ago.

Furthermore WhisperSystems has repeatedly demanded other people not distribute modified and unmodified versions of their software. While I believe that WhisperSystems is sincere about security, they seem to have no problem with the security implications of proprietary software, sharing meta-data with Google (by means of Google Push) and now working for WhatsApp / Facebook. This is all a sad example for a project that does license its code under Free licenses, but that otherwise is between uninterested and hostile towards community involvement and the Free Software landscape.

Fortunately, not all is lost! The other program mentioned already a year ago, Kontalk, is doing great. Kontalk is community-based and is transparently financed through donations. It is based on XMPP, actively develops new extensions and proposals for XMPP and their developers are very friendly towards suggestions and community involvement. The server side is even implemented as extensions on top of an existing XMPP server and you can of course run your own (the server isn’t even hardcoded in the app, can be changed via the options). It runs without any proprietary components and is available in F-Droid. There is also a desktop client, although I haven’t tried it, yet.

Some of Kontalk’s features are:

• contact discovery via phone numbers

• transport and end-to-end encryption

• working picture and file sharing

• customizable privacy settings (per-user in future versions)

It is currently still in beta, but some of the expected features for the 3.0 are: • group chats

• perfect forward secrecy

• sharing of message history between multiple clients

• federation with regular jabber servers(!!!)

I use it day to day and have experienced only few issues. You should it give it a try! And maybe you can help with spreading the word, reporting bugs or even contributing code?

edit: see I am not big for Valentine’s day, but maybe this counts a slightly delayed #ilovefs for Kontalk ;)

Encrypting cron's daily mail

Tue, 04 Nov 2014 20:51:28 +0000

Most you have probably set your system aliases to receive root’s e-mail, and that of course is a very good idea so you are kept up to date. On the other hand you do send a lot of information about your system through the wire, often package diagnostics with information about locally installed vulnerable software and many other things that might help a person or entity gain access to your computer. Now living in a world, where we know that all unencrypted mail is automatically parsed and possibly filtered and stored that is something you might want to avoid. The natural answer is to encrypt the mail which is what I am going to explain in the following.

This HowTo pertains to FreeBSD in particular, but I am sure all you GNUsers out there will figure out the necessary changes for their system. A note of warning: I will not be redirecting all of root’s mail, I will just be sending out the log files. If you want a more complete solution, you might want to check out this. For my situation it definitely was overkill, as it involves installing and configuring a new MTA, new user accounts, some other unmaintained software…

First of all, get GnuPG from your ports or packages (security/gnupg)

pkg install security/gnupg

and add the public key of the receiver to your keyring:

gpg --import your_pkey.asc

I am doing this as root, but you can also setup an extra user for it or even use your regular account. As long as they are in the wheel group, the permissions should be fine. Double-check that the key was added correctly by printing the list of keys!

Next, tell your periodic script not to mail the log files to root, but instead save them to disk. Do so by appending the following lines to (or creating) /etc/periodic.conf :

daily_output="/var/log/daily.log"
daily_status_security_inline="YES"
weekly_output="/var/log/weekly.log"
weekly_status_security_inline="YES"
monthly_output="/var/log/monthly.log"
monthly_status_security_inline="YES"

Now paste the following text into /root/bin/gpgcron.sh :

#!/bin/sh

# verify argc
[ $# -ne 1 ] && exit 1

# verify argv
[ $1 != "daily" ] &&  [ $1 != "weekly" ] && [ $1 != "monthly" ] \
 && exit 1

LOG="/var/log/${1}.log"
SENDER="something@valid.com"                # could be == $RECEIVER
RECEIVER="your@email.address"               # duh.
RECEIVER_KEY_ADDR="public.keys@address.com" # usually == $RECEIVER
SUBJECT="${1} run on $(hostname -s)"        # could be something else

cat "$LOG" | /usr/local/bin/gpg -e -a -r "$RECEIVER_KEY_ADDR" \
 --trust-model always --batch | mail -s "$SUBJECT" "$RECEIVER" \
-f "$SENDER"

and don’t forget to make the file executable.

Verify that the script works by placing some random text in /var/log/daily.log (iff it doesn’t exist) and running

/root/bin/gpgcron.sh daily

You should receive an encrypted mail now that your MUA will decrypt for you. If this works, the last step is adding the script to your /etc/crontab. I always have them run half an hour after the original script to make sure that it completed (although 5min might be enough):

# Perform daily/weekly/monthly maintenance.
1      3      *      *      *     root   periodic daily
30     3      *      *      *     root   /root/bin/gpgcron.sh daily
15     4      *      *      6     root   periodic weekly
45     4      *      *      6     root   /root/bin/gpgcron.sh weekly
30     5      1      *      *     root   periodic monthly
1      6      1      *      *     root   /root/bin/gpgcron.sh monthly

(only the lines with gpgcron were added!)

Your next daily mail should come to you encrypted. Happy hacking!

edit: give the full path to GPG if your crontab overwrites $PATH

Photo of the Month — 2014-09

Sat, 13 Sep 2014 15:07:33 +0000

With a minor delay, here is the photo of the month for September. It was taken this February in Zermatt, Switzerland.

It is a Panorama consisting of eight individual Portrait-orientated shots, stitched together with Hugin. Unfortunately (again) the size limitations of this blog prevent me from giving you a better resolution / quality. The original is 8512x4634 and roughly 200MB big.

Developing and editing was tricky on this one, and used the opportunity to learn about regional masks in Darktable. Before all editing, the stitch looked like this:

After the usual corrections (basecurve, lighting, local contrast), the picture looked like this:

As you can see, the snow on the Matterhorn itself was a little darker/greyer then the other snow which results in poor contrast with the surroundings, likely a result of different contrast in the original individual pictures. By creating a manually drawn mask around the Matterhorn and applying another brightness increase of about 0,10 EV just to the region the final photo is attained. I love darktable, have I mentioned that?

Photo of the Month — 2014-08

Fri, 01 Aug 2014 04:00:32 +0000

Originally I had wanted to post a different picture, but sadly the war against the Palestinian people has seen yet another level of escalation, so here’s a picture in solidarity with the Palestinians and all other people suffering from war and oppression. I shot it in Ramallah, Westbank, 2011.

Many things have been said about the situation and I don’t want to engage in political debate on this blog, so I will not say more than the solidarity expressed above. If you haven’t yet thought about the whole conflict, here’s a good video by the Jewish Voice for Peace. If you agree, support groups and actions in your area.

Concerning the technical background of the picture, this is the original shot taken, walking down a street, without stopping to get the angle right and better exposure:

It was edited with Darktable using base curve and color curve corrections (weak sigmoidal curves for both), local contrast enhancement, lense correction, cropping and angle correction. The latter is not to my full satisfaction, but the best I could get from the picture, I think.

[advertising] A nice Indie-Game

Tue, 01 Jul 2014 18:42:07 +0000

I have just backed a wonderful looking independent dystopian RPG on Kickstarter! It’s called Insomnia, it looks like it uses the Free Software Engine Ogre3d (although its not Free Software itself) and it will feature native Linux support. They still need money, so if you enjoy special video games, go help them out and pre-order your DRM-free copy!

Photo of the Month -- 2014-07

Mon, 30 Jun 2014 23:04:13 +0000

I decided to start a photo-of-the-month series; hopefully I will be able to keep it up. Unfortunately the upload limitation on this platform is 1MB per picture so you won’t get high-res for now. The picture was taken last fall in Oberengadin, Switzerland, close to Salecina House. It was a breath-taking sunset and I was very tired from a day of hiking, but thankful for having been there right at that moment!

These three bracketed shots were originally taken (+0.66EV, -1.33EV, +2.66EV):

They were then joined into an HDR with Luminance HDR and afterwards two tonemappings were created, one with the Fattal algorithm and one with Mantiuk06:

I have presets for the algorithms that I usually use, but I am not 100% sure if they were used in this case. For Fattal I usually use default settings, for Mantiuk06 I usually increase Saturation factor a little (to ~1.0) and sometime decrease contrast factor. These two images where then blended 50% each with ImageMagick into this shot:

This was then finalized with Darktable, only tuning the base curve (influences brightness and contrast) to the shot you see in the beginning. If it seems a little dark to you, that’s something I also realized when looking at my pictures on other screens. My color correctness is not very good and I usually have brightness cranked up to 100% because it gives a better picture… I hope to get some nice 4k screen with better color correctness once they are cheaper.

All my photos, as usual, dual-licensed under CC-by-nc-sa and GPLv3.

Secure Texting and why FSFE cares

Wed, 25 Jun 2014 14:08:55 +0000

Heard of WhatsApp? If you haven’t used it before (I e.g. haven’t), you can think of it as a free-of-charge messaging app that knows which of your contacts also have the app and automatically routes messages to them over your dataplan instead of SMS, so it’s (usually) free of charge. In the face of NSA and WhatsApp’s recent acquisition by Facebook, many of the million WhatsApp users are looking for secure and trustworthy alternatives.

Because this effects so many people, we at the Free Software Foundation Europe would like to promote an alternative that respects your freedom and privacy. Therefore we decided to do some research and to hold a workshop on WhatsApp alternatives during our German-speaking FSFE team meeting ten days ago.

While most tech-people including myself really didn’t see the point in WhatsApp – after all there is XMPP – I now understand better why it’s so popular. Easy integration with the operating system and automatic contact discovery seem to be crucial features for the masses. And I do have to admit that both, encryption (OTR) and file transfers, are not yet solved reliably and/or conveniently with popular XMPP clients.

So we ruled out promoting Xabber or ChatSecure as a good alternatives to WhatsApp and turned our focus to other currently popular apps:

· Kontalk

· Telegram

· TextSecure

· Threema

· Surespot

Of these, Threema seems to be quite popular and is recommended by many people although it is completely proprietary. And Telegram is proprietary on the server side, so we immediately ruled those two out. After all, the interception of communication (man-in-the-middle) is not the only surveillance scenario; the client software needs to be trustworthy as well. And depending on a single Server definitely is also not the path to freedom.

Surespot is Free Software, but doesn’t have automatic contact discovery through phone numbers, so we think it will probably not be competitive in the market segment that is currently dominated by WhatsApp.

TextSecure and Kontalk are both good apps in our eyes, however, TextSecure has a much larger adoption and its protocol has gone through more reviews. The protool is integrated into CyanogenMod, recommended by leading security experts and the project just recently gained lots of media attention and $400.000 funding. So we believe if we are to have a chance at migrating people away from WhatsApp than TextSecure is the way to go.

Unfortunately, TextSecure relies on Google Cloud Messaging for pushing messages to the user. This comes in form of a dependency on a proprietary Google library, part of the “Google Play Services”. This is unacceptable for many reasons, this page sums up some of them.

As a consequence we have starting testing and supporting a version of TextSecure that provides an alternative mechanism for message distribution. We are confident that the issue will be solved sooner or later (as the announced desktop and iOS clients will have to work with something else anyway), but if you have free time, please help the effort.

Once this is achieved, TextSecure will be a great alternative to WhatsApp, easy to use, free as in freedom and respecting your privacy.

FreeBSD Port of luminance-hdr updated

Sun, 20 Apr 2014 00:24:04 +0000

I finally managed to update graphics/luminance to 2.3.1. Thanks to lme@freebsd and pawel@freebsd for feedback and commit.

This is last version that runs with Qt4. As we recently got Qt5 into the ports I will try to update the port soon again. I also have some new ports coming and hope to get around to fixing/updating my other ones. Stay tuned.

I am making strong use of RedPorts now. In case you don’t know the service, I really recommend it.

A Good way to publish Photos copyleft

Mon, 17 Jun 2013 21:29:36 +0000

I am looking for a good way to publish images as copyleft. The current state of the art in licensing non-software seems to be creative commons, but its copyleft is so weak, that I don’t see how it protects the end users’ freedom. To be more precise, here is how I perceive the legal effects (please correct me if I am wrong):

· I publish a digital picture in high resolution, with intermediate files under CC-by-sa · Magazine XY takes the picture, touches it up and prints it as part of some article in a small size · user Z sees the picture in the article and doesn’t even know the associated rights, unless she looks in the table of figures; and even if she does she can only demand the ready-to-print down-scaled version of the photo, possibly with text covering it, in a non editable format

→ what’s the point? Where is the freedom? What do I gain as an author, if I let commercial magazines use my photos, that in turn, don’t share with my fellow citizens?

Why do we not treat pictures like software? Obviously there is a preferred way of making changes, which might be a format with layer and effects information, or even a non-destrucitve instruction set, like with darktable. This is != to the preferred form of distribution, which is some form of bitmap, so it makes sense to speak of source and non source in my opinion.

I want to release my photos in a way that forces people to make the source of my photos as well as the source of their changes available to all third parties. So I thought about just stuffing the original image, with intermediate formats and/or instructions in a tarball, add a GPLv3 file and a note, on how I define that the GPL applies. Does that make sense? Or have you come over other, easier ways to achieve a similar effect?

A KISS GNU/Linux distro?

Sun, 02 Jun 2013 21:31:13 +0000

As most of you know, I use FreeBSD as my main Desktop-OS. But I also keep a ArchLinux around that I use (almost exclusively) for gaming. After being disappointed with the way it is headed since a while already, I am now completely fed up with it and am desperately seeking an alternative.

[rant]ArchLinux claims to keep it simple, but over the last year or two it has accumulated an increasing about of bloatware, especially all the stuff that Lennart Poettering is spewing out. Yes, I am conservative in a lot of techie ways, I am a little biased, as I heard lots of bad things about the software before actually using it, but I did give all the stuff a fair try and it just failed epicly. For PulseAudio I was at least able to replace it with OSSv4, which had proper support on ArchLinux at the time. Not really proper, as some programs stopped producing audio (e.g. Chromium), but at least all the gaming stuff worked. Then came systemd and it really is worse then everything I heard about it. Upgrading to systemd severely broke my system (I would never have switched, had I not been forced to by Arch), it took me a countless number of hours to make it boot again. But it is still really broken, and not broken, in the “I need to find the right config”-way, but in the “Let’s roll the dice”-Windows way were everything is unpredictable. First of all one in three boots fails, because systemd messes up the order of the nfs-mounts. OSS now has some weirdness as well, where it sometimes just doesn’t work and restarting the computer makes it work again. Shutting down the system actually never works, because on shutting down, it again messes up the order of nfs unmounts and never manages to stop OSS (the things actually spawns jobs to stop other processes, see the irony?!). But the biggest issues is that after a random amount of minutes systemd or logind or whatever is responsible for logins just crashes, throwing me unto a console, which asks me to enter my root password or press Ctrl-D to continue and where my keyboard doesn’t work anymore (apparently that is handled by the same stuff). Wild button-pressing sometimes retuns me back to X, which then seems to just keep on working, with the exception of an ocasional flicker. And of course – thanks to systemd – there is nothing logged anywhere in any form that I can read… Really, most sysadmins I know warned me about the unreliability of systemd, but I didn’t expect it to be that bad. And all of it just to boot one second faster. The GNU/Linux desktop has really fallen far IMHO and I seriously doubt that the whole concept of making everything more integrated and removing modularity and other core unix concepts will help the Free Software movement in the end. Where we win people with some eye-candy and a faster boot, we will alienate more because the software becomes unstable, unreliable and unpredictable. Rebooting should never be the fix to a problem! [/rant]

Ok, back to topic, do you know a GNU/Linux Distro that adheres to keep-it-simple philosophy, does not use Poettering-Ware and has a somewhat recent Xorg?

I thought about switching back to Debian, but they seem to be going the Poettering-road as well, as recent polls indicate. Draco GNU/Linux was my first idea, since they use OSSv4 as default and lots of BSD-stuff, but the Xorg available is quite old, and the projects future is a little unkown… Slackware was my next idea, but it doesn’t even include my default window manager (“awesome”). What are your thoughts on this? Any suggestions?

Autumn Leaves

Sat, 25 May 2013 14:41:06 +0000

This is my current wallpaper on my desktop. Not a winter picture, but autumn is ok, too. I took the picture in Brandenburg in fall of 2011. Had I had more time, I might have taken some better ones, because the light was really great. I would really have liked to make one without the »Hochstand« (high seat?). But actually I was quite in a hurry, because there were other people walking in the forest, and the picture was taken just when the person in front of me was concealed by a tree.

The only editing done on the picture is manual contrast/gamma correction and transformation to black-and-white with a green filter. I might post a tutorial on the digikam workflow involved with making this picture, if you are interested in that, let me know.

Free commenting disabled

Tue, 21 May 2013 16:57:08 +0000

Since I have had more and more spam coming through the filters and annoying me no end, I decided to restrict commenting to registered users (which are only FSFE fellows). I regret to do this, but the ratio of spam to content is 5,245 to 22 (do the math yourself) and I don’t want to spend what time I have for the blog with removing spam. I hope to change it in the future again, once a better anti-spam solution has been found.

Winter in Berlin

Thu, 09 May 2013 19:21:46 +0000

While spring and early summer are claiming Berlin now and most folks are out enjoying the change, I thought I’d post a late winter picture.

People who know me, know that I am more of a winter person, so I usually have pictures of winter or fall around when the sun shines too bright outside. This picture was taken about two months ago in Pankow, Berlin. It is an HDR rendering made of three individual shots using Luminance HDR and the Mantiuk06 algorithm. Color and Contrast were corrected with digikam.

I like the way the different elements in the picture contrast each other… the classical architecture and the graffiti; the coziness of the christmas-like tree and the cold wind… it somehow doesn’t want to fit together and still does.

HowTo: Dual-Boot Ubuntu [arm] and CM10.1 on the TF700t

Sat, 20 Apr 2013 18:48:32 +0000

I thought I’d cover how I got to my current dual-boot on my Transformer. The main difference to other setups is, that my Transformer’s data partition is encrypted, which makes some steps more difficult.

The Goal

After following these instructions you will have a regular dual-boot on the transformer, like on any notebook or desktop, i.e. when booting the device you get to select which OS to start. The Android experience is in no way diminished, everything works as before. The Ubuntu experience is just like you would have on a notebook, with RAM being the only limiting factor and some rough edges. Wifi and (proprietary) 3d- and video-acceleration work. Unity works, although I use awesome wm, as everywhere else. The rough edges are missing suspend support, some rendering glitches here and there, no VTs… for more info, see the XDA-thread of the port. This is also the place, in case anything described here doesn’t work.

Preperation

You should know what I am talking about in the following. If not you should probably get to know your device better, before doing this.

What you need: · Unlocked Transformer TF700t, with CyanogenMod 10.x, stock Android 4.1.x or CleanRom 4.1.x [yes, CM-Versions of Android 4.2 work, others don’t] ·· if the transformer is not yet encrypted, do this now (through Settings->Security) · one dedicated µSD-Card with at least 10GB. I would recommend at least class 10, better UHS1 to get decent performance, as Ubuntu will reside on this. You can get the latter for 22€ these days… [“SD1”] · a second µSD-Card with 1GiB free space [“SD2”] · a µSD to SD-Converter (you get these free with µSDs usually)

Get the following files and put them on SD2 · Kernel/Bootloader · Ubuntu RootFS · A current CM-Nightly without the boot-blob inside (just open the zip and remove the boot-blob)

Action

Make backups of your data. Just in case.
Put SD2 into the µSD-Slot of the transformer and SD1 into the converter and into the SD-Slot of the transformer.
Reboot the transformer into Recovery and select the “linux-install-*.zip” as Update
Follow the on-screen instructions, selecting the Dock-SD-Card (SD1) as target
After you are done, reboot into Ubuntu, see if it works.
Reboot again, into CM, see that it doesn’t work (loading and loading).
Reboot into Recovery again, install the prepared CM-update (without the boot-blob, or you will overwrite the loader!)
Reboot into CM, and see if it works (will still take longer to load on first load)
Switch the SD-Cards to have the Dock’s reader free again.
be happy

Open Ends

As mentioned, there are still some issues, the main one for me being, that I have not yet successfully decrypted the android data partition from Ubuntu, which I would like to share as home directory. But altogether I am quite happy to have a proper OS, whenever I need to do serious things :) Here’s a screenshot:

Emergency Exit

Thu, 11 Apr 2013 20:49:06 +0000

Here’s the picture that is used in the header and which inspired the title of the blog. The picture was taken with my transformer pad through the window of a bus, in Northern Ireland, last fall. I ran it through some filters of the android image program, not much was done on the computer.

I kind of like the sound of »emergency exit«, and it fits with the picture. You are stuck in a place that isn’t good for you – but things aren’t hopeless, there is a way out. Can be understood as a allegory on our society in general or the Software world or whatever the viewer might see in it.

Then again the picture has a more gloomy touch to it, the sky that is inscribed with “EMERGENCY EXIT” (or at least parts of the words) and symbolising salvation is distant and unreal, the viewer remains in darkness and the surrounding world as well. Both, the vignetting effect applied to the picture and the fact that there is glass between the viewer and the scene create a separateness that hints at the possibility that you might actually not reach the emergency exit in time.

Blog Overhaul

Thu, 28 Mar 2013 19:30:34 +0000

Dear readers,

I have changed the look and feel on my blog a little. It is more streamlined now, and it will contain pictures and notes on photography as well. I will mostly write about technical details of the the picture taking and editing process, but the pictures themself might be interesting for not-technical users as well :)

Hopefully some people who know more about digital photography will find their way to this blog and leave helpful comments, so I can improvemy skills!

I will write more on the choice of title and title picture in another post…

Tracking memory usage Revisited

Fri, 08 Mar 2013 13:36:45 +0000

I posted an article on tracking memory usage a while ago. Unfortunately it doesn’t work for a lot of cases, i.e. when programs are statically linked, or are written in a way where they don’t go through the dynamic linker, e.g. programs written in Java.

I now actually pgrep the child processes of the invoked command and sum up all the processes’ resident set memory, as found in /proc, repeatedly, preserving the max. The temporal resolution of this of course depends on the interval at which you measure, but especially on commands executing over a long time this seem to be be very close to VmHWM. I currently check every 300ms.

You can find the script here here.

Missing (Free) Android Apps - II - E-Mail

Wed, 26 Dec 2012 22:11:46 +0000

This is a pretty serious one: there are no proper E-Mail apps for Android tablets. Yes, I know there is K9, but K9 is only good for phones. Seeing that the K9 people also develop the proprietary (and for-money) Kaitan Mail, which is for tablet and high-resolution phones, I have little hope of seeing K9 being upgraded any time soon to being usable on tablets.

The only alternative I have seen so far is the stock Jelly-Bean(and after) Mail program, which looks decent on my screen and which I use right now. But it also lacks so many crucial features, that I am not happy with it at all. Just to name a few:

editable and top-placable quotations
identities (you won’t believe, I actually set up the same email account multiple times to get different sender-addresses – and I had to cheat the program to be able to do it…)
gnupg
a white-on-black theme …

My transformer also shipped with an E-Mail program that was a little better than stock Android (e.g. it had support for showing things in a threaded view), but it was proprietary. Then again it looked very much like a fork of the stock E-Mail app, maybe one should investigate if the proprietary fork was legitimate.

Anyway, the overall situation is kind of dissapointing. I mean, I really have lowered my expectations of Android, but emailing really is the most basic functionality a “mobile device” should provide…

Hacking encryption into Android

Tue, 04 Dec 2012 10:52:51 +0000

Next weekend we are going to have a small hackathon in Berlin to port/fix/implement disk encryption on various Android devices.

Android has full disk encryption since 4.0, but it only works when using regular filesystems, e.g. ext. If you have a device that doesn’t offer proper block devices, because the hardware doesn’t do wear-leveling et cetera, you will probably have YAFFS2 as a file system or something similar that does this on software side. Unfortunately this prevents standard encryption (luks, dm-crypt…) from working. But there is some outdated code floating around that is supposed to implement this, which we will try to update and integrate with current CyanogenMod. More details on the issue can be found here, here and here (in parts).

Since many phones with hardware keyboards (Xperia Pro, Xperia Mini Pro) are affected, and a lot of hackers have those, maybe you are one of those and want to help! Or maybe you just want to help, which would also really be appreciated. Actually anyone with Android, Java and/or C(++) skills is really welcome!

The hackathon facts: · 2012.12.08 11:00 ↔ 2012.12.09 18:00 · graciously hosted by IN-Berlin e.V. (OpenStreetMap)

We will have a limited amount of sponsored Club Mate. Other drinks are available for inexpensive purchase. Food will have to find its way to us by delivery. Bring your own phone, bring your own laptop. Hope to see you there!

Tracking memory usage

Thu, 29 Nov 2012 10:33:30 +0000

… or why using Linux can be a pain, when you come from the BSD-world.

On BSD (and MacOSX and probable other UNIXes), if you want to track a program’s memory usage, you can simply issue the following in a CSHELL set time= ( 0 "%D KB avg / %K KB total / %M KB max" ) or just use /usr/bin/time’s l-parameter.

On GNU/Linux on the other hand, rusage as defined by POSIX is not completely implemented, so you cannot do this. A workaround is preloading /lib/libmemusage.so which plugs into your program’s memory allocation, keeps track of stuff and prints funky statistics in the end.

That’s of course better than nothing, but it does create extra overhead, and especially the huge amount of colored (sic!) output that is added to your program’s stderr really comes in unhandy if you are working on a benchmarking infrastructure and have program pipelines, where you want stdout and stderr preserved.

Thankfully I found a different approach, that also hooks into the program by preloading a library, but not interfering with all the memory allocations. It uses /proc-fs on Linux to also get the maximum resident set memory of a process just before it terminates and prints this.

I changed it to print to the fourth file descriptor instead of stderr, so the output can be processed seperately. And I made output more easily parsable.

Then I wrote a wrapper script that you can pass your command call, and it will execute it (preserving stdout and stderr) and print the total run-time and the maximum resident set memory usage of your process (or a sub-process spawned by it) to FD4.

Here are the files: · mymemusage.c · wrapper.sh

I might adapt it to detect, if it is running on *BSD and then do the easy way (right now this will only work on Linux).

The only situation where this doesn’t work, is where your process spawns multiple processes in parallel and you want the maximum memory usage of the group. Then again, I think there is no solution for this that doesn’t involve polling /proc regularly for your process groups current memory usage and summing that up… Luckily parallelization is done mostly by threads nowadays.

Missing (Free) Android Apps - I - Office

Tue, 06 Nov 2012 15:04:11 +0000

Ok, so you know that I am using Android, and that I am not too happy with the situation. Ultimately Android is a JAVA-based operating system for phones, so maybe I shouldn’t be surprised that trying to replace a FreeBSD (or proper GNU/Linux) laptop with it, will cause some grief.

But, than Android does have some cool things (which I might elaborate on in another post), and the availibility just forces us as political Free Software users to take the OS seriously and care about its users.

So whats preventing me from be as productive on my device, as I want to be? Since I am not rooted and still running ASUS-Android 4.1.1 I will not talk about platform problems, like keyboard integration, but only apps. Maybe you know of some apps that could help, or see some Apps that you could help out with testing or hacking, who knows.

Part 1 of this series is on Word-processing

Ok, I don’t digg Office-apps in general, because I prefer tex, but especially in my current ultra-mobile use case, where I often need to take some formatted notes, or hand the Transformer to someone else, I need a working office app. My device came with Polaris Office preinstalled, which is proprietary and only creates docx files which totally rules it out.

I looked around for a while, but I really did not find a working free replacement. LibreOffice seems to have someone working on it, but judging from my previous experiencewith OOo it is probably such a memory hog that I doubt it will work nicely. But maybe I am too skeptic.

Then there is Calligra Active, the mobile port of former KOffice, which would probably be right for my usecase and looks like it is way more portable than LibreOffice, for which they are developing a completely new interface… BUT unfortunately the developer builds of Calligra Active crash on my Transformer. Have any of you tried them, yet? Is there some way this development could be spead up? I have the feeling some projects would be a lot further, if they had half the (wo)man-power other big projects have…

ASUS Transformer Pad Infinity

Wed, 31 Oct 2012 11:54:46 +0000

I have this tablet since two or three months now and I must say, I pleasently surprised with the hardware. People who know me, know that I don’t say stuff like that often. My taste in hardware just doesn’t align with the mainstream, I guess. I want hardware keyboards, I need high resolution displays, I like metal bodies and battery time is important. I don’t care about weight, style issues or fancy accesories, like pink fur covers or stuff like that.

I am not going go into details about the hardware, since you can look that up anywhere and there are lots of reviews available. Let’s just say, it has a great screen and long battery life. The only thing they could improve is giving the keyboard backlights, that would help working in the dark!

The issues I do have with the device are all related to software, I will go into that in a seperate post…

Long time no see

Wed, 31 Oct 2012 11:40:32 +0000

Hi everyone, ok so there haven’t been posts in while, because I was incredibly busy, but I do plan on finishing the line of posts concerning my audio setup. I also have a post in line regarding my (possibly irrelevant) thoughts on KDE and I have a few things to say about Android which I have been using a lot lately, so stay tuned.

The spam issue is getting kind of out-of-hand (over 1000 spam comments VS a handful of real ones), so if your comments don’t appear immediately don’t get angry. If in doubt of being regarded as a robot, just write me an email and I will whitelist you!

A versatile, open and future-proof audio setup [ part 2 ] Storage

Thu, 14 Jun 2012 13:22:09 +0000

This post covers storage and organisation of the files.

General Layout

MUSICDIR
  │
  ├── FILES
  │    ├── _flac
  │    ├── _flac_vorb
  │    ├── _other
  │    └── _vorb
  │
  ├── LINKS
  │    ├── best
  │    ├── smallest
  │    └── vorb
  │
  └── meta

Ok, the obvious stuff: · MUSICDIR is where you keep your music. · FILES is a sub directory that contains your actual files. · LINKS contains links to files from FILES, based on certain criteria. · meta contains maintenance scripts and other stuff that is not music.

Files

FILES contains four subdirectories. _flac contains my music in the FLAC-format. FLAC is a lossless audio codec, for more information click on the link. To be honest, I am not sure if I can hear the difference between FLAC and Ogg Vorbis at high bitrates, but that’s not the point. I don’t want to worry about quality and I want to be able to convert formats, if a better one appears, which is only feasible for lossless audio (you should never convert between lossy formats as it just increases lossiness).

Now while I am slowly reripping my music collection to FLAC, there is also some music where I only have the lossy versions on my hard-disk (e.g. broken or lost CDs…). These files go to _vorb if they are in Ogg Vorbis, or to _other if they are MP3, MPC or some other weird format. The _flac_vorb folder is special, it contains all the music I have as FLAC, but converted to low quality Ogg Vorbis (Quality 2.5). This is, because in some situations FLAC is not handy, e.g. when your bandwidth is limited or your storage is limited… the details will be explained later. A Script that converts your FLAC to Vorbis while preserving the file system hierarchy and Tags is available here.

Note that it is important that beneath the codec-directories you have the same hierarchy, i.e. if you organize your music by genre, you should do so for all codecs. Any other way of sorting is also good, but don’t have all your files just flying around on the topmost level. Actually the scripts do not allow any files on the topmost level, just directories. In case you were wondering, this is how I currently file my music beneath the aforementioned levels, although it is completely unimportant for the setup:

.../genre/artist/year - albumtitle/artist - tracknumber - tracktitle.ext

Links

The folders in LINKS are based on certain criteria. Since there is some redundancy in my files (songs both available as FLAC and Vorbis), I want to remove that. Also when actually playing music, of course I don’t want to worry about which format it is in, I want two CDs by the same artist to appear in the same folder, even if they are in different formats. So I wrote a little script that “joins” the folders from FILES by creating lots of symlinks in LINKS. It currently creates three “joins”: · best : contains the highest quality version of a song. This folder is the folder you want to index with computers on the local network (or just your desktop) · smallest : contains the smallest file of each song. This folder is the folder you want to index from computers that access the music from remote locations. It is also the directory you will want to sync to your portable audio player, your phone, whatever… · vorbis : contains just the vorbis files (original and flac-converted ones). I am not using this yet, but I want to serve this with OwnCloud, so I can listen to the music with html5 from other places. It might also be useful, if you want to stream through icecast or so.

Note that these are really many links if your music collection is big. I don’t know if that’s an issue on certain filesystems (I use ZFS, so I know it doesn’t matter).

Serving the files

Ok, now you have the music organized. You can think about how to serve it. I have it on a server, and can access it from different locations. I think that’s a good idea, but you can do all the stuff explained here on your regular computer as well.

Known Issues

· The scripts currently don’t do any cleanup, which is not optimal. I will fix that, as soon as I get time. When it’s fixed, it should be feasible to have the scripts run by cron. · When you change something in the hierarchy (e.g. split a genre folder up into two, want to delete an artist,…) you have to do this in all the folders in FILES which may be annoying (LINKS will be updated automatically, though).

stay tuned for the next post about portable audio!

A versatile, open and future-proof audio setup [ part 1 ] Overview

Wed, 06 Jun 2012 12:55:41 +0000

Here are the general components of the setup, as well as planned components. Fear not, most of everything that follows does not imply this setup and is generally usable. You don’t even need a server for the most part.

Server · acts as storage for local and remote access · handles maintenance jobs · plays music directly, remote controlled [planned] · runs FreeBSD and ZFS · more details in separate article

Regular Clients · some Laptops and Desktops · running GNU/Linux and FreeBSD · access files through NFS (+OpenVPN) · arbitrary player software

Portable Audio Player · Sandisk SansaClip+ · multiple µSD-Cards · more details (esp. on syncing) in separate article

Android Clients [planned] · one or multiple Android clients · access files through NFS or WebDAV · or µSD-Cards (see above)

Universal Access [planned] · based on OwnCloud

A versatile, open and future-proof audio setup [ part 0 ]

Mon, 04 Jun 2012 11:13:23 +0000

Since it might be useful to some other people I want to document my audio setup in a few blog posts. By audio setup I mean the storage, organization and playback of my music collection. It covers the following: file systems, file formats, network access, choice of portable hardware, my own maintenance software (conversion, synchronization etc) and briefly other software (operating systems, playback software).

The overall features of the system, as I will describe later:

fully compatible and based on Free Software codecs
uses Free Software in all parts, except playback on portable
inexpensive hardware, especially the portable part
scalable to very large music collections, expandable to growing music collections
easily adaptable to different or new situations (e.g. individual parts of the systems may be used independently)

The system works, but is not completely finished, i.e. some maintenance operations have yet to be added or tuned for performance or convenience, but I think it will be useful nevertheless. If you are skilled in bourne syntax, you might spot some rough edges or have ideas for the TODOs. If so, please let me know.

PDFReaders 2.0 - Your help is needed!

Mon, 21 May 2012 13:41:13 +0000

Hi everyone,

we are currently preparing a major update to PDFReaders.org, adding:

a more appealing and cleaner “home”-page, with one recommendation for the auto-detected platform
Free pdf reader recommendations for mobile platforms
Free pdf browser plugin recommmendations

BUT, we need some help with gathering/verifying information, deciding what to recommend…

Please have a look at the current Reader-Overview and the TODOs there.

Please tell us…

if you know any other Free pdf readers that we haven’t listed, yet (for any plaform, but especially OSX and iOS)
which Android reader you would recommend
any other ideas you might have!

You can reach us via mail or comment directly in the blog. If you have an account at the fellowship-wiki you can commit changes to the overview directly.

Thanks for your help!

Improving E-Mail Privacy

Sat, 12 May 2012 19:07:37 +0000

I have recently decided to use PGP / GnuPG to sign and encrypt emails, and I also recently switched from KMail (after using it for ~10 years) to Thunderbird [the why of the latter is a longer story I might tell some other time].

So, after not caring about email privacy for pretty long, I now got it all setup, although the setup on my laptop produces faulty inline PGP once in a while, where it doesn’t even recognize signed content itself…

Anyway, what I want to discuss today is the email header. As you all know, it contains all sorts of information, even your IP-Address – if you have a nasty provider. I kind of had the intuition that Thunderbird and Enigmail would reduce this info, or at least not add to it, but apparently that is not true.

Stuff you normally transmit with every (signed) E-Mail:

1) Thunderbird user agent I was kind of confused by it, as I thought this information should go into the “X-Mailer” info, but obviously Mozilla has their own way, as you can see below. And it contains quite a bit of information, beside the mail client: My operating system, my windowing system, my cpu-architecture, even the Gecko-revision (what has that got to do with anything?).

User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:10.0.3) Gecko/20120407 Thunderbird/10.0.3

2) Enigmail user agent Enigmail has its own client field, where it tells the world that it is installed on my system, and what version. Since Enigmail is specific to Thunderbird, this also tells people I am using Thunderbird (even if we get around 1) ).

 X-Enigmail-Version: 1.4

3) GnuPG Identifier Then, inside the PGP-Signature, GnuPG informs everyone of its version and my operating system:

 Version: GnuPG v2.0.19 (FreeBSD)

4) Enigmail advertising In case people hadn’t noticed from 2) or the combination of 1) and 3), here comes Enigmail again:

Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

Now, you might ask yourself why I worry about this information. Well first of all, it fulfills no real purpose. The receiving party doesn’t act on this information and it doesn’t need this information, as E-Mail is pretty standardized. This alone should be reason enough not to transmit the information. But here are some real scenarios:

If an attacker knows the applications you are using and even their versions, it might make it easier for him to send you prepared code that acts on certain known vulnerabilities of these application versions.
If you use more than one computer, people can identify which computer you wrote your E-Mail from based on different CPU-architectures, operating systems or application versions. This is less of an issue than the IP, but can still be used against you in different situations.

I didn’t find any place that describes how to easily fix this, so I gathered the solutions from different web-sites and man-pages… to fix all of the above, go to Edit->Preferences->Advanced->General->Config Editor and set these variables (they fix the above problems in the exact order):

general.useragent.override=""
extensions.enigmail.addHeaders=false
extensions.enigmail.agentAdditionalParam="--no-emit-version"
extensions.enigmail.useDefaultComment=true

The first two variables have to be created(string and bool), as they are non-default and there is no gui-way to set them. And yes the value for the last one is ’true'.

Hope this is usefull to some of you and saves you some googling or scroogling ;).

my awesome (wm) config

Sat, 05 May 2012 12:57:40 +0000

Since quite a few people have been asking me for my awesome config, I thought I’d upload it.

For those of you who don’t know awesome, check it out, it’s awesome! SCNR

Main differences from the default are:

remove all panels and stuff, because they are ugly
load xfce4-panel instead, because it’s light, but provides stuff like mpd-controls &c
create 12 tags (virtual desktops)
bind certain type of applications to certain desktops
map Meta to ALt instead of Winkey, because my main kbd doesn’t have a winkey
map the tags to F* instead of the number keys, so I get tags with Alt+F* (original unix style)
use cairo-compmgr where available (unfortunately not on FreeBSD) and pre-start some other stuff (e.g. launchy for running things)

Thats basically it. I have thought about using shifty for dynamic tagging, but haven’t gotten around to trying it, yet.

Here it is: rc.lua

(in case you consider that bit of code as (c)-able I hereby release into CC0 / public domain)

An inside view on the Great Chinese Firewall

Fri, 20 Apr 2012 14:08:33 +0000

As I am currently located in China, I thought I’d give all of you some technical infos on the current censorship techniques employed here. My experience differs a little from what Wikipedia tells us.

What you see as a user: youtube.com, facebook.com, twitter.com are all not reachable. Google.com is on and off, usually redirects to google.com.hk (which is still the less censored version of google.cn). Google.de is however available. A nice site that lets you test which web-sites are unavailable is http://www.greatfirewallofchina.org, from my experience it is reliable.

Now funny enough all things political that I checked are available, be it critical (western) newspaper sites, blogs et cetera. Even Wikipedia is available. All sites that I checked, especially Wikipedia and Google.de do offer SSL (valid!) and so looking at https://en.wikipedia.org/wiki/Tiananmen_Square_protests_of_1989 is possible (note that some sources claim that the western propaganda on this event is at least partly wrong, so don’t take this as defending the WP-article). Note also that I did not try without SSL, because I was told, that that would be detected and can get your internet service cut.

Whats going on under the hood: Like I said, I did not experiment too much with entering sensitive terms in search engines, but regarding the web-site blocking I can say that

the DNS-Servers here resolve certain adresses to other IPs (obvious one). Strangely enough first neblock-owner checks reveal these IPs to be outside of China, even EU… will investigate this further.
dns-requests to foreign (probably non-whitelisted) DNS-servers are either all rerouted (based on port or deep packet inspection), or filtered and manipulated in case the requested host matches a blacklist or an expression (based on deep packet inspection). So just setting a different DNS-Server is no good, even when they are not widely known (contrary to what WP says)
traffic to the real IPs of the sites is blocked, so if you happen to know where twitter.com is, it still doesn’t get you there. This is probably achieved by resetting the TCP connection. Note that this happens early on, a traceroute to twiiter only goes one hop before dissappearing.

I currently go online trough SSH port forwarding, tunneling and proxies, will update that to proper VPN, as soon as I get the time. Anything encrypted really works, no issues with ssh and https on regular ports. OpenVPN also works…

FSFE Affiliate Userscript v0.3 released.

Thu, 05 Apr 2012 09:08:38 +0000

Thanks to Hugo, for finding a bug in the version 0.2 with non-[.de|com] domains. Should be fixed now. Get the new version on my blog, as usual.

FSFE Affiliate Userscript v0.2 released.

Thu, 29 Mar 2012 19:17:30 +0000

I decided to rewrite the script.

What’s new:

only one 5-line function from another script is used, the rest is original code under CC0 / Public Domain
Amazon code is cleaner, should produce less false-positive replacements (in theory ;) )
Support for Libri.de / Bookzilla.de is added. Any and all links to Libri are replaced with links to Bookzilla. Actions on Libri.de itself are also “transferred”.

FSFE-Affiliate Userscript now available

Thu, 29 Mar 2012 11:56:26 +0000

I have uploaded a userscript that you can use to support FSFE when buying at Amazon. Automatic support for redirecting from libri.de to bookzilla.de is planned, aswell as any other affiliate programs FSFE might take part in.

In contrast to the AffiliateFox-Plugin, the userscript is compatible with non-Mozilla browsers. But it works on those, too, in case you prefer to install something where you can see the source, before installing it ;)

For more information see post/userscripts/

Blog and Micro-Blogs now working!

Wed, 28 Mar 2012 15:11:06 +0000

After the last test post, I am now convinced that the blog and the micro-blogs work. You can visit this site regularly or follow me on identi.ca or Twitter.

Should we call Free Software releases "products"?

Wed, 28 Mar 2012 14:46:28 +0000

I have posted a lengthy answer on Bernhard’s blog[1], why I don’t think “product” is a suitable term. What are your thoughts on this? Reply here or on Bernhards’ blog!

[1] http://blogs.fsfe.org/bernhard/2012/03/lets-end-all-free-software-projects-quickly/