This is a problem because I use the dmesg log to monitor important hardware related messages such as the kernel recognizing a USB device or diagnosing bluetooth/wifi issues. When I plug in a USB drive, the first thing I do is check dmesg for the following messages:

[10701.359834] usb 2-4.4: new high-speed USB device number 8 using ehci-pci
[10701.394801] usb 2-4.4: New USB device found, idVendor=12f7, idProduct=0313, bcdDevice= 1.10
[10701.394807] usb 2-4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[10701.394810] usb 2-4.4: Product: MerryGoRound
[10701.394813] usb 2-4.4: Manufacturer: Memorex
[10701.394816] usb 2-4.4: SerialNumber: AAAAAAAAAAAA
[10701.395182] usb-storage 2-4.4:1.0: USB Mass Storage device detected
[10701.398885] scsi host7: usb-storage 2-4.4:1.0
[10702.401161] scsi 7:0:0:0: Direct-Access     Memorex  MerryGoRound     PMAP PQ: 0 ANSI: 0 CCS
[10702.401710] sd 7:0:0:0: Attached scsi generic sg6 type 0
[10702.651720] sd 7:0:0:0: [sde] 15654912 512-byte logical blocks: (8.02 GB/7.46 GiB)
[10702.652341] sd 7:0:0:0: [sde] Write Protect is off
[10702.652346] sd 7:0:0:0: [sde] Mode Sense: 23 00 00 00
[10702.652961] sd 7:0:0:0: [sde] No Caching mode page found
[10702.652965] sd 7:0:0:0: [sde] Assuming drive cache: write through
[10702.681473]  sde: sde1 sde2
[10702.684869] sd 7:0:0:0: [sde] Attached SCSI removable disk

This output reports that a USB device was detected, where it is plugged in, what is the vendor/product information, what USB speed it is using, the size of the storage, the device name (/dev/sde), and its partitions (/dev/sde1, /dev/sde2). There are a lot of other messages written out to dmesg, such as the kernel detecting a bad USB cable, segementation faults, and so on.

Given the importance of the above log output, I have developed a habit of running dmesg -w to monitor such kernel events. The -w tells dmesg to monitor for new messages. The long option is --follow.

In addition to dmesg -w not working as intended, syslogng log entries written to /var/log/kern.log are not written as they occur; instead the log is written in "bursts", which suggests sysklogd occasionally reopens /dev/kmsg, thereby reading in new log messages, but all the timestamps are the same time for each "burst" read.

I have two systems with a virtually identical OS installation; one is a workstation named snowcrash with an AMD FX-8350 on an ASRock M5A97 R2.0 motherboard; the other is a HP Elitebook 820 G4 named cyberdemon with an Intel Core i5-7300U. Curiously enough, the strange dmesg -w hang does not occur on cyberdemon, but does occur on snowcrash. Both hosts run Linux mainline, with both machines on 5.6.4. Looking through my /var/log/kern.log files, this behavior was apparent on a 5.4.25 kernel. As we will see later, this coincides with the affected versions that others have reported

Additionally, I asked my friend tyil who happens to also use an AMD FX-8350 with Gentoo to check for the bug; he also had the problem on 5.6.0.

First thing I did was find a way to reproduce the issue. I recorded an asciinema recording. You can watch it here. I then shared the recording on IRC, hoping somebody would know of a solution. I got some helpful and encouraging feedback, but nobody knew of this particular bug. See the recording below, or click here:

The next step was to figure out if there was something wrong with /bin/dmesg. Running strace -o dmesg-strace.log dmesg -w shows the follow pertinent lines:

openat(AT_FDCWD, "/dev/kmsg", O_RDONLY) = 3
lseek(3, 0, SEEK_DATA)                  = 0
read(3, "6,242,717857,-;futex hash table "..., 8191) = 79
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x16), ...}) = 0
openat(AT_FDCWD, "/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=26388, ...}) = 0
mmap(NULL, 26388, PROT_READ, MAP_SHARED, 4, 0) = 0x7fed92688000
close(4)                                = 0
futex(0x7fed925f9a14, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(1, "\33[32m[    0.717857] \33[0m\33[33mfut"..., 97) = 97
… SNIP …
read(3, "6,1853,137347289701,-;input: Mic"..., 8191) = 128
write(1, "\33[32m[137347.289701] \33[0m\33[33min"..., 140) = 140
read(3,

The last line indicates a pending read() that never completes. Note the 3 file description refers to the /dev/kmsg device. It appears nothing out of the ordinary occurs, except for the fact that the read() simply hangs.

I was a bit at a loss to explain the hanged read(). I was really lost honestly. So I went on, and inspected the changes to /bin/dmesg shipped by util-linux, and did not find any sign of significant changes. I did run dmesg from master just to be sure. See the commit log of dmesg.c here. Additionally I also searched the util-linux bug tracker and found nothing relevant.

Given I had no solution yet, I decided to resort to Googling things, hoping somebody had discussed this bug before. Keywords I tried are:

• dmesg follow no longer working
• dmesg kmsg no more messages
• linux kmsg read hang
• "/dev/kmsg" hang
• "dmesg -w" hangs

None of these came up with anything useful. I was using DuckDuckGo mainly, with some Google queries sprinkled on top.

I then visited the torvalds/linux GitHub repository, searched for "kmsg", and did not find a commit that looked like a fix. I picked on through reading commits that /dev/kmsg is written to via the printk functions, so on a whim I decided to look changes made to kernel/printk/printk.c. Reading through the commit logs of printk.c, I realized the last commit is likely the fix:

commit ab6f762f0f53162d41497708b33c9a3236d3609e
Author: Sergey Senozhatsky <protected@email>
Date:   Tue Mar 3 20:30:02 2020 +0900

    printk: queue wake_up_klogd irq_work only if per-CPU areas are ready
    
    printk_deferred(), similarly to printk_safe/printk_nmi, does not
    immediately attempt to print a new message on the consoles, avoiding
    calls into non-reentrant kernel paths, e.g. scheduler or timekeeping,
    which potentially can deadlock the system.
    
    Those printk() flavors, instead, rely on per-CPU flush irq_work to print
    messages from safer contexts.  For same reasons (recursive scheduler or
    timekeeping calls) printk() uses per-CPU irq_work in order to wake up
    user space syslog/kmsg readers.
    
    However, only printk_safe/printk_nmi do make sure that per-CPU areas
    have been initialised and that it's safe to modify per-CPU irq_work.
    This means that, for instance, should printk_deferred() be invoked "too
    early", that is before per-CPU areas are initialised, printk_deferred()
    will perform illegal per-CPU access.
    
    Lech Perczak [0] reports that after commit 1b710b1b10ef ("char/random:
    silence a lockdep splat with printk()") user-space syslog/kmsg readers
    are not able to read new kernel messages.
    
    The reason is printk_deferred() being called too early (as was pointed
    out by Petr and John).
    
    Fix printk_deferred() and do not queue per-CPU irq_work before per-CPU
    areas are initialized.
    
    Link: https://lore.kernel.org/lkml/aa0732c6-5c4e-8a8b-a1c1-75ebe3dca05b@camlintechnologies.com/
    Reported-by: Lech Perczak <protected@email>
    Signed-off-by: Sergey Senozhatsky <protected@email>
    Tested-by: Jann Horn <protected@email>
    Reviewed-by: Petr Mladek <protected@email>
    Cc: Greg Kroah-Hartman <protected@email>
    Cc: Theodore Ts'o <protected@email>
    Cc: John Ogness <protected@email>
    Signed-off-by: Linus Torvalds <protected@email>

Unfortunately my understanding of the linux kernel architecture is not comprehensive, let alone competent, the commit message describes

1. syslog/kmsg readers — which includes dmesg and syslogng,
2. certain functions don't immediate attempt to print a new message to console,
3. and syslog/kmsg readers might not wake up.

Indeed it's a bit hard for me to wrap my minimal kernel understanding around, however, reading the linked email list thread clears things up significantly:

After upgrading kernel on our boards from v4.19.105 to v4.19.106 we found out that syslog fails to read the messages after ones read initially after opening /proc/kmsg just after booting. I also found out, that output of 'dmesg –follow' also doesn't react on new printks appearing for whatever reason - to read new messages, reopening /proc/kmsg or /dev/kmsg was needed. I bisected this down to commit 15341b1dd409749fa5625e4b632013b6ba81609b ("char/random: silence a lockdep splat with printk()"), and reverting it on top of v4.19.106 restored correct behaviour.

— Lech Perczak

Now that, sounds like the issue I'm having! The thread also discusses the bug is apparent on 4.19.106 (fixed in 4.19.107 — see this commit), and affects users of 5.5.9, 5.5.15, 5.6.3 (see the PATCHv2 thread).

Next step is to apply the patch in order to test and verify this fixes the issue.

Since Fall last year, I have used sys-kernel/vanilla-kernel to compile, install, and create an initramfs for my two machines. This is a great ebuild because it uses a kernel .config based off of Archlinux's, so it is compatible with most machines. It is also streamlined in that it does all the work for you — no more manually configuring and remembering which make invocations are necessary to update the kernel. It's not hard to get right, but it's not particularly interesting in my use-case. Additionally, using sys-kernel/vanilla-kernel, the kernel & its modules are now packaged, and can be distributed to my other machine as a binpkg. This streamlines deployment significantly.

In order to add the patch to this ebuild, I simply have to drop the patch file into /etc/portage/patches/sys-kernel/vanilla-kernel. In my case I chose to drop it in /etc/portage/patches/sys-kernel/vanilla-kernel:5.6.4 because I rather the patch only be applied no the current kernel I have installed, than all versions of sys-kernel/vanilla-kernel. This ensures when I upgrade to to the upcoming 5.7 release (which has the fix included), the patch won't be applied and emerge won't fail due to the patch not being applied cleanly.

The commands (commit to my /etc/portage):

mkdir -p /etc/portage/patches/sys-kernel/vanilla-kernel:5.2.6
curl -o /etc/portage/patches/sys-kernel/vanilla-kernel:5.2.6/fix-dmesg--follow.patch \
        https://github.com/torvalds/linux/commit/ab6f762f0f53162d41497708b33c9a3236d3609e.patch
emerge -1av sys-kernel/vanilla-kernel:5.2.6

An hour later and the kernel is installed. After the reboot, indeed dmesg -w works once again! And the log messages in /var/log/kern.log have timestamps that correctly reflect the kernel time!

Even kernels have regressions. As discussed on IRC, I was reminded that the kernel project is not responsible for the userland, so it's possible such testcases might not be on the radar of most kernel developers. Perhaps it's the distros' responsibilities to execute integrated system testing to catch bugs like this. In any case it is still a surprise to see such a regression occur. We like to think of the kernel as this infallible magical machine that doesn't break except when you do something patently wrong, but this isn't really the case. We're all human.

I want to thank Tyil, Sergey (the patch author), Lech (the bug reporter) and some folks from the #linux IRC channel for helping me pinpoint this issue. The reader may think this is a lot of effort to go through to fix such a simple bug — but it's really important for the kernel to work — if the kernel misbehaves, anything is up for grab. It's not like your bug-laden browser product we accept will have crashing bugs in it — if the kernel crashes or misbehaves, the ramifications are almost as bad as if the hardware is failing — you'll lose your application data, productivity, and trust in the operating system itself.

It is important to mention LTS (Long term support) kernels exist. Given the amount of trouble I went to address this issue, and the fact I rather not have things breaking, I don't think I should be running a mainline kernel at the moment. Perhaps I can install both side by side. then pick 'n choose which kernel to use de jure.

I am very interested to hear of you, the reader's, suggestions for kernel maintenance and version selection strategies. You can find my contact details at https://winny.tech/ . Thank you for reading.

I spent a bunch of time trying to get crash dumps from Zathura, and was largely unsuccessful, until I realized the wonkiness I was dealing with (see below).

Knowing GDB is a powerful, useful skill. Nothing beats understanding your debugger. Not even printf debugging.

Let's start with the source of the crash:

(gdb) frame 0
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
120             movdqu  (%rax), %xmm4
(gdb) list
115     #ifdef AS_STRNLEN
116             andq    $-16, %rax
117             FIND_ZERO
118     #else
119             /* Test first 16 bytes unaligned.  */
120             movdqu  (%rax), %xmm4
121             PCMPEQ  %xmm0, %xmm4
122             pmovmskb        %xmm4, %edx
123             test    %edx, %edx
124             je      L(next48_bytes)
(gdb) info registers rax
rax            0x61                97

So strlen is trying to derefence address 0x61, that doesn't look right. Cheking the output of info proc mappings shows zathura doesn't have mapped memory that corresponds to the value in rax.

(gdb) info proc mappings
process 12314                                                                                                                  
Mapped address spaces:                                                                                                         

          Start Addr           End Addr       Size     Offset objfile                                                          
      0x555555554000     0x55555555f000     0xb000        0x0 /usr/bin/zathura                                                 
      0x55555555f000     0x555555581000    0x22000     0xb000 /usr/bin/zathura                                                 

…SNIP…

      0x7ffff7ffd000     0x7ffff7ffe000     0x1000    0x27000 /lib64/ld-2.29.so
      0x7ffff7ffe000     0x7ffff7fff000     0x1000        0x0 
      0x7ffffffdd000     0x7ffffffff000    0x22000        0x0 [stack]
  0xffffffffff600000 0xffffffffff601000     0x1000        0x0 [vsyscall]

Now let's carry on with the second frame.

(gdb) frame 1
#1  0x00007ffff721680d in g_strjoinv (separator=separator@entry=0x7fffec702546 "-", str_array=str_array@entry=0x555555e56ac0)
    at ../glib-2.60.7/glib/gstrfuncs.c:2585
2585          for (i = 1; str_array[i] != NULL; i++)
(gdb) info frame
Stack level 1, frame at 0x7fffffffd140:
 rip = 0x7ffff721680d in g_strjoinv (../glib-2.60.7/glib/gstrfuncs.c:2585); saved rip = 0x7fffec6fd31b
 called by frame at 0x7fffffffd200, caller of frame at 0x7fffffffd0f0
 source language c.
 Arglist at 0x7fffffffd0e8, args: separator=separator@entry=0x7fffec702546 "-", str_array=str_array@entry=0x555555e56ac0
 Locals at 0x7fffffffd0e8, Previous frame's sp is 0x7fffffffd140
 Saved registers:
  rbx at 0x7fffffffd108, rbp at 0x7fffffffd110, r12 at 0x7fffffffd118, r13 at 0x7fffffffd120, r14 at 0x7fffffffd128,
  r15 at 0x7fffffffd130, rip at 0x7fffffffd138
(gdb) list
2580          gsize separator_len;
2581
2582          separator_len = strlen (separator);
2583          /* First part, getting length */
2584          len = 1 + strlen (str_array[0]);
2585          for (i = 1; str_array[i] != NULL; i++)
2586            len += strlen (str_array[i]);
2587          len += separator_len * (i - 1);
2588
2589          /* Second part, building string */
(gdb) print *str_array
$1 = (gchar *) 0x555555cea670 "Canon"
(gdb) print str_array[1]
$2 = (gchar *) 0x555555dac870 "MF632C"
(gdb) print str_array[2]
$3 = (gchar *) 0x555555e87150 "634C"
(gdb) print str_array[3]
$4 = (gchar *) 0x61 <error: Cannot access memory at address 0x61>

Indeed, we cannot access memory of address 0x61. And looking at the source and documentation for g_strjoinv, the str_array argument should be a NUL terminated array of strings.

Let's look at the third frame.

(gdb) frame 2
#2  0x00007fffec6fd31b in avahi_service_resolver_cb (source_object=<optimized out>, res=<optimized out>, 
    user_data=user_data@entry=0x555555e06040)
    at /usr/src/debug/x11-libs/gtk+-3.24.13/gtk+-3.24.13/modules/printbackends/cups/gtkprintbackendcups.c:3223
3223                  data->printer_name = g_strjoinv ("-", printer_name_compressed_strv);
(gdb) info frame
Stack level 2, frame at 0x7fffffffd200:
 rip = 0x7fffec6fd31b in avahi_service_resolver_cb
    (/usr/src/debug/x11-libs/gtk+-3.24.13/gtk+-3.24.13/modules/printbackends/cups/gtkprintbackendcups.c:3223); 
    saved rip = 0x7ffff73caf79
 called by frame at 0x7fffffffd220, caller of frame at 0x7fffffffd140
 source language c.
 Arglist at 0x7fffffffd138, args: source_object=<optimized out>, res=<optimized out>, user_data=user_data@entry=0x555555e06040
 Locals at 0x7fffffffd138, Previous frame's sp is 0x7fffffffd200
 Saved registers:
  rbx at 0x7fffffffd1c8, rbp at 0x7fffffffd1d0, r12 at 0x7fffffffd1d8, r13 at 0x7fffffffd1e0, r14 at 0x7fffffffd1e8,
  r15 at 0x7fffffffd1f0, rip at 0x7fffffffd1f8
(gdb) list
3218                          printer_name_compressed_strv[j] = printer_name_strv[i];
3219                          j++;
3220                        }
3221                    }
3222
3223                  data->printer_name = g_strjoinv ("-", printer_name_compressed_strv);
3224
3225                  g_strfreev (printer_name_strv);
3226                  g_free (printer_name_compressed_strv);
3227                  g_free (printer_name);
(gdb) print printer_name_compressed_strv 
$5 = (gchar **) 0x555555e56ac0

Note the value of printer_name_compressed_strv of 0x555555e56ac0 corresponds to the value of str_array in the previous frame (g_strjoinv()). The full definition of avahi_sernvice_resolver_cb can be read on Gnome's GitLab.

As mentioned above, we found the sentinel value of the string array was not NUL. Looking at the following code, do you see the bug? I honestly didn't:

printer_name = g_strdup (name);
g_strcanon (printer_name, PRINTER_NAME_ALLOWED_CHARACTERS, '-');

printer_name_strv = g_strsplit_set (printer_name, "-", -1);
printer_name_compressed_strv = g_new0 (gchar *, g_strv_length (printer_name_strv));
for (i = 0, j = 0; printer_name_strv[i] != NULL; i++)
  {
    if (printer_name_strv[i][0] != '\0')
      {
        printer_name_compressed_strv[j] = printer_name_strv[i];
        j++;
      }
  }

data->printer_name = g_strjoinv ("-", printer_name_compressed_strv);

After spending some time refamiliarizing myself with glib, GTK+, and Googling, it became apparently after I found the commit that fixed it. Let me preface the found commit with a brief explaination.

1. g_strcanon() replaces characters not in PRINTER_NAME_ALLOW_CHARACTERS with a hyphen, i.e. "Canon MF632C/634C" becomes "Canon-MF632C-634C"

2. g_strsplit_set() splits printer_name on "-", giving the following array:

   (gdb) print *printer_name_strv@g_strv_length(printer_name_strv)+1
   $12 = {0x555555eff250 "Canon", 0x555555ece120 "MF632C", 0x555555f397e0 "634C", 0x0}
   

3. g_new0() initializes a zero-filled array of pointers of length 3, the number of splitted elements returned by g_strsplit_set()
4. The for loop copies over the contents of printer_namestrv, but skips empty elements - e.g. in the case that the above string had two adjacent hypens.
5. The g_strjoinv() joins each string in the printer_name_compressed_strv array of strings, joining them on "-".

The problem occurs because the call to g_new0 does not account for the extra array sentinel element. Indeed that is what the GitLab commit discusses.

In this case, I did what is best for my time effort. GTK+ 3.24.14 has been out for a couple months, and GTK+ 3.24.13 is not much older. So instead of dealing with backports, that is making a patch for the older version of GTK+ and adding it to my install, I took the liberty to bump my local GTK+ 3 install to GTK+ 3.24.14.

Either is not too tricky, in all honesty, given adding patches to a Gentoo system is as easy as placing the patch in the correct path. And bumping the ebuild usually entails simply unmasking it via accepting the keyworded version (in my case ~amd64).

As such this is all I had to do to fix the issue:

snowcrash ~ # echo '~x11-libs/gtk+-3.24.14 ~amd64' >> /etc/portage/package.accept_keywords/gtk
snowcrash ~ # emerge -uDU -av --changed-deps --verbose-conflicts @world

These are the packages that would be merged, in order:



[ebuild     U ~] x11-libs/gtk+-3.24.14:3::gentoo [3.24.13:3::gentoo] USE="X cups examples introspection xinerama (-aqua) -broadway -cloudprint -colord -gtk-doc -test -vim-syntax -wayland" ABI_X86="(64) -32 (-x32)" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

Would you like to merge these packages? [Yes/No] 

And viola. I am able to print!

In this post, I described several related challenges:

1. How to get a backtrace
2. What happens when seccomp blocks ptrace
3. How to install debug symbols and source code on Gentoo
4. What it looks like to pick apart a backtrace
5. And how I fixed this particular issue

In retrospect, I should have reported the bug to the gentoo tracker, because this was a bug due to the selection of patches cherry picked off the gtk git repository. Thankfully the affected versions of GTK are no longer in the official gentoo ebuild repository. I'll be sure to report such bugs going forward! It pleases me how easy Gentoo makes it to debug stuff.

I found the entire experience here informative, but also incredibly irritating. I'm no stranger to debugging crashes and grabbing debug symbols. When something gets in the way of getting backtraces, things really get very frustrating. The debugging part is the fun part; dealing with wildcards like seccomp preventing ptrace() with no meaningful error messages is a huge time waster.

The lack of literature about debugging seccomp enabled applications was a factor in this frustration. I only figured out the issue due to taking the time to read the source code, grepping for ptrace, and understanding seccomp as its used in Zathura. Had I read the README I could have saved some time; It's important to read all the documentation.

If you made it this far, you have a lot of patience for ramblings and hobbyist computing. You're terrific! Next time you run into a segfault, put what you've learned to good use!

1. Eat the exact same thing every day: Eggs, Bacon, Corn Tortillas. Take any dietary supplements/vitamins.
2. Make coffee — Aeropressed. Currently sourcing coffee from Ruby. I wish to see more Ethiopia/Kenya/Peru coffee but even a more earthy Colombia coffee is agreeable.
3. Spend 30–60 minutes checking email, news, IRC.
4. Wash up
5. Spend about 2-3 hours on current tasks — schoolwork, bugfixing, packaging, or researching.
6. Eat a lunch, make some tea
7. Spend another 2-3 hours on same tasks.
8. Take a break, preferably away from computer
9. Spend another couple hours
10. Have a dinner
11. Spend some time with housemates, make sure we're all on the same page
12. Spend another couple hours on tasks
13. Wash up, goto bed

In retrospect, I think I should replace one of those task blocks with off-task things such as gaming, reading books, and so on. That's way too much time on task, and knowing myself I end up being less productive due to too much time on task.

For a long time now, I've aimed to keep all my system-wide software packaged in the OS package manager. This allows me to easily rebuild my systems, or deploy the OS on new computers. This also means updates become a lot easier, because the package manager can track things such as rebuilding all library dependencies, and ensuring dependencies are installed and are the correct versions. Depending on your OS it's pretty easy. In my case Gentoo makes it extremely easy.

In 2015 I purchased a Nvidia GTX 760 used for $50. It was a great investment. At the time AMD driver quality is pretty poor. This is the post-fglrx horror years, but the drivers were still subpar compared to Nvidia's proprietary drivers. You could not expect to get Windows-par graphics performance on an AMD card in 2015. On the other hand one could expect Windows-par graphics performance on a Nvidia card.

I found using a graphics tablet to be valuable to my math class. I can take notes in xournal, and write problems step by step. You might wonder what's wrong with paper, but it seems when in front of a computer watching lectures and interacting with online learning management systems, it is difficult to split attention between the computer and the notebook. As such I simply decided to digitize the notebook.

To make the experience more tolerable, I have been using youtube-dl to grab all the videos I can, and play them locally in MPV. This ensures I have global multimedia shortcuts to control video playback, have better control over frame advance, do not require internet access 24/7, and have better control over playback speed.

As I finally packaged the software used for one of my classes I can do all that class's work locally except for submission. This is fantastic because I can use my Emacs 26 setup and do not require a 24/7 Internet connection.

My office is located in a room that can get down to the low 60°'s at night, and I found many times I'd want to do work, I could barely focus because I was so unevenly cold. The floor would be 60-65 but the room would be 70. I feel like an old man complaining about this, but really getting a space heater did wonders for my productivity and focus. This is a schoolwork problem, because it's the most tedious sort of productivity.

This has been a rather long post. I really wanted to describe some of the things I've been up to, and some of the challenges I've been facing. I am very happy to have removed my workstation Nvidia dependency. I am very excited about graduating soon, and adjusting in this time, and keeping that in mind, has been a challenge. As usual packaging software and fixing keeps my computers usable and maintainable.

I hope to write more in the near future. I had started some posts on debugging a GTK bug, and some other topics, but the amount of material to cover kept growing, much like this post keeps growing. Stay tuned to read about seccomp madness.

Initially I wanted to segment my devices on its own RFC1918 network. The idea is the main Amplifi LAN would be on 192.168.182.0/24 and my network would be on 10.9.8.0/24. I know I can ensure an address from each subnet is on each router, and the routers should then be able to route between each other without any additional configuration.

Unfortunately Amplifi does not offer a facility to add additional IP addresses to the LAN IP configuration. Think about how absurd that is—a router doesn't offer facility to add multiple addresses on an interface or bridge. Yep pretty silly. This leads me to the second approach: add static routes to make every segment on the network routable. Unfortunately, yet again, Ubiquiti Amplifi does not offer this standard router feature.

Given the limitations of Amplifi to not offer static routing or multiple LAN IP addresses, I think the only approach where I could maintain an additional subnet is with NAT (Network Address Translation) configured, which totally defeats the purpose of having a second subnet which could host any number of network services that should be reachable anywhere on the network, without any port forwarding.

My examples assume the user is logged into the CLI via SSH. Make sure to read through the RouterOS Wiki documentation and upcoming replacement wiki when things are not clear.

In particular check out the guides for scripting, using the console, first-time setup, and troubleshooting tools.

The steps are this:

Though a casual reader might consider this too many steps to configure a network device, it is only a handful of operations. A Router/Switch device's web configuration GUI might help streamline this, but that doesn't mean it is any simpler to configure. Chances are one will have to fill in more boxes and tick more fields in such a scenario, since this is a somewhat weird and awkward network topology.

As usual the Mikrotik Wiki helped me get this project finished in little time. Doubly so thanks to the Mikrotik IRC channel. I rather enjoy working with RouterOS because it feels rather state-less. When I specify a configuration, it is usually idempotent, every line of configuration feels relevant to the use-case, and in general feels rather DWIM (Do what I Mean).

I hope this helps somebody, because this took some mild mental aerobics to figure out how this works. In particular I knew that WiFi and Ethernet have different frame formats, and Layer 2 bridging between the two requires weird hacks that are unique to the vendor's software/hardware. In this case using station-pseudobridge, the Routerboard does Layer 2 bridging for certain traffic, and falls back on Layer 3 for the rest. I don't fully understand it, but the results are satisfactory.

Caddy is very easy to get started with, but it has its own set of trade-offs. Over the last few years, I’ve noticed multiple hard-to-isolate performance quirks, some of which were likely related to the official Docker image. In particular, I had built a Docker image of Caddy with webdav support, and the overall performance tanked to seconds per request, even with webdav disabled. I still have no clue what happened there; instrumenting Caddy through Docker appeared nontrivial, so I gave up on webdav support, reverted to my old Docker based setup, and everything was fast, once again.

There is a good amount of inflexibility in Caddy, such as the git plugin’s limitation to deploy to a non-root folder of the web root. And its rewrite logic is usually what you want, but not nearly as flexible as nginx’s.

Asking questions on their IRC is usually met with no response of any kind, which indicates to me that the project’s community isn’t very active.

The move to Caddy v2 is unwelcoming; I don’t want to relearn yet another set of config files and quirks, especially weeding through the layer of configuration file format adapters and the abstracted-away configuration options, so I rather just use Certbot and some other HTTPD that won’t change everything for the fun of it.²

Until recently Caddy experimented with a pretty dubious monetizing strategy. HackerNoon published an article detailing how it worked. In short: they plastered text all over their website claiming you “need to buy a license” to use Caddy commercially, though that claim was never true. Caddy was always covered by Apache License 2.0. Instead, you needed a commercial license in the narrow use-case that your organization wants to use Caddy’s prebuilt release binaries as offered on their website. It is good they stopped this scheme, but it leaves a bad taste with the community, and with me, and discourages me from relying on the project moving forward.

I have used both GitHub Pages and GitLab Pages in the past. My experience with GitHub Pages is it’s relatively inflexible and difficult to see what is going to be published, and has a CI/CD setup only useful for certain Jekyll based sites. GitLab pages, on the other hand, lets you set up any old Docker-based CI/CD workflow, so it is possible to render a blog with GitLab CI of any static site generating software. The IEEE-CS student chapter I am a part of does just this. We use a combination of static redirect sites and a Pelican-powered static website. There are a large number of example repositories for most of the popular ways to publish a static website, including Gatsby, Hugo, and Sphinx. Needless to say GitLab Pages puts GitHub pages to shame in terms of flexibility.

There are two steps in setting up GitLab Pages. These are the most important ideas related to GitLab pages; how to navigate the site is something the reader must experience for oneself. Nothing beats experimentation and reading the docs. Make sure to refer to the official GitLab Pages documentation for further details.

There are a few gotchas about GitLab Pages. Some of them are related to GitLab Pages users not being familiar with all of the DNS RFCs. Others are simply because GitLab Pages has quirks too.

GitLab pages isn’t perfect, but this should streamline what services my VPS hosts, and give me more freedom to fiddle with my VPS configuration and deployment. I look forward to rebuilding my VPS with cdist, ansible, or saltstack. While that happens, my website will be up thanks to GitLab pages. Also, I imagine GitLab Pages is a bit more resilient to downtime than a budget VPS provider.

The repositories with .gitlab-ci.yml files for both this site, and winny.tech are public on GitLab official hosting. Presently it is the simplest setup possible, simply deploying pre-generated content already checked into git, but the possibilities are endless.

IT is a good idea to verify if your kernel is missing a driver at boot, or is missing firmware or both. Boot up a Live USB with good hardware compatibility, such as GRML¹ or Ubuntu’s, and let’s see what framebuffer driver our host is trying to use²:

$ dmesg | grep -i 'frame.*buffer'
[    4.790570] efifb: framebuffer at 0xe0000000, using 8128k, total 8128k
[    4.790611] fb0: EFI VGA frame buffer device
[    4.820637] Console: switching to colour frame buffer device 240x67
[    6.643895] i915 0000:00:02.0: fb1: i915drmfb frame buffer device

Se we can see the efifb is initially used for a couple seconds, then i915 is used for the rest of the computer’s uptime. Now let’s look at if firmware is necessary, first checking if modinfo(8) knows of any firmware:

$ modinfo i915 -F firmware
i915/bxt_dmc_ver1_07.bin
i915/skl_dmc_ver1_27.bin
i915/kbl_dmc_ver1_04.bin
... SNIP ...
i915/kbl_guc_33.0.0.bin
i915/icl_huc_ver8_4_3238.bin
i915/icl_guc_33.0.0.bin

This indicates this driver will load firmware when available, and if necessary for the particular mode of operation or hardware.

Now let’s look at dmesg to see if any firmware is loaded:

[    0.222906] Spectre V2 : Enabling Restricted Speculation for firmware calls
[    5.511731] [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)
[   25.579703] iwlwifi 0000:02:00.0: loaded firmware version 36.77d01142.0 op_mode iwlmvm
[   25.612759] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[   25.620251] Bluetooth: hci0: Found device firmware: intel/ibt-12-16.sfi
[   25.712793] iwlwifi 0000:02:00.0: Allocated 0x00400000 bytes for firmware monitor.
[   27.042080] Bluetooth: hci0: Waiting for firmware download to complete

Aha! So it appears we need i915/kbl_dmc_ver1_04.bin for i915. In the case case one doesn’t need firmware, it won’t show anything related to drm or a line with your driver name in it.

By the way, it is a good idea to check dmesg for hints about missing firmware, or alternative drivers, for example my trackpad is supported by both i2c and synaptics based trackpad drivers, and the kernel was kind enough to tell me.

On Gentoo install sys-kernel/linux-firmware. You will have to agree to some non-free licenses; nothing too inane, but worth mentioning. Now just run emerge -av sys-kernel/linux-firmware. (On other distros it might be this easy, or more difficult; for example—in my experience Debian does not ship every single firmware like Gentoo does, so YMMV.)

Since most of my systems run Gentoo, it is business as usual to deploy a kernel with most excess drivers disabled except for common hot-swappable components such as USB network interfaces, audio devices, and so on. For example, this laptop’s config was originally derived from genkernel’ stock amd64 config with most extra drivers disabled, then augmented with support for an Acer ES1-111M-C7DE, and finally with support for this Elitebook.

I had compiled the kernel with i915 support built into the image, as opposed to an additional kernel module. Unfortunately this meant the kernel is unable to load firmware from filesystem, because it appears only kernel modules can load firmware from filesystem. To work around this without resorting to making i915 a kernel module, we can include the drivers within the kernel image (vmlinuz). Including firmware and drivers both in the vmlinuz has a couple benefits. First it will always be available. There is no need to figure out how to load the driver and firmware from initrd, let alone getting the initrd generator one is using, to cooperate. A downside is it makes the kernel very specific to the machine, because perhaps a different Intel machine needs a different firmware file compiled in.

To achieve including the firmware in kernel, I set the following values in my kernel config (.config in your kernel source tree).

CONFIG_EXTRA_FIRMWARE="i915/kbl_dmc_ver1_04.bin"
CONFIG_EXTRA_FIRMWARE_DIR="/lib/firmware"

Note, if you’re using menuconfig, you can type /EXTRA_FIRMWARE (slash for search, then the text) followed by keyboard return to find where these settings exist in the menu system.

Then I verified i915 is indeed not a kernel module, but built into the kernel image (it would be m if it’s a module):

CONFIG_DRM_I915=y

After compiling & installing the kernel (and generating a dracut initrd for cryptsetup/lvm), I was able to reboot and get an early pre-mounted-root framebuffer on this device.

I discovered the Gentoo devs have begun shipping an ebuild that builds and installs a kernel with a portable, livecd friendly config. In addition this package will optionally generates an initrd with dracut as a pkg_postinst step, making it very suitable as a replacement for users who just want a working kernel, and don’t mind a excessive compatibility (at a cost to size and build time).

This presents a different challenge, because while this package does allow the user to drop in their own .config, it is not very multiple-machine-deployment friendly to hard-code each individual firmware into the kernel. Instead we tell dracut to include our framebuffer driver. As mentioned above I found this computer uses the i915 kernel driver for framebuffer. Let’s tell dracut to include the driver:

cat > /etc/dracut.conf.d/i915.conf <<EOF
add_drivers+=" i915 "
EOF

Dracut is smart enough to pick up the firmware the kernel module needs, provided they are installed. To get an idea what firmware dracut will include, run modinfo i915 -F firmware which will print out a bunch of firmware relative paths.

After applying this fix, just regenerate your initrd using dracut; in my case I let portage do the work: emerge -1av sys-kernel/vanilla-kernel. Finally reboot.

Check dmesg. Always check dmesg. We found two ways to deploy firmware, in-kernel and in-initrd. The in-kernel technique is best for a device-specific kernel, the in-initrd is best for a portable kernel. I am a big fan of the second technique because it scales well to many machines.

I did not touch on the political side of using binary blobs. It would be nice to not use any non-free software, but I rather have a working system with a couple small non-free components, than a non-working system. Which is more valuable, your freedom, or reduced capacity of your tools?

I should mention unison is a superb tool for synchronizing your data. It shows the user a list of changes to each directory being synchronized, waits for the user to decide which way each file should be synchronized:

1. Send file from host A to B
2. Send file from host B to A
3. Ignore the file this time
4. Ignore the file permanently
5. Merge the files

Because unison doesn’t try to be fancy or automatic, it is easy to understand what is happening.

Here are the steps I used to get started with Blink, as it was relatively unclear where to find documentation and which settings are necessary to configure. Usage information can be found via the help command, the README on GitHub⁷, and fiddling with the UI.

1. Run the config command.
   1. Default User -> set to your preferred username on most of your servers
   2. Add a new font via: Appearance -> Add a new font. Open the gallery and grab the raw CSS file (you can get a raw URL after switching to GitHub desktop), then paste it into the URL Address Field. I prefer Droid Sans Mono for its readability at all sizes.

   3. Keys -> + -> Create New

      Please use ssh keypair authentication. Password authentication requires you to memorize a password to log in which can be bruteforced or otherwise leaked when typing it in the wrong context. Also, be sure to use a unique key on each of your devices to make revocation easier when a device is no longer used.

      Note: it appears keys are always stored as plaintext, which is acceptable for your uses, but it appears ssh-keygen can create a key with a passphrase. I wasn't able to get Blink to work with Passphrase protected keys, but I didn't try very hard. YMMV.

      • Type: RSA
      • Bits: 4096 (why not?)
      • Name: id_rsa
   4. Hosts -> +
      • Host: the name you gave your host when running ssh host or mosh host. This is not related to the server's hostname, though it can be the same. I prefer simple names, usually the hostname before the first dot (e.g. worf.winny.tech becomes simply worf)
      • User: make sure this is correct
      • Key: select a key to use.
2. Now you need to install the SSH public key:
   1. Run config again
   2. Keys -> id_rsa -> Copy Public Key

   3. Install it some way.

      I usually prefer to visit https://ptpb.pw/f and paste the public key, hit Paste, then copy the url with the text of "created". You can then curl https://ptpb.pw/PasteId on the server via another login setup (or ssh from a set up machine) and add it to your ~/.ssh/authorized_keys. Make sure that file is mode 0600 (chmod 0600 ~/.ssh/authorized_keys). Also make sure the .ssh directory is 0700 (chmod 0700 ~/.ssh). OpenSSH refuses to use authorized_keys if these permissions are world readable.

3. Run ssh worf. Congratuations, you now have a SSH session on the best iOS client available at this time.
4. To use mosh, ensure mosh is installed on the server.⁸ Then run mosh worf.

After reviewing a list of org-mode¹ capable static website generators², I decided to see if org-static-blog³ could suffice my simple needs. My criteria for choosing an org-mode static site generator was:

• it must be actively maintained,
• it must be simple to set up with customizations,
• and it must work with Emacs 26 and later.

This ruled out quite a few right away. I didn't attempt using org-publish, as it looked like a great deal of configuration to achieve a minimum viable web-page for this project.

Following the org-static-blog README documentation, it is very straight forward to get a minimal viable website generated. I added the following to my init.el:

(add-to-list 'auto-mode-alist (cons (concat org-static-blog-posts-directory ".*\\.org\\'") 'org-static-blog-mode))
(setq org-static-blog-publish-title "blog.winny.tech")
(setq org-static-blog-publish-url "https://blog.winny.tech/")
(setq org-static-blog-publish-directory "~/projects/blog/")
(setq org-static-blog-posts-directory "~/projects/blog/posts/")
(setq org-static-blog-drafts-directory "~/projects/blog/drafts/")
(setq org-static-blog-enable-tags t)

(setq org-static-blog-page-header
      "
<link href=\"static/style.css\" rel=\"stylesheet\" type=\"text/css\" />
")

I opted to setq all the configuration variables; I will likely switch to using M-x customize-group RET org-static-blog RET in the future however. It's a better experience.

Then I simply create a new post with M-x org-static-blog-create-new-post RET <TITLE> RET, edit the buffer, save it, then run M-x org-static-blog-publish RET. I also added some styling to my style.css⁴ based off the Tachyons CSS framework⁵. I had previously used Bootstrap⁶ for styling but was hoping to avoid adding framework and other extra tooling that shouldn't be necessary to generate a simple site like this one.

I am a fan of the Caddy⁷ web server which offers automatic HTTPS via LetsEncrypt with only a few lines of configuration. In addition Caddy has a plugin named git which offers the ability to automatically deploy content from git repositories with webhook support. To deploy the following steps are taken:

1. M-x org-static-blog-publish RET from Emacs to regenerate the static site,
2. commit the changes in git,
3. and finally push the git branch to GitHub.

After these steps, GitHub automatically sends a HTTP POST request to my Caddy server with information about the new git commits and Caddy pulls the git repository. If everything went well and the webhook successfully fired, the website is now deployed.

With this simple setup I can write posts. I can easily move the configuration to a new host at will. In addition my setup does not depend on future use of GitHub as GitLab, Gogs, and other git hosts offer webhook support in the same way. Most importantly, I can author org-mode files and have a better balance between features and ease of use than what markdown offers.

Web dev is one of my least favorite programming exercises. Between all the testing necessary to ensure a simple site works across many platforms, the trend to use a very complex system such as webpack and many other tools to produce simple websites, and the perpetual flux-and-flow between vendors only partially implementing good features, web dev just doesn't do it for me. Hence, I am very pleased how simple this project turned out to be.