18 Nov 2019

Won MSOE x Google Cloud Hackathon

About a fortnight ago (Nov 9th) I went to the MSOE x Google Cloud hackathon.1 There was pizza, soda, and Google Cloud gear. Each group was given a Google AIY Computer Vision kit to assemble, and build a proof of concept around.

The kit contained a Raspberry Pi Zero W, the Raspberry Pi Camera Add-on, a breakout board to provide simplified pin-outs for a button with an integrated light, an additional LED that mounted next to the camera to indicate if the camera was active, and a piezo buzzer.2 All these components fit into a carefully engineered cardboard box that folded onto itself, held together with adhesive tape. The assembled device was remarkably robust and easy to operate.

Challenges

We were among the first group to finish building the kit. It turned out the software on the included SD card was not exactly what we needed, and the SD card writing software for Windows (Etcher) was a bit unreliable and did not clearly indicate to the user of a successful write. After a second attempt we had bootable SD card.

The system took a couple minutes to boot and resize itself. Mind you, we did not have a Mini-HDMI to HDMI cable, nor a monitor to output the Raspberry Pi display. Thus we had to wait, chat, and eat pizza.

The next challenge was to “pair” the device with a wifi network so one could SSH into it. There is an android app for this, and at first we paired it with a spare android device acting as a hotspot. Unfortunately this configuration did not give us internet access when connected to this wireless access point. We were able to verify the device was working, SSH in, and inspect the images we took via the kit’s included camera.

We moved to a more central location, as the most cognitively demanding part was complete — construction of the device, and ensuring it works. This lead to more networking challenge. We wanted a way to network this device to the internet so we could not only log into it via SSH, but access PyPi from the device, and access StackOverflow from our laptops on the same network. With a little brainstroming we came up with this network topology:

msoe-network-topology.png

Yes, as you can see the path from the internet is (A) public wifi (B) my friend Karl’s android phone (C) his laptop via bluetooth tethering (D) finally a wifi network via his laptop’s built in wifi. We had an intemittent hiccup with nameserver configuration not set up correctly on the host wifi network — as such any DNS would not resolve. A quick tweak to Karl’s network manager settings mitigated this. And like that… we were networked together on a private wifi network complete with internet access.

There were some other techincal issues. Because of our network topology, the round trip time to the internet was very high, occasionally over a second from the pi, and pip has a relatively low timeout when installing stuff. The workaround was to tell pip to calm down, and be patient. I had installed tmux so we could share a session across the table (for pair programming), and the apt man-db triggers took around 5-15 minutes; with the crunch time we had, this felt unacceptable. The other technical issue we had was the fact the raspbian image starts up a lot of unnecessary services by default which eat into the rasperry pi zero w’s very limited memory. This caused pip to crash due to failure to allocate memory. We had to disable lightdm (GUI) and the default vision kit demo. Had this been a device I’d use for more than a couple hours, I’d go through and disable things like GIO services, and other bloat that we never would use.3

Meanwhile our other two group members worked on a proof of concept game. The idea the device comes up with a common word, and the user carries the device around, showing it various text on the wall or on paper. Using a cloud OCR service, it can recognize the words seen by the camera. It then will buzz happily when it is shown the correct word, or buzz sadly if shown the wrong word. Then the process repeats. It’s an “iSpy” with computer vision and words — a word hunt!

Completion

While our PoC was not deployed in time for presentations, we were about five to ten minutes away from setting it up, and demoing it. During the presentations, we found that many of the other groups experienced the same issues with the Google AIY Vision Kit — due to the Raspberry Pi platform, Raspbian, and the way one pairs it to wifi. At least one other group managed to get some non-default code running on their Pi. We were only given around 3-4 hours from start to finish, with a time loss factor due to slow internet speeds to download the initial SD card image, man-db triggers rendering the device unusable for awhile, and dealing with the lack of suitable networking configuration.

Given all these challenges, I think we did very well. As did every other group that participated. We did have competition for 1st place, because some of the other groups had PoC’s (though some did not get the Pi completely working) and others did get the Pi working but did not have a PoC. We were selected as 1st place, and each was given either a Google Home Mini or a Google Cloud hoodie. I went with the hoodie because I don’t want to use Google’s creepy spyware voice assistant. And the modding/reverse engineering community has done very little with this product; nobody has loaded custom firmware on it, for example.

Conclusion

Our issue in seeking out a usable networking topology had be thinking: if I simply had a device with two wifi radios, I could run one as an Acesss Point and the other connected to Public WiFi. This device would then yield this topology, which would be ideal for these sorts of impromptu projects and activities:

dual-wifi.png

In addition, it would be perfect for demonstrations of man in the middle attacks on public wifi, or experimenting with multipath tcp and wifi.

With these sorts of events that start early on a Saturday morning, it’s been useful to agree with a friend to attend the same event. That way both parties are more likely to show up, because it wouldn’t be very personable to cancel last minute. We also had a wonderful team. I later met up with most of the same team for another event (post incoming). Hackathons are a great way to meet new people. I enjoyed this event thoroughly.

See you at the next hackathon :)

Footnotes:

1

There isn’t a link online yet :(. I will update this note when I find a link to the event.

2

Full list of materials here.

3

Stay tuned as I explore why I find distros like Debian not ideal in practice in a future post ☺

Tags: community
06 Nov 2019

GDG Milwaukee 2019 DevFest - We participated!

I attended the GDG Milwaukee 2019 DevFest last Saturday. This was my second hackathon. Around 6-9 teams participated. We coded for six hours, and I learned a lot about team dynamics. We formed a team of eight participants. We encountered a couple significant challenges.

The stack matters

Initially we decided to use Python and the Django framework. This turned out to be a grave error, because picking up Django quickly while staying productive is challenging. This challenge is multiplied by unfamiliarity with MVC/MVP web frameworks.

A couple hours in we sat down and decided Django had to go. We realized two of us had prior experience with Flask. Combined with flaskapi this could be a wonderfully simple way to build a RESTful API backend.

For the frontend we used Create React App. I did not work with it much, but I found it easy to run, deploy, and tweak. I think it was a solid choice.

The hardest part of our stack was integrating our hand-rolled RESTful API into the React.js based frontend. In fact we weren't able to complete this, but we got really close. It was a lot of work to get as far as we did as a team.

Most of the other groups also used Python. Multiple groups used Django, and one group even used Django Rest Framework. They appeared to be facing the same challenge we were having with getting Django to do anything productive in the allotted time. I know at my next hackathon I won't be recommending Django to the uninitiated.

The winning team used Firebase. Every project I've seen done in Firebase was rapidly prototyped, indicating it is extremely suitable for hackathons. I have deep reservations about using a proprietary PaaS, but maybe I can put this concern aside for my next hackathon. :)

Java. At my university and most others in my area, Java is the first language we learn. In some cases it may be the only language one really learns well. In this light, a team member mentioned perhaps we could use Tomcat or some other Java web framework in future. This seems like a superb idea, I hope to explore this in an upcoming project with classmates.

Debian surprised me (again)

A funny experience during the hackathon was discovering a rather surprising patch Debian's virtualenv package ships. On CentOS 7, Gentoo, and possibly anything not Debian derived, running python3 -m virtualenv ./venv will create a Python3 virtual environment. This is not the case on Debian. Instead Debian will always default to installing python2 in the virtual environment. One must pass -p python3 to install python3. Sure seems wonky to me!

Demos never work

We almost had a working demo, but the part that got us was deployment. I spun up a Vultr VPS, installed npm, node, caddy, virtualenv. I got the API backend running, and built the Create React App pages, and tied it all together with a Caddyfile, but it simply wouldn't work. There was too many moving parts, and manual deployment was too tedious to get right within the time frame.

There is something to be said for working in containerized workflows; this would have been a non-issue. Drop in a docker-compose.yml into the project and just run docker-compose up. Next time :)

Teamwork is essential

We had a team of eight members, and it was challenging to find tasks for everybody. Given we had two major components — frontend and backend — and we had quite a few members who needed instruction to get started, it was challenging to give both the coding and instructing enough attention. In future I strive to have more balanced teams so everybody can feel more involved. Perhaps a good rule of thumb is to pair at most one beginner to one intermediate, never more than one.

Something else I think that will help is ensure nobody gets pigeonholed into managing the project; rather, share the responsibility. Project managers are likely not effective in a half-day Hackathon.

Keep morale up. Don't let negativity distract from the team tasks. Redirect negativity into going for a walk, playing video games, or simply taking a break. Make sure to smile.

Conclusion

I had fun at GDG Milwaukee DevFest. Good food, good company. We found our initial choice of Django was not productive in true hackathon spirit. Flask was better for this. Maybe next time we'll consider Firebase. If I had a nickle for every Debian patch that violated my own idea of least surprise, I'd have laundry money. Demos are hard, deployments should be automated or otherwise streamlined. Finally, teamwork is vital. Keep the team small, and make sure everybody has things to do.

See you next year GDG.

Tags: community
01 Nov 2019

Track down basic Emacs bugs & hangs

I ran into a hang today with only ivy enabled and nothing else configured or installed. The behavior was such that after I typed a hostname with a TLD (such as not.existant.com1), then typed C-x d to visit a directory or C-x C-f to find a file, Emacs would hang. My mouse would turn into a pin-wheel. My only recourse was to send the 'quit' command via C-g to cancel the operation. Unfortunately this meant I could not do one of the simplest operations in Emacs - visiting files and directories.

Enter debug-on-quit

After some searching I found the debug-on-quit and debug-on-error variables2, with their corresponding commands toggle-debug-on-quit and toggle-debug-on-error. Enabling debug-on-quit means I can get a backtrace when I type C-g of what was happening before I signaled Emacs to quit the current command. Here is an example backtrace:

Debugger entered--Lisp error: (quit)
  make-network-process(:name "ffap-machine-p" :buffer nil :host "not.existant.com" :service "discard" :nowait nil :tls-parameters nil)
  open-network-stream("ffap-machine-p" nil "not.existant.com" "discard")
  ffap-machine-p("not.existant.com")
  ffap-machine-at-point()
  ffap-guesser()
  ffap-guess-file-name-at-point()
  run-hook-with-args-until-success(ffap-guess-file-name-at-point)
  ivy-thing-at-point()
  ivy--reset-state(#s(ivy-state :prompt "Find file: " :collection read-file-name-internal :predicate nil :require-match confirm-after-completion :initial-input nil :history file-name-history :preselect nil :keymap (keymap (96 lambda (&optional arg) "nil (`nil')" (interactive "p") (if (string= "" ivy-text) (execute-kbd-macro (kbd "M-o b")) (self-insert-command arg))) (C-backspace . counsel-up-directory) (67108991 . counsel-up-directory)) :update-fn nil :sort nil :frame #<frame (Emacs) dummy.txt [Text] /tmp/dummy.txt 0x70a1e20> :window #<window 7 on dummy.txt> :buffer #<buffer dummy.txt> :text nil :action (1 ("o" counsel-find-file-action "default") ("i" #f(compiled-function (x) #<bytecode 0x1b79a11>) "insert") ("w" #f(compiled-function (x) #<bytecode 0x1b79a21>) "copy") ("j" find-file-other-window "other window") ("f" find-file-other-frame "other frame") ("b" counsel-find-file-cd-bookmark-action "cd bookmark") ("x" counsel-find-file-extern "open externally") ("r" counsel-find-file-as-root "open as root") ("R" find-file-read-only "read only") ("k" counsel-find-file-delete "delete") ("c" counsel-find-file-copy "copy file") ("m" counsel-find-file-move "move or rename") ("d" counsel-find-file-mkdir-action "mkdir")) :unwind nil :re-builder nil :matcher counsel--find-file-matcher :dynamic-collection nil :display-transformer-fn ivy-read-file-transformer :directory "/tmp/" :caller counsel-find-file :current nil :def nil :ignore t :multi-action nil :extra-props nil))
  ivy-read("Find file: " read-file-name-internal :matcher counsel--find-file-matcher :initial-input nil :action counsel-find-file-action :preselect nil :require-match confirm-after-completion :history file-name-history :keymap (keymap (96 lambda (&optional arg) "nil (`nil')" (interactive "p") (if (string= "" ivy-text) (execute-kbd-macro (kbd "M-o b")) (self-insert-command arg))) (C-backspace . counsel-up-directory) (67108991 . counsel-up-directory)) :caller counsel-find-file)
  counsel--find-file-1("Find file: " nil counsel-find-file-action counsel-find-file)
  counsel-find-file()
  funcall-interactively(counsel-find-file)
  call-interactively(counsel-find-file nil nil)
  command-execute(counsel-find-file)

If one tries this, they'll also notice every function is a link to the corresponding Emacs lisp or C source function. Pressing RET with point on an item will find its definition. Use C-. and C-, as usual to follow and return from function definitions.

Thanks to this feature, I was able to quickly focus on a "bug" related to the find file at point subsystem as used by ivy, which causes Emacs to ping random hostname-looking addresses under the point. I reported it, maybe a default will get changed; most importantly, it's documented somewhere now.

Don't forget about debug-on-error

debug-on-error is also extremely useful, especially when extending Emacs with your own Elisp code. When I write Elisp, I almost always enabled this when testing. The Elisp language and its libraries are somewhat anachronistic and each component is seemingly tightly coupled with the next, making it hard to understand any particular subsystem. You will write bad Elisp, even if you're a wizard. Everybody writes buggy Elisp. That's why I find an Emacs bug every few months. It's just a very awkward language. Regardless, Emacs is probably the most flexible and powerful editor out there, and in this case, Worse is Better. And don't forget, a good challenge is a fun challenge.

Footnotes:

1

I know I misspelled existent. What's important is it's recognized as a valid fully qualified domain name by Emacs.

2

By the way, you can just read the info pages in Emacs (C-h i). Don't be like me, and read the infopages in the web browser, that's silly. Emacs is the best documetation browser. :)

Tags: emacs productivity
02 Aug 2019

The Danger of fuzzy matching over one's PATH

Awhile back I noticed my personal mnt/ directory, my (empty) personal tmp/ directory, and a few symbolic links disappeared from my home directory. I only noticed because I use unison1 to synchronize my desktop and laptop homedirs. The actual amount of removed directories and symbolic links were staggering, and it costed me five minutes of extra effort to search through the unison UI to ignore files I don’t want to synchronize. Repeat this a few times a day, with the problem occurring at seemingly random intervals, and you’ve wasted minutes out of every day, which adds up to hours every month.

For months I had not figured out what the problem was. By chance I had noticed while using my application launcher, I had accidentally not ran links -g 2 but instead had ran cleanlinks. I wonder to myself what was I running by accident, as I had done this before, but had not thought anything of it, assuming it was a program that would print usage or perform a no-operation by default.

I was wrong.

Turns out cleanlinks searches the current working directory for empty directories and broken symbolic links. Both are useful. For example I keep empty directories in ~/mnt/ to mount sshfs stuff, and I prefer to use ~/tmp/ as a work directory because no system scripts will touch it.3 I had a few broken symbolic links scattered about, from weird git repositories working trees to some stale user-level systemd unit links from my archlinux install.

Making things more interesting, if you run cleanlinks --help, or with any flags, it operates as usual. So it’s a mistake to also do cleanlinks /some/directory/i/want/to/clean. As a part of imake,4 the old X11 ecosystem build tools, cleanlinks will be installed on many systems and it’s not safe to run it lest you enjoy random stuff being messed about with in your current directory.

How did I manage to run cleanlinks so many times? I did not have links installed on the affected machine. And even after I did install it, I forgot to remove cleanlinks from my rofi runcache. So it had a higher precedence to match than links in certain cases. Hence I ran it a few times on accident even after installing links.

Therefore, I strongly recommend one doesn’t fuzzy match over their PATH. Who knows what other nasty tools ship on your system that will lay waste your productivity, or worse, damage your personal files.

Regardless, I have yet to heed my own warning. Maybe I should just use .desktop files, but then again, maybe there exists a cleanlinks.desktop… Ideally, I’ll create a directory of symlinks to programs I want to launch from rofi. Someday :)

About Unison

I should mention unison is a superb tool for synchronizing your data. It shows the user a list of changes to each directory being synchronized, waits for the user to decide which way each file should be synchronized:

  1. Send file from host A to B
  2. Send file from host B to A
  3. Ignore the file this time
  4. Ignore the file permanently
  5. Merge the files

Because unison doesn’t try to be fancy or automatic, it is easy to understand what is happening.

Footnotes:

2

Links 2 is the best web 1.0 browser. It even shows images and different text sizes. Screenshots on this page.

3

/var/tmp/ could also work, but this way I know nobody is gunna mess with my files and I won’t accidentally mess up permissions on sensitive data.

4

imake on freedesktop’s GitLab. See also what packages depend on imake in Arch Linux. I use Gentoo across my laptop and workstation, so it’s necessary to have imake installed.

Tags: computing rant
28 Jul 2019

Open URL in existing Qutebrowser from Emacs Daemon on Gentoo

On my Gentoo desktops, I use Emacs Daemon via sys-emacs/emacs-daemon1 to ensure an Emacs instance is ready to go and always available from boot. This is done via creating a symbolic link like /etc/init.d/emacs.winston to /etc/init.d/emacs which will start Emacs for the given user. See the package README for more details.

A shortcoming of this setup is XDG_RUNTIME_DIR2 is not set, as this is set by my Desktop Session - maybe LightDM or consolekit set this? As a result, when I open a URL from Emacs Daemon, it opens a fresh qutebrowser session, loading the saved default session, and making a mess of my workflow.

One approach to fix this might be to instead run Emacs daemon from my .xsession script, but I rather not supervise daemons at the user level; if I were to consider this, I'd be better off to switch to systemd for user-level services anyway.

The solution I came up with is to add some lines to my init.el to ensure XDG_RUNTIME_DIR is set to the expected value:

(defun winny/ensure-XDG_RUNTIME_DIR ()
  "Ensure XDG_RUNTIME_DIR is set. Used by qutebrowser and other utilities."
  (let ((rd (getenv "XDG_RUNTIME_DIR")))
    (when (or (not rd) (string-empty-p rd))
      (setenv "XDG_RUNTIME_DIR" (format "/run/user/%d" (user-uid))))))

(add-hook 'after-init-hook #'winny/ensure-XDG_RUNTIME_DIR)

A strange emacs-ism: (user-uid) returns float or integer, despite the backing uid_t (on *nix) is guarenteed to be an integer type. I'll just assume this'll never return a float. Please contact me otherwise, I'd love to hear about this.

Footnotes:

Tags: emacs productivity computing gentoo qutebrowser
Other posts

© Winston Weinert (winny) — CC-BY-SA-4.0