Sillypaste migrated to fly.io

I’ve been operating Sillypaste (source code) - a simple Django pastebin created for dogfooding. In this post I hope to capture some of the painpoints of working with Django, Python, Heroku, and the migration to fly.io.

Ever since Heroku sold its soul to SalesForce its been on the decline. Customer service is worse than ever, giving wonderful canned responses to most questions. It used to be free to host Sillypaste on heroku, now is a $17/mo ordeal. See the screenshot for a breakdown. Notice how the “free scheduler”¹ is not even free anymore. They also dropped the ball when performing the pricing structure change: everyone and myself misunderstood if or when our databases would be nuked if we didn’t take any action. I opened a ticket about this and got another canned response from their support staff. Thanks heroku. The last straw that broke the camel’s back was how difficult it was to upgrade Python on Heroku. In the end I never did figure out how to upgrade my Django application to a newer Herok buildpack. I had all sorts of problems, most I have blanked out my brain because it was such a negative experience. Something to do with EOL buildpacks and trying to upgrade a requirements.txt.

From the above criticisms I see three primary issues with Python on Heroku:

1. Upgrading to a new buildpack is very difficult to test - you just keep pushing over and over again to test deploys.
2. Python setuptools and requirements.txt is kind of annoying to work with, but the buildpack doesn’t seem to work with some of the better package tools like Poetry.
3. Heroku is now not cost effective. You can pay for a a couple budget VPSes and gain the same reliability, and still save money.

As the article title suggests, I decided to cut my losses and migrate from Heroku to fly.io. If I understand correctly, fly.io will cost me about 0-5/month for the same services. I will update this article with my findings.

§Investigation #1: Ship the django applicatinon using a Nix flake

I was curious if i could ship Sillypaste as nix Flake. It’s probably possible, but I didn’t have good results. Here’s what I tried:

1. I looked into mach-nix, however the codebase is unmaintained, so I didn’t want to adopt it.
2. I looked into the mach-nix successor dream2nix, however it choked on my requirements.txt entry that pulls in package from a git repository URL due to sandboxing.
3. I noticed poetry2nix, but at this point I decided it was probably just another distraction from my end goal of migrating Sillypaste, given I had invested half a day on this research. I might investigate this later. If I do, I’ll edit this blog post.

Take away: nix might work, but it’s still a big time sink. Not a good use of my time for this specific project phase.

§Investigation #2: Ship the django application using a Dockerfile

One lesson I learned from the above nix foray is that many build tools work better if you can install your Python package like any other package in your requirements.txt. To achieve this I restructured the Sillypaste codebase into a single top level package sillypaste then updated various references in the codebase. I followed this Django tutorial on package-izing your application.

After doing this, I still kept running into issues with modifying requirements.txt (such as the case of removing Heroku support). I called it, requirements.txt bankruptcy, and migrated the project over to Poetry. As a nice bonus, Poetry also has a simple way to declare scripts and other configurations in pyproject.toml² Another nice benefit is Poetry can set up virtualenvs for you. I think the best feature of Poetry is its lockfile that separates specific version data from manually-specified dependencies.

So there I was, able to run poetry install, then run sillypaste serve to run the Django application in gunicorn. Pretty cool. For the curious here is the pyproject.toml (c.f. the poetry.lock in the same directory).

Next up was to write a Dockefile that installs my project as a proper python package. It is essentially the following - based off of this StackOverflow Answer:

FROM python:3.10-alpine

RUN pip install poetry
RUN apk add --no-cache libpq-dev build-base

WORKDIR /src
COPY . .
RUN poetry config virtualenvs.create false \
  && poetry install $(test "$YOUR_ENV" == production && echo "--no-dev") --no-interaction --no-ansi
COPY crontab /etc/crontabs/root

WORKDIR /app
EXPOSE 8000

CMD crond && sillypaste serve

Note the Dockerfile also runs a cron daemon. This is for running sillypaste expire on a cadence. This is how expired pastes and temporary user sessions are removed.

This isn’t a well optimized Dockerfile, however it works and I don’t think I’ll touch it again until I decide to slim it down. I also wrote a docker-compose.yml that does the bare minimum.

§Investigation #3: Deploying on fly.io and migrating from Heroku

So now I have the building blocks to deploy this application on fly.io. I have a Dockerfile that works well locally and seems to work on fly.io.

§Setting up the database

I created the user and database as follows:

CREATE USER sillypaste WITH PASSWORD 'some top secret password';
CREATE DATABASE sillypaste WITH OWNER sillypaste;

Once I got the application running at sillypaste.fly.dev, I did a quick database sync to test it. Here’s the command I used from the fly.io documentation:

pg_dump -Fc --no-acl --no-owner -d $HEROKU_DATABASE_URL |
    pg_restore --verbose --clean --no-acl --no-owner -d $DATABASE_URL

§And a little testing…

One weirdism I noticed was crond wouldn’t fire in some cases. This confused me to no end. So when I noticed that reducing the cronjob’s frequency to every minute, it seemed to fire reliably, I was like, that’s good enough. If it costs CPU time, I’ll be able to measure it and adjust if it’s incurring a service cost. (I’m wrote this paragraph as a reminder to myself to fix this weird behavior.)

Now the application has live data and it seems the temporary user pruning/paste expiration runs reliably, I’m confident fly.io will work better than Heroku.

§Performing the migration

To do the actual migration, all I had to do was scale back my Heroku to no dynamos, so nobody could make further database modifications via the web frontend:

heroku ps:scale web=0

Next I ran the pg_dump ... | pg_restore ... command once more to ensure the new database has the most recent data from the old production database.

Next I set up fly.io to use paste.winny.tech instead of sillypaste.fly.dev. I chose to set up A and AAAA records on paste.winny.tech then issue a fly certs create paste.winny.tech. I had to issue a fly certs check paste.winny.tech due to some slow DNS propgatation delays. Otherwise it worked perfectly and I was now serving content at https://paste.winny.tech/ .

§Redirecting traffic from Heroku to fly.io

I found an easy-to-use Heroku “Button” named heroku-redirect (I think it’s like a pre-packaged app) to redirect all traffic to a canonical URL.

I deployed it by:

1. Cloning the git repository somewhere
2. Add the current git repo to the existing sillypaste heroku app: heroku git:remote -a sillypaste. This will be used to push and also easily configure the application without specifying -a sillypaste again.
3. Run heroku config:add NEW_BASE_URL=http://paste.winny.tech to configure the application to use this URL.
4. Change the buildpack because yuck - heroku has all this configuration that isn’t in your git repo: heroku buildpacks:set heroku/nodejs
5. Overwrite the existing deployment and git repository: git push -f heroku master

If you reduced the scale of the workers using heroku ps:scale, be sure to set the workers to more than 0: heroku ps:scale web=1.

§Cleaning up - saving costs

It seems heroku will happily push you to use basic dynamos instead of eco dynamos when you change the scale from 0 to 1. This costs twice as much and you probably don’t want it for a hobbyist website. Run heroku ps:type eco to fix this.

Finally make sure to clean up your “free” scheduler and postgresql database, if any. It’s probably to do it from the CLI, but I just visited the dashboard and clicked delete a bunch.

§Other remarks

There were some weird weirdisms going on with my Django application. Some reason setting DEBUG to a boolean expression that evaluated to False did not actually disable DEBUG mode. Setting it manually to False fixed it. Bizarre (and yes there was no DJANGO_DEBUG envvar configured or set).

Another unrelated thing: if you find the fly.io remote builders slow, you can also run fly deploy --local-only - this’ll build the docker image locally then push to your registry.

Adopting poetry was a good move. Dumping heroku in the nearest trash receptacle was also a good move. I think it should be easier to operate Sillypaste in the future since it uses a Dockerfile - no need to deal with heroku’s weird buildpacks. I won’t miss heroku.