Sillypaste migrated to fly.io
I’ve been operating Sillypaste (source code) - a simple Django pastebin created for dogfooding. In this post I hope to capture some of the painpoints of working with Django, Python, Heroku, and the migration to fly.io.
Ever since Heroku sold its soul to SalesForce its been on the decline.
Customer service is worse than ever, giving wonderful canned responses to
most questions. It used to be free to host Sillypaste on heroku, now is a
$17/mo ordeal. See the screenshot for a breakdown. Notice how the “free
scheduler”1 is not even free anymore. They also dropped the ball when
performing the pricing structure change: everyone and myself misunderstood if
or when our databases would be nuked if we didn’t take any action. I opened a
ticket about this and got another canned response from their support staff.
Thanks heroku. The last straw that broke the camel’s back was how difficult it was
to upgrade Python on Heroku. In the end I never did figure out how to upgrade
my Django application to a newer Herok buildpack. I had all sorts of problems,
most I have blanked out my brain because it was such a negative experience.
Something to do with EOL buildpacks and trying to upgrade a
From the above criticisms I see three primary issues with Python on Heroku:
- Upgrading to a new buildpack is very difficult to test - you just keep pushing over and over again to test deploys.
- Python setuptools and requirements.txt is kind of annoying to work with, but the buildpack doesn’t seem to work with some of the better package tools like Poetry.
- Heroku is now not cost effective. You can pay for a a couple budget VPSes and gain the same reliability, and still save money.
As the article title suggests, I decided to cut my losses and migrate from Heroku to fly.io. If I understand correctly, fly.io will cost me about 0-5/month for the same services. I will update this article with my findings.
§Investigation #1: Ship the django applicatinon using a Nix flake
I was curious if i could ship Sillypaste as nix Flake. It’s probably possible, but I didn’t have good results. Here’s what I tried:
- I looked into mach-nix, however the codebase is unmaintained, so I didn’t want to adopt it.
- I looked into the mach-nix successor dream2nix, however it choked on my requirements.txt entry that pulls in package from a git repository URL due to sandboxing.
- I noticed poetry2nix, but at this point I decided it was probably just another distraction from my end goal of migrating Sillypaste, given I had invested half a day on this research. I might investigate this later. If I do, I’ll edit this blog post.
Take away: nix might work, but it’s still a big time sink. Not a good use of my time for this specific project phase.
§Investigation #2: Ship the django application using a Dockerfile
One lesson I learned from the above nix foray is that many build tools work
better if you can install your Python package like any other package in your
requirements.txt. To achieve this I restructured the Sillypaste codebase
into a single top level package
sillypaste then updated various references in
the codebase. I followed this Django tutorial on package-izing your application.
After doing this, I still kept running into issues with modifying
requirements.txt (such as the case of removing Heroku support). I called it,
requirements.txt bankruptcy, and migrated the project over to Poetry. As a
nice bonus, Poetry also has a simple way to declare scripts and other
pyproject.toml2 Another nice benefit is Poetry can
set up virtualenvs for you. I think the best feature of Poetry is its lockfile
that separates specific version data from manually-specified dependencies.
So there I was, able to run poetry install, then run
sillypaste serve to run
the Django application in gunicorn. Pretty cool. For the curious here is the
pyproject.toml (c.f. the
poetry.lock in the same directory).
Next up was to write a Dockefile that installs my project as a proper python package. It is essentially the following - based off of this StackOverflow Answer:
FROM python:3.10-alpine RUN pip install poetry RUN apk add --no-cache libpq-dev build-base WORKDIR /src COPY . . RUN poetry config virtualenvs.create false \ && poetry install $(test "$YOUR_ENV" == production && echo "--no-dev") --no-interaction --no-ansi COPY crontab /etc/crontabs/root WORKDIR /app EXPOSE 8000 CMD crond && sillypaste serve
Note the Dockerfile also runs a cron daemon. This is for running
sillypaste expire on a cadence. This is how expired pastes and temporary user sessions are removed.
This isn’t a well optimized Dockerfile, however it works and I don’t think I’ll touch it again until I decide to slim it down. I also wrote a docker-compose.yml that does the bare minimum.
§Investigation #3: Deploying on fly.io and migrating from Heroku
So now I have the building blocks to deploy this application on fly.io. I have a Dockerfile that works well locally and seems to work on fly.io.
§Setting up the database
I created the user and database as follows:
CREATE USER sillypaste WITH PASSWORD 'some top secret password'; CREATE DATABASE sillypaste WITH OWNER sillypaste;
Once I got the application running at
sillypaste.fly.dev, I did a quick
database sync to test it. Here’s the command I used from the fly.io documentation:
pg_dump -Fc --no-acl --no-owner -d $HEROKU_DATABASE_URL | pg_restore --verbose --clean --no-acl --no-owner -d $DATABASE_URL
§And a little testing…
One weirdism I noticed was crond wouldn’t fire in some cases. This confused me to no end. So when I noticed that reducing the cronjob’s frequency to every minute, it seemed to fire reliably, I was like, that’s good enough. If it costs CPU time, I’ll be able to measure it and adjust if it’s incurring a service cost. (I’m wrote this paragraph as a reminder to myself to fix this weird behavior.)
Now the application has live data and it seems the temporary user pruning/paste expiration runs reliably, I’m confident fly.io will work better than Heroku.
§Performing the migration
To do the actual migration, all I had to do was scale back my Heroku to no dynamos, so nobody could make further database modifications via the web frontend:
heroku ps:scale web=0
Next I ran the
pg_dump ... | pg_restore ... command once more to ensure the
new database has the most recent data from the old production database.
Next I set up fly.io to use
paste.winny.tech instead of
sillypaste.fly.dev. I chose to set up A and AAAA records on
paste.winny.tech then issue a
fly certs create paste.winny.tech. I had to
fly certs check paste.winny.tech due to some slow DNS propgatation
delays. Otherwise it worked perfectly and I was now serving content at
§Redirecting traffic from Heroku to fly.io
I found an easy-to-use Heroku “Button” named
heroku-redirect (I think it’s
like a pre-packaged app) to redirect all traffic to a canonical URL.
I deployed it by:
- Cloning the git repository somewhere
- Add the current git repo to the existing sillypaste heroku app:
heroku git:remote -a sillypaste. This will be used to push and also easily configure the application without specifying
heroku config:add NEW_BASE_URL=http://paste.winny.techto configure the application to use this URL.
- Change the buildpack because yuck - heroku has all this configuration that isn’t in
your git repo:
heroku buildpacks:set heroku/nodejs
- Overwrite the existing deployment and git repository:
git push -f heroku master
If you reduced the scale of the workers using
heroku ps:scale, be sure to set
the workers to more than 0:
heroku ps:scale web=1.
§Cleaning up - saving costs
It seems heroku will happily push you to use basic dynamos instead of eco
dynamos when you change the scale from 0 to 1. This costs twice as much and
you probably don’t want it for a hobbyist website. Run
heroku ps:type eco to
Finally make sure to clean up your “free” scheduler and postgresql database, if any. It’s probably to do it from the CLI, but I just visited the dashboard and clicked delete a bunch.
There were some weird weirdisms going on with my Django application. Some
DEBUG to a boolean expression that evaluated to
not actually disable
DEBUG mode. Setting it manually to False fixed it.
Bizarre (and yes there was no
DJANGO_DEBUG envvar configured or set).
Another unrelated thing: if you find the fly.io remote builders slow, you can
fly deploy --local-only - this’ll build the docker image locally
then push to your registry.
Adopting poetry was a good move. Dumping heroku in the nearest trash receptacle was also a good move. I think it should be easier to operate Sillypaste in the future since it uses a Dockerfile - no need to deal with heroku’s weird buildpacks. I won’t miss heroku.