Moving Off Haskell (Part 3): Choosing a Tech Stack

I was going to start posting about this in Plan for Moving Off Haskell (Part 2): Technical Details, but I think the discussion here may get long / detailed in a way that will distract from that thread’s current focus on what changes need to happen & in what order. So here we are.

I would like the tech stack we pick to meet these criteria:

  1. Works (aka Correctness): This should go without saying.

  2. FLO: Building our platform on a proprietary technology is a non-starter, unless there is no usable FLO alternative (e.g. our current use of proprietary javascript from Stripe on the single page where we accept credit cards).

  3. In-team Expertise: Our platform must be maintainable by the core team members. It’s why we’re moving off Haskell in the first place.

  4. Boring: We already have a novel mechanism and governance structure. We don’t also need to break new ground with our tech stack.

    a. Not Too Boring! Boring technology is, well, boring. Probably not something volunteers would be excited to work on in their free time. Which is important to us as a 100% volunteer project.

  5. Convenient / Approachable: We are currently all volunteers, and often have limited time. If contributing is too much of a time suck, it’s unlikely to happen. This goes double for people outside the core team; it should be relatively easy for them to contribute.

The last two points suggest that something Mainstream is desirable.

For the frontend, we’ve already chosen Elm. It’s neither Boring nor Mainstream, but @Adroit’s enthusiasm for it brings enough momentum to trump those considerations, and it is either great or good-enough on the other criteria. Plus, if for some reason we needed to migrate away, it’s relatively easy to “eject” the static content back to vanilla html + css.

For the backend, I strongly suggest TypeScript (read: Unless someone objects, I consider this decided). It keeps us on the web frontend stack (html/css/js), and is the language I am most experienced with. It is less approachable than vanilla JavaScript, but writing applications that Work in vanilla JS is painful; I think that trade-off is worth it. This implies NodeJS, as the only Boring JS runtime.[1]

Beyond that, we have decisions to make.


  1. I know the blog I linked above cites Node as not Boring, but that was in 2015. Today, insufficiently mature alternatives are Deno or Bun. ↩︎

Hey @Salt, I had an idea at today’s meeting. What do you think about using civi as our auth backend?
Or, alternatively, using a CMS like payload and also replacing civi with it?

Moving here from the other thread:

I also ran into GitHub - curveball/a12n-server: An open source lightweight OAuth2 server today, which looks like pretty much what I’d want for auth… except that it’s not particularly well documented, and probably not Boring enough. It’s part of https://curveballjs.org/, which looks nice, but with the same caveat.

The most-boring option for a node api, would probably be https://expressjs.com/ + https://www.passportjs.org/, although Express and Passport are unopinionated enough that they are also not a complete stack here; they don’t come with e.g. a password reset flow built in.

This blog post on the password reset flow is stack-agnostic., concise and informative,

Their (non-boring) open source auth product might also be worth checking out.

I’ve been digging into authentication, taking in node/ts basics along the way. I’ve started with express-generator-typescript (the plain javascript version is actually embedded in npm I believe), which provides a nice skeleton for getting started.

As it happens, this package provides it’s own, ostensibly “production quality” client-side security + session support, utilizing jsonwebtoken for a session mechanism using signed cookies (jwt) and bcrypt for password hashing. As it’s merely a scaffold, it has no signup or reset mechanism, and uses an orm mock reading from a json file as the database.

However, things like github login, which with a passportjs + express-session combo are trivial to provide, would also need custom implementations, and I guess we needn’t reinvent the wheel…

Anyway, just surveying the landscape out loud. I’ll be around this evening for the meeting.

I looked into SuperTokens a little, and it seems like it is basically what I was originally looking for— a piece with only the auth components (password storage, password reset flow, updating email/password…) that we could plug in to our site. However, I do think it’s not quite mature enough to go with. In particular, as far as I can tell, it has no way for users to change their email. That’s a major downside, especially considering snowdrift.coop currently lacks that ability, so adding it is a major draw for switching to some off-the-shelf software.

That said, I’m glad you mentioned it, because their comparison with KeyCloak was helpful in understanding that KeyCloack is basically a more boring/mature version of the same thing. The main downsides I noticed are:

  • Somewhat complicated setup / bad official docs. It is mature enough that we can probably find 3rd party tutorials to get everything set up, but it’ll require some time investment and hair-pulling to figure out how to configure everything the way we want.
  • New addition to our tech stack: Java. This is not as big of a downside as it might be, because OSUOSL will handle deployment, so we don’t also need to get expertise in how to deploy java apps. However, any customization of the login interface (e.g. to use Snowdrift.coop branding) would need to be done in Java (I think), so it is still a significant addition to our tech stack footprint.

Overall my impression is that KeyCloak would be an acceptable solution for us to use, but not a first choice.

Small correction: You don’t need to write any Java code if all you want to do is change how the login screen / registration screen / account console / emails look. You mostly edit html, css and config files (see “creating a theme” in the docs and the source code of one of the bundled themes). You only need to write Java code if you want to change the behavior of Keycloak beyond what is configurable, e.g. if you want it to call a webhook whenever a user registers.

If you’re talking about the keycloak admin UI: There’s a lot of options, but you only need to touch some of them unless you want to do something special. You’d basically create a realm and within that realm, create a client. For both of these, you’d go through their config options and see what you need to change. Most attributes have a questionmark icon with a short description on click.

I think the most effort would be writing the custom theme, plus of course replacing the current authentication with an oauth2 client [edit 1: and importing the current users, see below]. (Assuming that OSUOSL handles deploying the instance + helps to set up a deployment pipeline for the custom theme)

An interesting question would also be what to do with the current users that are already in the database… It might be possible to configure the hashing algorithm etc sufficiently so that you can use the existing password hashes, but maybe that would take some luck :slight_smile:

(probably became irrelevant due to edit 2)

Can you maybe create a few test users in a local instance and publish their passwords and hashes? (plus salts and algorithm if stored separately, or anything else that seems relevant) Then someone could try to get them into an instance of e.g. keycloak as a fairly well-defined task. (It’s a bit hard for me to run an instance of the snowdrift.coop site locally, last time it didn’t work because the stripe dependency needs too much RAM to compile.) I’m not promising I will do this, but maybe if I’m motivated to play around with it, I might try and report the results

Edit 2: Passwords are currently hashed with PBKDF1 (see Plan for Moving Off Haskell (Part 1): Overview). Keycloak by default only supports PBKDF2. It is possible to add a custom hashing algorithm to keycloak by writing java code + this would allow migrating hashes from PBKDF1 to PBKDF2 on login. (See Keycloak password hashing alghoritm md5 - Stack Overflow)

Last week Two weeks ago, we discussed using Payload, which I think is our most likely option for the backend framework at this point.

Cons:

  • Requires mongodb
    • I lack experience with document database schema design compared to a “traditional” RDBMS.
    • I think much of the crowdmatching financial data actually is a good fit for a tabular data model.
  • Not very mature
  • Small degree of lock-in, inherent to using any framework

Pros:

  • Very little lock-in and “magic” relative to other frameworks and CMSes.
    • Relatively easy to “eject”— drop the framework and use access the underlying db directly —if needed.
  • Works well alongside custom code
    • If we reach framework limits, we can build what we need without scrapping everything.
    • Provides an incremental migration path away, if we decided we needed to drop it entirely.
  • We get quite a few things “for free” ⇒ fast feature development and iteration.
    • User accounts
    • Access controls
    • Auto-generated api (REST, GraphQL, and local nodejs)
    • Admin panel interface for us to manage data
      • Maybe also useful for projects to configure their project page and other things we might want to allow them to customize later.

The first 2 pros are why I don’t consider the cons to be deal-breakers. Still, I was hesitant at first, and I still have some gut reservations. And if I were only considering the migration off of Haskell, I would probably still be inclined against Payload.

However, thinking about the next steps on the road, supporting multiple projects and refining our crowdmatching mechanism, I’ve come around to the idea that a document db with a flexible schema is actually the right tool for the job.

I’m still open to being convinced that we ought to stick with postgres (or even use both?), though :slight_smile: (I recognize that I’m blurring the lines between Payload and MongoDB here, but my strongest reservations about Payload come from it using Mongo).

More Cons:

  • MongoDb moved to a questionable license, Payload happens to use it
    • License change does not really affect our usage, but the consensus is that it’s now non-FLO
    • Hopefully it’s only a matter of time before there’s a compatible fork, or swappable db support
  • Compared to big old CMSs like WordPress, there’s very few plugins yet

More Pros:

  • Backed by an active company, that has financially supported itself working on Payload before voluntarily making it FLO
  • Issues are tackled at an incredible pace, as smichel noted
  • We’d be using a setup that many orgs bigger than us are actually using, right now, in production
  • Designed for scalability, probably a non-issue even in the world-changing success story
  • Designed to be as nice as possible for developers to work with

About a month ago, we were discussing this on Matrix, and @davidak brought up a good point: using technology that is very well established/boring can make contributing to Snowdrift feel more like day-job programming (since corporations tend to be risk averse and choose boring technology). From a technical-social perspective, technology that is a little more fresh and exciting to work with can be a draw for potential contributors; we should balance that against tech-stack stability.

I wanted to call it out here because, while I initially missed it in the bullet points above (now edited in), it is a criteria that I have already been considering. In particular, Elm would not be my normal first choice for a frontend.

  • Technically, I think it would be simplest and best to stay with a monolithic server-side rendered site, not a Single Page App (SPA); it’s mostly static content and SPAs are best for highly interactive sites.
  • Aside from build times (which would be fixed if we had the expertise to update the Haskell site dependencies to a more recent Stackage snapshot), I think our current frontend issues are mostly due to complex CSS; I think the main benefit of the rewrite is switching our CSS to Tailwind.

And yet, I supported the rewrite… because @Adroit was excited about it. And that turned out to be probably the best decision I’ve made. Besides directly working on the frontend rewrite, @Adroit has also prompted me to join him for coworking, at times when I otherwise wouldn’t have felt like it. It helped us get out the the holding pattern we were in for 5 years.

Would that have happened if I’d insisted on the boring-est tech stack? Hard to say, but I doubt it. I’d make the same decision again in a heartbeat. Especially since the new frontend stack has relatively low lock-in. After all, it is mostly static content; if a SPA turns out to be a limiting choice, we can always take the generated html, stick it into a templating engine, and go from there ; )

You might notice some similarities between the previous few paragraphs and what I wrote earlier about Payload. That’s not a coincidence :wink:. They’re both relatively “exciting” tech that I have some technical reservations about, but they have minimal technical lock-in and I think they’re good social choices, and that’s worth a lot.


Technical side-note on Payload: Since it uses Passport.js for its authentication, and there is an OAuth plugin for Passport, we have the option to use Keycloak and Payload. I need to think more about it, but I might like this option best. Keycloak is very boring/stable and full-featured, and I’d very much like to never worry about an auth migration again.

2 Appreciations

Thanks for the appreciation!

I think the main benefit of the rewrite is switching away from Yesod.

1 Appreciation