Code – Page 2 – Prose Poetry Code

The SPA vacation

I am a “full-stack” developer, which means a lot of things. Specifically for the purposes of this post, if means that I work with every part of a web application, and those are traditionally divided into three layers. First is the front-end or client-side layer: that’s what you see when you visit the site in your browser. That client connects to the back-end or server-side layer to get and store its data and generally accomplish things. Underneath all that, the database keeps track of everything; it tends to be a special snowflake, so we’ll ignore it for the moment.

Client code, as it must run in a browser, uses a mix of three different languages: HTML for markup, CSS for styling, and Javascript for interactivity and communication. That’s already a lot to know, which is why full-stack developers must be more generalist than specialist. (And, incidentally, why I enjoy the field more than a specialized role.) But then you add in the server, which typically uses a fourth language. That can be PHP, as in my current job, or the Python I like to use for my hobby projects, or Ruby, Java, C#, Elixir, or just about anything. Really, it’s hard to find a language that doesn’t have the capability to be used for the serve side.

That includes Javascript, and the past decade has given us the rise of Node, along with a number of web frameworks based on it, such as Express and Nest. The draw for these is simple: if you use the same programming language on the front and back ends, that’s one less thing you need to learn.

JSON the killer

Node is a great thing. I use it on a daily basis, even if I’m not serving web pages with it. But this encroachment of Javascript into the server realm also brought about the rise of the Single Page Application, or SPA. Instead of the traditional web model of the server sending pages and the client (i.e., browser) rendering them, the SPA takes a different approach. Why not let the client do all the work of creating HTML elements?

It’s simple and even ingenious. The server just offers an API, so theoretically anybody could use it, and the client fetches data from that API, creating HTML on the fly based on a response in the form of JSON. (You could also use XML like in the old days, but…no. Let’s not go there.) Perfect separation of concerns, because why should the server care how the data is presented?

And we got some really great tools out of that, along with a whole new way of looking at the web as an application platform. Angular was the first to hit it big, while React and Vue are the top two now. All these share a focus on “reactive” programming, a style of event-driven programming where data changes automatically when its dependencies do. Very simple, fairly elegant, and…

Build me up, buttercup

…And a total mess to build. Angular, React, and Vue all share that problem, and it’s not getting any better. Building a web application using any of these frameworks is black magic, pure and simple. I’ve been doing this for 4 years now, and I still don’t think I’ve even scratched the surface of Webpack configuration. Seriously, I just use Laravel Mix, or copy a Webpack config file from somewhere else, because I’ve got better things to do with my time. Like, you know, writing the code!

The fault lies in the sheer complexity involved, and the fact that modules (one of the most integral parts of any programming environment since around the 1970s) were, in Javascript’s case, kind of bolted on. So we have to assume someone out there is using an old browser that doesn’t support them, or that we’re in an environment where we can’t load them, which means smashing everything into a single file. Oh, and we want to make that file as small as possible, because bandwidth is still an issue. Add in a minification step, then. Want to use that fancy new API Chrome just added? Well, here’s a polyfill for everything else on the market.

There’s a reason why node_modules is a meme. There’s no valid reason why it should be five hundred megabytes for a “starter” React app. Modularization and the borderline insane dependency lists of a web framework have turned the web dev environment into a mess. Not only that, but it’s nearly impossible to get an app built without running multiple processes simultaneously: Webpack, dev server, maybe a CSS postprocessor, and who knows what else. I understand that you do need a lot of tools to cover various use cases, but can’t we think about optimizing the developer’s experience a little, too?

Zero is better than nothing

When I wrote my first web page in 1996, it was easy. You didn’t need any specialized software, just a text editor, a browser, and maybe an FTP client. Sure, the internet has moved on since then, and pages are a lot more complex. They can do so much more that sites I use every day in 2021 would have been unfathomable to anyone 25 years ago, let alone a nerdy 13-year-old.

But does that increased complexity require a proportional increase in development complexity, or can we get back to making quality pages without all the cruft? In other words, can we make modern web applications without a client-side build process?

Until a few days ago, I didn’t think so. Because that was the received wisdom: modern apps are SPAs, and SPAs need a client framework. And that client framework just has to have a build step. Even Preact, the stripped-down cousin to React that boasts of being easy to install, requires a build step for its templates. Tailwind CSS needs one to cut its 1.7 MB of styling into something both manageable and incredibly sleek.

But…what if we didn’t use frameworks? What if we didn’t use JSON? What if the server sent back HTML instead? That’s exactly what I’m doing in the app I’ve been writing at my day job (as an aside, it still feels weird to write that), so what makes the SPA approach superior?

I’m not the only one thinking that. There’s a small but growing minority of web devs pushing for HTML over the wire, which is precisely what it sounds like. There are libraries that take advantage of this, using the abilities of modern Javascript and the browser platform to patch HTML on the fly.

You’re going to have to do the request/response roundtrip anyway, so cutting out the “turn JSON into HTML” step at the end will only save client time. Since that client might be a cheap Chinese Android phone, you want to save its processing power for more important things like actually being usable. AJAX and the DOM let us do some invasive surgery on a page without requiring a full refresh, but still providing for usable URL history. (In other words, we can make the Back button usable, too!) And it’s that much less Javascript for us to write, because we’re not worrying about computed properties or event handlers.

HTML over the wire takes us back to the simpler times of yore, while letting us use the tools we’ve invented in the meantime. From my perspective, that’s great, especially because it means that, in a lot of cases, you can get by with a front-end build process that looks like this:

Add a script tag to your “base” HTML.
That’s it. You’re done!

About as easy as it gets. So that’s why my redesign of Clave is going to use htmx and Alpine instead of a complicated SPA based on Vue. I’ll still have Tailwind for styling, so I don’t get away completely build-less, but you could swap in Bootstrap or Bulma or something and eliminate even that step.

The web is the center of many lives in this fallen time. We should be doing everything in our power to make it easier to contribute, and I think HTML over the wire is one way to do that on the development side. By making it easier to create complex applications, we’ll see more of them.

Most people would go to a spa for a vacation. I’m taking a vacation from the SPA, because the simpler life is so much better.

Forward and to the side

A little over four months ago, I started a new job. My first, in fact, where I wasn’t employed by myself or a family member, where I was a member of a team, not just a lone programmer writing code, running tech support, designing web pages, and handling the books in the meantime. It was a big jump, and I still find myself off balance some days. I wonder when I’m going to be exposed for the impostor I surely must be. I fret about letting everyone down.

Well, those fears are about to get worse.

From the beginning, my boss said I would be “transitioning” to full-time after 90 days. This would be a kind of grace period for me, a chance to show what I was capable of, while minimizing risk for the company. Understandable, from a business perspective, and I was honestly just happy to be hired in the first place, so I wasn’t going to complain.

Now, the grace period is over. The transition is done. Next week will be like starting over, in one sense. In another, it’s like jumping off a cliff, because I’m not going to be the full-stack developer I expected.

I’m going to be the CTO.

When he said that in the call where we discussed it, I think my heart stopped for a second. Sure, as he was quick to point out, a company that’s effectively a startup in size and revenue doesn’t have a lot of “prestige” in its titles. I’m not a C-level executive at Amazon or Microsoft or some other Big Tech corporation. I’ll effectively be running the tech department of a B2B company that…doesn’t really have much but the tech they (we) use and the sales it allows.

But that is a huge shift. It’s a major jump in responsibility. It turns me into not just a developer, but a manager. I had my first strategy meeting today—just an hour-long talk with the CEO-who-hates-that-title about next steps, but still. This is like nothing I’ve ever done. Or even imagined doing, except in my wildest dreams.

For so long, I’ve written about my depression and anxiety, and I lamented the fact that there just doesn’t seem to be anywhere I belong. I felt powerless, silenced by a world that didn’t want to listen to what I had to say. Now, someone does want to hear that. Someone does value my opinion and my perspective. And it’s overwhelming.

I know I’m not executive material. I don’t have an MBA. I never took any classes in business management. I barely understand half the industry-specific terms my boss throws around.

On the other hand, I do know programming. Almost 30 years ago, I wrote my first lines of code. Three decades spent trying to get somebody to see what I had created, to understand why I feel such joy in doing this job well. Now, I’m being thrust into a position where, paradoxically, I may be doing less actual coding.

I should hate that. Management is a running joke in the development community, much like how military non-coms look down on their commanding officers, and the reasons are the same: moving up the chain of command means getting farther away from the action. Oddly, however, I’m okay with it. Oh, I’m well aware that I’m in over my head, but…I am not alone in that. If anything, the only thing I fear now is letting down the team. I don’t want to be the one everything falls on. I don’t want to be the single point of failure. But then I’m grateful that I’m trusted enough to be given that responsibility, and there’s really only one thing I can say.

It’s about time.

Introducing Agena

I’ve been sick this past week. Sinus infections are always bad news, but this one has left me so out of sorts that I did something crazy. Okay, crazier than usual for me. Therefore, I give to you Agena.

What is it?

Agena is a server for the Gemini protocol, written in pure Python with no external dependencies. It supports static and dynamic routing, server-side scripting, virtual hosts, and wildcard SSL certificates while being light on resources and relatively easy to configure. It’s named after the Agena target vehicle, the unmanned rendezvous partner of the Gemini space missions, which was itself named after the star Agena, also known as Beta Centauri.

No, seriously, what is it?

Right. Let’s back up a step or two. First, Gemini. As you know, alt-tech is all the rage right now. If it isn’t where you live, it should be. Now that Google, Facebook, Twitter, and the other big players have shown themselves to be in opposition to basic human rights such as free speech and fair elections, while also exercising dictatorial control of their platforms by banning anyone whose ideology doesn’t perfectly align with that of the global elite, we need a change.

That change has already begun. Parler and Gab are two popular sites that have been attacked ruthlessly by Big Tech and the media for the crime of allowing free expression, while the superior alternative of the fediverse (note the link on this page) offers a truly decentralized option for social media.

But evading censorship isn’t the only reason to look at alt-tech. Some people like it because it’s new, because it’s a wide open space for experimentation, the way the internet was until it became overcommercialized in the last generation. (Wow. The internet has been around for generations now. I feel so old.)

Gemini, then, is one of a number of projects that aims to bring back some of the feel-good feel of old. Some of us still remember the glory days of Gopher, Usenet, and FTP sites, days when you didn’t need to download six megabytes of Javascript just to load a web page. Sure, those old platforms were limited, but that was by necessity; Gemini does it intentionally, replacing the HTTP protocol that underlies what we think of as the Web with a bare-bones alternative focused on content. There’s no CSS, client-side scripting, or even inline hyperlinks! In return, you get blazing speed and austere simplicity.

You get, in other words, something a decent programmer can write in a weekend.

The weekend project

Now, I’m not sure I’d be considered a decent programmer, but I did exactly that. To be fair, I needed a little longer, but that’s due to my own failings. I started on Wednesday, then slept. A lot. I haven’t been awake too much in the past few days, and most of my waking hours have been in the dead of night. The headaches and occasional dizziness make it hard to think straight sometimes.

Altogether, it took me about 8 hours of coding over 4 days to get a fully functional server. I could have finished it in a weekend, if I’d been physically capable. Sunday and Monday were for adding features: virtual hosts and server-side scripting, respectively.

No software project is ever complete. There are always bugs to be fixed, features to be added, and refactors to be, uh, refactored. Agena is no exception. I consider it beta quality (I’ve put it at version 0.4.2), and you probably shouldn’t use it for anything serious yet. That said, I’d like to keep working on it when I have the chance.

If you want to check it out, head to the Gitlab repo, where you can download a copy of the source, read the installation instructions, and all the usual Git goodness. It’s not often that I actually release something on the code side of things. It’s even rarer that it’s something I’m proud of.

But this is that time. I really feel a sense of accomplishment. Considering how down I’ve been the past few days, that’s saying something.

Full stack adventures 1: Meet the stack

I have a lot of different development projects. Most aren’t all that great, and some of the older ones are…well, they’re awful. (For the morbidly curious, I have many of them on GitHub, and my newer ones will start to appear on Gitlab.)

My current one, however, has been an adventure. It’s useful, it makes for a good learning experience, and it has a very wide range of technologies. I’m truly working with a full stack on this one, even if I may not be playing with a full deck.

What it is

The project is called Pixeme, a portmanteau of “picture” and “lexeme” that, in my opinion, captures the essence of what I’m trying to create. Basically, I took the old saying “A picture is worth a thousand words” far too literally.

My goal is to make a full web platform for what I describe in the documentation as a community for visual language learning. To put it simply: users post pictures with simple, descriptive captions. Nothing more than a single word or phrase, though they can add more text later. What separates this from, say, Instagram is that other users (or the same one) can add captions for other languages.

For example, I could post a photo of my stepdad’s dog with the caption “a dog”. Then, someone who speaks French might come along and add “un chien” to it. A Japanese user could then add “犬” to the mix, and so on. Together, we build a kind of cross-language picture dictionary. This can then be used by people who really are trying to learn languages and understand that visual reinforcement helps.

(I have plenty of other ideas, things like audio attachments, larger texts, Anki-style flashcards, adding in support for constructed or artificial languages, and even possible research uses, but those are all for much later. Let me get the site started first.)

All in all, Pixeme checks all the boxes for me. It involves programming and linguistics, two topics which any reader of this blog will know are among my favorites. It’s a long-term project, so it requires focus but gives the satisfaction of completion in return. And maybe I can even find some way to monetize it in the future. Even if I can’t, it’s still valuable experience with a number of different languages, frameworks, and libraries.

The stack

Every web application is layered. There’s no way around it. At the very least, you have the server side and the client side, but most modern apps add in multiple extra layers, forming a stack. Pixeme is no exception. For this project, I chose a stack which, for the most part, reflects my personal preferences while also allowing me to stay on top of current developments in both sides of the web equation.

The server-side framework I chose is Flask. I like Python. Even though I feel the 3.x series was a needless break in compatibility, it’s still one of the few languages I feel comfortable writing. I’ve joked before that I can write Python in my sleep—that’s how much experience I have, and how much it fits my mindset.

Flask calls itself a “micro-framework” for web applications, meaning that it comes with very few bells and whistles, but plenty of extensibility. And I like that, too. It’s a little harder to set up, as you have to track down a number of extensions for things like database connectivity, form validation, authentication, etc., but that’s okay. You only pay for what you use, and there’s nothing hidden.

Apart from Flask extensions, the rest of the back end is pretty standard. SQLAlchemy and PostgreSQL, which is pretty much Python Databases 101. Pytest for testing, because tests are important. (I’ll admit that I haven’t always kept those up to date in past projects!) The Marshmallow library for serialization and validation. And the usual host of minor packages for little things. You can’t get away from those.

The front end is where things get interesting. Pixeme has a REST API because that’s the cool thing to do these days, but also because it really does make things easier even when you don’t want an SPA. And I decided after writing half of one that I don’t want another SPA. It just doesn’t fit the vision I have for this platform. Oh, there will be an app eventually, and that’ll have web and mobile versions, but the site itself needs to be the initial focus.

Flask includes the Jinja template engine; you can use your own, but it’s easier to go with the default. So I’m writing the view layer (Pixeme mostly follows the typical MVC architecture, in case you were wondering) as HTML templates using Jinja. That’s sometimes harder than it looks, one reason why this post is the first in a series. We’ll get into some of the difficulties later, though.

HTML isn’t the only part of the web in 2020, so Pixeme has a whole client-side layer, and this is where my focus on minimalism ramps up. As I said, I gave up on an SPA for the initial version of the site. I personally like Vue over React, so that’s where I started. But then I saw how I was duplicating so much of the server functionality, so I decided to back up and make a more traditional site. Still, reactivity is great, so I turned to Alpine.js to fill that niche. It’s not perfect, but it’s very helpful, and I’ll explain that later in the series, too.

Last, CSS. I know what you’re thinking: Bootstrap. Or maybe Bulma or one of the other alternatives. And those are great. I’ve used them before, and I do enjoy them…when they fit. I wanted to try something different with Pixeme, something more customized, so I chose TailwindCSS instead. It takes more setting up, it doesn’t play perfectly well with Jinja (yet another topic for another post), but it suits me. Once I grasped the concepts behind it, it just made so much sense.

So that’s the Pixeme stack. Python, Flask, and Postgres are the server side; what OS the whole thing will run on is still an open question, but bet on some flavor of Linux. Jinja for templating and views. A front end built with Alpine and Tailwind. “Full stack” is always a juggling act, and this might be my most ambitious attempt yet. Stay tuned to see how it works out.

From the archive: Trireme

(I’m bored, and I’m tired of talking about the Wuhan virus. So let’s delve into my personal code archive. First, of course, we need a prologue, so bear with me.)

For as long as I can remember, I have been interested in recreational mathematics. And that phrasing is entirely literal: I’m 36, and I have memories from my early childhood involving numbers, math problems, and the fascination that the field produced in me. From a young age, I read through math textbooks for fun (algebra at age 4-5, calculus as early as 9), but I was more drawn to the strange and wonderful ways people used numbers. Puzzles involving math were great fun. I read condensed versions of The Phantom Tollbooth and Flatland while my contemporaries were struggling through Dr. Seuss. My aunt had a kind of children’s encyclopedia, where each volume revolved around a different topic; to this day, I have no idea what was in the other 12 or so, because I only ever read the math one.

Naturally, that led me to computers early on, and you could say that my career as a programmer started when I was 8, the day the teacher of my gifted class handed me a black binder, pointed me towards the Apple II in the back of the room, and sent me on my way into the world of 10 PRINT "MICHAEL RULES!"; 20 GOTO 10. I was hooked, and nearly three decades have not dimmed that fire one bit.

But I still have a passion for numbers, for mathematics in the recreational sense. As an adult, I discovered Donald Knuth’s The Art of Computer Programming, the seminal (and unfinished after some 50 years!) text on the underlying mathematics of programming, and that connected the twin loves of my online life.

Honestly, the books aren’t much help for learning how to code. The edition I have uses a positively ancient assembly language for its examples, and it’s easier for me to understand the concepts from the prose descriptions. But the lessons are still usable today…assuming you need them. Or, in my case, want them.

Chapter 4 is my favorite, as it discusses, well, numbers. A significant chunk of the chapter (itself half a book long) is concerned with the representation of numbers, whether in writing or in computer circuitry, and it was here that I rediscovered the spark of my childhood. And that leads me to one of my solo projects from 2019: Trireme.

What in the world?

Trireme is, to put it simply, a simulation of a fictitious CPU that uses non-binary arithmetic. Now, that doesn’t mean it rejects society’s notions of sexuality. Instead, it rejects society’s notion of how a computer should work. You see, all computers in use today deal in binary numbers. Base-2. 1s and 0s. (Sometimes, you’ll hear talk of hexadecimal, but that’s a human conceit: a single hex digit is nothing more than a group of 4 binary bits.)

But it wasn’t always that way. In the early days of computing, binary wasn’t a given. Quite a few computers from the 50s and 60s used decimal arithmetic. That was harder on the designers, and they often cheated by using some kind of binary-coded decimal scheme internally. (Even today’s x86 processors, such as the one you most likely have in your PC, still have instructions for this kind of number, but they’re disabled most of the time.)

Decimal’s fine. It’s what we use in the real world, so putting it in the virtual world isn’t too big a stretch. What I learned from Knuth’s book, then expanded upon in my online research much later, is that some people went for something even stranger. The Soviets, ever ready to be different from the US, had a weird little machine called Setun. It didn’t use binary numbers. It didn’t use decimal. No, its designers chose something called balanced ternary arithmetic: base-3, but instead of using 0, 1, and 2 as digits (like you’d expect), you use 0, 1, and -1. It’s crazy.

And, in my opinion, beautiful.

I’m a big fan of symmetry. To me, it is the largest component of what makes something aesthetically pleasing. Balanced ternary is a symmetric number system, and thus I find it more intrinsically beautiful than computer binary, where negative numbers have to be represented using either a sign bit (which gives you the possibility of a negative zero) or two’s complement arithmetic (where the maximum negative value doesn’t have a positive counterpart).

Eye of the beholder

I first read about Setun in the Knuth book, as he devotes a small section to balanced ternary for the same aesthetic reasons. From there, I learned the rudiments of the number system, how arithmetic works when some digits are negative by their very nature. And I thought little of it for a decade after that.

In 2009, I was bored (are you sensing a theme yet?), and I got one of my borderline-insane ideas. What if we made a balanced ternary computer today? What would it look like? Setun was a stack-based machine; I won’t go into the details here, but suffice to say, I find stack machines ugly and unwieldy. I much prefer load-store architectures similar to those I’ve programmed in the past: AVR, 6502, etc.

So I designed one. Recall that I have no formal training in CPU design or even electronics. I was just a man with a dream, and I got surprisingly far, considering what little I had to work with. I even went to the most fundamental level, designing logic circuits that could detect and operate on electrical signals that came in three levels, including positive and negative voltage.

(You’d be surprised how well it works. Two transistors of the right type can create a balanced ternary signal. A flip-flop—basically a memory cell—takes fewer than ten. A half adder? Not much bigger. With today’s miniaturization, we could do it, and it wouldn’t be too inefficient.)

In the end, however, I’m a programmer, so my main interest lay in the software to emulate this hypothetical architecture. My first attempt, a decade ago, was…not good. Okay, it wasn’t bad, but it could use a lot of work. The code was not organized well. It relied too much on what are now considered suboptimal structures, and it just didn’t do everything I wanted. Still, I called it a partial success, because I proved to myself that it was possible to make a modern-style processor using non-binary numbers, and that I could do it.

Fast forward

Skip ahead another decade, and I read a forum post mentioning Setun, even linking to an article written about a nearly forgotten experiment from behind the Iron Curtain. That hit me at the right time to rekindle the fire. It’s nothing more than coincidence, really. Perfect timing to snipe my mind.

Trireme was born that night. I chose the name because I wanted something that evoked the number 3 (to represent the ternary aspect), and I didn’t think Trident worked. Plus, I’m a big Civilization fanboy; the trireme is an iconic unit for the series, so why not honor it in this way?

With ten more years of experience, I was smarter about both aspects of the project. I understood more about computer architecture, what worked and what didn’t. As well, I’m a better programmer today than I was then, with much more breadth, and a better grasp on how to write code other people could stand to read.

I wrote the code in Modern C++, because I wanted something fast, but also because I really like the language. (I know, I’m weird that way.) It’s a real program, too, complete with a command-line interface, a rough outline of a plugin system, and a few bells and whistles. Is it complete? Not at all. I’d love to do more with it. Given the chance, I’d like to add more to what’s currently a microcontroller; Trireme needs simulated peripherals, an OS, and much, much more. Despite that, I’m proud to say that it’s mine.

The purpose of Trireme, in as much as it has one, is to answer one question: How much of modern computer science relies on computers using binary arithmetic? As it turns out, quite a lot. Many of our common algorithms are tuned to binary, for instance, as our most of the standards upon which our modern world is built. But I think that’s where something like Trireme can come in handy as a pedagogical tool. It’s always good to think outside the box. Studying such an “alien” computer might give us insight into how to improve our own.

If you’d like to check it out, I’ve put the code for Trireme up as a Github repository. It’s free and open source, and I would love to talk to people who are interested in it. Yes, it needs a lot of improvement, but that’s on the back burner as I work on a few more practical projects. Once I have the spare time, I do want to return to it, make it better.

Why? Because I love numbers. And, strange as it may seem, I have rarely had as much fun programming as when I was working on Trireme. Isn’t that enough?

Themis dev diary #6

Themis is done. Not complete, mind you, but done. I haven’t worked much on it over the summer, and there are many reasons for that: writing, relationships, and so on. Code falls behind, and I’ve been focusing more on Liblio, my federated creator platform.

It’s a bit different, owing to its different requirements. For instance, there’s no real threading for comments, and the central metaphor is an announcement rather than a discussion. Really, it’s more of a Tumblr/DeviantArt/etc. replacement than a serious platform for debate. (I’ll open up a public repo once I’m more confident in its state, just so you know.)

But designing and implementing Liblio has taught me a lot about Themis. Mostly, what I’ve learned is that it isn’t scalable. New features are very difficult to add. The front end is in dire need of refactoring. And the server stack just isn’t up to what I want. So that’s why I’ve decided to make a change. Thus, I announce the successor to Themis: D4.

What is D4, you might ask? Well, that’s easy. It’s Themis, but revised and revamped. The name itself alludes to alliteration, something I love: the 4 D’s are Declare, Discuss, Debate, Decentralize. That’s basically the project’s motto; branding and the like will come later.

Note that I am not simply throwing away the Themis code. All of the core functionality will move to D4. It will be a federated platform for discussion with a heavy emphasis on threaded conversation, rich filtering systems, topic-based grouping, and all the good things that come from opening up and having honest debate without the fear of reprisal. It will still speak ActivityPub, so you can (in theory) follow and participate even if you’re on Mastodon, Pleroma, or any of the other myriad fediverse platforms.

In the coming weeks, I hope beat the code into shape for a release. Once that’s done, I’ll go ahead and push the final Themis alpha, which will have a link to the project’s successor. Until then, here’s a bit of a postmortem.

Rationale

The premise behind Themis was sound, in my opinion. It still is. If I didn’t believe that, I wouldn’t be doing the same thing all over again with D4. And the world truly needs platforms like this. The flat timeline approach of Twitter and Mastodon makes it difficult to track a conversation. The commercialization and attendant censorship necessities on, say, Facebook and Reddit create a hostile environment for honest debate. Essentially every popular platform now promotes (by design or by accident) the polarization of social media that we see today. So we do need something different.

However, the software I was using to develop Themis just didn’t cut it. Nest is a great framework. Typescript is what JavaScript should be, if you ask me. Vue remains my favorite UI library. But everything I had done became too brittle. As any developer knows, it’s all too easy to turn your code castle into a house of cards, and that’s what happened to me.

It’s a poor craftsman who blames his tools, yes, but in this case some of the tools weren’t up to snuff. In particular, TypeORM, the database abstraction layer specifically recommended by the Nest documentation, is…not production-ready. It was at version 0.3 when I used it, and that beta label showed in too many respects. Thanks to Liblio (and the excellent SQLAlchemy library I’m using for it), I can see the deficiencies more clearly. Most apparent is that TypeORM has very limited support for nested or hierarchical data. For a back end based on nested messages, that’s far from ideal.

I have other problems with the stack I was using, but I don’t want this to turn into a rant, so let’s just say that I had to do some major hacking to make everything work right. (One example: Nest’s HTTP body parser, obviously necessary for requests like, oh, making a post, doesn’t allow you to use custom JSON mimetypes. Those are required by the ActivityPub spec.) And the documentation, as is far too common in the Node/JS/TS world, told me almost nothing. In a lot of cases, I discovered a bug or limitation only after hours of digging through Github issues, StackOverflow posts, or even the actual source code of a library. And it was just maddening.

But the final straw was my attempt at upgrading a few weeks back. I was using Vuetify for the Themis UI. (What can I say? I’m the weirdo who really likes Material Design.) Well, they put out a new major version, 2.0. So why not upgrade, right? Themis is still alpha, so breakage is to be expected.

Uh, nope. I still have no idea what happened, but upgrading from Vuetify 1.5 to 2.0 broke everything. I followed the instructions in their docs, but got a series of Typescript compilation errors that I could not resolve. Maybe it was something to do with my build setup. I don’t know. What I do know is that it frustrated me so much that I just plain quit. I don’t need that level of stress.

Compared to that, Liblio is a breath of fresh air, and it gave me the perspective to see what’s wrong with Themis and, more importantly, what I can do to fix it.

Fixing it, I think, means making it into something else, and that something is D4.

Themis dev diary #5

I’m up to the 8th alpha release of Themis, and things are looking up. Since my previous entry in this dev diary, I’ve made considerable work on implementing more of ActivityPub, but most of my time has been taken up by two things not absolutely necessary for a program to work, but vital to getting it adopted.

First is the bane of all programmers: documentation. I’ve spent most of the last month working on internal docs, source code comments, and the like. Fortunately, this all came at a time when I was facing a little bit of burnout on Themis, so the project itself didn’t suffer. Now, I still need to write the user documentation, including such minor tidbits like setting up a Themis server, logging into an existing one, and so on. But that’s for later. You know, when I think I might be ready for users.

Second, the front end. I hate doing UI work. I’m not an artistically-inclined person. I work better in text than art. I can’t draw, I have no real intuition when it comes to formatting or color schemes or anything like that. It’s just not who I am. On the plus side, ActivityPub is meant to be client-agnostic. Mastodon, Pleroma, and other platforms using the standard do have builtin clients, but you can also use the servers with your own apps. (I’ve mostly been using Fedilab, but I’ll likely switch from that now that its author has outed himself as a censor.) Themis will, I hope, reach the same state. For now, though, it has a simple, functional front-end inspired by traditional newsreader programs (such as my old favorite, Xnews).

Originally, I thought to use a three-panel system, in the vein of reader programs such as Pan and Forte (Free)Agent. Or even Thunderbird, although it’s geared more towards email than newsgroups. Thus, the left-hand panel would contain the groups, while the right side would be split between threads on the top and posts on the bottom.

I used that for the 0.0.6 version I described in the last dev diary, and it just didn’t work. Vuetify, the UI library I’m using on this project, made it extraordinarily difficult to align all the metadata columns while preserving the tree structure of a thread. As the tree is one of the two most defining features of Themis, this was unacceptable. On top of that, three-panel is hard to make work in the browser, especially when you’re working with widescreen monitors. (Back in the day, that wasn’t an issue. We had 4:3, and we liked it!) So I scrapped that and rewrote the whole thing. Now, Themis still uses 3 panels, but they’re set up as columns. Let’s take a quick look at how they work.

On the left is the list of groups your server hosts or otherwise knows about. Being an ActivityPub implementation, a Themis server can connect to other servers (instances, as they’re called). Some of these might not run Themis themselves; nothing’s stopping a Mastodon user, for example, from posting to a Themis group. But they can’t create groups, because their software doesn’t know how. Thus, you’ll only get groups from your server and any other Themis servers it has interacted with in the past. (Federation isn’t completely implemented yet, so take this as more of an ideal than the way things really are.)

Once you select a group, the middle panel appears. This shows the threads in the chosen group. Threads work here like they do in forums, email, or newsgroups. Each has a subject (interestingly enough, this also means all Themis posts will appear on Mastodon with content warnings) and other metadata, but you won’t get to see the text until you click.

That opens the right-hand panel, the thread view. This is what I couldn’t get working in previous versions, necessitating the rewrite. But I must admit a great amount of pride at my accomplishment. Internally, it’s a hand-rolled, block-level tree component with integrated event handling. For non-developers, just know that it shows every post in a thread in a nested format similar to comments on Reddit, Slashdot, Disqus, and many other sites/services. Best of all, you have an inline reply form, so you’ll always know where your post will appear in the thread.

Also coming in the rewrite was a new navigation system, easier to work with on the developer side. Version 0.0.8 adds in some of the places it can take you, namely your profile and your list of “favorites”, which is basically the same as on any other social media platform.

Some things remain unimplemented, but I feel I’m making great progress. Before the beta release—still targeted for October 1—I want to add a new, more comprehensive filtering system. That’s the big one, in my opinion. That’s what separates Themis from the pedestrian “block or mute” systems of other platforms, and it will, I hope, allow for a higher standard of discussion. “All that’s dead might live again,” as the song says. (And if you can name that tune, I’ll put you in the contributors section!)

Keep watching for future dev diaries, and track the progress of the project over at Github. If there’s any interest, I might set up a donations page, too.

Themis front page, version 0.0.8, showing the login form, site info, and “A Look Inside” counterpart

Themis 0.0.8 main page, with only Group View open

Themis 0.0.8 main page, with all panes open, showing nested replies

Themis 0.0.8 main page, with navigation menu open

Themis 0.0.8 main page, showing the New Thread composer

Themis 0.0.8 user profile page, showing mostly placeholders

Themis 0.0.8 main page, showing the Inline Reply composer

Themis dev diary #4

Since my last update, I’ve now released the 6th alpha of Themis. It’s still not ready for primetime, alas, but I’m making progress. This particular milestone marks the introduction of ActivityPub support, one of the cornerstones of the Themis project. So far, the server can handle incoming Create activities (or post objects by themselves, which it converts into these as per the spec) and follow requests for groups…as long as they come from local users.

In other words, we’re not federating yet, but it’s something. I’ve never implemented a spec of this complexity before, and I’m still working alone. It’s harder than I thought, especially since so much of the AP spec expects you to be online. (In fact, it outright tells you not to deliver to localhost, meaning that I can’t even call Themis compliant until I’m done testing!)

Now, though, I’m taking a break from that and shifting my focus back to the front end. For Themis, this is a kind of SPA using the excellent Vue.js framework and the Vuetify UI library. It’s Material Design, but I’m a strange person. (Okay, you already knew that much.) I like the look of Material Design. It’s clean, uncrowded, and it has what I see as a nice feel. No design spec is perfect, of course, and Material does have some glaring flaws. Vuetify has even more. Still, I think it’s a good starting point.

The screenshot below shows what will become the front page of a Themis server. The login box switches to an account creation form when requested, the right-hand column displays a list of all known groups, and the middle portion will hold a server-specific description that can contain local rules, admin info, or anything else you like. (Editing that information is yet to be implemented.) Other than that, it’s pretty much final. All that’s missing is a logo, which may take someone with more artistic skill—compared to me, that’s anybody.

Because so many people use mobile devices instead of proper desktops these days, I’m also doing my best to make this front-end fully responsive. It’s not mobile-first, because I don’t believe in that philosophy, but Android phones and tablets will be first-class citizens in the Themis network. (I can’t afford to test Apple products, so that’s why they’re not included.)

So, after all that hard work to get 1/3 of the ActivityPub spec implemented, expect some faster progress now. Themis still has a lot of work to go, but I hope to get at least a feature-complete beta version out by October 1st. That’s my target, as it has been all along. Now, I think I can do it.

Themis Dev Diary #3

This will be a much quicker post than the last two, and there’s a very good reason for that. You see, I’ve never implemented a spec before. ActivityPub isn’t the easiest, from what I can tell, and it’s exposed quite a few…deficiencies in my design for Themis. So, at the moment, I’m spinning my wheels a bit.

The crux of the issue is the way the spec expects me to communicate. ActivityPub uses activities for that (duh). These are objects with a number of properties, one of which is an ID. These have to be globally unique, and the easiest way to do that is to tie them to the originating server. So the server at example.com, for instance, can make IDs of the form example.com/activity/1234: the last number is different for each new activity, and it probably comes from the autogenerated database key. (An alternative is UUIDs, which I use elsewhere in Themis. Flake IDs—what Pleroma uses—are another option, if you’re looking for something that can be sorted chronologically, which is required by certain parts of the spec.)

So far, not so bad. But the AP spec wants these IDs to be URIs. And that means I have to format them properly. The problem is, a URI has a few necessary components. I have to account for subdomains, for instance. And the difference between HTTP and HTTPS, because somebody might use the former (I am for my dev instance, so why not?). Let’s not forget nonstandard ports, either. Listening on 80 or 443 requires root privileges on Linux, and NestJS defaults to 3000.

Putting all that together proves that my initial idea of just storing an origin host name alongside the names of groups and users is, to put it mildly, inadequate. Yesterday, I added a new Server object, which will store every part of a URI except the path. Hopefully, that’ll be enough to make ID generation a lot easier. And let’s also hope I don’t break too much in the process.

Anyway, once I get that done, I’m thinking the rest of ActivityPub will be relatively simple. Not easy, mind you, but I actually have made some progress on implementing the client-to-server portion of the spec, which is something even Mastodon isn’t doing. Give me a few more weeks, and I think I’ll be ready for Alpha 6. Until then, keep your fingers crossed that I don’t screw this up too much.

Themis Dev Diary #2

It’s been a few weeks, and this project of mine is still moving along. Maybe not as fast as I would like, but I am making progress. Since the last post, I’ve spent much of my coding time working on what I consider the biggest feature of Themis: filtering. Here, I want to talk a little bit about what I mean, and why it’s so important.

My computer, my rules

Today, essentially every discussion platform is moderated. What that means depends on the place, but let’s boil it down to its essence. Moderation is censorship, plain and simple. Sometimes it’s necessary, sometimes it serves a purpose, but a moderated community is one that has decided, either by collective choice or external fiat, to disallow certain topics. More importantly, the administrators of the platform (or their anointed assistants) have the power to remove such content, often without debate or repercussion.

Removing the users that post the prohibited content is the next step. If online communities were physical, such suspensions would be the equivalent of banishment. But a much larger site like Facebook or Twitter, so integrated into the fabric of our society, should be held to a higher standard. When so much in our lives exists only in these walled-off places, banning is, in fact, more akin to a death sentence.

It is my strong belief that none of this is necessary. Except in the most extreme cases—automated spamming, hacking attempts, or illegal content that disrupts the infrastructure of the site—there really isn’t a reason to completely bar someone from a place simply because of what others might think. Themis is modeled on Usenet, and Usenet didn’t have bans. True, your account on a specific server could be locked, but you could always make a new one somewhere else, yet retain the ability to communicate with the same set of people.

This is where Facebook, et al., fail by design. Facebook users can only talk to each other. You can’t post on Twitter timelines unless you have a Twitter account. On the other hand, the “fediverse” meta-platform of Mastodon, Pleroma, etc., returns to us this ability. It’s not perfect, but it’s there, which is more than we can say for traditional social media.

Out of sight, out of mind

But, you may be thinking, isn’t that bad? If nobody wants to see, say, propaganda from white supremacists in their discussions, then how is discussion better served by allowing those who would post that content to do so?

The answer is simple: because some people might want to see that. And because what is socially acceptable today may become verboten tomorrow. Times change, but the public square is timeless. As the purpose of Themis is to create an online public space, a place where all discussion is welcome, it must adhere to the well-known standards of the square.

This is where filtering comes in. Rather than give the power of life and death over content to administrators and moderators, I seek to place it back where it belongs: in the hands of the users. Many sites already allow blocklists, muting, and other simple filters, but Themis aims to do more.

Again, I must bring up the analogy of Usenet. The NNTP protocol itself has no provisions for filtering. Servers can drop or remove messages if they like, but this happens behind the scenes. Instead, users shape their own individual experiences through robust filtering mechanisms. The killfile is the simplest: a poster goes in, and all his posts are hidden from view. Most newsreader software supports this most basic weapon in our arsenal.

Others go the extra mile. The newsreader slrn, for instance, offers a complex scoring system. Different qualities of a post (sender, subject text, and so on) can be assigned a value, with the post itself earning a score that is the sum of all filters that affect it. Then, the software can be configured to show only those posts that meet a given threshold. In this way, everything a user doesn’t want to see is invisible, unless it has enough “good” in it to rise above the rest. Because there are diamonds in the rough.

Plans

The score system works, but it’s pretty hard to get into. So, by default, Themis won’t have it. But that doesn’t mean you can’t use it. The platform I’m building will be extensible. It will allow alternative clients, not just the one I’m making. Thus, somebody out there (maybe even me, once I have time) can create something that rivals slrn and those other newsreaders with scoring features.

But the basics have to be there. At the moment, that means two things. First is an option to allow a user to “mute” groups and posters. This does about what you’d expect. On the main group list (the first step in reading on Themis), muted groups will not be shown. In the conversation panel, posts by muted users will not be shown, instead replaced by a marker that indicates their absence. In the future, you’ll have the option to show these despite the blocks.

Second is the stronger filtering system, which appears in Alpha 4 at its most rudimentary stage. Again, groups and users can be filtered (posts themselves will come a little later), and the criteria include names, servers, and profile information. As of right now, it’s mostly simple string filtering, plus a regex option for more advanced users. More will come in time, so stay tuned.

In closing

This is why I started the project in the first place, and I hope you understand my reasoning. I do believe that open discussion is necessary, and that we can’t have that without, well, openness. By placing the bulk of the power back in the hands of the users, granting them the ability to create their own “filter bubbles” instead of imposing our own upon them, I think it’s possible. I think we can get past the idea that moderators, with all their foibles and imperfections, are an absolute necessity for an online forum. The result doesn’t have to be the anarchy of 4chan or Voat. We can have serious, civil conversations without being told how to have them. Hopefully, Themis will prove that.