Dumbing down tech

I recognize that I’m smarter than most people. I’ve known that as long as I can remember. When I was six years old, I took a standardized IQ test. You know, the kind whose results are actually usable. I apparently scored a minimum of 175; it wasn’t until a few years later, when I was studying statistics, that I understood both what that meant in relation to society at large and why it had to be a minimum. (IQ is a relative measurement, not an absolute one. Once you get to a certain point, small sample sizes make a precise evaluation impossible.)

There is, of course, a big difference between intelligence and wisdom, though I like to think I also have a good stock of the latter. In some fields, however, the intelligence factor is the defining one, and tech is one of those fields. Why? Because intelligence isn’t just being able to recite facts and formulas. It’s about reasoning. Logic, deduction, critical thinking, and all those other analytical skills that have been absent from most children’s curricula for decades. While some people literally do have brains that are better wired for that sort of thinking—I know, because I’m one of them—anyone can learn. Logic is a teachable skill. Deductive reasoning isn’t intuition.

Modern society, in a most unfortunate turn of events, has deemed analytical thinking to be a hindrance rather than an aid. While public schooling has always been about indoctrination first, and education a very distant second, recent years have only made the problem both more visible and more pronounced. I’ll get back to this point in a moment, but it bears some consideration: as a 40-year-old man, I grew up in a society that was indifferent to high intelligence, but I now find myself living in one that is actively hostile to it.


I’ve always enjoyed reading tech books. Even in the age of Medium tutorials, online docs, and video walkthroughs, I still find it easiest to learn a new technology from a book-length discussion of it. And these books used to be wonderful. Knuth’s The Art of Computer Programming has parts that are now approaching 60 years old, yet it’s still relevant today. The O’Reilly programming language books were often better than a language’s official documentation.

It’s been awhile since I really needed to read a book for a new technology. I’ve spent the past few years working with a single stack that doesn’t have a lot of "book presence" anyway, and the solo projects I’ve started have tended to use things I already knew. Now that I’m unemployed and back to the eternal job hunt, though, I wanted to look for something new, and I was tired of looking at online resources that are, by and large, poorly written, poorly edited, and almost completely useless beyond the beginner level. So I turned to books, hoping for something better.

I didn’t find it.

One book I tried was Real World OCaml. For a few years, I’ve been telling myself that I should learn a functional language. I hate functional programming in general, because I find it useless for real-world problems—the lone exception is Pandoc, which I use to create my novels, because text transformation is one of the few actual uses for FP. But it’s become all the rage, so I looked at what I felt to be the least objectionable of the lot.

The language itself isn’t bad. It has some questionable decisions, but it’s far more palatable than Haskell or Clojure. That comes from embracing imperative programming as valid, meaning that an OCaml program can actually accomplish things without getting lost in mathematical jargon or a sea of parentheses.

But the book…well, it wasn’t bad. It just didn’t live up to its title. There wasn’t much of the real world in it, just the same examples I’d get from a quick Brave search. The whirlwind tour of the language was probably the best part, because it was informative. Tech books work best when they inform.


Okay, maybe that’s a one-off. Maybe I ran into a bad example. So I tried again. I’m working on Pixeme again, the occasional side project I’ve rewritten half a dozen times now, and I decided that this iteration would use the stack I originally intended before I got, er, distracted. As it turns out, the authors of the intriguing Htmx library have also written a book about it, called Hypermedia Systems.

This was where I started getting worried. The book is less about helping you learn their library and more about advancing their agenda. Yes, that agenda has its good parts, and I agree with the core of it, that a full-stack app can offer a better experience for both developers and users than a bloated, Javascript-heavy SPA. The rest of it is mostly unhelpful.

As someone who has been ridiculed for pronouncing "GIF" correctly (like the peanut butter, as the format’s author said) and fighting to keep "hacker" from referring to blackhats, I have to laugh when the authors try to claim that a RESTful API isn’t really REST, and use an appeal to authority to state that the term can only apply to a server returning HTML or some reasonable facsimile.

Advocacy aside, the book was unhelpful in other ways. I can accept that you feel your technology is mostly for the front end, so you don’t want to bog down your book with the perils and pitfalls of a back-end server. But when you’re diving into a method of development that requires a symbiotic relationship between the two, using the academic "beyond the scope" cop-out to wall off any discussion of how to actually structure a back end to benefit from your library is doing your readers—your users—a great disservice. If the scope of your book doesn’t include patterns for taking advantage of a "hypermedia" API, then what does it include? A few new HTML attributes and your whining that people are ignoring a rant from three decades ago?


Alright, I thought after this, I’ll give it one more shot. Never let it be said that I’m not stubborn. The back end of this newest version of Pixeme is going to use Django. Mostly, that’s because I’m tired of having to build out or link together all the different parts of a server-side framework that FastAPI doesn’t include. Things like logins, upload handling, etc. I still want to use Python, because that’s become the language I’m most productive in, but I want something with batteries included.

The official documentation for Django is an excellent reference, but it’s just that: a reference. There’s a tutorial, but this ends very quickly, and offers almost no insight on, say, best practices. That, for me, is the key strength of a tech book: it has the space and the "weight" to explain the whys as well as the hows. So I went looking for a recent book on the topic, and I ended up with Ultimate Django for Web App Development Using Python. A bit of a mouthful, but it’s so new that it even uses the "on X, formerly Twitter" phrasing that mainstream media has adopted to refer to Twitter. (Seriously, nobody in the real world calls it X, just like nobody refers to the Google corporate entity as Alphabet.)

In this case, the book is somewhat informative, and it functions a lot like an expanded version of the official Django tutorial. If you’re new to the framework, then it’s probably not a bad guide to getting started. From something with "ultimate" in the title, I just expected…more. Outside of the tutorial bits, there’s not much there. The book has a brief overview of setting up a Docker container, but Docker deserves to be wiped off the face of the earth, so that’s not much help. And the last chapter introduced Django Ninja, a sort of FastAPI clone that would be incredible if not for the fact that its developers support child trafficking and religious persecution.

Beyond that, the text of the book is littered with typos and grammatical errors. Most of these cases have the telltale look of an ESL author or editor, a fact which is depressingly common in tech references of all kinds nowadays. Some parts are almost unreadable, and I made sure to look over any code samples I wanted to use very carefully. It’s like dealing with ChatGPT, except here I know there was a real human involved at some point, someone who looked at the text and said, "Yeah, this is right." That’s even worse.


Three strikes, and I’m out. Maybe I’m just unlucky, or maybe these three books are representative of modern tech literature. If it’s the latter, that only reinforces my earlier point: today’s society rewards mediocrity and punishes intelligence, even in fields where intelligence is paramount.

Especially in programming, where there is no room for alternate interpretations, the culture of "good enough" is not only depressing but actively harmful. We laugh wryly at the AAA video game with a 200 GB install size and a 50 GB patch on release day, but past experiences show that it doesn’t have to be that way. We can have smart developers. As with any evolutionary endeavor, we have to select for them. Intelligence needs to be rewarded at all stages of life. Otherwise, we’ll be stuck where we are now: with ESL-level books that recapitulate tutorials, screeds disguised as reference texts, and a thousand dead or paywalled links that have nothing of value.

As a case in point, I was looking just yesterday for information about code generation from abstract syntax trees. This is a fundamental part of compiler design, something every programming language has to deal with at some point. Finding a good, helpful resource should be easy, right?

Searching the web netted me a few link farms and a half-finished tutorial using Lisp. Looking for books wasn’t much better, because the only decent reference is still the Dragon Book, published in 1986! Yes, the state of the art has certainly advanced in the past 38 years, but…good luck finding out how.

That’s what needs to change. It isn’t only access to information. It isn’t only that this information isn’t being written down. It’s a confluence of factors. All of it happening all at once is making us dumber as a people. Worst of all is that we accept it. Whether you consider it the "price of democracy" or simply decide that there’s nothing you can do about it, accepting this rot has only let it fester.

Full stack adventures 1: Meet the stack

I have a lot of different development projects. Most aren’t all that great, and some of the older ones are…well, they’re awful. (For the morbidly curious, I have many of them on GitHub, and my newer ones will start to appear on Gitlab.)

My current one, however, has been an adventure. It’s useful, it makes for a good learning experience, and it has a very wide range of technologies. I’m truly working with a full stack on this one, even if I may not be playing with a full deck.

What it is

The project is called Pixeme, a portmanteau of “picture” and “lexeme” that, in my opinion, captures the essence of what I’m trying to create. Basically, I took the old saying “A picture is worth a thousand words” far too literally.

My goal is to make a full web platform for what I describe in the documentation as a community for visual language learning. To put it simply: users post pictures with simple, descriptive captions. Nothing more than a single word or phrase, though they can add more text later. What separates this from, say, Instagram is that other users (or the same one) can add captions for other languages.

For example, I could post a photo of my stepdad’s dog with the caption “a dog”. Then, someone who speaks French might come along and add “un chien” to it. A Japanese user could then add “犬” to the mix, and so on. Together, we build a kind of cross-language picture dictionary. This can then be used by people who really are trying to learn languages and understand that visual reinforcement helps.

(I have plenty of other ideas, things like audio attachments, larger texts, Anki-style flashcards, adding in support for constructed or artificial languages, and even possible research uses, but those are all for much later. Let me get the site started first.)

All in all, Pixeme checks all the boxes for me. It involves programming and linguistics, two topics which any reader of this blog will know are among my favorites. It’s a long-term project, so it requires focus but gives the satisfaction of completion in return. And maybe I can even find some way to monetize it in the future. Even if I can’t, it’s still valuable experience with a number of different languages, frameworks, and libraries.

The stack

Every web application is layered. There’s no way around it. At the very least, you have the server side and the client side, but most modern apps add in multiple extra layers, forming a stack. Pixeme is no exception. For this project, I chose a stack which, for the most part, reflects my personal preferences while also allowing me to stay on top of current developments in both sides of the web equation.

The server-side framework I chose is Flask. I like Python. Even though I feel the 3.x series was a needless break in compatibility, it’s still one of the few languages I feel comfortable writing. I’ve joked before that I can write Python in my sleep—that’s how much experience I have, and how much it fits my mindset.

Flask calls itself a “micro-framework” for web applications, meaning that it comes with very few bells and whistles, but plenty of extensibility. And I like that, too. It’s a little harder to set up, as you have to track down a number of extensions for things like database connectivity, form validation, authentication, etc., but that’s okay. You only pay for what you use, and there’s nothing hidden.

Apart from Flask extensions, the rest of the back end is pretty standard. SQLAlchemy and PostgreSQL, which is pretty much Python Databases 101. Pytest for testing, because tests are important. (I’ll admit that I haven’t always kept those up to date in past projects!) The Marshmallow library for serialization and validation. And the usual host of minor packages for little things. You can’t get away from those.

The front end is where things get interesting. Pixeme has a REST API because that’s the cool thing to do these days, but also because it really does make things easier even when you don’t want an SPA. And I decided after writing half of one that I don’t want another SPA. It just doesn’t fit the vision I have for this platform. Oh, there will be an app eventually, and that’ll have web and mobile versions, but the site itself needs to be the initial focus.

Flask includes the Jinja template engine; you can use your own, but it’s easier to go with the default. So I’m writing the view layer (Pixeme mostly follows the typical MVC architecture, in case you were wondering) as HTML templates using Jinja. That’s sometimes harder than it looks, one reason why this post is the first in a series. We’ll get into some of the difficulties later, though.

HTML isn’t the only part of the web in 2020, so Pixeme has a whole client-side layer, and this is where my focus on minimalism ramps up. As I said, I gave up on an SPA for the initial version of the site. I personally like Vue over React, so that’s where I started. But then I saw how I was duplicating so much of the server functionality, so I decided to back up and make a more traditional site. Still, reactivity is great, so I turned to Alpine.js to fill that niche. It’s not perfect, but it’s very helpful, and I’ll explain that later in the series, too.

Last, CSS. I know what you’re thinking: Bootstrap. Or maybe Bulma or one of the other alternatives. And those are great. I’ve used them before, and I do enjoy them…when they fit. I wanted to try something different with Pixeme, something more customized, so I chose TailwindCSS instead. It takes more setting up, it doesn’t play perfectly well with Jinja (yet another topic for another post), but it suits me. Once I grasped the concepts behind it, it just made so much sense.

So that’s the Pixeme stack. Python, Flask, and Postgres are the server side; what OS the whole thing will run on is still an open question, but bet on some flavor of Linux. Jinja for templating and views. A front end built with Alpine and Tailwind. “Full stack” is always a juggling act, and this might be my most ambitious attempt yet. Stay tuned to see how it works out.

Pandoc filter for books

Pandoc is a great tool, as I’ve already stated, but it can’t do everything. And some of the things it dies aren’t exactly what you’d want when creating a book. This is especially true when working on a print-ready PDF, as I’ve been doing.

Fortunately, there is a solution. Unfortunately, it’s not a pretty one. The way Pandoc works internally—I’m simplifying a lot here, so bear with me—is by turning your input document (Markdown, in my case) into an AST, then building your output from that. That’s basically the same thing a compiler does, so you could think of Pandoc as something like a Markdown compiler that can output PDF, EPUB, HTML, or whatever.

In addition to the usual “compiler” niceties, you’re also given access to an intermediate stage. If you tell Pandoc to let you, you can use filters to modify the AST before it’s sent off to the output stage. That lets you do a lot of modifications and effects that aren’t possible with plain Markdown or LaTeX or even HTML. It’s great, but…

Cruel and unusual

But Pandoc is written in Haskell. Haskell, if you’re not familiar with programming languages, is the tenth circle Dante didn’t tell you about. It’s awful, if you’ve ever written code in any other language, because it’s designed around a philosophy that doesn’t really match anything else in the programming world. (Seriously, go look up “monads” if you’re bored.) For us mere mortals, it’s sheer torture trying to understand a Haskell program, much less write one. And Pandoc’s default language for writing filters, alas, is this monstrosity.

If I had to do that, I’d have given up months ago. But I’m in luck, because Pandoc’s developer recognizes that we’re not all masochists, and he gave us the option to write filters in Python instead. Now that I can use. It’s not pretty. It’s not nice. But it gets the job done, and it does so without needing to install extra libraries or anything like that.

So, I’ve written a few filters that take care of some of the drudgery of converting Markdown into a decent-looking PDF. You can find them in this Gist if you want to see the gory details, and I’ll describe each of them below.

Fancy breaks

In Pandoc’s version of Markdown, you can get a horizontal rule (the HTML hr element) by making a line containing only asterisks with spaces between them: * * * is what I use these days. It’s simple enough, and you can use CSS to make it not appear as an actual line across the page, but as a nice vertical blank space that serves as a scene break. It even carries over into MOBI format when you use Kindlegen.

But it doesn’t work for PDFs. Well, it does, but there’s an even better way. Since I’m using Memoir, I get what are called “fancy” breaks. In print, they’re nothing more than a centered set of asterisks, stars, or any other icon you’d like to use. Those can be a bit tacky if they show up after every seen, though, so there’s another option that only shows the “fancy” breaks when they’d be at the end of a page, but instead puts in a “plain” blank otherwise. In Memoir, this is the \pfbreak command, and it’s smart enough to choose the right style every time.

So all the fancybreak.py filter does is swap out Pandoc’s HorizontalRule AST element, replacing it with the raw LaTeX code for Memoir’s “plain fancy break”. Take out the boilerplate, and it’s literally only three lines of code. Simple, even for me.

Writing links

Another difference between print and digital editions of a book comes from the formatting available. E-books are interactive in a way paper can’t be. They can use hyperlinks, and I do exactly that. But it’s impossible to click on a link in a paperback, and blue doesn’t show up in a black and white book, so I need to get rid of the link part. Ideally, I’d like to keep the address, though.

For this, I wrote the writelinks.py filter. This one’s a little bit harder to explain from a code point of view. From the reader’s perspective, though it’s easy: every link is removed, but its address is added to the text in parentheses instead. It comes out as preformatted (or verbatim) text, in whatever monospaced font I’m using. (I actually don’t remember which one.)

The guts of this filter are only 5 lines, and the hardest part was working out exactly what I had to do to get the link address. Pandoc’s API documentation isn’t very helpful in this regard, and it gets even worse in a moment.

Drop caps and raised initials

Here’s where I was ready to gouge my own eyeballs out. If you look at the code for dropcaps.py and raisedinitials.py, you’ll probably see why. Let’s back up just a second, though, so we can ask a simple question: What was I thinking? (Don’t answer that.)

I like the “raised initial” style for books. With this, the first letter of a chapter is printed bigger than the rest, and the rest of the first word is printed in regular-sized small caps. Other people like “drop caps”, where the initial letter hangs down into the first paragraph. Either way, one LaTeX package, lettrine, takes care of your needs. Using it with Memoir is a matter of importing it and adding a bit of markup at the beginning of each chapter.

Using it with Pandoc, on the other hand, takes more work. Since I don’t want to sprinkle LaTeX code all over my source documents, I made these filters to inject that code later in the process. And that was…not fun at all. After a lot of trial and error (going from Haskell to Python and back doesn’t give you a lot of useful diagnostics), I settled on the process I used in these filters. They’re the same thing, by the way. The only real difference is their output.

Well, and dropcaps.py has to break up a Quoted element so it doesn’t blow up the opening quotation mark instead of the first letter. Doing that required some trickery. If you’d like to try it for yourself, I suggest drinking heavily. If you don’t drink, well, you’ll want to by the time you’re done.

Limitations and future expansion

Anyway, after I finished this herculean task, I had a set of filters that would let me use my original source files but produce something much more suited to Memoir and the paperback format. Now I’ve got fancy scene breaks, links that write themselves out when they’re in a PDF, and those wonderfully enormous initial letters starting each chapter.

Can I do more? Of course I can. The last two filters don’t take into account “front matter” chapters. For my current novels, that’s not a problem, as I don’t use those. But if you need something with, say, an extended foreword, then you’d need to hack on the scripts to fix that.

There’s also nothing at all I can do for the opening pages of the book, the parts that come before the text. Here, the formats are too different even for filters. I’m talking about the title page, copyright page, dedication, and things like that. (These, in fact, are considered front matter, but they’re not part of a chapter, so the last paragraph doesn’t apply.) I still need to maintain two versions of those, and I don’t see any real way around that.

Still, what I’ve got so far is good. It was a lot of work, but it’s work I only have to do once. That’s the beauty of programming in a nutshell. It’s automation. Sure, I could have done the editing by hand instead of writing scripts to do it for me, and I probably would have been done sooner, but now I won’t have to do it all over again for Nocturne or any other book I write in the future.

To close out this miniseries, I have one more post in mind. This one will look at some of the additional LaTeX packages I used, like the lettrine one I mentioned above. By the time that comes out, maybe I’ll even have another book ready.