python – Prose Poetry Code

I have a lot of different development projects. Most aren’t all that great, and some of the older ones are…well, they’re awful. (For the morbidly curious, I have many of them on GitHub, and my newer ones will start to appear on Gitlab.)

My current one, however, has been an adventure. It’s useful, it makes for a good learning experience, and it has a very wide range of technologies. I’m truly working with a full stack on this one, even if I may not be playing with a full deck.

What it is

The project is called Pixeme, a portmanteau of “picture” and “lexeme” that, in my opinion, captures the essence of what I’m trying to create. Basically, I took the old saying “A picture is worth a thousand words” far too literally.

My goal is to make a full web platform for what I describe in the documentation as a community for visual language learning. To put it simply: users post pictures with simple, descriptive captions. Nothing more than a single word or phrase, though they can add more text later. What separates this from, say, Instagram is that other users (or the same one) can add captions for other languages.

For example, I could post a photo of my stepdad’s dog with the caption “a dog”. Then, someone who speaks French might come along and add “un chien” to it. A Japanese user could then add “犬” to the mix, and so on. Together, we build a kind of cross-language picture dictionary. This can then be used by people who really are trying to learn languages and understand that visual reinforcement helps.

(I have plenty of other ideas, things like audio attachments, larger texts, Anki-style flashcards, adding in support for constructed or artificial languages, and even possible research uses, but those are all for much later. Let me get the site started first.)

All in all, Pixeme checks all the boxes for me. It involves programming and linguistics, two topics which any reader of this blog will know are among my favorites. It’s a long-term project, so it requires focus but gives the satisfaction of completion in return. And maybe I can even find some way to monetize it in the future. Even if I can’t, it’s still valuable experience with a number of different languages, frameworks, and libraries.

The stack

Every web application is layered. There’s no way around it. At the very least, you have the server side and the client side, but most modern apps add in multiple extra layers, forming a stack. Pixeme is no exception. For this project, I chose a stack which, for the most part, reflects my personal preferences while also allowing me to stay on top of current developments in both sides of the web equation.

The server-side framework I chose is Flask. I like Python. Even though I feel the 3.x series was a needless break in compatibility, it’s still one of the few languages I feel comfortable writing. I’ve joked before that I can write Python in my sleep—that’s how much experience I have, and how much it fits my mindset.

Flask calls itself a “micro-framework” for web applications, meaning that it comes with very few bells and whistles, but plenty of extensibility. And I like that, too. It’s a little harder to set up, as you have to track down a number of extensions for things like database connectivity, form validation, authentication, etc., but that’s okay. You only pay for what you use, and there’s nothing hidden.

Apart from Flask extensions, the rest of the back end is pretty standard. SQLAlchemy and PostgreSQL, which is pretty much Python Databases 101. Pytest for testing, because tests are important. (I’ll admit that I haven’t always kept those up to date in past projects!) The Marshmallow library for serialization and validation. And the usual host of minor packages for little things. You can’t get away from those.

The front end is where things get interesting. Pixeme has a REST API because that’s the cool thing to do these days, but also because it really does make things easier even when you don’t want an SPA. And I decided after writing half of one that I don’t want another SPA. It just doesn’t fit the vision I have for this platform. Oh, there will be an app eventually, and that’ll have web and mobile versions, but the site itself needs to be the initial focus.

Flask includes the Jinja template engine; you can use your own, but it’s easier to go with the default. So I’m writing the view layer (Pixeme mostly follows the typical MVC architecture, in case you were wondering) as HTML templates using Jinja. That’s sometimes harder than it looks, one reason why this post is the first in a series. We’ll get into some of the difficulties later, though.

HTML isn’t the only part of the web in 2020, so Pixeme has a whole client-side layer, and this is where my focus on minimalism ramps up. As I said, I gave up on an SPA for the initial version of the site. I personally like Vue over React, so that’s where I started. But then I saw how I was duplicating so much of the server functionality, so I decided to back up and make a more traditional site. Still, reactivity is great, so I turned to Alpine.js to fill that niche. It’s not perfect, but it’s very helpful, and I’ll explain that later in the series, too.

Last, CSS. I know what you’re thinking: Bootstrap. Or maybe Bulma or one of the other alternatives. And those are great. I’ve used them before, and I do enjoy them…when they fit. I wanted to try something different with Pixeme, something more customized, so I chose TailwindCSS instead. It takes more setting up, it doesn’t play perfectly well with Jinja (yet another topic for another post), but it suits me. Once I grasped the concepts behind it, it just made so much sense.

So that’s the Pixeme stack. Python, Flask, and Postgres are the server side; what OS the whole thing will run on is still an open question, but bet on some flavor of Linux. Jinja for templating and views. A front end built with Alpine and Tailwind. “Full stack” is always a juggling act, and this might be my most ambitious attempt yet. Stay tuned to see how it works out.

Pandoc is a great tool, as I’ve already stated, but it can’t do everything. And some of the things it dies aren’t exactly what you’d want when creating a book. This is especially true when working on a print-ready PDF, as I’ve been doing.

Fortunately, there is a solution. Unfortunately, it’s not a pretty one. The way Pandoc works internally—I’m simplifying a lot here, so bear with me—is by turning your input document (Markdown, in my case) into an AST, then building your output from that. That’s basically the same thing a compiler does, so you could think of Pandoc as something like a Markdown compiler that can output PDF, EPUB, HTML, or whatever.

In addition to the usual “compiler” niceties, you’re also given access to an intermediate stage. If you tell Pandoc to let you, you can use filters to modify the AST before it’s sent off to the output stage. That lets you do a lot of modifications and effects that aren’t possible with plain Markdown or LaTeX or even HTML. It’s great, but…

Cruel and unusual

But Pandoc is written in Haskell. Haskell, if you’re not familiar with programming languages, is the tenth circle Dante didn’t tell you about. It’s awful, if you’ve ever written code in any other language, because it’s designed around a philosophy that doesn’t really match anything else in the programming world. (Seriously, go look up “monads” if you’re bored.) For us mere mortals, it’s sheer torture trying to understand a Haskell program, much less write one. And Pandoc’s default language for writing filters, alas, is this monstrosity.

If I had to do that, I’d have given up months ago. But I’m in luck, because Pandoc’s developer recognizes that we’re not all masochists, and he gave us the option to write filters in Python instead. Now that I can use. It’s not pretty. It’s not nice. But it gets the job done, and it does so without needing to install extra libraries or anything like that.

So, I’ve written a few filters that take care of some of the drudgery of converting Markdown into a decent-looking PDF. You can find them in this Gist if you want to see the gory details, and I’ll describe each of them below.

Fancy breaks

In Pandoc’s version of Markdown, you can get a horizontal rule (the HTML hr element) by making a line containing only asterisks with spaces between them: * * * is what I use these days. It’s simple enough, and you can use CSS to make it not appear as an actual line across the page, but as a nice vertical blank space that serves as a scene break. It even carries over into MOBI format when you use Kindlegen.

But it doesn’t work for PDFs. Well, it does, but there’s an even better way. Since I’m using Memoir, I get what are called “fancy” breaks. In print, they’re nothing more than a centered set of asterisks, stars, or any other icon you’d like to use. Those can be a bit tacky if they show up after every seen, though, so there’s another option that only shows the “fancy” breaks when they’d be at the end of a page, but instead puts in a “plain” blank otherwise. In Memoir, this is the \pfbreak command, and it’s smart enough to choose the right style every time.

So all the fancybreak.py filter does is swap out Pandoc’s HorizontalRule AST element, replacing it with the raw LaTeX code for Memoir’s “plain fancy break”. Take out the boilerplate, and it’s literally only three lines of code. Simple, even for me.

Writing links

Another difference between print and digital editions of a book comes from the formatting available. E-books are interactive in a way paper can’t be. They can use hyperlinks, and I do exactly that. But it’s impossible to click on a link in a paperback, and blue doesn’t show up in a black and white book, so I need to get rid of the link part. Ideally, I’d like to keep the address, though.

For this, I wrote the writelinks.py filter. This one’s a little bit harder to explain from a code point of view. From the reader’s perspective, though it’s easy: every link is removed, but its address is added to the text in parentheses instead. It comes out as preformatted (or verbatim) text, in whatever monospaced font I’m using. (I actually don’t remember which one.)

The guts of this filter are only 5 lines, and the hardest part was working out exactly what I had to do to get the link address. Pandoc’s API documentation isn’t very helpful in this regard, and it gets even worse in a moment.

Drop caps and raised initials

Here’s where I was ready to gouge my own eyeballs out. If you look at the code for dropcaps.py and raisedinitials.py, you’ll probably see why. Let’s back up just a second, though, so we can ask a simple question: What was I thinking? (Don’t answer that.)

I like the “raised initial” style for books. With this, the first letter of a chapter is printed bigger than the rest, and the rest of the first word is printed in regular-sized small caps. Other people like “drop caps”, where the initial letter hangs down into the first paragraph. Either way, one LaTeX package, lettrine, takes care of your needs. Using it with Memoir is a matter of importing it and adding a bit of markup at the beginning of each chapter.

Using it with Pandoc, on the other hand, takes more work. Since I don’t want to sprinkle LaTeX code all over my source documents, I made these filters to inject that code later in the process. And that was…not fun at all. After a lot of trial and error (going from Haskell to Python and back doesn’t give you a lot of useful diagnostics), I settled on the process I used in these filters. They’re the same thing, by the way. The only real difference is their output.

Well, and dropcaps.py has to break up a Quoted element so it doesn’t blow up the opening quotation mark instead of the first letter. Doing that required some trickery. If you’d like to try it for yourself, I suggest drinking heavily. If you don’t drink, well, you’ll want to by the time you’re done.

Limitations and future expansion

Anyway, after I finished this herculean task, I had a set of filters that would let me use my original source files but produce something much more suited to Memoir and the paperback format. Now I’ve got fancy scene breaks, links that write themselves out when they’re in a PDF, and those wonderfully enormous initial letters starting each chapter.

Can I do more? Of course I can. The last two filters don’t take into account “front matter” chapters. For my current novels, that’s not a problem, as I don’t use those. But if you need something with, say, an extended foreword, then you’d need to hack on the scripts to fix that.

There’s also nothing at all I can do for the opening pages of the book, the parts that come before the text. Here, the formats are too different even for filters. I’m talking about the title page, copyright page, dedication, and things like that. (These, in fact, are considered front matter, but they’re not part of a chapter, so the last paragraph doesn’t apply.) I still need to maintain two versions of those, and I don’t see any real way around that.

Still, what I’ve got so far is good. It was a lot of work, but it’s work I only have to do once. That’s the beauty of programming in a nutshell. It’s automation. Sure, I could have done the editing by hand instead of writing scripts to do it for me, and I probably would have been done sooner, but now I won’t have to do it all over again for Nocturne or any other book I write in the future.

To close out this miniseries, I have one more post in mind. This one will look at some of the additional LaTeX packages I used, like the lettrine one I mentioned above. By the time that comes out, maybe I’ll even have another book ready.

Tag: python

Full stack adventures 1: Meet the stack