The SPA vacation

I am a “full-stack” developer, which means a lot of things. Specifically for the purposes of this post, if means that I work with every part of a web application, and those are traditionally divided into three layers. First is the front-end or client-side layer: that’s what you see when you visit the site in your browser. That client connects to the back-end or server-side layer to get and store its data and generally accomplish things. Underneath all that, the database keeps track of everything; it tends to be a special snowflake, so we’ll ignore it for the moment.

Client code, as it must run in a browser, uses a mix of three different languages: HTML for markup, CSS for styling, and Javascript for interactivity and communication. That’s already a lot to know, which is why full-stack developers must be more generalist than specialist. (And, incidentally, why I enjoy the field more than a specialized role.) But then you add in the server, which typically uses a fourth language. That can be PHP, as in my current job, or the Python I like to use for my hobby projects, or Ruby, Java, C#, Elixir, or just about anything. Really, it’s hard to find a language that doesn’t have the capability to be used for the serve side.

That includes Javascript, and the past decade has given us the rise of Node, along with a number of web frameworks based on it, such as Express and Nest. The draw for these is simple: if you use the same programming language on the front and back ends, that’s one less thing you need to learn.

JSON the killer

Node is a great thing. I use it on a daily basis, even if I’m not serving web pages with it. But this encroachment of Javascript into the server realm also brought about the rise of the Single Page Application, or SPA. Instead of the traditional web model of the server sending pages and the client (i.e., browser) rendering them, the SPA takes a different approach. Why not let the client do all the work of creating HTML elements?

It’s simple and even ingenious. The server just offers an API, so theoretically anybody could use it, and the client fetches data from that API, creating HTML on the fly based on a response in the form of JSON. (You could also use XML like in the old days, but…no. Let’s not go there.) Perfect separation of concerns, because why should the server care how the data is presented?

And we got some really great tools out of that, along with a whole new way of looking at the web as an application platform. Angular was the first to hit it big, while React and Vue are the top two now. All these share a focus on “reactive” programming, a style of event-driven programming where data changes automatically when its dependencies do. Very simple, fairly elegant, and…

Build me up, buttercup

…And a total mess to build. Angular, React, and Vue all share that problem, and it’s not getting any better. Building a web application using any of these frameworks is black magic, pure and simple. I’ve been doing this for 4 years now, and I still don’t think I’ve even scratched the surface of Webpack configuration. Seriously, I just use Laravel Mix, or copy a Webpack config file from somewhere else, because I’ve got better things to do with my time. Like, you know, writing the code!

The fault lies in the sheer complexity involved, and the fact that modules (one of the most integral parts of any programming environment since around the 1970s) were, in Javascript’s case, kind of bolted on. So we have to assume someone out there is using an old browser that doesn’t support them, or that we’re in an environment where we can’t load them, which means smashing everything into a single file. Oh, and we want to make that file as small as possible, because bandwidth is still an issue. Add in a minification step, then. Want to use that fancy new API Chrome just added? Well, here’s a polyfill for everything else on the market.

There’s a reason why node_modules is a meme. There’s no valid reason why it should be five hundred megabytes for a “starter” React app. Modularization and the borderline insane dependency lists of a web framework have turned the web dev environment into a mess. Not only that, but it’s nearly impossible to get an app built without running multiple processes simultaneously: Webpack, dev server, maybe a CSS postprocessor, and who knows what else. I understand that you do need a lot of tools to cover various use cases, but can’t we think about optimizing the developer’s experience a little, too?

Zero is better than nothing

When I wrote my first web page in 1996, it was easy. You didn’t need any specialized software, just a text editor, a browser, and maybe an FTP client. Sure, the internet has moved on since then, and pages are a lot more complex. They can do so much more that sites I use every day in 2021 would have been unfathomable to anyone 25 years ago, let alone a nerdy 13-year-old.

But does that increased complexity require a proportional increase in development complexity, or can we get back to making quality pages without all the cruft? In other words, can we make modern web applications without a client-side build process?

Until a few days ago, I didn’t think so. Because that was the received wisdom: modern apps are SPAs, and SPAs need a client framework. And that client framework just has to have a build step. Even Preact, the stripped-down cousin to React that boasts of being easy to install, requires a build step for its templates. Tailwind CSS needs one to cut its 1.7 MB of styling into something both manageable and incredibly sleek.

But…what if we didn’t use frameworks? What if we didn’t use JSON? What if the server sent back HTML instead? That’s exactly what I’m doing in the app I’ve been writing at my day job (as an aside, it still feels weird to write that), so what makes the SPA approach superior?

I’m not the only one thinking that. There’s a small but growing minority of web devs pushing for HTML over the wire, which is precisely what it sounds like. There are libraries that take advantage of this, using the abilities of modern Javascript and the browser platform to patch HTML on the fly.

You’re going to have to do the request/response roundtrip anyway, so cutting out the “turn JSON into HTML” step at the end will only save client time. Since that client might be a cheap Chinese Android phone, you want to save its processing power for more important things like actually being usable. AJAX and the DOM let us do some invasive surgery on a page without requiring a full refresh, but still providing for usable URL history. (In other words, we can make the Back button usable, too!) And it’s that much less Javascript for us to write, because we’re not worrying about computed properties or event handlers.

HTML over the wire takes us back to the simpler times of yore, while letting us use the tools we’ve invented in the meantime. From my perspective, that’s great, especially because it means that, in a lot of cases, you can get by with a front-end build process that looks like this:

  1. Add a script tag to your “base” HTML.
  2. That’s it. You’re done!

About as easy as it gets. So that’s why my redesign of Clave is going to use htmx and Alpine instead of a complicated SPA based on Vue. I’ll still have Tailwind for styling, so I don’t get away completely build-less, but you could swap in Bootstrap or Bulma or something and eliminate even that step.

The web is the center of many lives in this fallen time. We should be doing everything in our power to make it easier to contribute, and I think HTML over the wire is one way to do that on the development side. By making it easier to create complex applications, we’ll see more of them.

Most people would go to a spa for a vacation. I’m taking a vacation from the SPA, because the simpler life is so much better.

Full stack adventures 1: Meet the stack

I have a lot of different development projects. Most aren’t all that great, and some of the older ones are…well, they’re awful. (For the morbidly curious, I have many of them on GitHub, and my newer ones will start to appear on Gitlab.)

My current one, however, has been an adventure. It’s useful, it makes for a good learning experience, and it has a very wide range of technologies. I’m truly working with a full stack on this one, even if I may not be playing with a full deck.

What it is

The project is called Pixeme, a portmanteau of “picture” and “lexeme” that, in my opinion, captures the essence of what I’m trying to create. Basically, I took the old saying “A picture is worth a thousand words” far too literally.

My goal is to make a full web platform for what I describe in the documentation as a community for visual language learning. To put it simply: users post pictures with simple, descriptive captions. Nothing more than a single word or phrase, though they can add more text later. What separates this from, say, Instagram is that other users (or the same one) can add captions for other languages.

For example, I could post a photo of my stepdad’s dog with the caption “a dog”. Then, someone who speaks French might come along and add “un chien” to it. A Japanese user could then add “犬” to the mix, and so on. Together, we build a kind of cross-language picture dictionary. This can then be used by people who really are trying to learn languages and understand that visual reinforcement helps.

(I have plenty of other ideas, things like audio attachments, larger texts, Anki-style flashcards, adding in support for constructed or artificial languages, and even possible research uses, but those are all for much later. Let me get the site started first.)

All in all, Pixeme checks all the boxes for me. It involves programming and linguistics, two topics which any reader of this blog will know are among my favorites. It’s a long-term project, so it requires focus but gives the satisfaction of completion in return. And maybe I can even find some way to monetize it in the future. Even if I can’t, it’s still valuable experience with a number of different languages, frameworks, and libraries.

The stack

Every web application is layered. There’s no way around it. At the very least, you have the server side and the client side, but most modern apps add in multiple extra layers, forming a stack. Pixeme is no exception. For this project, I chose a stack which, for the most part, reflects my personal preferences while also allowing me to stay on top of current developments in both sides of the web equation.

The server-side framework I chose is Flask. I like Python. Even though I feel the 3.x series was a needless break in compatibility, it’s still one of the few languages I feel comfortable writing. I’ve joked before that I can write Python in my sleep—that’s how much experience I have, and how much it fits my mindset.

Flask calls itself a “micro-framework” for web applications, meaning that it comes with very few bells and whistles, but plenty of extensibility. And I like that, too. It’s a little harder to set up, as you have to track down a number of extensions for things like database connectivity, form validation, authentication, etc., but that’s okay. You only pay for what you use, and there’s nothing hidden.

Apart from Flask extensions, the rest of the back end is pretty standard. SQLAlchemy and PostgreSQL, which is pretty much Python Databases 101. Pytest for testing, because tests are important. (I’ll admit that I haven’t always kept those up to date in past projects!) The Marshmallow library for serialization and validation. And the usual host of minor packages for little things. You can’t get away from those.

The front end is where things get interesting. Pixeme has a REST API because that’s the cool thing to do these days, but also because it really does make things easier even when you don’t want an SPA. And I decided after writing half of one that I don’t want another SPA. It just doesn’t fit the vision I have for this platform. Oh, there will be an app eventually, and that’ll have web and mobile versions, but the site itself needs to be the initial focus.

Flask includes the Jinja template engine; you can use your own, but it’s easier to go with the default. So I’m writing the view layer (Pixeme mostly follows the typical MVC architecture, in case you were wondering) as HTML templates using Jinja. That’s sometimes harder than it looks, one reason why this post is the first in a series. We’ll get into some of the difficulties later, though.

HTML isn’t the only part of the web in 2020, so Pixeme has a whole client-side layer, and this is where my focus on minimalism ramps up. As I said, I gave up on an SPA for the initial version of the site. I personally like Vue over React, so that’s where I started. But then I saw how I was duplicating so much of the server functionality, so I decided to back up and make a more traditional site. Still, reactivity is great, so I turned to Alpine.js to fill that niche. It’s not perfect, but it’s very helpful, and I’ll explain that later in the series, too.

Last, CSS. I know what you’re thinking: Bootstrap. Or maybe Bulma or one of the other alternatives. And those are great. I’ve used them before, and I do enjoy them…when they fit. I wanted to try something different with Pixeme, something more customized, so I chose TailwindCSS instead. It takes more setting up, it doesn’t play perfectly well with Jinja (yet another topic for another post), but it suits me. Once I grasped the concepts behind it, it just made so much sense.

So that’s the Pixeme stack. Python, Flask, and Postgres are the server side; what OS the whole thing will run on is still an open question, but bet on some flavor of Linux. Jinja for templating and views. A front end built with Alpine and Tailwind. “Full stack” is always a juggling act, and this might be my most ambitious attempt yet. Stay tuned to see how it works out.

The JavaScript package problem

One of the first lessons every budding programmer learns is a very simple, very logical one: don’t reinvent the wheel. Chances are, most of the internal work of whatever it is you’re coding has already been done before, most likely by someone better at it than you. So, instead of writing a complex math function like, say, an FFT, you should probably hunt down a library that’s already made, and use that. Indeed, that was one of the first advances in computer science, the notion of reusable code.

Of course, there are good reasons to reinvent the wheel. One, it’s a learning experience. You’ll never truly understand the steps of an algorithm until you implement them yourself. That doesn’t mean you should go and use your own string library instead of the standard one, but creating one for fun and education can be very rewarding.

Another reason is that you really might be able to improve on the “standard”. A custom version of a standard function or algorithm might be a better fit for the data you’ll be working with. Or, to take this in another direction, the existing libraries might all have some fatal flaw. Maybe they use the wrong license, or they’re too hard to integrate, or you’d have to write a wrapper that’s no less complicated than the library itself.

Last of all, there might not be a wheel already invented, at least not in the language you’re using. And that brings us to JavaScript.

Bare bones

JavaScript has three main incarnations these days. First, we have the “classic” JS in the browser, where it’s now used for pretty much everything. Next is the server-side style exemplified by Node, which is basically the same thing, but without the DOM. (Whether that’s a good or bad thing is debatable.) Finally, there’s embedded JavaScript, which uses something like Google’s V8 to make JS work as a general scripting language.

Each of these has its own quirks, but they all share one thing in common. JavaScript doesn’t have much of a standard library. It really doesn’t. I mean, we’re talking about a language that, in its standardized form, has no general I/O. (console.log isn’t required to exist, while alert and document.write only work in the browser.) It’s not like Python, where you get builtin functions for everything from creating ZIP files to parsing XML to sending email. No, JS is bare-bones.

Well, that’s not necessarily a problem. Every Perl coder knows about CPAN, a vast collection of modules that contains everything you want, most things you don’t, and a lot that make you question the sanity of their creators. (Question no longer. They’re Perl programmers. They’ve long since lost their sanity.) Other languages have created similar constructs, such as Python’s PyPi (or whatever they’re using these days), Ruby’s gems, the TeX CTAN collection, and so on. Whatever you use, chances are you’ve got a pretty good set of not-quite-standard libraries, modules, and the like just waiting to be used.

So what about JavaScript? What does it have? That would be npm, which quite transparently started out as the Node Package Manager. Thanks to the increase in JS tooling in recent years, it’s grown to become a kind of general JavaScript repository manager, and the site backing it contains far more than just Node packages today. It’s a lot more…democratic than some other languages, and JavaScript’s status as the hipster language du jour has given it a quality that sometimes seems a bit questionable, but there’s no denying that it covers almost everything a JS programmer could need. And therein lies the problem.

The little things

The UNIX philosophy is often stated as, “Do one thing, and do it well.” JavaScript programmers have taken that to heart, and they’ve taken it to the extreme, and that has caused a serious problem. See, because JS has such a small standard library, there are a lot of little utility functions, functions that pop up in almost any sizable codebase, that aren’t going to be there.

For most languages, this would be no trouble. With C++, for instance, you’d link in Boost, and the compiler would only add in the parts you actually use. Java or C#? If you don’t import it, it won’t go in. And so on down the line, with one glaring exception.

Because JavaScript was originally made for the browser—because it was never really intended for application development—it has no capability for importing or even basic encapsulation. Node and recent versions of ECMAScript are working on this, but support is far from universal at this point. Worse, since JavaScript comes as plain text, rather than some intermediate or native binary format, even unused code wastes space and bandwidth. There’s no compilation step between server and client, and there’s no way to take only the parts of a library that you need, so evolutionary pressure has caused the JavaScript ecosystem to create a somewhat surprising solution.

That is the NPM solution: lots of tiny packages with myriad interdependencies, and a package manager that integrates with the build system to put everything together in an optimized bundle. JavaScript, of course, has no end of build systems, which come in and out of style like seasonal fashions. I haven’t really looked into this space in about eight months, and my knowledge is already obsolete! (Who uses Bower anymore? It’s all Webpack…I think. Unless something else has replaced by the time this post goes up.)

This is a prime example of the UNIX philosophy in action, and it can work. Linux package managers do it all the time: for reference, my desktop runs Debian, and it has about 2000 packages installed, most of which are simple C or C++ libraries, or else “data” packages used by actual applications. But I’m not so sure it works for JavaScript.

Picking up the pieces

From that one design decision—JavaScript sent as plain text—comes the necessity of small packages, but some developers have taken that a bit too far. In what other language would you need to import a new module to test if a number is positive? Or to pad the left side of a string? The JS standard library provides neither function, so coders have created npm packages for both, and those are only two of the most egregious examples. (Some might even be jokes, like the one package that does nothing but return the number 5, but it’s often hard to tell what’s serious and what isn’t. Think Poe’s Law for programmers.)

These wouldn’t be so bad, but they’re one more thing to remember to import, one more step to add into the build system. And the JavaScript philosophy, along with the bandwidth requirements its design enforces, combine to make “utility” libraries a nonstarter. Almost nobody uses bigger libraries like Underscore or Lodash these days; why bother adding in all that extra code you don’t need? People have to download that! The same even goes for old standbys like jQuery.

The push, then, is for ever more tiny libraries, each with only one use, one function, one export. Which wouldn’t be so bad, except that larger packages—you know, applications—can depend on all these little pieces. Or they can depend on each other. Or both, with the end result a spaghetti tangle of interdependent parts. And what happens if one of those parts disappears?

You might think that’s crazy, but it did happen. That “left pad” function I mentioned earlier? That one actually did vanish, thanks to a rogue developer, and it broke everything. So many applications and app libraries depended on little old leftpad, often indirectly and without even noticing, that its disappearance left them unable to update, unable to even install. For a few brief moments, half the JavaScript world was paralyzed because a package providing a one-line function, something that, in a language with simple string concatenation, comes essentially for free, was removed from the main code-sharing repository.

Solutions?

Is there a middle ground? Can we balance the need for small, space-optimized codebases with the robustness necessary for building serious applications? Or is NPM destined to be nothing more than a pale imitation of CPAN crossed with an enthusiast’s idea of the perfect Linux distro? I wish I knew, because then I’d be rich. But I’ll give a few thoughts on the matter.

First off, the space requirement isn’t going away anytime soon. As long as we have mobile and home data caps, bandwidth will remain important, and wasting it on superfluous code is literally too expensive. In backwards Third World countries without net neutrality, like perhaps the US by the time this post goes up, it’ll be even worse. Bite-size packages work better for the Internet we have.

On the other hand, a lot of the more ridiculous JS packages wouldn’t be necessary if the language had a halfway decent standard library. I know the standards crew is giving it their best shot on this one, but compatibility is going to remain an issue for the foreseeable future. Yes, we can use polyfills, but then we’re back to our first problem, because that’s just more code that has to be sent down the wire, code that might not be needed in the first place.

The way the DOM is set up doesn’t really help us here, but there might be a solution hiding in there. Speculative loading, where a small shim comes first, checking for the existence of the needed functions. If they’re found, then the rest of the app can come along. Otherwise, send out a request for the right polyfills first. That would take some pretty heavy event hacking, but it might be possible to make it work. (The question is, can we make it work better than what we’ve got?)

As for the general problem of ever-multiplying dependencies, there might not be a good fix. But there also might not be a need to keep the old maxim in mind. Do we really need to import a whole new package to put padding at the beginning of a string? Yes, JS has a wacky type system, but " " + string is what the package would do anyway. (Probably with a lot of extra type checking, but you’re going to do that, too.) If you only need it once, why bother going to all the trouble of importing, adding in dependencies, and all that?

Ultimately, what has happened is that, as JavaScript lacks even the most basic systems for code reuse, its developers have reinvented them, but poorly. As it has a stunted standard library, the third party has had to fill those gaps. Again, they have done so poorly, at least in comparison to more mature languages. That’s not to say that it doesn’t work, because everything we use on the Internet today is proof that it does. But it could be better. There has to be a better way, so let’s find it.

Practical Typescript: dice roller

In the last few “code” posts of Prose Poetry Code, there’s been one thing missing. One very important thing, and you might have noticed it. That’s right: there’s no code! I’ve been writing about generalities and theory and the like for a while now, but I’ve been neglecting the ugly innards. Part of that is because I haven’t been in much of a coding mood these past few months. Writing fiction was more interesting at the time. But I’ve had a few spare hours, and I really do want to get back into coding, so here goes.

A while back, I mentioned Typescript, a neat wrapper that sits on top of JavaScript and generally makes it palatable. And at the end of that post, I said I’d be playing around with the language. So here’s what I’ve got, a practical example of using Typescript. Well, dice rollers aren’t exactly practical—they’re a dime a dozen, and nobody really needs them—but you get the idea.

The code

I’ve posted a full ZIP of the code, including a config file, an HTML skeleton, and the JavaScript output. But here’s the Typescript code (dice.ts) for your perusal.

// Simple function simulating a single die roll
function roll(sides: number): number {
    return 1 + Math.floor(Math.random() * sides);
}

// Our "back-end" will roll a number of simulated dice,
// possibly adding a bonus or subtracting a penalty.
// It will return an object containing the rolls and their total.
// (This is really for no reason other than to show off interfaces.)
interface RollResult {
    rolls: number[],
    total: number
}

// Roll a number of dice, and return the results,
// using the object we defined above.
function rollMany(count: number, sides: number): RollResult {
    let rolls : number[] = [];

    for (let i = 0; i < count; i++) {
        rolls.push(roll(sides));
    }

    let result = {
        rolls: rolls,
        total: rolls.reduce((a: number, b: number) => a+b, 0)
    };

    return result;
}

// This is where the interactive portion of the script begins.
// It's really just basic DOM stuff. In a moment, we'll actually
// hook it all up to the page.
function diceRoller() {
    console.log("Rolling...");

    let countBox = document.getElementById("count") as HTMLInputElement;
    let sidesBox = document.getElementById("sides") as HTMLInputElement;
    let addsBox = document.getElementById("adds") as HTMLInputElement;
    let resultRollsText = document.getElementById("result");
    let resultTotalText = document.getElementById("total");

    let count = +countBox.value;
    let sides = +sidesBox.value;
    let adds = +addsBox.value;

    let result = rollMany(count, sides);
    let totalRoll = result.total + adds;

    resultRollsText.innerHTML = result.rolls.join(" ");
    resultTotalText.innerHTML = ""+ totalRoll;
}

// This clears out our results and options.
function clearAll() {
    console.log("Clearing...");

    // Note that this gives us an HTMLCollection...
    let boxes = document.getElementsByClassName("entry");

    // ...which can't be used like an array in for/of...
    for (let b in boxes) {
        // ...and contains only generic HTMLElements.
        (boxes[b] as HTMLInputElement).value = "";
    }

    let resultText = document.getElementById("result");
    let totalText = document.getElementById("total");
    resultText.innerHTML = "";
    totalText.innerHTML = "";
}

// Here's where we connect our functions to the page.
document.addEventListener("DOMContentLoaded", function () {
    let clearButton = document.getElementById("clear");
    let rollButton = document.getElementById("roll");

    clearButton.onclick = clearAll;
    rollButton.onclick = diceRoller;
});

Honestly, if you’ve seen JavaScript, you should have a pretty good idea of what’s going on here. We start with a helper function, roll, which does the dirty work of generating a number from 1 to N, exactly as if you rolled a die with N sides. (It’s not perfect, as JavaScript uses pseudorandom numbers, but it’s the best we can do.) If you take out the two type declarations, you wouldn’t be able to tell this was Typescript.

Next comes something that inarguably identifies our source as not normal JS: an interface. We don’t really need it for something this simple, but it doesn’t cost anything, and it’s a good check on our typing. Our RollResult interface simply defines an object “layout” that we’ll pass from our back-end rolling function to the front-end output. If we screw up, the compiler lets us know—that’s the whole point of strong typing.

After this is the rollMany function. It builds on the simple roll, calling it multiple times and storing each result in an array. This array, along with the sum of its contents, will become the returned object, matching our interface.

We’ll skip over the diceRoller function briefly, as it’s the grand finale of our little app, and I wanted to save it for last. Instead, we move to clearAll. It would be an unremarkable DOM manipulation function, if not for two quirks in Typescript. First, our simple HTML page defines a few entry boxes: one for the number of dice to roll, one for how many sides each die has, and one for a flat bonus or penalty to add to the total. These are <input> text elements, and they all have a CSS class of entry. Iterating over them (and only them) should be a piece of cake, right?

Not quite. See, the obvious way to do this, document.getElementsByClassName, returns an HTMLCollection, which isn’t quite the same thing as a JavaScript array. And Typescript, unless you tell it to spit out the latest and greatest ECMAScript standard (and I haven’t), won’t let you use the easier for..of loop. Instead, you have to use for..in, which iterates over the keys of a collection—for arrays, these are the indexes.

And that brings us to our second problem. See, our DOM collection is full of HTMLElements. These could be anything at all, and are thus usable as essentially nothing. In particular, we can’t change the value property that all input textboxes have, because not every HTML element has them. As any Java or C# programmer knows, we need an explicit cast into the more specific type of HTMLInputElement. That seems like needless drudgery, but it pays off in more complex applications, and we could always use jQuery or something instead. In a “real” setting, we probably would be.

The areas where we output our rolls and their total are simple divs. We don’t have to do any casting with them; we can just clear their innerHTML properties. And the last bit is not much more than your usual “let’s set up some event handlers” block.

That leaves diceRoller. First up are a few variables to hold the DOM elements: 3 text boxes and 2 output areas. Again, we have to do a cast on anything that’s an input element, but that should be old hat by now.

Following that, we make a few variables that hold our input values in number form. I used the idiomatic “unary plus” conversion from string to number here.

Next comes result, which holds (naturally) our resulting roll. I threw in a totalRoll variable to hold the total (including the “adds” roll bonus/penalty), but you don’t really need it; you can calculate that as you’re putting it into the output field. Speaking of which, that’s where we end: joining the result array into a space-separated string (you could use commas or whatever), then putting that and the total into their proper places.

Conclusion

So Typescript isn’t that hard. Counting comments and blank lines, this little thing weighs in at about 80 lines. The resulting JavaScript is less than 50. The difference comes from the strong typing, an interface definition that is purely for the benefit of the Typescript compiler and the programmer, and a few other odds and ends that contribute nothing physical to the finished product.

It may seem silly. We’re writing more code, locking ourselves into the type system, and we’re not really getting much out of it. Sure, we’ve got protection from typos. If I’d misspelled the name of one of the RollResult fields, I’d get an error telling me where I went wrong, but that’s about it.

For something this simple, even that’s enough. I get that error when compiling. I don’t end up with a blank screen or unresponsive page and no indication as to why. JavaScript’s dynamic nature is great, but it’s also terrible. In coding, unlike in real life, there’s such a thing as too much freedom.

Now, if you like, feel free to take this tiny app and improve on it. The HTML, for instance, expects a stylesheet; you could make one. Add in some bells and whistles. Throw up a few preset buttons. Store the last 100 rolls in DOM local storage. Most of all, have fun with it.

First glance: Scala.js

I’ve learned and used a lot of different programming languages. If I had to pick one to take as the moneymaker, it’d probably be JavaScript right now. But JavaScript is pretty awful. It wouldn’t be my first choice if money was no object. That would likely be C++, but there’s one other contender, one language I liked from the first time I saw it. That language is Scala.

Scala has its problems, of course. It’s closely tied to the Java ecosystem, which…isn’t the greatest. Sure, it’s got a huge selection of libraries and frameworks for whatever you want to do, but the Java language itself is atrocious, the JVM has a well-deserved reputation for bloat, and it’s practically banned on the desktop. (And you run the risk of getting sued if you use it on mobile, as Google discovered.) So Scala isn’t exactly a popular option, for very good reason.

But it doesn’t have to be that way. Plenty of languages have been turned into tools for web development by the clever trick of compiling them into JavaScript. For Scala, that would be the job of Scala.js. It’s a relative newcomer (version 0.6.14 at the time of this writing), but it’s already looking good.

What is it?

Scala.js is, simply enough, a tool for turning Scala code into JavaScript. In that, it’s not much different from “transpilers” like TypeScript or Babel. Only the source language isn’t something vaguely reminiscent of the JS it will become. No, you write in Scala, so you have all the expressive power of that language.

I like Scala. It’s one of the few languages I’ve seen that accomplishes the feat of getting me interested in functional-style programming. I absolutely hate the “pure functional” approach of, e.g., Haskell, where you can’t actually do anything, because all those nasty “impurities” like, say, I/O, are restricted, and variables can’t, you know, vary. (Yeah, you can use monads, but they’re limited in how they interact with the rest of your code. That’s kinda the point.) Scala, instead, makes FP style encouraged, but not required. That’s much better.

On top of that, Scala is just a better Java than Java. It’s got most of what Gosling took out as “too hard”, like operator overloading. It’s got pattern matching. Lambdas exist, and they’re easy to write. Scala is what Java programmers wish they could use. Putting it in the browser? Sure.

The good

So we’ve got a better language, and that’s where most of the good comes from. Like what, you may ask? How about strong typing? Sure, you can get that in TypeScript, but Scala uses it to its full extent. The functional style is front and center, and it’s not hidden behind bad syntax like in JS. OOP is class-based, not prototype-based, and you don’t have to worry about browser compatibility. And then there are a lot of things Scala has that just aren’t feasible in JS, like case classes and traits.

Outside of the language proper, you’ve got a good HTML templating system, Scalatags, that has full type safety. And it’s not an addon, not in the JS sense. No, you’re working with something that’s based on Java, remember? You’ve got Java-derived build tools and libraries.

I could go on, but the main advantages of Scala.js come from the simple fact that you’re using Scala in place of JavaScript. Everything else great about it follows naturally from that. If you like Scala, that should be enough to give it a shot. If you hate JavaScript, same deal.

The bad

But it’s not all good. That same Java heritage is the greatest weakness of Scala.js. You’re using Java tools, a Java-based API, JVM-centric constructs. Not a fan of Java? Tough. You really can’t get away from it with Scala. (Really, though, this isn’t that much different from that thing Google did a few years ago. JWT? I can’t remember, and I’m too lazy to look it up.)

A worse shortcoming is the number of limitations Scala.js has on what Scala code you can use. Mostly, this makes sense—you can’t use parallel programming in the inherently single-threaded environment of the browser. You also can’t use any Java libraries; your code, and all the code you import, has to be pure Scala, with a very few exceptions. And then there are a few nits to pick, cases where the developers of Scala.js couldn’t figure out a way to make the construct work in JS. Reflection is one of these: you can’t use it at all, and you can’t use anything that uses it. Macros (the recommended replacement) are a poor substitute for reflection, just as templates are in C++.

Lastly, you’ve got the build environment. If you’ve used any sort of JavaScript tools made this decade, you’re probably familiar with Node, Grunt, Gulp, and the like. Scala.js inherits Scala’s sbt, which is yet another thing to learn. And IDEs don’t seem to like it very much; I’ve never managed to get an sbt-based project talking to an IDE, and Eclipse, Netbeans, and IDEA all laughed at my attempts to use sbt for their projects. Since Scala, like Java, really wants an IDE, this can be a problem. (Maybe they’ve fixed things since then, though.)

The verdict

Although the “bad” section outweighs the “good” in quantity, don’t let that make you think I don’t like Scala.js. I like the idea of it. I can even like the implementation. File sizes are a bit on the hefty side, but they’re working on that. Some of the API limitations can be changed. The thing’s still in beta, remember.

On the whole, if you can live with the restrictions it places on you, Scala.js looks to be a very interesting way of creating web apps without writing raw JavaScript. I know I’m keeping an eye on it.

Playing with TypeScript

JavaScript sucks, and we all know it. But it’s the lingua franca of the web (not like WebAssembly is ever taking off), so we have to learn to deal with it. That doesn’t, however, mean we have to write it. Oh, no. There’s a whole subculture of languages that compile into JavaScript, meaning you can write your code in something that looks sane (to you) without worrying about the end result being readable.

This trend really started a few years ago with CoffeeScript, which did an admirable job of making JS look like Ruby. From there, similar projects spun off, like Coco and LiveScript, which were far better in that they didn’t have anything to do with Ruby. And then Microsoft got involved. That’s where TypeScript comes from. Yes, that Microsoft.

But it’s not as bad as it sounds. The language and compiler use the Apache license, and that gives us a bit of protection from the worst of corporate silliness. And that means it might be interesting to look at TypeScript for what it is: an attempt at making a better JavaScript than JavaScript.

Up and running

TypeScript fits into the usual Node-hipster-modern ecosystem. You can install it through npm, and it works with most of the other big JavaScript tools like testing systems. Some of the big “web app” frameworks even use it by default now, meaning that I’m a little behind the times. Oh, and you can also use it with Visual Studio, but I can’t, because that’s Windows-only. (And VS Code is not the same.) But you don’t have to: the homepage even has a link to TypeScript support for the best text editor out there. I’ll leave it to you to decide which one that is, but I’ll tell you that it’s the same one I’m using to write this post.

My type

So you get TypeScript installed, and you’ve got a nifty little compiler that spits out perfectly fashionable JavaScript, but what about the source language? What’s so special about it? In a word: types. (You’ll note that this word makes up half the language’s name.)

Essentially, TypeScript is nothing more than JavaScript with strong typing. That may not seem like much, but it’s actually a huge change that alters the whole way you write code. Variables are typed, functions are typed, objects are typed. To be fair, they all are in JavaScript, too, but that language doesn’t do anything with those types. TypeScript is made to use them. By itself, that’s almost enough to overcome my instinctive hatred of all things Microsoft. Almost.

On top of the bare bones JavaScript gives us, we’ve got tuples, enums, generics (C#-like, so not the best we could get, but far better than what JS offers), and an any type that lets us ignore the type system when necessary. If you wanted to, you could just about get away with writing regular JavaScript, tagging everything as any, and running it through the TypeScript compiler. But why would you do that? You’re supposed to be using a better language, right? So use it! Besides, like any decent strongly-typed language, we can use type inference to save us most of the trouble of specifying what something should be.

The next level

Just having the added safety of stronger typing already gives TypeScript a leg up on its parent language. But if it didn’t offer anything else, I’d tell you not to bother. Fortunately for this post, it has a lot more.

JS is currently in a state of flux. Most (but not all) browsers support ES5, some let you use a lot of the new ES6 features, and the language standard itself has transitioned to the ridiculous rapid-release model that’s all the rage today, so it’s looking like we’ll soon be back in the bad old days of “Best viewed in {browser X}”. Polyfills can only take us so far, you see. For coders writing raw JavaScript, it’s a serious problem making something maximally compatible, and most just throw their hands up and say, “Just use Chrome like everyone else.”

That’s another case where I’ll give the MS team credit. TypeScript takes a lot of the newer JS additions and lets you use them now, just like Babel or other “transpilers”. You can declare your variables with let instead of var, and the compiler will do the right thing. You get the new class syntax for free, which is great if you never could wrap your head around prototype OOP. Generators aren’t even in JS yet, so TypeScript even looks a bit like the future in some cases.

Reservations

If there’s anything wrong with TypeScript, it’s not the kind of thing that shows up in a cursory inspection of the documentation. No, the main problems are twofold. One, it’s a project started by Microsoft, a company with a history of bad blood towards the open source community. The Apache license helps in that regard, though I don’t think it can ever completely alleviate some people’s fears.

Second, TypeScript has been around for a while now, but it’s only recently been picking up steam. Angular and React both like it a lot, as do some indie game engines like Phaser. But the JS community is fad-driven, and this new acceptance could be an indication of that. If TypeScript is simply “the next big thing”, then interest will fade once some other shiny thing catches the eyes of the hipsters. We can’t prevent that, but we can do our best to ignore it.

Will TypeScript become the future of web development? I can’t say. It’s definitely one option for the present, though. And it’s a pretty good option, from what I’ve seen. I think I’ll play around with it some more. Who knows? Maybe I’ll show you what I’ve made.

Wrapping text by code

Very many games require displaying significant amounts of text on the screen. Whether dialogue, signs, backstory “data log” entries, or lists of quests and objectives, text is absolutely a necessity. And we’re not talking about just a word here and there. No, this is serious text, entire sentences at least. More specifically, I want to look at the long strings of text that are commonly shown in dialog boxes or other GUI/HUD elements, not something that might be baked into a texture.

With single words, it’s not hard at all: you just throw the word on the screen at the right position. What game engine doesn’t let you do that? Even my failed attempt at a simple 2D engine a few years ago could manage that much!

It’s when you need to display longer snippets that things get hairy. Your screen is only so wide, so, like any word processor, you have to know when to wrap the text. Now, many engines do this for you, but you may not be using one of those, or you may be writing your own. So this post is designed to give you an idea of what lies ahead, and to provide a starting point for your own efforts.

Ground rules

First, let’s get this out of the way: I’m using a bit of pseudocode here. Actually, it’s JavaScript, but with a lot of references left undefined. Hopefully, you can convert that into your language and platform of choice.

Next, I’m going to assume you’re working in English or another language that uses an alphabetic script. Something like Chinese or Japanese will have its own rules regarding how long text is broken up, and I don’t entirely understand all of those. The same goes for Arabic and other RTL scripts, but they’re much closer to American and European usage, so some of the concepts described here might carry over.

We’ll also ignore hyphenation in this post. It’s a hideously complex topic, first of all, and nobody can quite agree on which words should (or even can) be hyphenated at the end of a line. Besides, what was the last game you saw that used printed-style hyphenation? In the same vein, we’ll be ignoring non-breaking spaces, dashes, and the like.

The last assumption I’m going to make is that we’re working with Unicode strings. That shouldn’t matter too much, though. We won’t be delving into individual bytes—if you do that with Unicode, you’ve gone wrong somewhere, or you’re at a far lower level this post.

Words

On the screen, our string will be divided into lines. Thus, our job is clear: take a string of text, divide it into lines so that no line is longer than the width of our dialog box. Simple, right? So let’s see how we can go about that.

English text is divided first into sentences. These sentences are then divided into words, sometimes with added punctuation. Each word is separated from its neighbors by blank space. It’s that space that lets us wrap our text.

For a first approximation, let’s take the example of fixed-width text. This was common in older games, and it’s still used for a retro feel. (And programmers demand it for editing code.) It’s “fixed” width because each letter, each bit of punctuation, and each space will have the same width. That width can be different from one font to another, but every character in a fixed-width string, even the blank ones, will be as wide as any other.

As we won’t be worrying about hyphenation, every line will have a whole number of words. Thus, our first task is to get those words out of our string:

// Split a string of text into a list of words
function splitWords(str) {
    return str.split(' ');
}

Basically, we’ve split our string up into words, and removed the spaces in the process. Don’t worry about that. We’ll put them back in a moment. First, we need to go on a little digression.

Building with boxes

Do you remember magnetic poetry? Those little kits you could buy at the bookstore (back when bookstores were a thing)? They came with a set of pre-printed words and a bunch of magnetic holders. Some of them even let you cut your own magnets, then slide the bits of laminated paper containing words into them. However you did it, you could then assemble the words into a short, tweet-level string that you could stick to your fridge. Longer words required longer magnets, while something like “a” or “I” took up almost no space at all. If you used one of the magnetic poetry kits that came with a little board, you had to do a bit of shuffling—and maybe some editing—to get the words to fit on a line.

That’s what we’re doing here. We’re taking the words of our text, and we’re assembling them on the screen in exactly the same way. Like the board, the screen is only so wide, so it can accommodate only so much. Any more, and we have to move to the next line. So wrapping text can be thought of like this. (It’s also the beginnings of computerized typesetting. We’re not really doing anything different from TeX; it just does the job a whole lot better.)

So we need to pack our words onto lines of no more than a given width. One way to do that is incrementally, by adding words (and spaces between them) until we run out of space, then moving on to the next line.

// Build lines of fixed-width text from a string
function buildTextLines(str, screenWidth, fontWidth) {
    let allWords = splitWords(str);
    let lines = [];

    let currentLine = "";
    let currentLineWidth = 0;
    let currentWord = "";

    while (allWords.length > 0) {
        currentWord = allWords.shift();

        let currentWordWidth = currentWord.length * fontWidth;

        if (currentLineWidth + currentWordWidth > screenWidth) {
            // Too long; wrap text here
            // Save the line we just finished, and start a new one
            lines.push(currentLine);
            currentLine = "";
            currentLineWidth = 0;
        }

        currentLine += currentWord;
        currentLineWidth += currentWord.length * fontWidth

        // Try to insert a space, unless we're at the edge of the box
        // (width of space is the same as anything else)
        if (!(currentLineWidth + fontWidth > screenWidth)) {
            currentLine += " ";
            currentLineWidth += fontWidth;
        }
    }

    return lines;
}

For fixed width, that’s really all you have to do. You can still treat the text as a sequence of characters that you’re transforming into lines full of space-delimited words. As a bonus, punctuation carries over effortlessly.

Varying the width

The real fun begins when you add in variable width. In printed text, an “m” is bigger than an “n”, which is wider than an “i”. Spaces tend to be fairly narrow, but they can also stretch. That’s how justified text works: extra bits of space are added here and there, where you likely wouldn’t even notice them, so that the far ends of the lines don’t look “ragged”. (That, by the way, is a technical term.) Two words might have an extra pixel or two of space, just to round out the line.

Doing that with simple strings is impossible, because I don’t know of any programming language having a basic string type that accounts for proportional fonts. (Well, TeX does, because that’s its job. That’s beside the point.) So we have to leave the world of strings and move onward. To display variable-width text, we have to start looking at baking it into screen images.

Since every engine and renderer is different, it’s a lot harder to give a code example for this one. Instead, I’ll lay out the steps you’ll need to take to make this “box” model work. (Note that this phrase wasn’t chosen by accident. HTML layout using CSS, as in a browser, uses a similar box-based display technique.)

  1. Using whatever capabilities for font metrics you have, find the minimum width of a space. This will probably be the width of the space character (0x20).

  2. Using this as a guide, break the text into lines—again based on the font metrics—but keep the words for each line in a list. Don’t combine them into a string just yet. Or at all, for that matter.

  3. For each line, calculate how much “stretch” space you’ll need to fill out the width. This will be the extra space left over after adding each word and the minimum-width spaces between them.

  4. Divide the stretch space equally among the inter-word spaces. This is aesthetically pleasing, and it ensures that you’ll have fully justified text.

  5. Render each word into the sprite or object you’re using as a container, putting these stretched spaces between them. You can do this a line at a time or all at once, but the first option may work better for text that can be scrolled.

In effect, this is a programmatic version of the “magnetic poetry” concept, and it’s not too much different from how browsers and typesetting engines actually work. Of course, they are much more sophisticated. So are many game engines. Those designed and intended for displaying text will have much of this work done for you. If you like reading code, dive into something like RenPy.

I hope this brief look whets your appetite. Text-based or text-heavy games require a bit more work for both the developer and the player, but they can be rewarding on a level a fully voiced game could never reach. They are, in a sense, like reading a story, but with a bit of extra visual aid. And games are primarily a visual medium.

Statistical sampling in JavaScript

It’s 2016, and that’s an election year, which means we’ll be spending the rest of the summer (and half of the fall) watching America’s most ridiculous spectator sport. The pundits and panels and polls are all quite fun, but I find that the methodology is far more interesting than the results.

One of the greatest weapons in the pollster’s arsenal is sampling, and one of those footnotes you’ll see in opinion polls in the coming weeks is the margin of error. These are basic concepts of statistics, but most people might not know how they work. Worse, some programmers might not. So here’s my attempt at rectifying that.

Definitions

Sampling is, in effect, a way of drawing conclusions about a population (such as a state or country) based on surveying only a small fraction of its members. It’s not perfect, but it turns out that, say, 500 people are actually a pretty good indicator of the rest of the nation…as long as you pick the right 500 people. In terms relatable to current events, a presidential poll that only asks people from rural parts of the South is going to get very different results from one that surveys nothing but New York City. That’s selection bias, and it’s one of the hardest things for pollsters to avoid. They’ve got a few ways around it, such as cold-calling random phone numbers, but it’s always a battle.

That very randomness is why sampling works in the first place. If you truly choose your data points (i.e., the people you ask) randomly, then they will, when put together, approximate the “true” nature of things. The more you get, the closer your picture is to the real thing. Eventually, as you’d expect, your sample size approaches the total population; at that limit, the sample obviously represents the whole to perfection.

For smaller samples, however, things aren’t so perfect. Let’s say we have two candidates: H and T. (You get no points for guessing what those stand for.) In “reality”, they aren’t quite even. Say that H has 50% of the vote, T has 45%, and the remaining 5% are undecided, independent, or whatever. Now, take some sort of random number generator and set it to give numbers from 1 to 100. Everything up to 50 is a “vote” for candidate H, 51-95 are for T, and 96-100 are undecided.

After a single generated number, you’ve got a score of 1 for one of the three, 0 for the other two. Not very predictive, but keep going. With a sample size of 10, my results were H 6, T 3, and 1 undecided. Yours might be different, but notice that that’s already looking a lot closer to the true numbers. Give it 100 tries, and it’s probably even better. (Doing this three times with a different generator, my results were: 52 T, 44 H, 4 undecided; 48 T, 47 H, 5; and 57 T, 40 H, 3. Clearly, this RNG leans right.)

The larger the sample size, the more likely the sample will match the population. If you don’t mind a bit of math, then we can look at just how good a match we can get. The basic formula is e = 1 / sqrt(N), where N is the sample size and e is the margin of error. So, for our sample size of 100 above, the math says that our expected error is somewhere within 1/sqrt(100) = 1/10 = 0.1, or 10% either way. Or, as the polls put it, ±10%. Most samples like this are assumed to be conducted at a 95% confidence interval; this basically means that there’s a 95% chance that the true results lie within that margin. (Note, however, that our third poll in the last paragraph didn’t. It’s an outlier. They happen.)

As counter-intuitive as it may seem, this formula doesn’t really depend on the population size at all, as long as the sample is sufficiently small in relation. For national polls surveying a thousand or so people, that assumption holds, so they can safely tout a margin of error of ±3% from their sample of 1,016.

The code

Now we’ll look at how you can do your own sampling. This isn’t just for opinion polls, though. Any kind of analytics could make use of sampling.

The basic function, in JavaScript, would look something like this:

/*
 * Select `k` random choices from a population
 */
function sample(population, k) {
    var psize = population.length;
    var choices = new Set();
    var result = [];

    // Choose `k` elements from the population, without repeats
    for (var i = 0; i < k; i++) {
        var ch;

        do {
            ch = Math.trunc(Math.random() * psize);
        } while (choices.has(ch))

        choices.add(ch);
    }

    for (var c of choices) {
        result.push(population[c]);
    }

    return result;
}

As always, this isn’t the only way to do what we’re trying to do, but it’s very close to what Python’s random.sample function does, so the idea is sound. To get our sample, we generate a number of array indexes, and the Set guarantees we won’t get any repeats. Our result is a new array containing only those elements we chose. We can then do whatever we need.

But how do we determine what sample size we need? Well, one way is to work backwards from the margin of error we want. Remember that this usually won’t depend on the size of the population.

/*
 * Given a desired margin of error,
 * find an appropriate sample size.
 */
function sampleSize(margin) {
    return Math.round(1 / (margin*margin), 0);
}

This is nothing more than a rearrangement of our formula. Instead of saying e = 1 / sqrt(N), we move things around to solve for N: N = 1 / e^2. Rounding to the nearest integer is just for convenience.

In a nutshell, that’s all there is behind those opinion polls you’ll be hearing so much about over the coming months. Pick enough people at random, and you’ll get a picture of the general populace. It’ll be a blurry picture, but even that’s enough to make out some of the larger details.

First glance: Superpowers

It seems like each new day brings us a new tool for game development, whether it’s an engine, a framework, a library, or any of a number of other things. Best of all, many of these up-and-coming dev tools are open source and free, so even the poorest game makers (like me!) can use them. And the quality is only going up, too. No longer must indies be content with alpha-level, code-only engines, uncompiled libraries, and NES-era assets. No, even the zero-cost level of game development is becoming packed with tools that even professionals of a few years ago wished they could have had.

The one I’m looking at today is called Superpowers, by Sparklin Labs. It’s yet another HTML5 game maker that has recently been released as open source software, under the ISC license. (ISC is functionally equivalent to MIT or “new” BSD; basically, it’s not much more than “do what you want, but give us credit”.) It’s not entirely a volunteer effort, and there are a couple of options for supporting it. Their download host, indie game publisher itch.io, gives you a donation option, but the primary way to send money is through Patreon. (There’s a link on the Superpowers main page.)

Let’s take a look

What does Superpowers bring to the table? Well, first of all, it’s an HTML5 engine. The maker itself runs as a nativized web app, and games can be compiled into standalone apps or exported in a browser-friendly format. There’s also a mobile option using the Intel XDK, but I haven’t really looked into that.

Second, and even more important, is the fact that this engine comes with a visual editor. That’s something sorely lacking in the free HTML5 arena. Granted, it’s not exactly up to the level of the editors for Unity or Unreal, but it’s much better than what we had, i.e., not much. It’s got all the usual bells and whistles: tree-based scene management, component editors (these seem a little buggy on my machine, but that’s probably just a local thing), drag-and-drop actors, and so on. For what’s technically still a beta, it’s pretty nice.

Coding works about the same way. You can attach scripts to the various parts of a scene, and they’ll know what to do. The whole thing is mostly behavior-driven, following the component style that is so popular these days. The scripts themselves are written in TypeScript, and I’m a little ambivalent about that. On the one hand, it’s an easier way of writing JavaScript (Superpowers is HTML5-based, so it’s going to end up as JavaScript eventually). On the other, TypeScript is a Microsoft project, so you never know what might happen.

One of the big features that looks interesting is the collaboration support. The Superpowers “app” has a client-server architecture, and it takes advantage of it. When you start it, it creates a server. Now, that’s pretty common in Node applications, but Superpowers actually uses it. After a little initial setup, you can have other people connect to your editor instance and work with you in real-time. I can’t tell you how well that works, since I’m just a lonely guy, but if it comes anywhere close to what’s advertised, then…wow.

There’s a lot more than this, and what I’m seeing looks very good. There’s support for prefabs (like those in Unity) in the editor, for instance, and the engine has all the usual suspects: 2D physics, multiple cameras, etc. Debugging works like in Chrome, since the whole thing runs on NW.js. (IMO, Chrome is a horrible browser, but an okay wrapper for web apps. The developer tools aren’t half bad, though.)

That’s not to say the Superpowers is perfect. Far from it. It’s early in development, and there’s bound to be a few unsquashed bugs here and there. There’s also the TypeScript dependency I mentioned above, but they’re working on that; the developers have an alpha (I think) version of the editor using Lua and the LÖVE engine. And, being on GitHub, I noticed a “Code of Conduct” file, which could be worrisome to free-speech advocates like myself. Also, there’s no online API documentation. You’re supposed to use the editor’s built-in docs. The developers’ reasoning (it boils down to “But there might be plugins!”) sounds weak to my ears. Every other HTML5 engine can do it, so why not this one?

In the end, I think the good outweighs the bad. Give it some time, and Superpowers might become one of the go-to tools for making indie games. Or it could bomb, and we’ll never hear from it again. I doubt that, though. Give me some proper online API docs, support for multiple languages (including pure JavaScript, preferably of the ES6 variety), and quite a bit more polish, and I’ll gladly put it up there with Phaser at the top of the list. For now, I’ll definitely be keeping an eye on this one.

Randomness and V8 (JS)

So I’ve seen this post linked in a few places recently, and I thought to myself, “This sounds interesting…and familiar.”

For those of you who don’t quite have the technical know-how to understand what it means, here’s a quick summary. Basically, the whole thing is a flaw in the V8 JavaScript engine’s random number generator. V8, if you don’t know, is what makes JavaScript possible for Chrome, Node, and plenty of others. (Firefox uses a different engine, and Microsoft’s browsers already have far bigger problems.) In JavaScript, the RNG is accessed through the function Math.random(). That function is far from perfect as it is. There’s no need to make it worse.

But V8, until one of the newest versions (4.9.40), actually did make it worse. An outline of their code is in the post above, but the main problems with it are easy to explain. First, Math.random() returns JavaScript numbers—i.e., 64-bit floating-point numbers—between 0 and 1. The way those numbers work leaves the algorithm 52 bits to play with, but V8’s code worked by converting a 32-bit integer into a floating-point number. That’s a pretty common operation, and there’s nothing really wrong on the face of it. Well, except for the part where you’re throwing away 20 out of your 52 random bits.

Because of the way V8’s RNG algorithm (MWC1616), this gets even better. MWC stands for “multiply with carry”, an apt description of what we’re dealing with. Internally, the code has two state variables, each a 32-bit unsigned integer, or uint32_t. These start off as seeded values (JavaScript programmers have no way of influencing this part, unfortunately), and each one undergoes a simple transformation: the low 16 bits are multiplied by one of two “magic” constants, then added to the high 16 bits. The function then creates its result in two parts, with the upper half of the result coming from one state variable’s lower half, while the lower 16 bits are taken from the other state’s upper half.

The whole thing, despite its shell-game shifting of bits, is not much more than a pair of linear congruential generators welded together. LCGs have a long history as random generators, because they’re easy to code, they’re fast, and they can give okay randomness for simple applications. But now that JavaScript is being used everywhere, the cracks are starting to appear.

Since V8’s Math.random() implementation uses 32-bit numbers and none of the “extra” state found in more involved RNGs, you’re never getting more than 2^32^ random numbers before they start repeating. And I do mean repeating, as linear congruential generators are periodic functions. Given the same state, they’ll produce the same result; generate enough random numbers, and you’ll repeat a state, which restarts the cycle. But that 2^32^ is a maximum, and you need some planning to get it. The magic numbers that make an LCG work have to be chosen carefully, or you can sabotage the whole thing. All the bit-shifting tricks are little more than a distraction.

So what can you do? Obviously, you, as a user of Chrome/Node/V8/whatever, can upgrade. The new algorithm they’re using is xorshift128+, which is highly regarded as a solid RNG for non-cryptographic work. (If you’re interested in how it works, but you don’t think you can read C++, check out the second link above, where I roll my own version in JavaScript.) Naturally, this doesn’t fix all the other problems with Math.random(), only the one that caused V8’s version of it to fail a bunch of the statistical tests used to quantify how “good” a specific RNG is. (The linked blog post has a great visualization to illustrate these.) Seeded, repeatable randomness, for example, you’ll still have to handle yourself. But what we’ve got is good enough for a lot of purposes, and it’s now a better foundation to build upon.