The JavaScript package problem

One of the first lessons every budding programmer learns is a very simple, very logical one: don’t reinvent the wheel. Chances are, most of the internal work of whatever it is you’re coding has already been done before, most likely by someone better at it than you. So, instead of writing a complex math function like, say, an FFT, you should probably hunt down a library that’s already made, and use that. Indeed, that was one of the first advances in computer science, the notion of reusable code.

Of course, there are good reasons to reinvent the wheel. One, it’s a learning experience. You’ll never truly understand the steps of an algorithm until you implement them yourself. That doesn’t mean you should go and use your own string library instead of the standard one, but creating one for fun and education can be very rewarding.

Another reason is that you really might be able to improve on the “standard”. A custom version of a standard function or algorithm might be a better fit for the data you’ll be working with. Or, to take this in another direction, the existing libraries might all have some fatal flaw. Maybe they use the wrong license, or they’re too hard to integrate, or you’d have to write a wrapper that’s no less complicated than the library itself.

Last of all, there might not be a wheel already invented, at least not in the language you’re using. And that brings us to JavaScript.

Bare bones

JavaScript has three main incarnations these days. First, we have the “classic” JS in the browser, where it’s now used for pretty much everything. Next is the server-side style exemplified by Node, which is basically the same thing, but without the DOM. (Whether that’s a good or bad thing is debatable.) Finally, there’s embedded JavaScript, which uses something like Google’s V8 to make JS work as a general scripting language.

Each of these has its own quirks, but they all share one thing in common. JavaScript doesn’t have much of a standard library. It really doesn’t. I mean, we’re talking about a language that, in its standardized form, has no general I/O. (console.log isn’t required to exist, while alert and document.write only work in the browser.) It’s not like Python, where you get builtin functions for everything from creating ZIP files to parsing XML to sending email. No, JS is bare-bones.

Well, that’s not necessarily a problem. Every Perl coder knows about CPAN, a vast collection of modules that contains everything you want, most things you don’t, and a lot that make you question the sanity of their creators. (Question no longer. They’re Perl programmers. They’ve long since lost their sanity.) Other languages have created similar constructs, such as Python’s PyPi (or whatever they’re using these days), Ruby’s gems, the TeX CTAN collection, and so on. Whatever you use, chances are you’ve got a pretty good set of not-quite-standard libraries, modules, and the like just waiting to be used.

So what about JavaScript? What does it have? That would be npm, which quite transparently started out as the Node Package Manager. Thanks to the increase in JS tooling in recent years, it’s grown to become a kind of general JavaScript repository manager, and the site backing it contains far more than just Node packages today. It’s a lot more…democratic than some other languages, and JavaScript’s status as the hipster language du jour has given it a quality that sometimes seems a bit questionable, but there’s no denying that it covers almost everything a JS programmer could need. And therein lies the problem.

The little things

The UNIX philosophy is often stated as, “Do one thing, and do it well.” JavaScript programmers have taken that to heart, and they’ve taken it to the extreme, and that has caused a serious problem. See, because JS has such a small standard library, there are a lot of little utility functions, functions that pop up in almost any sizable codebase, that aren’t going to be there.

For most languages, this would be no trouble. With C++, for instance, you’d link in Boost, and the compiler would only add in the parts you actually use. Java or C#? If you don’t import it, it won’t go in. And so on down the line, with one glaring exception.

Because JavaScript was originally made for the browser—because it was never really intended for application development—it has no capability for importing or even basic encapsulation. Node and recent versions of ECMAScript are working on this, but support is far from universal at this point. Worse, since JavaScript comes as plain text, rather than some intermediate or native binary format, even unused code wastes space and bandwidth. There’s no compilation step between server and client, and there’s no way to take only the parts of a library that you need, so evolutionary pressure has caused the JavaScript ecosystem to create a somewhat surprising solution.

That is the NPM solution: lots of tiny packages with myriad interdependencies, and a package manager that integrates with the build system to put everything together in an optimized bundle. JavaScript, of course, has no end of build systems, which come in and out of style like seasonal fashions. I haven’t really looked into this space in about eight months, and my knowledge is already obsolete! (Who uses Bower anymore? It’s all Webpack…I think. Unless something else has replaced by the time this post goes up.)

This is a prime example of the UNIX philosophy in action, and it can work. Linux package managers do it all the time: for reference, my desktop runs Debian, and it has about 2000 packages installed, most of which are simple C or C++ libraries, or else “data” packages used by actual applications. But I’m not so sure it works for JavaScript.

Picking up the pieces

From that one design decision—JavaScript sent as plain text—comes the necessity of small packages, but some developers have taken that a bit too far. In what other language would you need to import a new module to test if a number is positive? Or to pad the left side of a string? The JS standard library provides neither function, so coders have created npm packages for both, and those are only two of the most egregious examples. (Some might even be jokes, like the one package that does nothing but return the number 5, but it’s often hard to tell what’s serious and what isn’t. Think Poe’s Law for programmers.)

These wouldn’t be so bad, but they’re one more thing to remember to import, one more step to add into the build system. And the JavaScript philosophy, along with the bandwidth requirements its design enforces, combine to make “utility” libraries a nonstarter. Almost nobody uses bigger libraries like Underscore or Lodash these days; why bother adding in all that extra code you don’t need? People have to download that! The same even goes for old standbys like jQuery.

The push, then, is for ever more tiny libraries, each with only one use, one function, one export. Which wouldn’t be so bad, except that larger packages—you know, applications—can depend on all these little pieces. Or they can depend on each other. Or both, with the end result a spaghetti tangle of interdependent parts. And what happens if one of those parts disappears?

You might think that’s crazy, but it did happen. That “left pad” function I mentioned earlier? That one actually did vanish, thanks to a rogue developer, and it broke everything. So many applications and app libraries depended on little old leftpad, often indirectly and without even noticing, that its disappearance left them unable to update, unable to even install. For a few brief moments, half the JavaScript world was paralyzed because a package providing a one-line function, something that, in a language with simple string concatenation, comes essentially for free, was removed from the main code-sharing repository.

Solutions?

Is there a middle ground? Can we balance the need for small, space-optimized codebases with the robustness necessary for building serious applications? Or is NPM destined to be nothing more than a pale imitation of CPAN crossed with an enthusiast’s idea of the perfect Linux distro? I wish I knew, because then I’d be rich. But I’ll give a few thoughts on the matter.

First off, the space requirement isn’t going away anytime soon. As long as we have mobile and home data caps, bandwidth will remain important, and wasting it on superfluous code is literally too expensive. In backwards Third World countries without net neutrality, like perhaps the US by the time this post goes up, it’ll be even worse. Bite-size packages work better for the Internet we have.

On the other hand, a lot of the more ridiculous JS packages wouldn’t be necessary if the language had a halfway decent standard library. I know the standards crew is giving it their best shot on this one, but compatibility is going to remain an issue for the foreseeable future. Yes, we can use polyfills, but then we’re back to our first problem, because that’s just more code that has to be sent down the wire, code that might not be needed in the first place.

The way the DOM is set up doesn’t really help us here, but there might be a solution hiding in there. Speculative loading, where a small shim comes first, checking for the existence of the needed functions. If they’re found, then the rest of the app can come along. Otherwise, send out a request for the right polyfills first. That would take some pretty heavy event hacking, but it might be possible to make it work. (The question is, can we make it work better than what we’ve got?)

As for the general problem of ever-multiplying dependencies, there might not be a good fix. But there also might not be a need to keep the old maxim in mind. Do we really need to import a whole new package to put padding at the beginning of a string? Yes, JS has a wacky type system, but " " + string is what the package would do anyway. (Probably with a lot of extra type checking, but you’re going to do that, too.) If you only need it once, why bother going to all the trouble of importing, adding in dependencies, and all that?

Ultimately, what has happened is that, as JavaScript lacks even the most basic systems for code reuse, its developers have reinvented them, but poorly. As it has a stunted standard library, the third party has had to fill those gaps. Again, they have done so poorly, at least in comparison to more mature languages. That’s not to say that it doesn’t work, because everything we use on the Internet today is proof that it does. But it could be better. There has to be a better way, so let’s find it.

Practical Typescript: dice roller

In the last few “code” posts of Prose Poetry Code, there’s been one thing missing. One very important thing, and you might have noticed it. That’s right: there’s no code! I’ve been writing about generalities and theory and the like for a while now, but I’ve been neglecting the ugly innards. Part of that is because I haven’t been in much of a coding mood these past few months. Writing fiction was more interesting at the time. But I’ve had a few spare hours, and I really do want to get back into coding, so here goes.

A while back, I mentioned Typescript, a neat wrapper that sits on top of JavaScript and generally makes it palatable. And at the end of that post, I said I’d be playing around with the language. So here’s what I’ve got, a practical example of using Typescript. Well, dice rollers aren’t exactly practical—they’re a dime a dozen, and nobody really needs them—but you get the idea.

The code

I’ve posted a full ZIP of the code, including a config file, an HTML skeleton, and the JavaScript output. But here’s the Typescript code (dice.ts) for your perusal.

// Simple function simulating a single die roll
function roll(sides: number): number {
    return 1 + Math.floor(Math.random() * sides);
}

// Our "back-end" will roll a number of simulated dice,
// possibly adding a bonus or subtracting a penalty.
// It will return an object containing the rolls and their total.
// (This is really for no reason other than to show off interfaces.)
interface RollResult {
    rolls: number[],
    total: number
}

// Roll a number of dice, and return the results,
// using the object we defined above.
function rollMany(count: number, sides: number): RollResult {
    let rolls : number[] = [];

    for (let i = 0; i < count; i++) {
        rolls.push(roll(sides));
    }

    let result = {
        rolls: rolls,
        total: rolls.reduce((a: number, b: number) => a+b, 0)
    };

    return result;
}

// This is where the interactive portion of the script begins.
// It's really just basic DOM stuff. In a moment, we'll actually
// hook it all up to the page.
function diceRoller() {
    console.log("Rolling...");

    let countBox = document.getElementById("count") as HTMLInputElement;
    let sidesBox = document.getElementById("sides") as HTMLInputElement;
    let addsBox = document.getElementById("adds") as HTMLInputElement;
    let resultRollsText = document.getElementById("result");
    let resultTotalText = document.getElementById("total");

    let count = +countBox.value;
    let sides = +sidesBox.value;
    let adds = +addsBox.value;

    let result = rollMany(count, sides);
    let totalRoll = result.total + adds;

    resultRollsText.innerHTML = result.rolls.join(" ");
    resultTotalText.innerHTML = ""+ totalRoll;
}

// This clears out our results and options.
function clearAll() {
    console.log("Clearing...");

    // Note that this gives us an HTMLCollection...
    let boxes = document.getElementsByClassName("entry");

    // ...which can't be used like an array in for/of...
    for (let b in boxes) {
        // ...and contains only generic HTMLElements.
        (boxes[b] as HTMLInputElement).value = "";
    }

    let resultText = document.getElementById("result");
    let totalText = document.getElementById("total");
    resultText.innerHTML = "";
    totalText.innerHTML = "";
}

// Here's where we connect our functions to the page.
document.addEventListener("DOMContentLoaded", function () {
    let clearButton = document.getElementById("clear");
    let rollButton = document.getElementById("roll");

    clearButton.onclick = clearAll;
    rollButton.onclick = diceRoller;
});

Honestly, if you’ve seen JavaScript, you should have a pretty good idea of what’s going on here. We start with a helper function, roll, which does the dirty work of generating a number from 1 to N, exactly as if you rolled a die with N sides. (It’s not perfect, as JavaScript uses pseudorandom numbers, but it’s the best we can do.) If you take out the two type declarations, you wouldn’t be able to tell this was Typescript.

Next comes something that inarguably identifies our source as not normal JS: an interface. We don’t really need it for something this simple, but it doesn’t cost anything, and it’s a good check on our typing. Our RollResult interface simply defines an object “layout” that we’ll pass from our back-end rolling function to the front-end output. If we screw up, the compiler lets us know—that’s the whole point of strong typing.

After this is the rollMany function. It builds on the simple roll, calling it multiple times and storing each result in an array. This array, along with the sum of its contents, will become the returned object, matching our interface.

We’ll skip over the diceRoller function briefly, as it’s the grand finale of our little app, and I wanted to save it for last. Instead, we move to clearAll. It would be an unremarkable DOM manipulation function, if not for two quirks in Typescript. First, our simple HTML page defines a few entry boxes: one for the number of dice to roll, one for how many sides each die has, and one for a flat bonus or penalty to add to the total. These are <input> text elements, and they all have a CSS class of entry. Iterating over them (and only them) should be a piece of cake, right?

Not quite. See, the obvious way to do this, document.getElementsByClassName, returns an HTMLCollection, which isn’t quite the same thing as a JavaScript array. And Typescript, unless you tell it to spit out the latest and greatest ECMAScript standard (and I haven’t), won’t let you use the easier for..of loop. Instead, you have to use for..in, which iterates over the keys of a collection—for arrays, these are the indexes.

And that brings us to our second problem. See, our DOM collection is full of HTMLElements. These could be anything at all, and are thus usable as essentially nothing. In particular, we can’t change the value property that all input textboxes have, because not every HTML element has them. As any Java or C# programmer knows, we need an explicit cast into the more specific type of HTMLInputElement. That seems like needless drudgery, but it pays off in more complex applications, and we could always use jQuery or something instead. In a “real” setting, we probably would be.

The areas where we output our rolls and their total are simple divs. We don’t have to do any casting with them; we can just clear their innerHTML properties. And the last bit is not much more than your usual “let’s set up some event handlers” block.

That leaves diceRoller. First up are a few variables to hold the DOM elements: 3 text boxes and 2 output areas. Again, we have to do a cast on anything that’s an input element, but that should be old hat by now.

Following that, we make a few variables that hold our input values in number form. I used the idiomatic “unary plus” conversion from string to number here.

Next comes result, which holds (naturally) our resulting roll. I threw in a totalRoll variable to hold the total (including the “adds” roll bonus/penalty), but you don’t really need it; you can calculate that as you’re putting it into the output field. Speaking of which, that’s where we end: joining the result array into a space-separated string (you could use commas or whatever), then putting that and the total into their proper places.

Conclusion

So Typescript isn’t that hard. Counting comments and blank lines, this little thing weighs in at about 80 lines. The resulting JavaScript is less than 50. The difference comes from the strong typing, an interface definition that is purely for the benefit of the Typescript compiler and the programmer, and a few other odds and ends that contribute nothing physical to the finished product.

It may seem silly. We’re writing more code, locking ourselves into the type system, and we’re not really getting much out of it. Sure, we’ve got protection from typos. If I’d misspelled the name of one of the RollResult fields, I’d get an error telling me where I went wrong, but that’s about it.

For something this simple, even that’s enough. I get that error when compiling. I don’t end up with a blank screen or unresponsive page and no indication as to why. JavaScript’s dynamic nature is great, but it’s also terrible. In coding, unlike in real life, there’s such a thing as too much freedom.

Now, if you like, feel free to take this tiny app and improve on it. The HTML, for instance, expects a stylesheet; you could make one. Add in some bells and whistles. Throw up a few preset buttons. Store the last 100 rolls in DOM local storage. Most of all, have fun with it.

First glance: Scala.js

I’ve learned and used a lot of different programming languages. If I had to pick one to take as the moneymaker, it’d probably be JavaScript right now. But JavaScript is pretty awful. It wouldn’t be my first choice if money was no object. That would likely be C++, but there’s one other contender, one language I liked from the first time I saw it. That language is Scala.

Scala has its problems, of course. It’s closely tied to the Java ecosystem, which…isn’t the greatest. Sure, it’s got a huge selection of libraries and frameworks for whatever you want to do, but the Java language itself is atrocious, the JVM has a well-deserved reputation for bloat, and it’s practically banned on the desktop. (And you run the risk of getting sued if you use it on mobile, as Google discovered.) So Scala isn’t exactly a popular option, for very good reason.

But it doesn’t have to be that way. Plenty of languages have been turned into tools for web development by the clever trick of compiling them into JavaScript. For Scala, that would be the job of Scala.js. It’s a relative newcomer (version 0.6.14 at the time of this writing), but it’s already looking good.

What is it?

Scala.js is, simply enough, a tool for turning Scala code into JavaScript. In that, it’s not much different from “transpilers” like TypeScript or Babel. Only the source language isn’t something vaguely reminiscent of the JS it will become. No, you write in Scala, so you have all the expressive power of that language.

I like Scala. It’s one of the few languages I’ve seen that accomplishes the feat of getting me interested in functional-style programming. I absolutely hate the “pure functional” approach of, e.g., Haskell, where you can’t actually do anything, because all those nasty “impurities” like, say, I/O, are restricted, and variables can’t, you know, vary. (Yeah, you can use monads, but they’re limited in how they interact with the rest of your code. That’s kinda the point.) Scala, instead, makes FP style encouraged, but not required. That’s much better.

On top of that, Scala is just a better Java than Java. It’s got most of what Gosling took out as “too hard”, like operator overloading. It’s got pattern matching. Lambdas exist, and they’re easy to write. Scala is what Java programmers wish they could use. Putting it in the browser? Sure.

The good

So we’ve got a better language, and that’s where most of the good comes from. Like what, you may ask? How about strong typing? Sure, you can get that in TypeScript, but Scala uses it to its full extent. The functional style is front and center, and it’s not hidden behind bad syntax like in JS. OOP is class-based, not prototype-based, and you don’t have to worry about browser compatibility. And then there are a lot of things Scala has that just aren’t feasible in JS, like case classes and traits.

Outside of the language proper, you’ve got a good HTML templating system, Scalatags, that has full type safety. And it’s not an addon, not in the JS sense. No, you’re working with something that’s based on Java, remember? You’ve got Java-derived build tools and libraries.

I could go on, but the main advantages of Scala.js come from the simple fact that you’re using Scala in place of JavaScript. Everything else great about it follows naturally from that. If you like Scala, that should be enough to give it a shot. If you hate JavaScript, same deal.

The bad

But it’s not all good. That same Java heritage is the greatest weakness of Scala.js. You’re using Java tools, a Java-based API, JVM-centric constructs. Not a fan of Java? Tough. You really can’t get away from it with Scala. (Really, though, this isn’t that much different from that thing Google did a few years ago. JWT? I can’t remember, and I’m too lazy to look it up.)

A worse shortcoming is the number of limitations Scala.js has on what Scala code you can use. Mostly, this makes sense—you can’t use parallel programming in the inherently single-threaded environment of the browser. You also can’t use any Java libraries; your code, and all the code you import, has to be pure Scala, with a very few exceptions. And then there are a few nits to pick, cases where the developers of Scala.js couldn’t figure out a way to make the construct work in JS. Reflection is one of these: you can’t use it at all, and you can’t use anything that uses it. Macros (the recommended replacement) are a poor substitute for reflection, just as templates are in C++.

Lastly, you’ve got the build environment. If you’ve used any sort of JavaScript tools made this decade, you’re probably familiar with Node, Grunt, Gulp, and the like. Scala.js inherits Scala’s sbt, which is yet another thing to learn. And IDEs don’t seem to like it very much; I’ve never managed to get an sbt-based project talking to an IDE, and Eclipse, Netbeans, and IDEA all laughed at my attempts to use sbt for their projects. Since Scala, like Java, really wants an IDE, this can be a problem. (Maybe they’ve fixed things since then, though.)

The verdict

Although the “bad” section outweighs the “good” in quantity, don’t let that make you think I don’t like Scala.js. I like the idea of it. I can even like the implementation. File sizes are a bit on the hefty side, but they’re working on that. Some of the API limitations can be changed. The thing’s still in beta, remember.

On the whole, if you can live with the restrictions it places on you, Scala.js looks to be a very interesting way of creating web apps without writing raw JavaScript. I know I’m keeping an eye on it.

Playing with TypeScript

JavaScript sucks, and we all know it. But it’s the lingua franca of the web (not like WebAssembly is ever taking off), so we have to learn to deal with it. That doesn’t, however, mean we have to write it. Oh, no. There’s a whole subculture of languages that compile into JavaScript, meaning you can write your code in something that looks sane (to you) without worrying about the end result being readable.

This trend really started a few years ago with CoffeeScript, which did an admirable job of making JS look like Ruby. From there, similar projects spun off, like Coco and LiveScript, which were far better in that they didn’t have anything to do with Ruby. And then Microsoft got involved. That’s where TypeScript comes from. Yes, that Microsoft.

But it’s not as bad as it sounds. The language and compiler use the Apache license, and that gives us a bit of protection from the worst of corporate silliness. And that means it might be interesting to look at TypeScript for what it is: an attempt at making a better JavaScript than JavaScript.

Up and running

TypeScript fits into the usual Node-hipster-modern ecosystem. You can install it through npm, and it works with most of the other big JavaScript tools like testing systems. Some of the big “web app” frameworks even use it by default now, meaning that I’m a little behind the times. Oh, and you can also use it with Visual Studio, but I can’t, because that’s Windows-only. (And VS Code is not the same.) But you don’t have to: the homepage even has a link to TypeScript support for the best text editor out there. I’ll leave it to you to decide which one that is, but I’ll tell you that it’s the same one I’m using to write this post.

My type

So you get TypeScript installed, and you’ve got a nifty little compiler that spits out perfectly fashionable JavaScript, but what about the source language? What’s so special about it? In a word: types. (You’ll note that this word makes up half the language’s name.)

Essentially, TypeScript is nothing more than JavaScript with strong typing. That may not seem like much, but it’s actually a huge change that alters the whole way you write code. Variables are typed, functions are typed, objects are typed. To be fair, they all are in JavaScript, too, but that language doesn’t do anything with those types. TypeScript is made to use them. By itself, that’s almost enough to overcome my instinctive hatred of all things Microsoft. Almost.

On top of the bare bones JavaScript gives us, we’ve got tuples, enums, generics (C#-like, so not the best we could get, but far better than what JS offers), and an any type that lets us ignore the type system when necessary. If you wanted to, you could just about get away with writing regular JavaScript, tagging everything as any, and running it through the TypeScript compiler. But why would you do that? You’re supposed to be using a better language, right? So use it! Besides, like any decent strongly-typed language, we can use type inference to save us most of the trouble of specifying what something should be.

The next level

Just having the added safety of stronger typing already gives TypeScript a leg up on its parent language. But if it didn’t offer anything else, I’d tell you not to bother. Fortunately for this post, it has a lot more.

JS is currently in a state of flux. Most (but not all) browsers support ES5, some let you use a lot of the new ES6 features, and the language standard itself has transitioned to the ridiculous rapid-release model that’s all the rage today, so it’s looking like we’ll soon be back in the bad old days of “Best viewed in {browser X}”. Polyfills can only take us so far, you see. For coders writing raw JavaScript, it’s a serious problem making something maximally compatible, and most just throw their hands up and say, “Just use Chrome like everyone else.”

That’s another case where I’ll give the MS team credit. TypeScript takes a lot of the newer JS additions and lets you use them now, just like Babel or other “transpilers”. You can declare your variables with let instead of var, and the compiler will do the right thing. You get the new class syntax for free, which is great if you never could wrap your head around prototype OOP. Generators aren’t even in JS yet, so TypeScript even looks a bit like the future in some cases.

Reservations

If there’s anything wrong with TypeScript, it’s not the kind of thing that shows up in a cursory inspection of the documentation. No, the main problems are twofold. One, it’s a project started by Microsoft, a company with a history of bad blood towards the open source community. The Apache license helps in that regard, though I don’t think it can ever completely alleviate some people’s fears.

Second, TypeScript has been around for a while now, but it’s only recently been picking up steam. Angular and React both like it a lot, as do some indie game engines like Phaser. But the JS community is fad-driven, and this new acceptance could be an indication of that. If TypeScript is simply “the next big thing”, then interest will fade once some other shiny thing catches the eyes of the hipsters. We can’t prevent that, but we can do our best to ignore it.

Will TypeScript become the future of web development? I can’t say. It’s definitely one option for the present, though. And it’s a pretty good option, from what I’ve seen. I think I’ll play around with it some more. Who knows? Maybe I’ll show you what I’ve made.

Wrapping text by code

Very many games require displaying significant amounts of text on the screen. Whether dialogue, signs, backstory “data log” entries, or lists of quests and objectives, text is absolutely a necessity. And we’re not talking about just a word here and there. No, this is serious text, entire sentences at least. More specifically, I want to look at the long strings of text that are commonly shown in dialog boxes or other GUI/HUD elements, not something that might be baked into a texture.

With single words, it’s not hard at all: you just throw the word on the screen at the right position. What game engine doesn’t let you do that? Even my failed attempt at a simple 2D engine a few years ago could manage that much!

It’s when you need to display longer snippets that things get hairy. Your screen is only so wide, so, like any word processor, you have to know when to wrap the text. Now, many engines do this for you, but you may not be using one of those, or you may be writing your own. So this post is designed to give you an idea of what lies ahead, and to provide a starting point for your own efforts.

Ground rules

First, let’s get this out of the way: I’m using a bit of pseudocode here. Actually, it’s JavaScript, but with a lot of references left undefined. Hopefully, you can convert that into your language and platform of choice.

Next, I’m going to assume you’re working in English or another language that uses an alphabetic script. Something like Chinese or Japanese will have its own rules regarding how long text is broken up, and I don’t entirely understand all of those. The same goes for Arabic and other RTL scripts, but they’re much closer to American and European usage, so some of the concepts described here might carry over.

We’ll also ignore hyphenation in this post. It’s a hideously complex topic, first of all, and nobody can quite agree on which words should (or even can) be hyphenated at the end of a line. Besides, what was the last game you saw that used printed-style hyphenation? In the same vein, we’ll be ignoring non-breaking spaces, dashes, and the like.

The last assumption I’m going to make is that we’re working with Unicode strings. That shouldn’t matter too much, though. We won’t be delving into individual bytes—if you do that with Unicode, you’ve gone wrong somewhere, or you’re at a far lower level this post.

Words

On the screen, our string will be divided into lines. Thus, our job is clear: take a string of text, divide it into lines so that no line is longer than the width of our dialog box. Simple, right? So let’s see how we can go about that.

English text is divided first into sentences. These sentences are then divided into words, sometimes with added punctuation. Each word is separated from its neighbors by blank space. It’s that space that lets us wrap our text.

For a first approximation, let’s take the example of fixed-width text. This was common in older games, and it’s still used for a retro feel. (And programmers demand it for editing code.) It’s “fixed” width because each letter, each bit of punctuation, and each space will have the same width. That width can be different from one font to another, but every character in a fixed-width string, even the blank ones, will be as wide as any other.

As we won’t be worrying about hyphenation, every line will have a whole number of words. Thus, our first task is to get those words out of our string:

// Split a string of text into a list of words
function splitWords(str) {
    return str.split(' ');
}

Basically, we’ve split our string up into words, and removed the spaces in the process. Don’t worry about that. We’ll put them back in a moment. First, we need to go on a little digression.

Building with boxes

Do you remember magnetic poetry? Those little kits you could buy at the bookstore (back when bookstores were a thing)? They came with a set of pre-printed words and a bunch of magnetic holders. Some of them even let you cut your own magnets, then slide the bits of laminated paper containing words into them. However you did it, you could then assemble the words into a short, tweet-level string that you could stick to your fridge. Longer words required longer magnets, while something like “a” or “I” took up almost no space at all. If you used one of the magnetic poetry kits that came with a little board, you had to do a bit of shuffling—and maybe some editing—to get the words to fit on a line.

That’s what we’re doing here. We’re taking the words of our text, and we’re assembling them on the screen in exactly the same way. Like the board, the screen is only so wide, so it can accommodate only so much. Any more, and we have to move to the next line. So wrapping text can be thought of like this. (It’s also the beginnings of computerized typesetting. We’re not really doing anything different from TeX; it just does the job a whole lot better.)

So we need to pack our words onto lines of no more than a given width. One way to do that is incrementally, by adding words (and spaces between them) until we run out of space, then moving on to the next line.

// Build lines of fixed-width text from a string
function buildTextLines(str, screenWidth, fontWidth) {
    let allWords = splitWords(str);
    let lines = [];

    let currentLine = "";
    let currentLineWidth = 0;
    let currentWord = "";

    while (allWords.length > 0) {
        currentWord = allWords.shift();

        let currentWordWidth = currentWord.length * fontWidth;

        if (currentLineWidth + currentWordWidth > screenWidth) {
            // Too long; wrap text here
            // Save the line we just finished, and start a new one
            lines.push(currentLine);
            currentLine = "";
            currentLineWidth = 0;
        }

        currentLine += currentWord;
        currentLineWidth += currentWord.length * fontWidth

        // Try to insert a space, unless we're at the edge of the box
        // (width of space is the same as anything else)
        if (!(currentLineWidth + fontWidth > screenWidth)) {
            currentLine += " ";
            currentLineWidth += fontWidth;
        }
    }

    return lines;
}

For fixed width, that’s really all you have to do. You can still treat the text as a sequence of characters that you’re transforming into lines full of space-delimited words. As a bonus, punctuation carries over effortlessly.

Varying the width

The real fun begins when you add in variable width. In printed text, an “m” is bigger than an “n”, which is wider than an “i”. Spaces tend to be fairly narrow, but they can also stretch. That’s how justified text works: extra bits of space are added here and there, where you likely wouldn’t even notice them, so that the far ends of the lines don’t look “ragged”. (That, by the way, is a technical term.) Two words might have an extra pixel or two of space, just to round out the line.

Doing that with simple strings is impossible, because I don’t know of any programming language having a basic string type that accounts for proportional fonts. (Well, TeX does, because that’s its job. That’s beside the point.) So we have to leave the world of strings and move onward. To display variable-width text, we have to start looking at baking it into screen images.

Since every engine and renderer is different, it’s a lot harder to give a code example for this one. Instead, I’ll lay out the steps you’ll need to take to make this “box” model work. (Note that this phrase wasn’t chosen by accident. HTML layout using CSS, as in a browser, uses a similar box-based display technique.)

  1. Using whatever capabilities for font metrics you have, find the minimum width of a space. This will probably be the width of the space character (0x20).

  2. Using this as a guide, break the text into lines—again based on the font metrics—but keep the words for each line in a list. Don’t combine them into a string just yet. Or at all, for that matter.

  3. For each line, calculate how much “stretch” space you’ll need to fill out the width. This will be the extra space left over after adding each word and the minimum-width spaces between them.

  4. Divide the stretch space equally among the inter-word spaces. This is aesthetically pleasing, and it ensures that you’ll have fully justified text.

  5. Render each word into the sprite or object you’re using as a container, putting these stretched spaces between them. You can do this a line at a time or all at once, but the first option may work better for text that can be scrolled.

In effect, this is a programmatic version of the “magnetic poetry” concept, and it’s not too much different from how browsers and typesetting engines actually work. Of course, they are much more sophisticated. So are many game engines. Those designed and intended for displaying text will have much of this work done for you. If you like reading code, dive into something like RenPy.

I hope this brief look whets your appetite. Text-based or text-heavy games require a bit more work for both the developer and the player, but they can be rewarding on a level a fully voiced game could never reach. They are, in a sense, like reading a story, but with a bit of extra visual aid. And games are primarily a visual medium.

Statistical sampling in JavaScript

It’s 2016, and that’s an election year, which means we’ll be spending the rest of the summer (and half of the fall) watching America’s most ridiculous spectator sport. The pundits and panels and polls are all quite fun, but I find that the methodology is far more interesting than the results.

One of the greatest weapons in the pollster’s arsenal is sampling, and one of those footnotes you’ll see in opinion polls in the coming weeks is the margin of error. These are basic concepts of statistics, but most people might not know how they work. Worse, some programmers might not. So here’s my attempt at rectifying that.

Definitions

Sampling is, in effect, a way of drawing conclusions about a population (such as a state or country) based on surveying only a small fraction of its members. It’s not perfect, but it turns out that, say, 500 people are actually a pretty good indicator of the rest of the nation…as long as you pick the right 500 people. In terms relatable to current events, a presidential poll that only asks people from rural parts of the South is going to get very different results from one that surveys nothing but New York City. That’s selection bias, and it’s one of the hardest things for pollsters to avoid. They’ve got a few ways around it, such as cold-calling random phone numbers, but it’s always a battle.

That very randomness is why sampling works in the first place. If you truly choose your data points (i.e., the people you ask) randomly, then they will, when put together, approximate the “true” nature of things. The more you get, the closer your picture is to the real thing. Eventually, as you’d expect, your sample size approaches the total population; at that limit, the sample obviously represents the whole to perfection.

For smaller samples, however, things aren’t so perfect. Let’s say we have two candidates: H and T. (You get no points for guessing what those stand for.) In “reality”, they aren’t quite even. Say that H has 50% of the vote, T has 45%, and the remaining 5% are undecided, independent, or whatever. Now, take some sort of random number generator and set it to give numbers from 1 to 100. Everything up to 50 is a “vote” for candidate H, 51-95 are for T, and 96-100 are undecided.

After a single generated number, you’ve got a score of 1 for one of the three, 0 for the other two. Not very predictive, but keep going. With a sample size of 10, my results were H 6, T 3, and 1 undecided. Yours might be different, but notice that that’s already looking a lot closer to the true numbers. Give it 100 tries, and it’s probably even better. (Doing this three times with a different generator, my results were: 52 T, 44 H, 4 undecided; 48 T, 47 H, 5; and 57 T, 40 H, 3. Clearly, this RNG leans right.)

The larger the sample size, the more likely the sample will match the population. If you don’t mind a bit of math, then we can look at just how good a match we can get. The basic formula is e = 1 / sqrt(N), where N is the sample size and e is the margin of error. So, for our sample size of 100 above, the math says that our expected error is somewhere within 1/sqrt(100) = 1/10 = 0.1, or 10% either way. Or, as the polls put it, ±10%. Most samples like this are assumed to be conducted at a 95% confidence interval; this basically means that there’s a 95% chance that the true results lie within that margin. (Note, however, that our third poll in the last paragraph didn’t. It’s an outlier. They happen.)

As counter-intuitive as it may seem, this formula doesn’t really depend on the population size at all, as long as the sample is sufficiently small in relation. For national polls surveying a thousand or so people, that assumption holds, so they can safely tout a margin of error of ±3% from their sample of 1,016.

The code

Now we’ll look at how you can do your own sampling. This isn’t just for opinion polls, though. Any kind of analytics could make use of sampling.

The basic function, in JavaScript, would look something like this:

/*
 * Select `k` random choices from a population
 */
function sample(population, k) {
    var psize = population.length;
    var choices = new Set();
    var result = [];

    // Choose `k` elements from the population, without repeats
    for (var i = 0; i < k; i++) {
        var ch;

        do {
            ch = Math.trunc(Math.random() * psize);
        } while (choices.has(ch))

        choices.add(ch);
    }

    for (var c of choices) {
        result.push(population[c]);
    }

    return result;
}

As always, this isn’t the only way to do what we’re trying to do, but it’s very close to what Python’s random.sample function does, so the idea is sound. To get our sample, we generate a number of array indexes, and the Set guarantees we won’t get any repeats. Our result is a new array containing only those elements we chose. We can then do whatever we need.

But how do we determine what sample size we need? Well, one way is to work backwards from the margin of error we want. Remember that this usually won’t depend on the size of the population.

/*
 * Given a desired margin of error,
 * find an appropriate sample size.
 */
function sampleSize(margin) {
    return Math.round(1 / (margin*margin), 0);
}

This is nothing more than a rearrangement of our formula. Instead of saying e = 1 / sqrt(N), we move things around to solve for N: N = 1 / e^2. Rounding to the nearest integer is just for convenience.

In a nutshell, that’s all there is behind those opinion polls you’ll be hearing so much about over the coming months. Pick enough people at random, and you’ll get a picture of the general populace. It’ll be a blurry picture, but even that’s enough to make out some of the larger details.

First glance: Superpowers

It seems like each new day brings us a new tool for game development, whether it’s an engine, a framework, a library, or any of a number of other things. Best of all, many of these up-and-coming dev tools are open source and free, so even the poorest game makers (like me!) can use them. And the quality is only going up, too. No longer must indies be content with alpha-level, code-only engines, uncompiled libraries, and NES-era assets. No, even the zero-cost level of game development is becoming packed with tools that even professionals of a few years ago wished they could have had.

The one I’m looking at today is called Superpowers, by Sparklin Labs. It’s yet another HTML5 game maker that has recently been released as open source software, under the ISC license. (ISC is functionally equivalent to MIT or “new” BSD; basically, it’s not much more than “do what you want, but give us credit”.) It’s not entirely a volunteer effort, and there are a couple of options for supporting it. Their download host, indie game publisher itch.io, gives you a donation option, but the primary way to send money is through Patreon. (There’s a link on the Superpowers main page.)

Let’s take a look

What does Superpowers bring to the table? Well, first of all, it’s an HTML5 engine. The maker itself runs as a nativized web app, and games can be compiled into standalone apps or exported in a browser-friendly format. There’s also a mobile option using the Intel XDK, but I haven’t really looked into that.

Second, and even more important, is the fact that this engine comes with a visual editor. That’s something sorely lacking in the free HTML5 arena. Granted, it’s not exactly up to the level of the editors for Unity or Unreal, but it’s much better than what we had, i.e., not much. It’s got all the usual bells and whistles: tree-based scene management, component editors (these seem a little buggy on my machine, but that’s probably just a local thing), drag-and-drop actors, and so on. For what’s technically still a beta, it’s pretty nice.

Coding works about the same way. You can attach scripts to the various parts of a scene, and they’ll know what to do. The whole thing is mostly behavior-driven, following the component style that is so popular these days. The scripts themselves are written in TypeScript, and I’m a little ambivalent about that. On the one hand, it’s an easier way of writing JavaScript (Superpowers is HTML5-based, so it’s going to end up as JavaScript eventually). On the other, TypeScript is a Microsoft project, so you never know what might happen.

One of the big features that looks interesting is the collaboration support. The Superpowers “app” has a client-server architecture, and it takes advantage of it. When you start it, it creates a server. Now, that’s pretty common in Node applications, but Superpowers actually uses it. After a little initial setup, you can have other people connect to your editor instance and work with you in real-time. I can’t tell you how well that works, since I’m just a lonely guy, but if it comes anywhere close to what’s advertised, then…wow.

There’s a lot more than this, and what I’m seeing looks very good. There’s support for prefabs (like those in Unity) in the editor, for instance, and the engine has all the usual suspects: 2D physics, multiple cameras, etc. Debugging works like in Chrome, since the whole thing runs on NW.js. (IMO, Chrome is a horrible browser, but an okay wrapper for web apps. The developer tools aren’t half bad, though.)

That’s not to say the Superpowers is perfect. Far from it. It’s early in development, and there’s bound to be a few unsquashed bugs here and there. There’s also the TypeScript dependency I mentioned above, but they’re working on that; the developers have an alpha (I think) version of the editor using Lua and the LÖVE engine. And, being on GitHub, I noticed a “Code of Conduct” file, which could be worrisome to free-speech advocates like myself. Also, there’s no online API documentation. You’re supposed to use the editor’s built-in docs. The developers’ reasoning (it boils down to “But there might be plugins!”) sounds weak to my ears. Every other HTML5 engine can do it, so why not this one?

In the end, I think the good outweighs the bad. Give it some time, and Superpowers might become one of the go-to tools for making indie games. Or it could bomb, and we’ll never hear from it again. I doubt that, though. Give me some proper online API docs, support for multiple languages (including pure JavaScript, preferably of the ES6 variety), and quite a bit more polish, and I’ll gladly put it up there with Phaser at the top of the list. For now, I’ll definitely be keeping an eye on this one.

Randomness and V8 (JS)

So I’ve seen this post linked in a few places recently, and I thought to myself, “This sounds interesting…and familiar.”

For those of you who don’t quite have the technical know-how to understand what it means, here’s a quick summary. Basically, the whole thing is a flaw in the V8 JavaScript engine’s random number generator. V8, if you don’t know, is what makes JavaScript possible for Chrome, Node, and plenty of others. (Firefox uses a different engine, and Microsoft’s browsers already have far bigger problems.) In JavaScript, the RNG is accessed through the function Math.random(). That function is far from perfect as it is. There’s no need to make it worse.

But V8, until one of the newest versions (4.9.40), actually did make it worse. An outline of their code is in the post above, but the main problems with it are easy to explain. First, Math.random() returns JavaScript numbers—i.e., 64-bit floating-point numbers—between 0 and 1. The way those numbers work leaves the algorithm 52 bits to play with, but V8’s code worked by converting a 32-bit integer into a floating-point number. That’s a pretty common operation, and there’s nothing really wrong on the face of it. Well, except for the part where you’re throwing away 20 out of your 52 random bits.

Because of the way V8’s RNG algorithm (MWC1616), this gets even better. MWC stands for “multiply with carry”, an apt description of what we’re dealing with. Internally, the code has two state variables, each a 32-bit unsigned integer, or uint32_t. These start off as seeded values (JavaScript programmers have no way of influencing this part, unfortunately), and each one undergoes a simple transformation: the low 16 bits are multiplied by one of two “magic” constants, then added to the high 16 bits. The function then creates its result in two parts, with the upper half of the result coming from one state variable’s lower half, while the lower 16 bits are taken from the other state’s upper half.

The whole thing, despite its shell-game shifting of bits, is not much more than a pair of linear congruential generators welded together. LCGs have a long history as random generators, because they’re easy to code, they’re fast, and they can give okay randomness for simple applications. But now that JavaScript is being used everywhere, the cracks are starting to appear.

Since V8’s Math.random() implementation uses 32-bit numbers and none of the “extra” state found in more involved RNGs, you’re never getting more than 2^32^ random numbers before they start repeating. And I do mean repeating, as linear congruential generators are periodic functions. Given the same state, they’ll produce the same result; generate enough random numbers, and you’ll repeat a state, which restarts the cycle. But that 2^32^ is a maximum, and you need some planning to get it. The magic numbers that make an LCG work have to be chosen carefully, or you can sabotage the whole thing. All the bit-shifting tricks are little more than a distraction.

So what can you do? Obviously, you, as a user of Chrome/Node/V8/whatever, can upgrade. The new algorithm they’re using is xorshift128+, which is highly regarded as a solid RNG for non-cryptographic work. (If you’re interested in how it works, but you don’t think you can read C++, check out the second link above, where I roll my own version in JavaScript.) Naturally, this doesn’t fix all the other problems with Math.random(), only the one that caused V8’s version of it to fail a bunch of the statistical tests used to quantify how “good” a specific RNG is. (The linked blog post has a great visualization to illustrate these.) Seeded, repeatable randomness, for example, you’ll still have to handle yourself. But what we’ve got is good enough for a lot of purposes, and it’s now a better foundation to build upon.

First glance: Unreal.js

With Christmas coming up, I don’t exactly have the time to write those 2,000-word monologues that I’ve been doing. But I want to post something, if only because laziness would intervene otherwise. Inertia would take over, and I’d stop posting anything at all. I know it would happen. I’ve done it once before. So, these last few Wednesdays of 2015 will be short and sweet.

This time around, I want to talk about something I recently found on a wonderful site called Game From Scratch. It’s called Unreal.js, and it’s open-source (Apache license). What does it do? Well, that’s the interesting thing, and that’s what I’m going to ramble on about.

You’ve probably heard of UnrealEngine. It’s the latest iteration of the game engine used to power a wide array of games, from Unreal Tournament to AAA titles like the newest Street Fighter and Tekken to hundreds of up-and-coming indie games. The most recent version, UnrealEngine 4, is getting a lot of press mainly because of its remarkably open development and friendly pricing scheme. (Compared to other professional game engines, anyway.) Lately, Unreal has become a serious competitor to Unity for the middle tier of game development, and competition is an indisputably good thing.

But Unreal has a problem compared to Unity. You see, Unity runs on Microsoft’s .NET framework. (Strictly speaking, it runs on Mono, which is a Microsoft-approved knockoff of .NET that used to be fully open, to the point where most Linux distributions preinstalled it a few years ago. Now…not so much.) Anyway, Unity uses .NET, and one of the nifty things about .NET is that, like the JVM, it’s not restricted to a single language. Sure, you’re most likely to use C#, but you don’t have to. Unity explicitly supports JavaScript, and it used to have full support for a Python clone called Boo. (Supposedly, there are ways to get other languages like F# to work with it, but I don’t know why anyone would want to.)

Unreal, on the other hand, uses C++. From a performance perspective, that’s a great thing. C++ is fast, it can use far less memory than even C#, and it’s closer to the hardware, making it easier to take advantage of platform-specific optimizations. However, C++ is (in my experienced opinion) one of the hardest programming languages to learn well. It’s also fairly ugly. The recent C++11 standard helps a lot with both of these problems, but full support just isn’t there yet, even 4 years later. C++17 looks like it will go a few steps further in the “ease of use” direction, but you’ll be lucky to use it before 2020.

The makers of UnrealEngine know all of this, so they included a “visual” programming language, Blueprints. Great idea, in theory, but there are a lot of languages out there that you don’t need to invent. Why not use one of them? Well, that’s where Unreal.js comes in. Its developers (some guys called NCSoft; you may have heard of them) have made a plugin that connects the V8 JavaScript engine from Chrome/Safari/Node.js/everywhere into Unreal. The whole thing is still in a very early stage, but it’s shaping up to be something interesting.

If Unreal.js takes off, then it can put Unreal well ahead of Unity, even among hobbyists and lower-end indies. JavaScript is a lot easier on the brain than C++ (take it from someone who knows both). And it has a huge following, not just for webapps and server stuff. The Unreal.js project page claims support for “(Full) access to existing javascript libraries via npm, bower, …”

That’s huge. Sure, not all npm packages are of the highest quality, but there are plenty that are, and this would let you use all of them to help make a game. Game engines, historically, have been some of the worst about code reuse, 3rd-party libraries, and other niceties that “normal” applications get to use. Well, that can change.

And then there’s one other factor: other languages. Since Unreal.js is pretty much just the V8 engine from Node, and it can load most Node packages, that opens the possibility of using some of the many “transpiled” languages that are transformed to Node-friendly JavaScript. Think CoffeeScript, TypeScript (which recently released its new 1.7 version), or even my April Fools’ Day joke language Elan.

Maybe I’m wrong. Maybe Unreal.js will fizzle. Perhaps it’s destined to join the legions of other failed attempts at integrating game development with the rest of the programming world. I hope not. The past few years have seen a real move in the direction of democratizing the art of game-making again. I’d like to see that trend continue in 2016 and beyond.

ES6 iterators and generators

With ES6, JavaScript now has much better support for iteration. Before, all we had to work with was the usual for loop, either as a C-style loop or the property-based for...in. Then we got the functional programming tools for arrays: forEach, map, reduce, and so on. Now we have even more options that can save us from needing an error-prone C-style loop or the cumbersome for...in.

The new loop

ES6 adds a new subtype of for loop: for...of. At first glance, it looks almost exactly the same as for...in, but it has one very important difference: for...in works on property names, while for...of loops over property values. Prettified JavaScript variants (like CoffeeScript) have had this for years, but now it comes to the base language, and we get to do things like this:

var vowels = ['a','e','i','o','u','y'];

for (var v of vowels) {
    if (v != 'y') {
        console.log(v);
    } else {
        if (Math.random() < 0.5) {
            console.log("and sometimes " + v);
        }
    }
}

Most developers will, at first, use for...of to iterate through arrays, and it excels at that. Just giving the value in each iteration, instead of the index, will save the sanity of thousands of JavaScript programmers. And it’s a good substitute for Array.forEach() for those of you that don’t like the FP style of coding.

But for...of isn’t just for arrays. It works on other objects, too. For strings, it gives you each character (properly supporting Unicode, thanks to other ES6 updates), and the new Map and Set objects work in the way you’d expect (i.e., each entry, but in no particular order). Even better, you can write your own classes to support the new loop, because it can work with anything that uses the new iterable protocol.

Iterables

Protocols are technically a new addition to ES6. They lurked behind the scenes in ES5, but they were out of sight of programmers. Now, though, they’re front and center, and the iterable protocol is one such example.

If you’ve ever written code in Java, C#, C++, Python, or even TypeScript, then you already have a good idea of what a protocol entails. It’s an interface. An object conforms to a protocol if it (or some other object up its prototype chain) properly implements the protocol’s methods. That’s all there is to it.

The iterable protocol is almost too easy. For a custom iterable object, all you need to do is implement a method called @@iterator that returns an object that meets the iterator protocol.

Okay, I know you’re thinking, “How do I make a @@iterator method? I can’t use at-signs in names!” And you’re right. You can’t, and they don’t even want you to try. @@iterator is a special method name that basically means “a symbol with the name of iterator“.

So now we need to know what a symbol is. In ES6, it’s a new data type that we can use as a property identifier. There’s a lot of info about creating your own symbols out there, but we don’t actually need that for the iterable protocol. Instead, we can use a special symbol that comes built-in: Symbol.iterator. We can use it like this:

var myIterable = {
    [Symbol.iterator]: function() {
        // return an iterator object
    }
}

The square brackets mean we’re using a symbol as the name of the property, and Symbol.iterator internally converts to @@iterator, which is exactly what we need.

Iterators

That gets us halfway to a proper iterable, but now we need to create an object that conforms to the iterator protocol. That’s not that hard. The protocol only requires one method, next(), which must be callable without arguments. It returns another object that has two properties:

  • value: Whatever value the iterator wants to return. This can be a string, number, object, or anything you like. Internally, String returns each character, Array each successive value, and so on.

  • done: A boolean that states whether the iterator has reached the end of its sequence. If it’s true, then value becomes the return value of the whole iterator. Setting it to false is saying that you can keep getting more out of the iterator.

So, by implementing a single method, we can make any kind of sequence, like this:

var evens = function(limit) {
    return {
        [Symbol.iterator]: function() {
            var nextValue = 0;
            return {
                next: function() {
                    nextValue += 2;
                    return { done: nextValue > limit, value: nextValue };
                }
            }
        }
    }
}

for (var e of evens(20)) {
    console.log(e);
} // prints 2, 4, 6..., each on its own line

This is a toy example, but it shows the general layout of an iterable. It’s a great idea, and it’s very reminiscent of Python’s iteration support, but it’s not without its flaws. Mainly, just look at it. We have to go three objects deep to actually get to a return value. With ES6’s shorthand object literals, that’s a bit simplified, but it’s still unnecessary clutter.

Generators

Enter the generator. Another new addition to ES6, generators are special functions that give us most of the power of iterators with much cleaner syntax. To make a function that’s a generator, in fact, we only need to make two changes.

First, generators are defined as function*, not the usual function. The added star indicates a generator definition, and it can be used for function statements and expressions.

Second, generators don’t return like normal functions. Instead, they yield a value. The new yield keyword works just like its Python equivalent, immediately returning a value but “saving” their position. The next time the generator is called, it picks up right where it left off, immediately after the yield that ended it. You can have multiple yield statements, and they will be executed in order, one for each time you call the function:

function* threeYields() {
    yield "foo";
    yield "bar";
    yield "The End";
}

var gen = threeYields();

gen.next().value; // returns "foo"
gen.next().value; // returns "bar"
gen.next().value; // returns "The End"
gen.next().value; // undefined

You can also use a loop in your generators, giving us an easier way of writing our evens function above:

var evens = function(limit) {
    return {
        [Symbol.iterator]: function*() {
            var nextValue = 0;
            while (nextValue < limit) {
                nextValue += 2;
                yield nextValue;
            }
        }
    }
}

It’s still a little too deep, but it’s better than writing it all yourself.

Conclusion

Generators, iterators, and the for...of loop all have a common goal: to make it easier to work with sequences. With these new tools, we can now use a sequence as the sequence itself, getting its values as we need them instead of loading the whole thing into memory at once. This lazy loading is common in FP languages like Haskell, and it’s found its way into others like Python, but it’s new to JavaScript, and it will take some getting used to. But it allows a new way of programming. We can even have infinite sequences, which would have been impossible before now.

Iterators encapsulate state, meaning that generators can replace the old pattern of defining state variables and returning an IIFE that closes over them. (For a game-specific example, think of random number generators. These can now actually be generators.) Coroutines and async programming are two other areas where generators come into play, and a lot of people are already working on this kind of stuff. Looking ahead, there’s a very early ES7 proposal to add comprehensions, and these would be able to use generators, too.

Like most other ES6 features, these aren’t usable by everyone, at least not yet. Firefox and Chrome currently have most of the spec, while the others pretty much have nothing at all. For now, you’ll need to use something like Babel if you need to support all browsers, but it’s almost worth it.