From 200K lines of CoffeeScript to zero: making decaffeinate super-stable

The world of frontend development evolves so quickly that sometimes it feels impossible to keep up. Even worse, what do you do with your existing code when technologies shift? Innovations in languages, libraries, and practices are great, but they’re only useful if they can be used in practice. To enable new technologies, we need good modernization tools to aid in the transition process.

At . There’s a judgement call about what is “reasonable”, but decaffeinate needs to be trustworthy. decaffeinate’s goal is to speed up the conversion from CoffeeScript to JS, and if you need to extensively manually test your code after decaffeinate, it will take much, much longer. decaffeinate needs to be as stable as any compiler.

If you use decaffeinate, you’ll probably see Array.from all over the place. You can disable it with the–loose option, but I’d recommend instead looking through the resulting code and only removing Array.from when you’re confident that the object you’re working with is already an array.

Reaching super-stability

Between July and December 2016, decaffeinate slowly got more and more stable when run on the Benchling code. 500 files failing, then 350, then 200, then 50, then 10, then 5, then 0. After some cheers and excitement, I ran the tests for the newly-decaffeinated codebase, and they crashed immediately. More work to do, I guess. I fixed a bug, I fixed another bug, I tracked down and fixed an ESLint bug, I upgraded Babel to work around a Babel bug, I kept on iterating, and finally it got to a point of the tests passing.

So now what? decaffeinate seemed to work great on the code that was covered by tests, but what about the code that wasn’t covered by tests? I could write 100,000 lines of tests to try to get full code coverage, but that might take a while. I needed to find more test cases for decaffeinate, and fortunately, the internet has no shortage of CoffeeScript code. I set up a build system that would run decaffeinate on a bunch of projects, patch them to use Babel, then run all of the tests. A status board on the README was also a good motivating factor and a good way to see progress. Here were some of the results early on:

Testing out decaffeinate on codebases with a wide variety of authors and coding styles worked out great, and setting up the tests allowed me to discover the most critical bugs. There were lots and lots of bugs, but after a few months, I finally got everything working:

It’s rare that you ever get to say that a software project is complete, but I think this is one of those times. Technically, there’s still a bit more that would be useful, and I did some follow-up work to add a suggestions system and clean up usage, but decaffeinate is now in maintenance mode.

Getting to zero

With decaffeinate stable, we were finally at a point where it wasn’t crazy to run it on large swaths of code without extensive testing or review. So what do you do when you have a tool like that and 150,000 lines of CoffeeScript to convert? Somehow, converting it all at once seemed a little worrying.

Here’s the strategy we took: every Tuesday, we’d pick off a large chunk of code and run it through decaffeinate, first about 5000 lines, then larger and larger chunks, up to 20,000 lines at once. We always had a 100%-machine-automated set of commits, then two of us scanned through the converted code and made any safe cleanups we could find. That did not mean blindly removing all usages of Array.from or other decaffeinate artifacts; it meant fixing formatting, removing unused variables, renaming autogenerated variables to other names, etc. The code at the end wasn’t pristine, but it was JavaScript, and it was much better than what the CoffeeScript compiler gives.

The stability work paid off, and after repeating this process for about 10 weeks, we finally were able to get it completely to zero with very few issues. That also meant that it was possible to remove CoffeeScript from the build system, disable CoffeeLint, and delete our CoffeeScript style guide.

In a sense, the job still isn’t done. We still have lots of files with decaffeinate artifacts and disabled lint rules that should eventually be manually cleaned up. One of our biggest auto-converted files starts with this:

So it’s not perfect, but it’s pretty easy to do the remaining cleanup work as you go.

Let’s compare my original estimate with the actual cost of converting 200,000 lines of code:

Estimated cost without decaffeinate: 6,400 hours, 800 bugs.

Actual cost with decaffeinate: ~100 hours, ~5 bugs.

(Not including the work on decaffeinate, which was in my spare time. 😄).

Those 100 hours were mostly spent spot-checking and cleaning up the resulting JS, code reviewing those cleanups, manually testing the relevant features, and deploying the changes gradually to reduce risk. 2000 lines of code per hour seemed like a safe rate, but theoretically, it could have all been done at once, and if you’re in a hurry, you could probably go much faster.

What went wrong?

decaffeinate is extremely stable, but the conversion process still wasn’t without its bugs. By far, the largest source of bugs was human error. Let’s look at some code, before and after decaffeinate:

decaffeinate moved the default param to an if statement in order to be technically correct, wrappeditemIdsToKeep in Array.from, and changed thein operator to theincludes method. The if statement and the Array.from could both use cleanup, and in this case we played it safe and only removed the Array.from, since it clearly seemed like an array:

As it turns out, itemIdsToKeep was not always an array. Purely by mistake, it could sometimes be a DOM event instead. Both CoffeeScript and Array.from silently treat it as the empty array in that case, but removing Array.from exposed the crash.

This is an example of a theme that occurred a number of times: decaffeinate tends to break on code that is already buggy.

Let’s look at another example. Here’s some CoffeeScript before and JavaScript after:

Seems pretty simple, right? What could go wrong?

As it turns out, the problem wasn’t so much a conversion error as an unexpected consequence in switching compilers: switching from CoffeeScript to Babel enables strict mode. Without strict mode, assigning to a non-writable property is a no-op, and in strict mode it crashes. In this case, response was supposed to be an object, but instead was sometimes the empty string, which meant that the assignment simply did nothing. This was just a bug, but it was a benign bug before and switching to Babel made it a crashing bug.

How can you avoid running into these same problems? Some ideas:

  • Try to defer extensive manual cleanups until you’re working heavily with the code and are comfortable testing all of its edge cases. Over and over, we found that a cleanup that seemed safe actually had subtle flaws, and often times it was best to just go with the decaffeinate output.
  • You may want to move to strict mode separately from moving to JS. One way to do this is with the babel-plugin-transform-remove-strict-mode Babel plugin. That will keep you on the lookout for the types of errors that arise in strict mode.

Other things to know before using decaffeinate

Here are some more things to keep in mind:

  • The hardest problem that decaffeinate had to solve is handling this before super in constructors, which is allowed in CS but not in JS. There’s a hack to trick Babel and TypeScript into allowing it, but it’s an ugly hack that you should remove ASAP after running decaffeinate.
  • To help run decaffeinate along with other tools, I wrote a tool called bulk-decaffeinate that runs decaffeinate, several jscodeshift transforms (e.g. converting code to JSX), eslint –fix, and several other steps. You’ll probably want to either use that tool or one you write yourself, since decaffeinate is more of a building block.
  • There have been some great previous blog posts on decaffeinate: Converting a large React Codebase from Coffeescript to ES6 and Decaffeinating a Large CoffeeScript Codebase Without Losing Sleep.
  • The decaffeinate docs have some pages to help you through the process: the Conversion Guide has some practical advice on running decaffeinate on a big codebase, and the Cleanup Suggestions page lists most of the subtleties you’ll see in the converted code. You can also drop in on Gitter to ask questions!

Every project and team is different, though, so regardless of how you approach it, you’ll likely need some real effort and care. But hopefully, once the conversion strategy, the configuration, and any other details are figured out, the gruntwork will all be safely handled by decaffeinate!

Migration tools let you focus on what’s important

Programming is full of tradeoffs, and a common pitfall is to focus too much on your code when you should be focusing on the real problem that you’re solving. Whether it’s formatting, variable names, code structure, or even what programming language to use, none of it matters if the product doesn’t actually help people. Big code migrations can sometimes feel necessary and benefit in the long run, but they can also be a big distraction and a massive time sink. It’s often an awkward tradeoff, but with solid migration tools like decaffeinate, you can get the best of both worlds: you can benefit from the latest tools and practices while still focusing your time and brainpower on solving real problems and helping people.


From 200K lines of CoffeeScript to zero: making decaffeinate super-stable was originally published in Benchling Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Please follow and like us:
error0