IE10: How does it really stack up?

At Lucidchart, we really care about making a great diagramming application that is easy to use and intuitive for first time users, yet powerful enough for the sophisticated diagrams and mockups that many people need to create. That means that we want Lucidchart to be fast, even when a diagram gets very large and complex. And because we run inside the browser, we are subject to the limitations of each browser vendor. Internally, we’ve developed a benchmark that runs inside of the Lucidchart diagramming application. Our benchmark simulates many tasks that users typically do while diagramming, but on a much larger scale. For example, we connect a single block to dozens of others with lines, then rotate the entire diagram around dozens of times; we also add text to shapes and lines, move and resize objects, and so forth. Our entire benchmark includes sixteen different sections, each corresponding to a set of tasks that represent different actions users perform when using Lucidchart.

What does this benchmark measure?

We designed our benchmark to determine if a new release of our software is going to slow down the user experience. Not only do we measure overall runtime, but we actually instrument a number of critical functions so that we know, for example, how many times a piece of text is re-rendered. If those numbers go up, then we know we’ve introduced a new bug that hurts performance. We also use the benchmark to compare the performance of different browsers. With the recent release of Internet Explorer 10 on Windows 8 and some initial reviews of its performance (for example: here, here and here), we were interested to see how IE10 performance stacks up against the current versions of other browsers. Most browser benchmarks, such as SunSpider, the V8 benchmark suite, and so on, focus on important but narrow low-level functionality, such as mathematical calculation speed. Our benchmark is not directly comparable to these other benchmarks, as we’re attempting to measure user-visible speed of an entire graphical application. We are measuring a much wider variety of performance characteristics to answer questions such as:

• How quickly does the canvas API execute?
• How does the browser perform when we are rapidly changing the size of canvases?
• How quickly can major number-crunching happen on a complex data set in Javascript?
• How quickly can a very complex DOM including tens of thousands of elements be laid out and rendered?
• How quickly can we reorder, re-Z-index, re-style, render to, and move around thousands of elements in the DOM?

Enough already. Which browser is fastest?

The overall run time for the entire benchmark test in each browser is as follows: We ran our latest benchmark on the same Windows 8 laptop. Each test was run on a fresh launch of the browser in question. Unfortunately, we were unable to test IE9 under the same conditions to see how it stacks up against IE10, since you can’t install IE9 on Windows 8. Chrome still holds first place, but by a surprisingly small margin over IE10. Firefox repeatedly refused to complete the entire benchmark, crashing hard every time we attempted it. We gathered timing information for the parts of the benchmark that it did complete, and on those portions it performed comparably to IE10.

Is IE10 really that fast?

On a few of the sections, IE10 handily beats Chrome on timing. For example, in a test where we simulate typing 1500 characters into one of our custom rich-text editors, IE finished in 4.8 seconds vs. Chrome at 10.3 seconds. In a test of rotating a few dozen different shapes simultaneously, IE finished in 8.3 seconds vs. Chrome’s 17.3 seconds. However, when we actually watched the benchmark run, it was obvious that IE10 was skipping a lot of frames. All of the Javascript was run for each frame, but IE fully rendered about half of the frames to the actual screen. Chrome showed each and every frame of animation on-screen. Chrome absolutely crushed IE10 and every other browser on two sections that stress the CPU more than the rendering pipeline. In one, we have a dozen blocks, each connected to another via multiple long, curved lines, and each line has several pieces of text attached. The benchmark grabs half of these blocks and moves them around, resulting in a massive amount of calculations in Javascript to determine where lines intersect, where text should land, etc. In this section, Chrome finished in 19.1 seconds. IE10 finished in 71.7 seconds, Safari took 121 seconds, and Opera 166 seconds.

Performance breakdown

This chart shows the amount of time taken by each browser for each section of the benchmark (excluding Firefox, which could not complete the benchmark).

Conclusions

Chrome still holds the crown as the fastest overall performer for Lucidchart. IE10 made a surprisingly strong showing, performing far better than any other browser (though, under a more reasonable load, Firefox actually does quite well). There is one more strange thing about IE10 that I should mention. IE9 often maxes out one CPU core when a Lucidchart diagram is open, even when the user is completely idle. IE10 is somewhat better, but on our test machine it still used about 50% of one core when the user is idle. No other browser has any detectable CPU load under the same conditions. Whatever the cause may be, this is the one issue that keeps us from recommending IE10 as a first-class experience for Lucidchart.