Benchmark Compulsion

Did you ever wonder why as programmers, we reflexively reduce the complexity of an expression or function in code? Elimination of extra code, grouping similar bits of code, shortening statements?

It can become an obsession. We think that shorter code will give us a result that is easier to read, or faster execution, or less memory. When we go too far, contemporaries criticize us for engaging in “premature optimization”. We do battle finding shorter ways to rewrite each other’s code. And sometimes it can be a puzzle: “I bet I can make that code shorter”.

I wonder if the root of this reflex is part of our natural language ability, and that finding shorter ways to say things is a reflex of any human language speaker. Contractions of the form “do not” to “don’t” are certainly common in English. Is there some natural impulse to economize going on here? The topic of how natual languages relate to programming languages is certainly an interesting one.

This project is aimed at comparing the performance of different language systems for a problem bigger than typical toy problems used in benchmarks.

So how should we compare the results?

Memory and Time constraints

In embedded systems, memory is usually a constraint. This is less so as memory prices fall, but small devices still are memory constrained relative to desktop computers. Graphics cards or cell processors have unusual memory constraints. Making programs and their use of data small is a requirement in these environments.

In real-time systems, such as data-acquisition systems, or real-time control systems, speed of execution is a constraint. In financial exchanges, latency is a key competitive factor.

What is the question

One common battle in the programming language wars is about which language is faster. Proponents of each of the languages have their own position on this. C programmers say that nothing can be faster. A Fortran proponent will point out measurements of matrix manipulation or Mandelbrot computations where Fortran programs win hands down. High-level language proponents will point out that improvements in algorithms will trump line-by-line speed. We can look at generated instructions, reference optimization issues raised by aliasing. Better than theory, however, is to measure.

The canonical site for these discussions is The Computer Language Benchmarks Game. Browsing through the different kinds of problems reveals a different language winner in terms of speed, for each problem.

However, there are often several programs entered in a given category, tagged as “Fortran Intel” and “Fortran Intel #3”, where there have been different codings of the same algorithm in that language.

So the rankings here are really not “which language is fastest” but “which program is fastest”. Application of programmer skill will push the time down, as presumably better knowledge of the language runtime will result in a better time. And there is variance amongst compilers for a given language.

And the question “which language is fastest” can be interpreted as “which language gets me to the solution fastest”, not something exposed by the shootout site.

Other metrics for program speed

Total execution time for a program is the most common metric. For real-world problems, other metrics might be considered:

A bigger simulated problem

The problems considered in the shootout site seem to be chosen to be compute bound. There are certainly many real-world problems that have other pain points. If your engineering problem is a compiler, disk throughput will influence the outcome. If you are doing a real-time system, latency and predictability will be important design goals. If you are doing a transactional system, contention for resources will be a factor.

It would be interesting to see how a larger problem works out with respect to languages. I propose a simple financial exchange supporting real-time, high-frequency trading as one possibility.

Low-Latency trading benchmark

To outline the requirements of this prototype exchange,

What language to choose

Having convinced ourselves that speed is not exactly a property of a language, how might we determine which environment to use in a complex real-world application or in our simulate exchange?

Some considerations might be

I am sure you have seen the hidden agenda before in this kind of discussion. “Friends don’t let friends do Java” or “Everyone knows that C is fastest, just man up and deal with it”.

Let’s choose several

For our simple exchange, we will choose a set of languages to implement client and server in and compare key metrics for each implementation. In particular, the metrics, in order, are:

Since the skill of the programmers is limited to my skill, a small number of languages will be chosen. These are ones thought to be genuine real-world candidates for such a system, and ones with which I have experience. These are, in order of implementation:

With these choices, we can compare the performance of the three languages by the metrics above, beginning with latency.

Next Steps

In other chapters, let’s lay out the basics of operation of the exchange, and the details of communication between client and server.