David R. MacIver's Blog

The right tool for the job

James Iry recently wrote a post about type systems in which he ranked type systems according to two characteristics, safety and sophistication. We had an interesting discussion about it on IRC afterwards, in which one of the things discussed was whether these were really the right axes.

It occurred to me that we spend a lot of time talking about the major axes on which you compare programming languages, and that it would be interesting to gather some data on what these actually were instead of just plucking things that seem like they may sense out of the air. So I’ve created a little app to gather data for this: Check out The Right Tool, and I’d love it if you’d rank some languages on it.

Essentially the idea is this: I have a list of statements and a list of programming languages, and I want you to rank those languages in order of how well the statements apply to them.

At the moment I’m just data gathering: I don’t actually do anything with the data. The end goal is that I’m going to have a very high dimensional representation of each programming language in terms of what people think of them and why they use them which I will do some sort of dimension reduction to, hopefully giving rise to some interesting metrics with which to compare languages.

A couple notes:

Comments

rtaycher on 2010-05-09 10:12:51:

are rankings top (most) to bottom(least) or vice versa?

Harald Korneliussen on 2010-05-09 10:13:13:

I filled out a couple, but it’s really hard to try to rank all of these languages, some of which I have barely heard about, on all sorts of dimensions.

Some things that would make it easier and more reliable
1. Allow ranking two languages the same, and let everything be ranked equally in the beginning
2. Randomize the order of the entries, or you will see high in the alphabet languages on top a lot...
3. Most radical suggestion: exclude all but a couple of languages randomly on each question. At the most extreme, a “hot ot not”-like format where you only compare two languages(in one question) at a time. You’ll get less data, but better quality, and more people can take the time to participate at all.

Orion Edwards on 2010-05-09 12:23:34:

This is a fascinating idea, I’d love to see what eventually comes from this :-)

After the first question where I selected the 9 or so languages I know anything about, it was quite nice to use, but the initial act of selecting those 9 from the 40 odd candidates sucked. Drag/drop using a list which scrolls offscreen is terrible. Might I suggest a series of checkboxes for the initial “pick the languages that you know” step?

Cheers

david on 2010-05-09 19:50:25:

Yeah, I heard that a lot and finally listened. I’ve just changed around the UI to do what you suggest: There’s an initial “pick your languages” phase and a separate UI for choosing them. It does seem to be a big improvement.

In terms of what eventually comes of that, most of that is still nascent, but here’s a first peek at it: http://therighttool.hammerprinciple.com/languages and http://therighttool.hammerprinciple.com/statements will take you to breakdowns of statement by language and language by statement. The individual languages are fascinating.

Peter Niederwieser on 2010-05-09 16:28:48:

When I hit “Rank!”, I get: Internal Server Error

david on 2010-05-09 16:31:51:

Bah, bloody typical. In working around a Heroku bug I inadvertently introduced a bug of my own. Ruby definitely goes down in my ranking of “software I write in this language is reliable”. ;-)

Try again now. It should work.

Rafael de F. Ferreira on 2010-05-09 20:55:09:

The description talks about Right and Left hand lists¸ but I only see a single list of movable boxes. What am I doing wrong?

david on 2010-05-09 20:56:43:

My fault! The description is of the old UI. I’ll fix.

Rohan on 2010-05-10 14:54:15:

Can you add a “skip to last” button? it gets boring to rank all 64 statements.

great effort though, looking forward to more programmers ranking this.

John Haugeland on 2010-05-12 00:24:28:

I’d like to see Delphi, Mozart-Oz, FormulaONE, Fortress, Mythryl and Coq added to the list, please.

John Haugeland on 2010-05-12 00:24:52:

Oh, and SQL.

david on 2010-05-12 08:59:38:

I’ve added Delphi, Mozart-Oz and Coq. For reasons that make sense in my head but I can’t articulate I’ve decided against FormulaONE, Fortress and Mythryl now (there has to be some cutoff - people are already complaining about the number of languages in the system).

I deliberately chose to omit SQL because this more about general purpose programming languages, so I don’t think it really fits.

Alexey Romanov on 2010-05-16 12:52:04:

When I looked at the “see and edit your previous rankings” button, I saw rankings for the questions I’ve skipped as well.

david on 2010-05-16 13:11:29:

Did you see empty rankings there? If so, that’s by design. If not, that’s a bug and I’ll look into it.

Alexey Romanov on 2010-05-16 13:24:28:

No, not empty. I think they are in order they were ranked when I clicked “Skip”

david on 2010-05-16 13:28:15:

Hmm. That’s quite unfortunate! Thanks, I’ll look into it.

(On the plus side, the order they’ll be in at the start is deliberately random, so hopefully this shouldn’t have introduced too much bias)

david on 2010-05-16 13:34:10:

Ok. This should be fixed now. Thanks for the bug report.

Nathan E. Rasmussen on 2010-05-16 22:27:05:

I’ll be watching with interest.

Any thoughts of running rank-order correlation between pairs of statements? I’d like to see which statements turn out to be synonyms/antonyms/independent based on aggregate responses.

david on 2010-05-16 22:39:10:

Yes, it’s very much on the to do list. I just haven’t gotten around to it yet.

k on 2010-05-18 17:45:14:

A more useful thing might be to do a principal components/factor analysis on the rankings to identify major dimensions underlying the rankings, and scale the languages on those dimensions, rather than on a per-statement basis.

david on 2010-05-19 08:40:40:

I’ve tried a few variations on that: It was actually the original intent of the project. there are some more variations I could try, but the problem is that the various axes for the data are highly non orthogonal, so the principle component results are somewhat dubious.

k on 2010-05-21 23:00:58:

I tried to respond to your response, but it didn’t let me for some reason...

Anyway, you could rotate the loadings to a non-orthogonal solution (e.g., using promax or geomin or something), if that hasn’t been tried.

It’s very interesting to me that the axes are best thought of as nonorthogonal (which makes sense).

Anton Ertl on 2010-05-17 09:39:54:

Cool concept. Maybe you could provide an interface that works without Javascript
to avoid excluding those (like me) who are to concerned (or paranoid) about security
to enable JavaScript. I have to admit that this will probably result in a pretty cumbersome
interface.

david on 2010-05-17 10:22:40:

Err. Pass, sorry. :-) I’ve no idea how we’d do a good pure html “rearrange these items” list.

We might do something about this later, but it’s going to be pretty low priority.

Graham on 2011-10-28 14:42:51:

Comma seperated list in text box ?

eg. 3,2,1,5,4

quarc on 2010-05-20 19:57:19:

Well, even with Javascript enabled, I cannot proceed beyond the first
question where I tick those languages I know. After sending, I am
presented with the same sheet (entries cleared) again.
It is a bit boring. What strange browser configuration do you need
for this?

david on 2010-05-20 20:13:24:

I dunno. What strange browser configuration you got? We’ve tested and verified it works fine in chrome, firefox and safari, and have tested and verified that it works with a few minor visual quirks in IE. What are you using?

david on 2010-05-20 20:18:31:

Oh. You have cookies disabled, right? Site won’t work without them.

Dan on 2010-07-26 16:46:39:

Lovely project. I’d be keen to see graphical programming languages in the list so that we could get some text-vs-graphics opinions emerging. In my work (audio+music) there are visual programming languages Max/MSP and Puredata (among others), and text languages SuperCollider and Csound (among others). It’s a constant and interesting topic of discussion to compare them - text-vs-graphical is never the only difference, of course.

Andrés on 2010-10-01 20:47:55:

Trying to add new statements gives an “Internal Server Error”.

Another step down for the reliability ranking of Ruby! :P

Anonymous on 2011-07-23 23:01:41:

Trying to add new statement or programming language item results in “500” error. I list it in here.

* Programming Languages: *
CWEB
TeX
LaTeX
INTERCAL
WEB
METAFONT
PostScript
FurryScript
TROFF
Icoruma
Inform
Game Maker
QBASIC
GWBASIC
Inform 7
LLVM
dc
SQL
dBase
Csound
Puredata
MegaZeux
MegaZeux Forth
ZZT

* Statements: *
I DISLIKE THIS PROGRAMMING LANGUAGE
THIS PROGRAMMING LANGUAGE IS GOOD FOR GAMES
THIS PROGRAMMING LANGUAGE IS GOOD FOR GRAPHICS
THIS PROGRAMMING LANGUAGE IS GOOD FOR TYPESETTING
THIS PROGRAMMING LANGUAGE IS GOOD FOR EVERYTHING
THIS PROGRAMMING LANGUAGE IS GOOD FOR NOTHING
THIS PROGRAMMING LANGUAGE IS GOOD FOR AUDIO
THIS PROGRAMMING LANGUAGE IS STUPID
THIS QUESTION IS STUPID
THIS PROGRAMMING LANGUAGE IS GOOD FOR DESCRIBING AND EXPLAINING PROGRAMS
THIS PROGRAMMING LANGUAGE IS BAD FOR DESCRIBING AND EXPLAINING PROGRAMS

Anonymous on 2011-07-23 23:06:51:

I forgot some programming languages:

Lisp
C--
BASIC
bc
A+
Cyclone

Aaron Davies on 2011-10-19 22:46:15:

awesome site, it’s amazing how much valid info you already have with <150 respondents.

any plans for visualizations? clusters of languages/features, etc., would be cool

Louis on 2011-10-20 03:28:34:

Here is a big one you forgot. Actionscript ( 3 ) You have Haxe... but they are very different.

Zack on 2011-10-20 19:28:52:

I second the nomination of AS3 (Actionscript 3).

Joe on 2012-03-30 13:54:32:

I’m new to programming and I’ve been Googling around to get my bearings before diving deep into any one programming language. Clicking around your site was a very efficient way for me to form a general idea of the space. Thank you so much!
I like how you’ve expanded into other domains like gin and databases. I’d like to use it to rank professors, fraternities, courses and rotation sites at my school.

Arne Babenhauserheide on 2012-07-23 15:07:13:

A nice addition might be build tools:

* pure make
* autotools
* cmake+make
* scons
* waf
* …

david on 2012-07-23 15:09:38:

Oh, that’s a good idea, thanks.

servicio tecnico on 2014-11-11 01:00:41:

There’s certainly a lot to learn about this subject. I love all the points you’ve made.