Abstract
We're all lazy, to so some extent, and prefer to think about things that are easy to understand. In order to keep your focus on what's important, recognize that people have this tendency to get distracted by what's easy. I don't have an advice beyond that -- you'll have to do the hard work yourself.
Start
The founder of Clojure gave a talk called Simple Made Easy on simplicity and complexity. What I remember from it is the idea the we, as humans, reach for ideas that are easy and at hand. Furthermore, ideas that are easy and at hand aren’t necessary simple. They can be complex and yada yada yada now you should to use this convoluted map-reduce paradigm. transcript
Examples
We reach for ideas that are easy and at hand and you see examples of this all the time:
- Tabs vs. Spaces
- Code commenting style
- Emacs vs. Vim
- Speed of code
- Python vs. R
Search
I don’t know why computer scientists are obsessed with speed. For example, Google reports – proudly – how quickly they return their results. This is nice but as a consumer I don’t care. I care about accuracy first and foremost and that's why everyone used Google in the first place. Whether the results take 0.4 or 0.6 seconds doesn’t matter – it just needs to be fast enough.
You can imagine bosses like the speed metric (which is why it's so prominent). It’s an easy thing to test and and easy thing to understand. Testing accuracy, on the other hand, is hard. You have to figure out what the correct results are supposed to be – and once you have that you have to figure out what metric to use. Mean average precision at 10? You have to order things, avoid bad results but also not miss results...It quickly becomes hard to measure and understand. You want to go back to thinking about speed!
My only point is a warning. When you hear people in deep learning talking about the speed of a backward pass...think about what’s really important. It’s accuracy! Who cares how long it takes? We wouldn’t care about Google if all they did was return search results quickly.
Cars
Point 2: statistics don’t capture everything. Take a sports cars, we care about horsepower. More is better. The savvy consumer cares about torque (“Horsepower sells cars, torque wins races.”) and the power to weight ratio. There’s the cornering acceleration (often over 1g), weight ratio, and on and on. But none of these things tell you what you really care about: What is it like to drive?
So it is with programming languages. There are things you care about that are hard to quantify: What's it like to debug code? What's the learning curve like? What’s it like getting help from the documentation or community?
Deep Learning
There's a popular benchmark for deep learning maintained by Soumith Chintala at Facebook AI. It's useful information but keep it in perspective.
Conclusion
Don’t let things that are easy to measure distract you from the things you really care about.