Experience is a Poor Proxy for Skill
Experience is a Poor Proxy for Skill
Incident 1: An rpm install from source failed. The fail was in a python script. I found the bug in the script and fixed it in a couple of minutes.
Incident 2: A coworker came to me with a problem php script. I found and fixed the bug in a minute.
Incident 3: A C utility to inform the kernel that memory had been added to the system was failing. I tracked down the missing data, and fixed the utility in less than a week. (Not job time.)
What's the big deal? I just described my first encounter with the C, php, and python languages. And while I did pick up a C manual in order to read the Amiga kernel sources, I did not read any documentation on php or python. By definition, the originator of the bugs had to have had more experience than I.
I am not claiming that at the time of each incident that I was better at the given language than the person that produced the bug. But I will suggest that I might well have been a better programmer overall. More pointedly, I will suggest that I would be a better hire for a year-long project.
Domain-specific information is important--especially for lead positions. But often there is less there than one might thinks. For instance, the pain I went through hitting every "may" in the IEEE-754 standard for floating point arithmetic while building an emulator prepared me very well for the "surprise" I got when I first opened my standards-compliant website in Internet Explorer, ten years later.
Almost every job I look at wants a fast learner with three years experience in the tools they use. Think about it.
Nathan Zook
http://www.linkedin.com/in/nathanzook
O.K., what are you trying to get at? I know that most job descriptions are messed up and what employers are looking for isn't always the right thing, but how do you go about explaining that to them? I hate to say it but delving into minutia of technical details tends to have the result of making most hiring manager's eyes glaze over.
What I'm getting at is that we need better tools.
If the hiring manager is not technical, he has to use someone who is for that part of the evaluation. If necessary, ask a trusted friend for someone that can do it. The technical part of the interview needs to be focused on separating the expert from the average from the liar. An expert programmer is likely to come up with solutions that surprise the interviewer. An expert programmer is going to have a very dim view of poorly written code--even if it is in a language he has never seen.
The premise of my post is that the managers know that they want skilled programmers. My assumption is that they use experience as a proxy because it is they only tool in their box. My claim is that this screw is not adequately inserted via hammer.
I don't think that managers necessarily use experience as a proxy because it is their only tool... just that in an employer's market it is an easy way to weed people out. There is such a huge glut of skilled and experienced programmers available in the market today that employers can afford to be extremely demanding and picky. When they routinely get 500+ resumes within a few days for every IT job posting they start to look for arbitrary criteria for which to narrow the number down to a manageable level to call for phone screens because they probably don't want to do more than a couple dozen before they select the handful that will get in person interviews. So if someone isn't a 110% buzzword match to their current and expected future technology footprint and doesn't have an arbitrary number of years of experience with their key skills commensurate with the level they are hiring for, they file-13 the resume. They leave the subjective judgments as to who is expert/average/liar for the people who make it past the pre-interview "gauntlet".
On the other hand, I would certainly be interested in hearing your suggestions on ways to improve the process.
Nathan,
Great post, thanks for your insight.
I think what you're describing though is 'problem solving'. You definitely want clever, sharp people with some quasi-related experience to help you with problem solving. This is where interview trick questions and brain teasers come in.
However, when I build a team, I would much rather have a 3:1 or a 4:1 ratio of "Problem Preventers" to "Problem Solvers". You have to have a problem solver, no doubts there. Someone has to come in when all else has failed and get their hands dirty.
An ounce of prevention is worth a pound of cure. I'd rather have a bunch of preventers keeping most problems from happening in the first place, then have a bunch of problem solvers doing things wrong from the get-go, but very effective at solving the problems that they, themselves, usually created.
Disclaimer: I was, and still am to a lesser extent, one of these people. I was the hot-shot whiz-kid who could blast out code and solve problems left and right. But I left a wasteland of unmaintainable, buggy code behind me.
Shameless plug, I just recently wrote a blog post on this subject:
http://www.lostechies.com/blogs/chad_myers/archive/2008/04/25/the-proble...
------
This message brought to you by the Department of Problem Prevention
------
blog: http://chadmyers.lostechies.com
Here is a blog idea for you... how about a "top ten" list of commonly encountered software development problems along with some strategies/best practices to avoid them?
Good idea.
I something sort of like that, but not quite as explicit.
What do you think of this one?
http://www.lostechies.com/blogs/chad_myers/archive/2008/03/16/time-to-lo...
------
This message brought to you by the Department of Problem Prevention
------
blog: http://chadmyers.lostechies.com
Interesting, but as you say not quite as explicit as what I was thinking of. Most of what that blog article talks about is just building the prerequisites of a project. One would hope that those things could generally be taken for granted although I know that many shops aren't that mature. What I was thinking of was assuming you have a reasonably mature development environment, methodologies and processes in place, what are some of the most common pitfalls and how to avoid them. I've seen articles on how to identify and correct pitfalls like that in the past, but moving to being proactive is pushing things to another level.
So I was trying to think of the biggest mistakes I've seen that were the most costly. Usually, it was not necessarily coding mistakes.
1.) Only having binaries and no one knows where the source is (no source control, or not committing often enough)
2.) Not having an automated build, so the code can only be compiled on one box that has the correct magic applied to it
3.) Not having an automated deployment, so the code can only be assembled and deployed from one box by one person who knows all the correct incantations to get the software to a running state
4.) Not having any sort of smoke test or independent verification that the build and/or deployment was successful, resulting in the 'Works on my machine!' syndrome.
http://www.codinghorror.com/blog/archives/000818.html
5.) Not working towards automating testing as much as possible, creating huge friction during the testing phase of the project. Many times, regression testing is simply not performed as much as it should be because the cost/time is so great.
6.) Silo coding, solitary code ownership, code fiefdoms.
7.) Massive changes done in isolation for too long resulting in really nasty merges or, worse yet, undetected conflicts in code that lead to bugs
I'm running out of steam here. I'll have to think up 8, 9, and 10.
Anyhow, every single one of those 7 dysfunctions would be eliminated or nearly eliminated by doing those things I mentioned in that TTLS post. :)
See, it's all about problem prevention!
------
This message brought to you by the Department of Problem Prevention
------
blog: http://chadmyers.lostechies.com
If that's all there is to it, then you'll quickly put yourself out of work... :-) Well, #6 can be a tough one to deal with if you've got people with certain personality types in play.
I'm advocating quite the opposite. The implication was supposed to be than since I found these bugs so easy to fix, that I can reasonably claim that I would not make them in the first place.
If you create problems "in order to" solve others, your value to the company must subtract the debug and fix time for your bugs.
Consider the classic case of a for loop in C. There is nothing in the language to prevent placing the increment inside the block, and the culture actually encourages it. But this practice creates a problem--the next person to look at the code is going to have to find the increment. The fact that in the general case, state may be updated in the body does not make it acceptable to do so in every case.
I know this. I'm a problem preventer, and it is precisely my experience _outside_ of C that brought this particular matter to my attention.
I think that employers need both people who can write code "right" the first time (of course there are varying opinions as to what is right), and people who can solve problems (which will happen). The problem is how do you measure those skills in a meaningful way? I think that using experience as a substitute measurement often comes in because generally most talented programmers will hopefully learn a certain amount of both how to do things right (often by doing it wrong and seeing the results) and how to solve problems over a given period of time.
Both of these things are important, but even at that they are only a couple of factors in what it takes to make a great programmer. I've known and worked with people who were brilliant, wrote great code but still weren't overall great... for various reasons... some couldn't work inside a "system" or methodology or in a team and some were just plain insufferable to deal with.
Skills aren't buzzwords; I think that recruiters (though many are pretty bright) too often tend to focus on the experience buzzwords when they're really looking for someone interested in and capable of performing certain needed functions.
It's a bit like requiring candidates to list on their resumes the exact brands and models of cars they've driven, when all the company needs is a capable driver, and disqualifying those who haven't driven the same model(s) the company uses.
Similarly, I've known plenty of folks who could justly list experience with C++ on their resumes, but don't know the first thing about object-oriented programming, or using C++ (correctly) for any particular application. I've also known folks who've earned CS degrees and still have no idea how the computer makes their "Hello, World!" programs work, so we really can't use having a degree as a differentiator, either.
So, what can we do? I'm sure most folks have been told to try and quantify the results of their efforts. I'm pretty shy to claim credit for group efforts, especially when part of a great team. Any thoughts?
FYI: This thread is starting to get some interesting crossover with softwarejanitor's thread Survey response.