I wanted to start something about defensive metrics because there are a lot of things people don't understand.
Misconception 1: Defensive metrics don't mean anything because UZR and DRS disagree a lot. This has nothing to do with the usual argument against defensive metrics. UZR and DRS have the same data, they just have different methodologies. This is the equivalent of saying ERA estimators are completely unreliable because SIERA and FIP and xFIP disagree a lot.
Misconception 2: Total zone. Anything about total zone. Most people don't know that total zone does not use hit location data. It just guesses based on batter and pitcher handedness. Something else it does is it says the player who picked up the ball should have made the play. So it will credit bad players with not having to make the play because they couldn't get to it.
If this is true, this is huge for defensive metrics. He says that if every borderline play was in a different bucket, the most a fielding metric could be off by would be +/- 3 runs. He also says he compared gameday hit location data to Matt Thomas' Cardinals data using cameras and there was no evidence that there was bias.
The UZR part 1 and 2 articles and the Fangraphs UZR primer contain the methodology. I also linked a great post by MGL at the book blog stating the differences between UZR and DRS.
Here are all the metrics and their strengths and weaknesses.
UZR: Strengths: very good all around metric. Uses hit location and adjusts for hit speed. Weaknesses: Hit location may be biased.
DRS: Strengths: Uses hit location like UZR and uses hang time instead of subjective hit speed which is huge. Weaknesses: Hit location bias may be even worse as the buckets in DRS are even smaller and the ball could be in a different bucket much more easily than in UZR. Even if hit location is not biased, the very small buckets may have sample size issues for performance in them.
Total Zone: Uses PBP data, doesn't have any bias with hit location. Weaknesses: Guesses hit location, the picking up the ball problem
FRAA: Strengths: Doesn't have any bias in data. Weaknesses: Doesn't even use PBP data, I would never use it.
Fans Scouting Report: Strengths: People who see the games a lot evaluate the players. Weaknesses: Very, very subjective, if you complain about UZR or DRS being too subjective, this is on anther level.