Once we got the right size in each pair, we wore them for three months while hiking around Las Vegas, NV. Our real world testing included putting dozens of miles on each model and comparing them side-by-side, sometimes even switching shoes halfway through a hike. We then evaluated them on the following metrics: Comfort, Support, Traction, Water Resistance, Weight, and Durability. Here is how we specifically scored and tested each metric.
When determining the comfort of a hiking shoe we specifically only compared the comfort level of the midsole. While some models were more comfortable because they fit us properly (our head tester has long but somewhat narrow feet), a narrower shoe might feel terrible on someone with a wider foot. We did notice if there were pressure points from the laces and/or tongue, and if our toes fell asleep, but we tried to mostly keep the rating to something that we would all experience regardless of foot shape.
For support we looked at several components of the shoe: the arch support (both internal and the insole), lateral support, and support provided by the lacing system. Again, arch support is somewhat relative, but a good hiking shoe should have a least a modicum of arch support to prevent fallen arches over time. We rated the lateral support based on hiking experience (did the sole cave in on us, did we feel unstable on rocky ground, did our ankles cave in at all), and we also twisted each pair like wringing out a towel. The lacing on each pair and the adjustability they provided affect how tightly we could secure our ankles and if it minimized heel lift at all.
There were also several aspects that went into our traction score. A hiking shoe should provide traction on a variety of terrain, both going uphill and downhill. We noted if we had any slippage issues on the trails, and then took all the shoes out at once and went up and down rock slabs in each one. We noted how sticky, soft or hard the rubber was, and the effectiveness of the tread pattern.
We tested these shoes in and around Las Vegas, NV, which is usually not the best environment to determine water resistance. Luckily it was a wet year here, so we had plenty of streams and puddles to splash in. We also did a ten minute bucket test with each pair to determine the effectiveness of the waterproof liner, and how much moisture the uppers absorbed.
Manufacturers state their product's weight in a variety of shoe sizes, so we weighed each pair on a calibrated scale to at least have a relative measurement that we could compare.
While we spent hours in the field testing these shoes and hiked umpteen miles and thousands of vertical feet, most hiking shoes fail at around 300-500 miles, which we could not achieve in three months with ten pairs of shoes. Instead, we examined each pair for any signs of wear or weak spots, and then researched other online user reviews to see if there were any consistent patterns of wear. We were also able to examine some well-used pairs in our friends' shoe racks to get a sense of how they fared long term.