The bikes on our racks are often worth way more than the cars they are attached to. So we figured riders deserve the same rigorous and unbiased side-by-side bike reviews that they expect from the auto industry. In April of 2015, we set out to create an accurate time trial process to quantitatively compare one bike to another. We succeeded. We now call that process benchmark testing, and we take it seriously. It involves a lot of painful uphill sprinting, but we do it for you. Combining the time trial results with the riding experiences of our opinionated, expert test riders — each varying in riding style and body type — gives us a multifaceted view of each bike — where it sends, where it struggles, what type of trails and riders it is best suited for, and how fun it is along the way.
We used this protocol to test five 2016 enduro bikes, then six 2017 trail bikes. Each time we tweak the process to suit the bike category in question. This time it was five hardtail and five short-travel 2017 trail bikes. We had two separate test teams run them through the exact same process to help us define when and why a rider might choose one over the other. Here are things from the short-travel trail side.
Addressing Bike Build Issues
We test complete bikes as they were constructed by their manufacturers. While we attempt to standardize the level of bike builds to give a fair comparison, in this case, we chose bikes between $-- and $--, manufacturers must make trade-off decisions in build choices that affect how their bike performs. Rather than custom build a bunch of bikes that aren't available for you to purchase, we're testing the bikes that are available to you. Where we think a relatively small component change, often tires, could greatly benefit your experience, we'll let you know.
Bike Wheels and tires
Yeah, they're all different and their all as-spec'd, and some elements are pretty easy to change (i.e. tires). But all these switches cost money, and we want to show you what you are getting when the bike arrives in your life like the entrance of a movie hero, ready and willing to rock your world. Tires can be pretty location specific, these time trials were on well-drained hardpack that was pretty darn consistent throughout the test.
Benchmark Time Trial Testing
We time short laps for speed testing and we ride long laps for feel. Obviously, the feel and the fun is the important bit for a non-race focused trail ride, but pushing it on a bike and looking at hard numbers yields surprising and informative results. Often it confirms first impressions, but sometimes, it really challenges them, i.e. a fast bike feels slow or a slow bike feels fast, queuing us to dig deeper into what is going on with a bike. That's why we do benchmark time trial testing, here's how.
The Test Trail
First, we have to find the right trail, or trails, for the category. For hardtail and short travel trail bikes, we wanted mellow hardback punctuated by a few techy sections and some switchbacks. We found it in the form of a 1.5-mile loop trail. We ride about three minutes uphill and two minutes down with a short connector section, perfect for a bit of a warm up on each bike and a rolling start to ensure consistent speed coming into the course.
We call it The Burger Burner as the 1.5-mile loop outlines the upper and lower buns of a contour sandwich. (That, and we developed a post ride In-N-Out Burger habit — great recovery fuel.) One edge of the Burger is bitten up by switchbacks and the other rolls down a gently sloping side with few sesame-sized features.
We use the Freelap Timing System to record lap times, which tracks values up to one-hundredth of a second. Attaching a radio microchip (pictured left) to each bike's handlebars records the time when the bike passes a series of gates (pictured right) — in this case a start gate at the beginning of the climb, a lap gate set at the top of the climb, and a finish gate a the bottom of the descent.
Number of Testers and Laps Needed
The more times we have to average, the more confident we are in those times. We aim for 4 times per rider per bike, with a minimum of three test riders. This gives us 12 lap times per bike. So far, comparing times to performance comments and the individual order of each test rider's fastest bikes gives us a high level of confidence in our numbers. When our confidence levels are low, we have our testers run more laps. If our confidence is really low, we don't publish results. More on that later
Minimizing External Factors
To isolate the relative speeds of the bikes themselves, we had to think through how to mitigate many complicating factors such as trail conditions, rider fitness and fatigue, line choice and trail etiquette. Here's a list of the hoops we pedal through.
Bike Order — Each bike is ridden twice throughout testing to catch shifts in trail and rider conditions. We also switch up the order that each bike is ridden on test days, if it is ridden first one day it will be last the next, giving each bike an equal chance to shine. If a tester rides three bikes in a row, then she or he either rides it first and last or in the middle both days. Equality man. We're for it.
Laps per Day — The number of repeatable laps that each rider can complete depends on the length of the course in question and the fitness level of the tester throughout the test. We started all the testers out at four laps per test day, two on one bike and two on another. As riders demonstrated that they could complete six consistent laps, two on each of three bikes, we allowed them to do so.
The Human Factor — To reliably compare benchmarks early in the test to those completed later on, we need the energy and trail expertise of the testers to be extremely consistent throughout the test. So we cover the following points with our testers:
- Kill the competition. We don't want the testers to feel competitive pressure that could slowly speed up their times. So we give them plenty of space between riders so there is no tendency to want to catch the rider in front of you or out-pedal the one behind. We also don't let them share time results with one another.
- Be in shape. If a tester gained too much fitness during the test they could skew results. They must maintain a baseline fitness.
- Know the trails. Riders must be very familiar with the trails so they don't gain speed with increased comfort.
- Be real good at biking. They have to be comfortable pushing their speed, even on the technical elements.
- Think interval training. While we want repeatable efforts, we want them to ride as fast as they possibly can. We want to push the performance of these bikes under pressure.
- Have a uniform. Riders need to wear a similar kit every day to make sure drag is consistent.
- Trail Conditions — We always shoot to get testing done quickly - one because we're excited to tell you all about it, and two because it reduces the amount that trail conditions can change. We aim to find less variable trails, those with rock armoring or with nice, resilient hardpack. Then we dodge the weather. Driving all of us insane with schedule shifts to let trails dry out, get wet down or to avoid inconsistent head or tail winds. Luckily every day is a good day to ride so we all put up with the controlled chaos.
Even after all of those precautions, our downhill time trials for short-travel trail bikes failed to produce reliable results. The aspect of our descending course is strongly wind affected and we were unable to avoid the weather consistently enough to feel confident in our rankings.
Ride Quality, i.e. Flow Trail, Testing
Each tester also takes each bike out on their own pedal track, one that they are extremely familiar with riding on their personal bikes. They range from 10 to 15 miles and have more technical terrain than our test track with steeper, longer descents. Riders fill out questionnaires every test day, and after every flow lap. When they finish up all the testing on each bike, we interview them about their overall impressions and how it compares to the other bikes in the test.
We don't let the testers talk about testing to one another until the last day of testing, Ranking Day. We do this to preserve individual opinion and experience as they come to their own conclusions about the bikes. They fill out questionnaires throughout the test process and our interviewed by the project manager about each of the bikes.
This gives us a great deal of confidence in the conclusions that these very different riders come to separately. Then we come together to present you with a cohesive explanation of the bike's relative capabilities. Where our riders disagree, we figure out why so we can help you figure out which rider's opinion might mean more to you (e.g. body type or riding style).
In the end, we rank each bike's ability in a variety of metrics from 1 (truly terrible) to 10 (couldn't imagine anything better). We weight the metrics according to their relative importance. Fun, for instance, gets a 35% for this round. We add up the metrics to get each bike's score and award winning status.
Geometry measurements aren't standardized across the industry, so we measure all the bikes in a consistent way so that you can compare one directly to another. The measurements appear in the 2017 short-travel trail bike review's specification table. Our super sophisticated measuring tools include a laser beam, six-foot box beam level, a six-foot and a three-foot straightedge, a tape measure, a digital angle gauge, a park tool digital scale, and a grease pen. Here's how we take each measurement:
Effective Top Tube Length — We mark the center of the seat post directly across from the center of the head tube with the help of a six-foot beam level. Then we measure the distance with a six-foot straight edge.
Reach — Using the markings on the head tube and seat post, we hold a straight edge horizontally across the effective top tube and run a vertical laser beam through the center of the bottom bracket to the straightedge. This gives us the reach measurement.
Head Tube Angle — We attached an Intercomp Digital Angle Gauge to the head tube, often attaching it to the stanchions and fork lowers, to find its angle and cross-referenced results with a digital protractor goniometer angle finder to ensure accuracy.
Seat Tube Angle — We put the Intercomp Digital Angle Gauge on the seat tube when possible (there is a lot of interference), but the digital protractor goniometer angle finder often worked better. We placed it on the ground under the bottom bracket and adjusted the upper arm to be mimic the angle of the seat tube.
Bottom Bracket Height — We measure from the ground to the center of the bottom bracket to find it's height.
Standover Height — We set up the vertical laser beam 7" in front of the bottom bracket to measure the standover height because that's where most of our test riders actually stand over the bikes.
Chainstay Length — We use the six-foot straightedge to measure between the center of the rear axle to the center of the bottom bracket.
Wheelbase — Lining up the six foot straightedge with the center of both wheel axles gives us each bike's wheelbase.
Weight — A Park Tool Digital Scale gave us the weight of each complete bike build.
Our measurements and those of the manufacturers are in the chart below. Please note that the Trek Fuel EX and Santa Cruz Tallboy were measured in their low flip chip settings as we ran them on 29er wheelsets.