Can you please give us an overview of Evidenced-Based Scheduling (EBS)?
FogBugz uses a sophisticated statistical algorithm called Evidence-Based Scheduling (EBS) to produce ship date probability distributions. EBS was developed at Fog Creek Software, and is exclusive to FogBugz.
EBS uses a statistical method called bootstrapping that we have found does a very good job of predicting software schedules.
The first time you use EBS, it will probably come up with a ship date that seems much later than you were hoping for, and you might despair. But after a while, you'll understand why it's doing this, and you'll see that the schedule that EBS generated, pessimistic though it seemed, was actually right on the mark. EBS is an algorithm that understands why software projects come in late and actually incorporates that into its estimates, mathematically. EBS actually looks at the historical track records of every person who enters estimates, and assumes that the future will look a lot like the past.
How the algorithm works
EBS doesn't give you a single ship date. Instead, it produces a probability distribution curve. That means that for any given date, it tells you what the probability is that you will ship by that date.
To do this, EBS needs to keep track of the historical track record of each estimator. An estimator's track record is just a list of all the features that they estimated which were implemented and closed. In order to be considered a historical estimate for a user, a case must be estimated by that user, have time charged against it, and then be resolved and closed. The resolution must be "Completed", "Fixed", "Implemented", or "Responded." For each completed feature, EBS looks at the original estimate entered and the actual time elapsed. It divides estimate by elapsed to get the velocity of that feature's implementation. A feature that takes twice as long as expected has velocity 0.5.
EBS collects as many as 200 recent velocities from each estimator. If an estimator has no historical track record or has fewer than six completed features, EBS backfills that track record with six velocities that reflect a very poor estimator. This way, in the absence of historical data, EBS assumes the worst about the estimator.
Once EBS has a track record for each estimator, it can simulate the future. EBS does this simulation 60 times. In each round of the simulation, and for every feature that needs to be implemented, it uses a velocity randomly chosen from the estimator's track record.
When EBS calculates the schedule repeatedly, for 60 possible futures, it assumes that each possible future will occur with 1/60th probability. This gives it a probability distribution curve with 60 data points.
To make a schedule for one developer, EBS adds up all the estimates of that developer's features to get a total number of hours of work. But instead of actually taking the estimates at face value, it uses adjusted estimates, which reflect the estimator's historical track record.
To get an adjusted estimate, EBS starts with the estimates entered into FogBugz (minus elapsed time). Then it picks a pseudo-random number n between 1 and x, where x is the number of velocities in the estimator's track record. It then sorts the velocities in ascending order and picks out the nth velocity from the list. Since n is real, it actually picks the two closest velocities - the (floor(n))th and the (floor(n+1))th - and interpolates linearly between them to get a single velocity. In other words, EBS picks a random velocity with the same distribution as that estimator's historical velocities.
Next, EBS divides the entered estimate for the feature by the randomly chosen velocity and uses this as the adjusted estimate for this particular imagined future. Unless every estimator has a perfect track record, each of the 60 futures will be a little bit different.
EBS scales adjusted estimates if the developer does not work 100% on that project; for a developer who works 20% time on that project, a 1 hour feature will be scaled to 5 hours.
Once EBS has the sum of a developer's adjusted estimates, it spreads those hours out on the calendar, the developer's work schedule, vacations and holidays, and figures out the calendar date when they would be done. It also takes any project dependencies into account, and assumes that the developer will do all priority 1 features first, followed by all priority 2 features, etc.
In each iteration, EBS calculates the schedule for each individual developer, and then assumes that the project will ship on the date that the last developer finishes their last task.
Why the algorithm works
EBS is not affected by the details of how estimators come up with their estimates. For example, it doesn't really care if they include coffee breaks in their estimates or not, as long as they're consistent about it.
EBS degrades very gracefully in the absence of historical data.
EBS handles rare outliers very well.
EBS handles estimators whose estimates improve.
EBS correctly accounts for real-world estimating errors.
EBS correctly accounts for vacations and holidays.
EBS correctly accounts for lazy developers.
EBS correctly accounts for team leaders who still like to meddle in the code.
Although there is a topic related to EBS and pair programming on Stack Overflow, it would be cool to get an official answer here as to how pairing and EBS work together.
Should estimates be made in calendar time or in person-hours? (In other words, John and Laurie are going to pair on a case that they think they can complete together in 6 hours. Do you put in an estimate of 6 hours, or do you put in an estimate of 12?)
Probably depending on the answer to the above, should both developers record time against the case, or should only one of them do so? Having them both record time is beneficial, since that means we can pull that data for billing purposes.
How do the answers above affect the ability to look forward and see completion predictions for a team?
I realize there are probably a number of ways to tackle this issue, but I'd like to know what the "official recommendation" from Fog Creek is.