Friday, March 6, 2009

Fitter comparison

Here's a comparison of the fitters that I've currently got coded up. The fitters that are used are:

  • "chi2" - chi2 using the template errors and the r_xx values in the error calculation
  • "chi2NoRValues" - same as above but without the r_xx values in the error as pointed out by Klaus
  • "chi2NoErrorFromTemplates" - only using the error from the data in the chi2 error
  • "likelihood" - simple likelihood fit, but not quite the same as Roberval was using from the Barlow-Beeston paper. It's [d*ln(f)-f - d*ln(d)+d] which I got from the NIM paper I mentioned a while ago (NIM 221 (1984) 437-442).
  • "BarlowBeeston" - The Barlow Beeston algorithm, as best as I've managed to implement it.
I've also tried it ignoring bins where the data is less than 10. I've not had a proper think about whether this is a fair thing to do yet, but the results seem good.

r_bb
r_bb ignoring bins where the data is less than 10

r_cc
r_cc ignoring bins where the data is less than 10

r_gg
r_gg ignoring bins where the data is less than 10

I also had a look at what they're like when cheating the background contribution, and it's pretty much a small constant shift towards the true value. It's not huge though so I haven't posted the plots - the efficiency of the current selection is quite reliable.

I'm not that keen on using the chi2 anymore because the results aren't very good without the r_xx values in the errors. Barlow-Beeston is still giving some strange results, not that visible here but the pulls on the toy study look strange (I'll post them once I've done a bit more work on that). I think it's more to do with my (currently) buggy implementation more than the method. As Roberval pointed out though, there is the approximation of Binomial errors to Poisson which is only valid when the entries in a bin are much less than the total entries.
The likelihood looks okay, so Im edging towards that. All of the r_bb results have a bias though, which I don't understand.

Everything is improved when cutting out bins with less than 10 data, so I guess a final decission depends on if we agree that is a fair thing to do. I'll also try each method with the toy study - multiple test might shed a bit more light.

No comments: