Membership questions? Log in issues? Email info@brewersassociation.org

Author Topic: Entry position within a large flight  (Read 2321 times)

S. cerevisiae

  • Guest
Entry position within a large flight
« on: May 12, 2014, 03:17:40 pm »
Has anyone else noticed that where an entry appears in a large flight can affect its score?

Offline tschmidlin

  • I must live here
  • **********
  • Posts: 8198
  • Redmond, WA
Re: Entry position within a large flight
« Reply #1 on: May 12, 2014, 03:20:10 pm »
Order bias is a well known issue in tastings of any sort.
Tom Schmidlin

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7781
  • Underhill VT
    • The Best Artist in the WORLD!!!!!
Re: Entry position within a large flight
« Reply #2 on: May 12, 2014, 03:21:53 pm »
I was reading/listening to an article the other day indicating that it can be an issue in hiring processes as well.
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline dmtaylor

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 4724
  • Lord Idiot the Lazy
    • YEAST MASTER Perma-Living
Re: Entry position within a large flight
« Reply #3 on: May 12, 2014, 03:35:26 pm »
It is a very real phenomenon.  The only resolution that I can think of, off the top of my head, is to generate very small flights of only 4-5 beers each, then taste all the beers twice, blindly, in a different order for each half of the total flight, then each judge compare their own scores for each beer at the end and choose the scoresheet that best depicts the true score.  A great judge should be within 2 points of their own score each time.  A heavily biased judge should be given three chances and then executed by whichever means of capital punishment is approved in their state.

;)
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

S. cerevisiae

  • Guest
Re: Entry position within a large flight
« Reply #4 on: May 12, 2014, 03:41:35 pm »
I am glad that I am not imagining things.  I gave a beer in the middle of the pack a 46 that placed 4th in a mini-BOS.  The 1st, 2nd, and 3rd place beers in the mini-BOS received lower scores within their respective flights.  It was a great beer that followed mostly mediocre to darn near undrinkable entries.  The difference between the beer that I gave a 46 the other beers in the flight was like the difference between night and day.  The flavors were so bright and clean.  The beer was also visually beautiful. The other judge on my flight also gave it a high score.  I felt really stupid for giving the beer such a high score; however, a very senior judge at the competition told me that it occurs very often, especially on large flights.  He called it the "middle of the flight effect." 

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7781
  • Underhill VT
    • The Best Artist in the WORLD!!!!!
Re: Entry position within a large flight
« Reply #5 on: May 12, 2014, 03:46:52 pm »
minimal judging experience speaking here:

I would guess an important skill to learn while judging is remembering that you are not a machine. You are, just like the rest of the judges, a human using what are essentially flawed and incredibly subjective tools in an attempt to create a reasonably objective result. keeping this in mind will help one both be a better judge (recognizing when bias might be entering into your observations) and be less stung when your judgement is not accurate to the judgement of others.
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline dkfick

  • Senior Brewmaster
  • ******
  • Posts: 1054
Re: Entry position within a large flight
« Reply #6 on: May 12, 2014, 04:27:49 pm »
I am glad that I am not imagining things.  I gave a beer in the middle of the pack a 46 that placed 4th in a mini-BOS.  The 1st, 2nd, and 3rd place beers in the mini-BOS received lower scores within their respective flights.  It was a great beer that followed mostly mediocre to darn near undrinkable entries.  The difference between the beer that I gave a 46 the other beers in the flight was like the difference between night and day.  The flavors were so bright and clean.  The beer was also visually beautiful. The other judge on my flight also gave it a high score.  I felt really stupid for giving the beer such a high score; however, a very senior judge at the competition told me that it occurs very often, especially on large flights.  He called it the "middle of the flight effect."
Same thing can also happen if the beginning of the flight is full of poor beers then you all of a sudden get that beer that is pretty good.  You can think that pretty good = awesome when in fact it's really just pretty good.
BJCP A0936 National Beer Judge and Mead Judge
Cicerone Certified Beer Server
AHA Member
CRAFT Homebrew Club
Sons of Liberty Homebrew Club
HBT "mors"

Offline sambates

  • Cellarman
  • **
  • Posts: 83
Re: Entry position within a large flight
« Reply #7 on: May 12, 2014, 10:09:01 pm »
This is more unlikely, but it's possible the bottle you got was bottled better than the bottle that got sent to best of show round. I don't think they could differ so much, but 1-2 points here or there could drop ya in BOS
Tap: Brandy Barrel Sour, Brett IPA, Tart Cherry Berliner, Sour Coffee Porter, Witbier, Helles Bock, Helles Bock Braggot, Strawberry "Nebraskambic," Raspberry Flanders
Secondary: OatRye Blonde, Plout Sour, Blueberry Ginger Sour
Primary:Belgian Dark Strong, Scottish Wee Heavy, American Session Wild Ale, Brandy Barrel "Lincolnambic"

Offline MDixon

  • Brewmaster General
  • *******
  • Posts: 2313
Re: Entry position within a large flight
« Reply #8 on: May 13, 2014, 06:40:53 am »
It sounds to me as if your judge pair was scoring higher than the other judge pair. A 46 is a rare score. I'm not saying it should not and isn't awarded, but if you put it at a 46 and the mini-BOS kicked it out entirely, your score was too high.
It's not a popularity contest, it's beer!

Offline hopfenundmalz

  • Global Moderator
  • I must live here
  • *****
  • Posts: 10678
  • Milford, MI
Re: Entry position within a large flight
« Reply #9 on: May 13, 2014, 06:48:52 am »
A couple of times when judging the senior judge would look over the scores, and would have a fairly high scoring early beer pulled to compare to a high scoring later beer to check for drift in the judging during the flight. This has only happened a couple of times, but it was a good technique. Of course this does not fix the difference in judge pairs and what happens in a mini-BOS.
« Last Edit: May 13, 2014, 06:50:37 am by hopfenundmalz »
Jeff Rankert
AHA Lifetime Member
BJCP National
Ann Arbor Brewers Guild
Home-brewing, not just a hobby, it is a lifestyle!

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 27093
  • Noti OR [1991.4, 287.6deg] AR
    • Dennybrew
Re: Entry position within a large flight
« Reply #10 on: May 13, 2014, 08:42:28 am »
Has anyone else noticed that where an entry appears in a large flight can affect its score?

Drew and I write about that in the evaluation chapter in our book.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

S. cerevisiae

  • Guest
Re: Entry position within a large flight
« Reply #11 on: May 13, 2014, 04:22:14 pm »
It sounds to me as if your judge pair was scoring higher than the other judge pair. A 46 is a rare score. I'm not saying it should not and isn't awarded, but if you put it at a 46 and the mini-BOS kicked it out entirely, your score was too high.

It was the first time that I have ever given a beer a score in the forties.  The other judge on the flight is a well-respected national judge, and he gave it a 44.   

I did not care for the beer that won the mini-BOS (I was given the opportunity to sample the mini-BOS winner beer after judging was complete).  It was visually unappealing and tasted more like a hop delivery device than a beer.  In my humble opinion, the beer won the round because it was hopped so much higher than the other beers in the round.  It had a ton of trash in the bottle.


I would like add that this contest was not your typical regional home brew contest.  It was the Brewer’s $6K.  American pale ales had to compete against taste bud destroying DIPAs.  The BOS winner is going to walk away from the contest $6,000.00 richer (the per entry fee was $40.00).  The winners will be announced this weekend during the Beer Bacon Music festival.

http://beerbaconmusic.com/homebrew-competition/
« Last Edit: May 13, 2014, 08:41:28 pm by S. cerevisiae »

Offline ajk

  • Brewer
  • ****
  • Posts: 279
  • Foam Blowers of Indiana (FBI)
Re: Entry position within a large flight
« Reply #12 on: May 13, 2014, 08:14:42 pm »
Order bias manifests in many ways. The first beer usually scores in the middle (judges hesitate to go straight for the extremes, figuring their palates just aren't yet acclimated/calibrated). It's also rare for a judge to assign very low or very high scores to more than 3  entries in a row (such clusters are rare but still possible, and the odds of "another 40 point beer" are just the same as the first). And we haven't even touched on palate fatigue!