Author Topic: Entry position within a large flight  (Read 716 times)

Offline S. cerevisiae

  • Senior Brewmaster
  • ******
  • Posts: 1639
  • deus ex machina
    • View Profile
Entry position within a large flight
« on: May 12, 2014, 02:17:40 PM »
Has anyone else noticed that where an entry appears in a large flight can affect its score?
Mark V.

Just say "no" to yeast rinsing
https://www.homebrewersassociation.org/forum/index.php?topic=19850.msg252492#msg252492

Friends don't let friends use Star San as their primary sanitizer

"Acid-anionic sanitizers are broad spectrum against bacteria and viruses, but not very effective against yeasts and molds."

Offline tschmidlin

  • I must live here
  • **********
  • Posts: 8197
  • Redmond, WA
    • View Profile
Re: Entry position within a large flight
« Reply #1 on: May 12, 2014, 02:20:10 PM »
Order bias is a well known issue in tastings of any sort.
Tom Schmidlin

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7487
  • Underhill VT
    • View Profile
    • The Best Artist in the WORLD!!!!!
Re: Entry position within a large flight
« Reply #2 on: May 12, 2014, 02:21:53 PM »
I was reading/listening to an article the other day indicating that it can be an issue in hiring processes as well.
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline dmtaylor

  • Senior Brewmaster
  • ******
  • Posts: 1453
  • Two Rivers, WI
    • View Profile
Re: Entry position within a large flight
« Reply #3 on: May 12, 2014, 02:35:26 PM »
It is a very real phenomenon.  The only resolution that I can think of, off the top of my head, is to generate very small flights of only 4-5 beers each, then taste all the beers twice, blindly, in a different order for each half of the total flight, then each judge compare their own scores for each beer at the end and choose the scoresheet that best depicts the true score.  A great judge should be within 2 points of their own score each time.  A heavily biased judge should be given three chances and then executed by whichever means of capital punishment is approved in their state.

;)
Dave

"This is grain, which any fool can eat, but for which the Lord intended a more divine means of consumption. Let us give praise to our Maker, and glory to His bounty, by learning about... BEER!" - Friar Tuck (Robin Hood - Prince of Thieves)

Offline S. cerevisiae

  • Senior Brewmaster
  • ******
  • Posts: 1639
  • deus ex machina
    • View Profile
Re: Entry position within a large flight
« Reply #4 on: May 12, 2014, 02:41:35 PM »
I am glad that I am not imagining things.  I gave a beer in the middle of the pack a 46 that placed 4th in a mini-BOS.  The 1st, 2nd, and 3rd place beers in the mini-BOS received lower scores within their respective flights.  It was a great beer that followed mostly mediocre to darn near undrinkable entries.  The difference between the beer that I gave a 46 the other beers in the flight was like the difference between night and day.  The flavors were so bright and clean.  The beer was also visually beautiful. The other judge on my flight also gave it a high score.  I felt really stupid for giving the beer such a high score; however, a very senior judge at the competition told me that it occurs very often, especially on large flights.  He called it the "middle of the flight effect." 
Mark V.

Just say "no" to yeast rinsing
https://www.homebrewersassociation.org/forum/index.php?topic=19850.msg252492#msg252492

Friends don't let friends use Star San as their primary sanitizer

"Acid-anionic sanitizers are broad spectrum against bacteria and viruses, but not very effective against yeasts and molds."

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7487
  • Underhill VT
    • View Profile
    • The Best Artist in the WORLD!!!!!
Re: Entry position within a large flight
« Reply #5 on: May 12, 2014, 02:46:52 PM »
minimal judging experience speaking here:

I would guess an important skill to learn while judging is remembering that you are not a machine. You are, just like the rest of the judges, a human using what are essentially flawed and incredibly subjective tools in an attempt to create a reasonably objective result. keeping this in mind will help one both be a better judge (recognizing when bias might be entering into your observations) and be less stung when your judgement is not accurate to the judgement of others.
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline dkfick

  • Senior Brewmaster
  • ******
  • Posts: 1014
    • View Profile
Re: Entry position within a large flight
« Reply #6 on: May 12, 2014, 03:27:49 PM »
I am glad that I am not imagining things.  I gave a beer in the middle of the pack a 46 that placed 4th in a mini-BOS.  The 1st, 2nd, and 3rd place beers in the mini-BOS received lower scores within their respective flights.  It was a great beer that followed mostly mediocre to darn near undrinkable entries.  The difference between the beer that I gave a 46 the other beers in the flight was like the difference between night and day.  The flavors were so bright and clean.  The beer was also visually beautiful. The other judge on my flight also gave it a high score.  I felt really stupid for giving the beer such a high score; however, a very senior judge at the competition told me that it occurs very often, especially on large flights.  He called it the "middle of the flight effect."
Same thing can also happen if the beginning of the flight is full of poor beers then you all of a sudden get that beer that is pretty good.  You can think that pretty good = awesome when in fact it's really just pretty good.
BJCP A0936 National Beer Judge and Mead Judge
Cicerone Certified Beer Server
AHA Member
CRAFT Homebrew Club
Sons of Liberty Homebrew Club
HBT "mors"

Offline sambates

  • Cellarman
  • **
  • Posts: 74
    • View Profile
Re: Entry position within a large flight
« Reply #7 on: May 12, 2014, 09:09:01 PM »
This is more unlikely, but it's possible the bottle you got was bottled better than the bottle that got sent to best of show round. I don't think they could differ so much, but 1-2 points here or there could drop ya in BOS
Tap: Doppelbock, American Pale Ale, Hefeweizen, Kettle Soured Berliner Weisse, Black Saison, Scottish 80/-, Wild Cider, Tart Cherry Porter, RyePA
Secondary: Flanders Red, Lambic, Flemish Barrel Sour, Dark Strong Sour
Primary:

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1263
    • View Profile
    • Mike's Homebrewing Page
Re: Entry position within a large flight
« Reply #8 on: May 13, 2014, 05:40:53 AM »
It sounds to me as if your judge pair was scoring higher than the other judge pair. A 46 is a rare score. I'm not saying it should not and isn't awarded, but if you put it at a 46 and the mini-BOS kicked it out entirely, your score was too high.
It's not a popularity contest, it's beer!

Offline hopfenundmalz

  • Official Poobah of No Life.
  • *
  • Posts: 6312
  • Milford, MI
    • View Profile
Re: Entry position within a large flight
« Reply #9 on: May 13, 2014, 05:48:52 AM »
A couple of times when judging the senior judge would look over the scores, and would have a fairly high scoring early beer pulled to compare to a high scoring later beer to check for drift in the judging during the flight. This has only happened a couple of times, but it was a good technique. Of course this does not fix the difference in judge pairs and what happens in a mini-BOS.
« Last Edit: May 13, 2014, 05:50:37 AM by hopfenundmalz »
Jeff Rankert
Ann Arbor Brewers Guild, AHA Member, BJCP Certified
Home-brewing, not just a hobby, it is a lifestyle!

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 13970
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Entry position within a large flight
« Reply #10 on: May 13, 2014, 07:42:28 AM »
Has anyone else noticed that where an entry appears in a large flight can affect its score?

Drew and I write about that in the evaluation chapter in our book.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

Offline S. cerevisiae

  • Senior Brewmaster
  • ******
  • Posts: 1639
  • deus ex machina
    • View Profile
Re: Entry position within a large flight
« Reply #11 on: May 13, 2014, 03:22:14 PM »
It sounds to me as if your judge pair was scoring higher than the other judge pair. A 46 is a rare score. I'm not saying it should not and isn't awarded, but if you put it at a 46 and the mini-BOS kicked it out entirely, your score was too high.

It was the first time that I have ever given a beer a score in the forties.  The other judge on the flight is a well-respected national judge, and he gave it a 44.   

I did not care for the beer that won the mini-BOS (I was given the opportunity to sample the mini-BOS winner beer after judging was complete).  It was visually unappealing and tasted more like a hop delivery device than a beer.  In my humble opinion, the beer won the round because it was hopped so much higher than the other beers in the round.  It had a ton of trash in the bottle.


I would like add that this contest was not your typical regional home brew contest.  It was the Brewer’s $6K.  American pale ales had to compete against taste bud destroying DIPAs.  The BOS winner is going to walk away from the contest $6,000.00 richer (the per entry fee was $40.00).  The winners will be announced this weekend during the Beer Bacon Music festival.

http://beerbaconmusic.com/homebrew-competition/
« Last Edit: May 13, 2014, 07:41:28 PM by S. cerevisiae »
Mark V.

Just say "no" to yeast rinsing
https://www.homebrewersassociation.org/forum/index.php?topic=19850.msg252492#msg252492

Friends don't let friends use Star San as their primary sanitizer

"Acid-anionic sanitizers are broad spectrum against bacteria and viruses, but not very effective against yeasts and molds."

Offline ajk

  • Assistant Brewer
  • ***
  • Posts: 239
  • Foam Blowers of Indiana (FBI)
    • View Profile
Re: Entry position within a large flight
« Reply #12 on: May 13, 2014, 07:14:42 PM »
Order bias manifests in many ways. The first beer usually scores in the middle (judges hesitate to go straight for the extremes, figuring their palates just aren't yet acclimated/calibrated). It's also rare for a judge to assign very low or very high scores to more than 3  entries in a row (such clusters are rare but still possible, and the odds of "another 40 point beer" are just the same as the first). And we haven't even touched on palate fatigue!