The Review Process

Thank you for agreeing to be a reviewer for NOAA Grant Applications. NOAA relies on the expertise and perspectives of grant reviewers to ensure a fair and thorough evaluation of grant applications. Reviewers play a critical role in helping NOAA identify the most promising projects that align with its mission to understand and protect the common types of bias, how bias can influence our decisions, and how to mitigate the impact of bias in the review process. 

Everyone has bias. Bias is a mental shortcut that helps humans process information quickly and efficiently. Sometimes, though, our biases cause us to process information through a narrow lens. Understanding the types of unconscious biases can help you be more aware of how they affect your decisions. As the various types of biases are described, consider pausing the video and think about how each of the biases could lead to incomplete or unfair assessments and decisions in the merit review process.

We will now go over the most common types of bias. First is diagnosis bias. Diagnosis bias occurs when you judge someone based on your first impression and refuse to change your opinion in the face of new evidence. For example, a reviewer may see a poorly written executive summary and assume the entire application is substandard, ignoring evidence of a strong project plan and team credentials in other sections. Next, we have value attribution bias. This happens when you assign something a value and then allow that perception to cloud all subsequent interactions or new information. For example, a reviewer notices an applicant from a well-known organization with a strong reputation and assumes the proposed project will also be high quality.

Overlooking flaws in the methodology or budget. Confirmation bias occurs when you only pay attention to information that confirms your preexisting opinions, discounting anything that would disprove them. For example, if a reviewer rates a proposal highly, they might overlook weaknesses in the proposal. Commitment confirmation bias happens when you grow so attached to an idea that you make bad choices in order to appear consistent. For example, after advocating for a proposal in an initial review meeting, a reviewer continues to defend it during discussions despite new concerns raised by peers about feasibility or alignment with NOAA's priorities.

Anchoring bias takes place when your decision is overly influenced by the first piece of information that you receive about something. For example, a reviewer may feel that an applicant had a weak personal statement and hear that all other reviewers felt this way, too. Therefore, they conclude that they must be right. Attentional bias happens when you base a decision on only one thing, to the exclusion of all other relevant evidence. For example, a reviewer fixates on the budget being slightly over the recommended amount, ignoring the overall strengths of the project. Primacy bias takes place when you only focus on what happened first, ignoring all other aspects of a situation. For example, a reviewer strongly favors an application they read first because it sets the tone for their expectations, giving less attention or consideration to subsequent applications.

Recency bias happens when you judge something based only on what happened most recently. Instead of considering the whole situation, for example, a reviewer judges an application they read last as the strongest -- simply because it is fresh in their mind -- even if earlier applications had more merit. And finally, there's the halo effect. This refers to a tendency to apply an overall impression of a person to a specific trait, and then extend that impression to all aspects of the individual. For example, a reviewer notices that an applicant has an eye catching resume and therefore assumes their whole application is strong.

It's important to remember these and keep them in mind whenever reviewing a grant or a proposal. Again, everyone has bias. Understanding the types of unconscious biases can help you to be more aware of how they affect your decisions. There are many ways to help individuals and groups minimize bias to ensure fair assessments and decisions.

How you respond and acknowledge your bias will help you become the best reviewer you can be. A few tips for you as you enter the review process. Consider reviewing applications side by side with any scoring criteria or rubrics provided. Read applications completely and in the same order every time, and consider the order in which you are reading materials. Take notes. Make them as specific as possible about the information in the application. Many review processes require comments about the scores provided. Use your notes to support the score given.

Additionally, applicants may receive these comments, which can be used as a learning and improvement tool. Read every application as if it is your first one. This will reduce the tendency to compare applications against each other rather than against the scoring criteria. Give yourself plenty of time. Try and read only a few applications a session to keep yourself fresh and give each application your full focus. Just these few tips can make a huge difference in mitigating bias.

As an individual, you can do things to manage your bias as you review each application or proposal. Ask yourself critical questions such as: does the proposal remind you of yourself or others? Have you assigned value to this proposal or application based on details other than its merit? Have you given this and all applications equal attention and time? As you participate in interviews or review panel discussions? You should consider questions like, are you updating your opinion of the applicant based on new information received? Are you updating your assessment of the applicant based on merit only? Are there other details influencing your assessment? If after reviewing this short introductory video, you want to learn more, we have provided additional resources in the description below.

Also, make sure to review any resources your program has provided to you. Thank you for your commitment to the merit review process and to reducing the influence of bias.