National Institute for Learning Outcomes Assessment | 1
knowledge accountability connection self-reflection educate action understand communicate listen learn access quality innovation success
ingenuity intellect curiosity challenge create achievement connection self-reflection educate action understand communicate listen learn access
quality innovation success ingenuity intellect curiosity challenge knowledge accountability connection understand communicate listen learn
access quality innovation success ingenuity self-reflection educate action understand intellect knowledge accountability connection self-reflection
educate action understand communicate curiosity challenge create achievement connection self-reflection curiosity challenge create achievement
connection self-reflection knowledge accountability connection self-reflection educate action understand communicate listen learn access quality
innovation success ingenuity intellect curiosity challenge educate innovation success ingenuity intellect curiosity challenge create achievement
knowledge accountability connection self-reflection educate action understand communicate curiosity challenge create achievement connection
self-reflection understand communicate listen learn access quality action educate action understand communicate listen learn action understand
communicate listen learn access quality innovation success ingenuity intellect curiosity challenge knowledge accountability connection access
quality self-reflection curiosity challenge create achievement learn access quality innovation success ingenuity self-reflection educate action
understand intellect knowledge accountability connection self-reflection educate action understand knowledge accountability connection self-
reflection educate action understand communicate listen learn access quality innovation success ingenuity intellect curiosity challenge connection
knowledge accountability connection self-reflection educate action understand communicate listen learn access quality innovation success ingenuity
challenge create achievement connection self-reflection educate action understand connection self-reflection understand communicate listen learn
access quality action create achievement connection self-reflection educate action understand communicate listen learn access quality innovation
success educate action communicate listen learn access quality action educate action understand communicate educate innovation success self-
reflection knowledge accountability communicate listen learn achievement connection self-reflection educate action understand communicate listen
learn access quality innovation success ingenuity intellect access quality innovation success self-reflection curiosity challenge create achievement
connection self-reflection understand educate action understand communicate listen learn action understand communicate listen learn access
quality innovation success ingenuity curiosity challenge create achievement connection self-reflection understand communicate listen learn
Equity and Assessment:
Moving Towards Culturally Responsive Assessment
Erick Montenegro and Natasha A. Jankowski
National Institute for Learning Outcomes Assessment
January 2017
Occasional Paper #29
www.learningoutcomesassessment.org
National Institute for Learning Outcomes Assessment | 2
NILOA Mission
e National Institute for Learning
Outcomes Assessments (NILOA) primary
objective is to discover and disseminate
the ways that academic programs and
institutions can productively use assessment
data internally to inform and strengthen
undergraduate education, and externally to
communicate with policy makers, families,
and other stakeholders.
Abstract....3
Equity and Assessment:
Moving Towards Culturally Responsive Assessment...4
Limiting Learning Demonstration...6
Culturally Responsive Assessment...8
Student Learning Outcomes Statements...11
Assessment Approaches...12
Use of Assessment Results...13
Final oughts...14
References...17
NILOA National Advisory Panel...21
About NILOA...22
Table of Contents
Please cite as:
Montenegro, E., & Jankowski, N. A. (2017, January). Equity and assessment: Moving towards culturally responsive assessment
(Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA).
National Institute for Learning Outcomes Assessment | 3
Abstract
As colleges educate a more diverse and global student population, there is increased need to ensure every student
succeeds regardless of their dierences. is paper explores the relationship between equity and assessment,
addressing the question: how consequential can assessment be to learning when assessment approaches may not be
inclusive of diverse learners? e paper argues that for assessment to meet the goal of improving student learning
and authentically document what students know and can do, a culturally responsive approach to assessment is
needed. In describing what culturally responsive assessment entails, this paper oers a rationale as to why change
is necessary, proposes a way to conceptualize the place of students and culture in assessment, and introduces three
ways to help make assessment culturally responsive.
National Institute for Learning Outcomes Assessment | 4
Equity and Assessment:
Moving Towards Culturally Responsive Assessment
Erick Montenegro and Natasha A. Jankowski
Introduction
College enrollment has become increasingly diverse in terms of students
race, ethnicity, gender identity, socioeconomic status, sexual orientation,
age, ability, etc. is trend is only expected to continue as the United
States moves into a majority-minority nation by the year 2050, and college
enrollments continue to increase. Conducting assessment in a manner that
takes into consideration the various needs of dierent student populations is
a responsibility of higher education. For one, underrepresented students are
more likely to be low-income and rst-generation (Del Rios & Leegwater,
2008; Li & Carroll, 2007; Benitez, 1998), and there are vast dierences
between the needs of lesbian, gay, bisexual, transgender, queer, intersex, asexual
(LGBTQIA) (Check & Ballard, 2014; Mallory, 2009), undocumented (Kim
& Diaz, 2013; Perez, 2010), nontraditional (Macqueen, 2012), and special-
needs students attending higher education institutions (Froese-Germain &
McGahey, 2012). Further, students are increasingly mobile, with transfer
students coming from mostly traditionally underrepresented backgrounds,
attending multiple institutions (Backes & Velez, 2015; Shapiro et al, 2012)
and facing their own challenges in higher education (Tobolowski & Cox,
2012).
Various areas of higher education are aware of the need to accommodate
dierent student populations because “individual dierences are clearly
important to student success” (Strange & Banning, 2015, p. 61). For
example, approaches to teaching, student development, student services,
and campus programs have been analyzed and altered to improve outcomes
for specic student groups (Ladson-Billings, 1995a; Ladson-Billings, 1995b;
Schuh, Jones, Harper, & Associates, 2011; Kezar, 2011; Lara & Wood,
2015; Bowen, Chingos, & McPherson, 2009, Aronson & Laughter, 2016).
Within the eld of campus advising, the issue of microaggressions through
lack of cultural awareness has been raised (Chu, 2016) and the work of the
Association of American Colleges and Universities (AAC&U) in their Equity
Imperative outlines the need to understand who students are, disaggregate
data to look for inequities, and explore policy changes for unintended
impacts on student groups. Conversations in K-12 have addressed the notion
of equity from the standpoint of equity traps (McKenzie & Scheurich, 2004)
within schools and the need to prepare school leaders to not only expose but
address them through courageous conversations about inequities (Singleton,
2012). In a literature review of culturally responsive school leadership,
Khalifa, Gooden, and Davis (2016) argue that culturally responsive leaders
need to continuously support minoritized students through examination of
assumptions about race and culture. Further, they argue that as demographics
continue to shift, so should practice that responds to student needs, nding
that it is “deleterious for students to have their cultural identities rejected in
school and unacknowledged as integral to student learning” (p. 1285).
Conducting assessment in
a manner that takes into
consideration the various
needs of dierent student
populations is a responsibility
of higher education.
National Institute for Learning Outcomes Assessment | 5
Culturally relevant and culturally responsive pedagogies sought to outline
ways in which teachers could address unique learning needs of diverse student
populations. Gloria Ladson-Billings (1995a, 1995b) recommends using
culturally relevant pedagogy as a way to allow students in populations outside
of the majority to maintain their cultural integrity all-the-while succeeding
academically. Culturally relevant pedagogy aims to “produce students who
can achieve academically…demonstrate cultural competence, and develop
students who can both understand and critique the existing social order
(Ladson-Billings, 1995a, p. 474). In culturally responsive pedagogy, teachers
use aspects of students’ cultures in an asset-based approach as opposed to
decit-based to make the course material relevant to them, and increase
their skill acquisition, engagement, and learning outcomes (Ladson-Billings,
1995a). Yet, Geneva Gay (2010) has argued that solely modifying teaching
practices cannot solve the challenges faced by ‘minoritized’ students.
In terms of assessing student learning, the eld has been largely quiet when
it comes to issues of equity. Assessment, if not done with equity in mind,
privileges and validates certain types of learning and evidence of learning
over others, can hinder the validation of multiple means of demonstration,
and can reinforce within students the false notion that they do not belong in
higher education. For equity gaps to be addressed, an entire institution needs
to explore the combination of solutions and supports needed for students
to be successful (Jones, 2015; Methvin & Markham, 2015), of which
assessment is one. However, little of the conversation thus far has focused on
the connection points between demonstration of student learning and issues
of equity. Instead, assessment has remained largely unchanged in regards to
inclusivity, and little urgency has been given to ensuring that students are
provided with just and equitable means to demonstrate their learning. ere
is a dierence between assessing all students in the same way in relation to a
specic outcome of interest and making sure assessments are appropriate and
inclusive of all students. Being attentive to how students may understand
questions, tasks, and assignments dierently, as well as feedback regarding
their learning, is not only benecial to students but to internal improvement
eorts as well. Intentionally choosing appropriate assessment tools or
approaches that oer the greatest chance for various types of students to
demonstrate their learning so that assessment results may benet students
from all backgrounds advances our collective interest in student success.
Without examining issues of equity the students who may stand the most to
gain from assessment eorts may have the least benet since their learning
is not accurately assessed and feedback may not be relevant to impact
learning. If assessments are to be holistic in their goal of improving student
learning, then incorporating a culturally responsive approach to assessment
is a priority. As C. Carney Strange and James Banning (2015) state, student
cultures “can play an important role, for good or otherwise, in introducing
students to and maintaining their engagement in the learning process” (p.
53). It also creates opportunities for students to experience deep learning
(Entwistle, 2001) by honoring students’ prior knowledge and experience.
However, before we present the concept of culturally responsive assessment,
it is useful to unpack an assumption that hinders consideration of diverse
learner needs within assessment—that while learners may take multiple
paths to and through learning, they must demonstrate their knowledge and
skills in the same way.
Assessment, if not done with
equity in mind, privileges
and validates certain types
of learning and evidence
of learning over others,
can hinder the validation
of multiple means of
demonstration, and can
reinforce within students the
false notion that they do not
belong in higher education.
National Institute for Learning Outcomes Assessment | 6
Limiting Learning Demonstration
ere is an assumption at play within the eld of assessment that while
there are multiple ways for students to learn, students need to demonstrate
learning in specic ways for it to count. For instance, in a specic course
dierent approaches may be used to engage students in the material, but
demonstration of a students’ knowledge, skills, and abilities are done
uniformly in the same assignment or approach—so while there may be
multiple approaches and methods used across a program or institution for
assessing student learning, at each instance of demonstration a single approach
is employed. Regardless of the literature on the multiple ways students acquire
knowledge, assessment asks students, at each instance of demonstration,
to show they have the knowledge and skills of interest through the same
means. William Sedlacek (1994) discusses the need for the development of
multicultural assessment standards within the Association for Assessment in
Counseling (AAC). While the focus is upon assessment within the context
of counseling support and services, the interest of addressing the needs of
those with “cultural experiences dierent from…White middle-class men
of European descent, those with less power to control their lives, and those
who experience discrimination in the United States” (p. 550), remains the
same for assessment of or for learning in higher education. Sedlacek (1994)
identies ve fallacies related to culture and assessment, stressing that most
measures were not designed with nontraditional or underserved populations
in mind, that few assessment specialists are trained in developing measures
for use with nontraditional populations, and that larger issues exist that
need to be explored and addressed when promoting diversity, equity, and
inclusivity through assessment. Of note is the fallacy referred to as the three
musketeers, which is the idea that in order to make a measure equally valid
for everyone, everyone completes the same measure—all for one and one
for all—as a means to ensure fairness instead of using dierent measures
for dierent groups. Yet, Sedlacek (1994) argues, “if dierent groups have
dierent experiences and dierent ways of presenting their attributes and
abilities…it is unlikely that we could develop a single measure or test item
that would be equally valid for all” (p. 550); further arguing that there is no
need to employ the same measure when what is desired is equity of results,
not process.
ere are institutions providing students with support and opportunity to
choose from a variety of approaches or even design how they will be assessed
in cooperation with faculty members, presenting students with agency and
choice in the assessment process (Singer-Freeman & Bastone, 2016) and
most institutions use a combination of assessment methods to gauge learning
(Kuh et al., 2014). In a study at the University of East London, students were
allowed to choose how they were assessed, signicantly improving attainment
among learners without an academic background (Grove, 2016). Instead of
completing exams based on coursework, students were given the option to do
a presentation, poster, or debate. Using the alternative assessment techniques
“helped mitigate the fact that many rst-year students had not been in
formal education for some time” allowing them space to demonstrate their
learning, not their exam-taking abilities (Grove, 2016). Further, a similar
approach was used at the University of Dublin where students were able to
make a poster instead of taking an exam. In both instances, students had to
ere is an assumption at play
within the eld of assessment
that while there are multiple
ways for students to learn,
students need to demonstrate
learning in specic ways for it
to count.
National Institute for Learning Outcomes Assessment | 7
e manner in which students
demonstrate learning is
irrelevent when student
demonstration is held to the
same learning outcomes and
evaluative criteria.
demonstrate their learning on the same learning outcomes and evaluative
criteria, but the manner in which they did so was irrelevant. Rubrics were
used such that the evaluation of the work was the same, thus quality ensured,
but the demonstration could be dierent. In Canada, a study was undertaken
within a large, third-year psychology class regarding dierentiated evaluation
to examine student engagement, quality of learning experience, and address
challenges associated with increased student diversity (Gosselin & Gagné,
2014). Dierentiated evaluation allowed students to choose how they would
be evaluated though all students were still required to take mid-term and
nal exams. Students had the option of adding a term project through
preparing a mini-class or participating in a community service learning
program. e study found positive impact on student achievement and on
the learning experience, with students performing below class average seeing
grade improvement when completing a term project. Further, students who
completed the project performed better on the nal exam in comparison to
those that did not, and the option helped to alleviate stress of sitting for an
exam. Qualitative responses from students that selected the project option
indicated that they saw the alternative as an opportunity to demonstrate
their learning through a format over which they felt more control. Gosselin
and Gagné (2014) argued that there are “methods of assessment that can
foster inclusiveness and academic success whilst upholding high standards
for the quality of student learning” yet interestingly “most innovations in
this context have focused on teaching rather than on student learning” (p. 6).
e dierentiated evaluation approach complemented the existing structure
and allowed the relationship between faculty and student to shift to one
of collaboration instead of power, regarding decisions about how students
demonstrate their learning.
e need to fold in culture and student experience into assessment is stressed
in the everyday expertise framework—a perspective of learning that takes
into account how students demonstrate knowledge and skills in their daily
life with the other people around them (Toomey Zimmerman & Bell,
2012). e framework allows for learning to have multiple dimensions
including individual, social, and cultural, requiring a broad consideration
of how people learn within and across learning environments, noting that
learners do not act with equal competency in all settings, even if the content
is the same. Toomey Zimmerman and Bell (2012) argue that the dierence
in performance indicates that learners competent in informal and everyday
settings may falter in more formalized learning settings, requiring alternative
means to demonstrate their knowledge outside of the traditional classroom.
Beyond the many benets from engaging students in co-curricular
experiences (Meents‐DeCaigny & Sanders, 2015; Schuh, Jones, Harper, &
Associates, 2011; Schuh, 2009; Pascarella & Terenzini, 2005; Del Rios &
Leegwater, 2008), co-curricular learning provides a means to address the
issues raised by the everyday expertise framework by widening our lens of
where learning happens to include experiences beyond the classroom. In
addition to conceptions shifting where learning happens, there has been a
rise in competency-based education (CBE) which releases the time structure
in which learning occurs in terms of credit hours. CBE programs stress that
authentic artifacts, or demonstrations of student learning, need to come
from a variety of sources to engage learners with curricula and assessment
National Institute for Learning Outcomes Assessment | 8
that reect not just multiple ways to learn but multiple ways to demonstrate
mastery of a competency (Jobs for the Future, 2016). However, there are
calls for signicant research to determine how best to design assessments
for underprepared learners that also elevate and validate their skills through
alternative measures (Person, Goble, & Bruch, 2014). While learning may
happen anywhere and learners may need dierent lengths of time in their
learning process, there is still the issue of who gets to validate that learning
has occurred, or that demonstrations of learning are of the ‘right type.
While there is movement to more inclusive means of assessment and active
engagement with students as partners in learning, it is clear that the challenges
of various minority groups on campus dier from those of the majority
(Ellis & Chen, 2013; Rankin, Weber, Blumenfeld, & Frazer, 2010; Miller,
Bradbury, & Pedley, 1998), yet higher education still privileges certain types
of learners, certain ways of demonstrating knowledge, and certain learning
spaces by not consistently oering transparency, dierentiated assessments,
or empowering students in their own learning. Students need to develop
and apply their knowledge and skills across multiple contexts in dierent
courses through a range of methods (Newman, Carpenter, Grawe, & Jaret-
McKinstry, 2014, p. 14) with integrative liberal learning requiring students
to engage in “ongoing demonstration to themselves and to others, of the
gains made through curricula, programs, and the educational experience
as a whole” (Ferren & Paris, 2015, p. 5). Yet, the signals education sends
to students about what is validated or counts as demonstration of learning
can be detrimental and reinforce for marginalized students that they do not
belong because their learning ‘doesnt count.’ What is needed is collaboration,
where students, faculty and sta “draw together their life experiences and
aspirations with classroom, co-curricular, and community opportunities”
(Ferren & Paris, 2015, p. 20).
Culturally Responsive Assessment
Dening “culture” and explaining what is meant by culturally responsive
assessment is complicated. e issue is that culture, whether speaking
about it in terms of an organization, a campus, or an individual, has been
historically dicult to dene. Higher education has a tendency to group
student dierences and issues around race under the term ‘diversity,’ which
is often discussed in relation to benets to White students as opposed to
African Americans, Latinx, Asian Americans, and Native Americans who
continue to be underrepresented in higher education (Dowd & Bensimon,
2015). While diversity eorts on college campuses have brought attention
to the vast dierences among students—including gender, religion, sexual
orientation, etc.—the term diversity fails to address issues surrounding race/
ethnicity and does not account for the dierent histories, needs, interests, and
issues aecting distinct groups of students on campus. With this in mind,
one can see why it would be benecial to use culture instead of diversity as
the imperative to refocus assessment into a more inclusive endeavor.
is paper draws from and expands on past denitions of culture to develop
an understanding that culture should be thought of as: (1) the explicit
elements that makes people identiable to a specic group(s) including
behaviors, practices, customs, roles, attitudes, appearance, expressions of
While learning may happen
anywhere and learners
may need dierent lenghts
of time in their learning
process, there is still the issue
of who gets to validate that
learning has occurred, or that
demonstrations of learning are
of the ‘right type.’
National Institute for Learning Outcomes Assessment | 9
identity, language, housing region, heritage, race/ethnicity, rituals, religion;
(2) the implicit elements that combine a group of people which include
their beliefs, values, ethics, gender identity, sexual orientation, common
experiences (e.g. military veterans and foster children), social identity;
and (3) cognitive elements or the ways that the lived experiences of a
group of people aect their acquisition of knowledge, behavior, cognition,
communication, expression of knowledge, perceptions of self and others,
work ethic, collaboration, and so on. e culturally relevant component
involves assuring that the assessment process—beginning with student
learning outcome statements and ending with improvements in student
learning—is mindful of student dierences and employs assessment methods
appropriate for dierent student groups. Underlying the culturally relevant
component is the focus on students—the importance of keeping students at
the center, which requires their involvement at every step in the assessment
process and builds upon their lived experience.
In addition, it is important to understand the concept of intersectionality and
its eect on culture. Traditionally, intersectionality is thought of in racial/
ethnic identity intersecting with class, gender, and sexual orientation to shape
how people of color experience oppression (Solorzano & Yosso, 2002; Huber,
2010; Cho, 1997). However, for purposes of this paper, intersectionality is
the way that aspects of a persons identity cannot be fully separated from one
another, play a central role in peoples’ experiences and making meaning of
those experiences. is is related to Susan Jones and Marylu McEwens (2000)
multiple dimensions of identity which treats a students identity as dynamic
and changing depending on the relative contextual salience of other elements
of ones identity (e.g. race/ethnicity, sexual orientation, gender, religion);
with no single aspect of ones identity understood singularly, but only in
relation to the other dimensions. For example, a White male that identies
as a member of the lesbian, gay, bisexual, transgender, queer, intersex, and
asexual (LGBTQIA) community and practices Judaism is shaped by the
intersectionality of these four elements. A Latina that is a single-mother from
a low-socioeconomic background is shaped by the intersectionality of these
elements. An undocumented English as a second language, rst-generation
student will experience college, acquire knowledge, and demonstrate
knowledge dierently than an international English as a second language
rst-generation student. e culture—the explicit, implicit, and cognitive
elements—of the people in these examples shape their college experiences,
and while one aspect of their culture may manifest itself more than another
in specic contexts, they all aect the outcomes being assessed.
inking of culture in the way that it is dened here can serve as a reference
point for what to consider when engaging in assessment and developing/
choosing/implementing assessment tools and methods. Culture is by no
means simple, and it is by no means easily denable. It is dependent on
the context in which culture is discussed. Culture permeates the individual,
group, entire institutions, countries, and continents; and at the same time
the individuals that comprise cultural groups are multicultural through
intersectionality. Perhaps Lang (1997) stated it best when he said “attempts
at dening culture in a denite way are futile” (p. 389). However, developing
an inclusive understanding of culture, and making it explicit that culture is
much more than race/ethnicity and aects students’ lives on multiple levels,
e culturally relevant
component involves assuring
that the assessment process
beginning with student
learning outcome statements
and ending with improvements
in student learning
is
mindful of student dierences
and employs assessment
methods appropriate for
dierent student groups.
National Institute for Learning Outcomes Assessment | 10
including learning and how they demonstrate learning, will help ensure
culturally responsive assessment and increase the eectiveness and impact of
learning outcomes assessment eorts.
In addition to the term of culture, it is important to note the use of responsive
to indicate “an action-based, urgent need to create contexts and curriculum
that responds to the social, political, cultural, and educational needs of
students; it is armative and seeks to identify and institutionalize practices
that arm indigenous and authentic cultural practices of students” (Khalifa,
Gooden, & Davis, 2016, p. 1278). Students who experience validation from
faculty and integrate academically and socially are more likely to persist and
be successful (Karp, Hughes, & O’Gara, 2011). Assessment approaches
and processes can help reinforce a sense of belonging or add to students
belief that they do not belong because their learning or experiences are not
deemed as valid or important. Susan Headden and Sarah McKay (2015)
stress this point, arguing that student motivation is connected to students
beliefs that they are able to do the work and have a sense of control over
the work. For rst-generation college goers and African American students
stereotypes about academic performance can turn into self-fullling
prophecies…even feedback on papers can reinforce or foster learning…that
students are cared about and respected as learners” (Headden & McKay,
2015, p. 15). An environment focused on students’ unique learning interests
and needs enables students to incorporate prior and everyday experiences in
meaning construction (Land, Hannan, & Oliver, 2012). Involvement with
culture is also important as Cathleen Spinelli (2008) argues that there are a
disproportionate number of students with cultural and linguistic dierences
that are misidentied as learning disabled. As a result, students are classied
incorrectly, not academically challenged, and do not receive appropriate
services. Spinelli (2008) further argues that when looking specically at the
case of English language learners, informal assessment provided a solution
to the need of assessment of learning, but in a manner adaptable to language
and cultural diversity, individual learning styles, and personal challenge
while also informing instruction.
Culturally responsive assessment is thus thought of as assessment that is
mindful of the student populations the institution serves, using language
that is appropriate for all students when developing learning outcomes,
acknowledging students’ dierences in the planning phases of an assessment
eort, developing and/or using assessment tools that are appropriate for
dierent students, and being intentional in using assessment results to improve
learning for all students. Culturally responsive assessment involves being
student-focused, which does not simply mean being mindful of students.
Instead, being student-focused calls for student involvement throughout the
entire assessment process including the development of learning outcome
statements, assessment tool selection/development process, data collection
and interpretation, and use of results. An essential aspect of maintaining
focus on students is truly understanding the student population at the
institution and/or level at which the assessment is being conducted. Once we
understand who our students are we can begin to tailor assessment processes
and materials to have the greatest impact for their learning. Institutions with
high enrollment of traditionally underrepresented students have already
begun tailoring their learning outcomes assessment approaches based on
Assessment approaches and
processes can help reinforce
a sense of belonging or add
to students’ belief that they
do not belong because their
learning or experiences are not
deemed as valid or important.
National Institute for Learning Outcomes Assessment | 11
the student populations that they serve (Montenegro & Jankowski, 2015;
Nunley, Bers, & Manning, 2011; Baker, Jankowski, Provezis, & Kinzie,
2012). Minority-Serving Institutions (MSIs) have been found to increase
self-esteem, engagement, critical thinking skills, leadership skills and
opportunities, and help the identity formation processes for traditionally
underrepresented students; which helps increase students’ persistence
through college (Conrad & Gasman, 2015; Del Rios & Leegwater, 2008;
Conrad et al, 2013). e work at these institutions can serve as guideposts
for the development of culturally responsive assessment practices.
Student Learning Outcomes Statements
Learning outcomes assessment as a process begins with developing learning
outcome statements that clearly state what students should know and be
able to demonstrate upon completion of a course, academic program,
college, making use of student services, etc. To develop student learning
outcomes statements using a cultural lens necessarily involves students in
the development process. Poorly constructed learning outcomes make it
dicult for students to demonstrate their learning for a myriad of reasons
(e.g. not understanding what is expected of them, not understanding how
the course/program is expected to contribute to their learning). In addition,
it is students that will directly benet from the feedback they receive as a
result of assessment. Clarity of outcomes and curricular structure matters
in general education (Gaston, 2015), assignment design (Winkelmes et al,
2016), co-construction of knowledge for deep learning (Juvova et al, 2015;
von Glasersfeld, 2005), and new course design models like competency-based
education (Jobs for the Future, 2016). Further, in the National Research
Council report, How People Learn, (2000) principles for designing learner-
centered environments emphasized the importance of individual social and
cultural contexts in learning. Such perspectives require dierent approaches
to curricular design, teaching, and assessment, and squarely place learner
preconceptions and experiences as an integral part of the learning process.
Assessment is a eld of alignment, and this also originates from learning
outcomes statements. Hutchings (2016) denes alignment as “the linking of
intended student learning outcomes with the processes and practices needed
to foster those outcomes” (p. 5). Similarly to how academic programs, student
services, and other institutional programs aim to align with and promote
the mission of the college or university, learning outcomes statements of
departments, programs, and courses should align with those of the institution.
Outcome statements need to be culturally responsive because they align with
assignments, evaluative criteria, and institutional and departmental goals. If
outcome statements are not culturally responsive, then there are implications
for various levels of the institution; not just for students. Learning outcome
statements which are written to inform educational policy and practice,
and are clear about expected prociencies make it possible for programs,
departments, institutions, and students to meet their goals (National
Institute for Learning Outcomes Assessment, 2016). e language and
operative verbs in learning outcome statements serve as a guide for students
to understand departmental/program expectations, as well as understand
how their educational experiences prepare them for their careers and lives
after college (National Institute for Learning Outcomes Assessment, 2016).
However, if learning outcomes statements are not written with attention to
To develop student learning
outcomes statements using
a cultural lens necessarily
involves students in the
development process.
National Institute for Learning Outcomes Assessment | 12
cultural relevancy, then it becomes dicult to accurately infer the learning
gains of dierent groups of students.
Cli Adelman (2015) speaks about the importance of being intentional and
mindful of language when writing learning outcomes statements as this can
lead to creating assignments that allow for genuine judgement of student
achievement. However, taking this a step further and being mindful of how
the language of learning outcomes statements might be appropriate for/
inclusive of certain student groups but not others can lead to more holistic
assessments. Flawed assessment designs may unintentionally skew scores
for certain student populations and ensuring this does not happen begins
with the writing of culturally responsive learning outcomes statements that
consider students, their dierent ways of learning, and the diverse ways
they demonstrate learning. One way to make statements more culturally
responsive is to explicitly dene terms and use scenarios or examples that
are relatable to various student groups. A sample tool that incorporates these
elements of being intentional and explicit in writing learning outcomes and
clearly dening learning is the Degree Qualications Prole (DQP) and
Tuning process. e DQP was developed by Lumina Foundation (2014),
and by coupling Tuning processes to it serves as a way to clearly outline
what students know and should be able to do after attaining a degree (Ewell,
2013).
Traditionally, learning outcomes statements are written by and for faculty
and administrators. As a result, faculty and administrators dene the
intended learning outcomes and what it looks like to demonstrate those
outcomes. If, instead, we write learning outcome statements for and with
students, then we increase the chances of students understanding what is
expected of them. In addition, instead of students’ knowledge conforming
to how we traditionally measure it, students would now have agency in how
to demonstrate learning. is would result in learning outcomes, as well as
the assessment process, becoming a more inclusive endeavor.
Assessment Approaches
ere is a need for assessments that allow students to demonstrate their
learning in various ways while also being transparent about the learning
that is taking place, help students reect on their learning experiences, and
allow students to actively participate in the learning and assessment process.
Course-level assessments such as culturally responsive rubrics, portfolios, and
capstone projects can lead to more valid, appropriate, holistic, and formative
assessment where results are more indicative of what all students can do
or lead to more targeted improvements in teaching and learning. Rubrics,
which help instructors gauge student learning, skills development, and
acquisition of learning outcomes, provide criteria by which to assess whether
or not the learning outcome was demonstrated. Rubrics, when they undergo
a culturally conscious development process and are shared with students,
can be a way to accurately assess learning for all students while allowing
variation in how the learning is demonstrated. While rubrics are at times
created by individual faculty members to t the context of specic courses or
programs, the Association of American Colleges and Universities (AAC&U)
Valid Assessment of Learning in Undergraduate Education (VALUE)
Rubrics serve as examples of rubrics for institutions to employ. In addition,
ere is a need for assessments
that allow students to
demonstrate their learning
in various ways while also
being transparent about the
learning that is taking place,
help students reect on their
learning experiences, and allow
students to actively participate
in the learning and assessment
process.
National Institute for Learning Outcomes Assessment | 13
capstones—which can be entire courses or student projects—can also be
an avenue through which the learning of students can be better gauged by
allowing students to design their own projects in partnership with faculty.
ey provide tangible, visible, self-explanatory evidence of exactly what
students have and havent learned” (Suskie, 2004, p. 95). As comprehensive,
culminating experiences, capstones allow students to demonstrate a wide-
range of skills and knowledge that oftentimes draws from previous work,
experiences, and learning that occurred throughout their coursework.
Finally, portfolios oer a similar freedom for students to demonstrate
their learning and provide a more holistic representation of what students
know and can do. e use of portfolios provides students the option to
select demonstrations and add commentary and reection, furthering
their agency in the process and selection of assessment evidence. Portfolios
represent student work over time and demonstrate various forms of learning
(Kuh et al, 2015; Banta, Grin, Flateby, & Kahn, 2009) which may not
be easily captured by other forms of assessment. Portfolios are “authentic
assessment that draws on the work students do in regular course activities
and assignments” and “reconnect assessment to the ongoing work of
teaching and learning and to the work of faculty, raising the prospects for
productive use” (Kuh et al, 2015, p. 36). Portfolios provide the opportunity
to get students invested into the course beyond grade attainment, and help
to deepen students’ educational experiences through allowing them to
make connections between conceptual issues, theoretical knowledge, and
real world experiences (Singer-Freeman & Bastone, 2016). Additionally,
portfolios can be made available online. Eportfolios can be easily accessed
by potential employers, as well as other institutions, which provides students
in the job market or looking to transfer a means to easily demonstrate their
knowledge and skills. Kuh et al (2015) mention a few of the advantages
that portfolios have for assessment, including advancing student success,
catalyzing change, and making learning more visible for students. ese
impacts can be furthered by applying a cultural lens when assessing student
portfolios. By being mindful of how culture aects students’ meaning-
making processes, cognition, and demonstrations of learning, we can better
understand and appreciate the learning gains that students make. In fact,
at the program-level, assessment approaches such as rubrics and portfolios
are used more often than surveys and other approaches (Ewell, Paulson, &
Kinzie, 2011).
Use of Assessment Results
Implementing formative assessment methods means very little if assessment
data are not used to inform learning at various levels of the institution or if it
has no meaning to students to improve their own learning. e rst step in
creating change is analyzing the data by student populations. Disaggregating
the data is instrumental in informing changes to higher education. While
the data may tell a positive story about overall learning, disaggregation may
yield the observation that rst-generation students are struggling in a course,
female students are making use of resources aimed at supporting their
education at disproportionate rates, or Latinx students are not reaching the
same institutional learning outcomes as other racial/ethnic groups. In either
hypothetical case, disaggregating the data allows researchers, administrators,
and practitioners to see themes that they otherwise would have missed and
By being mindful of how culture
aects students’ meaning-
making processes, cognition, and
demonstrations of learning, we can
better understand and appreciate
the learning gains that students
make.
National Institute for Learning Outcomes Assessment | 14
could inform changes that would positively impact students’ education.
In addition, disaggregation of assessment data should not only be used to
uncover surface-level ndings such as Latinx students excel at “ABC” while
rst-generation students struggle with “XYZ.” As Aydin Bal & Audrey Trainor
(2016) state, “researchers must also include an examination of processes
(e.g. the racialization of disability… and the institutional acts of exclusion
based on ability dierences) and institutions…that reproduce, regardless of
intentionality, disparities” (p. 330-331). is means that disaggregating data
should explore why the condition exists in the rst place, and then be used
to inform/develop possible solutions.
In using assessment results, it is also useful to be mindful of our own
assumptions. Similar to how a researchers bias cannot be fully removed from
his/her/zers study and can either harm or enhance his/her/zers research, so
can the biases of faculty and sta aect assessment eorts and use of results.
It is unrealistic and counterproductive for assessment professionals to think
they are approaching their work from an impartial stance or to assume that
the students being assessed also operate from an impartial stance. Failing to
recognize how culture and our own experiences aect the assessment process
can limit the impact of assessment. In discussing the need for faculty to be
attentive to the changes in the institutions student population, Goldrick-
Rab and Cook (2011) warn against comparing all students against the
researchers subconscious idea of what students do/should do. Failing to be
aware of our own biases or subconscious ideas and failing to disaggregate
assessment data in a culturally responsive manner may cause the assessment
endeavor to implement outdated norms as a means of comparison, which
can misclassify certain students as underachievers, confusing, or outliers; and
can also lead to the mistake of failing to connect the data to the actual lived
experiences and realities of the students the institution serves (Goldrick-Rab
& Cook, 2011). is can also lead to unintentionally reinforcing negative
assumptions about certain student groups. Treating dierent racial/ethnic
groups under an aggregate umbrella, as has been the recent case with the
term “underrepresented minorities,” minimizes the voice of various groups
and ignores their salient dierences (Dowd & Bensimon, 2015) which
impact their needs, experiences, learning, and demonstration of that
learning. Finally, it would be worthwhile to connect assessment results to
other campus assessment strategies. While certain data collection eorts
on campus may seem unrelated, occurrences on campus seldom happen in
isolation. Connecting dierent assessment eorts and resulting data sets can
better inform issues related to student attrition, success, campus climate,
pedagogy, and others (Hurtado & Halualani, 2014).
Final oughts
Students’ college experiences are inseparable from other daily experiences such
as those encountered at work, microaggressions endured on campus, family
life, and employment. More often than not, students’ college experiences are
aected by students’ own culture and cultural dierences with faculty, sta,
and peers. It has long been known that students of dierent backgrounds
experience college dierently and respond dierently to similar situations,
stimuli, experiences, requests, questions, etc. So, if we also know that students
from dierent cultures who have similar education backgrounds respond
Students’ college experiences
are inseparable from other
daily experiences such as
those encountered at work,
microaggressions endured
on campus, family life, and
employment.
National Institute for Learning Outcomes Assessment | 15
and perform signicantly dierent, why would we design assessments,
execute them, and then make changes based on assessment results without
considering the cultural relevance of the assessment eort and analyze how
the assessment might aect all students/benet certain population(s) and
hinder others? Why would we not include students in the assessment process
to improve our approaches?
e focus of assessment as a means to improve student learning is an agreed
upon purpose of the work. As Kuh, Ikenberry, Jankowski, Cain, Ewell,
Hutchings, and Kinzie (2015) state, “gathering information about collegiate
outcomes has a practical goal: using it to improve both student learning
and institutional performance” (p. 51); and “harnessing evidence of student
learning, making it consequential in the improvement of student success
and strengthened institutional performance is what matters” (p. 4). Yet,
how consequential can assessment truly be when assessment approaches are
minimally inclusive of our current student populations? Using assessment
tools and approaches that work for the majority of students but are less
mindful of students identifying with groups outside of the majority
population places a signicant portion of students at a disadvantage, leads
to a decrease in the quality of education, creates a disconnect between
students and the institution, and contributes to achievement gaps (Slee,
2010; Sullivan, 2010; Qualls, 1998). Assessment that overlooks issues of
diversity and equity contributes to inequalities in outcomes (Bal & Trainor,
2016). e same can be said for assessment approaches that do not take into
account students’ culture.
Students have dierent ways to demonstrate their knowledge and we need
to use assessment metrics that appropriately elicit demonstrations of what
students know. One example of the diverse ways students can demonstrate
learning comes from Nick Sousanis’ (2015) published dissertation exploring
how people construct knowledge. Instead of writing a typical manuscript,
Sousanis demonstrated his knowledge in a graphic novel format. At times,
the illustrations said more than the words on the page, and both pictures
and words united to tell a powerful academic story. is way of presenting
scholarly work, while unconventional in academia, is still a powerful
demonstration of learning. Sousanis’ chosen method of demonstrating his
knowledge on a specic topic is not wrong, it is just dierent. We undo
boundaries through the awareness that “it is our [own] vision, and not what
we are viewing, that is limited” (Sousanis, 2015, p. 42). How assessment is
often operationalized or experienced by students has not moved to a position
where it continuously regards students’ diverse methods of demonstrating
knowledge as appropriate. Instead, dierent can often be marked as wrong.
If assessment is about demonstrating learning, then we need to allow students
the space to show their knowledge. Students are highly varied in customs,
identity, and understanding, and it is all shaped by culture which aects
learning; and thus, should aect how we measure learning. If assessment
is done for improvement and with the goal of using the results to benet
student learning, then having outcome assessments that appropriately tell
the stories of what students know and can do is of imperative importance.
Our assessments approaches—how we assess and the process of assessment
itself—should align with the students we have, empowering them with
narratives to share and document their learning journey.
Assessment that overlooks
issues of diversity and equity
contributes to inequalities in
outcomes. e same can be
said for assessment approaches
that do not take into account
students’ culture.
National Institute for Learning Outcomes Assessment | 16
What is needed is not to help
learners conform to the ways
of higher education, thus
reinforcing inequities and
expectations based on ideologies
the students may ascribe to,
but to empower students for
success through intentional
eorts to address inequality
within our structures, create
clear transparent pathways,
and ensure that credits and
credentials are awarded by
demonstration of learning, in
whatever form that may take.
In summary, assessing students in the same way without paying attention
to their dierences works if students are all privy to the same educational
opportunities, are all at the same academic standing, have similar experiences
on campus, work through knowledge in similar fashion, understand questions
in similar ways, and benet from the same programs, pedagogical styles,
support services, and interactions. However, we know this is not the case.
“While absolute growth in the college-going population helped shape todays
college milieu, compositional changes also impacted the college experience,
turning it into a set of highly diverse experiences that led to very dierent
outcomes” (Goldrick-Rab & Cook, 2011, p. 257). Sara Goldrick-Rab &
Marjorie Cook (2011) continue to say that “as the student body grew more
diverse, so did the kinds of colleges and universities serving them; at the same
time, opportunities both expanded in number and became more distinct
and disparate, reecting and preserving key aspects of the inequality of
opportunity and outcomes” (p. 255). Continuing to assess students as if there
are no dierences will only work to preserve key aspects of inequality and
widen the achievement gap. It is no secret that there is a disparity between the
academic attainment of students based on race/ethnicity (Bowen, Chingos,
& McPherson, 2009; Condron, Tope, Steidl, & Freeman, 2013; Santiago,
Galdeano, & Taylor, 2015; Carnevale & Strohl, 2013) and social class (Kezar,
2011; Bowen, Chingos, & McPherson, 2009). We need to ask ourselves, is it
that we want students to demonstrate their knowledge and skills or attainment
of learning outcomes in a particular way, or that they demonstrate their
learning? What is needed is not to help learners conform to the ways of higher
education, thus reinforcing inequities and expectations based on ideologies
the students may not ascribe to, but to empower students for success through
intentional eorts to address inequality within our structures, create clear
transparent pathways, and ensure that credits and credentials are awarded by
demonstration of learning, in whatever form that may take.
National Institute for Learning Outcomes Assessment | 17
References
Adelman, C. (2015). To imagine a verb: e language and syntax of learning outcomes statements. Urbana, IL: University of
Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Aronson, B., & Laughter, J. (2016). e theory and practice of culturally relevant education: A synthesis of research across
content areas. Review of Education Research, 86(1), 163-206.
Backes, B., & Velez, E. D. (2015). Who transfers and where do they go? Community college students in Florida. Washington,
DC: National Center for Analysis of Longitudinal Data in Education Research (CALDER).
Baker, G. R., Jankowski, N., Provezis, S. & Kinzie, J. (2012). Using assessment results: Promising practices of institutions
that do it well. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment (NILOA).
Bal, A., & Trainor, A. A. (2016). Culturally responsive experimental intervention studies: e development of a rubric for
paradigm expansion. Review of Education Research, 86(2), 319-359.
Banta, T. W., Grin, M., Flateby, T. L., & Kahn, S. (2009, December). Three promising alternatives for assessing college
students’ knowledge and skills. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment.
Benitez, M. (1998). Hispanic-serving institutions: Challenges and opportunities. New Directions for Higher Education,
102(2), 57-68.
Bowen, W. G., Chingos, M. M. & McPherson, M. S. (2009). Crossing the nish line: Completing college at Americas public
universities. Princeton, NJ: Princeton University Press.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school.
Washington, DC: National Academy Press.
Carnevale, A. P., & Strohl, J. (2013). Separate and unequal: How higher education reinforces the intergenerational reproduction
of White racial privilege. Washington, DC; Georgetown University, Georgetown Public Policy Institute, Center on
Education and the Workforce.
Check, E., & Ballard, K. (2014). Navigating emotional, intellectual, and physical violence directed toward LGBTQQ
students and educators. Art Education, 67(3), 6-11.
Cho, S. K. (1997). Converging stereotypes in racialized sexual harassment: Where the model minority meets Suzie Wong.
In A. K. Wing, (Ed). Critical Race Feminism: A Reader (2nd. Ed). New York, NY: New York University Press.
Chu, J. Y. (2016, October 21). e need to be culturally aware when advising students (essay). Inside Higher Ed. Retrieved
from https://www.insidehighered.com/views/2016/10/21/need-be-culturally-aware-when-advising-students-essay.
Condron, D. J., Tope, D., Steidl, C. R., & Freeman, K. J. (2013). Racial segregation and the Black/White achievement
gap, 1992 to 2009. e Sociological Quarterly, 54, 130-157.
Conrad, C., Gasman, M. Lundberg, T., Nguyen, T., Commodore, F., & A. C. Samayoa (2013). Using educational data
to increase learning, retention, and degree attainment at Minority Serving Institutions. Philadelphia, PA: Penn Center for
Minority Serving Institutions. Retrieved from http://www.gse.upenn.edu/pdf/cmsi/using_educational_data.pdf.
Conrad, C., & Gasman, M. (2015). Educating a diverse nation: Lessons from Minority Serving Institutions. Cambridge,
MA: Harvard University Press.
Del Rios, M., & Leegwater, L. (2008). Increasing student success at minority serving institutions: Findings from the BEAMS
project. Washington, DC: Institute for Higher Education Policy (IHEP).
Dowd, A. G., & Bensimon, E. M. (2015). Engaging the “race question”: Accountability and equity in U.S. higher education.
New York, NY: Teachers College at Columbia University.
Ellis, L. M., & Chen, E. C. (2013). Negotiating identity development among undocumented immigrant college students:
A grounded theory study. Journal of Counseling Psychology, 60(2), 251-264.
Entwistle, N. (2001). Styles of learning and approaches to studying in higher education. Kybernetes, 30(5/6), 593-603.
Ewell, P. T. (2013). e Lumina Degree Qualications Prole: Implications for assessment. Urbana, IL: University of Illinois
and Indiana University, National Institute for Learning Outcomes Assessment.
Ewell, P. T., Paulson, K., & Kinzie, J. (2011). Down and in: Assessment practices at the program level. Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
National Institute for Learning Outcomes Assessment | 18
Ferren, A. S., & Paris, D. C. (2015). Faculty leadership for integrative liberal learning. Washington, DC: Association of
American Colleges and Universities.
Froese-Germain, B., & McGahey, B. (2012). Optimizing conditions of teacher’s professional practice to support students with
special educational needs. Ottawa, ON, Canada: Canadian Teacher’s Federation. Retrieved from http://les.eric.ed.gov/
fulltext/ED532565.pdf
Gaston, P. L. (2015). General education transformed: How we can, why we must. Washington, DC: Association of American
Colleges and Universities.
Gay, G. (2011). Culturally responsive teaching: eory, research, and practice (2nd ed.). New York, NY: Teacher College
Press.
Goldrick-Rab, S., & Cook, M. A. E. (2011). College students in changing contexts. In P. G. Altbach, P. J. Gumport, &
R. O. Berdahl, (Eds) American higher education in the twenty-rst century: Social, political, and economic challenges (3rd
Ed.) pp. 254-278. Baltimore, MD: e John Hopkins University Press.
Gosselin, J., & Gagné, A. (2014). Dierentiated evaluation: An inclusive evaluation strategy aimed at promoting student
engagement and student learning in undergraduate classrooms. Toronto, Canada: Higher Education Quality Council of
Ontario.
Grove, J. (2016, September, 29). Mature students ‘do better with non-written assessment’. Times Higher Education.
Retrieved from https://www.timeshighereducation.com/news/mature-students-do-better-non-written-assessment.
Headden, S., & McKay, S. (2015, July). Motivation matters: How new research can help teachers boost student engagement.
Stanford, CA: Carnegie Foundation for the Advancement of Teaching.
Huber, L. P. (2010). Using Latina/o critical race theory (LatCrit) and racist nativism to explore intersectionality in the
educational experiences of undocumented Chicana college students. Educational Foundations, 24, 77-96.
Hurtado, S., & Halualani, R. (2014). Diversity assessment, accountability, and action: Going beyond the numbers.
Diversity & Democracy, 17(4), 8-11.
Hutchings, P. (2016). Aligning educational outcomes and practices. Urbana, IL: University of Illinois and Indiana University,
National Institute for Learning Outcomes Assessment (NILOA).
Jobs for the Future. (2016, February). Expanding competency-based education for all learners. Washington, DC: Author.
Jones, S. (2015). e game changers: Strategies to boost college completion and close attainment gaps. Change: e
Magazine of Higher Learning, 47(2), 24-29.
Jones, S. R., & McEwen, M. K. (2000). A conceptual model of multiple dimensions of identity. Journal of College Student
Development, 41, 405-414.
Juvova, A., Chudy, S., Neumeister, P., Plischke, J., & Kvinotova, J. (2015). Reection of constructivist theories in current
educational practice. Universal Journal of Educational Research, 3(5), 345-349.
Karp, M. M., Hughes, K. L., & O’Gara, L. (2010). An exploration of Tintos integration framework for community
college students. Journal of College Student Retention: Research, eory & Practice, 12(1), 69-86.
Kezar, A. (2011). Recognizing and serving low-income students in higher education: An examination of institutional policies,
practices, and culture. New York, NY: Routledge.
Khalifa, M. A., Gooden, M. A., & Davis, J. E. (2016). Culturally responsive school leadership: A synthesis of the literature.
Review of Educational Research, 86(4), 1272-1311.
Kim, E., & Diaz, J. (2013). Undocumented students and higher education. ASHE Higher Education Report, 38(6), 77-90.
Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. T., Hutchings, P., & Kinzie, J. (2015). Using evidence
of student learning to improve higher education. San Francisco, CA: Jossey-Bass.
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing what students know and can do: e current
state of student learning outcomes assessment in US colleges and universities. Urbana, IL: University of Illinois and Indiana
University, National Institute for Learning Outcomes Assessment (NILOA).
Ladson-Billings, G. (1995a). Toward a theory of culturally relevant pedagogy. American Educational Research Journal,
32(3), 465-491.
Ladson-Billings, G. (1995b). But that’s just good teaching! e case for culturally relevant pedagogy. eory Into Practice,
34(3), 159-165.
National Institute for Learning Outcomes Assessment | 19
Land, S. M., Hannan, M. J., & Oliver, K. (2012). Student-centered learning environments: Foundations, assumptions and
design. In D. Jonassen, & S. Land (Eds.). eoretical foundations of learning environments (2nd ed.) pp. 3-26. New York,
NY: Routledge.
Lang, A. (1997). inking rich as well as simple: Boeschs cultural psychology in semiotic perspective. Culture & Psychology,
3, 383–394.
Lara, R., & Wood, J. L. (2015). Impact of non-cognitive behaviors on student engagement for Latino male students attending
a designated two-year Hispanic-serving institution. In J. P. Mendez, F. A. Bonner II, J. Mendez-Negrete, & R. T. Palmer
(Eds) Hispanic-serving institutions in American higher education: eir origin, and present and future challenges. Sterling, VA:
Stylus Publishing, LLC.
Li, X., & Carroll, C. D. (2007). Characteristics of minority-serving institutions and minority undergraduates enrolled in these
institutions. Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department
of Education.
Lumina Foundation (2014). e Degree Qualications Prole. Indianapolis, IN: Author.
Macqueen, E. (2012). Studying the experiences of non-traditional students in teacher education. Sydney, Australia: Joint AARE
APERA International Conference. Retrieved from http://les.eric.ed.gov/fulltext/ED544492.pdf.
Mallory, A. (2009). Access and equity for all students: Meeting the needs of LGBT students. California Postsecondary Education
Commission. Retrieved from http://les.eric.ed.gov/fulltext/ED507508.pdf.
McKenzie, K. B., & Scheurich, J. J. (2004). Equity traps: A useful construct for preparing principals to lead schools that are
successful with racially diverse students. Educational Administration Quarterly, 40, 601-632.
Meents-DeCaigny, E., & Sanders, L. (2015). Framing research and assessment of co-curricular learning. New Directions for
Institutional Research, 164, 15-25.
Methvin, P., & Markham, P. N. (2015). Turning the page: Addressing the challenge of remediation. Change: e Magazine
of Higher Learning, 47(4), 50-56.
Miller, R., Bradbury, J., & Pedley, J. (1998). Academic performance of rst and second language students: Disadvantages and
underpreparedness. South African Journal of Science, 95, 30-34.
Montenegro, E., & Jankowski, N. A. (2015). Focused on what matters: Assessment of student learning outcomes at minority-
serving institutions. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment (NILOA).
National Institute for Learning Outcomes Assessment. (2016, May). Higher education quality: Why documenting learning
matters. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.
Newman, L. E., Carpenter, S., Grawe, N., & Jaret-McKinstry, S. (2014). Creating a culture conducive to integrative learning.
Peer Review, 16(4), 14-15.
Nunley, C., Bers, T, & Manning, T. (2011, July). Learning outcomes assessment in community colleges. Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Pascarella, E. T., & Terenzini, P. T. (2005). How college aects students. San Francisco, CA: Jossey-Bass.
Perez, W. (2010). Higher education access for undocumented students: Recommendations for counseling professionals.
Journal of College Admission, 206, 32-35.
Person, A. E., Goble, L., & Bruch, J. (2014, April). Developing competency-based program models in three community colleges.
Oakland, CA: Mathematica Policy Research.
Qualls, A. L. (1998). Culturally responsive assessment: Development strategies and validity issues. e Journal of Negro
Education, 67(3), 296-301.
Rankin, S., Weber, G., Blumenfeld, W., & Frazer, S. (2010). 2010 state of higher education for lesbian, gay, bisexual, and
transgender people. Charlotte, NC; Campus Pride.
Santiago, D. A., Galdeano, E. C., & Taylor, M. (2015). e condition of Latinos in education: 2015 factbook. Washington,
DC; Excelencia in Education.
Schuh, J. H. (2009). Assessment methods for student aairs. San Francisco, CA: Jossey-Bass.
National Institute for Learning Outcomes Assessment | 20
Schuh, J. H., Jones, S. R., Harper, S. R., & Associates. (2011). Student services: A handbook for the profession (5th Ed). San
Francisco, CA: Jossey-Bass.
Sedlacek, W. E. (1994). Issues in advancing diversity through assessment. Journal of Counseling & Development, 72, 549-553.
Shapiro, D., Dundar, A., Chen, J., Ziskin, M., Park, E., Torres, V., & Chiang, Y. (2012). Completing college: A national view
of student attainment rates. Herndon, VA: National Student Clearinghouse Research Center.
Singer-Freeman, K. & Bastone, L. (2016, July). Pedagogical choices make large classes feel small. (Occasional Paper No. 27).
Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Singleton, G. E. (2012). More courageous conversations about race. ousand Oaks, CA: Corwin.
Slee, J. (2010). A systemic approach to culturally responsive assessment practices and evaluation. Higher Education Quarterly,
64(3), 246-260.
Solorzano, D. G., & Yosso, T. J. (2002). Critical race methodology: Counter-storytelling as an analytical framework for
education research. Qualitative Inquiry, 8(1), 23-44.
Sousanis, N. (2015). Unattening. Cambridge, MA: Harvard University Press.
Spinelli, C. G. (2008). Addressing the issue of cultural and linguistic diversity and assessment: Informal evaluation measures
for English language learners. Reading & Writing Quarterly, 24(1), 101-118.
Strange, C. C., & Banning, J. H. (2015). Designing for learning: Creating campus environments for student success (2nd Ed.).
San Francisco, CA: Jossey-Bass.
Sullivan, A. L. (2010). Preventing disproportionality: A framework for culturally responsive assessment. Communique, 39(3),
24-26.
Suskie, L. (2004). Assessing student learning: A common sense guide. Boston, MA: Anker Publishing Company.
Tobolowski, B. F., & Cox, B. E. (2012). Rationalizing neglect: An institutional response to transfer students. Journal of
Higher Education, 83(3), 389-410.
Toomey Zimmerman, H., & Bell, P. (2012). Everyday expertise: Learning within and across formal and informal settings.
In D, Jonassen & S. Land (Eds.). eoretical foundations of learning environments (2nd ed.) pp. 224-241. New York, NY:
Routledge.
von Glasersfeld, E. (2005). Instroduction: Aspects of constructivism. In C. T. Fosnot (Ed.), Constructivism: eory, Perspectives,
and practice (2nd ed.), pp. 3-7. New York, NY: Teachers College Press.
Winkelmes, M., Bernacki, M., Butler, J., Zochowski, M., Golanics, J., & Harriss Weavil, K. (2016). A teaching intervention
that increases underserved college students’ success. Peer Review, 18(1-2), 31-36.
National Institute for Learning Outcomes Assessment | 21
NILOA National Advisory Panel
James Anderson
Interim Dean
Edward William and Jane Marr Gutsgell
Professor
University of Illinois Urbana-Champaign
Wallace Boston
CEO
American Public University System
Molly Corbett Broad
President
American Council on Education
Judith Eaton
President
Council for Higher Education Accreditation
Richard Ekman
President
Council of Independent Colleges
Keston Fulcher
Director of the Center for Assessment and
Research Studies
Associate Professor, Graduate Psychology
James Madison University
Paul Gaston, III
Trustees Professor
Kent State University
Mildred Garcia
President
California State University,
Fullerton
Susan Johnston
Executive Vice President
Association of Governing Boards
Norman Jones
Professor
Utah State University
Peggy Maki
Higher Education Consultant
George Mehay
Vice President for
Academic Leadership and Change
American Association of State Colleges and
Universities
Lynn Pasquerella
President
Association of American Colleges &
Universities
George Pernsteiner
President
State Higher Education Executive Ocers
Association
Mary Ellen Petrisko
President
WASC Senior College and University
Commission
Kent Phillippe
Associate Vice President, Research and
Student Success
American Association of Community Colleges
Robert Sheets
Research Professor
George Washington Institute of Public Policy
Ralph Wol
Founder and President
e Quality ASsurance Commons for Higher
and Postsecondary Education
Ex-Ocio Members
Peter Ewell
President Emeritus
National Center for Higher Education
Management Systems
Stanley Ikenberry
President Emeritus and Regent Professor
University of Illinois
Natasha Jankowski
Director, NILOA
Research Assistant Professor
George Kuh
Founding Director, National Institute for
Learning Outcomes Assessment
Adjunct Research Professor, University of
Illinois Urbana-Champaign
Chancellor’s Professor of Higher Education
Emeritus, Indiana University
Paul Lingenfelter
President Emeritus
State Higher Education Executive Ocers
NILOA Mission
NILOAs primary objective is to
discover and disseminate ways that
academic programs and institutions
can productively use assessment data
internally to inform and strengthen
undergraduate education, and exter-
nally to communicate with policy
makers, families and other stake-
holders.
NILOA Occasional Paper Series
NILOA Occasional Papers
are commissioned to examine
contemporary issues that will inform
the academic community of the
current state-of-the art of assessing
learning outcomes in American higher
education. e authors are asked to
write for a general audience in order
to provide comprehensive, accurate
information about how institutions and
other organizations can become more
procient at assessing and reporting
student learning outcomes for the
purposes of improving student learning
and responsibly fullling expectations
for transparency and accountability
to policy makers and other external
audiences.
Comments and questions about this
paper should be sent to:
National Institute for Learning Outcomes Assessment | 22
About NILOA
e National Institute for Learning Outcomes Assessment (NILOA) was estab-
lished in December 2008.
NILOA is co-located at the University of Illinois and Indiana
University.
e NILOA website contains free assessment resources and can be found at http://
www.learningoutcomesassessment.org/.
e NILOA research team has scanned institutional websites, surveyed chief
academic ocers, and commissioned a series of occasional papers.
NILOAs Founding Director, George Kuh, founded the National Survey for
Student Engagement (NSSE).
e other co-principal investigator for NILOA, Stanley Ikenberry, was president
of the University of Illinois from 1979 to 1995 and of the American Council of
Education from 1996 to 2001.
NILOA Sta
NATIONAL INSTITUTE FOR LEARNING OUTCOMES ASSESSMENT
Stanley Ikenberry, Co-Principal Investigator
George Kuh, Founding Director, Senior Scholar, and Co-Principal Investigator
Natasha Jankowski, Director
Gianina Baker, Assistant Director
Filip Przybysz, Communications Coordinator
Katie Schultz, Project Manager
Peter Ewell, Senior Scholar
Pat Hutchings, Senior Scholar
Jillian Kinzie, Senior Scholar
Paul Lingenfelter, Senior Scholar
David Marshall, Senior Scholar
Erick Montenegro, Research Analyst
Verna F. Orr, Research Analyst
Anthony B. Sullers, Jr., Research Analyst
Emily Teitelbaum, Research Analyst
Terry Vaughan III, Research Analyst
Karie Brown-Tess, Research Analyst
NILOA Sponsors
Lumina Foundation for Education
University of Illinois, College of Education
Produced by Creative Services | Public Aairs at the University of Illinois for NILOA. 10.032
National Institute for Learning Outcomes Assessment | 23
National Institute for Learning Outcomes Assessment
For more information, please contact:
National Institute for Learning Outcomes Assessment (NILOA)
University of Illinois at Urbana-Champaign
360 Education Building
Champaign, IL 61820
learningoutcomesassessment.org
Phone: 217.244.2155
knowledge accountability connection self-reflection educate action understand communicate listen learn access quality innovation success ingenuity
intellect curiosity challenge create achievement connection self-reflection educate action understand communicate listen learn access quality innovation
success ingenuity intellect curiosity challenge knowledge accountability connection understand communicate listen learn access quality innovation success
ingenuity self-reflection educate action understand intellect knowledge accountability connection self-reflection educate action understand communicate
curiosity challenge create achievement connection self-reflection curiosity challenge create achievement connection self-reflection knowledge accountability
connection self-reflection educate action understand communicate listen learn access quality innovation success ingenuity intellect curiosity challenge educate
innovation success ingenuity intellect curiosity challenge create achievement knowledge accountability connection self-reflection educate action understand
communicate curiosity challenge create achievement connection self-reflection understand communicate listen learn access quality action educate action
understand communicate listen learn action understand communicate listen learn access quality innovation success ingenuity intellect curiosity challenge
knowledge accountability connection access quality self-reflection curiosity challenge create achievement learn access quality innovation success ingenuity
self-reflection educate action understand intellect knowledge accountability connection self-reflection educate action understand knowledge accountability
connection self-reflection educate action understand communicate listen learn access quality innovation success ingenuity intellect curiosity challenge
connection knowledge accountability connection self-reflection educate action understand communicate listen learn access quality innovation success
ingenuity challenge create achievement connection self-reflection educate action understand connection self-reflection understand communicate listen learn
access quality action create achievement connection self-reflection educate action understand communicate listen learn access quality innovation success
educate action communicate listen learn access quality action educate action understand communicate educate innovation success self-reflection knowledge
accountability communicate listen learn achievement connection self-reflection educate action understand communicate listen learn access quality innovation
success ingenuity intellect access quality innovation success self-reflection curiosity challenge create achievement connection self-reflection understand
educate action understand communicate listen learn action understand communicate listen learn access quality innovation success ingenuity curiosity
challenge create achievement connection self-reflection understand communicate listen learn access quality action create achievement connection self-
reflection educate action understand communicate listen learn access quality innovation success educate action communicate listen learn access quality
action educate action understand create achievement connection self-reflection understand communicate listen learn access quality action create achievement
connection self-reflection educate action understand communicate listen communicate educate innovation success self-reflection knowledge accountability
connection self-reflection educate action understand communicate listen learn access quality innovation ingenuity intellect connection self-reflection
understand communicate listen learn access quality action create achievement connection self-reflection educate action understand communicate listen learn