Pennsylvania State University
AC-6: Sustainability Literacy Assessment

The survey asked 22 questions, a sampling of which are below. The entire survey is attached above.
To begin, students were given the following direction: "For each of the following statements, indicate your best judgment regarding whether the statement is true or false using the response options provided."

A total of 15 Knowledge questions were multiple choice, single answer questions. These included:

Sustainability is one-dimensional and focuses only on protecting the environment.
o I’m certain this statement is true. (1)
o I think this statement is true but I’m uncertain. (2)
o I think this statement is false but I’m uncertain. (3)
o I’m certain this statement is false. (4)

The common/standard definition of environmental justice is to protect habitat necessary to reduce the likelihood of adding species to the International Union for Conservation of Nature (IUCN)’s Red List.
o I’m certain this statement is true. (1)
o I think this statement is true but I’m uncertain. (2)
o I think this statement is false but I’m uncertain. (3)
o I’m certain this statement is false. (4)

Residents of Pennsylvania will experience increased frequency and intensity of flooding events as a result of climate change.
o I’m certain this statement is true. (1)
o I think this statement is true but I’m uncertain. (2)
o I think this statement is false but I’m uncertain. (3)
o I’m certain this statement is false. (4)

Reducing and properly treating waste increases your ecological footprint.
o I’m certain this statement is true. (1)
o I think this statement is true but I’m uncertain. (2)
o I think this statement is false but I’m uncertain. (3)
o I’m certain this statement is false. (4)

The Six Americas Super Short Survey (SASSY) – a nationally recognized assessment of climate change beliefs that allowed us to compare student responses to national averages – was used to assess student beliefs and to compare them to knowledge responses.

Questions were asked with a rating scale of Strongly Agree, Agree, Neither agree nor disagree, Disagree, and Strongly Disagree. These included statements such as: I believe that my PSU education will prepare me to take on the challenge of climate change; I believe that Penn State should improve educational outreach to encourage greater campus participation in waste reduction and recycling efforts; and, The courses I have taken at PSU have transformed the way I think about sustainability.

In the spring semester of 2022 the student government University Park Undergraduate Association (UPUA) distributed the Sulitest to students. The survey is an online, multiple-choice-question format. Sulitest is aligned with the SDGs framework and was developed by entities including UN agencies, academic institutions, and civil society. According to Sulitest, it was developed "to expand sustainable knowledge, skills, and mindset that motivates individuals to become deeply committed to build a sustainable future and to make informed and effective decisions." However, distribution was disrupted due to cost constraints and a change in UPUA leadership.

In response to deficiencies related to what the Sulitest tracks as compared to university and sustainability learning objectives, a survey was developed by three Penn State Sustainability staff, a student serving as the Department of Environmental Sustainability chair of the UPUA and two faculty members (Dr. Janet Swim, Psychology, and Dr. Julie Stanton, Business).

This literacy survey was a pilot effort to determine if students were, in fact, learning material corresponding to university Learning Outcomes.

A goal of this pilot survey was to test out the format and use it to support system wide distribution and to provide preliminary recommendation to faculty, staff, and administration. These include:
1) Develop an annual, university-wide Sustainability Literacy Assessment
2) Engage students in sustainability in classrooms
3) Support an integrated approach to sustainability education in the curriculum (sustainable learning outcomes, sustainability designation, university requirement, etc.).

The survey instrument was divided into four sections that assessed sustainability competencies through socio-demographic, cognitive, affective, and behavioral questions (Waltner et al., 2019) and attempted to capture a snapshot of student knowledge and beliefs about sustainability through an expanded version of True/False knowledge questions about fifteen different areas of sustainability, the four Six Americas Super Short Survey (SASSY!) questions, agreement statements pulled from the Penn State Waste Stream Task Force recommendations and Penn State Carbon Emissions Reduction Task Force recommendations, the Awaken State survey, and Penn State-approved demographic questions.

It was designed to address the following questions:
- What sustainability knowledge do Penn State students currently have?
- What beliefs about sustainability do Penn State students currently have?
- Is there an association between knowledge and beliefs, and if so, what is it?

The same version was distributed to students in the College of Liberal Arts, and their results are included.

Penn State Waste Stream Task Force recommendations
https://wastestream.psu.edu/

Penn State Carbon Emissions Reduction Task Force recommendations
https://www.dept.psu.edu/ur/newsdocuments/PSU-CERTF-Report_5-02-22.pdf

Waltner, E. M., Rieß, W., & Mischo, C. (2019). Development and validation of an instrument for measuring student sustainability competencies. Sustainability, 11(6), 1717.

The Sulitest survey was first distributed in the spring of 2022, with data collection continuing through the fall of 2022. The UPUA did not strive for a representative sample but instead used multiple communication channels to recruit students to participate. Due to leadership change in the UPUA, the test was abandoned and no data is available.

As the Sulitest outreach began, a rising 4th year student with the incoming UPUA administration brought concerns about the assessment and the distribution process to Sustainability Institute staff. Several staff began working with the student and two faculty members to develop a pilot literacy test that would help determine if students were, in fact, learning material corresponding to university Learning Outcomes.

Once the survey was complete, students were recruited by tabling in the HUB-Robeson Center (the student union), promotional flyers in over 15 high-traffic buildings, social media posts, a promotional article by Onward State, and instructor referral. Over the course of three months, the survey received 1,315 total responses. Demographics of respondents were compared to those of the overall student body.

Throughout survey development, distribution, and data compilation, the Director of Student Affairs Research and Assessment was kept informed as they are supportive of and will be planning to distribute the survey across all campuses at some point between FA23-FA25.

While this will not allow comparison between the two time periods for each demographic group, it will enable Penn State to track and assess literacy related to the learning outcomes developed for Penn State students.

The attached file presents an analysis of performance on the literacy portion of the pilot survey. The Liberal Arts survey results can be accessed at

On a categorical basis, students performed best on waste-related questions (49% certain and correct), worst on climate change-related questions (18%), and in between on ecosystem services (39%), environmental justice (34%), and systems-thinking (31%) questions.

On average, participants correctly answered 72% of the fifteen questions, but they were only certain and correct on 32% of the questions. When interpreting results, the level of certainty indicates how confident participants were in their answer choices. Across items, participants were certain on 39% of the questions, independent of whether they got it correct or incorrect.

People who knew more about sustainability also were more supportive of sustainability education. The more likely they are to agree with the statements in the beliefs section.

This survey differs from the one done in 2018 (based on survey developed by Zwickle and Jones, 2018) which was planned to be repeated in Fall 2021. Staffing changes at the Sustainability Institute and other survey needs in the wake of the pandemic intervened and thus did not result in a follow up survey that would allow comparison between the two time periods for each demographic group.

Plans are in place to refine the survey piloted by the UPUA that will guide academic, co-curricular, and student affairs responses vs one which assesses content knowledge that is not related to goals of any of those programs, as was the case with the 2018 survey.

Zwickle,A. and K. Jones. 2018. Sustainability Knowledge and Attitudes—Assessing Latent Constructs. In Springer International Publishing AG 2018W. Leal Filho et al. (eds.), Handbook of Sustainability and Social Science Research, World Sustainability Series, https://doi.org/10.1007/978-3-319-67122-2_25