Summary Findings

Mary Wright, Assistant Director, Evaluation, & Assistant Research Scientist, CRLT
Jim Barber, Assistant for Programs & Research, CRLT

This report presents summary results of an evaluation of LSA’s Quantitative Reasoning requirement. According to The LSA Bulletin, “The goal of the Quantitative Reasoning requirement is to ensure that every graduate of the College achieves a certain level of proficiency in using and analyzing quantitative information.” Students may fulfill the requirement by passing one course designated for QR credit (QR1) or two courses designed for half credit (QR/2). Because a wide margin (82%) of students fulfill the requirement through the former option, this evaluation focuses specifically on the impact of QR1 courses.

Index

A. Evaluation Methods
B. Key questions and findings:

1. Do LSA students perceive that QR1 courses contribute to gains in their quantitative reasoning skills?
2. Do these reported learning outcomes differ by subgroups of students, e.g., by gender, race/ethnicity, incoming math proficiency, and class rank? Do they differ by discipline?
3. Are these gains comparable to those reported by other LSA students who have never taken a QR1 class, or are there statistically significant differences between students who take QR1 courses and those who do not (overall and by specific skill)?
4. Controlling for student demographics, what is the impact of taking a QR1 course on students’ reported learning gains? Given that QR/2 courses also can be used to fulfill the requirement, does having taken at least one QR1 or QR/2 class have a greater impact on students’ reported gains?
5. Do students who took a QR1 course perceive their course met the goals of the LSA Quantitative Reasoning requirement?
6. Can students who took a QR1 course give an example of how they are better able to use and analyze quantitative information?
7. Is a requirement necessary? In other words, would students likely take a course that emphasizes the development of quantitative reasoning even if they were not required to do so?
8. What instructional activities do students find useful for the development of quantitative reasoning skills?

C. Key implications of these findings

A. Evaluation Methods

The evaluation was based on a survey of LSA first- and second-year students about quantitative reasoning gains they reported making from their QR1 or Fall Term courses. Most of the survey was derived from a University of Wisconsin assessment study of its QR requirement, which found that student self-reports about learning gains on 14 different quantitative reasoning skills were as valuable a source of data as pre- and post-tests of authentic QR-related problems (Halaby, 2005). The instrument was developed by a study team that included UW’s Director of Testing & Evaluation and other quantitative researchers. In addition to the 14 UW gains items, the U-M survey asked students if they felt that the course met LSA’s goals for the QR requirement, if they could give an example of an application of the course, and what instructional methods helped them learn.  The survey was pilot tested on U-M students in six QR1 courses, as well as a control group, in Spring/Summer 2009. When asked to give their feedback about the survey, students reacted very favorably. The 14 QR gain items had a Cronbach’s alpha of 0.95, indicating the high consistency of the items in measuring the same underlying concept. 

In Fall 2009, all first- and second-year LSA students who were taking a QR1 course for the first time were surveyed at the end of the term.  The survey was distributed to 2,624 QR takers and 1,419 responded, a 54% return rate. Additionally, a parallel survey was distributed to a control group of 900 first- and second-year students who had never taken a QR1 course.  384 of these students replied to the survey, a 43% response rate.  Both groups of students were asked, “Compared to when I started [QR course or the Fall Term], I am now BETTER able to:” in regard to 14 different skills, which were aggregated into an overall mean QR score (1=Strongly Disagree to 5=Strongly Agree, N/A).

Back to Index

B. Key questions and findings were:

1) Do LSA students perceive that QR1 courses contribute to gains in their quantitative reasoning skills?

For first-time QR takers, the overall mean QR score was 3.52, indicating that these students tend to agree that they gained key quantitative reasoning skills in their Fall Term QR1 courses (1=strongly disagree, 5=strongly agree that gains were made in quantitative reasoning skills).. Strongest gains were reported for “solve problems using arithmetic, algebra or statistics” (M=3.83) and “use quantitative information to solve problems” (M=3.79). Weakest reported gains for all QR takers, hovering around a neutral response, were “using statistics to evaluate factual claims” (M=3.22), “understand randomness, uncertainty and risk” (M=3.18) and “understand the difference between correlation and causation” (M=3.16).

Back to Index

2) Do these reported learning outcomes differ by subgroups of students, e.g., by gender, race/ethnicity, incoming math proficiency, and class rank? Do they differ by discipline?

Looking at first-time QR takers only, there was no statistically significant difference in mean QR score between men and women, nor between underrepresented minorities and others.Dividing up QR takers’ ACT math scores into quartiles, there was no statistically significant difference between the four groups of students. However, the general trend is that students with the least math proficiency going into a QR1 course perceive that they gain the most. The mean QR score was significantly higher for sophomores, compared to first-year students, a 0.19 gap

In Fall 2009, eight departments offered QR1 courses. Looking by department, there was a range of 3.42 to 4.39 on the mean QR score (but an important caveat is that there was a small sample size in some departments due to the sampling procedures). Reported gains for the eight departments offering QR1 courses are generally similar, indicating that the variety of courses that fulfill the QR1 requirement function well to achieve relatively equivalent outcomes. 

Back to Index

3) Are these gains comparable to those reported by other LSA students who have never taken a QR1 class, or are there statistically significant differences between students who take QR1 courses and those who do not (overall and by specific skill)?

Compared with the control group (first- and second-year LSA students who had never taken a QR1 course), QR1 takers had a slightly higher (+.10), but significantly different, overall mean QR score (p<.05). On individual items, QR1 students scored significantly higher than the control group on seven of the 14 measures of QR skills:

• Solve problems using arithmetic, algebra or statistics (+.95 difference between the two groups, p<0.001)
• Use quantitative information to solve problems (+.55, p<0.001)
• Understand charts and graphs showing quantitative information (+.45 difference, p<0.001)
• Understand rates and percentages (+.45 difference, p<0.001)
• Express ideas using quantitative information (+.34 difference, p<0.001)
• Use statistics to evaluate factual claims (+.14 difference, p<0.05)
• Solve problems using formal logic (+.14 difference, p<0.05)

On three items, non-QR students rated their gains higher:

• Recognize when arguments use evidence well (-.54 difference, p<0.001)
• Recognize logically sound arguments (-.43 difference, p<0.001)
• Know when it is valid to infer that one thing causes another (-.31 difference, p<0.001)

Interestingly, in the UW study, the non-QR group scored higher than the QR sample on all three of these items as well (Halaby, 2005).  These seem to be the only items that could just as easily apply to general analytical skills, as opposed to quantitative reasoning specifically.

There was no statistically significant difference between the two groups on the remaining four items:

• Use quantitative information to evaluate an argument
• Understand the difference between correlation and causation
• Understand randomness, uncertainty and risk
• Understand how data can be used to test a hypothesis

Back to Index

4) Controlling for student demographics, what is the impact of taking a QR1 course on students’ reported learning gains? Given that QR/2 courses also can be used to fulfill the requirement, does having taken at least one QR1 or QR/2 class have a greater impact on students’ reported gains?

Regression models indicate that enrollment in a QR1 course is a significant predictor of the overall mean QR score, even when controlling for gender, race/ethnicity, class rank, incoming math proficiency, and participation in a QR/2 course. We also can conclude that both QR1 and QR/2 courses have a positive impact on students’ perceived learning gains, but QR1 is a slightly stronger predictor of self-reported development of quantitative reasoning. Sophomore status is also significant, showing a small positive effect on students’ reported learning gains as well.

Back to Index

5) Do students who took a QR1 course perceive their course met the goals of the LSA Quantitative Reasoning requirement?

QR students were presented with information about the goals of the QR requirement and asked if they felt that the course met these objectives. Students tended to agree (3.9) that their course fulfilled LSA’s goals (1=strongly disagree, 5=strongly agree).

Back to Index

6) Can students who took a QR1 course give an example of how they are better able to use and analyze quantitative information?

Back to Index

When asked to describe an example of the way in which they left their QR course being better able to use and analyze quantitative information, fewer than half the respondents (42%) offered a response.

Although many themes were course-dependent, two themes that cut across the QR courses were students’ development of evaluative skills (e.g., evaluation of ways that the media presents statistics) and applications to everyday problem-solving tasks.

7) Is a requirement necessary? In other words, would students likely take a course that emphasizes the development of quantitative reasoning even if they were not required to do so?

Students who took a QR course were asked, “If LSA did not have a Quantitative Reasoning requirement, would you still plan to take a course at U-M that met these goals?” Two-thirds of students indicated that they would, although for social science/humanities, the proportion dropped to less than half (48%). Given that LSA’s goal is to develop quantitative reasoning in every graduate of the College, this response suggests that a requirement is necessary.

Back to Index

8) What instructional activities do students find useful for the development of quantitative reasoning skills?
When asked about instructional activities that are helpful to QR learning, only about a third (35%) of students selected “solving real-world problems,” which may help explain why the majority of students did not provide an example of the way in which they could apply their QR course concepts/skills.

By department, there were several distinctive trends in what students reported was useful for QR learning:

• Three quarters (76%) of Communication Studies students indicated that receiving feedback on exams or quizzes was helpful to their QR learning, compared to 49% overall.
• Reviewing notes was selected by 81% of Statistics students, but only 63% of the larger group.
• Nearly all (92%) of the Philosophy respondents chose “listening to lectures,” compared to 62% of the overall sample.
• The percentage of Math students who selected “participating in group work in class” was 45%, a higher rate than all other departments (except Economics, at 50%, which had very few students in the sample).
• The small sample size makes it difficult to draw conclusions, but it is interesting that in Economics, Sociology, and Political Science, all respondents unanimously pointed to “GSI explanations in section,” which may speak to the influential GSI role in a social science QR course or solid GSI training in these units.
• Completing problem sets was labeled a helpful teaching approach by 72% of Physics respondents, compared to 64% overall.

Back to Index

C. Key implications of these findings include:

• The QR requirement appears to achieve LSA’s goals for student learning, and it should be maintained. Additionally, reported gains for the eight departments offering QR1 courses are generally similar, indicating that the variety of courses that fulfill the QR1 requirement function well to achieve relatively equivalent outcomes.
• Academic advisers should be aware that sophomores report greater learning gains from their QR1 courses compared to first-year students. If students have a choice about when to take a QR1 course, they should be advised to take it later in their educational careers.
• It is clear that some departments are doing particularly well at maximizing the learning that students report drawing from specific instructional methods, and it would be helpful to disseminate these strategies as best practices for developing students’ quantitative reasoning skills. This information sharing may be especially helpful for pedagogies that help students “solve real-world problems,” because many students do not perceive that their instructors are doing so (or doing so effectively) in their classes.
• Future studies should examine the equivalency of different vehicles for fulfilling the QR requirement (one QR1 course vs. two QR/2 classes).

Reference: Halaby, C. N. (2005, October). Two assessment studies of the general education quantitative reasoning “A” requirement at the University of Wisconsin-Madison. Report to the General Education Assessment Council Subcommittee on the Effectiveness of the Quantitative Reasoning “A” Requirement. Madison, Wisconsin: University of Wisconsin-Madison. Available: https://gened.wisc.edu/wp-content/uploads/sites/991/2019/05/qr-study04-05.pdf

back to top