Evaluating Proof Blocks Problems as Exam Questions
Seth Poulsen Mahesh Viswanathan
University of Illinois at Urbana-Champaign University of Illinois at Urbana-Champaign
USA USA
Geoffrey L. Herman Matthew West
University of Illinois at Urbana-Champaign University of Illinois at Urbana-Champaign
USA USA
ABSTRACT [12, 16, 30]. A panel of 21 experts using a Delphi process agreed that
Proof Blocks is a novel software tool which enables students to 6 of the 11 most difficult topics in a typical discrete mathematics
write mathematical proofs by dragging and dropping prewritten course are related to proofs and logic [9].
lines into the correct order, rather than writing a proof completely There are many aspects of writing mathematical proofs that are
from scratch. We used Proof Blocks problems as exam questions for difficult. Many students fail to produce the basic building blocks
a discrete mathematics course with hundreds of students, allowing that proofs have, such as properly declaring variables or referenc-
us to collect thousands of student responses to Proof Blocks prob- ing theorems [23]. Students get stuck working through the details
lems. Using this data, we provide statistical evidence that Proof of algebraic manipulations. They have a tendency to commit cer-
Blocks are easier than written proofs, which are typically very tain logical fallacies such as confusing a proposisiton with its con-
difficult. We also show that Proof Blocks problems provide about verse [23, 27]. Studies have shown that even when students have all
as much information about student knowledge as written proofs. the prerequesite content knowledge to write a mathematical proof,
Survey results show that students believe that the Proof Blocks user they still struggle to construct one [32]. Thus, there is a gap that
interface is easy to use, and that the questions accurately represent needs to be filled between students having the content knowledge
their ability to write proofs. to write a proof and the aptitude to actually construct one.
Vygotsky’s theory of psychological development posits that be-
CCS CONCEPTS tween the tasks which a person can and cannot do, there is a so-
called zone of proximal development: a set of tasks which a person
· Mathematics of computing → Discrete mathematics; · So-
cannot perform unaided, but which they can perform when given
cial and professional topics → Computing education; · Ap-
help and support, called scaffolding [31, 36]. Computer science
plied computing → Computer-assisted instruction.
instructors and researchers have used various approaches to scaf-
folding students learning to write code for the first time. Block based
KEYWORDS
programming languages such as Scratch and Blockly [8, 15] scaffold
discrete mathematics, CS education, automatic grading, proofs students by providing them with building blocks from which to
ACM Reference Format: assemble their programs and guarding against the struggles of syn-
Seth Poulsen, Mahesh Viswanathan, Geoffrey L. Herman, and Matthew West. tax errors. Research has shown that using block based languages
2021. Evaluating Proof Blocks Problems as Exam Questions. In Proceedings can accelerate the student learning process when first learning to
of the 17th ACM Conference on International Computing Education Research program [34]. Parson’s problems are a kind of homework and exam
(ICER 2021), August 16ś19, 2021, Virtual Event, USA. ACM, New York, NY,
question where students are asked to assemble prewritten lines of
USA, 12 pages. https://doi.org/10.1145/3446871.3469741
code into a correct program [17]. Researchers have shown Parson’s
problems to be useful both as test questions [4] and as a learning
1 INTRODUCTION tool for helping to accelerate the learning process for beginners
Understanding and writing mathematical proofs is one of the crit- learning to write code [7].
ical yet difficult skills that students must learn as a part of the Following from the success of Parson’s problem and similar ap-
discrete mathematics curriculum. Proofs and proof techniques are proaches to teach programming, we propose Proof Blocks. Proof
included by the ACM curricular guidelines as a core knowledge Blocks allows students to construct mathematical proofs by drag-
area that should be understood by any student obtaining a degree in ging and dropping prewritten proof lines into the correct order,
computer engineering, computer science, or software engineering rather than having to write the entire proof from scratch. Figure 1
shows an example of a Proof Blocks problem. Proof Blocks provides
a scaffolded environment, enabling students to construct mathe-
This work is licensed under a Creative Commons matical proofs without needing to worry about coming up with
Attribution-NonCommercial-ShareAlike International 4.0 License. all of the details on their own. A Proof Blocks problem may also
ICER 2021, August 16ś19, 2021, Virtual Event, USA
contain distractor lines which are not a part of any correct solution.
© 2021 Copyright held by the owner/author(s). The design of the Proof Blocks grader [20] is flexible in allowing
ACM ISBN 978-1-4503-8326-4/21/08. any correct arrangement of the lines of the proof. This is enabled by
https://doi.org/10.1145/3446871.3469741
157
, ICER 2021, August 16ś19, 2021, Virtual Event, USA Poulsen et al.
the instructor specifying which lines of the proof depend on which 2.1 Parson’s Problems
other lines (the full dependence graph of the lines of the proof in The use of scrambled code problems was first documented by Par-
Figure 1 can be seen in Figure 2). Students who fail to construct sons [17]. They have since been studied for their desirable proper-
a correct proof on their first try can then receive automated feed- ties both in assessment and learning [4, 5, 7]. The desirable proper-
back from the computer, as shown in Figure 3, before being given ties of Parson’s problems were a major inspiration for the creation
additional attempts at the discretion of the instructor. of Proof Blocks.
Proof Blocks problems are also very promising for saving time for Denny et al. [4] showed that Parson’s problems are easier to
both students and course staff. Many computer science departments grade than free-form code writing questions, and yet still offer rich
are experiencing a huge increase in enrollments. This increase in information about student knowledge. We will show the same to
enrollments means course staff lose more time to grading, making it be true with Proof Blocks problems in relation to free-form proof
more difficult for them to spend the time they need helping students writing questions. Ericson et al. [7] showed that students learning
individually. Proof Blocks helps to alleviate this strain by providing to write code using Parson’s problems learn at an accelerated rate
a way to test some of students’ proof skills in a way that can be in the early stages of learning compared to students being taught
automated, saving grading time and allowing course staff more to fix code or write code from scratch.
time for other activities that help students such as office hours and
review sessions. 2.2 Research on Teaching and Learning Proofs
The ability to receive automated feedback is also a boon to stu-
dents. Due to staff time constraints, students in a discrete mathemat- There are many threads of research in seeking to illuminate stu-
ics course may not be able to receive feedback on the correctness of dents’ understandings and misunderstandings about proofs [24, 26,
proofs they write until long after they have completed them. Proof 27]. One thread establishes that, as they learn, students go through
Blocks also helps with this, as it allows students to receive feedback different phases in the complexity of ways they are able to think
instantly, just as they receive instant feedback from the compiler about solving proof problems [33]. Another study demonstrated
and from automated testing suites as they write code. that even when students had all of the knowledge required to write
In using a new kind of test question with our students, we wanted a proof and were able to apply that knowledge in other types of
to ensure that we were testing students on the correct set of skills questions, they were still unable to write a proof [32], thus high-
and that we were providing them with a fair and equitable learning lighting the need to scaffold students through the proof-writing
experience. process.
In this paper, we seek to answer the following three research On the other hand, there is little research on concrete educational
questions: interventions for improving the proof learning process [11, 27]. In-
deed, a recent review of the literature on teaching and learning
proofs concluded: łmore intervention-oriented studies in the area
RQ1: What statistical information about student knowl- of proof are sorely neededž [27]. Hodds et al. [11] showed that train-
edge do Proof Blocks problems provide relative to ing students to engage more with proofs through self-explanation
other course content? increased student comprehension of proofs in a lasting way. Proof
RQ2: What is the relationship between the knowl- Blocks problems similarly force deliberate engagement with proof
edge required to complete Proof Blocks problems and content, as close reading is necessary to determine the correct
other types of problems in a discrete mathematics arrangement of lines. Proof Blocks also shows promise as a tool
course? that can provide scaffolding that students are so in need of when
RQ3: What are students’ perceptions about the fair- learning to write proofs.
ness, usability, and authenticity of being assessed by
using Proof Blocks problems? 2.3 Educational Theorem Proving Software
A few other software tools have been created to enable students to
create proofs in the computer in such a way that they can receive
automated feedback. Some use text-based representations, while
others use visual representations of proofs.
2 RELATED WORK Polymorphic Blocks [13] is a novel user interface which presents
Anecdotally, we have heard of instructors using scrambled proofs to propositions as colorful blocks with uniquely shaped connectors
assess student knowledge both in euclidean geometry and in higher- as a signifier of which types of propositions can be connected
level mathematics. In theory, instructors may have offered such in a proof. While the user interface has been shown to engage
questions on paper even before the advent of computers, though we students in learning proofs, it supports only propositional logic. The
can find no explicit record of this. Additionally, to our knowledge Incredible Proof Machine [3] guides students through constructing
there has been no research into the merits of these questions either proofs as graphs. As with Polymorphic Blocks, the user interface is
for learning or for assessment. engaging, but the formality of the system limits the topics which
We will give a brief overview of related work including Parson’s can be effectively covered.
problems, research on teaching and learning proofs, and software Jape [2] is a łProof calculator,ž which guides students through
tools for constructing mathematical proofs in an educational con- the process of constructing formal proofs in mathematical notation
text. with the help of the computer. While Jape can allow students to
158
Seth Poulsen Mahesh Viswanathan
University of Illinois at Urbana-Champaign University of Illinois at Urbana-Champaign
USA USA
Geoffrey L. Herman Matthew West
University of Illinois at Urbana-Champaign University of Illinois at Urbana-Champaign
USA USA
ABSTRACT [12, 16, 30]. A panel of 21 experts using a Delphi process agreed that
Proof Blocks is a novel software tool which enables students to 6 of the 11 most difficult topics in a typical discrete mathematics
write mathematical proofs by dragging and dropping prewritten course are related to proofs and logic [9].
lines into the correct order, rather than writing a proof completely There are many aspects of writing mathematical proofs that are
from scratch. We used Proof Blocks problems as exam questions for difficult. Many students fail to produce the basic building blocks
a discrete mathematics course with hundreds of students, allowing that proofs have, such as properly declaring variables or referenc-
us to collect thousands of student responses to Proof Blocks prob- ing theorems [23]. Students get stuck working through the details
lems. Using this data, we provide statistical evidence that Proof of algebraic manipulations. They have a tendency to commit cer-
Blocks are easier than written proofs, which are typically very tain logical fallacies such as confusing a proposisiton with its con-
difficult. We also show that Proof Blocks problems provide about verse [23, 27]. Studies have shown that even when students have all
as much information about student knowledge as written proofs. the prerequesite content knowledge to write a mathematical proof,
Survey results show that students believe that the Proof Blocks user they still struggle to construct one [32]. Thus, there is a gap that
interface is easy to use, and that the questions accurately represent needs to be filled between students having the content knowledge
their ability to write proofs. to write a proof and the aptitude to actually construct one.
Vygotsky’s theory of psychological development posits that be-
CCS CONCEPTS tween the tasks which a person can and cannot do, there is a so-
called zone of proximal development: a set of tasks which a person
· Mathematics of computing → Discrete mathematics; · So-
cannot perform unaided, but which they can perform when given
cial and professional topics → Computing education; · Ap-
help and support, called scaffolding [31, 36]. Computer science
plied computing → Computer-assisted instruction.
instructors and researchers have used various approaches to scaf-
folding students learning to write code for the first time. Block based
KEYWORDS
programming languages such as Scratch and Blockly [8, 15] scaffold
discrete mathematics, CS education, automatic grading, proofs students by providing them with building blocks from which to
ACM Reference Format: assemble their programs and guarding against the struggles of syn-
Seth Poulsen, Mahesh Viswanathan, Geoffrey L. Herman, and Matthew West. tax errors. Research has shown that using block based languages
2021. Evaluating Proof Blocks Problems as Exam Questions. In Proceedings can accelerate the student learning process when first learning to
of the 17th ACM Conference on International Computing Education Research program [34]. Parson’s problems are a kind of homework and exam
(ICER 2021), August 16ś19, 2021, Virtual Event, USA. ACM, New York, NY,
question where students are asked to assemble prewritten lines of
USA, 12 pages. https://doi.org/10.1145/3446871.3469741
code into a correct program [17]. Researchers have shown Parson’s
problems to be useful both as test questions [4] and as a learning
1 INTRODUCTION tool for helping to accelerate the learning process for beginners
Understanding and writing mathematical proofs is one of the crit- learning to write code [7].
ical yet difficult skills that students must learn as a part of the Following from the success of Parson’s problem and similar ap-
discrete mathematics curriculum. Proofs and proof techniques are proaches to teach programming, we propose Proof Blocks. Proof
included by the ACM curricular guidelines as a core knowledge Blocks allows students to construct mathematical proofs by drag-
area that should be understood by any student obtaining a degree in ging and dropping prewritten proof lines into the correct order,
computer engineering, computer science, or software engineering rather than having to write the entire proof from scratch. Figure 1
shows an example of a Proof Blocks problem. Proof Blocks provides
a scaffolded environment, enabling students to construct mathe-
This work is licensed under a Creative Commons matical proofs without needing to worry about coming up with
Attribution-NonCommercial-ShareAlike International 4.0 License. all of the details on their own. A Proof Blocks problem may also
ICER 2021, August 16ś19, 2021, Virtual Event, USA
contain distractor lines which are not a part of any correct solution.
© 2021 Copyright held by the owner/author(s). The design of the Proof Blocks grader [20] is flexible in allowing
ACM ISBN 978-1-4503-8326-4/21/08. any correct arrangement of the lines of the proof. This is enabled by
https://doi.org/10.1145/3446871.3469741
157
, ICER 2021, August 16ś19, 2021, Virtual Event, USA Poulsen et al.
the instructor specifying which lines of the proof depend on which 2.1 Parson’s Problems
other lines (the full dependence graph of the lines of the proof in The use of scrambled code problems was first documented by Par-
Figure 1 can be seen in Figure 2). Students who fail to construct sons [17]. They have since been studied for their desirable proper-
a correct proof on their first try can then receive automated feed- ties both in assessment and learning [4, 5, 7]. The desirable proper-
back from the computer, as shown in Figure 3, before being given ties of Parson’s problems were a major inspiration for the creation
additional attempts at the discretion of the instructor. of Proof Blocks.
Proof Blocks problems are also very promising for saving time for Denny et al. [4] showed that Parson’s problems are easier to
both students and course staff. Many computer science departments grade than free-form code writing questions, and yet still offer rich
are experiencing a huge increase in enrollments. This increase in information about student knowledge. We will show the same to
enrollments means course staff lose more time to grading, making it be true with Proof Blocks problems in relation to free-form proof
more difficult for them to spend the time they need helping students writing questions. Ericson et al. [7] showed that students learning
individually. Proof Blocks helps to alleviate this strain by providing to write code using Parson’s problems learn at an accelerated rate
a way to test some of students’ proof skills in a way that can be in the early stages of learning compared to students being taught
automated, saving grading time and allowing course staff more to fix code or write code from scratch.
time for other activities that help students such as office hours and
review sessions. 2.2 Research on Teaching and Learning Proofs
The ability to receive automated feedback is also a boon to stu-
dents. Due to staff time constraints, students in a discrete mathemat- There are many threads of research in seeking to illuminate stu-
ics course may not be able to receive feedback on the correctness of dents’ understandings and misunderstandings about proofs [24, 26,
proofs they write until long after they have completed them. Proof 27]. One thread establishes that, as they learn, students go through
Blocks also helps with this, as it allows students to receive feedback different phases in the complexity of ways they are able to think
instantly, just as they receive instant feedback from the compiler about solving proof problems [33]. Another study demonstrated
and from automated testing suites as they write code. that even when students had all of the knowledge required to write
In using a new kind of test question with our students, we wanted a proof and were able to apply that knowledge in other types of
to ensure that we were testing students on the correct set of skills questions, they were still unable to write a proof [32], thus high-
and that we were providing them with a fair and equitable learning lighting the need to scaffold students through the proof-writing
experience. process.
In this paper, we seek to answer the following three research On the other hand, there is little research on concrete educational
questions: interventions for improving the proof learning process [11, 27]. In-
deed, a recent review of the literature on teaching and learning
proofs concluded: łmore intervention-oriented studies in the area
RQ1: What statistical information about student knowl- of proof are sorely neededž [27]. Hodds et al. [11] showed that train-
edge do Proof Blocks problems provide relative to ing students to engage more with proofs through self-explanation
other course content? increased student comprehension of proofs in a lasting way. Proof
RQ2: What is the relationship between the knowl- Blocks problems similarly force deliberate engagement with proof
edge required to complete Proof Blocks problems and content, as close reading is necessary to determine the correct
other types of problems in a discrete mathematics arrangement of lines. Proof Blocks also shows promise as a tool
course? that can provide scaffolding that students are so in need of when
RQ3: What are students’ perceptions about the fair- learning to write proofs.
ness, usability, and authenticity of being assessed by
using Proof Blocks problems? 2.3 Educational Theorem Proving Software
A few other software tools have been created to enable students to
create proofs in the computer in such a way that they can receive
automated feedback. Some use text-based representations, while
others use visual representations of proofs.
2 RELATED WORK Polymorphic Blocks [13] is a novel user interface which presents
Anecdotally, we have heard of instructors using scrambled proofs to propositions as colorful blocks with uniquely shaped connectors
assess student knowledge both in euclidean geometry and in higher- as a signifier of which types of propositions can be connected
level mathematics. In theory, instructors may have offered such in a proof. While the user interface has been shown to engage
questions on paper even before the advent of computers, though we students in learning proofs, it supports only propositional logic. The
can find no explicit record of this. Additionally, to our knowledge Incredible Proof Machine [3] guides students through constructing
there has been no research into the merits of these questions either proofs as graphs. As with Polymorphic Blocks, the user interface is
for learning or for assessment. engaging, but the formality of the system limits the topics which
We will give a brief overview of related work including Parson’s can be effectively covered.
problems, research on teaching and learning proofs, and software Jape [2] is a łProof calculator,ž which guides students through
tools for constructing mathematical proofs in an educational con- the process of constructing formal proofs in mathematical notation
text. with the help of the computer. While Jape can allow students to
158
