A Framework for the Evaluation of Parallel and
Distributed Computing Educational Resources
David W. Brown Vitaly Ford Sheikh K. Ghafoor
Computer Science and Information Systems Computer Science and Math Computer Science
Elmhurst College Arcadia University Tennessee Tech University
Elmhurst, IL, United States Glenside, PA, United States Cookeville, TN, United States
Abstract—This paper proposes a classification scheme for cate- out of about 4,000 CS faculty) indicated the lack of proper
gorization of PDC educational resources. We have also proposed instructional materials as one of the major challenges of
an evaluation framework for assessing the PDC resources. Under integrating PDC. Also, a recent NSF funded workshop that was
the proposed framework, each resource type has a set of criteria
and an associated score. A PDC resource will obtain a score if participated by thirty thought leaders from Academia, Industry,
evaluated under our proposed framework that is the sum of the National Lab, and Government Agencies also identified a lack
scores of the criteria that the resource satisfy. The evaluation of appropriate instructional resources among one of the major
of whether a resource met a criteria is subjective. We have impediment to effectively integrate PDC in undergraduate
also presented our evaluation of PDC educational resources computing classes.
appropriate for CS1, CS2, and Data Structures and Algorithms
available on the web using our proposed framework. Many educators individually or in groups has been working
Index Terms—PDC Educational Resource, Resource Evalua- in creating curriculum and instructional materials, organizing
tion Framework, CS1, CS2, DS/A workshops, symposiums, and integrating PDC in undergrad-
uate CS classes. Lots of PDC educational resources have
I. I NTRODUCTION been developed/created in the last decade by these individuals
Parallel and Distributed Computing (PDC) has become and groups. The Center for Parallel and Distributed Com-
pervasive, from supercomputers and server farms containing puting Curriculum Development and Educational Resources
multi-core CPUs and GPUs, to individual PCs, laptops, and (CDER) [36] has been a pioneer in curriculum development
mobile devices. Even casual users of computers now depend and creating a repository for PDC educational resources. There
on parallel processing. Therefore, it is important for every are different types of resources (reading materials, slides,
computing professional (and especially every programmer) to syllabi, lab assignments, etc.) available on the web. These
understand how parallelism and distributed computing affect resources differ with respect to quality, maturity, complete-
problem solving. It is essential for educators to impart a range ness, usability, and adoptability. The community need more
of PDC and HPC knowledge and skills at multiple levels instructional resources, but there is also a need for systemic
within the educational fabric woven by Computer Science categorization, cataloging, and evaluations of the existing
(CS), Computer Engineering (CE), and related computational resources. Such categorization and evaluation of the available
curricula including data science. The need for including PDC resources will help the community.
in undergraduate computing curricula has drawn the attention This work is a preliminary attempt of such an effort, and it
of the CS education community. NSF/IEEE-TCPP Curriculum focuses on investigating what resources are available that will
Initiative on Parallel and Distributed Computing – Core Topics help faculty to integrate PDC in undergraduate curriculum.
for Undergraduates [36] released a curriculum recommenda- We want to examine what are the different type of resources
tion that contains what PDC topics should be included in first available, how easily adaptable they are, what PDC concepts
two years of computing curriculum. PDC has been included these resources cover at what level, and the quality of the
as a required knowledge area in ACM/IEEE 2013 curriculum resources. While we were searching for resources, we limited
recommendation [2]. Accreditation Board for Engineering and our focus on early CS classes (CS1, CS2, and data structures
Technology (ABET) [1] has also required exposure to PDC and algorithms). One of the challenges we faced was that there
for accreditation of computing programs. Additionally, federal is no catalog which would help to find such resources with the
funding agencies has been funding efforts in developing PDC exception of the commonly known CDER courseware [23],
curriculum and instructional materials in recent years. and the Internet search engines do not return any meaningful
There many challenges in integrating PDC in early CS results. We conducted our search based on our knowledge
classes. The major challenges are: 1) lack of trained faculty of the community efforts and looking through the recent
in PDC, 2) lack of instructional resources, and 3) lack of publications on PDC education. Due to time limitation, our
awareness among CS faculty. A recent survey by the authors search was primarily focused on the USA and by no means
of this paper in 2017 among the CS faculty (135 responded it was exhaustive. We are sure that we have missed many
, Fig. 1. Resource Classification Tree
resources. or instructors to understand or illustrate PDC concepts, iii)
In evaluating the PDC resources, we have proposed a libraries that can help to visualize PDC concepts or a parallel
resource classification scheme that can be used to classify a program, iv) auto grader and or assessment software that can
PDC educational resource. We have also proposed a resource help instructors and students. We are primarily focusing on
evaluation framework that can be used to assess a PDC edu- tools that are free or easily affordable.
cational resource and obtain a score indicative of the quality The available PDC educational resources needs to be eval-
of the resource. The proposed classification and evaluation uated based on some criteria that would be an additional help
framework is a preliminary attempt that needs to be examined, for an instructor to determine whether a particular material
deliberated, and improved further in the future. is suitable for her. We have proposed a resource evaluation
framework for different resources in our classification scheme.
II. E VALUATION C RITERIA AND C LASSIFICATIONS There are several criteria with an assigned score for each type
We categorize the available PDC educational resources into of resource. The total score of a resource is the sum of the
three major classes: those designed to aid in the classroom scores of all criteria met by the resource. In our proposed
instruction (III-A), those that provide access to hardware framework, instructional resources have 8 criteria, hardware
platforms for hands-on PDC exercises (III-B), and software resources have 5 criteria, and tools have 6 criteria. These
tools, libraries and games that can help with the instruction of criteria and the assigned score are our preliminary proposal
PDC (III-C). that requires further deliberation and refinement.
Instructional: These are resources that can help in teaching Following are the criteria for different types of resources
PDC concepts. These are materials that can be easily combined and their associated score:
with existing current course structures and materials with little Instructional - inspired by the Likert scale, these scores vary
or no effort. Example of these types resources are lecture on the scale of 5–10: 5 being the least applicable for a specified
slides, programming assignments, reading handout, course criteria, and 10 being the most applicable.
syllabi, questions banks, and unplugged activities. Some of 1) IC1 - All supplied documentation is complete (score 10)
these resources are meant to be used by the instructor only or simply in outline form (score 5)
and some may be by both instructor as well students. Some 2) IC2 - All material is current (score 10) or based on older
resources may have a different version for instructors and technology (score 5)
students. These will help students learn PDC concepts. These 3) IC3 - Supplied documents are easily editable (score 10)
resources can be further categorized based on what level of or do they exist only in a static form (score 5)
undergraduate classes (CS1, CS2, networking etc.) they are 4) IC4 - No special requirements are necessary for adoption
targeting or what PDC concept(s) they are covering. (score 10) or are there significant prerequisites (score 5)
Hardware: These are generally parallel distributed com- 5) IC5 - Solutions to any problem sets are supplied (score
puting platforms where students and instructors can perform 10) or is it the responsibility of the adopter to develop
hands-on PDC exercises. example of these could be a multi- them (score 5)
core server, a multi-node cluster with or without GPU capabil- 6) IC6 - Material is self-explanatory (score 10) or is
ity, Hadoop cluster, etc. The main characteristic of these types specialized knowledge required (score 5)
of resources is the that they are freely available and easily 7) IC7 - Separate resources are supplied for student or in-
obtainable to students and instructors. structor as appropriate (score 10) or are some component
Tools: These types of resources encompass i) software such resources missing (score 5)
as a java OpenMP thread library that can be used to write 8) IC8 - Setup and configuration instructions are complete
parallel programs, ii) games that can be used by the students and easy to follow (score 10) or are they vague or
Distributed Computing Educational Resources
David W. Brown Vitaly Ford Sheikh K. Ghafoor
Computer Science and Information Systems Computer Science and Math Computer Science
Elmhurst College Arcadia University Tennessee Tech University
Elmhurst, IL, United States Glenside, PA, United States Cookeville, TN, United States
Abstract—This paper proposes a classification scheme for cate- out of about 4,000 CS faculty) indicated the lack of proper
gorization of PDC educational resources. We have also proposed instructional materials as one of the major challenges of
an evaluation framework for assessing the PDC resources. Under integrating PDC. Also, a recent NSF funded workshop that was
the proposed framework, each resource type has a set of criteria
and an associated score. A PDC resource will obtain a score if participated by thirty thought leaders from Academia, Industry,
evaluated under our proposed framework that is the sum of the National Lab, and Government Agencies also identified a lack
scores of the criteria that the resource satisfy. The evaluation of appropriate instructional resources among one of the major
of whether a resource met a criteria is subjective. We have impediment to effectively integrate PDC in undergraduate
also presented our evaluation of PDC educational resources computing classes.
appropriate for CS1, CS2, and Data Structures and Algorithms
available on the web using our proposed framework. Many educators individually or in groups has been working
Index Terms—PDC Educational Resource, Resource Evalua- in creating curriculum and instructional materials, organizing
tion Framework, CS1, CS2, DS/A workshops, symposiums, and integrating PDC in undergrad-
uate CS classes. Lots of PDC educational resources have
I. I NTRODUCTION been developed/created in the last decade by these individuals
Parallel and Distributed Computing (PDC) has become and groups. The Center for Parallel and Distributed Com-
pervasive, from supercomputers and server farms containing puting Curriculum Development and Educational Resources
multi-core CPUs and GPUs, to individual PCs, laptops, and (CDER) [36] has been a pioneer in curriculum development
mobile devices. Even casual users of computers now depend and creating a repository for PDC educational resources. There
on parallel processing. Therefore, it is important for every are different types of resources (reading materials, slides,
computing professional (and especially every programmer) to syllabi, lab assignments, etc.) available on the web. These
understand how parallelism and distributed computing affect resources differ with respect to quality, maturity, complete-
problem solving. It is essential for educators to impart a range ness, usability, and adoptability. The community need more
of PDC and HPC knowledge and skills at multiple levels instructional resources, but there is also a need for systemic
within the educational fabric woven by Computer Science categorization, cataloging, and evaluations of the existing
(CS), Computer Engineering (CE), and related computational resources. Such categorization and evaluation of the available
curricula including data science. The need for including PDC resources will help the community.
in undergraduate computing curricula has drawn the attention This work is a preliminary attempt of such an effort, and it
of the CS education community. NSF/IEEE-TCPP Curriculum focuses on investigating what resources are available that will
Initiative on Parallel and Distributed Computing – Core Topics help faculty to integrate PDC in undergraduate curriculum.
for Undergraduates [36] released a curriculum recommenda- We want to examine what are the different type of resources
tion that contains what PDC topics should be included in first available, how easily adaptable they are, what PDC concepts
two years of computing curriculum. PDC has been included these resources cover at what level, and the quality of the
as a required knowledge area in ACM/IEEE 2013 curriculum resources. While we were searching for resources, we limited
recommendation [2]. Accreditation Board for Engineering and our focus on early CS classes (CS1, CS2, and data structures
Technology (ABET) [1] has also required exposure to PDC and algorithms). One of the challenges we faced was that there
for accreditation of computing programs. Additionally, federal is no catalog which would help to find such resources with the
funding agencies has been funding efforts in developing PDC exception of the commonly known CDER courseware [23],
curriculum and instructional materials in recent years. and the Internet search engines do not return any meaningful
There many challenges in integrating PDC in early CS results. We conducted our search based on our knowledge
classes. The major challenges are: 1) lack of trained faculty of the community efforts and looking through the recent
in PDC, 2) lack of instructional resources, and 3) lack of publications on PDC education. Due to time limitation, our
awareness among CS faculty. A recent survey by the authors search was primarily focused on the USA and by no means
of this paper in 2017 among the CS faculty (135 responded it was exhaustive. We are sure that we have missed many
, Fig. 1. Resource Classification Tree
resources. or instructors to understand or illustrate PDC concepts, iii)
In evaluating the PDC resources, we have proposed a libraries that can help to visualize PDC concepts or a parallel
resource classification scheme that can be used to classify a program, iv) auto grader and or assessment software that can
PDC educational resource. We have also proposed a resource help instructors and students. We are primarily focusing on
evaluation framework that can be used to assess a PDC edu- tools that are free or easily affordable.
cational resource and obtain a score indicative of the quality The available PDC educational resources needs to be eval-
of the resource. The proposed classification and evaluation uated based on some criteria that would be an additional help
framework is a preliminary attempt that needs to be examined, for an instructor to determine whether a particular material
deliberated, and improved further in the future. is suitable for her. We have proposed a resource evaluation
framework for different resources in our classification scheme.
II. E VALUATION C RITERIA AND C LASSIFICATIONS There are several criteria with an assigned score for each type
We categorize the available PDC educational resources into of resource. The total score of a resource is the sum of the
three major classes: those designed to aid in the classroom scores of all criteria met by the resource. In our proposed
instruction (III-A), those that provide access to hardware framework, instructional resources have 8 criteria, hardware
platforms for hands-on PDC exercises (III-B), and software resources have 5 criteria, and tools have 6 criteria. These
tools, libraries and games that can help with the instruction of criteria and the assigned score are our preliminary proposal
PDC (III-C). that requires further deliberation and refinement.
Instructional: These are resources that can help in teaching Following are the criteria for different types of resources
PDC concepts. These are materials that can be easily combined and their associated score:
with existing current course structures and materials with little Instructional - inspired by the Likert scale, these scores vary
or no effort. Example of these types resources are lecture on the scale of 5–10: 5 being the least applicable for a specified
slides, programming assignments, reading handout, course criteria, and 10 being the most applicable.
syllabi, questions banks, and unplugged activities. Some of 1) IC1 - All supplied documentation is complete (score 10)
these resources are meant to be used by the instructor only or simply in outline form (score 5)
and some may be by both instructor as well students. Some 2) IC2 - All material is current (score 10) or based on older
resources may have a different version for instructors and technology (score 5)
students. These will help students learn PDC concepts. These 3) IC3 - Supplied documents are easily editable (score 10)
resources can be further categorized based on what level of or do they exist only in a static form (score 5)
undergraduate classes (CS1, CS2, networking etc.) they are 4) IC4 - No special requirements are necessary for adoption
targeting or what PDC concept(s) they are covering. (score 10) or are there significant prerequisites (score 5)
Hardware: These are generally parallel distributed com- 5) IC5 - Solutions to any problem sets are supplied (score
puting platforms where students and instructors can perform 10) or is it the responsibility of the adopter to develop
hands-on PDC exercises. example of these could be a multi- them (score 5)
core server, a multi-node cluster with or without GPU capabil- 6) IC6 - Material is self-explanatory (score 10) or is
ity, Hadoop cluster, etc. The main characteristic of these types specialized knowledge required (score 5)
of resources is the that they are freely available and easily 7) IC7 - Separate resources are supplied for student or in-
obtainable to students and instructors. structor as appropriate (score 10) or are some component
Tools: These types of resources encompass i) software such resources missing (score 5)
as a java OpenMP thread library that can be used to write 8) IC8 - Setup and configuration instructions are complete
parallel programs, ii) games that can be used by the students and easy to follow (score 10) or are they vague or
