See discussions, stats, and author profiles for this publication at: [Link]

net/publication/282615679

A Performance-Based Evaluation Rubric for Assessing and Enhancing

Engineering Design Skills in Introductory Engineering Design Courses* KI-
IL SONG**, DO HYOUNG SHIN and SOOBONG SHI...

Article  in  International Journal of Engineering Education · July 2015

CITATIONS READS

0 429

1 author:

Sung-Hee Jin
Inha University
16 PUBLICATIONS   37 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Sung-Hee Jin on 07 October 2015.

The user has requested enhancement of the downloaded file.

International Journal of Engineering Education Vol. 31, No. 4, pp. 1007–1020, 2015 0949-149X/91 $3.00+0.00
Printed in Great Britain # 2015 TEMPUS Publications.

A Performance-Based Evaluation Rubric for Assessing and

Enhancing Engineering Design Skills in Introductory
Engineering Design Courses*
SUNG-HEE JIN
Academy of Convergence Education, Inha University, 100 Inha-ro, Incheon, 402-751, South Korea.

KI-IL SONG**, DO HYOUNG SHIN and SOOBONG SHIN

Department of Civil Engineering, Inha University, 100 Inha-ro, Incheon, 402-751, South Korea. E-mail: ksong@[Link]

Using the developmental research methodology, this study develops a performance-based evaluation rubric that can assess
and enhance students’ engineering design skills in introductory engineering design courses. The effectiveness of the
proposed performance-based evaluation rubric was validated in the context of engineering design activities for students
and practical directions for instructors to evaluate student performance. The proposed rubric included four phases, namely
the problem, the solution, implementation, and process management, based on seven performance criteria and 21
measureable scales.
Keywords: engineering design skill; engineering design assessment; evaluation rubric; performance-based evaluation

1. Introduction projects because of a lack of clear guidelines. A few

assessment methods such as student portfolios,
Introductory design courses have been developed to verbal protocol analyses, and the ‘‘Creative Engi-
enhance students’ motivation for engineering, neering Design Assessment’’ (CEDA) method have
increase student retention, and gain basic design been used to evaluate students’ design achievement
skills [1]. Many studies have reported that introduc- and processes [7–10]. These methods focus on eval-
tory engineering design courses have positive effects uating students’ design competencies and processes
on students’ intellectual development, retention, from instructors’ point of view but do not fully
and motivation for upper-division courses [2–5]. guide students’ design activities. Further, there is a
In the early 1990s, introductory engineering design well-established method for assessing and enhan-
courses have been introduced to engineering col- cing students’ design skills. This study develops a
leges in South Korea by the Accreditation Board performance-based evaluation rubric that can
Engineering Education of Korea. According to assess and enhance students’ engineering design
Taurasi (2007), about 65% of all engineers from skills and can be used in introductory engineering
mechanical, applied, and manufacturing engineer- design courses. In particular, the literature on gen-
ing firms surveyed agreed on a need for today’s eral characteristics, important issues, and assess-
engineers to be more creative and innovative to be ment methods for introductory engineering design
globally competitive Because the need for creativity courses is reviewed to establish some directions for
in engineering emphasizes creativity in engineering developing a rubric. The developed performance-
design activities [6], the introductory engineering based evaluation rubric is applied to introductory
design course in Korea is entitled ‘‘Creative Engi- engineering design courses, and its feasibility and
neering Design,’’ and many engineering educators effectiveness are validated.
have made efforts to prepare and teach this subject.
Engineering educators must be able to develop 2. Theoretical background
students’ design skills and assess their skills appro-
priately through engineering design courses. One 2.1 Introductory engineering design
important issue in teaching introductory design
Engineering design reflects a systematic, intelligent
courses is the assessment of students’ engineering
generation, and evaluation of specifications for
design capability. In addition, many engineering
artifacts whose form and function achieve stated
educators have difficulty assessing students’ design
objectives and satisfy specified constraints [11].
skills, and students struggle with engineering design
According to the ABET, ‘‘engineering design is a
decision-making process, in which the basic
** Corresponding author. sciences, mathematics, and engineering sciences

* Accepted 17 March 2015. 1007

1008 Sung-Hee Jin et al.

are applied to convert resources optimally to meet a engineering design processes to solve ill-structured
stated objective’’ [12]. and ill-defined design problems, and these problems
Introductory engineering design courses focus must be associated with daily life [21].
more on conceptual design methods and less on To define a design problem, students have to
discipline-specific artifacts [13]. First-year students actively analyze needs of target users and gather
can be guided to perform reasonable conceptual information from experts, patent databases, or
design without detailed technical knowledge. electronic databases. Students look for underlying
Marra, Palmer and Litzinger (2000) found the needs by observations, interviews, or survey meth-
introductory engineering design course at Penn ods and search for sufficient information on the
State University to be a project-focused, active initial problem, such as patents, products, or profes-
course requiring students to work in teams and sional knowledge to analyze reasons and identify
interact with instructors to solve open-ended pro- design requirements and constraints. Design
blems based on the design process [5]. The first-year requirements are objectives that should be ulti-
engineering design course in the department of mately satisfied. Design constraints reflect a
mechanical engineering at Stanford University number of limitations on conditions under which
was found to give students exposure to the creative the development of a system takes place, such as
nature of engineering through design projects, forms, functions, technologies, budgets, and time
hands-on laboratory exercises, and open-ended periods.
problem solving [14]. A creative engineering
course at Sungkyunkwan University was found to
provide learning opportunities during the pursuit of 2.2.2 Design process
design solutions through collaborative efforts of Introductory engineering design courses typically
design teams as well as innovative and effective focus on producing conceptual solutions, and there-
design ideation methods and design presentation fore engineering design processes are generally
skills [15]. According to the definition of engineering recognized according to the following steps: pro-
design and an overview of introductory design blem recognition or understanding ! information
courses, introductory engineering design can be gathering ! problem analysis ! idea generation !
defined as suggesting creative conceptual solutions idea evaluation ! validation ! communication
for open-ended problems through engineering [1, 16, 22, 23]. Introductory engineering design
design processes [16]. In summary, three issues in activities can be classified into three types according
engineering design activities include the design pro- to two variables: whether to suggest a design pro-
blem, the design process, and the design output blem and whether to provide information or knowl-
(type) [17], and the next section addresses these edge needed to solve a problem [24]. The first type is
issues in detail. ‘‘guided engineering design,’’ in which core con-
cepts or knowledge is learned through lectures
2.2 Three issues: The problem, the process, and the before design activities start and design problems
output are provided by instructors. The second type is
‘‘engineering design by problem-based learning,’’
2.2.1 Design problem in which a design problem is presented by instruc-
Kaufmann (1988) defined a ‘‘problem’’ as a gap tors but students have to learn core concepts or
between the current and desired states [18]. The knowledge needed to solve the problem by them-
definition of a problem should have at least three selves. Finally, the third type is ‘‘engineering design
characteristics: The problem exists in the current by self-directed learning,’’ in which students have to
state with particular givens such as circumstances, find a design problem as well as learn core concepts
objects, or pieces of information; the desired state or knowledge to solve the problem through self-
requires pondering to transform the current state directed learning. Despite the importance of pro-
into the desired state; and there is no direct and blem finding [25], engineering educators have
obvious way to solve the problem [19]. Many focused mainly on solving problems in introductory
engineering education researchers have agreed design courses and on making design problems.
that design problems are open-ended and ill-struc- Most design processes or creativity support tools
tured [11, 20]. When a design problem is initiated, guide students to generate possible solutions, but
the goals are usually vague, and constraints and they do not identify new problems [26]. In the
criteria need to be clarified. These problems cannot present study, the emphasis is placed on engineering
be resolved mechanically or automatically by using design activities of the third type. That is, students
known methods, and there are a considerable are guided to find their own design problem and
number of solutions [16]. In an introductory engi- learn information or knowledge related to it by
neering design course, students should conduct themselves.
A Performance-Based Evaluation Rubric for Assessing and Enhancing Engineering Design Skills 1009

2.2.3 Design output Documents can include portfolios, concept maps,

Design outputs result from the engineering design and paper-based assessment tools. A portfolio is a
process and are normally documented in models, collection of students’ work over time and is used to
drawings, engineering analyses, and other docu- holistically assess student learning. Olds and Pave-
ments. Pahl and Beitz (1984) offered three types of lich (1996) suggested that portfolios can be inte-
design outputs [27]: First, ‘‘original design’’ aims to grated into engineering design activities [9]. Concept
create an original solution principle for a system maps have been applied to assess students’ design
with the same, similar, or new task. Second, ‘‘adap- knowledge by asking them to draw a concept map of
tive design’’ is for adapting known solution princi- the design process at the beginning and end of first-
ples to satisfy new tasks. Finally, ‘‘variant design’’ year design courses [35]. Although it is clear that the
varies certain aspects of the system, leaving function concept map technique is an effective tool for
and solution principles unchanged. These different assessing students’ integrated design knowledge, it
design types are clearly related to the design process, is difficult to be used as an evaluation tool.
and therefore an introductory engineering design Some researchers have developed paper-based
focusing on conceptual design can be classified as assessment tools to evaluate students’ design skills.
original design. Therefore, the output of original Baily, Szabo, and Sabers (2004) used analytical
design should be original (new idea) and applicable writing to assess design knowledge by asking stu-
(useful) [17, 28]. In engineering design, patentability dents to critique two ill-structured design processes
is a generally acceptable perspective when evaluat- [30]. Frank, Strong, Boudreau, and Pap (2009)
ing the design output. A design must satisfy the developed a paper-based design skill assessment
following criteria to produce a patent [29]: (1) it is method in which students are provided open-
new in that the design output has never been ended engineering scenarios and asked to outline
released to the public, (2) it is capable of industrial the process they follow to solve a problem [36].
application, and (3) it involves an inventive step. Charyton and Merrill (2009) developed the ‘‘Crea-
Because freshman and sophomore engineering stu- tive Engineering Design Assessment’’ (CEAD)
dents may have difficulty achieving technical com- method [8], in which students are asked to sketch
pleteness, the design output should be assessed in designs that incorporate one or several three-dimen-
terms of whether the solution for the initially stated sional objects, to list potential users, and to perform
problem is novel and applicable in a real setting [16]. problem finding as well as problem solving in
response to specific functional goals. These docu-
ment methods are assessed using analytic rubrics.
2.2.4 Assessment of engineering design skills Feasible quantitative techniques include self-
Although previous assessments have typically relied assessment and test methods. Gintili et al. (1999)
on team grades on design reports, a team’s perfor- used a self-assessment method to measure students’
mance in terms of design artifacts, and the team’s perceived growth in terms of their design capability
ability to answer design questions on tests [30], and a self-assessment instrument consisting of seven
various approaches have been considered to assess categories, including information gathering, pro-
engineering design skills. Assessment techniques can blem definitions, idea generation, evaluation/deci-
be classified into quantitative and qualitative assess- sion making, implementation, teamwork, and
ment tools. Because of characteristics of engineering communication [37]. Okudan, Ogot, Zappe, and
design skills, qualitative assessment techniques have Gupta (2007) developed the ‘‘Comprehensive
been applied and developed to assess design skills in Assessment of Knowledge on Engineering
introductory engineering design courses. Design,’’ which can be used as a pretest/posttest
The major qualitative assessment techniques instrument for assessing the knowledge level in
applied to introductory design courses include engineering design [38]. This instrument consists
verbal protocols, interviews, and documents [31]. of 20 questions designed to measure skills and
In verbal protocols, by collecting and analyzing concepts that can be taught in introductory engi-
verbal data on the cognitive processing of design neering design courses.
activities, students are monitored and asked to think These existing assessment tools suggest some
aloud during design activities. Several studies have important practical implications for assessing stu-
analyzed verbal protocols to identify important dents’ engineering design skills in various ways.
factors in engineering design [32–34]. Interviews Most qualitative methods can be costly and time
have also been used to obtain in-depth information consuming if they are used only when instructors
on students’ design activities. Marra, Palmer, and evaluate students’ capability. Therefore, this study
Litzinger (2000) used the interview technique to develops an evaluation rubric that can guide team-
examine the effects of first-year design courses on based engineering design activities as well as assess
engineering students’ intellectual development [5]. students’ engineering design skills.
1010 Sung-Hee Jin et al.

3. Directions for design assessment 3.3 Scoring scales

Scoring scales are necessary for determining student
3.1 Three phases of engineering design performance relative to the desired level of achieve-
The purpose of an introductory engineering design ment. In this study, a four-point scoring scale was
course is to enhance the student’s engineering design used (1: poor; 2: marginal; 3: satisfactory; 4: excel-
skills through engineering design activities to make lent). This four-point scoring scale was used because
conceptual solutions based on an engineering design most evaluators tend to give average scores. In this
process for open-ended and ill-structured problems. scale, 1 represents a low level (i.e., a student not
The core learning outcomes of introductory engi- satisfying most of the performance criteria); 2, a
neering design courses include engineering design marginal level (a student partially satisfying perfor-
skills. Engineering design activities are usually mance criteria for each design step); 3, a satisfactory
team-based, and with the consideration of this level (a student satisfying performance criteria for
characteristic, teamwork and communication each design step); and 4, a high level (a student
skills are secondary learning outcomes. Therefore, exceeding performance criteria set for the introduc-
this study develops a rubric that can evaluate the tory design course). Table 2 shows the scoring scales
following three learning outcomes: engineering for idea generation in the solution phase.
design, teamwork, and communication skills.
The engineering design process can be classified 4. Validation of the performance-based
into three phases: (1) the problem phase, (2) the evaluation rubric
solution phase, and (3) the implementation phase.
In the problem phase, students identify and analyze The performance-based evaluation rubric (here-
a problem and finally define a design problem after ‘‘the PBER’’) for introductory engineering
including design requirements and constraints. Stu- design courses was validated using three different
dents investigate prior art and related information, methods based on Tracey and Richey (2007): an
analyze causes of the problem, and define the expert review, a usability test, and a field evaluation
problem through the process of problem recogni- [39]. These three validation methods were used to
tion, information gathering, and problem defini- find implications for modifying the PBER as well as
tions. A team project proposal can be an to validate its effectiveness in guiding or assessing
evaluation tool in the problem phase. In the solution students’ design activities. More specifically, the
phase, students develop several possible solutions first expert review was conducted to validate the
for the defined problem through the use of creative developed PBER. A usability test was conducted for
idea generation methods and select optimal solu- the instructor to examine the effects of the PBER
tions. A midterm report and a peer evaluation can and obtain opinions on the rubric after its classroom
be evaluation tools in the solution phase. The use. The second expert review was conducted to
selected solution is evaluated in terms of its origin- verify the modified PBER based on the first review.
ality and applicability. In the implementation Finally, a field evaluation was employed to investi-
phase, students develop a prototype of the selected gate the effectiveness of the PBER as guidelines for
solution and verify its compliance with design students’ engineering design activities.
requirements and constraints. In addition, they
4.1 Expert review
write a final report for their whole design process
and make a presentation. A final report, a presenta- 4.1.1 Procedures and participants
tion, and a peer evaluation can be evaluation tools Ten experts participated in expert reviews to vali-
in the implementation phase. date the PBER. All were engineering professors who
taught introductory engineering design courses
3.2 Performance criteria and tasks more than three times. The expert group included
To evaluate performance in terms of accomplishing two civil engineering professors (A and B), two
learning objectives of three phases (e.g., problem, geoinformatic engineering professors (C and D),
solution, and implementation phases), performance one mechanical engineering professor (E), one
criteria should be established. Performance criteria industrial engineering professor (F), one chemical
should specify observable details that evidence the engineering professor (G), one architectural engi-
desired state [1]. Performance tasks are goal-direc- neering professor (H), and two material engineering
ted activities completed by students and then judged professors (I and J). The purpose of the PBER
by instructors based on specific performance cri- research and development was introduced to seven
teria. The introductory engineering design process experts (A, B, C, E, F, G, and J), and these
generally consists of eight steps. Table 1 shows the individuals were invited to participate in an expert
performance criteria and tasks for each step. review. In addition, an expert review instrument was
A Performance-Based Evaluation Rubric for Assessing and Enhancing Engineering Design Skills 1011

Table 1. Performance criteria and tasks for engineering design skills

Phases Steps Performance criteria Performance tasks

Problem Problem recognition Students can identify open-ended, Determination of open-ended , challenging,
challenging, and impactive design problems and impactive design problems

Information Students can survey and analyze related A needs analysis

gathering information for a design problem Information investigation

Problem definition Students can define design problems, A gap analysis

including design requirements and Problem definitions
constraints
Requirements and constraints

Solution Idea generation Students can develop several possible The climate and attitudes
solutions for design problems by applying Creative methods
creative idea generation methods with team
members The number of ideas

Optimal solution Students can select optimal solutions from Idea evaluation
selection several possible solutions based on their Originality
originality and applicability
Applicability

Implementation Solution Students can elaborate on selected optimal Prototypes

improvement solutions with drawings or prototypes Completeness
/punctuality

Validation Students can verify the compliance of Requirements

selected solutions with design requirements Constraints
and constraints

Presentation and Students can effectively deliver engineering Structures and fidelity
reporting design processes and their results with Speaking, listening, and responding
writings and words

Table 2. Scoring scales for the idea generation step

Scales Poor Marginal Satisfactory Excellent

1 2 3 4

Climate and attitudes The team atmosphere was Students respected other Students were willing to Students actively
critical and stifling, and members’ opinions but suggest their own creative suggested their own
students did not actively did not suggest their own ideas and encouraged and creative ideas and
participate in team work. ideas. supported other motivated and stimulated
members’ opinions. other members’ opinions.

Creative methods Students did not use any Students developed their Students developed their Students developed their
creative idea generation solutions with one solutions with two solutions with three or
methods. creative idea generation creative idea generation more creative idea
method. methods. generation methods.

Quantity Students did not develop Students developed one Students developed two Students developed three
any possible solutions for possible solution for a possible solutions for a or more possible
a defined problem. defined problem. defined problem. solutions for a defined
problem.

provided and collected by e-mail for the other three the appropriateness of dividing engineering design
experts (D, H, and I) after a brief introduction to the activities into three phases (problem, solution, and
purpose of the PBER research and development by implementation phases), three items for the appro-
phone. priateness of a stepwise design process for each
phase, eight items for the appropriateness of per-
4.1.2 Instrument and analysis formance criteria for each phase, eight items for the
The instrument for the expert review was designed appropriateness of evaluation items for each phase,
to evaluate the appropriateness of the proposed and eight items for the appropriateness of rating
PBER by using a four-point Likert-type scale. The criteria for each phase. These experts were asked to
instrument was composed of 28 items: one item for comment on these items if they had any. The expert
1012 Sung-Hee Jin et al.

review instrument was circulated within the expert uncomfortable things in their daily lives through
group to verify its understandability and improve its team activities.
validity based on comments raised by the expert
group. 4.2.2 Instrument and analysis
The validity of expert responses was analyzed The instrument in the usability test was composed of
using the content validity index (CVI) and the three parts. The first part included five items for the
inter-relater agreement (IRA) adopted in Rubio et usability of the PBER from the instructor’s point of
al. (2003) [40]. The CVI is the number of experts view. The second part included eight items to
evaluating each item as valid divided by the total investigate the instructor’s perception of the educa-
number of experts. The CVI provides a ratio of tional effectiveness of the PBER in each design
experts assessing items to be valid, and a CVI > 0.80 stage. The third part included three open-ended
is considered valid. The IRA gives a score for the items to find the PBER’s strengths, weaknesses,
trustworthiness of experts’ evaluation and is calcu- and possible improvements.
lated by dividing the number of items evaluated The test items to measure the usability of the
equally by the total number of items. An IRA > 0.80 PBER from the instructor’s point of view were
is considered valid. In this study, CVI and IRA adopted from Lee and Jin (2014) [16] and modified
scores were measured on a four-point Likert-type to accommodate the purpose of the present study.
scale: ‘‘strongly disagree’’ (1), ‘‘disagree’’ (2), Five items asked about the efficiency, effectiveness,
‘‘agree’’ (3), and ‘‘strongly agree’’ (4). generality, learnability, and satisfaction of the
PBER. To verify the educational effectiveness of
4.1.3 Results of an expert review the PBER as a guide for a creative engineering
design process, eight items were derived to assess
According to the expert review of the PBER, the
the helpfulness of the PBER to the design process. A
average score was 3.65 (out of 4.0), and all CVI and
total of 13 items were reviewed by experts (M and N)
IRA values for each item exceeded 0.8 (Table 3).
with Ph.D. degrees in education and teaching
These results indicate all elements of the PBER to be
experience in introductory engineering design
highly valid and reliable. However, the rating cri-
courses. Responses to the questionnaire were pre-
teria for information gathering (experts C and I)
sented along with items, and collected data from
and idea generation (experts B and C) were evalu-
open-ended items were grouped by considering the
ated as inappropriate because of their mechanical
theme of opinions.
judgments. In addition, those items in the PBER
were evaluated as appropriate by most experts, and 4.2.3 Results of the usability test
therefore its appropriateness was reexamined
Table 4 shows the responses of three instructors to
through a usability test and a field evaluation.
the usability test. The results show that the instruc-
tors were satisfied with the PBER in terms of its
4.2 Usability test above-average efficiency, effectiveness, generality,
learnability, and satisfaction. In addition, the
4.2.1 Procedures and participants instructors recognized the PBER to be very helpful
A usability test was conducted to assess the effec- for guiding students in relevant activities through
tiveness of the PBER and identify possible improve- engineering design stages.
ments from instructors after their use of the PBER With respect to the items for strengths of the
in their introductory engineering design courses. PBER, expert B reported that the PBER presented
Three experts (expert B, who participated in the required activity levels and content and had educa-
expert review, K, and L) who conducted introduc- tional effectiveness, showing the student the overall
tory engineering design courses at I university in engineering design process. Expert K responded
Korea in the spring semester of 2013 participated in that the PBER enables a quantitative and objective
the usability test. The purpose of the study and the evaluation based on the rubric in each design stage
PBER were explained to these three experts, who instead of on some rough and qualitative evaluation
were asked to provide the PBER to students and use criteria. According to expert L, students can apply
it for evaluating team projects at the end of seme- the PBER to other design activities even after the
ster. In the introductory engineering design course, introductory engineering design course. This
students learned about the engineering design pro- implies that the PBER can be generalized to evalu-
cess and employed various creative thinking tech- ate engineering design activities.
niques during the first half of the semester. Then In terms of the items for shortcomings and
they carried out team projects until the end of the weaknesses, expert B suggested that the PBER
semester. Problems in team projects were not sug- should be improved to allow for a sufficient evalua-
gested by instructors, but students actively found tion of project quality because it was observed that
A Performance-Based Evaluation Rubric for Assessing and Enhancing Engineering Design Skills 1013

Table 3. Review results for the structure and components of the PBER and the validity of expert responses

Expert responses

Items A B C D E F G H I J CVI IRA

Three phases 4 4 3 4 3 4 3 4 4 3 1 1

Learning objectives Problem phase 4 4 3 3 3 4 4 4 4 4 1

Solution phase 4 4 3 3 4 4 4 4 4 4 1
Implementation phase 4 4 4 3 4 4 4 4 4 4 1

Design process Problem phase 4 3 3 3 3 4 3 4 3 3 1

Solution phase 4 4 4 4 4 4 4 4 4 4 1
Implementation phase 4 4 4 4 4 4 4 4 4 4 1

Problem recognition Performance criteria 4 3 4 3 3 4 4 4 4 4 1 1

Evaluation items 4 3 4 3 3 4 4 4 4 4 1
Rating criteria 4 3 4 2 4 4 4 4 4 4 0.9

Information gathering Performance criteria 4 4 3 3 4 4 4 4 4 4 1 1

Evaluation items 4 3 3 3 4 4 4 3 3 4 1
Rating criteria 4 3 2 3 4 4 4 3 2 4 0.8

Problem definition Performance criteria 4 4 3 3 3 4 4 4 3 4 1 1

Evaluation items 4 3 3 3 3 4 4 4 3 4 1
Rating criteria 4 3 3 3 3 4 4 4 4 4 1

Idea generation Performance criteria 4 4 3 3 3 4 4 4 4 4 1 1

Evaluation items 4 3 3 3 3 4 4 3 3 4 1
Rating criteria 4 2 2 3 3 4 4 3 3 4 0.8

Optimal solution Performance criteria 4 4 3 3 3 4 4 4 4 4 1 1

selection
Evaluation items 4 3 3 3 3 4 4 4 3 4 1
Rating criteria 4 3 3 3 3 4 4 4 4 4 1

Solution improvement Performance criteria 4 4 3 3 3 4 4 4 4 4 1 1

Evaluation items 4 4 3 3 3 4 4 4 3 4 1
Rating criteria 4 4 3 3 3 4 4 4 4 4 1

Validation Performance criteria 4 4 3 3 3 4 4 4 4 4 1 1

Evaluation items 4 3 3 3 3 4 4 4 4 4 1
Rating criteria 4 3 4 3 3 4 4 4 4 4 1

Presentation and Performance criteria 4 4 4 3 4 4 4 4 4 4 1 1

reporting
Evaluation items 4 4 4 3 4 4 4 3 3 4 1
Rating criteria 4 4 4 3 4 4 4 4 4 4 1

Table 4. Experts’ responses to the usability test questionnaire

Instructors’ responses

Items B* K* L*

Usability of the PBER Efficiency 4 5 5

Effectiveness 4 5 5
Generality 4 5 5
Learnability 4 5 5
Satisfaction 4 5 5

Learning effectiveness of the PBER Problem recognition 4 4 4

Information gathering 4 4 4
Problem definition 4 4 5
Idea generation 3 3 4
Optimal solution selection 4 4 5
Solution improvement 4 4 5
Validation 4 4 5
Presentation and reporting 4 4 5

* Three experts who participated in the usability test (see Section 4.2.1).
1014 Sung-Hee Jin et al.

some students tried to satisfy quantitative criteria design courses at I University participated in this
such as numbers of ideas and solutions without study. The experiment was conducted as part of the
considering this quality only to secure full credit. students’ regular coursework.
Given this suggestion, rating criteria for informa-
tion gathering and idea generation were amended to 4.3.2 Study design and procedures
allow for a qualitative evaluation. The students were randomly assigned to the experi-
Expert K suggested the separation of activities mental and control groups. Their motivation and
related to the design process from those related to knowledge of the design process in the introductory
the management of design projects. In the solution engineering design course were pretested to verify
improvement stage under the implementation the homogeneity of each group. Based on the
phase, the completeness and punctuality of perfor- pretest, two groups were considered homogeneous.
mance tasks were the items evaluated through the Based on the developed PBER, information on
engineering design process. Therefore, a new item engineering design processes and activities that
‘‘design process management’’ was included to needed to take place at each stage was provided to
implement the suggestion of expert K. all students, who completed their exercises in the
Expert L agreed that the problem definition classroom for the given case study. Team projects
needed to follow the recognition of the problem were assigned to each group. The developed PBER
and the gathering of relevant information in the was provided to the experimental group but not to
problem phase. After using the original rubric, the control group for the team project. The intro-
however, he suggested that problem recognition ductory engineering design course entailed the fol-
and information gathering needed to be combined lowing activities: an introduction to the subject and
to enhance the effectiveness of the PBER because a a pretest in the first week; instructions for the
problem can be found through information-gather- engineering design process and team-based exer-
ing activities and various solutions to a particular cises between week 2 and week 8; team projects
problem can be provided. To reflect the suggestion between week 9 and week 13 (students actively work
by expert L, problem recognition and information as a group to follow the engineering design process,
gathering in the problem phase were combined in including problem, solution, and implementation
the PBER. Given the suggestions by the three phases); the team project presentation between
experts participating in the usability test, the origi- week 14 and week 15; and information obtained
nal PBER was enhanced and updated. from students through a survey in week 16.
A second expert review was conducted to validate
the appropriateness of the improved PBER (see 4.3.3 Measures
Appendix 1: The modified PBER). Four experts The following variables were measured: the study
(expert B, K, and L, participants in the usability time per week, study efforts, learning outcomes for
test, and professor M, a civil engineering specialist engineering design skills, and student satisfaction.
with 16 years of teaching experience) participated in The study time per week was assessed using one item
the second expert review. The initial expert review (multiple choice) covering five alternatives (less than
instrument was modified to reflect the updated 3 hours per week (1) over 12 hours per week (5)).
rubric. The instrument was received by e-mail Study efforts were assessed using one item based on
from experts B, K, and L. The research purpose a five-point Likert-type scale ranging from ‘‘extre-
and process were introduced to expert M in person, mely low’’ (1) to ‘‘extremely high’’ (5) [41]. The
and the expert review instrument was provided and scales for assessing engineering design skills were
collected by e-mail. According to the results of the developed based on performance criteria for each
second expert review, the average score was 3.77 process of the PBER and consisted of seven items
(out of 4.0), and all CVI and IRA values for each (a = 0.89). The scales for perceived student satisfac-
item were 1, indicating that all PBER elements were tion were developed by referring to those items in
highly valid and reliable. Arbaugh (2000) [42]. Perceived student satisfaction
was assessed using three items (a = 0.86), including
4.3 Field evaluation ‘‘I was satisfied with this course’’ and ‘‘My choice to
take this course was a wise one.’’ In addition, seven
4.3.1 Participation items (a = 0.85) were developed to assess the help-
The purpose of the field evaluation was to identify fulness of the PBER for managing activities in the
the educational rationale for the PBER. To reveal overall engineering design process and the sixth
the practical applicability of the PBER as a guide for stage of the engineering design process. These
students’ design activity, a field evaluation can be items were designed for the experimental group to
essential. A total of 77 undergraduate students assess the learning effectiveness of the PBER. Per-
(64 males) enrolled in introductory engineering ceived engineering design skills, perceived student
A Performance-Based Evaluation Rubric for Assessing and Enhancing Engineering Design Skills 1015

Table 5. Descriptive statistics for dependent variables for two groups

Dependent measures Treatment

Experiment group (n = 39); control group (n = 38)

M (SD) M (SD) t value d

a
Study time per week 2.84 (0.85) 2.36 (0.63) 2.511*
0.77
Study efforts 4.00 (0.70) 3.75 (0.69) 1.376
Perceived engineering Design skills 4.20 (0.62) 3.74 (0.66) 2.844**
0.70
Perceived student satisfaction 4.24 (0.62) 4.01 (0.70) 1.268

a
(1) Less than 3 hours; (2) 3  6 hours; (3) 6  9 hours; (4) 9  12 hours; (5) over 12 hours.

satisfaction, and learning effectiveness for the 5. Discussion

PBER were measured using a five-point Likert-
type scale ranging from ‘‘strongly disagree’’ (1) to
‘‘strongly agree’’ (5). Two experts (M and N) who 5.1 Theoretical contributions
participated in the usability test examined the Education-related evaluation methods used in a
validity of the developed items. specific course generally entail objective-based eva-
luation techniques to determine the level of student
4.3.4 Analysis achievement in terms of learning objectives [44].
The collected data were analyzed through an inde- Therefore, many studies have focused mainly on
pendent samples t-test to determine the impact of evaluating the level of engineering design skill
the PBER in the context of introductory engineer- acquired by students [8, 36, 38]. The objectives of
ing design courses. The dependent measures this educational evaluation or assessment were to
included the study time, study efforts, the develop- verify the level of student achievement in terms of
ment of engineering design skills, and student learning goals while reflecting evaluation results to
satisfaction. Effect size was measured using Cohen’s improve students’ competency and educational
d if a significant effect was found. Effect size was quality. In this regard, this study provides instruc-
interpreted as follows: small <0.2, moderate 0.5, tors with objective criteria for assessing students’
and large >0.8 [43]. A basic statistical analysis was engineering design skills along with scaffolding
conducted for the learning effectiveness of the students during engineering design activities. The
PBER targeting the experimental group. proposed rubric can provide students with a clear
level of the instructor’s expectations to increase
4.3.5 Results of the field evaluation their performance in areas such as engineering
As shown in Table 5, the mean values for the design skills. This result provides support for
experimental group (with the PBER) were higher Reddy and Andrade (2010), who suggested that
than those for the control group in terms of the the rubric can promote learning as well as help
study time per week, study efforts, perceived engi- teachers evaluate student performance [45]. The
neering design skills, and perceived student satisfac- PBER can help students guide engineering design
tion. In particular, the independent samples t-test activities by specifically stating a desired perfor-
revealed significant effects on the study time per mance level in engineering design. Scaffolding has
week (M = 2.84, SD = 0.85 vs. M = 2.36, SD = 0.63, been defined as a learning strategy supporting
d = 0.77) and perceived engineering design skills learners by limiting the complexity of the learning
(M = 4.20, SD = 0.62 vs. M = 3.74, SD = 0.66, d =
0.70), and here the effect size was moderate. These Table 6. Students’ perception of the learning effectiveness of the
results demonstrate that those students exposed to PBER
the PBER invested more time in performing team- Items M SD
based design projects and were more likely to
perceive the PBER to increase their performance Learning Problem recognition 4.12 0.72
effectiveness Problem definition 4.04 0.61
in activities in each engineering design stage. of the PBER
Idea generation 3.96 0.61
According to the students’ perception of the learn-
Optimal solution selection 4.16 0.47
ing effectiveness of the PBER (Table 6), the PBER
Solution improvement 4.00 0.64
was helpful for student activities and could be used Presentation and reporting 4.16 0.68
as a useful guide for successful engineering design Design process management 4.12 0.60
activities.
1016 Sung-Hee Jin et al.

context [46]. Among the four types of scaffolding their engineering design outcome by using the
suggested in Hannafin, Hannafin, Land, and Oliver PBER.
(1997) [47], two are relevant to the PBER. First, the
PBER can serve as some conceptual scaffolding to 6. Conclusions
help student decide what to do at each engineering
design stage. Second, it can act as a procedural form The present study proposes the performance-based
of scaffolding to help students determine which evaluation rubric (PBER), which can support stu-
procedures they should take to solve a design dents’ engineering design activities and provide
problem. The results verify the usability of the practical directions for instructors to evaluate stu-
PBER from the instructor’s perspective in terms of dent performance. Students are expected to suggest
its efficiency, effectiveness, generality, learnability, creative conceptual solutions to open-ended pro-
and satisfaction. The effectiveness of the PBER was blems through engineering design processes by
verified based on the instructor’s perception and working with team members in introductory engi-
field evaluation. neering design courses. In this regard, the PBER
reflects review outcomes for three issues of design
5.2 Practical contributions programs, processes, and outputs identified in prior
research on the assessment of engineering design.
The major difficulties facing instructors in mana-
The results of expert reviews, usability tests, field
ging introductory engineering design courses can be
evaluations validate the PBER for two semesters.
summarized into two areas: (1) monitoring and
The PBER can provide scaffolding to support
providing feedback on engineering design activities
students’ engineering design activities and offer
conducted through team-based project learning
instructors with objective criteria for evaluating
methods and (2) objectively evaluating engineering
students’ engineering design skills. The PBER con-
design processes and outcomes. Students want to
sists of four phases, namely problem, solution,
receive consultation on their engineering design
implementation, and process management phases,
performance and ask for the desired level of achieve-
and include seven performance criteria and 21
ment in introductory engineering design courses. In
measureable scales. Evaluation tools for each
addition, because evaluation scores are not based on
phase are as follows: a team project proposal for
tests, students demand the justification of their
the problem phase, a progress report and a peer
grades on their engineering design projects. This
review for the solution phase, and a final report, a
study focuses mainly on addressing difficulties in
presentation, and a peer review for implementation
educational settings in the real world and thus
and process management phases.
proposes the PBER to provide specific performance
objectives and criteria in conjunction with specific Acknowledgements—This research was supported by the Basic
evaluation guidelines. The results suggests that the Science Research Program of the National Research Foundation
PBER’s clear specification for performance and (NRF) of Korea funded by the Ministry of Education (NRF-
2013R1A1A2063928) and by the INHA UNIVERSITY
evaluation guidelines in each engineering design Research Grant.
process allow instructors to not only evaluate stu-
dents’ engineering design processes and outcomes
based on objective criteria but also observe a References
secondary benefit of spending less time providing 1. D. Davis, K. Gentili, M. Trevisan and D. Calkins, Engineer-
feedback or answers. In addition, students are ing Design Assessment Processes and Scoring Scales for
Program Improvement and Accountability, Journal of Engi-
clearly aware of their goals in each engineering neering Education, 92(2), 2002, pp. 211–221.
design process to avoid any mistakes in skipping 2. A. Bright and C. L. Dym, General Engineering at Harvey
essential activities and thus can invest more time in Mudd: 1957–2003, In Proceedings, ASEE Conference and
Exhibition, Session 1471, 2004.
learning to enhance their task performance. 3. C. L. Dym and L. Little, Engineering Design: A Project-Based
Introduction, 2nd edn, John Wiley, New York, 2003.
4. D. Knight, L. Carlson and J. Sullivan, Improving engineer-
5.3 Limitation and future directions ing student retention through hands-on, team-based, first-
The participants were from two classes and divided year design projects, In Proceedings of the 31st International
Conference on Research in Engineering Education, 2007, pp.
into 10 teams. Therefore, the effectiveness of the 1–13.
PBER was measured based on perceived engineer- 5. R. M. Marra, B. Palmer and T. A. Litzinger, The Effects of a
ing design skills, perceived student satisfaction, and First-Year Engineering Design Course on Student Intellec-
tual Development as Measured by the Perry Scheme, Journal
perceived learning effectiveness for the PBER. of Engineering Education, 89(1), 2000, pp. 39–45.
Future research should extend this study by analyz- 6. E. M. Taurasi, Forging a path for today’s engineers, Design
ing the effects of the PBER on students’ engineering News, 62(11), 2007, pp. 62–66.
7. C. J. Atman and K. M. Bursic, Verbal protocol analysis as a
design output. In addition, it should be useful to method to document engineering student design processes,
investigate the effects of students’ self-evaluation on Journal of Engineering Education, 87(2), 1998, pp. 121–131.
A Performance-Based Evaluation Rubric for Assessing and Enhancing Engineering Design Skills 1017

8. C. Charyton and J. A. Merrill, Assessing general creativity Courses, In Proceedings of ASEE Conference & Exposition,
and creative engineering design in first year engineering 2004, pp. 753–766.
students, Journal of Engineering Education, 98(2), 2009, pp. 31. I. Moazzen, T. Hansen, M. Miller, P. Wild, A. Hadwin and
145–156. L. Jackson. Literature Review on Engineering Design
9. B. M. Olds and M. J. Pavelich, A portfolio-based assessment Assessment Tools, In Proceedings of the Canadian Engineer-
program, In Proceedings of the 1996 American Society for ing Education Association, 2013.
Engineering Education Annual Conference and Exposition, 32. C. J. Atman, D. Kilgore and A. McKenna, Characterizing
1996. Design Learning: A Mixed-Methods Study of Engineering
10. R. P. Smith and P. Tjandra, Experimental observation of Designers’ Use of Language, Journal of Engineering Educa-
iteration in engineering design, Research in Engineering tion, 97(3), 2008, pp. 309–326.
Design, 10(2), 1998, pp. 107–117. 33. L. L. Bogusch, J. Turns and C. J. Atman, Engineering design
11. C. L. Dym, Engineering design: a synthesis of views, Cam- factors: How broadly do students define problems?, In
bridge University Press, 1994. ASEE/IEEE Frontiers in Education Conference S3A-7, 2000.
12. ABET (Accreditation Board for Engineering and Technol- 34. J. E. Sims-Knight, R. L. Upchurch and P. Fortier, ASSES-
ogy), ABET engineering criteria 2000, 1997. SING STUDENTS KNOWLEDGE OF DESIGN PRO-
13. C. L. Dym, A. M. Agogino, O. Eris, D. D. Frey and L. J. CESS IN A DESIGN TASK, In Frontiers in Education
Leifer, Engineering design thinking, teaching and learning, Conference, 35(3), STIPES, 2005, p. S2J.
Journal of Engineering Education, 94(1), 2005, pp. 103–120. 35. J. Turns, C. J. Atman and R. Adams, Concept maps for
14. S. Sheppard and R. Jenison, Examples of Freshman Design engineering education: A cognitively motivated tool sup-
Education, International Journal of Engineering Education, porting varied assessment functions, Education, IEEE Trans-
13(4), 1997, 248–261. actions on, 43(2), 2000, pp. 164–173.
15. Y. Kim and B. Kang, Personal characteristics and design- 36. B. Frank, D. Strong, J. Boudreau and A. Pap, Design skill
related performances in a creative engineering design course, assessment from pre-university to third year, In CDEN/
In Proceedings of the 6th Asian Design Conference, Tsukuba, C2E2, 2009.
Japan, 2003. 37. K. L. Gentili, J. F. McCauley, R. K. Christianson, D. C.
16. Y. Lee and S. Jin, Rolling Discussion Technique for Facil- Davis, M. S. Trevisan, D. Calkins, and M. D. Cook,
itating Collaborative Engineering Design Activities, Inter- Assessing students design capabilities in an introductory
national Journal of Engineering Education, 30(2), 2014, pp. design class, In Frontiers in Education Conference, FIE’99,
449–457. 29th Annual, 3, 1999, pp. 13B1–8.
17. T. J. Howard, S. J. Culley and E. Dekoninck, Describing the 38. G. Okudan, M. Ogot, S. Zappe and S. Gupta, Assessment of
creative design process by the integration of engineering learning and its retention in the engineering design class-
design and cognitive psychology literature, Design Studies, room, In American Society for Engineering Education Annual
29(2), 2008, pp. 160–180. Conference, 2007.
18. G. Kaufmann, Problem solving and creativity, In K. Grøn- 39. M. W. Tracey and R. C. Richey, ID model construction and
haug and G. Kaufmann (Eds.), Innovation: A cross-disciplin- validation: A multiple intelligences case, Educational Tech-
ary perspective (pp. 87–137), Norwegian University Press, nology Research and Development, 55(4), 2007, pp. 369–390.
Oslo, Norway, 1988. 40. D. M. Rubio, M. Berg-Weger, S. S. Tebb, E. S. Lee and S.
19. R. E. Mayer, Thinking, problem solving, cognition, W. H. Rauch, Objectifying content validity: Conducting a content
Freeman and Company, New York, 1983. validity study in social work research, Social work research,
20. N. Cross, Engineering design methods: strategies for product 27(2), 2003, pp. 94–104.
design, John Wiley & Sons, 2008. 41. S. H. Jin and S. Shin, The effect of teacher feedback to
21. E. De Graaf and A. Kolmos, Characteristics of problem- students’ question-asking in large-sized engineering classes:
based learning, International Journal of Engineering Educa- A perspective of instructional effectiveness and efficiency,
tion, 19(5), 2003, pp. 657–662. The Asia-Pacific Education Researcher, 21(3), 2012, pp. 497–
22. Design Council, The double diamond design process model, 506.
Design Council, 2005. 42. J. B. Arbaugh, Virtual classroom characteristics and student
23. M. Ray, Elements of design engineering, Prentice-Hall Inter- satisfaction with internet-based MBA courses, Journal of
national, UK, 1985. Management Education, 24 (1), 2000, pp. 32–54.
24. B. Wilpert, Psychology and human factors engineering, 43. J. Cohen, Statistical power analysis for the behavioral
Cognition, Technology & Work, 10(1), 2008, pp. 15–21. sciences, Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.
25. S. Jin and T. Kim, A Research Review on Major Variables in 44. S. Alexander and J. G. Hedberg, Evaluating technology-
PBL Designs of Engineering Courses, Educational Technol- based learning: Which model?, In Proceedings of the IFIP
ogy International, 12(1), pp. 95–124. TC3/WG3. 2 working conference on the Seign, implementation
26. B. Shneiderman, Creativity support tools: Accelerating dis- and valuation of interactive multimedia in university settings:
covery and innovation, Communications of the AMC, 50(12), Designing for change in teaching and learning, Elsevier Science
2007, pp. 20–32. Inc., 1994, pp. 33–244.
27. G. Pahl and W. Beitz, Engineering design, The Design 45. Y. M. Reddy and H. Andrade, A review of rubric use in
Council, London, 1984. higher education, Assessment & Evaluation in Higher Educa-
28. C. Charyton, R. Jagacinski, J. Merrill, W. Clifton and W. tion, 35(4), 2010, pp. 435–448.
Dedios, Assessing creativity specific to engineering with the 46. N. Dabbagh, Scaffolding: An Important Teacher Compe-
revised creative engineering design assessment, Journal of tency in Online Learning, TechTrends, 47(2), 2003, pp. 39–
Engineering Education, 100(4), 2011, pp. 778–799. 44.
29. Intellectual Property Office, What is a Patent?, The Patent 47. M. J. Hannafin, K. M. Hannafin, S. M. Land and K. Oliver,
Office, 2007. Grounded Practice and the Design of Constructivist Learn-
30. R. Bailey, Z. Szabo and D. Sabers, Assessing Student ing Environments, Educational Technology Research and
Learning about Engineering Design in Project-Based Development, 45(3), 1997, pp. 101–117.
1018 Sung-Hee Jin et al.

Appendix 1. The Performance-based evaluation rubric

[Problem phase] Learning Objectives: Students can define problems in detail with design requirements and constraints by identifying
problems and analyzing related information.

Problem recognition Performance criteria: Students can set open-ended, challenging, and impactive design problems and analyze
diverse information.

Scales Poor Marginal Satisfactory Excellent

1 2 3 4

Problem finding Students found problems Students found an Students found an Students found a problem
based on the experience of inconvenient problem in inconvenient problem in valuable for improving
team members. daily life by using daily life by using daily life by using
observations, interviews, observations, interviews, observations, interviews,
experiences, or surveys. experiences, or surveys experiences, surveys, and
under a plan. some other methods
under a systematic plan.

Open-ended, challenging, Students set problems Students set a problem Students set problems Students set a problem
and impactive problems whose level is below their whose level is below their matching their capability whose level was above
capability and whose capability and which may and also had many their capability and which
solution is self-evident. not have many different different solutions whose may had many different
solutions. effects were expected but useful solutions.
temporarily.

Information investigation Students did not survey Students surveyed and Students carried out a Students carried out a
and analyze the set analyzed only accessible demand survey with demand survey with
problem and related information through accessible users (friends target users, collected
information. websites related to the set or colleagues) and information (prior
problem. surveyed part of technologies and
information related to the products), and surveyed
set problem. and analyzed diverse
information through
interviews with related
experts.

Problem definition Performance criteria: Students can define design problems, including design requirements and constraints.

Gap analysis The current state and The current state and The current state and The current state and
desirable state were not desirable state were desirable state were desirable state were
defined. defined, but the reason for defined, and the reason defined, and the reason
the gap was neither for the gap was partially for the gap was analyzed
presented nor clear. presented. from various
perspectives.

Definition The problem was not The problem was defined The problem was clearly Information on the
defined. but vague and unclear. presented, but operating zone and time
information on the of the problem was clearly
operating zone or time of defined.
the problem was not
included.

Requirements and Design requirements and Only design requirements Design requirements and Design requirements and
constraints constraints were not were presented. constraints were defined constraints (e.g., the
presented. at a proper level. economy, society, the
environment, ethics, and
standards) were specific
and defined
comprehensively.

[Solution phase] Learning objectives: Students can develop several solutions for design problems by using creative idea generation methods
and select optimal solutions.

Idea generation Performance criteria: Students can develop several possible solutions for design problems by applying creative
idea generation methods with team members.

Climate and attitudes The team atmosphere was Students respected other Students were willing to Students suggested their
critical and stifling, and members’ opinions but suggest their own creative own creative ideas
students did not did not suggest their own ideas and also encouraged actively and also
participate actively. ideas. and supported other motivated and stimulated
students’ opinions. other students’ opinions.
A Performance-Based Evaluation Rubric for Assessing and Enhancing Engineering Design Skills 1019

Scales Poor Marginal Satisfactory Excellent

1 2 3 4

Creative methods Students could generate Students generated ideas Students generated ideas Students generated ideas
their ideas only by team for solutions by using 23 for solutions by using 23 for solutions by using
members’ brainstorming. creative thinking creative thinking various creative thinking
methods, but the process methods, and the methods, and the process
was unclear. deduction process was was clearly stated.
clearly stated.

Quantity Students could not Students generated ideas Students generated an Students generated a
generate ideas for for solutions partially. optimal number of ideas sufficient number of ideas
solutions. for solutions. for solutions from diverse
viewpoints.

Optimal solution selection Performance criteria: Students can select optimal solutions from several possible solutions based on originality
and applicability.

Idea evaluation Students did not evaluate Students selected an Students selected an Students selected an
generated ideas for optimal solution through optimal solution by optimal solution by
solutions. team discussions. evaluating generated evaluating generated
ideas based on evaluation ideas in conductions with
indices predetermined by outsiders (experts and
team members. users) based on
evaluation indices
predetermined by team
members.

Originality The selected solution was The selected solution was The selected solution was The selected solution was
plain and simple. interesting. unique and different. insightful or innovative.

Applicability (useful) The selected solution was The selected solution was The selected solution was The selected solution was
not useful. somewhat useful. very useful. indispensable.

[Implementation phase] Learning Objectives: Students can prepare reports and present them by using the whole design process and results
after giving some shape to solutions.

Solution improvement Performance criteria: Students can elaborate selected optimal solutions with drawings or prototypes and
examine whether they reflect design requirements and constraints.

Prototype The solution did not Part of the final solution The final solution was The final solution
materialize. was materialized by approximately materialized exquisitely
drawings or prototypes. materialized by drawings by drawings or
or prototypes. prototypes.

Correctness There were many logical There were serious flaws The final solution and There was no logical or
and technical problems in in the final solution and prototype looked technical problem in the
the final solution and prototype. generally correct, but final solution and
prototype. there were some prototype.
unimportant flaws.

Validation The question of whether The final solution and The question of whether The question of whether
the final solution and prototype examined the final solution and the final solution and
prototype reflected design either design prototype satisfied design prototype satisfied design
requirements and requirements or requirements and requirements and
constraints was not constraints. reflected all constraints reflected all constraints
addressed. was addressed. was carefully addressed.

Presentation and Performance criteria: Students can effectively deliver engineering design processes and results with writings
reporting and words.

Structure and fidelity The system of the The content of each label The basic system of the The system for the
engineering design report was written in detail, but engineering design report engineering design report
was unclear, and the logic the system of the whole was prepared, but was made, and evidential
was lacking in its content. report was insufficient. evidential materials for materials for content at
content for each label each level were described
were not presented. faithfully.

Speaking Students just read the Students tried to present Students could present Students could present
presentation material their materials after being their materials by using their materials effectively
without understanding fully aware of the easy terms for the within the allowed time
the engineering design engineering design engineering design by using easy terms for
process of their own team, process, but their voice process, and their voice the engineering design
and therefore the delivery tone, eye contact, and tone, eye contact, and process, and their natural
of their content was very gesture were not natural. gesture were natural. voice tone, eye contact,
poor. However, they could not and gesture were
control the presentation appropriate.
time.
 1020 Sung-Hee Jin et al.

Scales Poor Marginal Satisfactory Excellent

1 2 3 4

listening and responding Students’ manner was Students’ manner was Students listened to other Students listened to other
bad in listening to other bad in listening to other teams’ presentations and teams’ presentations and
teams’ presentation, and teams’ presentations, but questions, but they could questions and answered
they could not answer they tried to answer not answer questions questions correctly.
questions. questions. sufficiently.

[Process management] Learning objectives: Students can manage the design process systematically through collaboration with team
members.

Design process Performance criteria: Students can finish their design activities within the time appointed by managing the
management design process through team activities.

Management It was not possible to Parts of content for the Students set and carried Students set and carried
determine whether prior planning of the out prior planning and out phased prior planning
students set prior design process and parts left records for results. and left records for
planning, and there was of records for results were results. They periodically
no record for results. left. examined whether each
phased goal was achieved.

Teamwork Team members were not Team members were Team members tried to Team members did their
aware of performance aware of performance carry out their own duties best to satisfy
criteria for each design criteria, but each focused by collaborating with performance criteria for
process and barely only on his or her own others. each design process by
participated in design duties. taking responsibility for
activities. the engineering design
project and by
collaborating with others.

Completeness / Students did not submit a Students completed the Students submitted Students submitted
punctuality report. report generally but did reports by the deadline, reports within the
not submit it by the but their completeness deadline, and they were
deadline. was somewhat sufficient.
insufficient.

Sung-Hee Jin is an assistant professor at the Academy of Convergence Education, Inha University, South Korea. She is
involved in conducting and coordinating engineering education innovation projects and programs for engineering
education accreditation. She earned her Ph.D. degree in education (educational technology), and her current research
interests include teaching strategies for engineering expository lectures, design thinking, and creative problem solving.

Ki-Il Song is an assistant professor in the Department of Civil Engineering, Inha University, South Korea. He obtained his
Ph.D. degree at the Korea Advanced Institute of Science and Technology in 2009. He was an assistant professor at the
University of Nottingham Malaysia Campus before joining Inha University in 2012. His current research interests include
underground spaces, nondestructive techniques, and energy geotechnology.

Do Hyoung Shin is an assistant professor in the Department of Civil Engineering, Inha University, South Korea. He
received his M.S. and Ph.D. degrees at Purdue University. He worked as a continuing lecturer at Purdue University and as
a senior researcher at HanmiParsons before joining Inha University in 2009. His current research interests include big data
for urban infrastructure and augmented reality in construction.

Soobong Shin is a professor in the Department of Civil Engineering, Inha University, South Korea. He received his Ph.D.
degree at the University of Illinois at Urbana-Champaign in 1994. He started his academic career at Dong-A University
and moved to Inha University in 2002. His current research interests include structural health monitoring and
infrastructure maintenance.

View publication stats