Sparking self-sustained learning: report on a design experiment to build technological fluency and bridge divides

In this article we report assessment results from two studies in an ongoing design experiment intended to provide a single school system with a sequence of secondary school level (ages 14–18) computer technology courses. In our first study, we share data on students’ learning as a function of the required introductory course and their pre-course history of technological experience. In order to go beyond traditional assessments of learning we assessed two aspects of students’ “learning ecologies”: their use of a variety of learning resources and the extent to which they share their knowledge about technology with others. In our second study we present patterns of course taking by male and female students who have almost completed their secondary schooling. In addition, we share case studies of students who elected to take more technology classes and leveraged their course experiences for internships, further education, and jobs. The quantitative and qualitative data are consistent with our hypothesis that students would become more technologically fluent and that their learning ecologies would diversify as a result of their project-based experiences.     

Investigating the feasibility of using digital representations of work for performance assessment in engineering

This paper reports on the results of a 3-year study conducted at the Centre for Schooling and Learning Technologies (CSaLT) at Edith Cowan University in collaboration with the Curriculum Council of Western Australia which concerns the potential to use digital technologies to represent the output from assessment tasks in the senior secondary course, Engineering Studies. The general aim of this study is to explore the potential of various digitally-based forms for external assessment for senior secondary courses in terms of manageability, cost, validity and reliability. The problem being addressed was the need to provide students with assessment opportunities in new courses, that are on one hand authentic, where many outcomes do not lend themselves to being assessed using pen and paper over a three hour period, while on the other hand being able to be reliably and manageably assessed by external examiners. That is, the external assessment for a course needs to accurately and reliably assess the outcomes without a huge increase in the cost of assessment. A computer managed examination was designed that consisted of a design task that was broken down into a number of timed activities. Students were paced through each activity, recording their input in the form of a portfolio. Input consisted of text, graphics through a camera, video and voice. The exam outputs were uploaded to a online repository. The students’ work was marked by external assessors using a standards based rubric that allowed the students work to be ranked though Rasch Modelling.     

Implementing an exemplar-based approach in an interaction design subject: enhancing students’ awareness of the need to be creative

In higher education effective teaching includes making learning goals and standards clear to students. In architecture and design education in particular, goals and standards around assessment are often not well articulated. There is good evidence that when teachers engage students before an assessment in marking exemplars, and explain why the exemplars received the grades they did, students’ performance in their written assessment is consequently enhanced. However some teachers are concerned that exemplars may discourage students from being creative; this concern is particularly important in design education. In this small-scale mixed methods study we explore interaction design students’ perceptions of the effect of an exemplar-based teaching approach on their work in a design task. Our quantitative and qualitative results show that students developed their understanding of task criteria and standards; far from discouraging their creativity, most students perceived that their experience of the approach enhanced their awareness of the need to produce an original design. The exemplar-based approach used in this study complements the studio-based teaching approach in design education, and helps to make clear the design goals and standards that teachers expect their students to strive for.     

Progression in Technology Education in New Zealand: Components of Practice as a Way Forward

Understanding and undertaking technological practice is fundamental to student learning in technology education in New Zealand, and the enhancement of student technological literacy. The implementation of technology into New Zealand’s core curriculum has reached the stage where it has become critical that learning programmes are based on student progression to allow for a seamless education in technology from early primary to senior secondary. For this to occur, teachers and students need to focus learning on key features of technology education. This paper is based on research initiated in 2001 which explored the nature of progression of student learning in technology. It draws on findings from research undertaken in New Zealand classrooms in 1999–2000 that resulted in the development of the technology assessment framework (TAF), (as reported in detail Compton & Harwood 2003). The 1999–2001 research was funded by the New Zealand Ministry of Education. Findings from the 2001 research allowed for the identification of key features of technology education that are relevant across all age groups, contexts and technological areas. These key features were collectively termed components of practice. The three components of practice established to date are brief development, planning for practice, and outcome development and evaluation. This paper discusses the development of progression matrices for each of these and provides illustrative examples of student work levelled against the matrix indicators of progression for brief development.     

Swedish technology teachers’ views on assessing student understandings of technological systems

Technology education is a new school subject in comparison with other subjects within the Swedish compulsory school system. Research in technology education shows that technology teachers lack experience of and support for assessment in comparison with the long-term experiences that other teachers use in their subjects. This becomes especially apparent when technology teachers assess students’ knowledge in and about technological systems. This study thematically analysed the assessment views of eleven technology teachers in a Swedish context. Through the use of in-depth semi-structured qualitative interviews, their elaborated thoughts on assessing knowledge about technological systems within the technology subject (for ages 13–16) were analysed. The aim was to describe the teachers’ assessment views in terms of types of knowledge, and essential knowledge in relation to a progression from basic to advanced understanding of technological systems. The results showed three main themes that the interviewed teachers said they consider when performing their assessment of technological systems; understanding (a) a system’s structure, (b) its relations outside the system boundary and (c) its historical context and technological change. Each theme included several underlying items that the teachers said they use in a progressive manner when they assess their students’ basic, intermediate and advanced level of understanding technological systems. In conclusion, the results suggest that the analysed themes can provide a basis for further discussion about defining a progression for assessing students’ understanding about technological systems. However, the findings also need to be examined critically as the interviewed teachers’ views on required assessment levels showed an imbalance; few students were said to reach beyond the basic level, but at the same time most assessment items lay on the intermediate and advanced levels.     

Using software to engage design students in academic writing

First year undergraduate design students have found difficulties in realising the standards expected for academic writing at university level. An assessment initiative was used to engage students with criteria and standards for a core interdisciplinary design subject notable for its demanding assessment of academic writing. The same graduate attribute categories linked to assessment criteria and web-based software (REVIEW?) were used for assessing students’ other design assignments. Students engaged with criterion-referenced assessment of an essay exemplar in order to reflect on their own essay writing process. Tutor marking of the exemplar and student essays used a visual mark on a grading scale to reveal the variation between the tutor’s marks and students’ own judgments against each criterion. Data from the software and post-semester focus group discussions and questionnaires showed that the initiative promoted engagement and dialogue between tutors and students and fostered independence and confidence. Results suggest that students’ understanding of the required academic writing standards was improved by this reflective intervention and increased their appreciation that writing and research skills are important attributes for designers.     

Designers as Teachers and Learners: Transferring Workplace Design Practice into Educational Settings

The nature of the design process and how to develop this skill in novice designers has been of considerable interest to technology educators. The relationship between workplace and school-based design is one area in which a need for further research has been identified by Hill and Anning (2001, International Journal of Technology and Design Education 11, 111–136). The research project described in this article had two aims. The first was to compare the workplace practice of six experienced designers with their practice when working on a technological assessment task as part of a pre-service teacher education programme. The second aim was to investigate their experience on teaching practicum in developing design skills with secondary school students. Findings from the research are described and the paper concludes with a discussion of the implications for teaching technology education.     

Assessment is for learning: supporting feedback

This paper describes an action research, school situated project conducted with partnership funding from Learning and Teaching Scotland, Scottish Qualifications Authority and Becta, the UK government’s agency for communications technology in education. Based on e-scape (e-solutions for creative assessment in portfolio environments), developed by Goldsmiths, University of London, the Scottish project focussed on integrating innovative methods of capturing evidence of creative performance with providing formative feedback to learners. Classroom trials were conducted with Primary 7 through to Secondary 3 learners (10–15 year olds) in 2 different local education authorities. Learner and teacher thoughts were recorded through blogs, e-forums and interviews; the authored design challenges were shared through the web-based e-scape authoring system and e-folios reviewed by participant teachers and researchers. This paper provides a summary of the reactions and responses from teacher practitioner, learner and researcher perspectives based on their experiences and the results of the classroom trials. It discusses the potential contribution in terms of supporting learning, teaching and assessment within the framework of Scotland’s Curriculum for Excellence (2004) Technologies (2009) learning area. It is hoped that e-scape scotland supports pedagogies which enable the capture of creative thinking in real time for authentic and formative assessment and addresses some of the issues for classroom practice and practitioners.     

Evaluating students with online testing modules in engineering economics: A comparision of student performance with online testing and with traditional assessments

Abstract

Engineering economics courses often require students to take time-constrained, in-class exams in which they solve problems by hand, possibly referring to interest rate tables. Many students rely on partial credit to successfully pass exams. Outside of the classroom, professionals rely on computers to solve engineering economics problems, which raises the question of whether engineering economics courses are correctly assessing student performance. This article describes the study of a large engineering economics class using a non-conventional testing method. Student performance was evaluated using online testing modules with a stringent passing criterion, and the tests could be taken multiple times. The questions for each testing attempt were pulled from a database so that students received a new question every time. We compare the performance of students who were assessed using traditional methods with the performance of students assessed with these online testing modules. Our analysis shows that, overall, students who were assessed using the online testing modules earned better grades than students who were assessed via traditional methods. The analysis also discusses several benefits and drawbacks to using online assessments compared with traditional methods. The online assessment method could be useful in large engineering courses that are formula-based.     

Assessment for Learning: Using Solo Taxonomy to Measure Design Performance of Design & Technology Students

One objective of Design &; Technology (D&;T) is to enable students to be inventive in designing practical solutions to problems. D&;T is viewed as being successful when students can demonstrate the ability to recognize problems, undertake inquiries by themselves, and contribute ideas accordingly. This article will discuss a study which investigated an alternative approach to assessing students’ design performances. In the study, a new item format was designed and a new criterion framework of assessment based on Biggs’ SOLO Taxonomy was developed. The evidence from this study indicates strong face validity for the new approach which maps closely to the goals and purposes of learning D&;T.     

Evolving project e-scape for national assessment

In the opening paper in this Special Edition I outlined the major issues that led to the establishment of project e-scape. The project was intended to develop systems and approaches that enabled learners to build real-time web-based portfolios of their performance (initially) in design & technology and additionally to build systems and approaches to facilitate the web-based assessment of those portfolios. The project was commissioned by the Qualifications and Curriculum Authority (QCA) with additional ‘buy-in’ from Awarding Bodies—who were seen by QCA as the leading beneficiaries of a successful project. The project was designed in three phases. I have outlined—in the Introduction to this Special Edition—the early exploratory work that we undertook within phase 1, the aim of which was to prove the viability of the concept. This was achieved, and QCA then commissioned phase 2 with a brief to build a working prototype system and run it through a national pilot-testing programme in 2006. Age 15 was the target age-group, aligning as closely as we could with the Awarding Body requirements for the General Certificate of Secondary Education (GCSE) that runs with age 16 learners. The successes of the phase 2 prototype—both as classroom activity and as reliable assessment—led QCA and Becta (the body responsible for funding ICT developments in schools) to commission phase 3 in which we explored the potential of the e-scape system for wider application. Specifically, we were required to demonstrate the transferability of the system to other curriculum areas beyond design & technology, and the scalability of the system if it were to be used for national assessment purposes, with hundreds of thousands of candidates. In this paper, I outline the approach that we adopted through the e-scape research; describe the major elements of the work both in terms of classroom/curriculum practice and in terms of new approaches to assessment; and analyse some of the key issues that arise from it.     

Sustainability awareness in engineering curriculum through a proposed teaching and assessment framework

Engineers play a vital role in enabling the sustainable development of their societies. Thus, it is necessary to teach sustainability in universities, especially for engineering students, for a more sustainable future. The present work aims to measure and enhance engineering students’ awareness of sustainability by monitoring and evaluating learning and student outcomes during the last stages of an engineering curriculum and its culminating major design experience. A questionnaire to assess students’ awareness of sustainability and its tools is implemented. From this empirical study, obstacles to the awareness of sustainability through engineering education are discussed. The analysis of the results of the questionnaire implies the modification of the structure of senior projects and other capstone design courses in order to include a proposed teaching and assessment framework. This framework emphasizes students’ ability to understand and use effective sustainability tools, within the environment of their major design experience. The framework is composed of two paths. The first path concerns course design through which sustainability concepts and methods are included in several course elements. The second path addresses the assessment of the attainment of sustainability objectives by the students.

     

The clock project: gears as visual-tangible representations for mathematical concepts

As we have noticed from our own classroom experiences, children often find it difficult to identify the adequate operations learned in mathematics class when they are solving mechanical-operators problems in Technology class. We wanted to design a project that exploits the idea of a hands-on relationship between mathematics and technology to teach students the concept of ‘transmission coefficient’ in schools. Our purpose was to bridge mathematical knowledge and mechanical parts in technological devices. Our belief was that visual-tangible representations enhance human cognition by scaffolding the information process. We supplied a kit with rubber foam (Fomi) and balsa wood parts to construct an analogical clock. This included a series of gears to work with three different ratios. The project was conducted in 2007 and 2008, with 38 students ranging from 10 to 14 years of age. The students were from the 5th and 7th grades in two different schools. The project included six stages of both theory and hands-on work. In a final stage, the students were given a written test. All students were able to make workable systems of their design for a time-measuring device. This proved important for technological education. On the other hand, manipulating gears along with visual reasoning strategies allowed all students to achieve an understanding of operations with fractions. We think this is a major achievement in mathematical education.     

Assessing scientific and technological enquiry skills at age 11 using the e-scape system

This article reports on the outcomes from the e-scape Primary Scientific and Technological Understanding Assessment Project (2009–2010), which aimed to support primary teachers in developing valid portfolio-based tasks to assess pupils’ scientific and technological enquiry skills at age 11. This was part of the wider ‘e-scape’ project (2003-present), which has developed an innovative controlled alternative to design & technology and science public assessment at age 16. Teachers from eight primary schools were trained in the use of an online task-authoring tool to develop and trial assessment activities based on current classroom work. To compile their e-portfolios of assessment evidence, pupils used netbook devices, which afford multi-modal responses (text, drawing, photo, audio, video, spreadsheet) whilst leaving space on pupils’ tables for practical investigations. Once the pupil e-portfolios had been uploaded to the secure e-scape website, teachers assessed them using a ‘comparative judgement’ approach to produce a rank order with a high reliability coefficient. Participant teachers recognised the strength of the e-scape approach in terms of facilitating and managing pupils’ responses to assessment tasks in the classroom, which they successfully adapted to suit primary pedagogy. In particular, the benefits of scaffolding complex assessment tasks through the step-wise e-scape process in the authoring tool represented for some of the teachers a pedagogically significant development in terms of their planning.     

Assessment for Learning: Using Solo Taxonomy to Measure Design Performance of Design & Technology Students

One objective of Design & Technology (D&T) is to enable students to be inventive in designing practical solutions to problems. D&T is viewed as being successful when students can demonstrate the ability to recognize problems, undertake inquiries by themselves, and contribute ideas accordingly. This article will discuss a study which investigated an alternative approach to assessing students’ design performances. In the study, a new item format was designed and a new criterion framework of assessment based on Biggs’ SOLO Taxonomy was developed. The evidence from this study indicates strong face validity for the new approach which maps closely to the goals and purposes of learning D&T. This revised version was published online in July 2006 with corrections to the Cover Date.     

Innovation learning in comprehensive education?

The goal of this article is to clarify the concept of innovation and by presenting a research on the basic education outcome assessment data from an innovation learning perspective, answer to a question: Do students learn innovation in comprehensive education? The empirical information in this research is based on data collected in the national assessment of the subject craft, design and technology education (CDT) in Finland in 2010. The comprehensive education in Finland, the basic education, means grades 1–9 in comprehensive schools from age 7 to 16. This assessment included a design task, a test of knowledge and skills and an attitude test in CDT. This research focuses on two central concepts: (1) innovation is defined as a novel, inventive and usable solution, in either material or immaterial space: an end-product, process or method related to people’s practical needs and purposes and (2) innovation learning is defined as a problem based and creative process of using and implementing knowledge and skills in iterative and critical manner in designing and making a novel and practical solution with high usability. The assessment data was marked off to tasks which indicated the innovation learning (n = 661 out of the sample n = 4792). Brim quartiles were used as a methodological solution; the brim quartiles of usability formed the sample of this research. The statistical differences were tested using the Kruskal–Wallis test and the Pearson Chi Square test. Innovation learning includes the process of designing, planning, making and the practical solution itself. The national data allow general conclusions according to the level of innovation learning in comprehensive education. The central observation is that students learn innovation in comprehensive education varying from good to moderate levels. However, if students have not studied design and technology since 7th grade, they are twice as likely to be negative underachievers as to be either positive achievers or positive underachievers. This is useful for governments to know when trying to increase innovation on a national level, as well as when considering the well-being of people and society.     

An analysis of the impact of student–scientist interaction in a technology design activity,using the expectancy-value model of achievement related choice

Many education initiatives in science and technology education aim to create enthusiasm among young people to pursue a career in Science, Technology, Engineering, and Mathematics (STEM). Research suggests that personal interaction between secondary school students and scientists could be a success factor, but there is a need for more in-depth research on the actual effects of science education initiatives. This paper describes an in-depth, qualitative assessment of a technology design activity, using as a theoretical framework the expectancy-value model of academic choice Eccles and Wigfield (Annu Rev Psychol 53:109–132, 2002). A core element in the studied education initiative is the interaction between secondary school students and scientists. Semi-structured interviews were conducted with participating students and analysed qualitatively to disentangle the factors in their motivation to participate in this initiative and their experiences and memories gathered during participation. Last, this paper reflects on the use of the expectancy-value model for in-depth assessments of science education initiatives. Results show that interest-enjoyment values and attainment values are most important in the students’ motivation to participate in the studied activity. These values are connected to educational principles of authentic practice, and of providing meaningful contexts for scientific concepts. Furthermore, results show that the interaction between students and scientists is not automatically a success factor. Disappointment in this interaction, can cast a shadow on students’ whole experience. This leads us to propose to include an additional factor in the expectancy-value model of achievement related choice: educational environment, including ‘personal interaction’ as an element. Adding this factor would—in our opinion—create an even better framework for in-depth assessment of science education initiatives.     

Students’ interest and expectancy for success while engaged in analysis- and creative design activities

Inasmuch as design is a central activity in K-12 engineering education, understanding the students’ motivation during engaging in engineering design activities will help educators to develop and evaluate strategies for engineering design challenges, and improve curriculum. The objective of this study is to better understand the relationship between students’ interest and expectancy for success while engaged in two design activities in grades 9–12. The primary difference between the two activities was the strategy used to solve the design problems from a predictive analysis and a creative approach. Constructs of motivation for students’ interest include task value (TV) and intrinsic goal orientation (IGO) and extrinsic goal orientation (EGO). Expectancy for success includes control of learning beliefs and self-efficacy for learning and performance. In this study, students (n = 31) from three high schools that implement the Project Lead the Way curriculum in three states in the US participated in the study. Immediately after completing their design projects, each student was asked to complete a modified version of the Motivated Strategies for Learning Questionnaire survey instrument which evaluates their interest and expectancy for success. The results show that students were more intrinsically motivated to engage in a design activity that involves a predictive analysis than a creative approach. No significant correlation was found between students’ expectancy for success and EGO in design tasks that utilized either predictive analysis or creative approach. The study also found that TV and IGO were good predictors for students’ expectancy for success. Demographic information associated with students’ motivation in the design activities is also presented.     

A scientific writing pedagogy and mixed methods assessment for engineering education using open-coding and multi-dimensional scaling

Assessments of new pedagogical practices usually rely on instructor oriented surveys and questionnaires to measure student perceptions of teaching methods; however, fixed response categories in structured questionnaires might bias results. This paper demonstrates a mixed methods approach using open and multi-dimensional scaling (MDS) for a student-oriented exploratory analysis and visualization of perceptions of teaching methods. A scientific writing and research methodology course is a required course for first year PhD students in software engineering and geo-infomatics at Wuhan University, China. PhD students attending this course came from countries whose first language is not English, and from a variety of software engineering and geo-infomatics domains. The problem therefore, was to elicit un-mediated perceptions of course assignment, reduce and generalize the resulting data for interpretation. A graphical visualization of themes emerging in student responses to two open-ended questions about an assignment provided a basis for inferring student interests and needs. In this course, an assessment of a task oriented problem-solving experience was implemented through a mixed method strategy incorporating qualitative methods, exploratory data mining techniques, and cartographic visualization. The visualization shows that participating students generally perceived the exercise as challenging, helped them understand journal requirements, and develop ways to survey texts to extract information. The results also suggested that this consensus breaks down in terms of each participant’s own goals, domain, and research interests. Unstructured questions, open coding, and MDS visualization might also prove to be helpful in the process of devising and assessing other student centered pedagogies.     

Spatial ability learning through educational robotics

Several authors insist on the importance of students’ acquisition of spatial abilities and visualization in order to have academic success in areas such as science, technology or engineering. This paper proposes to discuss and analyse the use of educational robotics to develop spatial abilities in 12 year old students. First of all, a course to introduce robotics to 6th grade primary school students was designed. The key intention was to prepare practical and motivating sessions in order to foster the students’ involvement in hands-on learning. Hence, during the sessions of the course, challenges were provided for the students, in order to develop their capabilities as proficient problem solvers. The teacher assisted and guided the students, and the students were encouraged to solve the problems by themselves, working in 3-members teams. The main goal of this paper is to discuss and analyse the potential usefulness of educational robotics to develop spatial abilities. To carry out the analysis, students were randomly divided into an experimental group (EG), which participated in the robotics course, and a control group (CG), which did not take part in the robotics course. The extensive existing literature for spatial ability evaluation was analysed and reviewed and a pre-test and a post-test were prepared for use in the research study. Initially, the spatial ability of both EG and CG students was assessed with the pre-test. Then, after finishing the robotics course, the same sets of students were tested with the post-test. An extensive analysis of the results is provided in the paper. Results show that the positive change in spatial ability of the participants in the robotics course (EG) was greater than change evident in the students who did not join the course (CG). The improvement was statistically significant. The results also show that the overall performance of the students depends on the instruments used to evaluate their spatial abilities. Hence, this study manifests clearly the importance of the selection of those instruments.