Copy the page URI to the clipboard
Hall, Jon. G.; Rapanotti, Lucia; Self, Steven; Slaymaker, Mark and King, David
(2018).
DOI: https://doi.org/10.21125/iceri.2018.0356
URL: https://library.iated.org/view/HALL2018EVA
Abstract
The subject of this paper is a pedagogy adopted on a Masters in Computing offered by the Open University (OU), UK, one of Europe's largest distance education university. The OU has approximately 160,000 enrolled students, over 2 million alumni and a commitment to widening participation and providing lifelong learning with its long-standing mission to be ''open to people, places, methods and ideas, and innovation. Over 75 of OU students are in work and studying to improve their career outcomes. Therefore, our pedagogical imperative was to develop a professionally relevant Masters which married a deep understanding of discipline specific knowledge with a broad set of skills for collaboration and knowledge application across disciplines (so-called T-shaped abilities 4), to enhance our graduates employability outcomes in today's rapidly changing and complex economy. Specifically, in the Masters, we aimed for theory and practice to combine so that all learners develop a deep understanding of the discipline, and a wide range of critical skills, from the ability to interact effectively with others (e.g., communication, collaboration, leadership), to sense making and problem solving in the real world (e.g., reasoning, analysis, synthesis, evaluation, innovation), and to skills to allow personal growth (e.g., reflection, critical thinking, independent learning, self-direction). Moreover, flexibility of study was a must, so that our students can choose their own pathways through the Masters, working at different study intensity, often taking inter-module study breaks on their journey, can exit the qualification at intermediate points, such as certificate and diploma stages, or even just study individual modules for career and professional development, rather than for a degree. To meet both professional relevance and study flexibility, in our Masters we designed three modules with a shared pedagogy which embeds learning and assessment activities within each student's rich and unique context of practice 7, and is packaged as a re-usable framework to be instantiated for teaching of each specific subject: this allows a wide range of transferrable employability and research skills to be contextualised in specific professional knowledge and practice, with repetition increasing competence as students journey from one module to the next, without a need for pre-defined study progression and intensity, which would reduce the flexibility of our offer. The three modules cover software engineering, information security management and data management, respectively. The other modules in the Masters follow a more traditional pedagogy where no contextualisation of learning in the student's professional practice takes place. Students choosing to complete the whole Masters need to complete a capstone research project. In this paper we describe the pedagogy and its initial evaluation. We also discuss open challenges and how we plan to overcome them.
The pedagogy
The pedagogy, developed by the first and second authors, was inspired by the observation by Quine, one of the most influential philosophers of the 20th century, that 9: ''Total science, mathematical and natural and human, is ... underdetermined by experience. The edge of the system must be kept squared with experience; the rest ... has as its objective the simplicity of the laws.'' Quine's observation suggests that generative research in disciplines that span both theory and practice can fruitfully be conducted within a context of practice. Much applied Computing knowledge has developed in practice, and only epistemologically post-rationalised. As such, its theories exhibit many of the issues mentioned above. Our claim is that there is value added for both the practitioner/student and their organisation in allowing them to bring their rich professional context into supportive taught Computing modules, with teaching mechanisms based on the experience of the application of theory in satisfying the student's needs within that rich context. This value arises from, in Quine's words, `keeping the edge of the theory squared with experience': specifically, through the application of their learning to an organisational problem, students deliver into their rich professional context valuable developmental artefacts ranging from a detailed problem understanding through to candidate solutions. Quine's observation does not, however, say how to teach theory within a practical discipline. Issues include: are all parts of the theory relevant? If not, which are relevant and in which order should they be taught? How should theory be assessed from its practical application? What value is delivered if the student's rich context is used? This is the gap fill-in in by our pedagogical framework, which was initially developed for distance research students on professionally related Doctorates 5, then refined for application to individual professionally oriented taught masters modules 6 and finally, as reported here, generalised further for application across a whole Masters programme. Note that our pedagogy, in marrying theory and professional practice, is distinct from methods with similar aims adopted in traditional face to face education, such as problem-based learning 2,3,10, which aims to bring realistic problems to classroom, but, necessarily has no rich professional context of reference, or student placements, which may not have any theory of reference. A key challenge for the pedagogy is therefore how to bring each student's context of practice to bear on their learning, and vice versa allow students to reflect their learning into that context. This brings two further challenges: firstly, to match the module content to the type of problems our students may be facing in their practice; secondly, to ensure equitable assessment, both in terms of the independent work required of each student, the measures applied to its assessment, and the level of effort expected of tutors and examiners to assess it. So, in adopting this pedagogy, educators must organise their teaching to deliver fit-for-purpose conceptual tools before the student needs them, so as to enable the development of understanding of, and ability to, solve real-world problems in a wide range of student contexts. Disciplines with a process basis, like applied Computing, thrive on critical reflective practice, so that theories, principles and techniques are taught for application by a student within their rich real-world context. The student in then required to reflect critically on their experience and to build a commentary on the extent to which the applied theories, principles and techniques have been fruitful, appropriate, deficient, over-complex, or just plain wrong, therein. This theory-supported process application within the student's rich context, followed by guided critical reflection, is the main vehicle for assessment and validation of learning. The guided reflection also includes the rationale for their choices in the process application within their context, and lessons learnt, and a critical reflection on the teaching against real-world practice. From a tutor's perspective, our tutors --- all experienced Computing professionals --- are chosen for their ability to deal with the variance in the student's context, and each is assigned a tutor group of up to 15 students. A generic marking scheme provides consistent marking within and between tutors' student groups, despite the diversity of development work to assess 8. This marries qualitative marking criteria related to the module's learning outcomes with quantitative measures which allow the tutor to assign a final grade, one of fail, pass, merit or distinction. In meeting the requirements of postgraduate learning, the pedagogy also prepares students for independent learning and the application of generic research skills, the latter particularly relevant to the successful completion of the capstone research project. To summarise, the pedagogical framework requires that in each module learning design judicious choices are made to:
* identify, within the body of knowledge of each sub-discipline, content of broad applicability, hence suitable for a wide range of professional problems and contexts;
* enable situated learning, by allowing students to choose within each module their own specific professional problems, directly exercising therein knowledge and skills acquired while studying, and then reflecting on their learning;
* design authentic assessment tasks and rubrics to allow: students to exercise knowledge and demonstrate their skills on meaningful tasks in their own context of practice and reflect on that practice; and tutors to assess each student's individual work based on a set of explicit assessment criteria;
* provide learners with the research skills and opportunities to conduct and share their own independent enquiries into topics relevant to the module and their profession and/or at the leading edge of the subject, but not necessarily addressed within the module.
The evaluation
Our Masters degree was launched in 2013, so that a wide range of data have been collected over the years for evaluation purposes, including standard performance indicators around student retention, attainment and satisfaction, as well as submitted assignments, both formative and summative, and student activities in module online fora. In our evaluation, we were interested in effects of the pedagogy with respect to the development of relevant transferrable skills both in individual modules and cumulatively as students progress through their chosen study pathways, including the capstone project for those choosing to complete the whole Masters. For the evaluation of individual modules, we looked at traditional key performance indicators, and for comparison selected other modules on the course which do not use the same pedagogy. However, evaluating cumulative effects along chosen pathways posed a number of challenges due to the flexible nature of the Masters, specifically the fact that students can choose study pathways and intensity at their own will, and are not compelled to complete the whole qualification. This means there were no clearly identifiable cohorts progressing through it, as is case in traditional full-time courses or even in more traditional part-time courses. Therefore, a first step in the evaluation was to identify those students on pathways which included a combination of two or more of the modules under study, and to follow their progression through those pathways. Among them, for those who had competed the whole Masters, a comparison was also made with students on a different Masters which shares the capstone project module, but none of the modules adopting our pedagogy. Finally, as student numbers on each module were relative high (around 100 per module per year), we were faced with large qualitative data sets, accumulated over the years, particularly from on-line forum activities and assessed student work. This made manual analysis infeasible, so that we investigated a range of Natural Language Processing (NLP) techniques for data extraction and classification to improve the efficiency of subsequent manual analysis, and which we applied to an initial set of forum posts as proof-of-concept. Initial findings Looking at individual modules, attainment data indicate no significance difference in mean values of relevant assessment scores between the modules under study and their comparators. However, the distribution of pass, merit and distinction grades appears different, with the former having a lower percentage of merit grades and a higher percentage of pass grades. Students satisfaction data indicate that the modules under study perform overall at least as well as their comparators. From qualitative feedback in student module survey, which we run every year, there was clear evidence that students found the approach making a positive contribution towards their professional work and career prospects. Out of the hundreds of students on the course since 2013, only 16 have completed the whole Masters. Those students appeared to perform better on the capstone project than students on the comparator Masters, with slightly higher classification grades. NLP-based techniques were developed and applied for the selection and classification of students' forum posts, with a view to collect evidence of study skills development and progression. As proof-of-concept a classifier based on the Bloom's taxonomy of skills was developed and applied, and some evidence of skills development could be found in the data. The main threats to validity of these results is the small number of students having completed the whole Masters to date, and the still tentative and experimental nature of the NLP-based techniques we have applied.
Conclusion and future work
From a knowledge perspective, with the caveats expressed above, our pedagogy appears to be effective in delivering open and flexible Masters education which is relevant to professional practice and has the potential to improve our students' employability and career outcomes. More evaluation is still required, particularly an in-depth analysis of students assignments, which will be the subject of our next research step. From a methodological perspective, the combination of evaluation techniques developed and applied in this project can be seen as a first step towards a framework for pedagogical evaluation in the context of open and flexible distance education, which combines traditional quantitative analysis with novel, partially automated, qualitative analysis. More work is still required both to refine and fully evaluate the effectiveness of the automated techniques. This work is of particular relevance to educators involved in open and flexible learning particularly for a professionally oriented audience, and for programmes delivered through online learning platforms.
References
1 James F Allen, 2003. Natural language processing. (2003): 1218-1222.
2 Lyn Brodie, Hong Zhou, and Anthony Gibbons, 2008. Steps in developing an advanced software engineering course using problem based learning. Engineering education, 3(1): 2--12.
3 James D. Delaney, George G. Mitchell, and Sean Delaney, 2003. Software engineering meets problem-based learning. The Engineers Journal, 57(6).
4 David Guest, 1991. The hunt is on for the Renaissance Man of computing. The Independent (London) 17.
5 Jon G. Hall and Lucia Rapanotti, 2013. Enterprising research skills: academia's changing role. International Journal of Learning and Intellectual Capital 10(1): 1--17.
6 Jon G. Hall and Lucia Rapanotti, 2015, May. Masters-level Software Engineering Education and the Enriched Student Context. In Proceedings of the International Conference on Software Engineering (Joint Software Engineering Education and Training track), Volume 2 (p. 311-314). IEEE Press.
7 Jean Lave and Etienne Wenger, 1991. Situated learning: Legitimate peripheral participation. Cambridge university press.
8 Jon Mueller, 2009. Assessing critical skills. Linworth Books.
9 van Orman Quine, W., 1976. Two dogmas of empiricism. In Can Theories be Refuted? (pp. 41-64). Springer, Dordrecht. (Originally published in The Philosophical Review 60 (1951): 20-43) Text available from http://www.ditext.com/quine/quine.html.
10 Ita Richardson, Louise Reid, Stephen B. Seidman, Bob Pattinson, and Yvonne Delaney, 2011. Educating software engineers of the future: Software quality research through problem-based learning. In Proceedings of Software Engineering Education and Training (CSEE&T), pp. 91--100. IEEE Press, 2011.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)
