Peer Discussion Online:

A study of peer argumentation and its measurement         (OPENERS)

Project: Doctoral Research Program
Keywords: Educational Design, Discussion, Argumentation, CMC
PhD Title: Design, Dialectics and Argumentation
Author: S R McAlister
Institution: The Institute for Educational Technology, The Open University, Milton Keynes, UK
This study is based on Discussion PG4, undertaken 3^rd November 2001, and Discussion UG5, undertaken 28^th-31^st October 2001

Abstract

This research tested an educational design for university level students involved in online peer discussion, with the aim of facilitating an engaging, rational and evaluative discourse. Both structured activities and structured dialogue are used to introduce students to using evidence-based argument and critical thinking, in an active and experiential way. This study analysed online, synchronous discussions by two sets of students on a controversial topic using a mediating computer interface, and four activities per discussion.

Students selected a sentence opener, a starting phrase, for each contribution to the discussion, and completed it in their own words. Short contributions contained one speech act and were pre-classified, according to their opener, with an extremely low error rate for students who were fluent language speakers.

One aim of this study was to examine various methods of dialogue analysis to see how progress towards the research goals could be measured. Counts of ‘speaker intention’ by student were tabulated to characterise student skills and predispositions, together with dialogue moves and other summary measures. A ‘semantic score’ was determined for each contribution, based on dialogue move and qualitative assessment by a tutor. Finally, dialogue transactions, dialogue tactics and issues of measurement are discussed.

1. Introduction

1.1 Background and theory behind the research

Collaborative argumentation provides an approach for introducing university level students to academic thinking and practices, and is particularly well suited to complex and controversial domains. Reasoned, dialectic argument provides a way to deal with uncertainty and multiple viewpoints, and to engage students in a rational and evaluative manner. To date, unstructured peer discussion is generally viewed as ineffective, especially amongst inexperienced students. Increasingly, students’ presence online provides opportunities for peer discussion of course-related material, and this research addresses an educational design that should facilitate effective discussion.

The approach adopted is to provide a mediating discussion tool, which structures the dialogue according to some prescriptive norms of the discourse it is wished to facilitate. The theoretical underpinnings of this approach are found in Vygotskian ideas of internalisation (Vygotsky, 1978) of the norms of discourse within a ‘community of inquiry’ (Lipman, 1991). Practical implementation of these ideas can be found in Wegerif’s ‘Exploratory Talk’ and Lipman’s programme ‘Philosophy for children’. For example, Wegerif (1996) instituted a collaborative (spoken) discourse among year 4 schoolchildren using five simple rules.

The online synchronous dialogue in this research is typed (like Internet Chat), not spoken. Nevertheless, it has more in common with spoken dialogue than written texts, as it is generated in real-time with a minimum of time for reflection and editing. The discussion itself is structured, with activities for familiarisation, group discuss­ion, debating with a partner and summarising the discussion. The design of each activity is based upon groupwork theory (Salmon, 2000) and previous research into peer argumentation using computer mediated tools (Soller & Lesgold, 1999, Veerman, 2000, Quinnard & Baker, 1999).

1.2 Research and study aims

The aim of the research is to facilitate skills of argument development, collaboration, critical thinking and evaluation of arguments and evidence. This critical, evidence-based approach is subsumed by academic discourse, which encourages a rational and evaluative approach to topics. It stresses advancing an argument, reasoning and critical thinking as core skills. Production of evidence is required to support claims, and any such claims should undergo thorough investigation. These mediated discussions can introduce students, in an active and experiential way, to such academic discourse. The expectation is that once students learn how to use these skills, they will be transferable into their studies generally.

In addition to argumentation skills, students need to develop collaborative skills in order to create effective, well functioning, discussion groups. Such collaborative skills are noticing and drawing out less confident students in the conversation, supporting them, and showing understanding for their point of view. Extra skill and care is needed with online communication to ensure the message is perceived correctly. In general, collaborative skills enable students to engage in online dialogue without appearing to bully and threaten other’s self-esteem, or sound moralistic and judgemental. The participation and goodwill stemming from collaborative approaches enhance motivation and predisposition by students to engage in discussion (Zimmer & Alexander 1996, Rogers 1962).

2. Interface Design

2.1 Relay Academic Talk

The computer interface is adapted to the Chat service familiar to many users of the Internet. Each student can type a contribution that is sent to an Internet server, which relays it on to the other students in a private channel (known as a ‘chat room’). The mediating interface is more structured than simple Chat and supports the concept of threaded conversation (contributions related to the same topic are bundled together). Each contribution consists of a sentence opener plus the student’s chosen text to complete the sentence. The sentence openers are chosen from a Talk menu with sub-menus headed by possible intentions of the ‘speaker’. These intentions are Inform, Question, Challenge, Reason, Support, and Maintain. Having formed an intention to contribute the student will bring up the appropriate menu of openers. These openers are designed to channel the student’s thoughts by the process of completing the sentence in a way that fits with the opener. For example, under the Challenge menu the student is offered ‘I disagree because...‘ and in choosing this opener is implicitly prompted to provide the reasoning behind their disagreement by completing the sentence.

Threaded conversation requires each contribution to be a reply to an earlier contribution, except when starting a new thread. In order to make a contribution the student must first select a contribution to reply to, secondly select an opener, and finally complete the opener with their own text. Each opener usually has a small set of openers that have been predetermined as ‘preferred’ responses. In order to indicate this to the student during the selection of an opener, those preferred openers are bolded on the menu. For example, if a student were to respond to a contribution containing an opener ‘Can you elaborate?...’ they would find that ‘Let me elaborate...’ was bolded under the Inform menu (the menu name is bolded too to draw attention to it).

A moderator who welcomes the students as they arrive online manages the discussion activity. The students are held in a ‘lobby area’ using a Chat style interface so that they can chat amongst themselves while waiting for others to join them. The moderator signals the start of the activity by creating a new communal ‘Talk space’ and students can select it from the Window menu. The Talk space will minimally contain the topic they should be discussing as the first contribution, and the interface changes to that in Diagram 1. Several Talk spaces may be run concurrently on the same channel though students are free to move between them and contribute to any.

In practice there are occasional problems that arise with synchronous communication over the Internet. If it is too busy, an ISP or Chat server may not allow a student to logon, or a student may lose their connection during the discussion. If the student manages to reconnect they will have missed some of the discussion and the moderator needs to send a discussion update for them. At the end of the discussion each client interface records the discussion in a session file, since the Internet server keeps no central records and merely relays Chat between the participants.

A transcript file saves each contribution along with meta-level information such as intention, dialogue move, and dialogue move of the contribution that this is a response to. Summary tables are also prepared of intention counts and dialogue move counts. The program does not attempt to interpret the content of the contributions, but relies upon the opener as a guide to the classification of the content, and therefore it requires inspection to verify that this is the case. The next section looks in more detail at the chosen openers and dialogue moves, their derivation for this context, and why they should realise the aims of this research.     

2.2 Rationale for sentence openers and dialogue moves  (OPENERS: LIST)

The original openers set is based on Soller (2001), and earlier work of Soller & Lesgold (1999), who were using a set framed for a collaborative work environment in which colleagues were discussing programming problems using an object-oriented approach. An earlier work using sentence openers for children was Robertson (1998). In Soller (2001), at the design stage of problem decomposition, a class hierarchy of domain objects must be imposed, with responsibilities assigned to each class identified in the problem domain. This provides scope for different views of participants to be proposed and resolved through argumentation, since design problems rarely have a ‘best’ solution.

This original openers set from Soller needed to be modified to generalise it across knowledge domains, and to specifically accomplish the research aims which were to imbue skills of argument development, critical thinking, evaluative reasoning and collaboration. In particular, the openers needed extension to deal with evidence, which is not a major theme of design problems, but is a core component of academic concerns.

The full set of openers and the intention headings is shown in Table 1. The attribution of openers to any one intention set is problematic in some cases. ‘How is that relevant?...’ could be asked as a genuine question with the speaker expecting to be told information. However, in practice so far, it is almost always issued as a challenge when the speaker suspects that the other student has wandered into irrelevant matters (the opener was originally placed under the Question intention). The intention sets Inform, Question, Challenge and Support are naturally derived from argumentative speech acts, but Maintain is not. Maintain contains a mix of simple replies and acknowledgements, together with group actions that are necessary for collaboration. These group actions include collaboratively rewriting statements and suggesting a change in the current group focus. The intention Reason contains a number of indirect dialogue moves such as paraphrasing, qualifying, verifying, synthesising and researching.      .

(MOVES IN CONTEXT)

The sentence openers provide a reasonably complete toolset for discourse within an ‘academic’ style. Though it would always be possible to add more openers, the set needs to be kept to a minimum, but adequate, set for design and practical reasons. Although the intention of the speaker can be used to label each contribution, it is standard practice to assign to each opener a dialogue move, which indicates a specific illocutionary act associated with its use. However, although students are instructed to keep their contributions short and to the point, contributions may contain more than one speech act. One finding of earlier work within this research was a problem with openers designed to perform simple acknowledgement, in that the contribution often contained a different speech act, typically a Counter, as in “Yes… but isn’t it the case that …”. In order to deal with this problem, acknowledgement openers were restricted so that text could not be added, and this is indicated by a full stop at the end of the opener instead of ellipses. The contributor must therefore break up their original contribution into two, e.g. Support and then Counter, or use a direct Challenge. This restriction helps to preserve the one-to-one mapping between opener and the (major) speech act of the contribution.

One simple set of dialogue moves is Inform, Elicit, Disagree, Counter & Qualify. Early works on dialogic argument used a simple set of moves, but later works such as Ravenscroft & Pilkington (2000) have used more complex, multi-move tactics, such as Probe within an expert system tutoring programme. The purpose of assigning dialogue moves is to delineate significant features of the dialogue, and in the context of this research there are additional features to be highlighted. The simple set of dialogue moves has been supplemented with Reason, NewArg, Evidence, Support, Maintain & Action to account for a number of moves that either, do not easily fit into the simple classification, or are significant features in this context.

Support, Maintain and Action are collaborative moves that are unnecessary in a formal argumentative scheme in which silence is regarded as acquiescence (MacKenzie, 1979). Dialogue moves related to the request for, or supply of, evidence are significant since evidence is regarded as a core element of the ‘academic’ style of discussion. NewArg is associated with the opener ‘I think...‘. Although it is an informing move and could be subsumed under Inform, its use is sufficiently frequent, and it’s context of use is interesting enough, for it to be treated separately in this study. This set of dialogue moves is designed to allow analyses of discussion to track the noteworthy features of an ‘academic’ style of discussion.

Finally, the sentence openers may be classified by their likelihood of contributing to the four research aims. The first three aims have little overlap among the openers, so openers may be classified in only one of these, as i) argument development, ii) critical thinking, or iii) collaborative skills. The remaining aim of evaluative reasoning, weighing the evidence and being prepared to consider alternatives, may be achieved through a number of openers. However, it is also the case that any reasoned contribution may be evaluative with respect to argument and evidence, and that it needs to be judged qualitatively by inspection. A schema is developed in the later methodology section that provides a quantitative measure of how well the study aims have been achieved.      

BACK TO TOP 

3.  Methodology

3.1 The discussion activity

Besides the structuring of dialogue that takes place using the interface, it is possible, and desirable, to structure the activities within the discussion to seek an orderly sequence of events in learning terms. For instance, that knowledge refinement activity should follow any knowledge building activity, so that a learning process evolves. Quinnard and Baker (1999) propose a process model for learning that includes individual, group, and then pair working. Each stage of the process model is designed to support the student at their present level of knowledge. The student performs individual preparation with reading materials, which is followed by group discussion. This supports knowledge acquisition followed by knowledge building. When students are familiar with the issues, argumentation between pairs of students with conflicting views can support knowledge refinement.

This three-stage model for learning can be augmented. For collaborative tasks students need to demonstrate a joint product from their efforts, according to earlier research (Veerman 2000, p157). A further activity of jointly producing a summary of their work is required. The activity model adopted comprises the following stages.

1)       individual preparation in the topic domain from papers and articles proposing competing viewpoints and providing evidence

2)       group discussion of the topic

3)       paired debates with each student arguing for, or against, a topic statement

4)       paired summary of the main arguments and counter arguments

Group development is an important process in the collaborative context. One model of group formation focuses on stages of interpersonal relations with ‘Forming, Storming, Norming and Performing’ (Brown 1979). Another model proposes five stages of ability for an online e-group (Salmon 2000). In both development of group relationships and development of e-group skills, the models dictate prerequisite familiarisation, both with the medium, and with other members of the group. This familiarisation process may be quite short for experienced users of technology, like Usenet and Chat room users, but may be prolonged for the inexperienced.

In addition to the four activities in the process model, it is necessary to add in a familiarisation stage, using an icebreaker activity, the length of which will depend upon the group members’ experience. Having structured the activities within the discussion, we now turn to the content of the discussion activities and examine ways to measure what has taken place.

3.2 Transcript counts

Each contribution in the transcript is automatically classified by its sentence opener, on the assumption that the openers are a reliable guide to speaker intention and dialogue move. The transcript generated from a discussion shows first a summary line of meta-level information, followed by the contribution itself. The example below shows the entry number of the contribution, the speaker nickname, the originating entry number (for which this contribution is a reply), the speaker intention, and the pre-classified dialogue move for the meta-line.

“4     S      1      Question    B-Elicit

4    Why do you think that?...aren't a lot of what we call natural foods previous genetic modifications?(S replied to 1 with Question*)”

On the next line the opener and the actual text contribution (in brackets is an automatic comment for the students to determine the context). Preferred responses are recorded (marked *) for each student.

The transcript must be checked for use of appropriate openers throughout or the contributions will be wrongly classified. Occasionally, students choose an opener and develop a contribution, as they type, that no longer matches the opener they selected. They are able to re-choose an appropriate opener before sending, but may forget to do this. Where an inappropriate opener is recorded, the correct intention/move is recorded and the counts are adjusted as appropriate.

Finally, recorded by inspection is the number of contributions that start a new topic, and the number of contributions off-topic, by student. Counts give simple indicators of quantity and type of contribution, and can be used to characterise students within the discussion, particularly when the percentage split of contributions is examined by student. The analysis of student characteristics may be used to answer the following questions. What are the characteristics of students who are proficient in this type of discourse? If these can be identified, can these be used as feedback to students to change their own characteristics? Do these changes imply a real and constructive improvement in students’ discussion abilities? In this paper only the first question may be attempted.

3.3 Semantic scores

In the previous section a quantitative analysis was outlined, where student characteristics could be analysed through counts of the type of contributions. In this section the quality of the contributions is examined by assessing each under the headings of the research goals. A simple qualitative assessment is applied to each contribution, using a score of between one and three for each of the categories.

A qualitative assessment of the discussion should examine those aspects of a discussion that cannot be assessed by simple counts. Counts register that a dialogue move has taken place, but not whether that move is insightful, or banal. Qualitative assessment of contributions uses a tutor’s knowledge of the topic to score contributions on the following basis. Simple use of an opener without additional text scores one, recognising that at least a minimal contribution had been made. A contribution with reasoned text scores three, but a contribution with text of poor quality scores two. In cases where the same information is repeated from a previous contribution, a deduction of one point is made. The points are allocated to one of the three categories, (A) argument development, (B) collaborative working, (C) critical thinking, determined by the context of the associated dialogue move (see section 2.3).

Finally, each contribution is separately judged evaluative or not, and additional points are assigned to the evaluative category (E) in the same way as the other categories. Likely evaluative moves are Reason, Qualify, Counter and Evidence. They provide a good indication that the contribution should score in the evaluative category, as they are in some way concerned with weighing argument and evidence, or providing alternative views. The schema derived here and termed semantic score, while undoubtedly crude, is easy to administer and attempts to move the analysis beyond one dimensional counts to provide an indicator as to whether the research aims are being met. That is, a discussion with a higher semantic score than another will be the more successful discussion, other things being equal, according to the research aims.

3.4 Transactions

In this section extracts of dialogue are examined as a collection of related responses. With transactions we are interested in the interplay between students and what makes successful tactics. In Pilkington (1998, 1999) and Coulthard & Brazil (1992), rhetorical and exchange structure has been used to analyse spoken dialogue. Rhetorical and exchange structure provides a useful framework for analysing discussion, but does not advance the study goals presented here. The reason is that it is concerned with the grammar of exchanges, not with the content of the discourse itself. The grammar of exchanges is useful to this study as it provides basic tools for description of many aspects of discussions, including dialogue moves.

Rhetorical and exchange structure is focused on exchanges, particularly between tutor and student, in which an initiative communication provokes a response. Learning occurs through the social interaction of the individual with others, according to the sociocultural paradigm. Interaction can be viewed in dialogue terms as a series of meaningful dialogue exchanges between individuals. An exchange can be defined as a set of linked dialogue moves, in the sense of responses, in which a significant event in terms of knowledge building or critique occurs, and a transaction is defined as a set of exchanges on the same topic.

This study focuses on transactions rather than exchanges. Exchanges are the smallest unit of interactive dialogue that makes sense on its own. In well-planned tutor-student dialogue, on which much exchange theory is based, this is acceptable. For instance, this can be seen in this example from Pilkington:

Tutor:     “What is the capital of France?”
Student:     “Paris”
Tutor:     “Correct”

This is coded as I,R,Rc - Initiative, Response, Response Complement. In tutor-student dialogue a question from the tutor is often the initiative, and the dialogue is centred around the tutor’s view of the knowledge domain and their chosen path through it. Tutor-student dialogue is therefore more easily classified than peer dialogue, because it is well planned and the tutor holds the initiative. Peer dialogue in this study has important differences that make classification according to this scheme more difficult, as Francis & Hunstun (1992) report.

Use of small exchanges loses much of the semantic value present in a transaction. From the evidence of this study, no one student leads the discussion in a clear direction and controls the initiative, rather each have their turn to attempt to focus the group on their preferred topics. Peer dialogue is rarely authoritative, unlike the tutor-student dialogue of Pilkington (1999), rather a more subtle building of arguments takes place over several exchanges, often taking sharp turns in terms of the focus, but often coming back again to the original topic.

For these reasons complete transactions are presented rather than individual exchanges. The semantic scores attached to transactions provides a reasonably objective method of measuring the success or otherwise of transactions, according to the goals of the research.

<top>

References

Brown, A. (1979) Groupwork. London: Heinemann.

Coulthard, M. & Brazil, D. (1992) 'Exchange structure'. In Coulthard, M. (ed.), Advances in Spoken Discourse Analysis, pp50-78. London: Routledge, ISBN: 0415 066875.

Francis, G. & Hunston, S. (1992) 'Analysing everyday conversation'. In Coulthard, M. (ed.), Advances in Spoken Discourse Analysis, pp123-161. London: Routledge, ISBN: 0415 066875.

Lipman, M. (1991)  Thinking in education. Cambridge University Press.

MacKenzie, J.D. (1979) 'Question-begging in non-cumulative systems' in  J. of Philosophical Logic, 8, 117-133.

Pilkington, R. (1998) 'Dialogues in support of qualitative reasoning'. J. of Computer Assisted Learning, 14, 308-320.

Pilkington, R. (1999) Analysing educational discourse: the DISCOUNT scheme, CBLU Technical Report No. 99/2 Jan 1999. Leeds University ISBN: 1 901418 022.

Quinard, M. and Baker, M. (1999) 'Favouring modellable computer-mediated argumentative dialogue in collaborative problem-solving situations' in Lajoie S.P.& Vivet M. (eds.)  Proceedings of the 9th Int. conference on AI in Education, 129-136. Amsterdam: IOS Press.

Ravenscroft, A. & Pilkington, R.M. (2000) Investigation by Design: Developing Dialogue Models to Support Reasoning and Conceptual Change, International Journal of Artificial Intelligence in Education: Special Issue on Analysing Educational Dialogue Interaction: From Analysis to Models that Support Learning, Vol 11/1, 273-298, ISSN 1560-4292.

Robertson, J., Good, J. & Pain, H. (1998) 'BetterBlether: The design and evaluation of a discussion tool for education' in  Int. J. of AI in Education, vol 9, 219-236.

Rogers, C. R. (1962) ‘The interpersonal relationship: the core of guidance,’ in Stewart, J. (ed.) Bridges Not Walls: A Book About Interpersonal Communication, London, Addison-Wesley (1977),240-8 (reprinted from Harvard Educational Review, Vol. 32, Fall 1962, 416-29).

Salmon, G. (2000) E-moderating: the key to teaching and learning. Kogan Page ISBN: 0749431105.

Soller, A. & Lesgold, A. (1999) 'Analyzing Peer Dialogue from an Active Learning Perspective' in  Towards Models that Support Learning, http://lesgold42.lrdc.pitt.edu/documents/Soller-Lesgold-AI-ED-Workshop.pdf (Oct 2000). Proceedings of the AI-ED 99 Workshop: Analysing Educational Dialogue Interaction.

Soller, A. (2001) The Epsilon Project. The Learning Research and Development Center, University of Pittsburgh at http://lesgold42.lrdc.pitt.edu/EPSILON/Epsilon_research.html (Mar 2001).

Veerman, A. (2000)  Computer-Supported Collaborative Learning Through Argumentation, Published PhD Thesis. NL: University of Utrecht.

Vygotsky, L. (1978)  Mind and society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

Wegerif, R. (1996) 'Using computers to help coach exploratory talk across the curriculum' in  Computers & Education, vol 26/1-3, 51-60.

Zimmer, R. S. & Alexander, G. R. (1996) 'The Rogerian interface: For open, warm empathy in computer-mediated collaborative learning' Innovations in Education and Training International, vol 33 (1) 13-21.

Appendix A: Student questionnaire for feedback