Cohesion and Learning in a Tutorial Spoken Dialog System

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Less significant on testing data, “token with stops” level reduced to a trend

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Less significant on testing data, “token with stops” level reduced to a trend

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655

Results: Aptitude/Treatment (2003 data)

• No significant difference
• between amounts of
• (turn normalized) cohesion
• in high and low pre-test
• groups.
• Difference in correlation between high and low pre-testers not due to different amounts of cohesion.

Results: Interactivity (2003)

• Cohesion between tutor utterances is not correlated with learning

Results: Interactivity (2003)

• No evidence that cohesion between student productions is correlated with learning (but student utterances are very short with computer tutor)

Discussion

• Both high and low pre-testers successfully learned from these dialogs
• Our measure of lexical cohesion seems to reflect only what the low pre-testers do to learn, not correlated with what high pre-testers do.
• McNamara & Kintsch also found a positive correlation for low pre-testers, but a negative correlation for high pre-testers.

Discussion

• Our measures are slightly different:
• McNamara & Kintsch: Manipulated coherence in text
• Reader does not contribute to coherence
• Coherence is the extent to which semantic relations are spelled out in the text, rather than inferred by the reader.
• Low pre-testers probably learned because high coherence text allowed them to make inferences they couldn’t from the low cohesion text.
• Low pre-testers & low coherence: didn’t know the terms
• High coherence may allow a greater number of successful inferences for their low pre-testers
• Our work: Dialog
• Student does contribute to cohesion
• Higher cohesion means using more of same terms
• Speculation: High cohesion may indicate the number of successful inferences our low pre-testers already made.
• High pre-testers already know the terms, so new inferences are not involved in using them.

Summary

• We have taken automatically computable measures of cohesion from computational linguistics
• Applied them to tutorial dialog
• Found correlations with student learning

Conclusions

• Simple, automatically computable measures of lexical cohesion correlate with learning
• But only for students with low pre-test scores, even though low and high pre-testers showed similar amounts of cohesion.
• Correlation is robust to differences in type of measurement
• It’s the cohesion between student and tutor that’s important

Future Work

• Short term:
• Cohesion may also be related with learning in high pre-testers, but we’re measuring the wrong kind of cohesion
• Work underway to try “sense” level measures
• Halliday & Hassan’s “synonym” levels of reiteration
• “Acceleration” & “speeding up”
• New issues:
• Word sense disambiguation (one sense per discourse?)
• Or measuring it in the wrong places
• Try finding cohesion at impasses (VanLehn 03)
• Try finding change in cohesion over time (Pickering & Garrod 04)
• Is it the dialog, or the essay?
• Long term:
• Test by manipulating cohesion in ITSPOKE

Thanks

• Diane Litman
• ITSPOKE group

Questions?

Cohesion vs Coherence

• Cohesive Devices
• Things that “tie” different parts of a discourse together:
• Anaphora, repetition, etc…
• But still may not make sense:
• John hid Bill’s car keys. He likes spinach. (Jurafsky & Martin 00)
• Coherence relations
• Semantic relations between utterances.
• Result, Explanation, elaboration, etc. (Hobbs 79)

Britton & Gulgoz 91

• Original text:
• Air war in the North, 1965
• By the fall of 1964, Americans in both Saigon and Washington had begun to focus on Hanoi as the source of the continuing problem in the south.
• Modified text:
• Air war in North Vietnam, 1965
• By the beginning of 1965, Americans in both Saigon and Washington had begun to focus on Hanoi, capital of North Vietnam, as the source of the continuing problems in the south.