Cohesion and Learning in a Tutorial Spoken Dialog System

Tutoring Corpora

• Transcripts of tutoring sessions
• Training corpus (fall 2003):
• 20 students, 5 problems each
• 95 dialogs (5 had no dialog)
• 13 low pre-testers, 7 high pre-testers
• Testing corpus (spring 2005):
• 34 students, 5 problems each
• 163 dialogs (7 had no dialog)
• 18 low pre-testers, 16 high pre-testers

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Less significant on testing data, “token with stops” level reduced to a trend

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Slightly less significant on testing data

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Less significant on testing data, “token with stops” level reduced to a trend

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Less significant on testing data, “token with stops” level reduced to a trend

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655

Results: Aptitude/Treatment

• Test: partial correlation of post-test & cohesion count, controlling for pre-test
• Cohesion correlated with learning for low pre-test students
• Not for high pre-test students
• Little difference between types of measurement
• Less significant on testing data, “token with stops” level reduced to a trend

	Tests
	Train: 2003 Data		Test: 2005 Data
Students	R	P-Value	R	P-Value
Grouped by Token (with stop words)
All Students	0.380	0.098	0.207	0.239
Low Pretest	0.614	0.026	0.448	0.062
High Pretest	0.509	0.244	0.014	0.958
Grouped by Token (Stop words removed)
All Students	0.431	0.058	0.269	0.124
Low Pretest	0.676	0.011	0.481	0.043
High Pretest	0.606	0.149	0.132	0.627
Grouped by Stem (Stop words removed)
All Students	0.423	0.063	0.261	0.135
Low Pretest	0.685	0.010	0.474	0.047
High Pretest	0.633	0.127	0.121	0.655