Menu
Log in
Forgot password

 

Login

Log in
Forgot password

Join | Renew | Donate


Evaluating an Interface: Metacognitive Approaches to AI and Tutoring

Jessica Craig, Utah Valley University

LEARNING OUTCOMES/OBJECTIVES

After completing this training, tutors should be able to 

  • Evaluate generative AI output for its effectiveness
  • Navigate tutoring sessions with writers who use generative AI
  • Offer recommendations to writers on productive AI use
  • Use metacognitive strategies when reading written works, such as close reading, questioning the text, or connecting the text to previous knowledge

MATERIALS NEEDED

INSTRUCTIONAL PLAN

While a basic outline for the lesson is included here, the presentation slides specify each of these steps. Refer to the trainer guidelines and the presenter notes on the slides for additional information.

Introduction (slides 1 & 2)

  • Ask participants for existing thoughts about AI. (Slide 3)
  • Provide basic background information on generative AI. (Slide 4)
  • Give a brief overview of effective prompt creation. (Slide 5)
  • Provide basic background information on metacognition. (Slide 6)
  • Remind participants to discuss appropriate AI use with writers, based on instructor, department, and university expectations.

Body of Lesson (Slides 7 & 8)

  • Select 2-3 scenarios to input as prompts into a generative AI system.
  • Use the initial prompt provided to receive an output from the AI program.
  • Ask participants what they notice about the output, then model critical evaluations of AI output for the first prompt.
  • Write 3-4 follow-up prompts that could reasonably follow the previous prompt, as in a conversation.
  • Encourage participants to critically evaluate follow-up prompts. For an easier transition into this, participants can model their own responses after the trainer’s example of evaluating the text.
  • Repeat for remaining scenarios chosen.

Conclusion (slide 9)

  • Return to question in introduction about participant thoughts on AI.
  • Encourage further exploration and experimentation to practice using generative AI. (Slide 10)
  • End with some main takeaways and discussion-prompting questions (included in presentation slides). (Slide 11)

ASSESSING FOR UNDERSTANDING

The following are concrete and specific measures to determine if tutors have met the learning objectives or outcomes:

  • Tutors can use metacognition in tutorials by applying strategies from evaluating generative AI.
  • Tutors can navigate tutorials with writers who use generative AI and can provide suggestions for improving the efficacy of the output from generative AI.
  • Tutors can read and evaluate the efficacy outputs from generative AI.

EXTENSIONS AND ADAPTATIONS

Suggestions for extending or condensing the training

      • Adjust the number of scenarios for how long the training should last. With discussion and 3-4 follow-up prompts, one scenario takes approx. 20 minutes. Two scenarios are recommended for a 50-60 minute training session.
      • Add to or reduce the number of follow-up prompts to adjust for smaller amounts of time than for a full scenario.
      • If needed, select a generative AI program that has a text-to-voice feature, and read aloud often. For this adjustment, select only one or two scenarios for thorough participation.
      • Create a scenario specific to your writing center’s context using the template scenario provided. Trainers who have practice with using generative AI and prompting are encouraged to use the template.

Alternate activities

      • For a training more focused on effective prompting techniques, adjust a prompt given by simplifying it down to one sentence or question. Then, use the original prompt in another tab to compare the outputs.
      • To adjust for an asynchronous format, assign tutors to a scenario. Instruct them to put it into a generative AI program and give 4-5 follow-up prompts. Provide an example (written, audio, or audiovisual) of critically reading through the AI’s outputs. Encourage discussion among tutors in a shared online space or at another designated time.

RESOURCES

Amazon Web Services (AWS). (n.d.). What is prompt engineering?. Amazon Web Services. Retrieved December 1, 2024. https://aws.amazon.com/what-is/prompt-engineering/

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?  . FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. https://doi.org/10.1145/3442188.3445922

DAIR.AI. (2024). Prompt engineering guide. Prompt Engineering Guide. Retrieved December 1, 2024. https://www.promptingguide.ai/

Dobrin, S. (2023). AI and Writing. Broadview Press.

The Writing Center, University of North Carolina at Chapel Hill. (2023). Generative AI in academic writing. The Writing Center, University of North Carolina at Chapel Hill. https://writingcenter.unc.edu/tips-and-tools/generative-ai-in-academic-writing/

REFERENCES

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?  . FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. https://doi.org/10.1145/3442188.3445922

Dobrin, S. (2023). AI and Writing. Broadview Press.

McGovern, B. T. (2022). The writing center’s role in disciplinary writing development: Enhancing discourse community knowledge through metacognitive dialogue. Praxis: A Writing Center Journal, 19(2). https://www.praxisuwc.com/192-mcgovern

Ul Ain, Q., Thurston, A., MacKenzie, A., & Ozkaya, C. (2023). What does previous research tell us about the effects of peer tutoring on metacognition in primary and secondary schools?. International Journal of Educational Research Open, 4. https://doi.org/10.1016/j.ijedro.2023.100248

The Writing Center, University of North Carolina at Chapel Hill. (2023). Generative AI in academic writing. The Writing Center, University of North Carolina at Chapel Hill. https://writingcenter.unc.edu/tips-and-tools/generative-ai-in-academic-writing/




Follow our activities

© Wild Apricot teachers association. 

Powered by Wild Apricot Membership Software