Homework and Assessment

Assessment consists of reading responses (15%), discussion leading (15%), class participation (10%), and a final project (60%).

The assignments detailed below may change prior to the beginning of the course, depending on the size of the class. The current version is here mostly to help students determine if they wish to take this course before enrolling. Please check back on the first day of class for the finalized version.

Reading responses (15%)

Students write reading reactions and add discussion questions at least 24 hours before class. Reference specific sections of papers by page number when possible. If you reference optional readings, make sure your response is understandable to others who may not have read that paper. Consider writing about:

  • Your prior knowledge, beliefs, interests, or opinions before reading a paper, and how this paper changed or augmented them
  • Critiques of the authors’ assumptions or methodology
  • Open questions raised by a paper, and potential ways of investigating those new questions
  • For newer papers, how it may converse with or build upon older work
  • For older work, what parts of it stand or don’t stand today

Each response will be graded on a binary scale of 85% (surface-level engagement with readings) or 100% (reflective and thoughtful engagement with readings).

Leading discussion (15%)

Non-lecture classes will have discussion leaders. They will:

  • Make a presentation (~10 min) recapping one optional reading of your choice in lieu of a reading response. This is due the class before the target class period. Lucy will make edits (focusing on factuality or faithfulness to the original paper) or minor suggestions (to improve presentation clarity) the day before class that you should incorporate before you present.
  • Lead a 45 min discussion initiated by students’ reactions to your presentation and their reading responses to required readings

This will be graded on a 0-100 scale based on a 50/50 split of clarity and accuracy.

Class participation (10%)

Participation includes:

  • Active contribution to paper discussions
  • Collaboratively workshopping others’ projects
  • Substantive contribution to your team’s final project
  • Fifteen minute no-slide, just-chat updates from each project group, arranged in a rotation over the course of the summer. To prepare to give your update, consider the following:
    • What have you done since your last update?
    • What challenges are you facing right now?
    • What will you do next?

Participation is graded on a trinary scale of 85%, 100%, 110%, where the latter is extra credit. Halfway through the summer, students will be notified of their current class participation grade.

Final project (60%)

These are done in groups of 1-3, depending on the size of the class. The aim of this project is to produce a paper similar to the ones we’ve been reading throughout the course.

Brainstorming (10 pts)

Propose three project ideas accompanied by three relevant papers each. Project descriptions should include why the question/s being asked are important and obtainable datasets that you could use to address these questions (300-500 words per idea). The percentages below indicate how you will be graded.

  • Research question (2 pts). This question should be scoped so that it is addressable within the weeks we have remaining in this class.
  • Core contribution (2 pts): You should juxtapose your contributions against relevant papers, e.g. what do you add that past papers do not? Are you working on a new or spin-off problem that has not received the attention it deserves?
  • Method (3 pts): What models and tasks? You can also mention technical work you’ve done in the past that ensures that the difficulty of what you plan to implement will be feasible within this semester.
  • Data (1 pt): You can find datasets by reading papers that contribute metadata-rich resources or benchmarks. Also consider using APIs (e.g. there are ones available for Wikipedia, Reddit).
  • Relevant papers (2 pts): At least 4 papers, 1-2 sentence summary for each paper and how it relates to your RQ or will guide your direction.

Lucy will grade each project idea and description separately and take the average of their point totals to produce the resulting brainstorming report score.

Proposal (20 pts)

Expand on one project idea from the brainstorming phase and describe potential experiments and pipelines needed to carry out the project (700-1000 words, similar format as brainstorming with 2x the points allotted to each component). Also expand the literature review by including additional relevant papers.

Midterm report (25 pts)

This 4 page report includes progress made so far, data collected, challenges faced, preliminary results, and the timeline needed to complete the project. You need to have your dataset/s ready and in an analyzable state by this time. Write this report in Overleaf, using the ACL conference paper template.

  • Abstract (2 pts): Motivate the problem, highlight main goals, include any preliminary main findings.
  • Research question/s (2 pts): Try to limit this to less than three.
  • Related work (2 pts): Recap related work and indicate what is novel in your work compared to prior work.
  • Data (4 pts): Recap your dataset and basic statistics about it (e.g. total word count, number of sentences, metadata information).
  • Approach (5 pts): If your computational project advances a task methodologically, include baselines, parameter decisions, and evaluation. If you are running an experiment or applying a pre-existing method, include details on parameters and experimental design.
  • Results (8 pts): A written section that references key figures and tables, which include captions that are descriptive enough to stand alone.
  • Next steps (2 pts): Outline remaining steps you will do before submitting your final report. It is okay to indicate places in which your new goals deviate from your original ones.

Final presentation (20 pts)

Students will give short slide presentations (# of minutes TBD based on size of class) of their projects to the class.

Final report (25 pts)

Project teams submit their papers (8 pages), which build upon their midterm reports. Include a section at the end that details the contributions of each collaborator on your project team. If there are strong disparities in this contribution list, then some team members’ class participation grade may be affected. Grading is similar to the midterm report except the points for “Next steps” are reallocated to conclusion and future work. Grading for the results section depends on whether it shows substantial progress, careful consideration of Lucy’s midterm feedback, and fulfillment of goals since the midterm report. Failure to meet goals is okay if there is thoughtful discussion around what was challenging and attempts you made towards success, and negative results are also acceptable especially if accompanied by explanation and/or additional analysis.

Some of the guidelines on this page are partially inspired by Stanford’s CS224C.