CS 2710 / ISSP 2160: Homework 5

Draft and Revise a Review Paper; Provide Feedback on Review Papers written by your peers

Assigned: 11/01/10

Due dates:

  • 11/15/10: Submit your First Draft

  • 11/29/10: Review the First Drafts of 4 peers

  • 12/08/10: Submit your Final Version

  • 12/08/10: Submit your Back Reviews

  • 12/13/10: Evaluate the Final Version of 4 peers

    Introduction

    Via the writing and reading of commentaries used to facilitate class discussion, you have now both written a number of short reviews of research papers, and read many reviews written by your peers. The goal of this assignment is to more formally improve your skills in 1) writing short reviews of research papers, and 2) providing useful writing feedback to your peers. Both of these skills are very important for a successful graduate and research career.

    For this assignment, everyone will write a review paper discussing Unifying SAT-based and Graph-based Planning by Henry Kautz and Bart Selman, Proceedings International Joint Conferences on Artificial Intelligence (IJCAI), 1999. First, you will submit a First Draft of your paper. Second, you and your peers will engage in Peer Review, where everyone will provide feedback about how to improve the First Draft of others. Third, you will use this feedback to revise your First Draft, to produce a Final Version. Finally, your peers as well as the instructor will evaluate your Final Version.

    Paper submissions and peer reviewing will be done online using the SWORD system, and will be completely anonymous. Login and register now as a new SWORD user.

    I. First Draft (30 points)

    Submit a first draft of your Review Paper, using SWORD. Your paper should be approximately 1250-1750 words, and submitted as a pdf file. Make sure you do NOT include your name anywhere in your paper, so that the peer reviewing process is anonymous.

    Content

    As with the commentaries you have done to date, your Review Paper should not primarily summarize/reiterate the Kautz and Selman paper. Instead, your review should present your deeper understanding and include critical analysis. For this homework your review should in particular focus on the following three points:

  • Contributions and Importance: A one or two paragraph summary of the scientific contributions of the Kautz and Selman paper, along with a critical analysis of the contributions with respect to significance. What problem does this paper solve? What are the major contributions to the field, i.e. which important issues in the field does this paper illuminate and how? Was this paper worth doing, i.e. is the paper solving an important problem? What are the strengths as well as the limitations of the solution, particularly those not already identified by the authors? Would other evaluations have been more compelling? Recall from How to read a research paper that scientific contributions can take on many forms, depending on the kind of a paper it is.

  • Comparison of Approaches: In the paper the authors discuss pure satisfiability planning, pure graph planning, and their unified approach. Identify one or more planning problems that highlight the strengths and weaknesses of each approach and use them to illustrate the strengths as well as the limitations of each solution, particularly those not already identified by the authors.

  • Future work: What extensions or future directions do you see for the ideas in the paper, that are not already mentioned by the authors? For example, does the method described seem mature enough to use in real applications? Why or why not? If so, what applications seem particularly amenable to this approach? What good ideas does the problem formulation, the solution, the approach or the research method contain that could be applied to other areas of Artificial Intelligence? What would be good follow-on projects and why?

    Writing

    Things to consider with respect to your writing:

  • Make it clear: Make sure each paragraph has a clear point, and that the ordering of points from paragraph to paragraph makes sense. Consider using section headers. Give concrete examples.

  • Make logical arguments: You are trying to explain something to your peers. Help them understand why they should believe you. Provide evidence for claims.

  • Make it interesting: After reading your paper, your peers should feel like they learned something.

    II. 4 Peer Reviews (30 points)

    Access the 4 First Draft papers assigned to you for peer reviewing, using SWORD.

    After reading and evaluating your peers' papers, use SWORD to submit your 4 peer reviews. Your peer reviews will consist of textual comments (e.g., provide feedback comments that praise the Contributions discussion in the First Draft) as well as numerical ratings. Each textual comment should focus on a single idea, with the most important comments given first.

    Tips

    Peer reviewing involves providing constructive feedback on your peers' First Drafts, which your peers will then use to revise (and hopefully improve) their papers. Generating feedback for others should also help you improve your general writing skills. When peer reviewing, ask yourself: did you understand the review, did you believe the arguments, did you learn something?

    Other things to consider when peer reviewing:

  • Be nice: Mention the strengths of your peers' papers, so people know where they succeeded. Flaming is not helpful.

  • Be constructive and concrete: Give particular ideas for how to improve the paper. Don't just complain about a problem; give some ideas for how to fix it.

  • Be specific: Be precise about where particular problems occur, and give exact examples of problems

  • Be open-minded about style: There is no one fixed way these papers have to be written. For example, the paper doesn't have to be formal or informal. Also, do not focus on low-level writing problems, like typos and simple grammar problems. Please limit these types of comments to issues that are so bad that they make it hard to understand the text.

    III. Final Version (30 points)

    Access the peer reviews of your First Draft, using SWORD.

    Revise your First Draft based on the peer feedback, then use SWORD to submit your Final Version.

    Also use SWORD to submit Back Reviews (evaluations of the peer reviews you received).

    Tips

    To revise your First Draft, it will likely be helpful to first make a list of the things that you will need to change. Try to address all of the issues brought up by your peers. Even if you do not agree with the feedback, your peer was confused by something, so you should try to make your paper clearer.

    IV. 4 Peer Evaluations (10 points)

    Finally, access the Final Versions of the 4 papers originally assigned to you, using SWORD.

    Use SWORD to evaluate whether you think the Final Version improves over the First Draft.

    Acknowledgements

    This assignment borrows liberally from sample peer reviewing assignments kindly provided by Melissa Patchan. For psychological research on peer reviewing, see this paper. For computational work on enhancing web-based peer review systems, see this conference.