A corpus of articles from the world press has been manually annotated with private states (opinions, emotions, sentiments) and their properties (source, target, intensity, polarity). The MPQA opinion annotated corpus as well as the OpinionFinder automatic system are available at http://www.cs.pitt.edu/mpqa

This article describes the annotation scheme: Janyce Wiebe, Theresa Wilson, and Claire Cardie (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, volume 39, issue 2-3, pp. 165-210.

Here are Instructions for Performing Opinion Annotation in the Gate Annotation System. T. Wilson (2002) Note: this is currently being updated.

Swapna Somasundaran, Janyce Wiebe, and Josef Ruppenhofer (2008). Discourse Level Opinion Interpretation. COLING-2008.

Theresa Wilson, Janyce Wiebe, and Paul Hoffmann (to appear). Recognizing Contextual Polarity: an exploration of features for phrase-level sentiment analysis. To appear in Computational Linguistics.

Josef Ruppenhofer, Swapna Somasundaran, and Janyce Wiebe (2008). Finding the Sources and Targets of Subjective Expressions. LREC 2008.

Carmen Banea, Rada Mihalcea, and Janyce Wiebe (2008). A Bootstrapping Method for Building Subjectivity Lexicons for Languages with Scarce Resources. LREC 2008.

Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe (2008). Discourse Level Opinion Relations: An Annotation Study. SIGdial Workshop on Discourse and Dialogue, Columbus, Ohio, June 2008.

Rada Mihalcea , Carmen Banea , & Janyce Wiebe (2007). Learning Multilingual Subjective Language via Cross-Lingual Projections ACL-2007 .

Swapna Somasundaran, Theresa Wilson, Janyce Wiebe & Veselin Stoyanov (2007). QA with Attitude: Exploiting Opinion Type Analysis for Improving Question Answering in On-line Discussions and the News. International Conference on Weblogs and Social Media (ICWSM-2007) .

Swapna Somasundaran, Josef Ruppenhofer & Janyce Wiebe (2007) Detecting Arguing and Sentiment in Meetings , SIGdial Workshop on Discourse and Dialogue, Antwerp, Belgium, September 2007.

Janyce Wiebe and Rada Mihalcea (2006). Word Sense and Subjectivity. ACL-2006 . Here are presentation slides.

Theresa Wilson , Janyce Wiebe, & Rebecca Hwa (2006). Recognizing strong and weak opinion clauses. Computational Intelligence 22 (2): 73-99.

Ellen Riloff , Siddharth Patwardhan and Janyce Wiebe (2006). Feature Subsumption for Opinion Analysis. Conference on Empirical Methods in Natural Language Processing (EMNLP-2006)

Wei-Hao Lin, Theresa Wilson, Janyce Wiebe and Alexander Hauptmann (2006). Which Side are You on? Identifying Perspectives at the Document and Sentence Levels. CoNLL-2006

Swapna Somasundaran, Janyce Wiebe, Paul Hoffmann, & Diane Litman (2006). Manual Annotation of Opinion Categories in Meetings. COLING-ACL Workshop: Frontiers in Linguistically Annotated Corpora .

Theresa Wilson , Janyce Wiebe, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. HLT-EMNLP-2005. Here are presentation slides.

Veselin Stoyanov , Claire Cardie , and Janyce Wiebe (2005). Multi-Perspective Question Answering Using the OpQA Corpus. HLT-EMNLP-2005. Here is a version with a correction. Here are presentation slides.

Ellen Riloff, Janyce Wiebe, and William Phillips (2005). Exploiting subjectivity classification to improve information extraction. Proc. 20th National Conference on Artificial Intelligence (AAAI-2005). Here are presentation slides.

Janyce Wiebe, Theresa Wilson , and Claire Cardie (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, volume 39, issue 2-3, pp. 165-210.

Theresa Wilson and Janyce Wiebe (2005). Annotating attributions and private states. Proc. ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky.

Janyce Wiebe and Ellen Riloff (2005). Creating subjective and objective sentence classifiers from unannotated texts. Sixth International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2005) (Invited paper.)

Theresa Wilson , Janyce Wiebe, and Rebecca Hwa (2004). Just how mad are you? Finding strong and weak opinion clauses. Proc. 19th National Conference on Artificial Intelligence (AAAI-2004). Here are presentation slides.

Janyce Wiebe, Theresa Wilson , Rebecca Bruce , Matthew Bell , and Melanie Martin (2004). Learning subjective language. Computational Linguistics 30 (3).

Veselin Stoyanov, Claire Cardie , Diane Litman and Janyce Wiebe (2004). Evaluating an Opinion Annotation Scheme Using a New Multi-Perspective Question and Answer Corpus. AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications.

Ellen Riloff and Janyce Wiebe (2003). Learning Extraction Patterns for Subjective Expressions. Conference on Empirical Methods in Natural Language Processing (EMNLP-03). ACL SIGDAT. Pages 105-112. Here are presentation slides.

Ellen Riloff, Janyce Wiebe, and Theresa Wilson (2003). Learning Subjective Nouns Using Extraction Pattern Bootstrapping. Seventh Conference on Natural Language Learning (CoNLL-03). ACL SIGNLL. Pages 25-32.

Theresa Wilson and Janyce Wiebe (2003). Annotating Opinions in the World Press. 4th SIGdial Workshop on Discourse and Dialogue (SIGdial-03). ACL SIGdial.
Here is a corrected version of the paper that appears in the proceedings.
Here is the original.
Here are presentation slides.

Janyce Wiebe, Eric Breck, Chris Buckley, Claire Cardie , Paul Davis, Bruce Fraser, Diane Litman, David Pierce, Ellen Riloff, Theresa Wilson , David Day, Mark Maybury (2003). Recognizing and Organizing Opinions Expressed in the World Press. 2003 AAAI Spring Symposium on New Directions in Question Answering.

Claire Cardie , Janyce Wiebe, Theresa Wilson, and Diane Litman (2003). Combining Low-Level and Summary Representations of Opinions for Multi-Perspective Question Answering. 2003 AAAI Spring Symposium on New Directions in Question Answering.

Wiebe, J. & Wilson, T. (2002). Learning to disambiguate potentially subjective expressions. Sixth Conference on Natural Language Learning (CoNLL-2002). ACL SIGNLL. Taipei, Taiwan, August, 2002.

Pustejovsky, J., Wiebe, J., & Maybury, M. (2002). Multi-perspective and temporal question answering. Third International Conference on Language Resources and Evaluation (LREC 2002) Workshop on Question Answering: Strategy and Resources, Canary Islands, Spain, May 2002.

Wiebe, J., Bruce, R., Bell, M., Martin, M., & Wilson, T. (2001). A Corpus Study of Evaluative and Speculative Language. Proc. 2nd ACL SIGdial Workshop on Discourse and Dialogue. Aalborg, Denmark, September, 2001. The data are available by clicking here.

Janyce Wiebe, Theresa Wilson , & Matthew Bell (2001) Identifying Collocations for Recognizing Opinions. Proc. ACL 01 Workshop on Collocation. Toulouse, France, July 2001.

Janyce Wiebe (2000). Learning Subjective Adjectives from Corpora. Proc. 17th National Conference on Artificial Intelligence (AAAI-2000). Austin, Texas, July 2000. Presentation slides and lists of adjectives learned using the process are available by clicking here .

Hatzivassiloglou, Vasileios and Janyce Wiebe (2000). Effects of Adjective Orientation and Gradability on Sentence Subjectivity. International Conference on Computational Linguistics (COLING-2000). Lists of positive, negative, gradable, and dynamic adjectives are available by clicking here .

Bruce, Rebecca F. & Janyce Wiebe M. (1999). Recognizing subjectivity: a case study in manual tagging. Natural Language Engineering 5 (2).

Janyce Wiebe M., Bruce, Rebecca F., & O'Hara, Thomas P. (1999). Development and use of a gold standard data set for subjectivity classifications. In Proc. 37th Annual Meeting of the Assoc. for Computational Linguistics (ACL-99). Association for Computational Linguistics, University of Maryland, June, pp. 246-253. The Coding manual and data are available by clicking here

Wiebe, Janyce, McKeever, Kenneth, & Bruce, Rebecca. (1998). Mapping collocational properties into machine learning features. Proc. 6th Workshop on Very Large Corpora (WVLC-98). Association for Computational Linguistics SIGDAT, Montreal, Quebec, Canada, August 1998, pp. 225-233. The coding manual for the event categorization manual annotations is available here

Wiebe, Janyce, Bruce, Rebecca, & Duan, Lei. (1997). Probabilistic event categorization. In Recent Advances in Natural Language Processing (RANLP-97). Tsigov Chark, Bulgaria, Sept. 1997, pp. 163-170. The coding manual for the manual annotations is available here

Janyce Wiebe & McKeever, Kenneth (1998). Collocational properties in probabilistic classifiers for discourse categorization. Proc. AAAI-98 Spring Symposium Series Workshop on Applying Machine Learning to Discourse Processing, Stanford, California, March 1998.

Rapaport, William J., Shapiro, Stuart C., & Janyce Wiebe M. (1997). Quasi-indicators and knowledge reports. Cognitive Science 21 (1), 63-107.

Janyce Wiebe & Bruce, Rebecca (1995). Probabilistic classifiers for tracking point of view. In Proc. Symposium on Empirical Methods in Discourse Interpretation and Generation, AAAI 1995 Spring Symposium Series, pp. 181-187.

Janyce Wiebe M. (1994). Tracking point of view in narrative. Computational Linguistics 20 (2): 233-287.

Janyce Wiebe M. (1993). Issues in linguistic segmentation. In Proc. 1993 SIG ACL workshop, Intentionality and Structure in Discourse Relations, pp. 148-151.

Janyce Wiebe M. (1991). References in narrative text. Noûs (Special issue on Cognitive Science and AI) 25 (4): 457-486.

Janyce Wiebe M. (1991). Point of view and discourse processing. Working Notes of the 1991 AAAI Fall Symposium on Discourse Structure in Natural Language Understanding and Generation, pp. 136-137. Paper (short) or Transparencies (more complete).

Janyce Wiebe M. (1990). Identifying subjective characters in narrative. In Proc. 13th International Conference on Computational Linguistics (COLING-90), pp. 401-408.

Bruder, Gail A. & Janyce Wiebe M. (1990). A psychological test of an algorithm for recognizing subjectivity in narrative text. In Proc. 12th Annual Conference of the Cognitive Science Society, pp. 947-952.

Janyce Wiebe M. & Rapaport, William J. (1988). A computational theory of perspective and reference in narrative. In Proc. 26th Annual Meeting of the Assoc. for Computational Linguistics (ACL-88), pp. 131-138.

Janyce Wiebe M. & Rapaport, William J. (1986). Representing De Re and De Dicto belief reports in discourse and narrative. Proceedings of the IEEE (Special Issue on Knowledge Representation) 74: 1405-13.

If you would like a hardcopy, please contact Ashley Nagy: nagya at pitt dot edu

Janyce Wiebe M. (1990). Recognizing Subjective Sentences: A Computational Investigation of Narrative Text. (Ph.D. dissertation) Technical Report 90-03. (Buffalo: SUNY Buffalo Dept. of Computer Science).

Cardie, C., Wiebe, J., Wilson, T., & Litman, D. (2004). Low-Level Annotations and Summary Representations of Opinions for Multi-Perspective Question Answering. M. Maybury (ed.) New Directions in Question Answering . AAAI Press.

Wiebe, J. & Bruce, R. Probabilistic Classifiers for Tracking Point of View (2001). Barnet, George A. (ed.) New Directions in Computer Content Analysis. New York: Ablex, Advances in Communication Research Series. Revised version of Wiebe & Bruce 1995.

Rapaport, William J., Shapiro, Stuart C., & Janyce Wiebe M. (1998). Quasi-indicators and knowledge reports. In F. Orilia and W.J. Rapaport (eds.), Thought, Language, and Ontology, Netherlands, Kluwer Academic, pp. 235-294. Reprint of Rapaport, Shapiro, & Wiebe 1997.

Janyce Wiebe (1995). References in narrative text. In Judy Duchan, Gail Bruder, and Lynne Hewitt (eds.) Deixis in Narrative: A Cognitive Science Perspective. Lawrence Erlbaum Associates. Reprint of Wiebe 1991a.

Bruder, Gail & Janyce Wiebe (1995). Recognizing subjectivity and identifying subjective characters in third-person fictional narrative. In Judy Duchan, Gail Bruder, and Lynne Hewitt (eds.) Deixis in Narrative: A Cognitive Science Perspective. Lawrence Erlbaum Associates.