Natural Language Processing

The main question is this: how can we identify terms that are overrepresented in a given set of document?

Methods #

Log odds ratio informative Dirichlet prior #

When you want to contrast two corpora (e.g. Democrats vs. Republican). Needs a (big) background corpus. Seems to work really well in many cases.

• paper: Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict

An interesting application to restaurant menus: [http://uncommonculture.org/ojs/index.php/fm/article/view/4944/3863](Narrative framing of consumer sentiment in online restaurant reviews)

code: https://gist.github.com/yy/a2fff314073c4806fd5b

tf-idf #

Pointwise mutual information #

Incoming Links #

Related Articles (Article 0) #

• Natural language processing
• Pointwise mutual information
• Text analysis

Suggested Pages #

• 0.248 Data analysis
• 0.129 NLP
• 0.085 Convolutional neural network
• 0.047 StanfordNLP
• 0.047 CNN
• 0.025 Information theory
• 0.025 spaCy
• 0.024 Gensim
• 0.024 String metric
• 0.024 scikit-learn
• More suggestions...