Correlations between many elements in a system inevitably contain indirect correlations from other elements.

You should also be very careful when you calculate a correlation between Time series.

Topics #

Handling indirect correlation #

The indirect correlation leads us to inaccurate prediction. There are several ways to filter out those indirect edges with spurious correlation:

• Correlation of correlations
• Partial correlation
  • e.g. Genetic Co-Occurrence Network across Sequenced Microbes
• Mutual information based approach (citation needed).

• Methods for the Inverse Ising problem

• network deconvolution: http://www.nature.com/nbt/journal/v31/n8/abs/nbt.2635.html

• Network link prediction by global silencing of indirect correlations
  • Lior Patcher: The network nonsense of Albert-László Barabási

Aggregating correlation #

Sometimes, we want to average correlation values. A standard method is performing Fisher transformation and average the z values then transform it back to correlation coefficient.

• Averaging Correlations: Expected Values and Bias in Combined Pearson rs and Fisher's z Transformations

Constructing networks from correlation matrices #

• Constructing networks by filtering correlation matrices: A null model approach

Correlation Matrix #

• Correlation Matrix Spectra: A Tool for Detecting Non-apparent Correlations?

Correlation and causation #

• http://xkcd.com/552/
• Spurious Correlations
• http://www.google.com/trends/correlate

Generalized ways of inferring associations #

• Reshef et al. Detecting Novel Associations in Large Data Sets
• Kinney and Atwal, Equitability, mutual information, and the maximal information coefficient

Articles #

• http://www.johndcook.com/blog/2008/11/05/how-to-calculate-pearson-correlation-accurately/ - don't expect that the small difference between two large numbers will be accurate.
• Cosine similarity, Pearson correlation, and OLS coefficients

References #

• http://en.wikipedia.org/wiki/Partial correlation
• Graphical interaction models for multivariate time series by Rainer Dahlhaus
• Comparing association network algorithms for reverse engineering of large-scale gene regulatory networks: synthetic versus real data

• A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics

• PLoS biology: Large-Scale Mapping and Validation of Escherichia coli Transcriptional Regulation from a Compendium of Expression Profiles - Mutual information based method

• Genetic Co-Occurrence Network across Sequenced Microbes

• Removing spurious interactions in complex networks

• Unbiased community detection for correlation matrices

Suggested Pages #

• 0.025 Udny Yule
• 0.025 Prediction using social media
• 0.013 Social media
• 0.013 Stock market prediction
• 0.013 Collective intelligence
• 0.006 Pew Research
• 0.006 DERP
• 0.006 Crowdsourcing
• 0.006 Diet
• 0.006 SAX
• More suggestions...