<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">&lt;html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"&gt;&lt;head&gt;&lt;meta content="text/html; charset=UTF-8" http-equiv="content-type" /&gt;&lt;meta name="description" /&gt;&lt;meta content="clojure programming-collective-intelligence" name="keywords" /&gt;&lt;meta content="Nurullah Akkaya" name="author" /&gt;&lt;link href="/images/favicon.ico" rel="icon" type="image/x-icon" /&gt;&lt;link href="/images/favicon.ico" rel="shortcut icon" type="image/x-icon" /&gt;&lt;link href="/default.css" rel="stylesheet" type="text/css" /&gt;&lt;link href="/rss-feed" rel="alternate" title="An explorer&amp;apos;s log" type="application/rss+xml" /&gt;&lt;link href="http://nakkaya.com/2009/11/13/pearson-correlation-score/" rel="canonical" /&gt;&lt;script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"&gt;&lt;/script&gt;&lt;title&gt;Pearson Correlation Score&lt;/title&gt;&lt;/head&gt;&lt;body&gt;&lt;div id="wrap"&gt;&lt;div id="header"&gt;&lt;h1&gt;&lt;a href="/"&gt;nakkaya&lt;span class="fade-small"&gt;dot&lt;/span&gt;&lt;span class="fade"&gt;com&lt;/span&gt;&lt;/a&gt;&lt;/h1&gt;&lt;div class="pages"&gt;&lt;a class="page" href="/"&gt;Home&lt;/a&gt; | &lt;a class="page" href="/projects.html"&gt;Projects&lt;/a&gt; | &lt;a class="page" href="/archives.html"&gt;Archives&lt;/a&gt; | &lt;a class="page" href="/tags/"&gt;Tags&lt;/a&gt; | &lt;a class="page" href="/contact.html" rel="author"&gt;About&lt;/a&gt;&lt;form action="http://www.google.com/search" id="searchform" method="get"&gt;&lt;div&gt;&lt;input class="box" id="s" name="q" type="text" /&gt;&lt;input name="sitesearch" type="hidden" value="nakkaya.com" /&gt;&lt;/div&gt;&lt;/form&gt;&lt;/div&gt;&lt;/div&gt;&lt;div id="content"&gt;&lt;div id="post"&gt;&lt;h2 class="page-title"&gt;Pearson Correlation Score&lt;/h2&gt;&lt;p&gt;
This post will cover another topic from &lt;a href="http://oreilly.com/catalog/9780596529321"&gt;Programming Collective
Intelligence&lt;/a&gt; that is used to define similarities between items called
&lt;a href="http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient"&gt;Pearson correlation score&lt;/a&gt;, the formula for this algorithm looks like
the following,
&lt;/p&gt;

\begin{equation}
  r = \frac{\sum XY - \frac{\sum X \times \sum Y }{N}}
           {\sqrt{(\sum X^2- \frac{(\sum X)^2}{N}) - (\sum Y^2- \frac{(\sum Y)^2}{N})}}
\end{equation}

&lt;p&gt;
This calculation returns a value between -1 and 1. Two users with a
similarity of 1 have rated every item identically. Unlike &lt;a href="http://nakkaya.com/2009/11/11/euclidean-distance-score/"&gt;Euclidean Distance Score&lt;/a&gt;
this formula doesn't need to be normalized. Pearson correlation score,
also accounts for average ratings for each user, a user that rates
everything 5 and a user that rates everything 1 will have a similarity
of 1. This  may or may not be the behavior you want depending on your
situation.
&lt;/p&gt;

&lt;div class="org-src-container"&gt;

&lt;pre class="src src-clojure"&gt;(&lt;span style="color: #ff5f00; font-weight: bold;"&gt;defn&lt;/span&gt; &lt;span style="color: #d7af00; font-weight: bold;"&gt;pearson&lt;/span&gt; [x y]
  (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;let&lt;/span&gt; [shrd (filter x (keys y))] 
    (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;if&lt;/span&gt; (= 0 (count shrd))
      0
      (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;let&lt;/span&gt; [sum1  (reduce (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;fn&lt;/span&gt;[s mv] (+ s (x mv))) 0 shrd)
            sum2  (reduce (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;fn&lt;/span&gt;[s mv] (+ s (y mv))) 0 shrd)
            sum1sq  (reduce (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;fn&lt;/span&gt;[s mv] (+ s (&lt;span style="font-weight: bold; text-decoration: underline;"&gt;Math&lt;/span&gt;/pow (x mv) 2))) 0 shrd)
            sum2sq  (reduce (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;fn&lt;/span&gt;[s mv] (+ s (&lt;span style="font-weight: bold; text-decoration: underline;"&gt;Math&lt;/span&gt;/pow (y mv) 2))) 0 shrd)
            psum (reduce (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;fn&lt;/span&gt;[s mv] (+ s (* (x mv) (y mv)))) 0 shrd)
            num (- psum (/ (* sum1 sum2) (count shrd)))
            den (&lt;span style="font-weight: bold; text-decoration: underline;"&gt;Math&lt;/span&gt;/sqrt (* 
                            (- sum1sq (/ (&lt;span style="font-weight: bold; text-decoration: underline;"&gt;Math&lt;/span&gt;/pow sum1 2) (count shrd)))
                            (- sum2sq (/ (&lt;span style="font-weight: bold; text-decoration: underline;"&gt;Math&lt;/span&gt;/pow sum2 2) (count shrd)))))]
        (&lt;span style="color: #ff5f00; font-weight: bold;"&gt;if&lt;/span&gt; (= den 0)
          0
          (double (/ num den))) ))))
&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;
Using the same critics map from &lt;a href="http://nakkaya.com/2009/11/11/euclidean-distance-score/"&gt;Euclidean Distance Score&lt;/a&gt;,
&lt;/p&gt;

&lt;pre class="example"&gt;
user=&amp;gt; (pearson (critics "Lisa Rose") (critics "Gene Seymour"))
0.39605901719066977

user=&amp;gt; (pearson (critics "Lisa Rose") {})
0
&lt;/pre&gt;
&lt;div class="post-tags"&gt;Tags: &lt;a href="/tags/#clojure"&gt;clojure &lt;/a&gt;&lt;a href="/tags/#programming-collective-intelligence"&gt;programming-collective-intelligence &lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;div id="related"&gt;&lt;h3 class="random-posts"&gt;Random Posts&lt;/h3&gt;&lt;ul class="posts"&gt;&lt;li&gt;&lt;span&gt;20 Apr 2010&lt;/span&gt;&lt;a href="/2010/04/20/fractals-in-clojure-newton-fractal/"&gt;Fractals in Clojure - Newton Fractal&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;span&gt;15 Feb 2017&lt;/span&gt;&lt;a href="/2017/02/15/bare-metal-lisp-rc-control-using-ferret/"&gt;Bare Metal Lisp - RC Control using Ferret&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;span&gt;10 Oct 2009&lt;/span&gt;&lt;a href="/2009/10/10/processing-xml-with-clojure/"&gt;Processing XML With Clojure&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;span&gt;03 Jan 2010&lt;/span&gt;&lt;a href="/2010/01/03/clodiuno-a-clojure-api-for-the-firmata-protocol/"&gt;clodiuno - A Clojure API for the Firmata Protocol&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;span&gt;09 Jan 2010&lt;/span&gt;&lt;a href="/2010/01/09/a-simple-turtle-graphics-implementation-in-clojure/"&gt;A Simple Turtle Graphics Implementation in Clojure&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/div&gt;&lt;div id="disqus"&gt;&lt;div id="disqus_thread"&gt;&lt;/div&gt;&lt;script type="text/javascript" src="//disqus.com/forums/nakkaya/embed.js"&gt;&lt;/script&gt;&lt;noscript&gt;&lt;a href="//disqus.com/forums/nakkaya/?url=ref"&gt;View the discussion thread.&lt;/a&gt;&lt;/noscript&gt;&lt;a href="//disqus.com" class="dsq-brlink"&gt;blog comments powered by &lt;span class="logo-disqus"&gt;Disqus&lt;/span&gt;&lt;/a&gt;&lt;/div&gt;&lt;/div&gt;&lt;div id="footer"&gt;&lt;a href="/rss-feed"&gt; RSS Feed&lt;/a&gt;&lt;p&gt;&amp;copy; 2018&lt;a href="http://nakkaya.com"&gt; Nurullah Akkaya&lt;/a&gt;&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;script type="text/javascript"&gt;
//&lt;![CDATA[
(function() {
	     var links = document.getElementsByTagName('a');
	     var query = '?';
	     for(var i = 0; i &lt; links.length; i++) {
		     if(links[i].href.indexOf('#disqus_thread') &gt;= 0) {
								       query += 'url' + i + '=' + encodeURIComponent(links[i].href) + '&amp;';
								       }
		     }
	     document.write('&lt;script charset="utf-8" type="text/javascript" src="//disqus.com/forums/nakkaya/get_num_replies.js' + query + '"&gt;&lt;/' + 'script&gt;');
	     })();
//]]&gt;
&lt;/script&gt;&lt;/body&gt;&lt;/html&gt;</pre></body></html>