This workshop at ICML was a continuation of previous workshops held at AAAI that aim to cast a critical eye on the methods used in machine learning to experimentally evaluate the performance of algorithms.

It kicked off with a series of mini debates with Nathalie and Chris articulating the opposing sides. The questions included the following:

• Should we change how evaluation is done?
• Is evaluation central to empirical work?
• Are statistical tests critical to evaluation?
• Are the UCI data sets sufficient for evaluation?

There were three papers I particularly liked: Janez Demsar’s talk “On the Appropriateness of Statistical Tests in Machine Learning“, Edith Law’s “The Problem of Accuracy as an Evaluation Criterion“, and Chris Drummond’s call for a mild-mannered revolution “Finding a Balance between Anarchy and Orthodoxy“.

Janez’s talk touched on a number of criticisms that I had found in Jacob Cohen’s paper “The Earth is Round (p < 0.05)” making the case that people often incorrectly report and incorrectly interpret p-values for statistical tests. Unfortunately, as Janez points out, since machine learning is a discipline that (rightly) places emphasis on results it is difficult as a reviewer to reject a paper that presents an ill-motivated and confusing idea if its authors have shown that, statistically, it outperforms similar approaches.

Edith’s talk argued that accuracy is sometimes a poor measure of performance making all this concern over whether we are constructing statistical tests for it (or AUC) moot. In particular, for tasks like salient region detection in images, language translation and music tagging there is no single correct region, translation or tag. Whether or not a particular region/translation/tag is “correct” or not is impossible to determine independent of the more difficult tasks of image recognition/language understanding/music identification. Solving these for the purposes of evaluation would make a solution to the smaller tasks redundant. Instead of focusing on evaluation of the smaller tasks, Edith suggests ways in which games that humans play on the web — such as the ESP Game — can be used to evaluate machine performance on these tasks by playing learning algorithms against humans.

Finally, Chris’s talk made the bold claim that the way we approach evaluation in machine learning is an “impoverished realization of a controversial methodology”, namely statistical hypothesis testing. “Impoverished” because when we do do hypothesis testing it is in the narrowest of senses, mainly to test that my algorithm is better than yours on this handful of data sets. “Controversial” since many believe science to have social, exploratory and accidental aspects — much more than just the clinical proposing of hypotheses for careful testing.

What these papers and the workshop as a whole showed me was how unresolved my position is on these and other questions regarding evaluation. On the one hand I spent a lot of time painstakingly setting up, running and analysing experiments for my PhD research on inductive transfer in order to evaluate the methods I was proposing. I taught myself how to correctly control for confounding factors, use the Bonferroni correction to adjust significance levels and other esoterica of statistical testing. Applying all these procedures carefully to my work felt very scientific and I was able to create many pretty graphs and tables replete with confidence intervals, p-values and the like. On the other hand — and with sufficient hindsight — it’s not clear how much value this type of analysis added to the thesis overall (apart from demonstrating to my reviewers that I could do it).

The dilemma is this: when one algorithm or approach clearly dominates another details such as p-values, t-tests and the like only obscure the results; and when two algorithms are essentially indistinguishable using “significance” levels to pry them apart seems to be grasping at straws.

That’s not to say that we should get rid of empirical evaluation all together. Rather, we should carefully choose (or create) our data sets and empirical questions so as to gain as much insight as possible and go beyond “my algorithm is better than yours”. Statistical tests should not mark the end of an experimental evaluation but rather act as a starting point for further questions and carefully constructed experiments that resolve those questions.

Having played around with various wiki systems and discussion sites in the past, I volunteered to help set something up. As John has noted on his blog the discussion site is now up and running.

The main aim with this first attempt was to provide basic functionality: papers can be browsed by author, title and keyword; each paper has a discussion thread where anyone can leave comments. There are no comments at the time of writing this but I’m hoping this will change once the conference gets underway.

Provided there are no disasters, the site will remain up for as long as it is useful. Ultimately, I’d like to add earlier conference proceedings to the site and ensure future conferences can be added as well. We will see how it goes this year and incorporate and feedback into future versions of the site.

For those interested in the technical details, I used DokuWiki as the engine for the site along with a number of plugins, most importantly the discussion plugin.

The rest of this post describes the intersection of our interests — data from the Australian Common Reader Project — and the visualisation tool I wrote to explore it. The tool itself is based on a simple application of linear Principal Component Analysis (PCA). I’ll attempt to explain it here in such a way that readers who have not studied this technique might still be able to make use of the tool.

The Australian Common Reader Project

One of Julieanne’s research interests is the Australian audience of the late 19th and early 20th centuries. As part of her PhD, she made use of an amazing database that is part of the Australian Common Reader Project — a project that has collected and entered library borrowing records from Australian libraries along with annotations about when books were borrowed, their genres, borrower occupations, author information, etc. This sort of information makes it possible for Australian literature and cultural studies academics to ask empirical questions about Australian readers’ relationship with books and periodicals.

Ever on the lookout for interesting data sets, I suggested that we apply some basic data analysis tools to the database to see what kind of relationships between books and borrowers we might find. When asked if we could have access to the database, Tim Dolin graciously agreed and enlisted Jason Ensor to help with our technical questions.

Books and Borrowers

After an initial inspection, my first thought was to try to visualise the similarity of the books in the database as measured by the number of borrowers they have in common. The full database contains 99,692 loans of 7,078 different books from 11 libraries by one of the 2,642 people. To make this more manageable, I focused on books that had at least 20 different borrowers and only considered people who had borrowed one of these books. This distilled the database down to a simple table with each row representing one of 1,616 books and each column representing one of 2,473 people.

Conceptually, each cell in the table contains a 1 if the person associated with the cell’s column borrowed the book associated with the cell’s row. If there was no such loan between a given book and borrower the corresponding cell contains a 0. For example, Table 1 shows that book 2 was borrowed (at least once) by borrower 1 but never by borrower 2,473.

Book Similarity

The table view of the books and their borrowers does not readily lend itself to insight. The approach we took to get a better picture of this information was to plot each book as a point on a graph so that similar books are placed closer together than dissimilar books. To do this a notion of what “similar books” is required.

Mathematically, row of Table 1 can be represented as a vector of 1s and 0s. The value of the cell in the ^th column of that row will be denoted . For example, the 2^nd row in the table can be written as the vector and the value in its first column is .

A crude measure of the similarity between book 1 and book 2 can be computed from this table by counting how many borrowers they have in common. That is, the number of columns that have a 1 in the row for book 1 and the row for book 2.

In terms of the vector representation, this similarity measure is simply the “inner product” between and and is written where N = 2,473 is the total number of borrowers.

It turns out that simply counting the number of borrowers two books is not a great measure of similarity. The problem is that two very popular books, each with 100 borrowers, that only share 10% of their borrowers would be considered as similar as two books, each with 10 readers, that share all of their borrowers. An easy way to correct this is to “normalise” the borrower counts by making sure the similarity of a book with itself is always equal to 1. A common way of doing this is by dividing the inner product of two books by the “size” of each of the vectors for those books.

Mathematically, we will denote the size of a book vector as . The similarity between two books then becomes:

Principal Component Analysis

Now that we have a similarity measure between books the idea is to create a plot of points — one per book — so that similar books are placed close together and dissimilar books are kept far apart.

A standard technique for doing this is Principal Component Analysis. Intuitively, this technique aims to find a way of reducing the number of coordinates in each book vector in such a way that when the similarity between two books is computed using these smaller vectors it is as close as possible to the original similarity. That is, PCA creates a new table that represents books in terms of only two columns.

Table 2 gives an example of the book table after PCA that reduces the book vectors (rows) from 2,473 to two entries. The PCA columns cannot be as easily interpreted as the borrowers columns in Table 1 but the values in the columns are such that the similarity of the books in Table 2 are roughly as similar as if the values in Table 1 were used. That is, if and are the vectors for the first two rows of Table 2 then would be close to , the similarity of the first two rows in Table 1.¹

Visualising the Data

Figure 1 shows a plot of the PCA reduced book data. Each circle represents one of the 1,616 books, plotted according to the coordinates in a table like Table 2. The size of each circle indicates how many borrowers each book had and its colour indicates which library the book belongs to.²

Figure 1: A PCA plot of all the books in the ACRP database coloured according to which library they belong to. The size of each circle indicates the number of borrowers of the corresponding book.

One immediate observation is that books are clustered according to which library they belong to. This is not too surprising since the books in a library limit what borrowers from that library can read. This means it is likely that two voracious readers that frequent the same library will read the same books. This, in turn, will mean the similarity of two books from a library will be higher than books from different libraries as there are very few borrowers that use more than one library.

Drilling Down and Interacting

To get a better picture of the data, we decided to focus on books from a single library to avoid this clustering. The library we focused on was the Lambton Miners’ and Mechanics’ Institute in New South Wales. This library had the largest number of loans (20,253) and so was most likely to have interesting similarity data.

There are a total of 789 books in the Lambton institute and 469 borrowers of those books. A separate PCA reduction was performed on this restricted part of the database to create a plot of only the Lambton books.

To make it easier to explore this data, I wrote a simple tool that allows a viewer to interact with the PCA plot. A screenshot from this tool is shown in Figure 2. Once again, larger circles represent books with a larger number of borrowers.

Clicking on the figure will open a new window and, after a short delay, the tool will run. The same page can also be accessed from this link.

Figure 2: A screenshot of the ACRP visualisation tool showing books from the Lambton Institute. Click the image to run the tool in a new window.

Instructions describing how to use the tool can be found below it. In a nutshell: hovering over a circle will reveal the title of the book corresponding to that circle; clicking on a circle will draw lines to its most similar neighbours; altering the “Borrowers” bar will only show books with at least that many borrowers; and altering the “Similarity” bar will only draw lines to books with at least that proportion of books in common.

Future Work and Distant Reading

Julieanne and I are still at the early stages of our research using the ACRP database. The use of PCA for visualisation was a first step in our pursuit of what Franco Moretti calls “distant reading” — looking at books as objects and how they are read rather than the “close reading” of the text of individual books.

Now that we have this tool, we are able to quickly explore relationships between these books based on the reading habits of Australians at the turn of the century. Of course, there are many caveats that apply to any patterns we might see in these plots. For instance, the similarity between books is only based on habits of a small number of readers and will be influenced by the peculiarities of the libraries and the books they choose to buy. For this reason, these plots are not intended to provide conclusive answers to questions we might.

Instead we hope that exploring the ACRP database in this way will lead us to interesting questions about particular pairs or groups of books that can be followed up by a more thorough analysis of their readers, their text as well as other historical and cultural factors about them.

Data and Code

For the technically minded, I have made the code I used to do the visualisation is available on GitHub. It is a combination of SQL for data preprocessing, R for the PCA reduction and Processing for creating the visualisation tool. You will also find a number of images and some notes at the same location.

Access to the data that the code acts upon is not mine to give, so the code is primarily to show how I did the visualisation rather than a way to let others analyse the data. If the founders of the ACRP project decide to release the data to the public at a later date I will link to it from here.

Today, I was flicking through the entry on “Constructivist Mathematics” in the Stanford Encyclopedia of Philosophy and found a simple example of some of the problems with non-constructive take on what disjunction means in mathematical statements. The article calls it “well-worn” but I hadn’t seen it before.

Consider the statement:

There exists irrational numbers a and b such that a^b is rational.

The article gives a slick proof that this statement is true by invoking the law of the excluded middle (LEM). That is, every number must be either rational or irrational.

Now consider . By the LEM, this must rational or irrational:

• Case 1: If it is rational then we have proved the statement since we know is irrational.

• Case 2: If is irrational then choosing and as our two irrational numbers gives — a rational number.

Either way, we’ve proven the existence of two irrational numbers yielding a rational one. The problem with this is that this argument is non-constructive and so we don’t know which of case 1 and case 2 is true, we only know that one of them must be¹. This is a simple case of reductio ad absurdum in disguise.

As a born-again computer scientist (my undergraduate degree was pure maths and my PhD in computer science) I’ve become increasingly suspicious of these sorts of proof and more constructivist — even intuitionist — in my tastes. I think the seed of doubt was planted during the awkward discussions of the Axiom of Choice in my functional analysis lectures. The sense of unease is summed up nicely in the following joke:

The Axiom of Choice is obviously true, the well-ordering principle obviously false, and who can tell about Zorn’s lemma?

Of course, all those concepts are equivalent but that’s far from intuitive.

I don’t think I’m extremist enough to take a wholeheartedly computational view of mathematics — denying all but the computable real numbers and functions, thereby making all functions continuous — but it is a tempting view of the subject.

In machine learning, I think there is a fairly pragmatic take on the philosophy of mathematics. For example, classical theorems from functional analysis are used to derive results involving kernels but when it comes to implementation, estimations and approximations are used with abandon. In my opinion, this is a healthy way for the theory in this area to proceed. As in physics, if the experimental work reveals inconsistencies with a theory, revisit the maths. If that doesn’t work, talk to the philosophers.

If a reader cannot point to some tangible outcome from reading a book, then the reader may be overestimating the personal impact of the book.

With that in mind I tried to think of which books had a substantial impact on my research career.

Although I can barely remember any of it now, the manual that came with the Commodore Vic 20 computer I read when I was around seven got me hooked on programming. In primary and secondary school it was that book and the subsequent Commodore 64 and Amiga manuals that set me on the road to studying computer science and maths.

In my second year at university I had the great fortune of being recommended Hofstadter’s “Gödel, Escher, Bach” by a fellow student. It is centrally responsible for getting me to start thinking about thinking and subsequently doing a PhD in machine learning. The fanciful but extremely well written detours into everything from genetics to Zen Buddhism also broadened my horizons immensely.

I. J. Good’s “The Estimation of Probabilities” was the tiny 1965 monograph I bought second-hand for $2 that made my thesis take a huge change in direction by giving it a Bayesian flavour. I now realise that a lot of that work had since been superseded by much more sophisticated Bayesian methods but sometimes finding a theory before it has been over-polished means that there is much more expository writing to aid intuition. It also helps that Good is a fabulous technical writer.

Philosophically, Nelson Goodman’s “Fact, Fiction and Forecast” also shaped my thinking about induction quite a lot. His ideas on the “virtuous circle” of basing current induction on the successes and failures of the past provided me with a philosophical basis for the transfer learning aspects of my research. I found his views a refreshing alternative to Popper’s (also personally influential) take on induction in “The Logic of Scientific Discovery“. Whereas Popper beautifully characterises the line between metaphysical and scientific theories, Goodman tries to give an explanation of how we might practically come up with new theories in the first place given that there will be, in general, countless that adequately fit the available data. In a nutshell, his theory of “entrenchment” says that we accrete a network of terms by induction and use these terms as features for future induction depending on how successful they were when used in past inductive leaps. This is a view of induction inline with Hume’s “habits of the mind” and one I find quite satisfying.

While not directly related to machine learning or computer science, there are a few other books that helped me form opinions on the process of research in general. I read Scott’s “Write to the Point” over a decade ago now but it still makes me stop, look at my writing and simplify it. My attitude to presenting technical ideas was also greatly influenced by reading Feynman’s “QED” lectures. They are a perfect example of communicating extremely deep and difficult ideas to a non-technical audience without condescension and misrepresentation. Finally, I read Kennedy’s “Academic Duty” just as I started my current post-doc and found it immensely insightful. I plan to reread it as I (hopefully) hit various milestone’s in my academic career.

Of course, like Peter, there are innumerable other books, papers and web pages that have shaped my thinking but the ones above are the ones that leap to mind when I think about how my research interests have developed over time.