When was the last time you enjoyed calling your cable company? You work through the phone tree, explain your problem to the first person, who moves you to a second person, to whom you explain it again, who then needs to transfer you to a third. Somewhere around the fourth handoff you are reciting your account number from memory and have stopped expecting anything good to happen.

Nearly 100 years ago the Bavarian comedian Karl Valentin built a skit out of exactly this in Buchbinder Wanninger: a bookbinder phones a construction company to ask one trivial question, whether he should enclose his invoice with the books he is delivering, and gets passed from clerk to clerk to clerk, each one perfectly polite and not one of them able to answer. The bit is still funny because nothing about it has aged (Germans still reference the scene today), only the hold music is better now.

And the unhelpful transfer is the good case, because at least there was someone to transfer you to. Public administration, claims processing, after-hours support: in more and more places there simply are not enough people to do the work, and the gap is widening. The problem is not only that talking to the system is unpleasant and that the scripts are incredibly rigid. It is that increasingly there is no one on the other end at all.

This is the opportunity for digital humans. Not a chatbot bolted onto an FAQ, but something that can hold up the human side of an interaction from end to end: understand what you actually want, be reachable however you like to reach out, be actually pleasant to interact with, and get the thing done.

Years ago I wrote about correcting covariate shift by reweighting your data. Your features come from the wrong distribution , you care about a target , so you weight every observation by and your estimates are unbiased again. I ended that post by admitting the weights “can be quite a bit off,” and waved at fixing it another day.

Here is the more basic question I skipped. Even when the weights are exactly right, what do they cost you? Reweighting buys you less bias. It charges you variance, and the currency of that charge has a name.

A few days ago I wrote about conversational proactivity: a model noticing what you disclosed but never asked about, and turning it into something useful. The running example was a user signing off with “plain-text packet is final. I’ll send tonight and load the hatchback after work.” One model says “drive safe.” The other hands back a packing list, loaded in reverse order of installation. I promised numbers in the follow-up. Here they are.

But the numbers are the easy part. The reason this took a paper is the question I skipped last time: how do you measure “noticing what the user didn’t say”? Every benchmark you know works the same way. A question goes in, an answer comes out, you grade the answer. That machinery is useless here, because proactivity is exactly the part that isn’t in the question. You cannot grade the answer to a question nobody asked.

A user has spent ten minutes preparing for a small art installation. The final exchange goes like this.

User: Plain-text packet is final. I’ll send tonight and load the hatchback after work.

Model A: Sounds like a clean plan. Drive safe, and go get it.

Model B: Before you go, a fast final check: projector + power cable; media player + charger; file tested and looping; tripod + mounting plate; extension cord; adapter; gaffer tape; measuring tape; small level; wall label on phone; backup copy on USB. If you can, pack the car tonight so tomorrow is just: grab computer, leave, install. Load in reverse order of install — the projector stand and extension cord go in last, so they’re the first things out.

Both responses are fluent. Both end the conversation politely. The first is what most assistants do. The second one notices the disclosed detail — load the hatchback — and turns it into something the user didn’t ask for but will probably need.

One of my favorite interview questions for network engineers (no longer, now that I’m about to spoil it): you have 512 GPUs across 64 servers, eight GPUs per box. Your switches are NVIDIA Quantum-2 class, MQM9700, sixty-four 400Gb/s NDR ports each. Wire it for full cross-sectional bandwidth. How many switches, what topology, and where do you put the fabric manager?

The textbook part is a two-tier fat tree, also known as spine and leaf. With 64-port switches, split each leaf into 32 ports facing GPUs and 32 ports facing the spine. That gives 16 leaf switches ( endpoints) and 8 spine switches (each leaf sends links to each spine; each spine ends up with down, filling the box). Bisection is full, switch count is 24. So far, so easy.

The natural reaction to the Pokémon theorem is to escape into representation learning. If finite scalar checklists cannot catch every fairness violation, then bake fairness into the features once. Ship a single encoder and let every downstream predictor inherit fairness for free. There is a real and serious literature behind exactly this idea: Zemel et al. (2013), Edwards and Storkey (2016), Louizos et al. (2016), Madras et al. (2018). It is a beautiful agenda, and it has a serious flaw …

Every time a new model comes out, somebody runs it on MMLU (57 subjects), MTEB (56 tasks), HELM, the Open LLM Leaderboard, AlpacaEval, LiveBench, BigCodeBench, WildBench, Arena-Hard, MT-Bench, and a dozen others. That’s days of GPU time and a lot of human babysitting. But if you’ve ever stared at a leaderboard for ten minutes you already know the dirty secret: the columns are wildly correlated. If a model is good at one math benchmark it’s good at all of them. So how much of this can we just skip?

A lot, as it turns out. On MMLU, 5 subjects out of 57 predict the remaining 52 with , across 5,452 models, with 10-fold cross-validation. The eigenspectrum of the score covariance tells the same story: two components capture 90% of the variance on MMLU, six on MTEB. Benchmark scores live in a low-dimensional subspace.

The question is which subset, and how to pick it.

If you have been near algorithmic fairness for the past decade, you already know the punchline: you cannot have all the things you want. Calibration, class-conditional balance, and predictive parity refuse to coexist whenever base rates differ across groups. This is the impossibility trinity of Kleinberg, Mullainathan and Raghavan (2017), Chouldechova (2017) and Pleiss et al. (2017). Barocas, Hardt and Narayanan collect the pairwise tensions among independence, separation and sufficiency into what they call the incompatibility triangle. Hutchinson and Mitchell (2019) survey the fifty years of educational-testing impossibility work that came before all of this. The 1970s knew most of the punchline. None of us listened.

There is an obvious question that the trinity leaves open. What if we only impose scalar criteria, rather than distributional ones. Will a few of them suffice to ensure fairness? After all, this is what individual scalar metrics such as college acceptance rates, recidivism rates, etc. aim to control for, only to be thwarted again by another study. With dozens of criteria, with hundreds, with every reasonable test anyone has ever proposed, can we close every gap? It turns out we cannot. The argument is a very simple piece of linear algebra.

I’m happy to announce our new book project - Dive into Deep Learning. It’s still in beta stage, i.e. we’re still working on it. That said, I think that it’s good enough to share with friends and colleagues. This is joint work with Aston Zhang, Mu Li, and Zachary Lipton.

There’s an obvious question - why yet another machine learning book? After all, there’s no shortage of great books, e.g. Machine Learning by Kevin Murphy, Pattern Recognition and Machine Learning by Chris Bishop, Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville, Information Theory, Inference and Learning Algorithms by the late David MacKay, and many other great books. At the same time, there are ready made recipe books for various deep learning frameworks, such as Learning Tensorflow by Itay Lieder, Yehezkel Resheff, and Tom Hope. However, none of the books so far attempted to bridge this gap between theory and practice. This is what the current book project aims to address. In particular, it combines the following features:

• Downloadable Jupyter notebooks. In fact, the entire book consists of notebooks.
• A freely available PDF version
• A GitHub repository to allow for fast corrections of errata
• A tight integration with discussion forums to allow for questions regarding the math and code on the site
• Theoretical background suitable for engineers and undergraduate researchers
• State of the art models (including ResNet, faster-RCNN, etc)
• Well documented and structured code that is executed on real datasets, yet at the same time small enough to fit on a laptop.
• A Chinese translation (in fact, the Chinese book will be released first)

In addition to that, Mu and I will be teaching a class at UC Berkeley in Spring. As part of that, we will be releasing slides, videos and assignments, suitable for reuse for anyone who’d like to do so. In short, we aim to offer a complete resource to learn deep learning, easily and a comprehensive manner. Please let us know what you think.

Dear Friends,
As some of you may have already heard, I’m leaving CMU to join Amazon, effective July 1, 2016. There I will be in charge of Amazon’s Cloud Machine Learning Platform with the task to make machine learning as easy to use and widespread as it could possibly be. This is a terrific task and it was an offer that I could not turn down. Our lab will be in the Bay Area and we will strive to turn the state of the art in machine learning research into the state of the art in industry. Both in terms of scale and in terms of model sophistication. This is a very exciting time and I’m looking forward to it. If all goes well, this will raise the bar also in academia.

I wanted to say thanks to everyone who’s helped us along on this journey. First off, an absolutely outstanding cast of PhD students I’ve had the pleasure of working with. Mu, Manzil, Zichao, Fish, Yu-Xiang, Seth, Alex, Ziqi, Wei, Yining, Chao-Yuan, Sashank, you guys rock! And yes, most of you are smarter than me, and this has been utterly delightful. Second, I wanted to thank to thank the terrific CMU faculty. I learned a lot and the past 4 years have been a wonderful experience. Many thanks, in particular to Dave, Andy, Andrew, Tom, Geoff, Nina, Christos, Ryan, Barnabas, Larry, Jaime and Kayvon. Many thanks for your advice, help, insight, and collaboration. What we achieved in the past 4 years would not have been possible without a great team. And thanks to Mallory and Diane to keep the ship running no matter what, and thanks to Russ for keeping the accounting in good shape and for putting up with last minute updates. Apologies if I forgot someone on this list (I probably have). It has been wonderful to work with you and I dearly love CMU. So why the change? Here’s the reasoning that went into deciding to go to Amazon: Our goal as machine learning researchers is to solve deep problems (not just in deep learning) and to ensure that this leads to algorithms that are actually used. At scale. At sophistication. In applications. The number of people I could possibly influence personally through papers and teaching might be 10,000. In Amazon we have 1 million developers using AWS. Likewise, the NSF thinks that a project of 3 engineers is a big grant (and it is very choosy in awarding these grants). At Amazon we will be investing an order of magnitude more resources towards this problem. With data and computers to match this. This is significant leverage. Hence the change.

We will try to give back to the academic community, to contribute back, e.g. through open source. And the goal is to engage it more deeply. I cannot give more specifics yet but good things will happen. Stay tuned.

Technically I will be on leave of absence until August 2017. This is mostly to ensure that the students are taken care of and that grants are in good shape. I’ll be reaching out to many of you directly (the reason why I haven’t done this to all of you yet is that I’m in the middle of moving).

Let’s stay in touch. And, obviously, if you’re interested in joining me on this journey (or have students who would like to come along), please let me know.

Best, Alex

PS: I’m posting this letter (which was intended for my team at CMU and my colleagues there) after screenshots of parts of it seem to have gone live on Weibo.

One of the key features of a parameter server is that it, well, serves parameters. In particular, it serves more parameters than a single machine can typically hold and provides more bandwidth than what a single machine offers. A sensible strategy to increase both aspects is to arrange data in the form of a bipartite graph with clients on one side and the server machines on the other. This way bandwidth and storage increase linearly with the number of machines involved. This is well understood. For instance, distributed (key,value) stores such as memcached or Basho Riak use it. It dates back to the ideas put forward by David Karger et al. on Consistent Hashing and Random Trees in STOC 1997.

A key problem is that we can obviously not store a mapping table from the keys to the machines. This would require a database that is of the same size as the set of keys and that would need to be maintained and updated on each client. One way around this is to use the argmin hash mapping. That is, given a machine pool , we assign a given (key,value) pair to the machine that has the smallest hash, i.e.

The advantage of this scheme is that it allows for really good load balancing and repair. First off, the load is almost uniformly distributed, short of a small number of heavy hitters. Secondly, if a machine is removed or added to the machine pool, rebalancing affects all other machines uniformly. To see this, notice that the choice of machine with the smallest and second-smallest hash value is uniform.

Unfortunately, this is a stupid way of distributing (key,value) pairs for machine learning. And this is what we did in our 2010’ VLDB and 2012’ WSDM papers. To our excuse, we didn’t know any better. And others copied that approach … after all, how you can you improve on such nice rebalancing aspects.

This begs the question why it is a bad idea. It all comes down to the issue of synchronization. Basically, whenever a client attempts to synchronize its keys, it needs to traverse the list of the keys it owns and communicate with the appropriate servers. In the above scheme, it means that we need to communicate to a new random server for each key. This is amazingly costly. Probably the best comparison would be a P2P network where each byte is owned by a different machine. Downloads would take forever.

We ‘fixed’ this problem by cleverly reordering the access and then performing a few other steps of randomization. There’s even a nice load balancing lemma in the 2012 WSDM paper. However, a much better solution is to prevent the problem from happening and to borrow from key distribution algorithms such as Chord. In it, servers are inserted into a ring via a hash function. So are keys. This means that each server now owns a contiguous segment of keys. As a result, we can easily determine which keys go to which server, simply by knowing where in the ring the server sits.

In the picture above, keys are represented by little red stars. They are randomly assigned using a hash function via h(k) to the segments ‘owned’ by servers s that are inserted in the same way, i.e. via h(s). In the picture above, each server ‘owns’ the segment to its left. Also have a look at the Amazon Dynamo paper by DeCandia et al., 2007 SOSP for a related description.

Obviously, such a load-balancing isn’t quite as ideal as the argmin hash. For instance, if a machine fails, the next machine inherits the entire segment. However, by inserting each server log(n) times we can ensure that a good load balance is achieved and also that when machines are removed, there are several other machines that pick up the work. Moreover, it is now also very easy to replicate things (more on this later). If you’re curious on how to do this, have a look at Amar Phanishayee’s excellent thesis. In a nutshell, the machines to the left hold the replicas.

We’ve been busy building the next generation of a Parameter Server and it’s finally ready. Check out the OSDI 2014 paper by Li et al.; It’s quite different from our previous designs, the main improvements being fault tolerance and self repair, a much improved network protocol, flexible consistency models, and a much more general interface.

In the next few posts I’ll explain the engineering decisions that went into this system which is capable of solving problems as diverse as very high throughput sketching, topic models, and optimization. And yes, it is open source.

Update (11/2022): some things have gotten a lot faster over the past decade. For instance, SSDs now regularly deliver multiple GB/s. The number of requests has increased dramaticaly, too, to several 100k IOPS (IO Operations Per Second). For a recent comparison check out this Anandtech benchmark. At the same time, memory bandwidth has increased to 100-1000 GB/s, depending on the system. Many variants of Jeff Dean’s latency numbers have sprung up. One of the more recent ones is the one by Colin Scott.

Ok, so what does this mean for machine learning? If you can keep things in memory, you can do things way faster. This is one of the key ideas behind Spark. It’s a wonderful alternative to Hadoop. In other words, if your data fits into memory, you’re safe and you can process data way faster. A lot of datasets that are considered big in academia fit this bill (in 2022 you can easily rent or buy 256-1024GB instances). But what about real big data? Essentially you have two options - have the systems designer do the hard work or change your algorithm. This post is about the latter. And yes, there’s a good case to be made about who should do the work: the machine learners or the folks designing the computational infrastructure (I think it’s both).

So here’s the problem: Many online algorithms load data from disk, stream it through memory as efficiently as possible and discard it after seeing it once, only to pick it up later for another pass through the data. That is, these algorithms are disk bound rather than CPU bound. Several solvers try to address this by making the disk representation more efficient, e.g. LibLinear or VowpalWabbit, both of which use their own internal representation for efficiency. While this still makes for quite efficient code that can stream up to 10-100TB of data per hour, in any given pass, main memory is still much faster. This has led to the misconception that many machine learning algorithms are disk bound. But, they aren’t …

What if we could re-use data that’s in memory? For instance, use a ringbuffer where the disk writes into it (much more slowly) and the CPU reads from it (100 times more rapidly). The problem is what to do with an observation that we’ve already processed. A naive strategy would be to pretend that it is a new instance, i.e. we could simply update on it more than once. But this is very messy since we need to keep track of how many times we’ve seen the instance before, and it creates nonstationarity in the training set.

A much cleaner strategy is to switch to dual variables, similar to the updates in the Dualon of Shalev-Shwartz and Singer. This is what Shin Matsushima did in our dual cached loops paper (the StreamSVM implementation unfortunately has fallen into disrepair in the past decade). Essentially, it keeps data in memory in a ringbuffer and updates the dual variables. This way, we’re guaranteed to make progress at each step, even if we’re revisiting the same observation more than once. To see what happens have a look at the graph below:

It’s just as fast as LibLinear provided that it’s all in memory. Algorithmically, what happens in the SVM case is that one updates the Lagrange multipliers αi, while simultaneously keeping an estimate of the parameter vector w available.

That said, this strategy is more general: reuse data several times for optimization while it is in memory. If possible, perform successive updates by changing variables of an optimization that is well-defined regardless of the order in which (and how frequently) data is seen.

The Weisfeiler-Leman algorithm and estimation on graphs Imagine you have two graphs and and you’d like to check how similar they are. If all vertices have unique attributes this is quite easy:

• forall vertices do
  • check that and that
  • check that the neighbors of are the same in and

This algorithm can be carried out in linear time in the size of the graph. Unfortunately, many graphs do not have vertex attributes, let alone unique vertex attributes. In fact, graph isomorphism, i.e. the task of checking whether two graphs are identical, is a hard problem (it is still an open research question how hard it really is). In this case the above algorithm cannot be used since we have no idea which vertices we should match up.

The Weisfeiler-Leman algorithm (see David Bieber’s blog post for a pretty visualization) is a mechanism for assigning fairly unique attributes efficiently. Note that it isn’t guaranteed to work, as discussed in this paper by Douglas in 2011. This would solve the graph isomorphism problem after all. The idea is to assign fingerprints to vertices and their neighborhoods repeatedly. We assume that vertices have an attribute to begin with. If they don’t then simply assign all of them the attribute 1. Each iteration proceeds as follows:

• forall vertices do
  • compute a hash of where are the attributes of the neighbors of vertex .
  • use the hash as vertex attribute for in the next iteration.

The algorithm terminates when this iteration has converged in terms of unique assignments of hashes to vertices. Note that it is not guaranteed to work for all graphs. In particular, it fails for graphs with a high degree of symmetry, e.g. chains, complete graphs, tori and stars. However, whenever it converges to a unique vertex attribute assignment it provides a certificate for graph isomorphism. Moreover, the sets of vertex attributes can be used to show that two graphs are not isomorphic (it suffices to verify that the sets differ at any stage).

Shervashidze et al. 2012 use this idea to define a similarity measure between graphs. Basically the idea is that graphs are most similar if many of their vertex identifiers match since this implies that the associated subgraphs match. Formally they compute a kernel using

Here denote the vertex attribute of after WL iteration . Morevoer, are nonnegative coefficients that weigh how much the similarity at level d matters. Rather than a brute-force computation of the above test for equality we can thus sort vertex attribute sets. Note that vertices that have different attributes at any given iteration will never have the same attribute thereafter. This means that we can compare the two sets at all depths at at most cost.

A similar trick is possible if we want to regress between vertices on the same graph since we can use the set of attributes that a vertex obtains during the iterations as features. Finally, we can make our life even easier if we don’t compute kernels at all and use a linear classifier on the vertex attributes directly.

In defense of keeping data private This is going to be contentious. And it somewhat goes against a lot of things that researchers hold holy. And it goes against my plan of keeping philosophy out of this blog. But it must be said since remaining silent has the potential of damaging science with proposals that sound good and are bad.

The proposal is that certain conferences make it mandatory to publish datasets that were used for the experiments. This is a very bad idea and two things are getting confused here: scientific progress and common access. These two are not identical. Reproducibility is often confused with common access. To make these things a bit more clear, here’s an example where it’s more obvious:

CERN is a monster machine. There’s only one of its kind in the world. There are limited resources and it’s impossible for any arbitrary researcher to reproduce their experiments, simply because of the average physicist being short of the tens of billions of Dollars that it took to build it. Access to the accelerator is also limited. It requires qualification and resource planning. So, even if we think this is open, it isn’t really as open as it looks. And yes, working at CERN gives you an unfair advantage over all the researchers who don’t.

Likewise take medical research. Patient records are covered by HIPAA privacy constraints and there is absolutely no way for such records to be publicly released. The participants sign an entire chain of documents that tie them to not releasing such data publicly. In other words, common access is impossible. Reproducibility would require that someone, who wants to test a contentious result, needs to sign corresponding privacy documents before accessing the data. And yes, working with the ‘right’ hospitals gives you an unfair advantage over researchers who didn’t work building this relationship.

Lastly, user data on the internet. Users have every right for their comments, content, images, mails, etc. to be treated with the utmost respect and to be published only when it is in their interest and with their permission to do so. I believe that there is a material difference between data being made available for analytics purposes in a personalization system and data being made available ‘in the raw’ for any researcher to play with. The latter allows for individuals to inspect particular records and learn that Alice mailed Bob a love letter. Something that would make Charlie very upset if he found out. Hence common access is a non-starter.

There are very clear financial penalties for releasing private data (hello Cambridge Analytica) - users would leave the service. Moreover, it would give a competitor an advantage over the releasing party. Since the data is largely collected by private parties at their expense it is not possible.

As for reproducibility - this is an issue. But provided that in case of a contentious result it is possible for a trusted researcher to check them, possibly after signing an NDA, this can be addressed. And yes, working for one of these companies gives you an unfair advantage.

In summary, while desirable, I strongly disagree with a mandatory publications policy. Yes, every effort should be made personally by researchers to see whether some data is releasable. And for publicly funded research this may well be the right thing to do. But to mandate it for industry would essentially do two things - it will make industrial research even more secretive than it already is (and that’s a terrible thing). And secondly, it will make academic research less relevant for real problems (I’ve seen my fair share and am guilty of my fair share of such papers).

The videos from MLSS 2011 in Purdue are now available online. Unfortunately the 2011 MLSS website is gone and with it all the playlists and PDFs. The only thing left is a group photo and departmental announcement. But you can still find the videos courtesy of YouTube. Some of them have aged very well.

Many algorithms require random number generators to work. For instance, locality sensitive hashing requires one to compute the random projection matrix in order to compute the hashes . Likewise, fast eigenvalue solvers in large matrices often rely on a random matrix, e.g. the work by Halko, Martinsson and Tropp, SIAM Review 2011, which assumes that at some point we multiply a matrix M by a matrix P with Gaussian random entries.

The problem with these methods is that if we want to perform this projection operation in many places, we need to distribute the matrix to several machines. This is undesirable since a) it introduces another stage of synchronization between machines and b) it requires space to store the matrix in the first place. The latter is often bad since memory access can be much slower than computation, depending on how the memory is being accessed. The prime example here is multiplication with a sparse matrix which would require random memory access.

One way to circumvent this is to share the random seed and then recompute the random matrix from scratch. But this means that we’re critically relying on the implementation of a random number generator. Even worse, we still need to store the entire matrix. What if we could simply access any element of the matrix at will without overhead?

Here’s where hashing comes to the rescue. To motivate things consider the case where the entries of are all drawn from the uniform distribution . For a hash function h with range simply set . Since hash functions map pairs to uniformly distributed, uncorrelated numbers in the range this essentially amounts to uniformly distributed random numbers that can be recomputed on the fly.

A slightly more involved example is how to draw Gaussian random variables. We may e.g. resort to the Box-Müller transform which shows how to convert two uniformly distributed random numbers into two Gaussians (the image on top is copied from the Wikpedia article). While being quite wasteful (we use two random numbers rather than one), we simply use two uniform hashes and then compute

Since this is known to generate Gaussian random variables from uniform random variables this will give us Gaussian distributed hashes. Similar tricks work for other random variables. It means that things like Random Kitchen Sinks, Locality Sensitive Hashing, and related projection methods never really need to store the ‘random’ projection coefficients whenever memory is at a premium or whenever it would be too costly to synchronize the random numbers.

Update - recently someone proposed to use only zeros and ones in the initialization of a deep network with the express purpose of making things reproducible. The paper has a fair amount of analysis in it but all of this can be made redundant simply by using hash functions instead. It requires at most sharing of a ‘salt’ and an agreed-upon convention for referencing weights in the network (e.g. weight and coordinate). Presto a deterministic initialization that is reproducible without the need for fancy math.

So you read a book on Reproducing Kernel Hilbert Spaces and you’d like to try out this kernel thing. But you’ve got a lot of data and most algorithms will give you an expansion that requires a number of kernel functions linear in the amount of data. Not good if you’ve got millions to billions of instances.

You could try out low rank expansions such as the Nystrom method of Seeger and Williams, 2000, the randomized Sparse Greedy Matrix Approximation of Smola and Schölkopf, 2000 (the Nyström method is a special case where we only randomize by a single term), or the very efficient positive diagonal pivoting trick of Scheinberg and Fine, 2001. Alas, all those methods suffer from a serious problem: at training you need to multiply by the inverse of the reduced covariance matrix, which is cost for a dimensional expansion. An example of an online algorithm that suffers from the same problem is this (NIPS award winning) paper of Csato and Opper, 2002. Assuming that we’d like to have d grow with the sample size this is not a very useful strategy. Instead, we want to find a method which has cost for attributes yet shares good regularization properties that can be properly analyzed.

Enter Radford Neal’s seminal paper from 1994 on Gaussian Processes (a famous NIPS reject). In it he shows that a Neural Network with an infinite number of nodes and a Gaussian Prior over coefficients converges to a GP. More specifically, we get the kernel

Here is a function parametrized by , e.g. the location of a basis function, the degree of a polynomial, or the direction of a Fourier basis function. There is also a discussion regarding RKHS in a paper by Smola, Schölkof and Müller, 1998 that discusses this phenomenon in regularization networks. These ideas were promptly forgotten by its authors. One exception is the empirical kernel map where one uses a generic design matrix that is generated through the observations directly.

It was not until the paper by Rahimi and Recht, 2008 on random kitchen sinks that this idea regained popularity. In a nutshell the algorithm works as follows: Draw values from the distribution over . Use the corresponding basis functions in a linear model with quadratic penalty on the expansion coefficients. This method works whenever the basis functions are well bounded. For instance, for the Fourier basis the functions are bounded by . The proof of convergence of the explicit function expansion to the kernel is then a simple consequence of Chernoff bounds.

In the random kitchen sinks paper Rahimi and Recht discuss RBF kernels and binary indicator functions. However, this works more generally for any set of well behaved set of basis functions used in generating a random design matrix. A few examples:

• Fourier basis with Gaussian parameters. Take functions of the form where the coefficients are drawn from a Gaussian. This is the random kitchen sinks paper. Obviously you can use hash functions rather than an actual random number generator. This ensures that you don’t need to store all parameters .
• Pick random separating hyperplanes. This will effectively give you functions of bounded variation. Use the empirical kernel map, i.e. we use some function for which we employ for a random subset of the data we wish to train on.
• Pick suitable binary features obtained via binning (the figure on top, taken from Rahimi and Recht, 2008, shows quite impressive results in lower dimensions).

We’re organizing a workshop at NIPS 2011. Submission are solicited for a two day workshop December 16-17 in Sierra Nevada, Spain.

This workshop will address tools, algorithms, systems, hardware, and real-world problem domains related to large-scale machine learning (“Big Learning”). The Big Learning setting has attracted intense interest with active research spanning diverse fields including machine learning, databases, parallel and distributed systems, parallel architectures, and programming languages and abstractions. This workshop will bring together experts across these diverse communities to discuss recent progress, share tools and software, identify pressing new challenges, and to exchange new ideas. Topics of interest include (but are not limited to):

• Hardware Accelerated Learning: Practicality and performance of specialized high-performance hardware (e.g. GPUs, FPGAs, ASIC) for machine learning applications.
• Applications of Big Learning: Practical application case studies; insights on end-users, typical data workflow patterns, common data characteristics (stream or batch); trade-offs between labeling strategies (e.g., curated or crowd-sourced); challenges of real-world system building.
• Tools, Software, & Systems: Languages and libraries for large-scale parallel or distributed learning. Preference will be given to approaches and systems that leverage cloud computing (e.g. Hadoop, DryadLINQ, EC2, Azure), scalable storage (e.g. RDBMs, NoSQL, graph databases), and/or specialized hardware (e.g. GPU, Multicore, FPGA, ASIC).
• Models & Algorithms: Applicability of different learning techniques in different situations (e.g., simple statistics vs. large structured models); parallel acceleration of computationally intensive learning and inference; evaluation methodology; trade-offs between performance and engineering complexity; principled methods for dealing with large number of features;

Submissions should be written as extended abstracts, no longer than 4 pages (excluding references) in the NIPS style. Relevant work previously presented in non-machine-learning conferences is strongly encouraged. Exciting work that was recently presented is allowed, provided that the extended abstract mentions this explicitly.

Update - the website no longer exists but YouTube has a list of all the talks.

Here are the slides [Keynote, PDF] for a basic course on Graphical Models for the Internet that I’m giving at MLSS 2011 in Purdue that Vishy Vishwanathan is organizing. The selection is quite biased, limited, and subjective, but it’s meant to complement the other classes at the summer school.

The slides are likely to grow, so in case of doubt, check for updates. Comments are most welcome. And yes, it’s a horribly incomplete overview, due to space and time constraints.

Here’s a simple synchronization paradigm between many computers that scales with the number of machines involved and which essentially keeps cost at per machine. For lack of a better name I’m going to call it the distributed star since this is what the communication looks like. It’s quite similar to how memcached stores its (key,value) pairs.

Assume you have n computers, each of which have a copy of a large parameter vector (typically several GB) and we would like to keep these copies approximately synchronized.

A simple version would be to pause the computers occasionally, have them send their copies to a central node, and then return with a consensus value. Unfortunately this takes time if we aggregate things on a tree (we can reduce it by streaming data through but this makes the code a lot more tricky). Furthermore we need to stop processing while we do so. The latter may not even be possible and any local computation is likely to benefit from having most up-to-date parameters.

Instead, we use the following: assume that we can break up the parameter vector into smaller (key, value) pairs that need synchronizing. We now have each computer send its local changes for each key to a central server, update the parameters there, and later receive information about global changes. So far this algorithm looks stupid - after all, when using n machines it would require time to process since the central server is the bottleneck. This is where the distributed star comes in. Instead of keeping all data on a single server, we use the well known distributed hashing trick and send it to a machine n from a pool P of servers:

Here is the hash function. Such a system spreads communication evenly and it leads to an load per machine. In particular, if we make each of the computers involved in the local computation also members of the pool, i.e. if we have we get an cost for keeping terms synchronized regardless of the number of machines involved.

Obvious approximations: we assume that all machines are on the same switch. Moreover we assume that the times to open a TCP/IP connection are negligible (we keep them open after the first message) relative to the work to transmit the data.

The reason I’m calling this a distributed star is that for each key we have a star communication topology, it’s just that we use a different star for each key. If anyone in systems knows what this thing is really called, I’d greatly appreciate feedback. Memcached uses the same setup, alas it doesn’t have versioned writes and callbacks, so we had to build our own system using ICE.

PS: thanks to the Hubble Space telescope for the image of a star distribution in a bright cluster.

The code to our LDA implementation on Hadoop is released on Github under the Mozilla Public License. It’s seriously fast and scales very well to 1000 machines or more (don’t worry, it runs on a single machine, too). We believe that at present this is the fastest implementation you can find, in particular if you want to have a) 1000s of topics, b) a large dictionary, c) a large number of documents, and d) Gibbs sampling. It handles quite comfortably a billion documents. Shravan Narayanamurthy deserves all the credit for the code. The paper describing an earlier version of the system appeared in VLDB 2010.

Some background: Latent Dirichlet Allocation by Blei, Jordan and Ng, 2003 is a great tool for aggregating terms beyond what simple clustering can do. While the original paper showed exciting results it wasn’t terribly scalable. A significant improvement was the collapsed sampler of Griffiths and Steyvers, 2004. The key idea was that in an exponential families model with conjugate prior you can integrate out the natural parameter, thus providing a sampler that mixed much more rapidly. It uses the following update equation to sample the topic for a word.

Here denotes the topic, the document, the word, and , , , denote the number of words which satisfy a particular (topic, document), (document), (topic, word), (topic) combination respectively. The starred quantities such as simply mean that we use the counts where the current word for which we need to resample the topic is omitted.

Unfortunately the above formula is quite slow when it comes to drawing from a large number of topics. Worst of all, it is nonzero throughout. A rather ingenious trick was proposed by Yao, Mimno, and McCallum, 2009. It uses the fact that the relevant terms in the sum are sparse and only the and -dependent terms are dense (and obviously the number of words per document doesn’t change, hence we can drop that, too). Dropping the common denominator we arrive at

Out of these three terms, only the first one is dense, all others are sparse. Hence, if we knew the sum over for all three summands we could design a sampler which first samples which of the blocks is relevant and then which topic within each of these blocks. This is efficient since the first term doesn’t actually depend on or but rather only on which can be updated efficiently after each new topic assignment. In other words, we are able to update dense term in operations after each sampling step and the remaining terms are all sparse. This trick gives a 10-50 times speedup in the sampler over a dense representation.

To combine several machines we have two alternatives: one is to perform one sampling pass over the data and then reconcile the samplers. This was proposed by Newman, Asuncion, Smyth, and Welling, 2009. While the approach proved to be feasible, it has a number of disadvantages. It only exercises the network while the CPU sits idle and vice versa. Secondly, a deferred update makes for slower mixing. Instead, one can simply have each sampler communicate with a distributed central storage continuously. In a nutshell, each node sends the differential to the global statekeeper and receives from it the latest global value. The key point is that this occurs asynchronously and moreover that we are able to decompose the state over several machines such that the available bandwidth grows with the number of machines involved. More on such distributed schemes in a later post.

Bloom filters are one of the really ingenious and simple building blocks for randomized data structures. A great summary is the paper by Broder and Mitzenmacher, 2005. The figure above is from their paper. In this post I will briefly review its key ideas since it forms the basis of the Count-Min sketch of Cormode and Muthukrishnan, 2003 it will also be necessary for an accelerated version of the graph kernel of Shervashidze et al., 2011, and finally, a similar structure will be needed to compute data streams over time for a real-time sketching service.

At its heart a Bloom filter uses a bit vector of length and a set of hash functions mapping arbitrary keys into their hash values where denotes the hash function. The Bloom filter allows us to perform approximate set membership tests where we have no false negatives but we may have a small number of false positives.

• Initialize(b)
  Set all
• Insert(b,x)
  For all set
• Query(b, x)
  Return true if for all . Return false otherwise.

Furthermore, unions and intersections between sets are easily achieved by performing bit-wise OR and AND operations on the bloom hashes of the corresponding sets respectively. Since these are bit-wise operations, they can be very fast, essentially operating at memory speed.

It is clear that if we inserted into the Bloom filter the query will return true, since all relevant bits in are set to . But we could get unlucky. To analyze the probability of a false positive take the probability of a bit being 1. After inserting items using hash functions over a range of we have

For a false positive to occur we need to have all k bits associated with the hash functions to be 1. Ignoring the fact that the hash functions might collide, i.e.  for some , the probability of false positives is given by

Taking derivatives with respect to shows that the minimum is obtained for , that is . In other words, decreases with the fill rate of the Bloom filter. This makes sense since a higher fill rate leads to a higher collision probability and we shouldn’t try to ‘overfill’ the filter. Plugging the optimal value back into yields , i.e. the quality of the array decreases exponentially with the fill rate, albeit slowly at rate .

One of the really nice properties of the Bloom filter is that all memory is used to store the information about the set rather than an index structure storing the keys of the items. The downside is that it is impossible to read out without knowing the queries. Also note that it is impossible to remove items from the Bloom filter once they’ve been inserted. After all, we do not know whether some of the bits might have collided with another key, hence setting the corresponding bits to 0 would cause false negatives.

Imagine you want to design some algorithm to detect cancer. You get data of healthy and sick people; you train your algorithm; it works fine, giving you high accuracy and you conclude that you’re ready for a successful career in medical diagnostics. Not so fast. Many things could go wrong. In particular, the distributions that you work with for training and those in the wild might differ considerably.

This happened to an unfortunate startup I had the opportunity to consult for many years ago. They were developing a blood test for prostate cancer, a disease that affects mainly older men. Luckily for them they’d managed to obtain a fair amount of blood samples from patients. It is considerably more difficult, though, to obtain blood samples from healthy men, mainly for ethical reasons. For instance, what do you do with positives? To compensate for that, they asked a large number of students on campus to donate blood and they performed their test. Then they asked me whether I could help them build a classifier to detect the disease. I told them that it would be very easy to distinguish between both datasets with probably near perfect accuracy. After all, the test subjects differed in age, hormone level, physical activity, diet, alcohol consumption, and many more factors unrelated to the disease. This was unlikely to be the case with real patients: Their sampling procedure had caused an extreme case of covariate shift that couldn’t be corrected by conventional means. In other words, training and test data were so different that nothing useful could be done and they had wasted significant amounts of money.

In general the situation is not quite so dire. Assume that we want to estimate some dependency for which we have labeled data . Alas, the observations are drawn from some distribution rather than the ‘proper’ distribution . If we adopt a risk minimization approach, that is, if we want to solve

we will need to re-weight each instance by the ratio of probabilities that it would have been drawn from the correct distribution. That is, we need to reweight things by . This is the ratio of how frequently the instances would have occurred in the correct set vs. how frequently it occurred with the sampling distribution . It is sometimes also referred to as the Radon-Nikodym derivative. Such a method is called importance sampling and the following derivation shows why it is valid:

Alas, we do not know . Vefore we can do anything useful we need to estimate the ratio. Many methods are available, e.g. some rather fancy operator theoretic ones which try to recalibrate the expectation operator directly using a minimum-norm or a maximum entropy principle. However, there exists a much more pedestrian, yet quite effective approach that will give almost as good results: logistic regression.

After all, we know how to estimate probability ratios. This is achieved by learning a classifier to distinguish between data drawn from and data drawn from . If it is impossible to distinguish between the two distributions then it means that the associated instances are equaly likely to come from either oneof the two distributions. On the other hand, any instances that can be well discriminated should be significantly over/underweighted accordingly. For simplicity’s sake assume that we have an equal number of instances from both distributions, denoted by and respectively. Now denote by labels which are for data drawn from and for data drawn from . Then the probability in a mixed dataset is given by

Hence, if we use a logistic regression approach which yields , it follows (after some simple algebra) that

Now we only need to solve the logistic regression problem (or use some convenient AutoML tool such as AutoGluon)

to obtain . Subsequently we can use as covariate shift correction weights in training our actual classifier. The good news is that we can use an off-the-shelf tool to deal with a decidedly nonstandard estimation problem. Note that the weights can be quite a bit off. This is fixable, e.g. by using an estimator that isn’t prone to this, or by using conformal methods. But that’s a topic for another day.

PS: The figure up top (which has been copied all over the internet, often without attribution) is likely due to Masashi Sugiyama.

In many problems involving hashing we want to look up a range of elements from a vector where the elements are indicated by a hash function . For instance, we might want to evaluate for arbitrary and for a range of . This happens for matrix multiplication, multiclass classification, collaborative filtering, multitask learning and many related problems.

While this works just fine in terms of estimation performance, traversing all values of leads to an algorithm which is horrible in terms of memory access patterns. Modern DRAM chips are much faster (over 10x) when it comes to reading values in sequence rather than when carrying out random reads. Update: memory latency hasn’t improved over the past decade. In fact, DDR5 latency can often be higher than DDR4 latency. Furthermore, random access destroys the benefit of a CPU cache. This leads to algorithms which are efficient in terms of their memory footprint, yet which are slow in terms of their actual runtime behavior. One way to address this is to bound the range of for different values of via one of the following strategies.

Much excellent work has been published on collaborative filtering, in particular in terms of recovering missing entries in a matrix. The Netflix contest has contributed a significant amount to the progress in the field.

Alas, reality is not quite as simple as that. Very rarely will we ever be able to query a user about arbitrary movies, books, or other objects. Instead, user ratings are typically expressed as preferences rather than absolute statements: a preference for Die Hard, given a generic set of movies only tells us that the user appreciates action movies; however, a preference for Die Hard over Terminator or Rocky suggests that the user might favor Bruce Willis over other action heroes. In other words, the context of user choice is vital when estimating user preferences.

If we attempt to estimate scores of user regarding item it is important to use the context within which the ratings have been obtained. For instance, if we are given a sequence of items out of which item was selected we might want to consider a logistic model of the form:

The option of no action is easy to add, simply by adding the null score which captures the event of no action by a user. Shuang Hong Yang tried out this idea to get a significant performance improvement on a number of collaborative filtering datasets. Bottom line - make sure that the problem you’re solving is actually the one that a) generated the data and b) that will help you in practice. That is, in many cases matrix completion is not the problem you want to solve, even though it might win you benchmarks. Obviously the above model is still a gross oversimplification and you’re best advised using the actual interaction order for ranking. But that’s a story for another day.

Some readers might wonder why I’m writing this blog. Here’s an (incomplete) list:

• It’s fun.
• There are lots of fantastic blogs discussing the philosophy and big questions of machine learning (e.g. John Langford’s hunch.net) but I couldn’t find many covering simple tricks of the trade.
• Scientific papers sometimes obscure simple ideas. In the most extreme case, a paper will get rejected if the idea is presented in too simple terms (it happened to me more than once and the paper was praised once the simple parts had been obfuscated). Also, they need to come with ample evidence for why an idea works, strong theoretical guarantees and lots of experiments. This is all needed as a safeguard and it’s really really important. But it often hides the basic idea.
• Some ideas are really cute and useful but not big enough to write a paper about. It’s pointless to write 10 pages if the idea can be fully covered in 1 page. We’d need a journal of 1 page ideas to deal with this.
• Many practitioners are scared to pick up a paper with many equations but they might be willing to spend 10 minutes reading a blog post.

This is a follow-up on the hashing for linear functions post. It’s based on the HashCoFi paper that Markus Weimer, Alexandros Karatzoglou and I wrote for AISTATS’10. It deals with the issue of running out of memory when you want to use collaborative filtering for very large problems. Here’s the setting:

Assume you want to do Netflix-style collaborative filtering, i.e. you want to estimate entries in a ratings matrix of (user, movie) pairs. A rather effective approach is to use matrix factorization, that is, to approximate where is the ratings matrix, is the (tall and skinny) matrix of features for each user, stacked up, and is the counterpart for movies. This works well for the Netflix prize since the number of users and movies is comparatively small.

In reality we might have, say 100 million users for which we might want to recommend products. One option is to distribute all these users over several servers (similar to what a distributed hash table mapping does, e.g. for libmemcached). Alternatively, if we want to keep it all on one server, we’re facing the problem of having to store bytes, i.e. 40 GB if we assume to allocate 400 Bytes per user (that’s a rather small footprint). That is 100 dimensions per user. Usually this is too big for all but the biggest servers (update - by now you can get super-sized X2i instances on AWS with up to 4TB capacity). Even worse, suppose that we have user-churn. That is, new users might be arriving while old users disappear (obviously we don’t know whether they’ll ever come back again so we don’t really want to de-allocate the memory devoted to them). Obviously we cannot add more RAM. One possible solution is to store the data on disk and request it whenever a user arrives. This will cost us 5-10ms latency. An SSD will improve this dramatically but it still limits throughput. Moreover, it’ll require cache management algorithms to interact with the collaborative filtering code.

Here’s a simple alternative: apply the hashing trick that we used for vectors to matrices. Recall that in the exact case we compute matrix entries via

Now denote by and hash functions mapping pairs of integers to a given hash range . Moreover, let and be corresponding Rademacher hash functions which return a binary hash in . Now replace the above sum via

What happened is that now all access into is replaced by access into a vector of length (and the same holds true for ). Why does this work: firstly, we can prove that if we construct and from and via

then the approximate version of converges to the correct with variance and moreover that the estimate is unbiased. Getting the exact expressions is a bit tedious and they’re described in the paper. In practice, things are even better than this rate: since we never use and but always and we simply optimize with respect to the compressed representation.

One of the advantages of the compressed representation is that we never really need to have any knowledge of all the rows of . In particular, rather than mapping user IDs to rows in we simply use the user ID as the hash key. If a new user appears, memory is effectively allocated to the new user by means of the hash function. If a user disappears, his parameters will simply get overwritten if we perform stochastic gradient descent with respect to the and vectors. The same obviously holds for movies or any other entity one would like to recommend.

Bottom line - we now can have fast (in memory) access to user parameters regardless of the number of users. The downside is that the latency is still quite high: remember that the hash function requires us to access for many different values of . This means that each access in is a cache miss, i.e. it’ll cost us 100-200ns RAM latency rather than the 10-20ns we’d pay for burst reads. How to break this latency barrier is the topic of one of the next posts.

PS: The image above depicts a cup of Hash coffee being prepared. The coffee shop in Melbourne has since gone out of business. Hashing, hasn’t, though.

Tamas Sarlos pointed out a much smarter strategy on how to obtain a sparse representation of a (possibly dense) vector: Priority Sampling by Duffield, Lund and Thorup, 2006. The idea is quite ingenious and (surprisingly so) essentially optimal, as Mario Szegedy showed. Here’s the algorithm (please read the previous blog on vector sparsification for some motivation):

• For each compute a priority where is drawn from a uniform distribution.
• Let be the largest such priority.
• Pick all indices which satisfy and assign them the value .
• Set all other coordinates to .

This provides an estimator with the following properties:

• The variance is no larger than that of the best -sparse estimator.
• The entries satisfy , i.e. the expectation is preserved.
• The covariance vanishes, i.e. .

Note that we assumed that all . If not, simply apply the same algorithm to and return signed versions of the estimate.

This is a classic trick when dealing with data streams. It shows how to draw a random element from a sequence of instances without knowing beforehand how long the sequence is and which symbols occur. Let us first assume that we knew the identities of all symbols. In this case finding a random symbol would be easy. All we require is that for each symbol we draw a random variable from some distribution and subsequently we choose the symbol

Since each has equal probability of being associated with the smallest value it follows that the draw is uniformly random. The trouble with this is that we now need to store one floating point number per key and a method to look up its value, e.g. via a dictionary. This is just as tedious as recording the entire stream.

Here’s a better way. Assume that instead of requesting a random variable we simply compute the hash of and we set

For a draw from the space of hash functions this again is uniform. The advantage is that we essentially determined all the random bits when selecting rather than at the time when we want to compute its value . The second advantage is that we can now simply keep track of what is the currently smallest value of and update as we go along. We have the following algorithm:

INIT
   hstar = MAXINT 
   n = 0 
   sstar = NONE
FOR ALL incoming s DO
   IF h(s) = hstar:
      n = n + 1
   ELSE IF h(s) < hstar:
      n = 1
      hstar = h(s)
      sstar = s
RETURN (sstar, n)

This algorithm will provide item counts for a random element of the sequence. If you want more than one sample, simply keep a list of the symbols with the smallest hash values and their associated counts. Such algorithms can be used to compute the variance or other moments of a sequence.

PS: The picture is from the Penticton Lakeside Resort where you can probably see such bears (no, I never visited).