In this post, we’ll take a look at the more general notion of stochastic processes and, in particular, a special stochastic process called martingale. This method helps us derive concentration bounds for certain functions of independent random variables - not just their sums. First, though, let’s review some of the basics.
Consider a sequence of -algebras such that ; that’s called a filtration. A sequence of random variables is a martingale if:
It’s easy to verify that the partial sum of independent zero-mean random variables is a martingale. That is, given a sequence of random variables with mean zero, for is a martingale with respect to the -algebra generated by ’s. So is the product of random variables with mean : .
The Doob martingale is another example, which is constructed as follows: Say we have a function and a sequence of random variables . If we define (or, more precisely, conditioning on the -algebra generated by the first random variables), and let and , then the sequence is a martingale if exists.
Proof: We have to show that the sequence has the properties of a martingale. First, notice that:
which exists by assumption. As for the second property:
A related notion is the martingale difference sequence: If a sequence with respect to a filtration has the property that is finite and that , then it is a martingale difference sequence. For example, when is a martingale with respect to some filtration , then where is a difference sequence.
Proof: The first property is trivial to show. As for the second property:
Let’s take a martingale difference sequence, with flitration , and impose the following constraint on it:
In other words, we require that is sub-exponential with parameters for all . Then it’s not hard to see the following result:
Any partial sum of the sub-exponential martingale differences, , with parameters for all , is itself a sub-exponential random variable.
Proof: Let’s start with the definition of a sub-exponential random variable:
Continuing with this process iteratively, we find that the partial sum is indeed sub-exponential with parameters .
As a result, we can apply tail bounds for sub-exponential random variables to the random variable . For example:
As a direct consequence of this result, it is again straightforward to see that, if a martingale difference sequence is bounded, it is sub-exponential, and as such bounds such as Hoeffding’s inequality hold:
If is a martingale difference sequence with filtration with the property that almost surely for all , then:
Proof: Because is bounded, it is sub-Gaussian with constant as shown here. This tells us that:
As a result, the sum of ’s is sub-Gaussian with constant , so that we may apply Hoeffding’s inequality and obtain the result above.
Sub-exponential martingale difference sequences are super useful! Well, really, the fact that bounded random variables are sub-Gaussian is a helpful result that, combined with martingale difference sequences allows us to apply tail bounds to stochastic processes.
Martingales aren’t limited to sums of independent random varaibles, however. So it is natural to ask if we can extend concentration results to a more general construction such as the Doob martingale. Why is such a generalization useful? To answer that, let’s have a look at one important example.
Suppose we have a function and we are interested in understanding the concentration of around its mean, . Now, what does the Doob martingale look like? Well, , , and . But we can write:
The right-hand side is the sum of martingale differences! So indeed we may be able to impose some conditions on the function and use our results from the previous section to bound the deviation of from its mean. The following result does exactly that:
McDiarmid’s Inequality: Suppose has the property that where denotes a copy of the vector that differs from in the -th dimension only. In other words, changing the -th coordinate of the input changes the output of by at most . If the random vector has independent components, then:
Proof: We have already seen that can be expressed as the sum of a martingale difference sequence. If we showed that belongs to an interval of length at most , then we can apply Hoeffding’s inequality to obtain the desired result. So we’ll focus on bounding ’s.
By definition, we have that:
Now define the following random variables:
and,
Notice that . But now:
where we used the fact that .
Let’s illustrate the result by applying it to a rather ubiquitous random variable: If is a vector collection, then:
where is a random Rademacher vector, is called the Rademacher complexity of the set . Now consider the random variable and notice that it is bounded in the sense of the previous theorem:
As a result:
where .
]]>
then we call an -subspace embedding of the column space of , or simply an -subspace embedding of . Essentially, preserves norms of the column space of up to amount of error. To simplify notation, I’ll write the above expression as: .
We can, without loss of generality, assume that has orthonormal columns. That is because, if that’s not the case, we can redefine where comes from the SVD of . Indeed, the sets and where are the same.
What that means is that, we can simplify the condition to: . We can also require that the above hold for unit vectors due to the linearity of the maps involved. That is in turn equivalent to where is the operator norm of its argument. To see this:
where we used the SVD and the fact that and have orthonormal columns.
What properties should have? Well, first and foremost, we want so that has as few rows as possible. That is, we want the smallest subspace of the column space of that offers a -approximation of the Euclidean norm. Additionally, we want to be able to compute rather quickly. That often means we prefer a sparser .
Suppose that we have a distribution over the space of all linear maps, where is a function of , with the following property: For every and any fixed matrix , with probability at least , is a -subspace embedding of . Then we call an oblivious -subspace embedding.
As a side-note, in addition to oblivious embeddings, we can design sampling-based sketching algorithms that are called Leverage Score Sampling. I’ll discuss these in a later post.
Theorem: Let where the entries of are iid standard normal random variables. If , then is a -subspace embedding for any matrix with probability at least .
Proof: Notice that, we are trying to find an -subspace embedding for the column space of . We can also focus only on unit vectors in the column space, due to linearity. The idea behind the proof is to first lay an -net over the column space of with the property that, a Johnson-Lindenstrauss Transform for the -net is a -subspace embedding for the entire space. Let’s get to it.
Consider the set and its -net . We can write every as the series: where is a scaled member of with the property that . To see that that’s always possible, take and write . Clearly because . Now write where for some . Then:
as desired. We can continue to expand the residual as above and reason by induction.
Now consider a JLT for , , so that with probability at least , we have that for all pairs of . If we applied the same map to any , then we’d get, with probability at least :
So, we have established that a JLT for the -net over the unit sphere in the column space of is an -subspace embedding for the column space of . We know that , so that as desired.
Definition: Define the set . An -net is a subset such that, for every , there exists a such that .
Lemma: For , there exists an -net for the set (defined above) whose size is at most .
Proof: Let’s consider the unit sphere in . Instead of finding a net for , we will argue by finding a net for . That’s because provides an isometry when operating on the unit sphere, so that a net for gives us a net for the image of under .
Finding a net over the unit sphere is trivial. We can choose a maximal set such that no two points in the set are within distance from each other. This means that balls of radius centered at points in are disjoint. These balls all fit in the ball with radius centered at the origin. The volume of this enlarged ball is times larger than the small balls. As such, there can be at most small balls, implying that . [This is the covering number of the unit sphere, which is a compact set.]
Now let . We can show by contradiction that, for any there must exist a point in such that . Suppose there exists no such for some for some . That implies that, there exists no point in for which . But that contradicts our construction of .
Recall that a random variable with mean is sub-Gaussian with constant if its moment generating function is bounded above as follows:
That proved useful because, if is sub-Gaussian, then we showed that the tail of its distribution is dominated by a Gaussian random variable’s with parameters and . In particular, we are able to apply the Chernoff bound to sub-Gaussian random variables and derive Hoeffding inequality-type bounds on the tail distribution.
Sub-Gaussianness, however, is somewhat restrictive. It requires that the bound on the moment generating function hold for all values of ! We can relax that requirement just a little bit and demand that the inequality hold for small values of around instead:
A random variable with mean for which inequality (2) holds is said to be sub-exponential with parameters for some non-negative (and defining ).
Before we continue, it’s worth mentioning that sub-exponentialness is preserved under summation of independent random variables, just as sub-Gaussiannes is. In other words, if we have sub-exponential random variables ’s with parameters , then their sum is sub-exponential:
which is valid when . So its parameters are .
There are other ways of characterizing a sub-exponential random variable that are equivalent to the definition in (2). For example, if the moment generating function of is finite for small values of , then is indeed a sub-exponential variable! This, and other definitions, are presented in the result below.
The following definitions are equivalent for a centered random variable :
i. There exist constants and such that inequality (2) holds.
ii. There exists a constant such that is finite for .
iii. There exist constants and such that for all .
iv. The quantity is finite.
Proof. (i) (ii) is straightforward.
(ii) (iii): Applying the Chernoff bound to gives:
So setting and gives the desired result.
(iii) (ii): We start from the definition of expectation of a positive random variable, for a positive :
when . Because and , we can conclude that is finite for .
(ii) (iv): We can argue using the power series expansion of the moment generating function:
Notice that the inverse of is the convergence radius of the series. As a result, if (ii) holds, then the series must converge, implying that the quantity in (iv) must be finite. If (iv) holds, then the series must converge, implying that the quantity in (ii) must be finite for within the convergence radius.
An alternative way to characterize sub-exponential random variables is by bounding the polynomial moments of , instead of its moment generating function. Concretely, if is a random variable with mean and variance , then we can show that, if the following condition, known as Bernstein’s condition with parameter , holds, then is a sub-exponential random variable:
Proof. Let’s start from the definition of a sub-exponential random variable and expand the left-hand-side of (2):
By Bernstein’s condition:
which, when is smaller than , is equal to:
when . That implies that is sub-exponential with parameters .
This is a neat result! For example, it is straightforward to see why Bernstein’s condition holds for bounded random variables with the property that . We proved previously that such random variables are sub-Gaussian. It turns out, they are sub-exponential too!
Why is this notion of sub-exponentialness useful anyway? Well, for one thing, the family of sub-exponential random variables includes sub-Gaussian random variables. But more importantly, limiting our focus to a neighborhood around enables us to derive more expressive bounds on the tail distribution of a sub-exponential . To see why, let’s first derive the basic tail bound and then discuss a more elaborate bound that contrasts the concentration of sub-Gaussian and sub-exponential variables.
For a sub-exponential random variable with parameters , the following holds:
Proof. For valid values of (i.e., ), we have, by definition, that:
When , we can solve the unconstrained minimization problem and obtain:
Substituting this value of into gives the desired result, as in the case of a sub-Gaussian random variable.
On the other hand, when , then reaches its minimum at the boundary point when . That’s because is monotonically decreasing in its first argument. Evaluating then leads to:
We can do something even more interesting if we consider Bernstein’s condition! If we stare at the proof of the statement above that Bernstein’s condition is sufficient to show that a random variable is sub-exponential, we notice an interesting result hiding in there. In particular, we derived, in that proof, the following bound when :
Setting results in the following concentration inequality:
Why is that significant? Consider a bounded random variable such that . We argued above that is sub-exponential as it meets Bernstein’s condition. The tail bound in (4) then introduces a dependence on, not just , as the Hoeffding inequality would (as a consequence of being sub-Gaussian), but also on the variance of !
Because for a bounded random variable, and for suitably small values of , (4) is never worse than the Hoeffding bound. In particular, when , then (4) offers a much sharper bound on the concentration of .
We have already discussed the case of bounded random variables. Indeed, such variables are sub-Gaussian and sub-exponential. Let’s now look at a random variable that’s sub-exponential but not sub-Gaussian: the random variable.
Recall that if is a Gaussian random variable, then is a random varaible whose mean is . Let’s inspect its moment generating function:
Obviously, when the moment generating function does not exist! When , then we can show that the moment generating function is bounded above by . As such, is sub-exponential with parameters .
As another example, consider the sum of iid random variables. Because each variable is itself a sub-exponential with parameters , we know from before that their sum, too, is sub-exponential with parameters . As a result, we can derive the following bound:
for all .
]]>One of the most basic inequalities governing the tail bounds of a distribution is Markov’s inequality. It says that, if is a non-negative random variable, then its tail probability has the following upper-bound:
Clearly, this only makes sense if and is finite.
The Markov inequality is tight, in the sense that in cannot be improved in general. To see this, consider the random variable taking values in with probability and . A straightforward calculation shows that the inequality in (1) becomes equality.
More generally, notice that:
As such, any random variable with the property that almost surely leads to equality in Markov’s bound.
A fairly trivial extension of Markov’s inequality is the Chebyshev’s inequality. It is arrived at by a simple application of Equation (1) to the non-negative random variable for an arbitrary random variable :
Similar to the Markov inequality, we can show that Chebyshev’s inequality is tight in general. It is easy to construct a random variable that results in equality if we notice that we must have . For example:
One can just as easily extend Chebyshev’s inequality to higher moments, so that:
Even more interesting is the fact that we can extend this formulation to functions other than polynomials! For example, consider the exponential function:
Optimizing our choice of so as to obtain the tightest result gives us the Chernoff bound!
We can show that, Equation (3) with the optimal choice of is never worse than Equation (4) with the optimal choice of . Concretely, suppose and that the moment generating function of exists in a neighborhood around . Then for any given , we can see that:
so that:
Consider Let’s derive the Chernoff bound for by optimizing our choice of . First, let’s derive the right-hand-side in Equation (4):
Differentiating with respect to and setting it to :
gives us the optimal choice . Plugging this value back in Equation (4) gives the Chernoff bound for our Gaussian random variable:
It is easy to see that, if the following holds for a random variable :
for some constant , then inequality (5) holds for . We call such random variables Sub-Gaussian with parameter .
Example (1): Rademacher random variables are sub-Gaussian with parameter . That is because:
Example (2): Any zero-mean bounded variable supported on the interval is sub-Gaussian with parameter at most .
Proving this requires the symmetrization argument. We first introduce an independent copy of and call it , and let be a Rademacher random variable. By the convexity of the exponential and Jensen’s inequality, we can write:
Using the fact that the distribution of is the same as the distribution of , and applying the result from Example (1), we can write:
Example (2) essentially gives us Hoeffding’s inequality. Concretely, because the sum of two sub-Gaussian random variables with constants and is itself a sub-Gaussian random variable with parameter , we can show that if ’s are independent sub-Gaussian random variables with parameter , then we have that:
For bounded random variables, we can derive the familiar form:
The result above can be further tightened by showing that, bounded random variable is indeed sub-Gaussian with parameter .
Proving that last statement is a bit involved. We’d have to define the following function:
The idea here is expand around :
Notice how the right-hand-side has the form we hope to get to in Equation (6): ? So if we showed that and , then we have proven our claim.
To that end, consider the derivative of with respect to which is simply:
It’s clear that , so that takes care of the first part. Its second derivative then is:
Now, we have to bound this to show that, when evaluated at , it is no greater than . Showing this is a bit tricky and involves a bit of measure theory. Let’s rearrange the terms in the last expression so it’s visually clearer what’s going on:
This looks almost like the expression for the variance of a random variable, but with an extra term . Here, we can appeal to the Radon-Nikodym theorem. Because the conditions of the theorem hold (i.e., measures are absolutely continuous), we can define a new distribution so that is its Radon-Nikodym derivative with respect to the distribution over . What that helps us accomplish is that, can now be expressed as the variance of a random variable drawn from :
So, using the fact that the mean is the minimizer of the mean squared error, and knowing that , we can write:
Putting all the pieces together, we have shown that , which shows that a random variable bounded to the interval is sub-Gaussian with constant .
We know how to apply a number of tricks from basic linear algebra to do just that. For example, we can obtain a low-rank approximation of by the simple act of projecting it onto the subspace spanned by its top singular vectors. That gives a matrix with rank at most such that is small.
In some applications, it’s not enough to approximately preserve the Frobenius norm of . For example, we may be interested, instead, in approximately preserving the pairwise Euclidean distance between the points—a property that is important in Nearest Neighbor Search. That is, if is the image of under our dimensionality reducing transformation (where presumably ), then we want the following to hold for all :
for some . Occasionally I’ll write the above inequalities more compactly as:
It’s fairly easy to see that a bound on the Frobenius norm does not necessarily translate to this -approximation of pairwise Euclidean distances! We might in fact be able to lower the error in the Frobenius norm by moving some points farther from each other and others closer to each other. In other words, approximately preserving a global notion of distance is not equivalent to approximately preserving our local distances required by Equation (2).
That’s where the seminal result due to Johnson and Lindenstrauss comes into the picture. It’s led to a series of works that offer a simple and elegant, linear and randomized machinery to reduce dimensionality and yet approximately preserve local distances. These probabilistic algorithms play an important role in randomized numerical linear algebra such as subspace embedding. So it’s worth reviewing the basics. Let’s dive into it.
Distributional Johnson-Lindenstrauss (JL) Lemma. For every positive and , there is a distribution over linear maps in with , such that, for any :
I’ll refer to Equation (3) as the JL Property and say that a random matrix has the JL property if Equation (2) holds for the distribution over ’s.
How does this lemma relate to Equation (2) and our search for a transformation that preserves pairwise Euclidean distances? By setting for all , and choosing to be at most , we can apply the union bound and obtain that with probability at least . In fact, that procedure gives the more familiar version of the Johnson-Lindenstrauss lemma, stated below:
Johnson-Lindenstrauss Lemma. Given a set of points in and , there exists a mapping where , such that for all pairs , .
Before we continue, I should mention a neat consequence of the distributional JL lemma. In particular, inner products too are approximately preserved! Concretely, if points and on the unit sphere then:
That’s because, we can write:
so that, with probability at least , we have that:
More generally, for arbitrary , the following statement holds:
which follows by defining and in Equation (4).
Let’s review a few popular constructions of the random matrix .
Let where the entries of are iid standard normal random variables. Then has the JL property.
Proof: First consider the case of . Notice that is a -dimensional Gaussian random vector whose entries are distributed as . That implies that is distributed as where is a random variable with degrees of freedom. We can then apply known tail bounds for a random variable with degrees of freedom, such as:
so that for our specific case, we have that:
Setting the right-hand-side to , we get that:
Extending this to is simply a matter of defining , which gives the JL property.
That’s so elegant, isn’t it? All we need to do is to sample a matrix whose entries are standard normal random variables, then scale it by , and apply the transformation to our points! That’s it! We get a whole new set of -dimensional points where the worst distortion of pairwise distances is bounded.
It turns out, while the result is simple to prove, it can be a bit expensive to apply in practice. There are other constructions that give much simpler functions that are faster to use. Sampling a matrix whose entries are Rademacher random variables, for example, is one example:
Let where the entries of are iid Rademacher random variables. Then has the JL property.
Proof: As in the previous proof, it is enough to consider points on the unit sphere. As before, we need to argue about the concentration of around its mean, which turns out to be :
where ’s are Rademacher random variables. The final step of bounding the tail probability (as in Equation (5)) is a bit involved, so I’ll just note that (Achlioptas, 2001) (Lemma 5) proves a bound of the following form:
Again setting the right-hand-side to gives the desired result: .
Finally, we can even make sparse by drawing the entries of from the following distribution:
There is even a faster-to-apply family of JL transformations that’s simply called Fast JL Transform or FJLT for short. Let’s see how it works!
Let where: is a diagonal matrix whose entries are Rademacher random variables; is an matrix with the property that and ; and is a matrix where each row is everywhere except in one uniformly random column. Then has the JL property with .
In general, a matrix with that property is called an unnormalized bounded orthogonal system. A concrete example is the Hadamard matrix.
Proof: As before, we do this analysis for a vector with .
Part I: First, consider . We can immediately notice a few things about the product . First, that its norm is simply:
Second, its -th entry is: . By following the same method to prove that a Rademacher random variable is sub-Gaussian, we can show that each term is a sub-Gaussian random variable with constant . As a result, , being the sum of sub-Gaussian random variables, is itself sub-Gaussian with constant .
Considering ’s are sub-Gaussian with mean , we know the following Chernoff bound holds:
so that if , then . By the application of the union bound, we can deduce the following, which is known as the flattening lemma:
From here on, let’s condition on the flattening lemma being true with probability .
Part II: Now, let’s consider . We have to show that the deviation of the norm of this product from the norm of is bounded. In other words, we want to assert that:
Let . This can be expressed as the sum of random variables ’s where for some uniformly random . As a result, and therefore .
Let us now bound . Notice that:
The variance of is therefore bounded by .
Conditioned on the flattening lemma, , making a bounded random variable, hence sub-exponential. Their sum, , is therefore sub-exponential as well. We can now apply Bernstein’s inequality as follows:
When , the above probability is at most .
Part III: We have thus far conditioned on the event where with probability , and showed that inequality (P1) holds. We must now remove our condition:
That’s where matrix norms come into the picture. You may be familiar with induced norms such as (maximum singular value), (maximum absolute column sum), (maximum absolute row sum), which capture a lot of information. , for example, tells you the largest amount of “stretching” is capable of as it transforms your space—to see this, it might help to go back to the definition of these norms: .
In this post, we’ll work with a different kind of matrix norm: the Schatten -norms, defined as follows: where is the rank of and is its -th singular value. When , for example, this is the familiar Frobenius norm; gives the spectral norm, which coincides with the induced -norm above. From here on, when I write , I’m referring to the Schatten -norm of , and not the induced norm.
One of the interesting facts about Schatten -norm is that they are invariant under rotation! That is, if I applied an orthogonal transformation to , wouldn’t change. Let’s see why: where is the transpose of its argument, and is the Singular Value Decomposition of . Indeed, because and are orthonormal matrices, too will be orthonormal, rendering the SVD of another matrix whose singular values match those of ’s. You can use a similar argument for transformations of from the right (i.e., where ).
We see why Schatten -norms are useful. Computing is also easy, if we are allowed to apply SVD to and extract its singular values. But, let’s say, we don’t have access to ; it’s being kept a secret by someone—let’s call this person the “oracle”—and they won’t let us apply SVD to the matrix! (It doesn’t have to be for nefarious reasons, it could simply be because is just way too large!) But we are allowed to transform any vector with and obtain the output, .
In what seems so magical, we can, in fact, approximate arbitrarily accurately by repeatedly asking for for random vectors ! The rest of this post explains how.
Suppose for now that and that it is symmetric—we will come back to the more general case in a bit. We want to obtain, with probability at least , a -approximation of for some valid choices of and . In other words, we want to find a quantity such that: Also, we’ll just focus on , rather than because it makes the math less cluttered.
Now to the main steps of the algorithm proposed by (Li et al., 2014). Repeat the following process times (for some constant to be defined later), indexed by :
In the end, output the following quantity: as the desired approximation of .
That was just the (sweet and simple) algorithm. We left out what should be and, more importantly, why this procedure even works! So let’s get to it!
Before we do though, let’s quickly consider the complexity of the algorithm. If the number of non-zero entries in is , then each iteration of the algorithm simply runs in time. We repeat the algorithm times, so that its complexity adds up to .
We assumed that was symmetric. That implies that it can be decomposed into the following form: for some orthogonal matrix (so that ) and diagonal matrix . What happens when we take the inner product inside the sum in Equation (1)? It reduces to: where we used the definition of to get the first equality, the definition of inner product for the second, and the decomposition of for the third. In the last expression, is the rank of and subscripts indicate specific coordinates.
The output of the algorithm is the mean of ’s. Considering each sample gives Equation (2), and that , the expected value of is an unbiased estimate of , which is exactly .
I promised we’ll get to at some point. Now that we have verified that the result of the algorithm is, in fact, an unbiased estimate of , it’s time to fill that gap.
Error analysis often begins with a derivation of or a bound on the variance. In this case, we should be looking at our random variable from Equation (2). It shouldn’t be too hard considering we’re dealing with iid Gaussian variables. So let’s do it:
So we know the variance of each sample of is less than that quantity. Great. We sample some times. Because the samples are independent, the variance scales down by a factor of .
Now we can finally get to our approximation error. We need to understand the maximum probability by which deviates from by a factor of at most . For that, we will appeal to standard results from concentration of measure. In particular, we simply apply Chebyshev’s inequality: to the random variable . That gives us:
We wanted the probability above to be at most . Setting the right-hand-side to , we get that . And there we have it!
We saw what we need to ask of the oracle to obtain an approximation of when is symmetric: We initialize a Gaussian vector, and repeatedly ask the oracle to return the matrix-vector product, which we then feed back to the oracle. Pretty straightforward. But what about the more general case where ?
It turns out, if we formed the following block matrix: then . It’s fairly easy to show this if you rewrite in terms of the SVD of :
We would have to update the algorithm with this logic to ensure it works for a general matrix .
I hope you found the elegance and simplicity of this randomized algorithm as compelling as I did and still do—ten years since its publication. There are numerous other algorithms that feel magical for norm approximation and other operations on vectors and matrices in this rich literature that I encourage you to study. There are a number of excellent monographs on the subject such as (Woodruff, 2014), (Murray et al., 2023), (Martinsson & Tropp, 2021) and references therein. I hope to highlight some of the more representative algorithms in future posts.
]]>Technical update. The content and figures in this post have been updated to reflect the latest version of the algorithm, described in our extended paper, Efficient Sketching and Nearest Neighbor Search Algorithms for Sparse Vector Sets (September 2025).
Picture this. You have a large collection of text: passages from legal documents, snippets from books, or pages from the web. Given a question, you want to find the passages most likely to help answer it. A learned embedding model gives us a way to express that task mathematically: encode the question and each passage as vectors, then retrieve the vectors with the largest similarity scores.
Our algorithm, Seismic, makes this search efficient for learned sparse vectors (Bruch et al., SIGIR 2024). It combines a sketch of the collection, a geometric organization of inverted lists, full-vector scoring, and an optional nearest-neighbor graph. The extended paper develops the sketching analysis and evaluates this combination on collections of up to 138 million vectors. Let’s work through how the pieces fit together.
A sparse vector has many zero coordinates. In learned sparse text representations, each dimension typically corresponds to a term in a fixed vocabulary, and its nonnegative value expresses that term’s importance. A model can give weight to a term absent from the input, allowing related passages to match even when they use different words. SPLADE and its variants are examples of this approach.
Only the nonzero coordinate–weight pairs need to be stored. A vector can therefore live in a vocabulary with tens of thousands of dimensions while requiring space for only a small fraction of them. Figure 1 illustrates the representation with invented weights; the encoder and its training procedure are separate from the search algorithm.
Sparsity helps with storage and arithmetic, but it does not make search trivial. A common term can still occur in millions of vectors. Efficient retrieval needs a way to avoid most of those vectors while retaining the ones that score highly for the query.
Write the query as q and a document vector as u. Their inner product is the sum of products in matching coordinates:
We seek the k documents with the largest scores: maximum inner product search, or MIPS. Geometrically, the score orders candidates by their projection along the query direction, as Figure 2 shows. Cosine similarity becomes an inner product after normalizing vectors. Euclidean nearest-neighbor search has the same ranking when candidate norms are equal; otherwise it needs an appropriate reduction.
A full scan computes every document’s score. An approximate index instead tries to discover a small, useful candidate set. The price of skipping work is that some true top-k documents can be missed.
The paper measures this with accuracy@k: the fraction of the exact top-k set present in the returned set. Returning nine of the true top ten gives 90% accuracy@10. This is agreement with exact vector search; the embedding model still determines how useful those documents are to a person.
Seismic separates candidate discovery from candidate scoring. Its inverted index maps a dimension to document identifiers, organized into blocks with compact summaries. Its forward index maps each identifier to the complete original vector. An optional κ-nearest-neighbor graph maps each document to a short list of similar documents, giving search another way to discover candidates.
The inverted index guides where to look; the forward index supplies exact inner products for the documents we actually inspect. The graph can then refine that candidate set. Figure 3 gives the full pipeline before we unpack each stage.
Many learned sparse vectors concentrate much of their weight in relatively few coordinates. The extended paper studies how to exploit that pattern, starting with randomized ℓ₁ threshold sampling and then developing deterministic subvector sketches. For its sketching analysis, the paper places nonnegative vectors in the ℓ₁ unit ball, with total coordinate mass at most one. The practical attraction of a subvector is simple: it retains some coordinates and their original values, so ordinary sparse inner-product routines can process it.
For a single vector, an α-mass subvector sketch, or α-MSS, sorts coordinates by decreasing absolute value and retains the smallest prefix whose total absolute weight reaches at least α times the vector’s ℓ₁ mass. Here α is a fraction of mass, rather than a fraction of coordinates. With nonnegative vectors, dropping coordinates makes the resulting inner-product estimate an underestimate. Section 3.3 of the paper bounds that error in terms of omitted mass and overlapping coordinates.
Seismic uses a related but different rule for the collection: Set α-MSS. Imagine documents as rows of a matrix and dimensions as columns. In each column, retain a fraction α of its nonzero entries, choosing the largest values. Figure 4 contrasts these two rules; Figure 5 applies the collection rule to a small matrix.
More precisely, if dimension i has mi nonzeros across the collection, retain its largest ⌈αmi⌉ entries and set the rest to zero. Equivalently, mi = ρi|S| in the paper’s density notation. This controls the retained entry count per dimension. It does not promise that every document retains an α fraction of its own mass: in the toy matrix, one document loses all its entries.
The analysis in Section 3.4 connects the collection sketch to mass retention under its distributional assumptions, including independent or weakly dependent dimensions. Empirically, the paper also examines mass concentration and preservation of inner-product rankings. Those results motivate the design; a particular sketch still has to be evaluated for the embeddings and accuracy target at hand.
After applying Set α-MSS, form an inverted list Li for each dimension: it contains the identifiers of documents whose retained coordinate in that dimension is nonzero. These lists are substantially smaller than the lists from the original collection when α is small. Original vectors remain available in the forward index.
Next, partition each retained list into geometrically cohesive blocks. The parameter β is a block-count factor: a list of length |Li| is organized into approximately β|Li| blocks. A larger β allows more, smaller blocks and more summaries to examine.
The paper uses a shallow clustering procedure. Sample representatives from the list, then assign each document to the representative with which its full vector has the largest inner product. Each resulting cluster becomes a block. Grouping similar vectors gives a summary a better chance of describing the block usefully. Lists are partitioned independently, so a document can appear in blocks belonging to several dimensions.
For a block B, first form its coordinate-wise maximum over the original document vectors. The summary’s coordinate i is φ(B)i = maxu ∈ B ui. For a nonnegative query, this full summary is conservative: its score is at least the score of every document in the block.
But a maximum can accumulate many nonzero coordinates. Seismic compresses it with a γ-MSS, retaining enough of its largest coordinates to reach a fraction γ of its mass. It also applies 8-bit scalar quantization to summary values, storing a subinterval identifier within the summary’s value range.
These smaller summaries are cheaper to store and score. They also lose the full maximum’s upper-bound guarantee. Figure 6 makes that loss concrete: a coordinate with little summary mass can matter greatly to a particular query. The compressed score is therefore a guide for approximate pruning.
The forward index retains every document’s complete sparse representation. Whenever a block is selected, Seismic fetches its documents and computes their full inner products with the query. The paper’s implementation uses prefetching to reduce the cost of these scattered accesses, since documents grouped together in one inverted list cannot generally be consecutive in every list.
With κ > 0, index construction also stores κ neighbors per document, measured by inner product. Finding exact neighbors for every document would be expensive, so the experiments build an approximate, high-accuracy graph using Seismic itself without graph refinement. Its storage grows with the collection size and κ, and its construction adds substantial work. Setting κ = 0 omits it.
The parameters now have distinct roles:
| Parameter | Role |
|---|---|
| α | Fraction of nonzero entries retained per dimension by Set α-MSS. |
| β | Factor determining the number of geometric blocks per retained list. |
| γ | Fraction of each maximum summary’s mass retained before quantization. |
| κ | Stored neighbors per document; zero disables graph refinement. |
| αq | Mass retained in the query sketch that selects inverted lists. |
heap_factor | Query-time factor controlling how aggressively blocks are skipped. |
Start by taking an αq-MSS of the query. Its nonzero coordinates select which inverted lists to visit. The full query is still used to score both summaries and document vectors. This distinction lets list selection be cheap without throwing away query information when evaluating a candidate.
Maintain a size-k min-heap of the best distinct documents examined so far. Within each selected list, compute every block’s summary score rj = ⟨q, Si,j⟩, then process blocks in decreasing score order. Promising blocks come first, helping establish a strong heap threshold early.
Once the heap contains k documents, let θ be its minimum score and write h for heap_factor, with 0 < h ≤ 1. Following Algorithm 3, skip a block when:
A smaller h raises this threshold and makes pruning more aggressive. Until the heap is full, this test does not skip blocks. For a block that passes, fetch its documents from the forward index, score them with the full query, and update the heap. Figure 7 shows how the threshold rises when better documents arrive.
The sketch, list selection, and compressed summaries all help reduce the search space. Full-vector scoring gives accurate scores for the candidates found, but it cannot restore a document that was never visited. The heap thus contains the best candidates examined, with accuracy controlled empirically through the index and query parameters.
Summary-score ordering and graph refinement were introduced in SeismicWave, our CIKM 2024 paper.
If the graph is enabled, take a snapshot of the document identifiers in the heap after inverted-index search. Add the stored neighbors of those initial candidates, compute their full-query scores, and retain the top k over the expanded set.
This is a one-hop expansion: newly discovered neighbors are not recursively expanded. A document missed by the selected lists or skipped blocks can still enter through a strong initial candidate’s neighborhood. Figure 8 shows the deduplicated candidate pool and the resulting reranking.
The draft evaluates MS MARCO v1 (8.8 million passages), Natural Questions, and, for scalability, MS MARCO v2 (138 million passages). Across its experiments it uses SPLADE, Efficient SPLADE, SPLADE-v3, and uniCOIL-T5. Query latency measures index search from an already embedded query to the returned neighbors, using a single query-processing thread; it excludes embedding time.
Figure 9 redraws one slice of Table 1: SPLADE on MS MARCO v1 at 95% accuracy@10. Seismic without graph refinement takes 324 microseconds per query. The other reported latencies range from 836 microseconds for kANNolo to 31,843 for Ioqp. In this setting, the relative speedup is about 2.6× against the fastest listed baseline and 98× against Ioqp. The comparison depends on the baseline, embeddings, and accuracy target.
Graph refinement adds another tradeoff. Table 3 tunes both variants under explicit memory budgets, expressed relative to forward-index size. With SPLADE on MS MARCO v1 and a 2× budget, the reported latency falls from 255 to 156 microseconds at 95% accuracy@10, and from 715 to 387 at 99%. These are separately tuned configurations within the budget. The graph is not faster for every model and target: on SPLADE-v3 at the same 2× budget, for example, the graph-free configuration is faster at 95%.
Figure 10 pairs those SPLADE query results with the larger collection’s construction costs. Table 4 reports MS MARCO v2 indexes of 98.5 GB without the graph and 98.4 GB with it, taking 0.5 and 16.6 hours to build, respectively. The two configurations allocate their memory differently. Similar total index sizes do not make the graph free: neighbor identifiers consume memory that can otherwise support the inverted index and summaries.
The scalability study builds on our ECIR 2025 paper and uses a different server with enough local memory for the large indexes. At its 2× memory budget and 98% accuracy cutoff, the extended paper reports Seismic with graph refinement as 6.9× faster than kANNolo, and the graph-free variant as 4.4× faster. These results support the overall design while showing why accuracy, query latency, memory, and construction time should be considered together. Section 5 of the extended paper provides the full tuning protocol, hardware, and results.
The extended paper contains the sketching definitions and analysis in Section 3, index construction and search in Section 4, and experiments in Section 5. Our open-source implementation is available on GitHub. This work is a collaboration with Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini.
For the development of Seismic, see the SIGIR 2024 paper, the CIKM 2024 SeismicWave paper, the ECIR 2025 scalability study (with Leonardo Venuta), and the extended paper.
The original SIGIR’24 paper, which gives this post its title, received the ACM SIGIR Best Paper Runner-Up Award. For the extended work described here, use:
@misc{bruch2025efficientsketching,
title = {Efficient Sketching and Nearest Neighbor Search
Algorithms for Sparse Vector Sets},
author = {Bruch, Sebastian and Nardini, Franco Maria and
Rulli, Cosimo and Venturini, Rossano},
year = {2025},
eprint = {2509.24815},
archivePrefix = {arXiv},
primaryClass = {cs.DS},
doi = {10.48550/arXiv.2509.24815},
url = {https://arxiv.org/abs/2509.24815}
}So I decided to write one, particularly geared towards graduate students, with an emphasis on the theoretical foundations of the field. I hope “Foundations of Vector Retrieval” proves a helpful reference, one that makes it a bit easier for you to pick up this topic, navigate the literature, continue this research, and make new discoveries!
I have made a copy freely available on arXiv.
If you prefer a hard copy, you can read it as
paperback.
You can cite the work using the following bibtex entry:
@book{Bruch_2024,
title={Foundations of Vector Retrieval},
ISBN={9783031551826},
publisher={Springer Nature Switzerland},
author={Bruch, Sebastian},
year={2024}
}