Assigning a visual code to a low-level image descriptor,
which we call code assignment, is the most computationally
expensive part of image classification algorithms
based on the bag of visual word (BoW) framework. This
paper proposes a fast computation method, Neighbor-to-
Neighbor (NTN) search, for this code assignment. Based
on the fact that image features from an adjacent region are
usually similar to each other, this algorithm effectively reduces
the cost of calculating the distance between a codeword
and a feature vector. This method can be applied not
only to a hard codebook constructed by vector quantization
(NTN-VQ), but also to a soft codebook, a Gaussian mixture
model (NTN-GMM). We evaluated this method on the
PASCAL VOC 2007 classification challenge task. NTN-VQ
reduced the assignment cost by 77.4% in super-vector coding,
and NTN-GMM reduced it by 89.3% in Fisher-vector
coding, without any significant degradation in classification
performance.