It is not only about visual similarity, but the semantics of the images as well (for example, all the dog photos should be close to each other, no matter the colours or the scene).

The simplest way to do that is probably to use one of the pretrained neural networks (like resnet), convert the images into embeddings, index them and use for search.

I'm sharing my article describing how to implement it: https://medium.com/p/5515270d27e3