Applying Cosine Similarity on a Large Scale

Fri, 30 Sep 2022 00:00:00 +0000

I wrote my first blog post about cosine similarity back in 2019 when the pandemic was out of sight, and most marketing people were unaware of “representational learning.” However, times have changed, and nowadays, many quantitative marketing papers apply word2vec, prod2vec, or similar approaches.

Thanks to gensim, training such a model is straightforward. For example, one can learn the representations of tracks within a few lines of python code:

from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence

model = Word2Vec(
 sentences=LineSentence('./data/tmp_isrc_training.txt'), 
 vector_size=50, window=99, epochs=5, min_count=1, workers=8,
 sg=1, negative=10
)

Parallel JSONLines in R

Wed, 06 Jan 2021 12:37:52 +0100

JSON is becoming more and more widely used in data lakes, especially as a replacement for CSV. In particular, JSONLines where one JSON object is stored per line is to be mentioned here. Although the format has some disadvantages (i.e., overhead, supports only few data types), the advantages mostly outweigh them. For example, the individual lines are easier to read by humans and the format is much more standardized. In addition, each line has its own JSON object that can be processed independently. The overhead caused by the repeating field names can be well managed with compression (e.g., gzip, zstd, brotli).

JSON on Christian Hotz-Behofsits

Applying Cosine Similarity on a Large Scale

Parallel JSONLines in R