Bigquery on Christian Hotz-Behofsits

Applying Cosine Similarity on a Large Scale

Fri, 30 Sep 2022 00:00:00 +0000

I wrote my first blog post about cosine similarity back in 2019 when the pandemic was out of sight, and most marketing people were unaware of “representational learning.” However, times have changed, and nowadays, many quantitative marketing papers apply word2vec, prod2vec, or similar approaches.

Thanks to gensim, training such a model is straightforward. For example, one can learn the representations of tracks within a few lines of python code:

from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence

model = Word2Vec(
 sentences=LineSentence('./data/tmp_isrc_training.txt'), 
 vector_size=50, window=99, epochs=5, min_count=1, workers=8,
 sg=1, negative=10
)

Bigquery: Least but not NULL

Thu, 14 Jan 2021 12:42:57 +0100

Let’s start with a simple SQL statement that returns the smallest value from each of several columns (similar to min only not across observations, but columns):

SELECT LEAST(date1, date2, date3) as min_date FROM tbl

In BigQuery it works nicely till you encounter NULL values, then it will always pick the NULL. Unfortunately it does not provide a IGNORE NULLS option such as ARRAY_AGG or others do. However, similar behaviour can be achieved with the following UDF:

Merge large CSV files with headers

Sat, 09 Jan 2021 13:54:18 +0100

If you have ever worked with classic data warehousing tools, you may already know the problem: The CSV export splits the output into different files, often with one file containing no more than 1 million observations. However, for analysis purposes, this format is not always optimal and must be merged into a single file. In bash this is not a big problem with cat, but only if there is no header present. The BigQuery CSV table export, for example, adds a header to each CSV file and this must be taken into account when merging.

Interpolate missing values in SQL

Sun, 24 May 2020 12:37:24 +0200

In this post I demonstrate how missing values can be interpolated in SQL. The code is written for BigQuery and I provide a user defined function, however, similar solutions may exist for other databases. Although, SQL may not be the first choice to do such tasks, it is possible. Before one can replace missing values, you have to have missing values. At least in my applications, often the whole rows are missing and not just selective fields. In those cases, try to insert the rows before.

Bigquery & Embeddings

Sun, 27 Oct 2019 08:48:29 +0100

math: true

One can argue if it is wise to store embeddings directly in bigquery or calculate the similarities in SQL. For sure, in some cases a library (e.g. gensim) or approximations (e.g. Facebook faiss) are more appropriate. However, in our setting we wanted to use BigQuery. Therefore, arrays are used to store the word vectors and I created SQL functions to calculate pairwise cosine similarities.

One downside of using BigQuery arrays to store vectors is that there is no guarantee that the ordering of the components is preserved. However, in most situations the order will not change.