gigl.src.post_process.utils.calculate_cosine_sim_between_embedding_tables

gigl.src.post_process.utils.cosine_similarity.calculate_cosine_sim_between_embedding_tables(bq_utils: BqUtils, table_1: str, table_2: str, n: int) → DataFrame

Return: a pd.Dataframe with columns: {DEFAULT_NODE_ID_FIELD, _emb_1, _emb_2, COSINE_SIM_FIELD} NOTE: Currently, the query below takes 17min for n=100M. If in future we wish to increase n to avoid the issue: results that exceed the BQ query limit, we can comment out the last lines. For, now we don’t do so as we don’t need to evaluate cosine similarity for more than 100M embeddings. Hence, there is no need to store an extra table in BQ.