Here is a snippet that can be reproduced with the attached data: train_voc.zip

import json
import gensim

with open("train_voc.json", "r") as file:
    sents = json.load(file)
model = gensim.models.fasttext.load_facebook_model("cc.de.300.bin")
model.build_vocab(sents, update=True)
model.train(sentences=sents, total_examples = len(sents), epochs=2)
gensim.models.fasttext.save_facebook_model(model, "cc.de.300.tuned.bin")

Link to the German model: https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.de.300.bin.gz

What I get in my debugging state:

is this something that I would be able to take a look at as it would be great to get the 3.9 Wheels on pypi.org?

Does the problem reproduce if you try in the 4.0.0-beta, as can be installed locally for you per instructions at https://groups.google.com/g/gensim/c/5mAeWrQN7lg/m/eMhYpktCAAAJ?

If it does not, then its likely an already-fixed bug – after all, this issue was closed-as-fixed after a test case was created that showed the problem in 3.8.x, and didn't in develop – and the recommendation would just be: use the 4.0.0 prerelease(s) as a workaround.

I understand that you cannot share commercial data. Can you reproduce the problem using other data that you can share?

While functionally they are the same, the session/resource stuff is not safe to use across multiple threads and subprocesses, and the client stuff is. The former is a bit easier to use directly than the latter, but this does not impact the user, because smart_open is doing all the work.

I also refactored the way we pass keyword parameters to boto3. Previously, we had a separate parameter for each boto3 function. This was a pain, because this made parameter lists longer for each new function, e.g.

• resource_kwargs
• multipart_upload_kwargs
• singlepart_upload_kwargs
• object_kwargs

This PR moves all of the above parameters into a single nested dict and introduces a wrapper that transparently injects the parameters into the required function call.

This does break backwards compatibility, so will need a major version bump when releasing.

I can't share the data to make the code reproducible because it is commercial data. I removed the paths because they have no influence on the code but added a name of the fasttext model that I am using. A column "text" in a csv file contains strings with tokens that are separated with ", ".

train_text_df = pd.read_csv("text_train_data.csv", index_col=[0])
sents = []
for text_val in train_text_df["text"].values:
         sents.append(text_val.split(", "))
# sents = [['lord', 'of', 'the', 'rings'], ['lord', 'of', 'the', 'flies']]
model = load_facebook_model("cc.de.300.bin")
model.build_vocab(sents, update=True)
model.train(sentences=sents, total_examples = len(sents), epochs=10)
gensim.models.fasttext.save_facebook_model(model, "cc.de.300.tuned.bin"))

Updated https://github.com/RaRe-Technologies/smart_open/blob/client/MIGRATING_FROM_OLDER_VERSIONS.rst, please have a look.

Thank you. What are the values of config.RESOURCE_PATH and config.MODEL_PATH?

Ideally, your reproducible example should be something that I (or another developer) can run here locally on my machine and get the same result as you do. We can then try to do something about the bug.

Sure `train_text_df = pd.read_csv(os.path.join("..", config.RESOURCE_PATH, "text_train_data.csv"), index_col=[0]) sents = [] for text_val in train_text_df["text"].values: sents.append(text_val.split(", "))

sent = [['lord', 'of', 'the', 'rings'], ['lord', 'of', 'the', 'semi-groups']]

model = load_facebook_model(os.path.join("..", config.MODEL_PATH)) model.build_vocab(sents, update=True) model.train(sentences=sents, total_examples = len(sents), epochs=10) gensim.models.fasttext.save_facebook_model(model, os.path.join("..", config.RESOURCE_PATH, "cc.de.300.tuned.bin"))`

Followed the tutorial from https://radimrehurek.com/gensim/models/fasttext.html#gensim.models.fasttext.load_facebook_model

Yup, that's why I closed this PR without merging it.

Still a work in progress, it has been fixed since.

Are you able to share code that reproduces your problem?

@mpenkov 3.8.3

One comment in LdaModel.inference() confused me: # The optimal phi_{dwk} is proportional to expElogthetad_k * expElogbetad_w. And I think it should modified to this so the comment would be more accurate: # The optimal phi_{dwk} is proportional to expElogthetad_k * expElogbetad_kw.

Reasons: 1. According to Algorithm 1 on paper: Online Learning for Latent Dirichlet Allocation, NIPS 2010 http://www.cs.princeton.edu/~mdhoffma, phi_dwk is proportional to exp(Elogtheta_dk + Elogbeta_kw). 2. phi_{dwk} is a scalar, expElogthetad_k is a scalar, and expElogbetad_wis a vector with K elements, so expElogthetad_k * expElogbetad_w will return a vector with K elements, which does not match the type of phi_{dwk}. If we use expElogthetad_k * expElogbetad_kw instead, in which expElogbetad_kwis a scalar, it matches.

Merged, thank you @Witiko !

Currently, all documentation, Jupyter notebooks and code examples assume that WordEmbeddingSimilarityIndex is located in gensim.models, although it lives in gensim.similarities since 4.0.0. This pull request fixes the documentation.

I re-ran the tests, they all passed.

Related PR: https://github.com/RaRe-Technologies/gensim/pull/2966

Fix #2852