Implementation options of task to add weight

[vtyushkevich]

I have two ideas how we can improve the algorithm by adding weight to tokens.

• we can define frequency of appearance of words in list of texts which we get in find_similar function. So we can sort all words by frequency and give them float scores, for example from 0 to 1 (the most frequent to the most rare) and then use it in find_similar during calculating final score of similarity.
• and other way, there is interesting python library (https://pypi.org/project/wordfreq/), so we can get global frequency of each word in text list and use it in find_similar too.
  @quillcraftsman What do you think about it?

[quillcraftsman]

I have two ideas how we can improve the algorithm by adding weight to tokens.

• we can define frequency of appearance of words in list of texts which we get in find_similar function. So we can sort all words by frequency and give them float scores, for example from 0 to 1 (the most frequent to the most rare) and then use it in find_similar during calculating final score of similarity.
• and other way, there is interesting python library (https://pypi.org/project/wordfreq/), so we can get global frequency of each word in text list and use it in find_similar too.
  @quillcraftsman What do you think about it?

wordfreq looks interesting. May be we can use this by default with some language (use this coefficients or not). But there is a question about custom coefficients, how can we set these for some specific task (And can we do it with wordfreq). I've noted that wordfreq uses some special file for other languages, may be we can use our custom file.
Anyway I think we can try to use wordfreq and check the difference in result report. May be this library will be really useful later (may be not :))

[vtyushkevich]

Interesting article
https://www.newscatcherapi.com/blog/ultimate-guide-to-text-similarity-with-python

[quillcraftsman]

Interesting article https://www.newscatcherapi.com/blog/ultimate-guide-to-text-similarity-with-python

Yes, got it. I need time to read it closely.

[vtyushkevich]

To start with something I suggest to pass to find_similar function a dict which contains important words with weight 0 or 1. When function is executed it counts the number of important words and saves it to the instance of TokenText like a cos. After that we can use it to sort result list of texts

[quillcraftsman]

@vtyushkevich , only 1 or 0?

[vtyushkevich]

May be at the first stage just to try it
Later it is possible to add ability to define weights from 0 to 1 with floating point, for example the most important word is 1, less important is 0.8 etc
I am thinking about how it will be implemented for user on the frontend. For example, user should only check important words (0 or 1 system), or user should range all words from most important to less, or user should define any integer weights to words?

[quillcraftsman]

May be at the first stage just to try it Later it is possible to add ability to define weights from 0 to 1 with floating point, for example the most important word is 1, less important is 0.8 etc I am thinking about how it will be implemented for user on the frontend. For example, user should only check important words (0 or 1 system), or user should range all words from most important to less, or user should define any integer weights to words?

Got it. If some text will be very long and all words will be important we will make many duplicates with words and important words. In this case may be better to set important words in TokenText object. But If we will have a few important words and big text it will be uncomfortable to convert str text to TokenText.

May be we can make both of this options:

• send important word to the find_similar function
• send important coefficient in the TokenText instance.

???

[vtyushkevich]

If am I understand right:
we send to the find_similar list of instances of TokenText where one instance is one important word with some coefficient
then we use it in calculations
Please, correct me, if it is wrong

[quillcraftsman]

If am I understand right: we send to the find_similar list of instances of TokenText where one instance is one important word with some coefficient then we use it in calculations Please, correct me, if it is wrong

1 variant:

text = 'one two three'
texts = [TokenText('one', important=1.0), TokenText('two', important=0.0), TokenText('three')]
result = find_similar(text, texts)

2 variant:

text = 'one two three'
texts = ['one', 'two', 'three']
important = [{'one': 1.0}, {'two': 0.0}]
result = find_similar(text, texts, important=important)

Is it yours variant what a you talking about above?

Theoretically we can combine both variants. May be you meant something else, in this case show an example please.

[vtyushkevich]

My variant was closer to 2 variant, something like that:

text = 'one two three'
texts = ['one two', 'one three', 'five six']
important = {'one': 1.0, 'two': 1.0}
result = find_similar(text, texts, important=important)

[quillcraftsman]

My variant was closer to 2 variant, something like that:

text = 'one two three'
texts = ['one two', 'one three', 'five six']
important = {'one': 1.0, 'two': 1.0}
result = find_similar(text, texts, important=important)

Okay, understand. Like I said we can implement this one or both. Let's start with this one.