FastText Language Model

Date: 23.05.18

Writer: 9tailwolf : doryeon514@gm.gist.ac.kr

Introduction

We used Word2Vec model. It is a effiquency language model but it has a problem with word that present rarely. And It can’t process unknown word. To solve that problem, Researcher suggested FastText model.

FastText Language Model

FastText Language Model is a method that encoding words by subword. For example, where can be divided at 3-gram model. $<$ or $>$ represented start of the word or end of the word.

$$ where = <wh, wh, whe, her, ere, re>, where $$

And the below figure is a all process of FastText.

Process of FastText

Process of FastText is simillar as Skip-gram Model. By minimizing the negative log-likehood function to find optimal coefficient.

$$ -\frac{1}{N} \Sigma_{n=1}^{N}y_{n} \log (f(BAx_{n})) $$

See Here for Skip gram model.