【論文読み会】Latent Dirichlet Allocation

潜在的ディリクレ配分法 (Latent Dirichlet Allocation) Blei et al., 2003 KaiRA 社会人メンバー 栗林 雷旗

Background Drawbacks of pLSI ( probabilistic Latent Semantic Indexing ) • The number of parameters grows linearly with the size of the corpus→Overfitting • It is not clear how to assign probability to a document outside of the training set. (Reference: Blei et al., 2003) Dimensionality Reduction Methods such as pLSI Bag of Words Assumption (Exchangeability) 2

LDA https://www.youtube.com/@SerranoAcademy 6

• https://www.youtube.com/@SerranoAcademy

LDA’s Role in Machine Learning Workflow LDA approaches the training phase in machine learning workflow. Data Collection Data Cleaning and Preparation Model Choosing Training Evaluation Tuning Deployment Monitoring and updating 7

LDA Architecture LDA ( Latent Dirichlet Allocation ) is a generative probabilistic model of a corpus. Creation of Topics Number of words in a given document 1 2 3 4 𝛽 Dataset 𝛼 𝜃𝑑 𝑧𝑑𝑛 𝑤𝑑𝑛 Frequency of topics per document N M observed word Document-topic Distribution Dirichlet parameter Word-topic assignment Number of documents D1 D2 D3 D4 8

LDA Generative Process Given the parameters 𝛼 and 𝛽, the joint distribution of 𝜃, 𝒛, 𝒘 is given by: 𝛼 𝑁 𝑝(𝜃, 𝒛, 𝒘) = 𝑝 𝜃 𝛼) ෑ 𝑝 𝑧𝑛 𝜃 𝑝(𝑤𝑛 |𝑧𝑛 , 𝛽) 𝑛=1 𝜃 𝜃: 𝑎 𝑡𝑜𝑝𝑖𝑐 𝑚𝑖𝑥𝑡𝑢𝑟𝑒 ቐ 𝒛: 𝑎 𝑠𝑒𝑡 𝑜𝑓 𝑁 𝑡𝑜𝑝𝑖𝑐𝑠 𝒘: 𝑎 𝑠𝑒𝑡 𝑜𝑓 𝑁 𝑤𝑜𝑟𝑑𝑠 𝑧𝑛 Example 𝜷 (𝒌 × 𝑽) 𝒘𝟏 𝒘𝟐 𝒘𝟑 𝒘𝟒 𝑧1 0.3 0.1 0.4 0.1 𝑧2 0.1 0.6 0.1 0.1 𝑧3 0.2 0.2 0.2 0.2 𝛽 𝑤𝑛 9

クラスタリングの流れ Step1: コーパスに変分ベイズ推論を適用してディリクレ分布のパラメータを推定 Step2: 推定されたパラメータの値を使って周辺化ギブスサンプリングを行い、各文書内の各単語にトピックを割り当て 10

Parameter Inference(Topic Distribution in Each Document) 𝛼Ԧ = (𝛼1 , 𝛼2 , … 𝛼𝑘 ) 11

Parameter Inference(Word Distribution in Each Topic) 𝛽Ԧ = (𝛽, 𝛽, … , 𝛽) 12