[DL輪読会]The Cramer Distance as a Solution to Biased Wasserstein Gradients

5IF
$SBNFS
%JTUBODF
BT
B
4PMVUJPO
UP
#JBTFE
8BTTFSTUFJO
(SBEJFOUT ॳ୩ྯ࣊

ϝλ৘ใɾϞνϕʔγϣϯ w #FMMFNBSF͞Μ
w 3FBDUPS
w 3FUSBDF
w 2Е
w 
w /*14
w ‫ػ‬ցֶश͸,-ͷ࠷খԽͷจ຺Ͱ‫ޠ‬ΒΕΔ͜ͱ͕ଟ͍͚Ͳɺ ͦΕͰ͍͍Μ͚ͩͬʁ

‫཭ڑ‬ͷ࿩ͷ෮श

,-μΠόʔδΣϯε ఆٛ P(x) KL(P || Q) = ∫ P(x)log dx Q(x) w ૬ରΤϯτϩϐʔ
w ৘ใརಘ
w ,-
JG
BOE
POMZ
JG
12
w ,-͸਺ֶత‫Ͱ཭ڑ‬͸ͳ͍
w ࡾ֯ෆ౳ࣜ☓
w ରশੑ☓

,-ͱ࠷໬ਪఆ KL(P || Q) = ∫ P(x)log P(x)dx − ∫ P(x)logQ(x)dx ఆ਺ ର਺໬౓ 1Λ‫ٻ‬Ί͍ͨ෼෍ɺ2Λֶश͍ͯ͠Δ෼෍ͱ͢Δͱ
,-ͷ࠷খԽ͸ର਺໬౓ͷ࠷େԽʹ౳͍͠

ճ‫ؼ‬ͷྫ ВͰύϥϝλϥΠζ͞ΕͨGʹ
/  М ͰαϯϓϦϯά͞ΕͨϊΠζΛՃ͑ͯ
֬཰෼෍ͱΈͳ͢ͱ 2⎞ ⎛ 1 Qθ (y | x) = A exp ⎜ − ( y − fθ (x)) ⎟ ⎝ 2 ⎠ − ∫ P(x)logQθ (x)dx = EP ⎡⎣ − logQθ ( x ) ⎤⎦ 1 2⎤ ⎡ = EP ⎢ − log A + ( y − fθ (x)) ⎥ 2 ⎣ ⎦ ೋ৐‫ࠩޡ‬

8BTTFSTUFJOμΠόʔδΣϯε W p ( µ,ν ) = inf p π ∈Γ( µ ,ν ) c(x, y) d π (x, y) ∫ p ෼෍ЖΛ෼෍W΁Ҡಈͤ͞Δ࣌ʹ͔͔Δίετ࿨ͷ࠷খ஋
෼෍Λ౔ͷΑ͏ͳ΋ͷͱͯ͠ߟ͑ΔͱЖͷ౔ΛWͷ౔ʹҠ͢
࠷దͳ༌ૹΛߟ͑Δ͜ͱͱಉ౳ Ж W

࿦จ֓ཁ ‫཭ڑ‬ई౓͕ຬͨͯ͠΄͍͠ҎԼͷੑ࣭ʹ͍ͭͯ
֬཰ม਺͕࣋ͭ‫ز‬Կֶతͳੑ࣭Λଊ͑Δ
‫ʹ཭ڑ‬ର͢Δޯ഑ਪఆ͕ෆภ
,-















OPU

BOE

8BTTFSTUFJO








BOE
OPU

$SBNFS















BOE

྆ํຬͨ͢$SBNFS‫཭ڑ‬ΛఏҊͯ͠("/ͱ͔SFHSFTTJPOͱ͔ɹɹɹ ৭ʑ࣮‫ͨ͠ݧ‬Α

֬཰ม਺ͷ‫ز‬Կֶతੑ࣭ w ,-Ͱ͸֬཰ͷີ౓ൺ͔͠‫͍ͨͳݟ‬Ίɺɹɹ

ɹɹɹ ֬཰ม਺ͷ‫ز‬Կֶతͳੑ࣭͸ଊ͑ΒΕ͍ͯͳ͍
w 8BTTFSTUFJO͸Dؔ਺͕֬཰ม਺ͷ‫ز‬ԿֶతͳҙຯΛ ΋ͭྔͱͳΔ ౔Λԕ͘ʹӡͿͷ͸ίετ͕େ͖͍
w ֬཰ม਺ͷ‫ز‬Կֶతੑ࣭Λ‫͕཭ڑ‬ଊ͑ͯ͘ΕΔͱ EJTKPJOUͳTVQQPSUͰ΋ֶशͰ͖Δ͠ɺɹɹɹɹɹɹ ը૾ͱ͔ྑͦ͞͏͡Όͳ͍ʁ

͜͜Β΁Μ͸໌‫͞ه‬Εͯ͸͍ͳ͍

֬཰ม਺ͷ‫ز‬Կֶతੑ࣭ͱ͸ʁ YZೋ࣍‫ݩ‬ฏ໘্ͷ఺͕
֬཰ม਺ͷ࣮‫ݱ‬஋ͷ࣌Λߟ͑Δ 1ͷTVQQPSU Z 2ͷTVQQPSU TVQQPSU

\Yc1 Y 㱠^
֬཰͕ΑΓେ͖͍֬཰ม਺ͷू߹ Y

֬཰ม਺ͷ‫ز‬Կֶతੑ࣭ͱ͸ʁ YZೋ࣍‫ݩ‬ฏ໘্ͷ఺͕
TVQQPSU͕EJTKPJOU
֬཰ม਺ͷ࣮‫ݱ‬஋ͷ࣌Λߟ͑Δ Z 1ͱ2ͷTVQQPSU͕ॏͳ͍ͬͯͳ͍ ͷͰ
1ͷTVQQPSU ,-͸ఆ਺ 2ͷTVQQPSU TVQQPSU

\Yc1 Y 㱠^
֬཰͕ΑΓେ͖͍֬཰ม਺ͷू߹ Y

֬཰ม਺ͷ‫ز‬Կֶతੑ࣭ͱ͸ʁ YZೋ࣍‫ݩ‬ฏ໘্ͷ఺͕
֬཰ม਺ͷ࣮‫ݱ‬஋ͷ࣌Λߟ͑Δ 1ͷTVQQPSU Z 2ͷTVQQPSU TVQQPSU

\Yc1 Y 㱠^
֬཰͕ΑΓେ͖͍֬཰ม਺ͷू߹ Y

֬཰ม਺ͷ‫ز‬Կֶతੑ࣭ͱ͸ʁ YZೋ࣍‫ݩ‬ฏ໘্ͷ఺͕
֬཰ม਺ͷ࣮‫ݱ‬஋ͷ࣌Λߟ͑Δ Z 8BTTFSTUFJO͕খ͍͞ 1ͷTVQQPSU 2ͷTVQQPSU TVQQPSU

\Yc1 Y 㱠^
֬཰͕ΑΓେ͖͍֬཰ม਺ͷू߹ Y

֬཰ม਺ͷ‫ز‬Կֶతੑ࣭ͱ͸ʁ YZೋ࣍‫ݩ‬ฏ໘্ͷ఺͕
֬཰ม਺ͷ࣮‫ݱ‬஋ͷ࣌Λߟ͑Δ Z 8BTTFSTUFJO͕େ͖͍ 1ͷTVQQPSU 2ͷTVQQPSU TVQQPSU

\Yc1 Y 㱠^
֬཰͕ΑΓେ͖͍֬཰ม਺ͷू߹ Y

ཧ૝తͳ‫཭ڑ‬ई౓ͷఆࣜԽ w 4DBMF
TFOTJUJWF 4
w TVN
*OWBSJBODF *

,-ͱ8BTTFSTUFJO w ,-͸ͲͪΒ΋ຬͨ͞ͳ͍
w 8BTTFSTUFJO͸ͲͪΒ΋ຬͨ͢

΍ͬͺ8BTTFSTUFJOͬ͛͢ʔʁ

ޯ഑ਪఆͷෆภੑ 6 ∞ ∫ δ ( x − a )dx = 1 −∞ δ ( x) = 0 ( x ≠ 0) w w ਅͷ෼෍͸खʹೖΒͳ͍ͷͰ%JSBDͷσϧλؔ਺Λ ࢖ͬͯಘΒΕͨ‫ݧܦ‬෼෍ʹରͯ͠‫཭ڑ‬Λ‫͠ࢉܭ‬ɺɹ ޯ഑๏Ͱ࠷దԽ͢Δ %FFQͰΑ͋͘ΔྲྀΕ
m P̂(x) = m1 ∑ δ Xi i=1 ͦͷ࣌ͷޯ഑ͷਪఆ͕ෆภͰͳ͍ͱֶश͕ɹɹɹɹ ҆ఆ͠ͳ͔ͬͨΓؒҧͬͨ࠷ద஋ʹऩଋ͢Δ

,-ͱ8BTTFSTUFJO w ,-ͷ‫ݧܦ‬෼෍ʹର͢Δޯ഑ਪఆ͸ෆภ
"QQFOEJY
" w 8BTTFSTUFJO͸ෆภͰͳ͍ʁ
8BTTFSTUFJOͷਪఆޯ഑ͱਅͷޯ഑ͷࠩʹԼ‫͕͋ݶ‬Δ͜ͱ
ؒҧͬͨ࠷దղʹऩଋ͢Δ͜ͱ
ܾఆతͳ෼෍ʹऩଋͯ͠͠·͏໨ඪ෼෍͕ଘࡏ͢Δ
ͱ͍͏͕̏ͭ#FSOPVMMJ෼෍ͷ৔߹ʹ‫͞໌ূͯͬݶ‬Ε͍ͯΔ
"QQFOEJY
" ͨͩɺVOCJBTFEͳ8BTTFSTUFJOͷޯ഑͕ಘΒΕΔ͔Ͳ͏͔ࣗମ͸PQFO
QSPCMFNͱ͍ͯ͠Δ

‫ط‬ଘ‫ڀݚ‬ w 8BTTFSTUFJOΛ࢖ͬͨ‫ػ‬ցֶश͸ɹɹɹɹɹɹɹɹ ݁‫ہ‬,-ͱ૊Έ߹Θ͞Ε͍ͯΔࣄ͕ଟ͍
w 8("/͸VOCJBTFEͳਪఆ͕Ͱ͖ͯΔͬΆ͍ʁ

$SBNFS
%JTUBODF ∞ l ( P,Q ) = ∫ | FP (x) − FQ (x) | dx p p p −∞ '͸ྦྷੵ෼෍ w 4 ͱ * Λຬͨ͠ͳ͕Βɺ 6 ΋ຬͨ͢৽͍͠‫཭ڑ‬ $SBNFS
%JTUBODFΛ࢖ͬͨΒྑͦ͞͏
w Qʹ͔͗ͬͯ 4 * 6 ͢΂ͯຬͨ͢ "QQFOEJY
"

༨ஊ Ұ࣍‫ʹݩ‬ஔ͍ͯ
QͰ͸8BTTFSTUFJOͱ$SBNFS͸‫׬‬શʹҰக͢Δ 1 W1 (P,Q) = ∫ | F (µ ) − F ( µ ) | d µ 0 ∞ −1 P −1 Q l1 ( P,Q ) = ∫ | FP (x) − FQ (x) | dx −∞

༨ஊҰ࣍‫ݩ‬8BTTFSTUFJO Y Y c(x1 , y1 ) + c(x2 , y2 ) < c(x1 , y2 ) + c(x2 , y1 ) ࠷ద༌ૹ͸୯ࣹͳ΋ͷͷΈʂʂ Z Z

༨ஊߴߍ਺ֶ ྦྷੵ෼෍ؔ਺ Z  ͔ͬͪ͜Β‫ݟ‬Δ  Y ͔ͬͪ͜Β‫ݟ‬Δ

&OFSHZ
%JTUBODF w ͜͜·Ͱͷ࿩Ұ࣍‫ͨͬͩݩ‬ͷͰɺ$SBNFSΛɹɹɹɹɹɹ ଟ࣍‫֦ʹݩ‬ு͠·͠ΐ͏
‎ &OFSHZ
%JTUBODF
΍ͬͺΓ͜Ε΋ 4 ͱ * ͱ 6 Λຬͨ͢ "QQFOEJY
"

‫཭ڑ‬ई౓ൺֱ࣮‫ݧ‬ w w ؆୯ͳ཭ࢄ֬཰෼෍ʹ੍‫ݶ‬ΛՃ͑ͯ
w ,-
w 8BTTFSTUFJO
w $SBNFS
Λ‫ޯͯ͠ͱ཭ڑ‬഑๏Ͱֶशͯ͠ൺֱ࣮‫ݧ‬Λߦͬͨ

‫཭ڑ‬ई౓ൺֱ࣮‫݁ݧ‬Ռ w Ұ൪ࠨ͕໨ඪͷ෼෍Ͱֶश͢Δ෼෍͸̍ͱ̍̌ͷɹ ֬཰͕ಉ͡ʹͳΔΑ͏ͳ੍‫͕ݶ‬Ճ͑ΒΕ͍ͯΔ
w $SBNFS͸ਅͷ8BTTFSTUFJOΛ࠷খʹ͢Δ෼෍ʹɹ Ұ൪͍ۙ‫ܗ‬Λֶश͍ͯ͠Δ

3FHSFTTJPO࣮‫ݧ‬ w ͍ͭͷ೥ͷՎ͔͋ͯΔճ‫ؼ‬໰୊
w ೥Λ཭ࢄΫϥεʹͯ͠෼෍ͱͯ͠ද‫
ݱ‬ w $SBNFS͕Ұ൪3.4&Λখͨ͘͞͠

$SBNFS
("/ w &OFSHZ
%JTUBODFΛར༻ͯ͠("/Λֶश
w ୯७ʹ&OFSHZ
%JTUBODFΛ࢖͏ͷͰ͸ͳ͘ɺ %JTDSJNJOBUPSͷΑ͏ͳUSBOTGPSNFSؔ਺Λ‫ͤ·ט‬ ͍ͯΔ
w ී௨ͷ("/ͷଞʹɺ൒෼ͷը૾͔Βɹɹɹɹɹɹ ΋͏൒෼Λ෮‫ͤ͞ݩ‬Δ("/Λ΍͍ͬͯΔ

$SBNFS
("/
"MHPSJUIN

$SBNFS
("/ଟ༷ੑ 8("/(1 w $SBNFS("/ 8("/(1ʹൺ΂ͯ෮‫ͨ͠ݩ‬൒෼ʹଟ༷ੑ͕͋Δ

$SBNFS
("/ଟ༷ੑ 8("/(1 w $SBNFS("/ 8("/(1ʹൺ΂ͯ෮‫ͨ͠ݩ‬൒෼ʹଟ༷ੑ͕͋Δ

$SBNFS
("/‫݈ؤ‬ੑ w $SJUJDͷߋ৽ճ਺Λม͑ͯΈͨάϥϑ
w $SBNFS
("/͸ߋ৽ճ਺ͷӨ‫ڹ‬Λड͚ͮΒ͍

5SBOTGPSNFSͷྗ

$SBNFS
("/·ͱΊ w ,-͕Կͷ‫࢖ʹ͠ͳؾ‬ΘΕ͍ͯΔ͚Ͳɺɹɹɹɹ 4 * 6 ͳͲͷ৚݅Λߟྀͯ͠‫ز‬Կֶతʹɹɹɹɹ ଥ౰ͳ‫཭ڑ‬Λߟ͑ͨ΄͏͕Α͍
w VOCJBTFEͰ͠8BTTFSTUFJOΛখ͘͢͞Δޯ഑Λɹ ਪఆͰ͖Δ$SBNFS
%JTUBODFΛఏҊ

$SBNFS
("/‫ײ‬૝ w ‫཭ڑ‬ई౓ͷຬͨ͢৚݅ʹ͍ͭͯͷߟ࡯͸͓΋͠Ζ͍
w 8BTTFSTUFJO͕CJBTFEͰ͋Δূ໌͕#FSOPVMMJ෼෍ ͷ͔࣌͠ͳ͍
w ‫ʹٸ‬5SBOTGPSNFSͱ͔(1͕ग़ͯ͘Δ
w #FMMFNBSF͞Μͷ*$.-௨ͬͨ‫ڧ‬Խֶश࿦จ͕Ҿ༻ ͞ΕͯΔ͚Ͳެ։͞Εͯͳ͍ͷͰ‫ͳʹؾ‬Δ