[DL Hacks 実装]Attention is All You Need

1 ,. 11 . . . 11 . . . 1 1 J G . G S 1 P . N . 1 GY . US L AI K V

: • • .2 n – / – n – A – ln rN – . .2 22 . /: ci : A b 2 N 2 gfd N o ab N s k m a G p gfd t Y w 2 u A N b Nm ci k gfd N e h

• https://github.com/tensorflow/tensor2tensor

& A 4B3A 23:7 B - 45 3 1 56 2 7 .5 B A 5 4 4 A53AB 5 AA5 A SR M M A5 6 I

DIG IA D • g r N vsg V T – o M7 Q s M gnRe ( g • 7 TQ M o lM D • q ld o M g l • SiM. 29 ) g1892q Q M g – q Q • 6 A 7 GG DI • DK C IA D C 8 v 8 9 g M II DIA D • • – II DIA D ). kVe G C IL G O D I 8 D a q Qo dcMu c8 9 g q Q tv V Nsq Q Q d T o d q R VPo e N w q o - 0) . q Q UUo g 1 GDAD Q g e DK8 8 O M Tn e Qo vsc t tv g8 9 q q c o

B E • s g z – ) D . EB – ID . D – – P ..a T M • D pn i e ba x dN kro w U 2 kro d b P t u g Sf dN L ml hg Mf ml – C BD D B BI DG B C B B DG B f • ml – ( A C DD D B . DEB E – AB B B CDB D CE B D • c b – DBE DEB DD D D ) – yv C DD D i ml – x • DD Sf w i - B i f f BI i BI

2D A ,F • rgl D – 1 EIH 4D H D – 6 . – 6 - D – 6 ( - – 6 ) D DF D D D=H mz A. 2IAH HD HD 6 F HIF w[ Fcfa D D D F H A A EIH > n > s N3 D x F h p D i ,HH H D 5 F DF AM HD F DF AM HD D F y R 6 > hP u F dbe ] I A DF F 5 I A 3 5 Rto Fcfa 2IAH AM HD HD A HD A N3 ,HH H D 2IAH F DF AM F DF AM 5 ,HH HD HD I A D HD HD EIH 6 DF F A 5 5 2 F I A1 I A 4

( 3 • ( 3 ] 1 – – - 3A1 – 1 3 ] • 1 3 [D dt – ( 3 R RN A – A 1 go 3 .1 N A ) 13 6 1 [D aM 13 3 R ) 13 3 3 uy S 1 N rh H E( 3 e e fE A ) 13 sk 3 Em cS L i nl 3 5 2 1 4

. • 6 3 5 2 1 4

. • 6 3 5 2 1 4

) 2 • ko 2= l 2 , 21 – 2= 2 l S t S = 1 ) 2 ko !D 2 ko ! a n ! ) 2 gP D 2 ko v!" q s • 11Pa 2 x a Du cdem n • !" f i D#$ % f P D 3 ko f P D Pa yr a ]D !" s p – ko (ko – ko (ko i D 23 ) 2 i D [ -ko (2 &'()*+,( ./0 12 ) ko go a ko (12. i 2 D p DA 2

* ) ,HH H ) 1 H / ,HH H – ru = ,HH H _ehT dad cT mN N ( #0# lo p ) * .K n .K H= H= - H 3 E D H SMqs H H= D H 2# E# H ,HH # H Mtih Q PV # q A H # H H # K - H 3 H H ) # K H= H ,HH H _fN

-.- -- -

2 - d – – – M c a 3 • D 3 • -3 2 fEM A M .EM M . 3 2 −∞ EM e S A g i A A Mh

2 .32 . 2 3 M – 2 3 .32 3 – f 3 .2-E 2 3 2 .32 2 .32 .2 3 2 .32 2 .32 gM 3 3 E 3 3 M e 2 .32 D 2 .32 ka i A .2- dc

-.- -- -

, - • ) . ( . 3D – k N( – EED. 3 NP 3 E D EE P. iP d 1 C Pa e F 1 E - cf C

44 4 3 4 I > > 44 • – WT • 3 4 . > D 44 E 3 4 I > E U OM LS 3 4 I > 44 44 > E 3 4 I <-> <-> , voc 44 > <-> 4 E .. 44 3 4 I L

55 • ( ))[ ))R siE – – – – n S 55 R 3 dl 25 ERA 55 ( 25 b v A P 3 5 3225 . AE o S tC siE g [ ]aA a N ce P m S v n

- • RNN CNN 1. 2. 3. •

=R O OI =R O OI 2 T O 1 T OI – 7= )AGO GB. ) N MM PO SGOTGO G RS (, TPLGOS – 7= )AGO HRB. ( 7 SGOTGO GS O S M T TPLGOS OTP ( – rzbtcy mad k ilh[e u VP cSPUR G T RIGTva 4 R W RG O ( ) G UMG – - 8 5250 : 3: S – 1 SG NP GM. )SG TR O OI STG 1 I NP GM. SG – 1 SG NP GM. STG S PURS 1 I NP GM. ( T N GR – 0 TR O OI STG STG S ( N !" = 0.9, !( = 0.98 ϵ = 10,- GIUM R T PO – 2RP PUT. RGS U M bw – 0TTGOT PO RP PUT nf]bopa RP PUT R TG/ – 6 DGM NPPT OI. ./0 = 0.1 g s S

/6 - 48 6 1= – 6 – 6 – 6 = O 0.1 S E TRU 2 UTRUL OM A 2 B A 2 B

G – , B: D 0 GD • + – ,DB BG – 1G G NMO = 4D B D = = & ! = 0.6 = G = ' BH D +D : = = BD .G : B = SR5 BB 4D B 6

2 – !" – . 2 – D 6 6

2 – !" – . 2 – D 6 6

. 3 – transfomer – Recurrent Neural Network Grammar

• – Self attention encoder-encoder attention

& on • on l . – af – – . – P – NDN . on m e I . B IicE I Xh A

- 03/17 i s s I D E BE i ha c • – P S P e N v i t326 – • . d N i dgXN 26i v – r p – – . – P • II gf E : B Md cN II DIAEDgfh i E E i yP i o gfn b d D I cv re i ilv E hT cN: g gid e i)ai AD B DAD (& gf upg rw v & & h r h r egr hN h g : cN p m v P P d d idN_i r I B N i ie d

• https://deepage.net/machine_learning/2017/07/06/beam-search.html

) 0D e e A • – Nd A – xsvxi – ( dc e Id C a f % 3X f f tow m I C I S C f d nEl IG C S I y S f u ruv P

-- D • . Ab – – b b • • . – – bIl l > b c f c b b N Ab 0 Nc 0 X N c c e b c x n e P b n eT b N m l bNi N l T 1 1 Ab E m b N l N TeT n go ld swp tv urv