[DL輪読会]DeepLearningと曲がったパラメータ空間 (Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation)

DEEP LEARNING JP [DL Papers] Deep Learning  
 Reiji Hatsugai, DeepX http://deeplearning.jp/ 1

• http://deeplearning.jp/

M\kxgn[UFW
• -6@@:3<YH,:@96?(.<7=?;3A:=<(03A?:B[UFW"O^P
• 
\'
–
–
–
–
–
–

^S]R\

$[

Ec

L\mqliXP

+/\&
tgjf{ow|xyplmz|i
hvls
*3A3@A?=>9:4(7=?86AA:<8
11\b
11\rxliulik

• -YH,.0\![UFW"O^P
• d:2352?=XFWF^P
–

VaHTSaP_^QeCC

• ISKNeWJ^PDD
– EYX %P`G[HKFW^PCX]d#Pc

[Zb^P
)

     $
   $
"&#   $!%&#

   !%
   %
#'$   %"& '$



 •  
   • 



F+,/

• RUSXOIL CK
•
BDE
WN?A@GK L>K
• HRUSXO!ITXMVPQ")
– *.JHL CK $3'5'#170;;64610<698%
–  I-.&26=3:53813L>K


, 

, https://www.jstage.jst.go.jp/article/sicejl1962/40/10/40_10_735/_pdf

(


  


  

&   '  •  #+/,*0-/.1'&$!' # )%"(


     
 

     $
   $
"&#   $!%&# 


•  $$!#.



• ",-,-3+(<
• 07.")&&(,-3+(<  2*0&/&'4%5
•  $$!#6+5 1 .+5:8;98

• 1
1



D.P.Kingma, Adam: A Method for Stochastic Optimization, https://arxiv.org/pdf/1412.6980.pdf










• "/201!&'(-, -*($5.1(+(6#1(-,
– ,3&/0& 3&$1-/./-%2$1! "# $@

<

;

>?

• /-,&$)&/#$1-/&%../-4(+#1&2/3#12/&
– @CDBA




• /-,&$)&/#$1-/&%!&$2/0(3&../-4(+#1(-,
– @CDBA

• 7<;




989=:

John Schulman et al, Trust Region Policy Optimization, https://arxiv.org/pdf/1502.05477.pdf
Yuhuai Wu et al, Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation,
https://arxiv.org/pdf/1708.05144.pdf
Roger Grose & James Martens, A Kronecker-factored approximate Fisher matrix for convolutional layers, https://arxiv.org/pdf/1602.01407.pdf
James Martens & Roger Grose, Optimizing neural network with kronecker-factored approximation curvature,
https://arxiv.org/pdf/1503.05671.pdf


• https://arxiv.org/pdf/1502.05477.pdf
• https://arxiv.org/pdf/1708.05144.pdf
• https://arxiv.org/pdf/1602.01407.pdf
• https://arxiv.org/pdf/1503.05671.pdf

^RTQeZafbc[WYdfS
• ^RTQeDCL<2
– @I\b_fVE CP?H;D 9=J><
–   H?1%*@3M 3?1CP?:MEFHPB782

• b]bU

• Θ∗ FDG>?2ME@1'-00.,/OHMA 
• `Xc48I IHKNM
• E)&+#O><5($*+ "D>?2M

5@6Mg

C.M.Bishop, PRML (
) 213p
Hippolyt Ritter et al, A Scaleble Laplace Approximation for Neural Networks, https://openreview.net/pdf?id=Skdvd2xAZ

!

• https://openreview.net/pdf?id=Skdvd2xAZ

$#"!'% &"!
• 987305/)*+., (
(.
– $#"!
–  &"!

• :<;=6 -214

-&$!""&-

-

Nitish Shirish Keskar et al, On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima,
https://arxiv.org/pdf/1609.04836.pdf



• https://arxiv.org/pdf/1609.04836.pdf

$5'1&/-*.(!%3%231/0)*&"/1('33*.(
• ##EBGLKJIHG;AB:

=GD9

– 6GLKJI <>CLKJ I =GALKJC=GI
– ?DLKJIC8G-4,3*3%2+,'%1.*.(BF


• LKJ@DMONPLDD

7

I<B9FLKJ I 

James Kirkpatrick et al, Overcoming catastrophic forgetting in neural networks, https://arxiv.org/pdf/1612.00796.pdf



• https://arxiv.org/pdf/1612.00796.pdf

6;A<>745@A1*
• +8<9A3#
– 
– 9:=0

•  *745@A1",
 (8<9A30 -%.
• .8<9A3*#?2 )'/&$ 0!."
–

*()

Yann Le Cun et al, Optimal Brain Damage, http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf
Yoojin Choi et al, Towards the Limit of Network Quantization, https://arxiv.org/pdf/1612.01543.pdf



• http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf
• https://arxiv.org/pdf/1612.01543.pdf


•

HTWPMUPMN
–  L
;D<E9
– $4%01!0)!, /%0230"!2).-

•
•


– 8KRZO6L 9C@BVRXF:>K7
8KQNSRZO!"#$" DGYNG

(22/$%%/,%!0-)-'*/3-$%012!-$)-'",!#+".5/0%$)#2).-14)!)-&,3%-#%&3-#2).-1
–

??G

F@=9C8JIA

Pang Wei Koh, Understanding Black-box Predictions via Influence Functions, https://arxiv.org/pdf/1703.04730.pdf



• https://arxiv.org/pdf/1703.04730.pdf

     $
   $
"&#   $!%&#

#++&"(&+%#*($)*'",&)(",*&-:EFH
• =?:

5

<>6CGEIA :
:EFH1



– !#+ #,:CGEIA 
–
8EFH.≈ 10$% J

• B@ID9=? <>:;2430

• 87:

/8








• (


)!" ('

– 
– &#!  %$%#!" !#%% 
– "% # 

• $$

C.M.Bishop, PRML p251
D.P.Kingma, Adam: A Method for Stochastic Optimization, https://arxiv.org/pdf/1412.6980.pdf
James Kirkpatrick et al, Overcoming catastrophic forgetting in neural networks, https://arxiv.org/pdf/1612.00796.pdf
Yann Le Cun et al, Optimal Brain Damage, http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf

• https://arxiv.org/pdf/1412.6980.pdf
• https://arxiv.org/pdf/1612.00796.pdf
• http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf

!"# $ (& "# $).  *,  •  &#'!"# $ () & "# $ )-( • 1/012$ .
!"+. * %! "# $ (& "# $).  3 

?6!":=
• !*)(-&#,%"+#$'%),/ 
•
C :5A C3 !" = $<.= !%& $C@A
• 
 <FDEGx;708()C5A4:192B>!%& $C

http://www.slis.tsukuba.ac.jp/~fujisawa.makoto.fu/cgibin/wiki/index.php?%CF%A2%CE%A91%BC%A1%CA%FD%C4%F8%BC%B0%A1%A7%B6%A6%CC%F2%B8%FB%C7%DB%CB
%A1



• http://www.slis.tsukuba.ac.jp/~fujisawa.makoto.fu/cgi-

!"# $ (& "# $)=



9; ,#)"

•  526!"# $ () & "# $. 8<7• @>?@A3 =
01:= 9/4! "# $ (& "# $)= 
• !((#$ !)%' '% *) +&

B

import tensorflow as tf
grads = tf.gradients(loss, params)
hvp = tf.gradients(tf.reduce_sum(grads*x), params)



•    


%#" % &#%$$%#)!&'%(&'% • 17865
2 %#" %4*. • %#" %1 ! "# 4 0 -3,/+ 4*3/  

&$#!& ($&&)&' *%%&$+ "( $#
• 

1.


• 3-

2

69:87

2/,0

45



.>@6 5!"7

859

• 

– .
–

$**%#&/207/-= ;2.

• !"# $ %& ' "# $? 

– CDF/. 

9=
7%+$)#+%'&741 ,(?3=/-=

• HIEB

– 

.

– 
:<



– GA.



92 7:<b[c
• NXVoqprm%,`K*^
– ."K*a_

SiYKh

NK

• ;7>52/;756_!Z)PYjg[$]92/7:<kcfiheL^]XV
– 0dL

bXYhQ\1/
Sf^I]*^jiYP[L

– 

'`UZ^jiYKh_Z85=[MD@H? C@?FEBEA[M

• nlqr +`URK
• 9@GGB?E_(&`bWbWJh
• [_
– 7:<N

]-jg

#^(&ZOheL^]XYOV_Z]^MT^]f]KMs

43