[DL輪読会]`強化学習のための状態表現学習 －より良い「世界モデル」の獲得に向けて－

455 Views

November 27, 18

#deep learning #State Representation Learning #machine learning #learning model #algorithm #theory

スライド概要

2018/10/26
Deep Learning JP:
http://deeplearning.jp/seminar-2/

profile-image
スライド一覧

DL輪読会資料

シェア

またはPlayer版

埋め込む »CMSなどでJSが使えない場合

（ダウンロード不可）

関連スライド

slide-thumbnail

【DL輪読会】KAN: Kolmogorov–Arnold Networks
user-img Deep Learning JP 79.7K
slide-thumbnail

【DL輪読会】Evolutionary Optimization of Model Merging Recipes モデルマージの進化的最適化
user-img Deep Learning JP 57.4K
slide-thumbnail

【拡散モデル勉強会】拡散モデルの数理
user-img Deep Learning JP 47.5K
slide-thumbnail

【拡散モデル勉強会】Introduction to Diffusion Models
user-img Deep Learning JP 35K
slide-thumbnail

【拡散モデル勉強会】拡散モデルのサンプラーまとめ
user-img Deep Learning JP 30.3K
slide-thumbnail

【DL輪読会】Generative Agents: Interactive Simulacra of Human Behavior
user-img Deep Learning JP 27.2K
各ページのテキスト
1.

‫ڧ‬ԽֶशͷͨΊͷঢ়ଶද‫ֶݱ‬श ɹʔΑΓྑ͍ʮੈքϞσϧʯͷ֫ಘʹ޲͚ͯʔ Tatsuya Matsushima @__tmats__ , Matsuo Lab 1

2.

ྠಡ಺༰ʹ͍ͭͯ ຊൃදͰϕʔεͱ͍ͯ͠Δ࿦จ  4UBUF3FQSFTFOUBUJPO-FBSOJOHGPS$POUSPM"O0WFSWJFX • IUUQTBSYJWPSHBCT -BTUSFWJTFE+VO  • 5JNPUIÉF-FTPSU /BUBMJB%ÍB[3PESÍHVF[ +FBO'SBOÇPJT(PVEPV %BWJE'JMMJBU • 43-5PPMCPYͱ͍͏πʔϧ΋࡞੒͍ͯ͠ΔIUUQTHJUIVCDPNBSBGGJOSPCPUJDTSMTSM • ੍‫ޚ‬λεΫʹ༻͍Δঢ়ଶͷද‫ֶݱ‬शʹؔ͢ΔϨϏϡʔ࿦จ • 6$#FSLFMFZΛத৺ʹ੝Μʹ‫͞ڀݚ‬Ε͍ͯΔ෼໺ • ೔ຊͰ͸͋Μ·Γ‫͕͢ؾ͍ͳݟ‬Δ • Χόʔ͞Ε͍ͯͳ͍ଞͷ࿦จ΋ຊൃදͰ͸௥Ճͨ͠ 2

3.

ঢ়ଶද‫ֶݱ‬श(SRL)ͱ͸ʁ ද‫ֶݱ‬श SFQSFTFOUBUJPOMFBSOJOH  • σʔλ͔ΒBCTUSVDUͳಛ௃Λ‫͚ͭݟ‬Δֶश ঢ়ଶද‫ֶݱ‬श TUBUFSFQSFTFOUBUJPOMFBSOJOH 43-  • ঢ়ଶ TUBUF ද‫ͱݱ‬͸ɼֶशͨ͠ಛ௃͕௿࣍‫Ͱݩ‬ɼ࣌ؒతʹൃల͠ɼΤʔδΣϯτͷߦಈͷ Ө‫ڹ‬Λड͚Δ΋ͷ • ͜ͷΑ͏ͳද‫ݱ‬͸ϩϘςΟΫε΍੍‫ޚ‬໰୊ʹ༗ӹͰ͋Δͱߟ͑ΒΕΔDG ࣍‫ݩ‬ͷढ͍ • ྫ ը૾৘ใ͸ඇৗʹߴ࣍‫͕ͩݩ‬ɼϩϘοτͷ੍‫ޚ‬ͷ໨తؔ਺͸΋ͬͱ௿࣍‫ʹݩ‬ද‫͞ݱ‬Ε͏Δ • ϚχϐϡϨʔγϣϯͷ৔߹ɼ෺ମͷ࣍‫ݩ‬ͷҐஔ৘ใ • ੜͷ‫؍‬ଌσʔλ͔Β͜ͷঢ়ଶද‫ݱ‬Λ‫͚ͭݟ‬Δख๏ͷ‫͕ڀݚ‬ओཁͳςʔϚ 3

4.

ੈքϞσϧ ஌ೳʹ͓͚Δ.PEFM#VJMEJOH<-BLF >ͷॏཁੑ • ਓؒ͸͋ΒΏΔ΋ͷΛ஌֮Ͱ͖ΔΘ͚Ͱ͸͘ɼ৘ใ ܹࢗ ͔ΒੈքΛϞσϧԽͨ͠಺෦Ϟ σϧΛ࡞Γɼਓؒͷ஌ೳʹେ͖ͳ໾ׂΛ୲͍ͬͯΔͱࢥΘΕΔ • ੈքϞσϧͱ΋͍͏ • <%-ྠಡձ>(2/ͱؔ࿈‫ڀݚ‬ɼੈքϞσϧͱͷؔ܎ʹ͍ͭͯ IUUQTXXXTMJEFTIBSFOFU%FFQ-FBSOJOH+1EMHRO • ʮࠓ·Ͱͷ‫ه‬Ա͔ΒະདྷΛ༧ଌ͢ΔྗɽͦΕ͕஌ೳͰ͋Δɽʯ • δΣϑɾϗʔΩϯεʰߟ͑Δ೴ɾߟ͑Δίϯϐϡʔλʱ • ֶशͨ͠಺෦Λ༻͍ͯະདྷΛγϛϡϨʔγϣϯ͠ͳ͕Βߦಈ͍ͯ͠Δ ͱߟ͑ΒΕΔ 4

5.

ྑ͍ද‫ͱݱ‬͸ʁ ੜͷ‫؍‬ଌ৘ใͷແؔ࿈ͳ෦෼Λແࢹͯ͠ɼ‫ڧ‬Խֶशʹར༻͢ΔͨΊʹඞཁෆՄܽͳ ৘ใΛΤϯίʔυ͢Δ͜ͱ͕ඞཁ <#ÖINFSFUBM >ʹΑΔྑ͍ঢ়ଶද‫ݱ‬ͷఆٛ • Ϛϧίϑੑ͕͋Δ • ‫ࡏݱ‬ͷঢ়ଶͷΈΛ‫ݟ‬Δ͚ͩͰɼ͋ΔํࡦΛ༻͍ͯߦಈΛબ୒͢Δ͜ͱ͕Ͱ͖Δ͙Β͍े෼ͳ৘ ใΛཁ໿͍ͯ͠Δ • ํࡦͷվળͷͨΊʹར༻Ͱ͖Δ • ಉ͡Α͏ͳಛ௃Λ࣋ͭ‫ͱͨ͜ݟ‬ͷͳ͍ঢ়ଶʹɼֶशͨ͠Ձ஋ؔ਺Λ൚ԽͰ͖Δ • ௿࣍‫͋Ͱݩ‬Δ 5

6.

SRLͷҰൠԽ ˜ st ∈ 𝒮 43-Ͱ͸ਅͷঢ়ଶɹɹɹΛ࢖Θͣʹɼ͜ΕΛۙࣅ͢ΔΑ͏ͳঢ়ଶɹɹɹΛֶश͢Δ s̃t ∈ 𝒮 o1:t st = ϕ (o1:t) st • ա‫ڈ‬ͷ‫؍‬ଌɹɹ͔Β‫ࡏݱ‬ͷঢ়ଶɹ΁ͷϚοϐϯάɹɹɹɹɹɹͷֶश ߦಈ at ∈ 𝒜 ਅͷঢ়ଶ ෆ໌ at ˜ s̃t ∈ 𝒮 s̃t ‫؍‬ଌ ot ot ∈ 𝒪 s̃t+1 ใु ot+1 6

7.

SRLͷΞϓϩʔν 43-ͷΞϓϩʔνʹ͸͍͔ͭ͘ύλʔϯ͕͋Δ • ࣗ‫߸ූݾ‬Խ‫ ث‬BVUPFODPEFS ͷར༻ • ॱϞσϧ GPSXBSENPEFM ͷར༻ • ‫ٯ‬Ϟσϧ JOWFSTFNPEFM ͷར༻ • ࣄલ஌ࣝ QSJPS ͷಋೖ 7

8.

SRLͷΞϓϩʔν ࣗ‫߸ූݾ‬Խ‫ ث‬BVUPFODPEFS ͷར༻ • ࠶ߏ੒‫ࠩޡ‬ͷ࠷খԽΛ͢Δ͜ͱͰɼΤϯίʔμɹͱσίʔμɹɹΛֶश ϕ ϕ −1 st • ͦͷࡍɼঢ়ଶɹ͕͋Δੑ࣭Λ࣋ͭΑ͏ʹ੍໿Λ͔͚Δ • ྫ ࣍‫ݩ‬ͷ੍໿ɼϊΠζͷআ‫ ڈ‬EFOPJTJOH ɼεύʔεੑͷ੍໿ Τϯίʔμ st = ϕ (ot; θϕ) σίʔμ ot̂ = ϕ −1 (st; θϕ−1) ࠶ߏ੒‫ࠩޡ‬ 8

9.

SRLͷΞϓϩʔν ॱϞσϧ GPSXBSENPEFM ͷར༻ st+1 at st f • ॱϞσϧɹ͸ঢ়ଶɹͱߦಈɹΛ༻͍ͯ࣍ͷঢ়ଶɹɹΛ༧ଌ • ॱϞσϧʹઢ‫ܗ‬ม‫Ͳͳ׵‬ͷ੍໿Λ͔͚Δ͜ͱ͕Ͱ͖Δ ϕ • Τϯίʔμɹ͸࣍ͷঢ়ଶͷ༧ଌ‫ࠩޡ‬Λ‫఻ٯ‬೻ͤ͞Δ͜ͱͰֶश͞ΕΔ ॱϞσϧ ̂ = f (st, at; θf wd) st+1 Τϯίʔμ st = ϕ (ot; θϕ) ࣍ͷঢ়ଶͷ༧ଌ‫ࠩޡ‬ 9

10.

SRLͷΞϓϩʔν ‫ٯ‬Ϟσϧ JOWFSTFNPEFM ͷར༻ st+1 at st • ঢ়ଶɹͱ࣍ͷঢ়ଶɹɹ͔Β࣮ࡍʹऔΒΕͨߦಈɹΛਪఆ͢Δ at ϕ • Τϯίʔμɹ͸࣮ࡍʹͱΒΕͨߦಈɹͷ༧ଌ‫ࠩޡ‬Λ‫఻ٯ‬೻ͤ͞Δ͜ͱͰֶश͞ΕΔ ࣮ࡍʹऔΒΕͨ ߦಈͷ༧ଌ‫ࠩޡ‬ ‫ٯ‬Ϟσϧ at̂ = g (st, st+1; θinv) Τϯίʔμ st = ϕ (ot; θϕ) 10

11.

SRLͷΞϓϩʔν ࣄલ஌ࣝ QSJPS ͷಋೖ • ಛఆͷ੍໿΍μΠφϛΫεʹؔ͢Δࣄલ஌ࣝΛར༻͢Δ • ྫ ࣌ؒతͳ࿈ଓੑ s1:n c • ࣄલ஌ࣝ͸͋Δ৚݅ɹͷ΋ͱͰɼঢ়ଶͷू߹ɹɹʹద༻͞ΕΔMPTTΛ௨ͯ͡ఆٛ͞ΕΔ • Loss = ℒprior (s1:n; θϕ | c) Τϯίʔμ st = ϕ (ot; θϕ) ঢ়ଶͷۭؒࣗମʹ ੍໿Λ͓͘ 11

12.

Why SRL? ͳͥ43-Λߟ͑Δ΂͖ͳͷ͔  • ੜͷ‫؍‬ଌ͔ΒFOEUPFOEʹ௚઀‫ڧ‬Խֶश͢Δͷ͸ίετ͕ߴ͍ • 43-Ͱྑ͍QSJPSΛೖΕͯ͋͛Δ͜ͱ͕Ͱ͖Δ͔΋ • ϚϧνϞʔμϧͳ‫؍‬ଌʹ֦ு͠ಘΔ • ؔ࿈ͨ͠λεΫΛࣄલʹղ͘͜ͱͰసҠֶशʹར༻Ͱ͖Δ • ਐԽઓུ &4 ͳͲͷɼ࣍‫ࡧ୳͕ݩ‬εϐʔυʹ௚݁͢ΔΑ͏ͳΞϧΰϦζϜΛ࠾༻͢Δ͜ͱ ͕ՄೳʹͳΔ 12

13.

‫ط‬ଘͷ‫ڀݚ‬ͷ঺հͱ෼ྨ 13

14.

‫ڀݚ‬ͷ෼ྨ ෼ྨͷํ๏ • ֶशͷ໨తؔ਺ • ‫؍‬ଌۭؒɾߦಈۭؒͷઃ‫ܭ‬ • ঢ়ଶද‫ݱ‬ͷධՁࢦඪ • ධՁʹ༻͍ΔλεΫ 14

15.

ֶशͷ໨తؔ਺ • ‫؍‬ଌͷ࠶ߏ੒ • ॱϞσϧ GPSXBSENPEFM ͷֶश • ‫ٯ‬Ϟσϧ JOWFSTFNPEFM ͷֶश • ಛ௃ͷఢରతֶशͷ‫༻׆‬ • ใुͷ‫༻׆‬ • ͦͷଞͷ໨తؔ਺ • ϋΠϒϦουͳ໨తؔ਺ 15

16.

ֶशͷ໨తؔ਺ ‫؍‬ଌͷ࠶ߏ੒ • ࣍‫ݩ‬ѹॖͱͯ͠Α͘࢖ΘΕΔํ๏ • ྫ 1$"<$VSSBO >ɼ%"&ɼ7"&<WBO)PPG >ɽ • ࣗ‫߸ූݾ‬Խ‫ ث‬BVUPFODPEFS Λ࢖͏ख๏͕ଟ͍ • ը૾ͷ‫؍‬ଌΛͦͷ··࢖͏<.BUUOFS > • ΦϒδΣΫτͷҐஔΛද‫͢ݱ‬ΔΑ͏ʹ੍໿͢Δྫ 4QBUJBM4PGUNBY<'JOO > • ‫؍‬ଌʹ໨ཱͭಛ௃͕ଘࡏͯ͠ͳ͍ͱ୯ʹ‫؍‬ଌΛ࠶ߏ੒͢Δ͚ͩͰ͸ྑ͍ද‫ʹݱ‬͸ͳΒͳ͍ • • ྫ ήʔϜʹ͓͚Δখ͍͞ΞΠςϜ ҧ͏࣌ؒεςοϓ͔Β࠶ߏ੒ͨ͠Γɼ࣌ؒൃలʹ੍ؔͯ͠໿Λ͔͚Δ͜ͱͰରԠ 16

17.

ֶशͷ໨తؔ਺ ॱϞσϧ GPSXBSENPEFM ͷֶश • ঢ়ଶ͕࣍ͷঢ়ଶΛ༧ଌ͢Δͷʹඞཁͳ৘ใΛΤϯίʔυ͢ΔΑ͏ʹ͢Δ • ‫؍‬ଌͷ࠶ߏ੒ͱΑ͘૊Έ߹ΘͤΒΕΔ • ঢ়ଶۭؒʹ͓͚ΔભҠΛઢ‫ͱܗ‬Ծఆ͢Δ͜ͱ͕ଟ͍ ̂ = Wst + Uat + V st+1 17

18.
[beta] 
E2C [Watter+ 2015]
&NCFEUP$POUSPM"-PDBMMZ-JOFBS-BUFOU%ZOBNJDT.PEFMGPS$POUSPMGSPN
3BX*NBHFT
• 7"&Λ༻͍ͨॱϞσϧɽঢ়ଶ જࡏද‫ݱ‬st ɹͷભҠΛઢ‫͋Ͱܗ‬ΔͱԾఆ
̂ ∼ 𝒩 (μ = Wst + Uat + V, σ)
st+1
ઢ‫ܗ‬

st+1
• ࣍ͷ࣌ؒεςοϓͷঢ়ଶͷ༧ଌɹɹͱͦͷঢ়ଶɹɹͷ,-Λ
̂
st+1
͚ۙͮΔ͜ͱͰॱϞσϧΛֶश
• ΧϧϚϯϑΟϧλͱͯ͠ఆࣜԽͨ͠΋ͷ΋͋Δ %7#'
<,BSM >

18
19.

World Model [Ha+ 2018] 8PSME.PEFMT • 7"&ͱ.%/3//Λར༻ͨ͠ॱϞσϧ • 7JTJPONPEFM 7 ߴ࣍‫ݩ‬ͷ‫؍‬ଌσʔλΛ7"&Λ༻͍ͯ ௿࣍‫ݩ‬ͷίʔυ ঢ়ଶ ʹѹॖ • .FNPSZ3// . ա‫ڈ‬ͷίʔυ͔Β࣍ͷεςοϓͷ ίʔυ ঢ়ଶ Λ༧ଌ <%-ྠಡձ>8PSME.PEFMT IUUQTXXXTMJEFTIBSFOFU%FFQ-FBSOJOH+1EMXPSMENPEFMT 19

20.

ֶशͷ໨తؔ਺ ‫ٯ‬Ϟσϧ JOWFSTFNPEFM ͷֶश • ͱͬͨߦಈΛਪఆͰ͖ΔΑ͏ʹঢ়ଶͷද‫੍ʹݱ‬໿Λ՝͢ • ྫ -FBSOJOHUP1PLFCZ1PLJOH<"HSBXBM > • ͍ͭͬͭͨҐஔ ɹ pt ɼ֯౓ ɹ θt ɼ‫ ཭ڑ‬ɹ lt Λਪఆ 20

21.

ICM [Pathak+ 2017] $VSJPTJUZESJWFO&YQMPSBUJPOCZ4FMGTVQFSWJTFE1SFEJDUJPO ℒf wd • ॱϞσϧͷ༧ଌ‫ࠩޡ‬ɹɹΛ‫ڧ‬Խֶशͷ಺తใुͱͯ͠ར༻ • ΤʔδΣϯτͷ֎෦͔Βͷใु͕εύʔεͳͱ͖ʹ୳ࡧΛଅਐ͢Δ 1 ℒf wd ϕ̂ (ot+1), f ̂ (ϕ̂ (ot), at) = ( ) 2 f ̂ (ϕ̂ (ot), at) − ϕ̂ (ot+1) 2 2 • ‫ٯ‬ϞσϧʹΑΔ-PTT΋ར༻ min [−λ𝔼π(st; θP) [Σtrt] + (1 − β)ℒinv + βℒf wd] θ ,θ ,θ P I F ֎తใु ‫ٯ‬Ϟσϧ ॱϞσϧ <%-ྠಡձ>-BSHF4DBMF4UVEZPG$VSJPTJUZ%SJWFO-FBSOJOH ઌिͷൃද IUUQTXXXTMJEFTIBSFOFU%FFQ-FBSOJOH+1EMMBSHFTDBMFTUVEZPGDVSJPTJUZESJWFOMFBSOJOH 21

22.

ֶशͷ໨తؔ਺ ಛ௃ͷఢରతֶश • ྫ $BVTBM*OGP("/<,VSVUBDI > • ("/ͷ໨తؔ਺ʹঢ়ଶͱ(FOFSBUPSͷग़ྗ ‫؍‬ଌͷϖΞ ͷ૬‫ޓ‬৘ใྔʹؔ͢Δਖ਼ଇԽ߲Λ௥Ճ min max G,Q,ℳ D V(G, D) − λIVLB(G, Q) ૬‫ޓ‬৘ใྔ 22

23.

ֶशͷ໨తؔ਺ ใुͷ‫༻׆‬ • 4-3ʹ͓͍ͯ͸ใुΛར༻͢Δ͜ͱ͸ඞͣ͠΋ඞཁͰ͸ͳ͍͕ɼঢ়ଶΛ۠ผ͢ΔͨΊͷ௥ Ճతͳ৘ใͱͯ͠ར༻͠͏Δ • ྫ 71/<0I > • ࣍ͷঢ়ଶͱͦͷঢ়ଶՁ஋΋༧ଌ ‫؍‬ଌ ঢ়ଶ ࣍ͷঢ়ଶՁ஋ ߦಈ ˞PQUJPOͷP ࣍ͷঢ়ଶ 23

24.

ֶशͷ໨తؔ਺ ͦͷଞͷ໨తؔ਺ • ࣮ੈքʹؔ͢Δࣄલ஌ࣝ QSJPS Λঢ়ଶۭؒʹ൓ө͢ΔͨΊʹɼ໨తؔ਺Λ޻෉͢Δ • ͍Ζ͍Ζͳ΋ͷ͕ఏҊ͞Ε͍ͯΔ • 4MPXOFTTQSJPS<-FTPSU  +POTDILPXTLJ > • ॏཁͳ΋ͷ͸Ώͬ͘Γͱ࿈ଓతʹಈ͖ɼ‫ͳܹٸ‬มԽ͕‫͜ى‬ΔՄೳੑ͸௿͍ ℒSlowness(D, ϕ) = 𝔼 [ Δst ] 2 • 7BSJBCJMJUZ<+POTDILPXTLJ > • ؔ܎ͷ͋Δ΋ͷ͸ಈ͘ͷͰɼঢ়ଶද‫ֶݱ‬श͸ಈ͍͍ͯΔ΋ͷʹ஫໨͢΂͖ ℒVariabilty(D, ϕ) = 𝔼 [e − st1 − st2 ] 24

25.

ֶशͷ໨తؔ਺ ͦͷଞͷ໨తؔ਺ • 3PCPUJD1SJPST<+POTDILPXTLJ >Ͱಋೖ͞Ε͍ͯΔ΋ͷ • 1SPQPSUJPOBMJUZ • ҧ͏ঢ়ଶͰ΋ಉ͡ߦಈΛͨ͠৔߹ʹ͸ɼঢ়ଶʹ‫͢΅ٴ‬Ө‫ڹ‬͸ಉఔ౓Ͱ͋Δ ℒProp(D, ϕ) = 𝔼 ( [ Δst2 − Δst1 ) | at1 = at2] 2 • 3FQFBUBCJMJUZ • ࣅͨঢ়ଶͰಉ͡ߦಈΛͨ͠৔߹ʹ͸ɼঢ়ଶʹ‫͢΅ٴ‬Ө‫ڹ‬͸ಉఔ౓ɾಉ͡ํ޲Ͱ͋Δ ℒRep(D, ϕ) = 𝔼 e [ − st2 − st1 2 Δst2 − Δst1 2 ] | at1 = at2 25

26.

ֶशͷ໨తؔ਺ ϋΠϒϦουͳ໨తؔ਺ • ࣮ࡍ͸ࠓ·Ͱʹ‫ͨ͛ڍ‬໨తؔ਺ͷ͏ͪɼෳ਺Λ૊Έ߹Θͤͯ43-͕ߦΘΕΔ͜ͱ͕ଟ͍ ߦಈ/࣍ͷঢ়ଶ ॱϞσϧ ※࣍ͷঢ়ଶͷ༧ଌ ͷ੍໿ E2C [Watter+ 2015] World Model [Ha+ 2018] ICM [Pathak+ 2017] Causal InfoGAN [Kurutach+ 2018] ࣍ͷ‫؍‬ଌͷ ༧ଌ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ [Oh+ 2017] [Jonschkowski+ 2015] ‫؍‬ଌͷ࠶ߏ੒ ✔ VPN Robotic Priors ‫ٯ‬Ϟσϧ ✔ ใुͷ‫༻׆‬ ✔ ✔ ✔ ✔ ✔ 26

27.

‫؍‬ଌɾঢ়ଶɾߦಈۭؒͷઃ‫ܭ‬ • ‫؍‬ଌɾঢ়ଶɾߦಈۭؒͷઃ‫ܭ‬͸໰୊ͷෳࡶੑʹӨ‫ڹ‬Λ‫͢΅ٴ‬ • Ͳͷ͘Β͍ͷ࣍‫ݩ‬ͷେ͖͔͞ɼߦಈ͕཭ࢄ͔ɾ࿈ଓ͔ • ௨ৗɼਅͷঢ়ଶΑΓ΋େ͖ͳঢ়ଶۭؒͷ࣍‫ݩ‬Λઃ‫͢ܭ‬Δ͜ͱ͕ଟ͍ • Ͳͷ͘Β͍ͷ࣍‫ݩ‬ͷେ͖͔͞ɼߦಈ͕཭ࢄ͔ɾ࿈ଓ͔ • ঢ়ଶΛͲͷ͙Β͍ͷ࣍‫͢ʹݩ‬Ε͹͍͍͔Α͘Θ͔Βͳ͍λεΫ΋ଟ͍ྫ "UBSJ Robotic Priors [Jon-schkowski+ 2015] E2C [Watter+ 2015] ICM [Pathak+ 2017] ‫ڥ؀‬ ‫؍‬ଌͷछྨ ‫؍‬ଌۭؒͷ࣍‫ݩ‬ ঢ়ଶͷ࣍‫ݩ‬ ߦಈ slot car racing ը૾ 16×16×3 2 ཭ࢄ(25) cart-pole ը૾ 80×80×3 8 ཭ࢄ Mario Bros. ը૾ 42×42×3 2 ཭ࢄ(14) 27

28.

ঢ়ଶද‫ݱ‬ͷධՁࢦඪ Ͳ͏΍ͬͯঢ়ଶද‫ݱ‬ͷྑ͞ΛධՁ͢Δ͔ʁ • ΤʔδΣϯτʹ࣮ࡍʹ‫ڧ‬ԽֶशλεΫΛղ͔ͤͯɼλεΫؒͰసҠ͙Β͍൚Խ͞Εͨද‫ݱ‬ ʹͳ͍ͬͯΔ͔Λௐ΂Δ • ΋ͬͱ΋Ұൠతͳํ๏͕ͩɼ࣮‫ݧ‬ίετ͕ߴ͍ • Ͳͷ‫ڧ‬ԽֶशΞϧΰϦζϜΛ࢖ͬͯධՁ͢Ε͹͍͍͔Θ͔Βͳ͍ • ͳͷͰɼֶशͨ͠ঢ়ଶද‫͔͏Ͳ͔͍ྑ͕ݱ‬ͷதؒతͳධՁख๏͕ཉ͍͠ • ࠷ۙ๣๏Λ࢖͏ • ࣭తධՁ • ྔతධՁ ,//.4&<-FTPSU > KNN − MSE(s) = 1 k ∑ s̃ − s̃′ 2 s′∈KNN(s,k) 28

29.

ঢ়ଶද‫ݱ‬ͷධՁࢦඪ Ͳ͏΍ͬͯঢ়ଶද‫ݱ‬ͷྑ͞ΛධՁ͢Δ͔ʁ • ΋ͭΕͷͳ͍ද‫ ݱ‬EJTFOUBOHMFE ͔Ͳ͏͔ΛΈΔ • EJTFOUBOHMFENFUSJDTDPSF<)JHHJOT > • σʔλͷഎ‫ޙ‬ͷੜ੒ཁҼ͕෼͔͍ͬͯΔલఏ • ༰ྔ͕খ͘͞7$࣍‫ݩ‬ͷখ͍͞൑ผ‫ث‬ͷBDDVSBDZΛ༻͍Δํ๏ • ਅͷঢ়ଶ΁ͷճ‫ؼ‬ϞσϧΛ࡞Δ<+POTDILPXTLJ > • ςετηοτͷਫ਼౓ΛධՁ͢Δ 29

30.

ঢ়ଶද‫ݱ‬ͷධՁࢦඪ Ͳ͏΍ͬͯঢ়ଶද‫ݱ‬ͷྑ͞ΛධՁ͢Δ͔ʁ 30

31.

ධՁʹ༻͍ΔλεΫ 43-Ͱఆ൪ͷλεΫ • ৼࢠɾ౗ཱৼࢠ • ϥϯμϜͳҐஔ͔Βελʔτ͢ΔৼࢠΛཱͯΔ • $BSU1PMF • ୆ंͷ͍ͭͨ౗ཱৼࢠΛཱͯΔ • ਨ௚ํ޲͔Β›ͣΕΔ͔த৺͔ΒϢχοτͿΜͣΕͯ͠·͏ͱΤϐιʔυ͕ऴྃ͢Δ 31

32.

ධՁʹ༻͍ΔλεΫ 43-Ͱఆ൪ͷλεΫ • ϏσΦήʔϜ • ྫ "UBSJɼ%PPNɼ4VQFS.BSJP#SPT • ෺ཧγϛϡϨʔλ • ྫ 0QFO"*(ZNɼ%FFQ.JOE-BCT • ࣮ϩϘοτ • ྫ ϚχϐϡϨʔγϣϯ<'JOO >ɼϘλϯԡ͠<-FTPSU >ɼ೺࣋<'JOO > 32

33.

S-RL Toolbox 43-ΞϧΰϦζϜͷධՁʹؔ͢Δ͍Ζ͍ΖΛղܾ͢Δπʔϧ<3BGGJO > • IUUQTHJUIVCDPNBSBGGJOSPCPUJDTSMTSM • ଟ༷ͳ‫ػ‬ೳ • छྨͷ‫ڧ‬ԽֶशΞϧΰϦζϜ • 0QFO"*(ZN‫ࣜܗ‬ͷΠϯλʔϑΣΠεΛ࣋ͭධՁ‫ڥ؀‬ • ϩΨʔɾՄࢹԽπʔϧ • ϋΠύʔύϥϝʔλαʔνπʔϧ • ࣮‫ػ‬ͷCBYUFSͰूΊͨσʔληοτ • 43-ͷ࣮૷ू΋43-;PPͱͯ͠‫·ؚ‬Ε͍ͯΔ • IUUQTHJUIVCDPNBSBGGJOTSM[PPUSFF EFBGBBCGBED • 1Z5PSDIͰ͏Ε͍͠ 33

34.

͓ΘΓʹ 34

35.

‫ײ‬૝ • ঢ়ଶද‫Ͳͯؔ͠ʹݱ‬Ε͚ͩෆ࣮֬ੑ͕͋Δͷ͔ΛධՁ͢Δ‫ڀݚ‬͸͋ΔͷͩΖ͏͔ʁ • ྫ͑͹ɼ࠷ॳͷϑϨʔϜ͚ͩ‫ͱ͖ͱͨݟ‬ɼϑϨʔϜ࿈ଓͰ‫Ͱ͖ͱͨݟ‬͸ͦͷঢ়ଶද‫ݱ‬ͷෆ ࣮֬ੑ͸ҟͳΔ͸ͣ • ͦͷෆ࣮֬ੑΛ൓өͨ͠QPMJDZ͕࡞ΕΕ͹ޮ཰తͳ୳ࡧʹ΋ͭͳ͕Δʁ • ͨ͘͞ΜͷλεΫΛղ͔ͤͯ43-ͯ͠ɼྑ͍43-ͷύϥϝʔλΛֶशͨ͠ͷͪɼGFXTIPU Ͱ৽͍͠λεΫʹద߹ͤ͞Δ.".-తͳΞϓϩʔν͕༗ޮ͔΋ • ͦ΋ͦ΋ɼ43-Λ͍ͨ͠‫ͪ࣋ؾ‬͸ɼͨ͘͞ΜͷλεΫͰ‫ڞ‬༗Ͱ͖Δද‫ݱ‬Λֶश͍͔ͨ͠Βͩͬ ͨͷͰ͸ʁ • • ·͋ɼ࣮‫ݧ‬ίετ͕ߴ͍ͷͰɼ࿦จ಺Ͱͨ͘͞ΜͷυϝΠϯΛ࢖ͬͨ‫ڧ‬ԽֶशΛͨ͘͠ͳ͍ͷ͸Θ ͔Δ͚Ͳ΋ʜ  .".-पΓͰ΋ͨ͘͞Μ࿦จ͕ग़͖͍ͯͯΔͷͰɼ࣍ճͷେ࿮͸ϝλϥʔχϯάʹ͠Α͏͔ͳ 35

36.

Appendix 36

37.
[beta] 
References
<"HSBXBM >1VMLJU"HSBXBM "TIWJO/BJS 1JFUFS"CCFFM +JUFOESB.BMJL 4FSHFZ-FWJOF  -FBSOJOHUP
1PLFCZ1PLJOH&YQFSJFOUJBM-FBSOJOHPG*OUVJUJWF1IZTJDTIUUQTBSYJWPSHBCT
<#ÖINFS >#ÖINFS 8 4QSJOHFOCFSH +5 #PFEFDLFS + 3JFENJMMFS . BOE0CFSNBZFS ,  
"VUPOPNPVTMFBSOJOHPGTUBUFSFQSFTFOUBUJPOTGPSDPOUSPM"OFNFSHJOHGJFMEBJNTUPBVUPOPNPVTMZMFBSOTUBUF
SFQSFTFOUBUJPOTGPSSFJOGPSDFNFOUMFBSOJOHBHFOUTGSPNUIFJSSFBMXPSMETFOTPSPCTFSWBUJPOT,*,ÛOTUMJDIF
*OUFMMJHFO[ QBHFTrIUUQXXXOJUVCFSMJOEFGJMFBENJOGHBSUJDMFTCPFINFSCQEG
<$VSSBO >8JMMJBN$VSSBO 5JN#SZT .BUUIFX5BZMPS 8JMMJBN4NBSU  6TJOH1$"UP&GGJDJFOUMZ
3FQSFTFOU4UBUF4QBDFTIUUQTBSYJWPSHBCT
<'JOO >$IFMTFB'JOO 9JO:V5BO :BO%VBO 5SFWPS%BSSFMM 4FSHFZ-FWJOF 1JFUFS"CCFFM  %FFQ
4QBUJBM"VUPFODPEFSTGPS7JTVPNPUPS-FBSOJOHIUUQTBSYJWPSHBCT
<)B >%BWJE)B +ÛSHFO4DINJEIVCFS  8PSME.PEFMTIUUQTBSYJWPSHBCT
<)JHHJOT >*SJOB)JHHJOT -PJD.BUUIFZ "SLB1BM $ISJTUPQIFS#VSHFTT 9BWJFS(MPSPU .BUUIFX
#PUWJOJDL 4IBLJS.PIBNFE "MFYBOEFS-FSDIOFS  CFUB7"&-FBSOJOH#BTJD7JTVBM$PODFQUTXJUIB
$POTUSBJOFE7BSJBUJPOBM'SBNFXPSLIUUQTPQFOSFWJFXOFUGPSVN JE4ZG[6HM

37
38.
[beta] 
References
<+POTDILPXTLJ >+POTDILPXTLJ 3BOE#SPDL 0  -FBSOJOHTUBUFSFQSFTFOUBUJPOTXJUISPCPUJD
QSJPST"VUPO3PCPUT   rIUUQXXXSPCPUJDTUVCFSMJOEFGJMFBENJOGH1VCMJLBUJPOFO@QEG
+POTDILPXTLJ"630QEG
<+POTDILPXTLJ >3JDP+POTDILPXTLJ 3PMBOE)BGOFS +POBUIBO4DIPM[ .BSUJO3JFENJMMFS  17&T
1PTJUJPO7FMPDJUZ&ODPEFSTGPS6OTVQFSWJTFE-FBSOJOHPG4USVDUVSFE4UBUF3FQSFTFOUBUJPOTIUUQTBSYJWPSHBCT

<,BSM >.BYJNJMJBO,BSM .BYJNJMJBO4PFMDI +VTUJO#BZFS 1BUSJDLWBOEFS4NBHU%FFQ7BSJBUJPOBM#BZFT
'JMUFST6OTVQFSWJTFE-FBSOJOHPG4UBUF4QBDF.PEFMTGSPN3BX%BUBIUUQTBSYJWPSHBCT
<,VSVUBDI >5IBOBSE,VSVUBDI "WJW5BNBS (F:BOH 4UVBSU3VTTFMM 1JFUFS"CCFFM  -FBSOJOH
1MBOOBCMF3FQSFTFOUBUJPOTXJUI$BVTBM*OGP("/IUUQTBSYJWPSHBCT
<-BLF ^#VJMEJOH.BDIJOFT5IBU-FBSOBOE5IJOL-JLF1FPQMF  #SFOEFO.-BLF 5PNFS%
6MMNBO +PTIVB#5FOFOCBVN 4BNVFM+(FSTINBOIUUQTBSYJWPSHBCT
<0I >+VOIZVL0I 4BUJOEFS4JOHI )POHMBL-FF  7BMVF1SFEJDUJPO/FUXPSLIUUQTBSYJWPSHBCT

<1BUIBL >%FFQBL1BUIBL 1VMLJU"HSBXBM "MFYFJ"&GSPT 5SFWPS%BSSFMM  $VSJPTJUZESJWFO
&YQMPSBUJPOCZ4FMGTVQFSWJTFE1SFEJDUJPOIUUQTBSYJWPSHBCT

38
39.
[beta] 
References
<3BGGJO >"OUPOJO3BGGJO "TIMFZ)JMM 3FOÉ5SBPSÉ 5JNPUIÉF-FTPSU /BUBMJB%ÍB[3PESÍHVF[ %BWJE'JMMJBU
 43-5PPMCPY&OWJSPONFOUT %BUBTFUTBOE&WBMVBUJPO.FUSJDTGPS4UBUF3FQSFTFOUBUJPO-FBSOJOHIUUQT
BSYJWPSHBCT
<-FTPSU >5JNPUIÉF-FTPSU .BUIJFV4FVSJO 9JOSVJ-J /BUBMJB%ÍB[3PESÍHVF[ %BWJE'JMMJBU  
6OTVQFSWJTFETUBUFSFQSFTFOUBUJPOMFBSOJOHXJUISPCPUJDQSJPSTBSPCVTUOFTTCFODINBSLIUUQTBSYJWPSHBCT

<.BUUOFS >.BUUOFS + -BOHF 4 BOE3JFENJMMFS ."  -FBSOUPTXJOHVQBOECBMBODFBSFBMQPMF
CBTFEPOSBXWJTVBMJOQVUEBUB*O/FVSBM*OGPSNBUJPO1SPDFTTJOHUI*OUFSOBUJPOBM$POGFSFODF *$0/*1 
%PIB 2BUBS /PWFNCFS  1SPDFFEJOHT 1BSU7 QBHFTrIUUQTJFFFYQMPSFJFFFPSHEPDVNFOU

<WBO)PPG >WBO)PPG ) $IFO / ,BSM . WBOEFS4NBHU 1 BOE1FUFST +  4UBCMFSFJOGPSDFNFOU
MFBSOJOHXJUIBVUPFODPEFSTGPSUBDUJMFBOEWJTVBMEBUB*O*&&&34+*OUFSOBUJPOBM$POGFSFODFPO*OUFMMJHFOU
3PCPUTBOE4ZTUFNT *304 QBHFTrIUUQTJFFFYQMPSFJFFFPSHEPDVNFOU
<8BUUFS >.BOVFM8BUUFS +PTU5PCJBT4QSJOHFOCFSH +PTDILB#PFEFDLFS .BSUJO3JFENJMMFS  &NCFE
UP$POUSPM"-PDBMMZ-JOFBS-BUFOU%ZOBNJDT.PEFMGPS$POUSPMGSPN3BX*NBHFTIUUQTBSYJWPSHBCT


39