【DL輪読会】Segment Anything

Segment Anything Shohei Taniguchi, Matsuo Lab

Segment Anything ॻࢽ৘ใ ஶऀ Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick ֓ཁ • Meta͕ެ։ͨ͠ηάϝϯςʔγϣϯͷͨΊͷ‫ج‬൫ϞσϧSAM • 1100ສຕͷը૾ʹ10ԯҎ্ͷϚεΫ͕Ξϊςʔγϣϯ͞Εͨσʔληοτ SA-1B΋ެ։ 2

֓ཁ Segment-Anything Model, SAM • ༷ʑͳϓϩϯϓτ͔Β෺ମͷϚεΫΛੜ੒Ͱ͖ΔϞσϧ ࢦࣔ఺ɾςΩετɾྖҬͳͲ

֓ཁ Segment-Anything Model, SAM • Τοδ༧ଌ΍text-to-mask΋zero-shotͰ݁ߏͰ͖Δ

ൃද֓ཁ • λεΫɿPromotable segmentation • ϞσϧɿSegment Anything Model • σʔλɿData engine • ࣮‫ݧ‬ • ·ͱΊ 5

എ‫ܠ‬ • ۙ೥ɼେ‫ن‬໛‫ޠݴ‬Ϟσϧͷൃల͕͍͢͝ ‣ PromptΛ༩͑ͨΒࣗࡏʹ‫ޠݴ‬Λੜ੒Ͱ͖Δ ‣ Scaling lawͰͲΜͲΜੑೳ্͕͕Δ ➡ίϯϐϡʔλϏδϣϯͰ΋ಉ͡Α͏ͳ͜ͱ͸ Ͱ͖ͳ͍ͷ͔ʁ https://j.gifs.com/Y7mBPW.gif 6

• https://j.gifs.com/Y7mBPW.gif

λεΫ Promptable Segmentation • ैདྷͷηάϝϯςʔγϣϯλεΫͱҧ͍ ηάϝϯτର৅ΛϓϩϯϓτͰࢦఆ͢Δ ‣ ࢦࣔ఺ɼྖҬɼςΩετͳͲ • ϓϩϯϓτ͸ᐆດੑΛ‫ؚ‬ΉͨΊ ਖ਼͍͠ϚεΫ͸1ͭͱ͸‫ݶ‬Βͳ͍ 7

Ϟσϧ Segment Anything Model, SAM • ߏ੒͸݁ߏγϯϓϧ 1. ը૾ͱϓϩϯϓτΛ ͦΕͧΕຒΊࠐΉ 2. TransformerϕʔεͷσίʔμͰ ຒΊࠐΈ͔ΒϚεΫΛੜ੒͢Δ 8

Ϟσϧ Segment Anything Model, SAM • Image encoder ‣ ը૾Λಛ௃ྔʹຒΊࠐΉ ‣ த਎͸ViT ‣ 1൪‫͕ࢉܭ‬ॏ͍෦෼͕ͩɼ ਪ࿦࣌ʹ͸ಛ௃ྔΛอ͓͚࣋ͯ͠͹ ϓϩϯϓτΛϦΞϧλΠϜͰ͍͡ΕΔ 9

Ϟσϧ Segment Anything Model, SAM • Prompt encoder (points, box) ‣ ϓϩϯϓτΛຒΊࠐΉ ‣ positional encodingʹͯ͠ ֶशՄೳͳຒΊࠐΈύϥϝʔλͱ ଍͠߹ΘͤΔ 10

Ϟσϧ Segment Anything Model, SAM • Prompt encoder (text) ‣ ϓϩϯϓτΛຒΊࠐΉ ‣ CLIPͷtext encoderΛ࢖͏ 11

Ϟσϧ Segment Anything Model, SAM • Prompt encoder (mask) ‣ ϓϩϯϓτΛຒΊࠐΉ ‣ ৞ΈࠐΈΛ͔͚ͨ΋ͷΛ ը૾ຒΊࠐΈͱ଍͠߹ΘͤΔ 12

Ϟσϧ Segment Anything Model, SAM • Mask decoder ‣ ϚεΫީิΛग़ྗ͢Δ ‣ த਎͸Transformerͷdecoder ‣ ϓϩϯϓτͷᐆດੑʹରॲ͢ΔͨΊʹ 3ͭͷީิΛग़ྗ͢Δ 13

Ϟσϧ Segment Anything Model, SAM • ֶश ‣ Focal lossͱdice lossΛ ૊Έ߹Θֶͤͯश ‣ ϓϩϯϓτ͸ϥϯμϜʹ αϯϓϧ͢Δ 14

σʔλ Data Engine • SAMΛΞϊςʔγϣϯʹ΋‫͢༻׆‬Δ ‣ Model-in-the-loop • 3ஈ֊ʹ෼͚ͯΞϊςʔγϣϯ͢Δ 15

σʔλ Data Engine 1. SAM͕༧ଌͨ͠ϚεΫΛमਖ਼͢Δ • SAM͸ॳΊʹผͷσʔληοτͰ ࣄલʹֶश͓ͤͯ͘͞ • σʔλ͕͋Δఔ౓ू·ͬͨΒ ͦΕΛ࢖ͬͯSAMΛֶशͤ͞Δ • 1ը૾͋ͨΓ30ඵҎ಺ʹ෇༩Ͱ͖ΔൣғͰ Ξϊςʔγϣϯ 16

σʔλ Data Engine 2. SAM͕༧ଌͨ͠΋ͷҎ֎ΛΞϊςʔγϣϯ • ΑΓࡉ͔͍෦෼ΛΞϊςʔγϣϯ • ͜ͷࡍʹ΋৽͘͠௥Ճͨ͠σʔλͰ SAMΛֶशͤ͞Δ • ͜͜·ͰͰ1020ສ‫ݸ‬ͷϚεΫ͕ಘΒΕΔ 17

σʔλ Data Engine 3. SAMͷ༧ଌͰΞϊςʔγϣϯ • 2ஈ֊໨ͰSAM͕͔ͳΓ͍͍ਫ਼౓ʹ ͳ͍ͬͯΔͨΊɼ༧ଌ݁ՌΛ΄ͱΜͲ ͦͷ··Ξϊςʔγϣϯͱͯ͠࢖͑Δ • Ϟσϧͷ֬৴౓͕ߴ͍΋ͷΛબΜͰ NMSͰॏෳΛআ‫͢ڈ‬Δ 18

σʔλ SA-1B • ࠷ऴతʹ1100ສຕͷը૾ʹ11ԯ‫ݸ‬ͷϚεΫ͕ ͍ͭͨσʔληοτ͕Ͱ͖Δ • ‫ط‬ଘͷσʔληοτʹൺ΂ͯɼ1ը૾͋ͨΓͷ ϚεΫͷ਺͕͍ͩͿଟ͍ 19

σʔλ SA-1B • ϚεΫͷҐஔͷόΠΞε΋গͳ͍ • ‫ط‬ଘͷ΋ͷ͸த৺෇ۙʹ͔ͳΓภ͍ͬͯΔ 20

࣮‫ݧ‬ ࢦࣔ఺͔ΒͷϚεΫ༧ଌ • ଟ͘ͷϕϯνϚʔΫͰZero-shotͰ‫ط‬ଘͷϞσϧΛ্ճΔੑೳ͕ग़Δ • Zero-shotɿ֤σʔληοτͰfinetune͍ͯ͠ͳ͍ 21

࣮‫ݧ‬ ͦͷଞͷzero-shotੑೳ Text-to-mask Τοδ༧ଌ 22

࣮‫ݧ‬ Ablation study • σʔλྔ΍ϞσϧαΠζʹΑͬͯੑೳ͕Ͳͷ͘Β͍มΘΔ͔ͷ෼ੳ • σʔλྔʹؔͯ͠͸100ສຕ͘Β͍Ͱ݁ߏανͬͯͦ͏ͳҹ৅

·ͱΊ • ϓϩϯϓτͰ੍‫ޚ‬Մೳͳηάϝϯςʔγϣϯ༻‫ج‬൫ϞσϧSAMΛఏҊ • SAMΛ࢖ͬͯmodel-in-the-loopͰσʔλΛऩूͨ͠SA-1Bσʔληοτ΋ެ։ • σϞ΋ެ։͞Ε͍ͯΔ https://segment-anything.com/demo • ϓϩϯϓτ͸ը૾‫Ͱܥ‬΋൚༻తʹ࢖͑ΔΞϓϩʔνʹͳΓͦ͏