visionOSで空間演出を実現する方法

iOSDC 2024 visionOSで空間演出を実現する方法 服部 智 @shmdevelop

https://www.apple.com/jp/newsroom/2024/06/apple-vision-pro-arrives-in-new-countries-and-regions-beginning-june-28/

• https://www.apple.com/jp/newsroom/2024/06/apple-vision-pro-arrives-in-new-countries-and-regions-beginning-june-28/

https://www.apple.com/jp/newsroom/2024/06/apple-vision-pro-arrives-in-new-countries-and-regions-beginning-june-28/

• https://www.apple.com/jp/newsroom/2024/06/apple-vision-pro-arrives-in-new-countries-and-regions-beginning-june-28/

空間的な演出が評判良いvisionOSのアプリ

3D CG キャラクター ポータル表示 + 前面の3Dオブジェクト 手の動きを使ったインタラクション "Kung Fu Panda: School of Chi" "What If…? An Immersive Story" 3D CG キャラクター ポータル表示、VR表示 現実空間が見える ハンドトラッキングなど

高品質な2D動画 空間的エフェクト インタラクティブ要素 ※ アップデートで体験要素追加あり "GUCCI" "Disney+" 2D動画視聴 3D動画もある 高品質のVR視聴環境

今回のセッションでは

GUCCI のような表現と機能をどう作るか

Satoshi Hattori xR Engineer Cyber AI Productions Cyber Agent: Next AR Experts Host of "visionOS Engineer Meetup" GitHub: satoshi0212 X: @shmdevelop

visionOS 30 Days Challenge

Image Board 5x5 3 lines static pictures tap to dynamic motion

Street View Images from Google StreetView API Combine images to a panoramic image covering 360°

Metal Shader Quote from ShaderToy Use CAMetalLayer, CADisplayLink visionOS doesn't support MTKView visionOS 2 has LowLevelTexture

https://www.amazon.co.jp/dp/4297143119/

• https://www.amazon.co.jp/dp/4297143119/

GUCCI のような表現と機能をどう作るか

image from "GUCCI" application

image from "GUCCI" application

空間的表現 2D動画 コントローラー

image from "GUCCI" application

image from "GUCCI" application

image from "GUCCI" application

image from "GUCCI" application

image from "GUCCI" application

image from "GUCCI" application

image from "GUCCI" application

image from "GUCCI" application

今回作るもの

線状パーティクル 雨パーティクル + 黒背景 + 花火 Environment(アプリ内)

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

今回の方式 HLSの指定時間に演出きっかけのtag埋め込み

他にも アプリ側で動画秒数で演出管理 別途切り出した設定ファイルで演出管理 等の手法がある

元となる動画を作成

ID3 TagとMacroファイルを作成

https://developer.apple.com/streaming/

• https://developer.apple.com/streaming/

https://developer.apple.com/documentation/http-live-streaming/using-apple-s-http-live-streaming-hls-tools

• https://developer.apple.com/documentation/http-live-streaming/using-apple-s-http-live-streaming-hls-tools

ID3 Tag Generator

$ id3taggenerator -o reset.id3 -t "c̲reset" -o | -output-file <file> Specifies the path where the generated ID3 tag is written. -t | -text <string> Inserts a text frame with the given string.

fi fi ff ff ff ff ff ff ff ff fi fi $ id3taggenerator -o reset.id3 -t "c̲reset" $ id3taggenerator -o line̲on.id3 -t "c̲on̲line̲particle" $ id3taggenerator -o line̲o .id3 -t "c̲o ̲line̲particle" $ id3taggenerator -o rain̲on.id3 -t "c̲on̲rain̲particle" $ id3taggenerator -o rain̲o .id3 -t "c̲o ̲rain̲particle" $ id3taggenerator -o reworks̲on.id3 -t "c̲on̲ reworks̲particle" $ id3taggenerator -o reworks̲o .id3 -t "c̲o ̲ reworks̲particle" $ id3taggenerator -o env̲01̲on.id3 -t "c̲on̲env̲01" $ id3taggenerator -o env̲01̲o .id3 -t "c̲o ̲env̲01"

Macro.txtを作成

Macro.txt ff ff ff ff fi fi 0 id3 ./reset.id3 2 id3 ./line̲on.id3 10 id3 ./line̲o .id3 11.5 id3 ./env̲01̲on.id3 20.5 id3 ./env̲01̲o .id3 21 id3 ./rain̲on.id3 30 id3 ./rain̲o .id3 32 id3 ./ reworks̲on.id3 40 id3 ./ reworks̲o .id3 44 id3 ./reset.id3

Media File SegmenterでHLSリソース生成

Media File Segmenter

$ media lesegmenter -f ./output/ -i index.m3u8 -B media- -t 1 \ -M ./macro.txt ./SpatialE ects001.mov -f | - le-base path Directory to store the media and index -i | -index- le les. leName This option de nes the index le name. The default is prog̲index.m3u8. It is recommended that the index le have an extension of .m3u8 or .m3u. -B | -base-media- le-name name This option de nes the base name of the media les. The default is leSequence. The current sequence number of the is appended, and an extension added. For example, specifying name as AppleMediaFile will generate le names that look like AppleMediaFile12.ts. -t | -target-duration duration Speci es a target duration for the media at the PTS/DTS in the source le. fi fi fi ff fi fi fi fi le to be used to insert timed metadata into the stream. fi fi fi fi fi fi fi fi fi fi fi fi Speci es the macro fi les. The default duration is 10 seconds. The duration is calculated by looking le fi -M | -meta-macro- le le

GitHub Pagesでホストする (開発用)

VLC (動画Player)

👍

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

Reality Composer Pro

線状のParticle

線状のParticle

線状のParticle

雨と黒背景

花火

Environment

https://developer.apple.com/documentation/realitykit/construct-an-immersive-environment-for-visionos

• https://developer.apple.com/documentation/realitykit/construct-an-immersive-environment-for-visionos

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

SpatialEffectsVideoPlayerApp.swift @main struct SpatialEffectsVideoPlayerApp: App { @State private var appModel = AppModel() @State private var playerViewModel = AVPlayerViewModel() @State private var surroundingsEffect: SurroundingsEffect? = .semiDark var body: some Scene { WindowGroup { if playerViewModel.isPlaying { AVPlayerView(viewModel: playerViewModel) } else { ContentView() .environment(appModel) } } .windowResizability(.contentSize) .windowStyle(.plain) ImmersiveSpace(id: appModel.immersiveSpaceID) { ImmersiveView() .environment(appModel) .environment(playerViewModel) .onAppear { appModel.immersiveSpaceState = .open } .onDisappear { appModel.immersiveSpaceState = .closed } .preferredSurroundingsEffect(surroundingsEffect) } .immersionStyle(selection: .constant(.mixed), in: .mixed) } }

SpatialEffectsVideoPlayerApp.swift @main struct SpatialEffectsVideoPlayerApp: App { @State private var appModel = AppModel() @State private var playerViewModel = AVPlayerViewModel() @State private var surroundingsEffect: SurroundingsEffect? = .semiDark var body: some Scene { WindowGroup { if playerViewModel.isPlaying { AVPlayerView(viewModel: playerViewModel) } else { ContentView() .environment(appModel) } } .windowResizability(.contentSize) .windowStyle(.plain) ImmersiveSpace(id: appModel.immersiveSpaceID) { ImmersiveView() .environment(appModel) .environment(playerViewModel) .onAppear { appModel.immersiveSpaceState = .open } .onDisappear { appModel.immersiveSpaceState = .closed } .preferredSurroundingsEffect(surroundingsEffect) } .immersionStyle(selection: .constant(.mixed), in: .mixed) } }

AVPlayerView.swift
import SwiftUI
struct AVPlayerView: UIViewControllerRepresentable {
let viewModel: AVPlayerViewModel
func makeUIViewController(context: Context) -> some UIViewController {
return viewModel.makePlayerViewController()
}
func updateUIViewController(_ uiViewController: UIViewControllerType, context: Context) {
// Update the AVPlayerViewController as needed
}
}

AVPlayerViewModel.swift
@Observable
final class AVPlayerViewModel: NSObject {
private(set) var isPlaying: Bool = false
private var avPlayerViewController: AVPlayerViewController?
private var avPlayer = AVPlayer()
private let videoURL: URL? = {
URL(string: "https://satoshi0212.github.io/hls/resources/index.m3u8")
}()
func makePlayerViewController() -> AVPlayerViewController {
let controller = AVPlayerViewController()
controller.player = avPlayer
controller.delegate = self
self.avPlayerViewController = controller
self.avPlayerViewController?.delegate = self
controller.modalPresentationStyle = .fullScreen
return controller
}
func play() {
guard !isPlaying, let videoURL else { return }
isPlaying = true
let item = AVPlayerItem(url: videoURL)
let metadataOutput = AVPlayerItemMetadataOutput(identifiers: nil)
metadataOutput.setDelegate(self, queue: DispatchQueue.main)
item.add(metadataOutput)
avPlayer.replaceCurrentItem(with: item)
avPlayer.play()
}
func reset() {
guard isPlaying else { return }
isPlaying = false
avPlayer.replaceCurrentItem(with: nil)
}
}

AVPlayerViewModel.swift
@Observable
final class AVPlayerViewModel: NSObject {
private(set) var isPlaying: Bool = false
private var avPlayerViewController: AVPlayerViewController?
private var avPlayer = AVPlayer()
private let videoURL: URL? = {
URL(string: "https://satoshi0212.github.io/hls/resources/index.m3u8")
}()
func makePlayerViewController() -> AVPlayerViewController {
let controller = AVPlayerViewController()
controller.player = avPlayer
controller.delegate = self
self.avPlayerViewController = controller
self.avPlayerViewController?.delegate = self
controller.modalPresentationStyle = .fullScreen
return controller
}
func play() {
guard !isPlaying, let videoURL else { return }
isPlaying = true
let item = AVPlayerItem(url: videoURL)
let metadataOutput = AVPlayerItemMetadataOutput(identifiers: nil)
metadataOutput.setDelegate(self, queue: DispatchQueue.main)
item.add(metadataOutput)
avPlayer.replaceCurrentItem(with: item)
avPlayer.play()
}
func reset() {
guard isPlaying else { return }
isPlaying = false
avPlayer.replaceCurrentItem(with: nil)
}
}

AVPlayerViewModel.swift
@Observable
final class AVPlayerViewModel: NSObject {
private(set) var isPlaying: Bool = false
private var avPlayerViewController: AVPlayerViewController?
private var avPlayer = AVPlayer()
private let videoURL: URL? = {
URL(string: "https://satoshi0212.github.io/hls/resources/index.m3u8")
}()
func makePlayerViewController() -> AVPlayerViewController {
let controller = AVPlayerViewController()
controller.player = avPlayer
controller.delegate = self
self.avPlayerViewController = controller
self.avPlayerViewController?.delegate = self
controller.modalPresentationStyle = .fullScreen
return controller
}
func play() {
guard !isPlaying, let videoURL else { return }
isPlaying = true
let item = AVPlayerItem(url: videoURL)
let metadataOutput = AVPlayerItemMetadataOutput(identifiers: nil)
metadataOutput.setDelegate(self, queue: DispatchQueue.main)
item.add(metadataOutput)
avPlayer.replaceCurrentItem(with: item)
avPlayer.play()
}
func reset() {
guard isPlaying else { return }
isPlaying = false
avPlayer.replaceCurrentItem(with: nil)
}
}

ImmersiveView.swift (抜粋)
struct ImmersiveView: View {
@Environment(AVPlayerViewModel.self) private var playerViewModel
@State var immersiveViewModel = ImmersiveViewModel()
var body: some View {
ZStack {
RealityView { content in
let entity = Entity()
content.add(entity)
immersiveViewModel.setup(entity: entity)
}
.gesture(SpatialTapGesture().targetedToAnyEntity()
.onEnded { value in
if value.entity.name == "StartButton" {
playerViewModel.play()
}
}
)
.onChange(of: playerViewModel.isPlaying, initial: false) { _, newValue in
immersiveViewModel.rootEntity?.getFirstChildByName(name: "StartButton")?.isEnabled = !newValue
}
.onDisappear {
playerViewModel.reset()
}
.transition(.opacity)
...
}
}
}

ImmersiveView.swift (抜粋)
struct ImmersiveView: View {
@Environment(AVPlayerViewModel.self) private var playerViewModel
@State var immersiveViewModel = ImmersiveViewModel()
var body: some View {
ZStack {
RealityView { content in
let entity = Entity()
content.add(entity)
immersiveViewModel.setup(entity: entity)
}
.gesture(SpatialTapGesture().targetedToAnyEntity()
.onEnded { value in
if value.entity.name == "StartButton" {
playerViewModel.play()
}
}
)
.onChange(of: playerViewModel.isPlaying, initial: false) { _, newValue in
immersiveViewModel.rootEntity?.getFirstChildByName(name: "StartButton")?.isEnabled = !newValue
}
.onDisappear {
playerViewModel.reset()
}
.transition(.opacity)
...
}
}
}

Metadata受信を確認

AVPlayerViewModel.swift extension AVPlayerViewModel: AVPlayerItemMetadataOutputPushDelegate { func metadataOutput(_ output: AVPlayerItemMetadataOutput, didOutputTimedMetadataGroups groups: [AVTimedMetadataGroup], from track: AVPlayerItemTrack?) { if let item = groups.first?.items.first, let metadataValue = item.value(forKey: "value") as? String { print("Metadata value: \(metadataValue)") // videoAction = VideoAction(rawValue: metadataValue) ?? .none } } }

🎉

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

演出系のViewは共通した構造にした

LineParticleView.swift import SwiftUI import RealityKit struct LineParticleView: View { static let viewName = "LineParticleView" @State var viewModel = LineParticleViewModel() var body: some View { RealityView { content in let entity = Entity() content.add(entity) viewModel.setup(entity: entity) } } }

RainParticleView.swift import SwiftUI import RealityKit struct RainParticleView: View { static let viewName = "RainParticleView" @State var viewModel = RainParticleViewModel() var body: some View { RealityView { content in let entity = Entity() content.add(entity) viewModel.setup(entity: entity) } } }

FireworksParticleView.swift import SwiftUI import RealityKit struct FireworksParticleView: View { static let viewName = "FireworksParticleView" @State var viewModel = FireworksParticleViewModel() var body: some View { RealityView { content in let entity = Entity() content.add(entity) viewModel.setup(entity: entity) } } }

Env01View.swift import SwiftUI import RealityKit struct Env01View: View { static let viewName = "Env01View" @State var viewModel = Env01ViewModel() var body: some View { RealityView { content in let entity = Entity() content.add(entity) viewModel.setup(entity: entity) } } }

ViewModelも基本的に共通した構造

LineParticleViewModel.swift
import RealityKit
import Observation
import RealityKitContent
@MainActor
@Observable
final class LineParticleViewModel: LiveSequenceOperation {
private var rootEntity: Entity?
func setup(entity: Entity) {
rootEntity = entity
rootEntity?.opacity = 0.0
Task {
guard let scene = try? await Entity(named: "LineParticle", in: realityKitContentBundle),
let particleEntity = scene.findEntity(named: "ParticleEmitter")
else { return }
particleEntity.name = "lineParticle"
particleEntity.position = [0.0, 1.2, -0.8]
rootEntity?.addChild(particleEntity)
}
}
...

LineParticleViewModel.swift
import RealityKit
import Observation
import RealityKitContent
@MainActor
@Observable
final class LineParticleViewModel: LiveSequenceOperation {
private var rootEntity: Entity?
func setup(entity: Entity) {
rootEntity = entity
rootEntity?.opacity = 0.0
Task {
guard let scene = try? await Entity(named: "LineParticle", in: realityKitContentBundle),
let particleEntity = scene.findEntity(named: "ParticleEmitter")
else { return }
particleEntity.name = "lineParticle"
particleEntity.position = [0.0, 1.2, -0.8]
rootEntity?.addChild(particleEntity)
}
}
...

LineParticleViewModel.swift
import RealityKit
import Observation
import RealityKitContent
@MainActor
@Observable
final class LineParticleViewModel: LiveSequenceOperation {
private var rootEntity: Entity?
func setup(entity: Entity) {
rootEntity = entity
rootEntity?.opacity = 0.0
Task {
guard let scene = try? await Entity(named: "LineParticle", in: realityKitContentBundle),
let particleEntity = scene.findEntity(named: "ParticleEmitter")
else { return }
particleEntity.name = "lineParticle"
particleEntity.position = [0.0, 1.2, -0.8]
rootEntity?.addChild(particleEntity)
}
}
...

LineParticleViewModel.swift
import RealityKit
import Observation
import RealityKitContent
@MainActor
@Observable
final class LineParticleViewModel: LiveSequenceOperation {
private var rootEntity: Entity?
func setup(entity: Entity) {
rootEntity = entity
rootEntity?.opacity = 0.0
Task {
guard let scene = try? await Entity(named: "LineParticle", in: realityKitContentBundle),
let particleEntity = scene.findEntity(named: "ParticleEmitter")
else { return }
particleEntity.name = "lineParticle"
particleEntity.position = [0.0, 1.2, -0.8]
rootEntity?.addChild(particleEntity)
}
}
...

protocol LiveSequenceOperation { func reset() async func play() async func fadeIn() async func fadeOut() async }

LineParticleViewModel.swift
import RealityKit
import Observation
import RealityKitContent
@MainActor
@Observable
final class LineParticleViewModel: LiveSequenceOperation {
private var rootEntity: Entity?
func setup(entity: Entity) {
rootEntity = entity
rootEntity?.opacity = 0.0
Task {
guard let scene = try? await Entity(named: "LineParticle", in: realityKitContentBundle),
let particleEntity = scene.findEntity(named: "ParticleEmitter")
else { return }
particleEntity.name = "lineParticle"
particleEntity.position = [0.0, 1.2, -0.8]
rootEntity?.addChild(particleEntity)
}
}
...

LineParticleViewModel.swift ... func reset() { rootEntity?.opacity = 0.0 } func play() { rootEntity?.getFirstChildByName(name: "lineParticle")?.isEnabled = true } func fadeIn() { Task { await rootEntity?.setOpacity(1.0, animated: true, duration: 0.4) } } func fadeOut() { Task { await rootEntity?.setOpacity(0.0, animated: true, duration: 0.4) } } }

ここで フェードイン、フェードアウトについて

LineParticleViewModel.swift ... func reset() { rootEntity?.opacity = 0.0 } func play() { rootEntity?.getFirstChildByName(name: "lineParticle")?.isEnabled = true } func fadeIn() { Task { await rootEntity?.setOpacity(1.0, animated: true, duration: 0.4) } } func fadeOut() { Task { await rootEntity?.setOpacity(0.0, animated: true, duration: 0.4) } } }

Entity+.swift
@MainActor
func setOpacity(_ opacity: Float, animated: Bool, duration: TimeInterval = 0.2,
delay: TimeInterval = 0, completion: (() -> Void) = {}) async {
guard animated, let scene else {
self.opacity = opacity
return
}
if !components.has(OpacityComponent.self) {
components[OpacityComponent.self] = OpacityComponent(opacity: 1.0)
}
let animation = FromToByAnimation(name: "Entity/setOpacity", to: opacity, duration: duration,
timing: .linear, isAdditive: false, bindTarget: .opacity, delay: delay)
do {
let animationResource: AnimationResource = try .generate(with: animation)
let animationPlaybackController = playAnimation(animationResource)
let filtered = scene.publisher(for: AnimationEvents.PlaybackTerminated.self)
.filter { $0.playbackController == animationPlaybackController }
_ = filtered.values.filter { await $0.playbackController.isComplete }
completion()
} catch {
print("Could not generate animation: \(error.localizedDescription)")
}
}

Entity+.swift
@MainActor
func setOpacity(_ opacity: Float, animated: Bool, duration: TimeInterval = 0.2,
delay: TimeInterval = 0, completion: (() -> Void) = {}) async {
guard animated, let scene else {
self.opacity = opacity
return
}
if !components.has(OpacityComponent.self) {
components[OpacityComponent.self] = OpacityComponent(opacity: 1.0)
}
let animation = FromToByAnimation(name: "Entity/setOpacity", to: opacity, duration: duration,
timing: .linear, isAdditive: false, bindTarget: .opacity, delay: delay)
do {
let animationResource: AnimationResource = try .generate(with: animation)
let animationPlaybackController = playAnimation(animationResource)
let filtered = scene.publisher(for: AnimationEvents.PlaybackTerminated.self)
.filter { $0.playbackController == animationPlaybackController }
_ = filtered.values.filter { await $0.playbackController.isComplete }
completion()
} catch {
print("Could not generate animation: \(error.localizedDescription)")
}
}

他のViewModel

RainParticleViewModel.swift
@MainActor
@Observable
final class RainParticleViewModel: LiveSequenceOperation {
private var rootEntity: Entity?
func setup(entity: Entity) {
rootEntity = entity
rootEntity.opacity = 0.0
let skyBoxEntity = Entity()
skyBoxEntity.components.set(ModelComponent(
mesh: .generateSphere(radius: 1000),
materials: [UnlitMaterial(color: .black)]
))
skyBoxEntity.scale *= .init(x: -1, y: 1, z: 1)
rootEntity.addChild(skyBoxEntity)
Task {
if let scene = try? await Entity(named: "RainParticle", in: realityKitContentBundle) {
let particleEntity = scene.findEntity(named: "ParticleEmitter")!
particleEntity.name = "rainParticle"
particleEntity.position = [0.0, 3.0, -2.0]
rootEntity.addChild(particleEntity)
}
}
}
...

LineParticleViewModel.swift ... func reset() { rootEntity?.opacity = 0.0 } func play() { rootEntity?.getFirstChildByName(name: "rainParticle")?.isEnabled = true } func fadeIn() { Task { await rootEntity?.setOpacity(1.0, animated: true, duration: 1.4) } } func fadeOut() { Task { await rootEntity?.setOpacity(0.0, animated: true, duration: 1.4) } } }

FireworksParticleViewModel.swift @MainActor @Observable final class FireworksParticleViewModel: LiveSequenceOperation { private var rootEntity: Entity? func setup(entity: Entity) { rootEntity = entity rootEntity?.opacity = 0.0 Task { guard let scene = try? await Entity(named: "Fireworks", in: realityKitContentBundle) else { return } rootEntity?.addChild(scene) } } ... }

Env01ViewModel.swift @MainActor @Observable final class Env01ViewModel: LiveSequenceOperation { private var rootEntity: Entity? func setup(entity: Entity) { rootEntity = entity rootEntity?.opacity = 0.0 Task { guard let scene = try? await Entity(named: "Env_01", in: realityKitContentBundle) else { return } rootEntity?.addChild(scene) } } ... }

Viewを生成し配置

ImmersiveViewModel.swift (抜粋) @MainActor @Observable class ImmersiveViewModel { private(set) var rootEntity: Entity? let lineParticleView: LineParticleView = .init() let rainParticleView: RainParticleView = .init() let fireworksParticleView: FireworksParticleView = .init() let env01View: Env01View = .init() @ObservationIgnored private lazy var effectViewModels: [String : LiveSequenceOperation] = { return [ LineParticleView.viewName : self.lineParticleView.viewModel, RainParticleView.viewName : self.rainParticleView.viewModel, FireworksParticleView.viewName : self.fireworksParticleView.viewModel, Env01View.viewName : self.env01View.viewModel, ] }() ...

ImmersiveViewModel.swift (抜粋)
struct ImmersiveView: View {
@State var immersiveViewModel = ImmersiveViewModel()
var body: some View {
ZStack {
RealityView { content in
let entity = Entity()
content.add(entity)
immersiveViewModel.setup(entity: entity)
}
.gesture(SpatialTapGesture().targetedToAnyEntity()
.onEnded { value in
if value.entity.name == "StartButton" {
playerViewModel.play()
}
}
)
.onChange(of: playerViewModel.videoAction, initial: true) { oldValue, newValue in
immersiveViewModel.processVideoAction(oldValue: oldValue, newValue: newValue)
}
.onChange(of: playerViewModel.isPlaying, initial: false) { _, newValue in
immersiveViewModel.rootEntity?.getFirstChildByName(name: "StartButton")?.isEnabled = !newValue
}
.onDisappear {
playerViewModel.reset()
}
.transition(.opacity)

}

}

}

// place effect views
immersiveViewModel.lineParticleView
immersiveViewModel.rainParticleView
immersiveViewModel.fireworksParticleView
immersiveViewModel.env01View

ImmersiveView.swift (抜粋)
struct ImmersiveView: View {
@State var immersiveViewModel = ImmersiveViewModel()
var body: some View {
ZStack {
RealityView { content in
let entity = Entity()
content.add(entity)
immersiveViewModel.setup(entity: entity)
}
.gesture(SpatialTapGesture().targetedToAnyEntity()
.onEnded { value in
if value.entity.name == "StartButton" {
playerViewModel.play()
}
}
)
.onChange(of: playerViewModel.videoAction, initial: true) { oldValue, newValue in
immersiveViewModel.processVideoAction(oldValue: oldValue, newValue: newValue)
}
.onChange(of: playerViewModel.isPlaying, initial: false) { _, newValue in
immersiveViewModel.rootEntity?.getFirstChildByName(name: "StartButton")?.isEnabled = !newValue
}
.onDisappear {
playerViewModel.reset()
}
.transition(.opacity)

}

}

}

// place effect views
immersiveViewModel.lineParticleView
immersiveViewModel.rainParticleView
immersiveViewModel.fireworksParticleView
immersiveViewModel.env01View

View、ViewModelの作成と配置ができた

Metadataのtagに応じてViewを表示

enum VideoAction: String { case none case c_reset case c_on_line_particle case c_off_line_particle case c_on_rain_particle case c_off_rain_particle case c_on_fireworks_particle case c_off_fireworks_particle case c_on_env_01 case c_off_env_01 }

AVPlayerViewModel.swift extension AVPlayerViewModel: AVPlayerItemMetadataOutputPushDelegate { func metadataOutput(_ output: AVPlayerItemMetadataOutput, didOutputTimedMetadataGroups groups: [AVTimedMetadataGroup], from track: AVPlayerItemTrack?) { if let item = groups.first?.items.first, let metadataValue = item.value(forKey: "value") as? String { print("Metadata value: \(metadataValue)") videoAction = VideoAction(rawValue: metadataValue) ?? .none } } }

AVPlayerViewModel.swift extension AVPlayerViewModel: AVPlayerItemMetadataOutputPushDelegate { func metadataOutput(_ output: AVPlayerItemMetadataOutput, didOutputTimedMetadataGroups groups: [AVTimedMetadataGroup], from track: AVPlayerItemTrack?) { if let item = groups.first?.items.first, let metadataValue = item.value(forKey: "value") as? String { print("Metadata value: \(metadataValue)") videoAction = VideoAction(rawValue: metadataValue) ?? .none } } }

ImmersiveView.swift (抜粋)
struct ImmersiveView: View {
@State var immersiveViewModel = ImmersiveViewModel()
var body: some View {
ZStack {
RealityView { content in
let entity = Entity()
content.add(entity)
immersiveViewModel.setup(entity: entity)
}
.gesture(SpatialTapGesture().targetedToAnyEntity()
.onEnded { value in
if value.entity.name == "StartButton" {
playerViewModel.play()
}
}
)
.onChange(of: playerViewModel.videoAction, initial: true) { oldValue, newValue in
immersiveViewModel.processVideoAction(oldValue: oldValue, newValue: newValue)
}
.onChange(of: playerViewModel.isPlaying, initial: false) { _, newValue in
immersiveViewModel.rootEntity?.getFirstChildByName(name: "StartButton")?.isEnabled = !newValue
}
.onDisappear {
playerViewModel.reset()
}
.transition(.opacity)

}

}

}

// place effect views
immersiveViewModel.lineParticleView
immersiveViewModel.rainParticleView
immersiveViewModel.fireworksParticleView
immersiveViewModel.env01View

ImmersiveViewModel.swift func processVideoAction(oldValue: VideoAction = .none, newValue: VideoAction = .none) { // avoid continuous firing of actions other than reset action if newValue != .c_reset && oldValue == newValue { return } switch newValue { case .none: break case .c_reset: resetAction() case .c_on_line_particle: Task { await play(viewName: LineParticleView.viewName) await fadeIn(viewName: LineParticleView.viewName) } case .c_off_line_particle: Task { await fadeOut(viewName: LineParticleView.viewName) } case .c_on_rain_particle: ...

ImmersiveViewModel.swift func processVideoAction(oldValue: VideoAction = .none, newValue: VideoAction = .none) { // avoid continuous firing of actions other than reset action if newValue != .c_reset && oldValue == newValue { return } switch newValue { case .none: break case .c_reset: resetAction() case .c_on_line_particle: Task { await play(viewName: LineParticleView.viewName) await fadeIn(viewName: LineParticleView.viewName) } case .c_off_line_particle: Task { await fadeOut(viewName: LineParticleView.viewName) } case .c_on_rain_particle: ...

ImmersiveViewModel.swift func processVideoAction(oldValue: VideoAction = .none, newValue: VideoAction = .none) { // avoid continuous firing of actions other than reset action if newValue != .c_reset && oldValue == newValue { return } switch newValue { case .none: break case .c_reset: resetAction() case .c_on_line_particle: Task { await play(viewName: LineParticleView.viewName) await fadeIn(viewName: LineParticleView.viewName) } case .c_off_line_particle: Task { await fadeOut(viewName: LineParticleView.viewName) } case .c_on_rain_particle: Task { await play(viewName: RainParticleView.viewName) await fadeIn(viewName: RainParticleView.viewName) } case .c_off_rain_particle: Task { await fadeOut(viewName: RainParticleView.viewName) } case .c_on_fireworks_particle: Task { await play(viewName: FireworksParticleView.viewName) await fadeIn(viewName: FireworksParticleView.viewName) } case .c_off_fireworks_particle: Task { await fadeOut(viewName: FireworksParticleView.viewName) } case .c_on_env_01: Task { await fadeIn(viewName: Env01View.viewName) } case .c_off_env_01: Task { await fadeOut(viewName: Env01View.viewName) } } }

これで全部つながった

🎉

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

Metadata付きHLSを作成し配置する 空間演出を作成する 動画再生する 空間演出を表示する 応用: 外部操作から演出を制御する

OSCで外部から制御する

https://www.elgato.com/jp/ja/p/stream-deck-mk2-black

• https://www.elgato.com/jp/ja/p/stream-deck-mk2-black

ImmersiveViewModel.swift import OSCKit class ImmersiveViewModel { private let oscClient = OSCClient() private let oscServer = OSCServer(port: 55535) private let addressSpace = OSCAddressSpace() func setup(entity: Entity) { rootEntity = entity ... setupOSC() } // MARK: - OSC private func setupOSC() { ... } }

ImmersiveViewModel.swift private func setupOSC() { ... addressSpace.register(localAddress: "/line_on") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_on_line_particle) } addressSpace.register(localAddress: "/line_off") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_off_line_particle) } oscServer.setHandler { [weak self] message, timeTag in do { try self?.handle(message: message, timeTag: timeTag) } catch { print(error) } } do { try oscServer.start() } catch { print(error) } } private func handle(message: OSCMessage, timeTag: OSCTimeTag) throws { let methodIDs = addressSpace.dispatch(message) if methodIDs.isEmpty { print("No method registered for:", message) } }

ImmersiveViewModel.swift private func setupOSC() { addressSpace.register(localAddress: "/reset") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_reset) } addressSpace.register(localAddress: "/line_on") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_on_line_particle) } addressSpace.register(localAddress: "/line_off") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_off_line_particle) } addressSpace.register(localAddress: "/rain_on") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_on_rain_particle) } addressSpace.register(localAddress: "/rain_off") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_off_rain_particle) } addressSpace.register(localAddress: "/fireworks_on") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_on_fireworks_particle) } addressSpace.register(localAddress: "/fireworks_off") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_off_fireworks_particle) } addressSpace.register(localAddress: "/env_01_on") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_on_env_01) } addressSpace.register(localAddress: "/env_01_off") { [weak self] _ in guard let self else { return } self.processVideoAction(newValue: .c_off_env_01) } oscServer.setHandler { [weak self] message, timeTag in do { try self?.handle( message: message, timeTag: timeTag ) } catch { print(error) } } do { } try oscServer.start() } catch { print(error) }

Stream Deck 設定アプリ

Stream Deck 設定アプリ

Stream Deck 設定アプリ

Demo Video

Demo

Demo Video

Wrap up 近日公開予定 Xcode16以降 visionOSシミュレータでも動作します Spatial E ects VideoPlayer HLS ID3 tag埋め込み コマンドファイル HLS Downloader HLS tag Viewer Touch Designer: OSC̲Dispatcher StreamDeck: pro le https://github.com/satoshi0212/visionOS_30Days fi ff https://github.com/satoshi0212/visionOS_2_30Days

• https://github.com/satoshi0212/visionOS_2_30Days
• https://github.com/satoshi0212/visionOS_30Days