visionOSアプリ開発 : Swiftで挑む最前線

Appleのサンプル群を解説！ Introductory visionOS samples: visionOS: RealityKit: Group Activities: TabletopKit: Compositor Services: AVFoundation: 8 1 1 1 2 3 1 1 1 1 1 Image I/O:

32.

Introductory visionOS samples から抜粋

33.

Creating D shapes with SwiftUI 2 https://developer.apple.com/documentation/visionos/creating-2d-shapes-in-visionos-with-swiftui

https://developer.apple.com/documentation/visionos/creating-2d-shapes-in-visionos-with-swiftui

34.

キモとなる処理

35.

[beta]

Triangle.swift
struct Triangle: Shape {
let vertex1: CGPoint
let vertex2: CGPoint
let vertex3: CGPoint
init(_ point1: CGPoint, _ point2: CGPoint, _ point3: CGPoint) {
vertex1 = point1
vertex2 = point2
vertex3 = point3
}
func path(in bounds: CGRect) -> Path {
/// The drawing path for the triangle shape.
var path = Path()
// Start at the first vertex.
path.move(to: vertex1)
// Draw the triangle's first two sides.
path.addLine(to: vertex2)
path.addLine(to: vertex3)
// Draw the triangle's third side by returning to the first vertex.
path.closeSubpath()
return path
}
}

36.

Creating D entities with RealityKit 3 https://developer.apple.com/documentation/visionos/creating-3d-shapes-in-visionos-with-realitykit

https://developer.apple.com/documentation/visionos/creating-3d-shapes-in-visionos-with-realitykit

37.

ShapesView.swift struct ShapesView: View { var body: some View { RealityView { content in addGeometryShapes(to: content) } } func addGeometryShapes(to content: RealityViewContent) { let allGeometryEntities = [ ShapesView.boxEntity, ShapesView.roundedBoxEntity, ShapesView.sphereEntity, ShapesView.coneEntity, ShapesView.cylinderEntity ] var xOffset: Float = -0.25 for entity in allGeometryEntities { entity.position.x = xOffset content.add(entity) xOffset += 0.125 } } }

38.

ShapesView.swift struct ShapesView: View { var body: some View { RealityView { content in addGeometryShapes(to: content) } } func addGeometryShapes(to content: RealityViewContent) { let allGeometryEntities = [ ShapesView.boxEntity, ShapesView.roundedBoxEntity, ShapesView.sphereEntity, ShapesView.coneEntity, ShapesView.cylinderEntity ] var xOffset: Float = -0.25 for entity in allGeometryEntities { entity.position.x = xOffset content.add(entity) xOffset += 0.125 } } }

39.

ShapesView+Entities.swift extension ShapesView { /// The white material that responds to lighting. static let whiteMaterial = SimpleMaterial(color: .white, isMetallic: false) /// The entity with a box geometry. static let boxEntity: Entity = { // Create a new entity instance. let entity = Entity() // Create a new mesh resource. let boxSize: Float = 0.1 let boxMesh = MeshResource.generateBox(size: boxSize) // Add the mesh resource to a model component, and add it to the entity. entity.components.set(ModelComponent(mesh: boxMesh, materials: [whiteMaterial])) return entity }() ... }

40.

Creating SwiftUI windows in visionOS https://developer.apple.com/documentation/visionos/creating-3d-shapes-in-visionos-with-realitykit

https://developer.apple.com/documentation/visionos/creating-3d-shapes-in-visionos-with-realitykit

41.

OpenWindowView.swift struct OpenWindowView: View { /// The `id` value that the main view uses to identify the SwiftUI window. @State var nextWindowID = NewWindowID(id: 1) /// The environment value for getting an `OpenWindowAction` instance. @Environment(\.openWindow) private var openWindow var body: some View { // Create a button in the center of the window that // launches a new SwiftUI window. Button("Open a new window") { // Open a new SwiftUI window with the assigned ID. openWindow(value: nextWindowID.id) // Increment the `id` value of the `nextWindowID` by 1. nextWindowID.id += 1 } } } fi /// A structure that gives each window a unique ID. struct NewWindowID: Identifiable { /// The unique identi er for the window. var id: Int }

42.

OpenWindowView.swift struct OpenWindowView: View { /// The `id` value that the main view uses to identify the SwiftUI window. @State var nextWindowID = NewWindowID(id: 1) /// The environment value for getting an `OpenWindowAction` instance. @Environment(\.openWindow) private var openWindow var body: some View { // Create a button in the center of the window that // launches a new SwiftUI window. Button("Open a new window") { // Open a new SwiftUI window with the assigned ID. openWindow(value: nextWindowID.id) // Increment the `id` value of the `nextWindowID` by 1. nextWindowID.id += 1 } } } fi /// A structure that gives each window a unique ID. struct NewWindowID: Identifiable { /// The unique identi er for the window. var id: Int }

43.

NewWindowView.swift struct NewWindowView: View { let id: Int var body: some View { // Create a text view that displays // the window's `id` value. Text("New window number \(id)") } }

44.

Creating an immersive space in visionOS https://developer.apple.com/documentation/visionos/creating-immersive-spaces-in-visionos-with-swiftui

https://developer.apple.com/documentation/visionos/creating-immersive-spaces-in-visionos-with-swiftui

45.

ImmersionView.swift struct ImmersiveView: View { let avgHeight: Float = 1.70 let speed: TimeInterval = 0.03 var body: some View { // Initiate a `RealityView` to create a ring // of rocks to orbit around a person. RealityView { content in /// The entity to contain the models. let rootEntity = Entity() // Set the y-axis position to the average human height. rootEntity.position.y += avgHeight // Create the halo effect with the `addHalo` method. rootEntity.addHalo() // Set the rotation speed for the rocks. rootEntity.components.set(TurnTableComponent(speed: speed)) // Register the `TurnTableSystem` to handle the rotation logic. TurnTableSystem.registerSystem() // Add the entity to the view. content.add(rootEntity) } } }

46.

ImmersionView.swift struct ImmersiveView: View { let avgHeight: Float = 1.70 let speed: TimeInterval = 0.03 var body: some View { // Initiate a `RealityView` to create a ring // of rocks to orbit around a person. RealityView { content in /// The entity to contain the models. let rootEntity = Entity() // Set the y-axis position to the average human height. rootEntity.position.y += avgHeight // Create the halo effect with the `addHalo` method. rootEntity.addHalo() // Set the rotation speed for the rocks. rootEntity.components.set(TurnTableComponent(speed: speed)) // Register the `TurnTableSystem` to handle the rotation logic. TurnTableSystem.registerSystem() // Add the entity to the view. content.add(rootEntity) } } }

47.

ImmersionView.swift rock.usdz

48.

ImmersionView.swift struct ImmersiveView: View { let avgHeight: Float = 1.70 let speed: TimeInterval = 0.03 var body: some View { // Initiate a `RealityView` to create a ring // of rocks to orbit around a person. RealityView { content in /// The entity to contain the models. let rootEntity = Entity() // Set the y-axis position to the average human height. rootEntity.position.y += avgHeight // Create the halo effect with the `addHalo` method. rootEntity.addHalo() // Set the rotation speed for the rocks. rootEntity.components.set(TurnTableComponent(speed: speed)) // Register the `TurnTableSystem` to handle the rotation logic. TurnTableSystem.registerSystem() // Add the entity to the view. content.add(rootEntity) } } }

49.

[beta]

TurnTableSystem.swift
struct TurnTableComponent: Component {
var time: TimeInterval = 0
var speed: TimeInterval
var axis: SIMD3<Float>
init(speed: TimeInterval = 1.0, axis: SIMD3<Float> = [0, 1, 0]) {
self.speed = speed
self.axis = axis
}
}
struct TurnTableSystem: System {
static let query = EntityQuery(where: .has(TurnTableComponent.self))
init(scene: RealityKit.Scene) { }
func update(context: SceneUpdateContext) {
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
var comp = entity.components[TurnTableComponent.self]!
comp.time += context.deltaTime
entity.components[TurnTableComponent.self] = comp
// Adjust the orientation to update the angle, speed, and axis of rotation.
entity.setOrientation(simd_quatf(angle: Float(0.1 * comp.speed), axis: comp.axis),
relativeTo: entity)
}
}
}

50.

[beta]

TurnTableSystem.swift
struct TurnTableComponent: Component {
var time: TimeInterval = 0
var speed: TimeInterval
var axis: SIMD3<Float>
init(speed: TimeInterval = 1.0, axis: SIMD3<Float> = [0, 1, 0]) {
self.speed = speed
self.axis = axis
}
}
struct TurnTableSystem: System {
static let query = EntityQuery(where: .has(TurnTableComponent.self))
init(scene: RealityKit.Scene) { }
func update(context: SceneUpdateContext) {
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
var comp = entity.components[TurnTableComponent.self]!
comp.time += context.deltaTime
entity.components[TurnTableComponent.self] = comp
// Adjust the orientation to update the angle, speed, and axis of rotation.
entity.setOrientation(simd_quatf(angle: Float(0.1 * comp.speed), axis: comp.axis),
relativeTo: entity)
}
}
}

51.

https://developer.apple.com/documentation/RealityKit/implementing-systems-for-entities-in-a-scene

https://developer.apple.com/documentation/RealityKit/implementing-systems-for-entities-in-a-scene

52.

Displaying a stereoscopic image https://developer.apple.com/documentation/visionos/creating-stereoscopic-image-in-visionos

https://developer.apple.com/documentation/visionos/creating-stereoscopic-image-in-visionos

53.

54.

LeftTexture RightTexture

55.

56.

https://developer.apple.com/documentation/ShaderGraph/realitykit/Camera-Index-Switch-(RealityKit)

https://developer.apple.com/documentation/ShaderGraph/realitykit/Camera-Index-Switch-(RealityKit)

57.

StereoImage.swift struct StereoImage: View { var body: some View { let spacing: CGFloat = 10.0 let padding: CGFloat = 40.0 VStack(spacing: spacing) { Text("Stereoscopic Image Example") .font(.largeTitle) RealityView { content in /// The creator instance of `StereoImageCreator`. let creator = StereoImageCreator() /// The image entity that `StereoImageCreator` generates. guard let entity = await creator.createImageEntity() else { print("Failed to create the stereoscopic image entity.") return } content.add(entity) } } .padding(padding) } }

58.

StereoImage.swift struct StereoImage: View { var body: some View { let spacing: CGFloat = 10.0 let padding: CGFloat = 40.0 VStack(spacing: spacing) { Text("Stereoscopic Image Example") .font(.largeTitle) RealityView { content in /// The creator instance of `StereoImageCreator`. let creator = StereoImageCreator() /// The image entity that `StereoImageCreator` generates. guard let entity = await creator.createImageEntity() else { print("Failed to create the stereoscopic image entity.") return } content.add(entity) } } .padding(padding) } }

59.

[beta]

StereoImageCreator.swift
@MainActor
public func createImageEntity() async -> ModelEntity? {
let materialRoot: String = "/Root/Material"
let materialName: String = "StereoscopicMaterial"
guard var material = try? await ShaderGraphMaterial(named: materialRoot, from: materialName) else {
print("Failed to load shader graph material.")
return nil
}
/// The size of the box in three dimensions.
let size: Float = 0.3
/// The z-axis scale to compress the box into an image placeholder.
let zScale: Float = 1E-3
// Generates the model entity in the shape of a box and applies the shader graph material.
let box = ModelEntity(
mesh: .generateBox(size: size),
materials: [material]
)
// Apply the z-axis scale to compress the box into a flat plane.
box.scale.z = zScale
// Load textures and apply to the box.
await applyTextureToEntity(box: box, material: &material)
return box
}

60.

[beta]

StereoImageCreator.swift
@MainActor
public func createImageEntity() async -> ModelEntity? {
let materialRoot: String = "/Root/Material"
let materialName: String = "StereoscopicMaterial"
guard var material = try? await ShaderGraphMaterial(named: materialRoot, from: materialName) else {
print("Failed to load shader graph material.")
return nil
}
/// The size of the box in three dimensions.
let size: Float = 0.3
/// The z-axis scale to compress the box into an image placeholder.
let zScale: Float = 1E-3
// Generates the model entity in the shape of a box and applies the shader graph material.
let box = ModelEntity(
mesh: .generateBox(size: size),
materials: [material]
)
// Apply the z-axis scale to compress the box into a flat plane.
box.scale.z = zScale
// Load textures and apply to the box.
await applyTextureToEntity(box: box, material: &material)
return box
}

61.

[beta]

StereoImageCreator.swift
@MainActor
private func applyTextureToEntity(box: ModelEntity, material: inout ShaderGraphMaterial) async {
do {
let leftFileName = "Shop_L"
let rightFileName = "Shop_R"
let leftTexture = try await TextureResource(named: leftFileName)
let rightTexture = try await TextureResource(named: rightFileName)
let leftParameter = "LeftTexture"
let rightParameter = "RightTexture"
// Set the textures into the shader graph material.
try material.setParameter(name: leftParameter, value: .textureResource(leftTexture))
try material.setParameter(name: rightParameter, value: .textureResource(rightTexture))
// Apply the results to the material and assign to the box's material.
box.model?.materials = [material]
} catch {
// Handle any errors that occur.
assertionFailure("\(error)")
}
}

62.

[beta]

StereoImageCreator.swift
@MainActor
private func applyTextureToEntity(box: ModelEntity, material: inout ShaderGraphMaterial) async {
do {
let leftFileName = "Shop_L"
let rightFileName = "Shop_R"
let leftTexture = try await TextureResource(named: leftFileName)
let rightTexture = try await TextureResource(named: rightFileName)
let leftParameter = "LeftTexture"
let rightParameter = "RightTexture"
// Set the textures into the shader graph material.
try material.setParameter(name: leftParameter, value: .textureResource(leftTexture))
try material.setParameter(name: rightParameter, value: .textureResource(rightTexture))
// Apply the results to the material and assign to the box's material.
box.model?.materials = [material]
} catch {
// Handle any errors that occur.
assertionFailure("\(error)")
}
}

63.

Displaying a D environment through a portal 3 https://developer.apple.com/documentation/visionos/displaying-a-3d-environment-through-a-portal

https://developer.apple.com/documentation/visionos/displaying-a-3d-environment-through-a-portal

64.

UIPortalView.swift struct UIPortalView: View { @Environment(AppModel.self) private var appModel private let root = Entity() private let portalPlane = ModelEntity( mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()] ) var body: some View { ZStack(alignment: .bottom) { if appModel.immersiveSpaceState == .closed { portalView } ToggleImmersiveSpaceButton() .padding(50) } } var portalView: some View { ... } func createPortal() async throws { ... } }

65.

UIPortalView.swift struct UIPortalView: View { @Environment(AppModel.self) private var appModel private let root = Entity() private let portalPlane = ModelEntity( mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()] ) var body: some View { ZStack(alignment: .bottom) { if appModel.immersiveSpaceState == .closed { portalView } ToggleImmersiveSpaceButton() .padding(50) } } var portalView: some View { ... } func createPortal() async throws { ... } }

66.

UIPortalView.swift var portalView: some View { GeometryReader3D { geometry in RealityView { content in try? await createPortal() content.add(root) } update: { content in // Resize the scene based on the size of the reality view content. let size = content.convert(geometry.size, from: .local, to: .scene) updatePortalSize(width: size.x, height: size.y) } .frame(depth: 0.4) } .frame(depth: 0.4) .frame(width: 1200, height: 800) }

67.

UIPortalView.swift var portalView: some View { GeometryReader3D { geometry in RealityView { content in try? await createPortal() content.add(root) } update: { content in // Resize the scene based on the size of the reality view content. let size = content.convert(geometry.size, from: .local, to: .scene) updatePortalSize(width: size.x, height: size.y) } .frame(depth: 0.4) } .frame(depth: 0.4) .frame(width: 1200, height: 800) }

68.

[beta]

UIPortalView.swift
/// Sets up the portal and adds it to the `root.`
func createPortal() async throws {
// Create the entity that stores the content within the portal.
let world = Entity()
// Shrink the portal world and update the position
// to make it fit into the portal view.
world.scale *= 0.5
world.position.y -= 0.5
world.position.z -= 0.5
// Allow the entity to be visible only through a portal.
world.components.set(WorldComponent())
// Create the box environment and add it to the root.
try await createEnvironment(on: world)
root.addChild(world)
// Set up the portal to show the content in the `world`.
portalPlane.components.set(PortalComponent(target: world))
root.addChild(portalPlane)
}

fi

/// Con gures the portal mesh's width and height.
func updatePortalSize(width: Float, height: Float) {
portalPlane.model?.mesh = .generatePlane(width: width, height: height, cornerRadius: 0.03)
}

69.

[beta]

UIPortalView.swift
/// Sets up the portal and adds it to the `root.`
func createPortal() async throws {
// Create the entity that stores the content within the portal.
let world = Entity()
// Shrink the portal world and update the position
// to make it fit into the portal view.
world.scale *= 0.5
world.position.y -= 0.5
world.position.z -= 0.5
// Allow the entity to be visible only through a portal.
world.components.set(WorldComponent())
// Create the box environment and add it to the root.
try await createEnvironment(on: world)
root.addChild(world)
// Set up the portal to show the content in the `world`.
portalPlane.components.set(PortalComponent(target: world))
root.addChild(portalPlane)
}

fi

/// Con gures the portal mesh's width and height.
func updatePortalSize(width: Float, height: Float) {
portalPlane.model?.mesh = .generatePlane(width: width, height: height, cornerRadius: 0.03)
}

70.

[beta]

ImmersiveView.swift
func createEnvironment(on root: Entity) async throws {
/// The root entity for the box environment.
let assetRoot = try await Entity(named: "CornellBox.usda")
// Convert the image-based lighting file into a URL, and load it as an environment resource.
guard let iblURL = Bundle.main.url(forResource: "TeapotIBL", withExtension: "exr") else {
fatalError("Failed to load the Image-Based Lighting file.")
}
let iblEnv = try await EnvironmentResource(fromImage: iblURL)
/// The entity to perform image-based lighting on the environment.
let iblEntity = await Entity()
/// The image-based lighting component that contains background and lighting information.
var iblComp = ImageBasedLightComponent(source: .single(iblEnv))
iblComp.inheritsRotation = true
// Add the image-based lighting component to the entity.
await iblEntity.components.set(iblComp)
// Set up image-based lighting for the box environment.
await assetRoot.components.set(ImageBasedLightReceiverComponent(imageBasedLight: iblEntity))
// Add the image-based lighting entity to the box environment.
await assetRoot.addChild(iblEntity)
// Add the box environment to `root`.
await root.addChild(assetRoot)
}

71.

[beta]

ImmersiveView.swift
func createEnvironment(on root: Entity) async throws {
/// The root entity for the box environment.
let assetRoot = try await Entity(named: "CornellBox.usda")
// Convert the image-based lighting file into a URL, and load it as an environment resource.
guard let iblURL = Bundle.main.url(forResource: "TeapotIBL", withExtension: "exr") else {
fatalError("Failed to load the Image-Based Lighting file.")
}
let iblEnv = try await EnvironmentResource(fromImage: iblURL)
/// The entity to perform image-based lighting on the environment.
let iblEntity = await Entity()
/// The image-based lighting component that contains background and lighting information.
var iblComp = ImageBasedLightComponent(source: .single(iblEnv))
iblComp.inheritsRotation = true
// Add the image-based lighting component to the entity.
await iblEntity.components.set(iblComp)
// Set up image-based lighting for the box environment.
await assetRoot.components.set(ImageBasedLightReceiverComponent(imageBasedLight: iblEntity))
// Add the image-based lighting entity to the box environment.
await assetRoot.addChild(iblEntity)
// Add the box environment to `root`.
await root.addChild(assetRoot)
}

72.

用他サンプルも有な情報が豊富！

73.

開発の情報源について Appleのサンプル群を解説！自用サンプルを応して独機能を作ってみる

74.

開発の情報源について Appleのサンプル群を解説！自用サンプルを応して独機能を作ってみる

75.

して独自用サンプルを応機能を作ってみる

76.

示ポータル複数表機能！

77.

元ネタ

78.

Displaying a D environment through a portal 3 https://developer.apple.com/documentation/visionos/displaying-a-3d-environment-through-a-portal

https://developer.apple.com/documentation/visionos/displaying-a-3d-environment-through-a-portal

79.

示複数シーンを作り同時にポータル表

80.

81.

82.

83.

84.

ContentView.swift struct ContentView: View { @Environment(\.openImmersiveSpace) var openImmersiveSpace @Environment(\.dismissImmersiveSpace) var dismissImmersiveSpace @Environment(\.dismissWindow) var dismissWindow var loader: EnvironmentLoader private let portalPlanes = [ ModelEntity(mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()]), ModelEntity(mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()]), ModelEntity(mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()]) ] var body: some View { HStack(spacing: 80) { portalView_Garden portalView_MyEnvironment portalView_CornellBox } .padding(80) } ... }

85.

ContentView.swift struct ContentView: View { @Environment(\.openImmersiveSpace) var openImmersiveSpace @Environment(\.dismissImmersiveSpace) var dismissImmersiveSpace @Environment(\.dismissWindow) var dismissWindow var loader: EnvironmentLoader private let portalPlanes = [ ModelEntity(mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()]), ModelEntity(mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()]), ModelEntity(mesh: .generatePlane(width: 1.0, height: 1.0), materials: [PortalMaterial()]) ] var body: some View { HStack(spacing: 80) { portalView_Garden portalView_MyEnvironment portalView_CornellBox } .padding(80) } ... }

86.

ContentView.swift private var portalView_Garden: some View { ZStack { GeometryReader3D { geometry in RealityView { content in let world = await makeWorldEntity(spaceType: .garden) content.add(world) let plane = getTargetPortalPlane(spaceType: .garden) plane.components.set(PortalComponent(target: world)) content.add(plane) } update: { content in let plane = getTargetPortalPlane(spaceType: .garden) let size = content.convert(geometry.size, from: .local, to: .scene) plane.model?.mesh = .generatePlane(width: size.x, height: size.y, cornerRadius: 0.03) }.frame(depth: 0.4) }.frame(depth: 0.4) VStack { Button("Enter") { Task { await openImmersiveSpace(id: Day14App.ImmersiveSpaceSelection.garden.rawValue) dismissWindow() } }.glassBackgroundEffect() } } .frame(width: Day14App.previewSize.width, height: Day14App.previewSize.height) }

87.

ContentView.swift private var portalView_Garden: some View { ZStack { GeometryReader3D { geometry in RealityView { content in let world = await makeWorldEntity(spaceType: .garden) content.add(world) let plane = getTargetPortalPlane(spaceType: .garden) plane.components.set(PortalComponent(target: world)) content.add(plane) } update: { content in let plane = getTargetPortalPlane(spaceType: .garden) let size = content.convert(geometry.size, from: .local, to: .scene) plane.model?.mesh = .generatePlane(width: size.x, height: size.y, cornerRadius: 0.03) }.frame(depth: 0.4) }.frame(depth: 0.4) VStack { Button("Enter") { Task { await openImmersiveSpace(id: Day14App.ImmersiveSpaceSelection.garden.rawValue) dismissWindow() } }.glassBackgroundEffect() } } .frame(width: Day14App.previewSize.width, height: Day14App.previewSize.height) }

88.

[beta]

ContentView.swift
private func makeWorldEntity(spaceType: Day14App.ImmersiveSpaceSelection) async -> Entity {
switch (spaceType) {
case .garden:
let world = Entity()
world.components.set(WorldComponent())
world.addChild(try! await loader.getEntity_Garden())
let scale: Float = 0.2
world.scale *= scale
world.position.y -= scale * 1.5
return world
case .myEnvironment:
let world = Entity()
world.components.set(WorldComponent())
world.addChild(try! await loader.getEntity_MyEnvironment())
let scale: Float = 0.2
world.scale *= scale
world.position.y -= scale * 1.5
return world
case .cornellBox:
let world = Entity()
world.components.set(WorldComponent())
world.scale *= 0.5
world.position.y -= 0.5
world.position.z -= 0.5
world.addChild(try! await loader.getEntity_CornellBox())
return world
}
}

89.

[beta]

ContentView.swift
private func makeWorldEntity(spaceType: Day14App.ImmersiveSpaceSelection) async -> Entity {
switch (spaceType) {
case .garden:
let world = Entity()
world.components.set(WorldComponent())
world.addChild(try! await loader.getEntity_Garden())
let scale: Float = 0.2
world.scale *= scale
world.position.y -= scale * 1.5
return world
case .myEnvironment:
let world = Entity()
world.components.set(WorldComponent())
world.addChild(try! await loader.getEntity_MyEnvironment())
let scale: Float = 0.2
world.scale *= scale
world.position.y -= scale * 1.5
return world
case .cornellBox:
let world = Entity()
world.components.set(WorldComponent())
world.scale *= 0.5
world.position.y -= 0.5
world.position.z -= 0.5
world.addChild(try! await loader.getEntity_CornellBox())
return world
}
}

90.

[beta]

EnvironmentLoader.swift
actor EnvironmentLoader {
private weak var entity_Garden: Entity?
private weak var entity_MyImmersive: Entity?
private weak var entity_CornellBox: Entity?
func getEntity_Garden() async throws -> Entity {
if let entity = entity_Garden { return entity }
let entity = try await Entity(named: "Garden", in: realityKitContentBundle)
entity_Garden = entity
return entity
}
func getEntity_MyImmersive() async throws -> Entity {
if let entity = entity_MyEnvironment { return entity }
let entity = try await Entity(named: "Immersive", in: realityKitContentBundle)
entity_MyEnvironment = entity
return entity
}
func getEntity_CornellBox() async throws -> Entity {
...
}
}

91.

[beta]

EnvironmentLoader.swift
actor EnvironmentLoader {
private weak var entity_Garden: Entity?
private weak var entity_MyImmersive: Entity?
private weak var entity_CornellBox: Entity?
func getEntity_Garden() async throws -> Entity {
if let entity = entity_Garden { return entity }
let entity = try await Entity(named: "Garden", in: realityKitContentBundle)
entity_Garden = entity
return entity
}
func getEntity_MyImmersive() async throws -> Entity {
if let entity = entity_MyEnvironment { return entity }
let entity = try await Entity(named: "Immersive", in: realityKitContentBundle)
entity_MyEnvironment = entity
return entity
}
func getEntity_CornellBox() async throws -> Entity {
...
}
}

92.

93.

🎉

94.

して独自用サンプルを応機能を作ってみる

95.

Shader Graphで動画からクロマキー処理！

96.

元ネタ

97.

Displaying a stereoscopic image https://developer.apple.com/documentation/visionos/creating-stereoscopic-image-in-visionos

https://developer.apple.com/documentation/visionos/creating-stereoscopic-image-in-visionos

98.

動画からリアルタイムで画像データを切り出す + クロマキー処理

99.

元素材

100.

101.

動画からリアルタイムで画像データを切り出す

102.

[beta]

ViewModel.swift
@MainActor
func setup() async {
...
displayLink = CADisplayLink(target: self, selector: #selector(displayLinkUpdated(link:)))
displayLink!.preferredFramesPerSecond = 60
displayLink!.add(to: .main, forMode: .default)
let urlPath = Bundle.main.path(forResource: "video001", ofType: "mov")!
let url = URL(fileURLWithPath: urlPath)
let videoAsset = AVURLAsset(url: url, options: [:])
videoPlayerItem = AVPlayerItem(asset: videoAsset)
let settings: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
kCVPixelBufferMetalCompatibilityKey as String: true
]
videoPlayerOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
videoPlayerItem?.add(videoPlayerOutput!)
player = AVPlayer(playerItem: videoPlayerItem)
player!.play()
}
func exit() {
displayLink?.invalidate()
player?.pause()
}

103.

[beta]

ViewModel.swift
@MainActor
func setup() async {
...
displayLink = CADisplayLink(target: self, selector: #selector(displayLinkUpdated(link:)))
displayLink!.preferredFramesPerSecond = 60
displayLink!.add(to: .main, forMode: .default)
let urlPath = Bundle.main.path(forResource: "video001", ofType: "mov")!
let url = URL(fileURLWithPath: urlPath)
let videoAsset = AVURLAsset(url: url, options: [:])
videoPlayerItem = AVPlayerItem(asset: videoAsset)
let settings: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
kCVPixelBufferMetalCompatibilityKey as String: true
]
videoPlayerOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
videoPlayerItem?.add(videoPlayerOutput!)
player = AVPlayer(playerItem: videoPlayerItem)
player!.play()
}

生

func exit() {
displayLink?.invalidate()
player?.pause()
}

動画再

104.

[beta]

ViewModel.swift
@MainActor
func setup() async {
...
displayLink = CADisplayLink(target: self, selector: #selector(displayLinkUpdated(link:)))
displayLink!.preferredFramesPerSecond = 60
displayLink!.add(to: .main, forMode: .default)
let urlPath = Bundle.main.path(forResource: "video001", ofType: "mov")!
let url = URL(fileURLWithPath: urlPath)
let videoAsset = AVURLAsset(url: url, options: [:])
videoPlayerItem = AVPlayerItem(asset: videoAsset)
let settings: [String: Any] = [
kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
kCVPixelBufferMetalCompatibilityKey as String: true
]
videoPlayerOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
videoPlayerItem?.add(videoPlayerOutput!)
player = AVPlayer(playerItem: videoPlayerItem)
player!.play()
}
func exit() {
displayLink?.invalidate()
player?.pause()
}

リアルタイム
切り出し

105.

[beta]

ViewModel.swift
@objc private func displayLinkUpdated(link: CADisplayLink) {
guard let textureResource else { return }
guard let cgImage = getCurrentFrameTexture()?.cgImage else { return }
try? textureResource.replace(withImage: cgImage,
options: TextureResource.CreateOptions(semantic: nil))
}
extension MTLTexture {
var cgImage: CGImage? {
guard let image = CIImage(mtlTexture: self, options: nil) else { return nil }
let flipped = image.transformed(by: CGAffineTransform(scaleX: 1, y: -1))
return context.createCGImage(flipped,
from: flipped.extent,
format: CIFormat.BGRA8,
colorSpace: CGColorSpace(name: CGColorSpace.linearDisplayP3)!)
}
}

106.

[beta]

ViewModel.swift
@objc private func displayLinkUpdated(link: CADisplayLink) {
guard let textureResource else { return }
guard let cgImage = getCurrentFrameTexture()?.cgImage else { return }
try? textureResource.replace(withImage: cgImage,
options: TextureResource.CreateOptions(semantic: nil))
}
extension MTLTexture {
var cgImage: CGImage? {
guard let image = CIImage(mtlTexture: self, options: nil) else { return nil }
let flipped = image.transformed(by: CGAffineTransform(scaleX: 1, y: -1))
return context.createCGImage(flipped,
from: flipped.extent,
format: CIFormat.BGRA8,
colorSpace: CGColorSpace(name: CGColorSpace.linearDisplayP3)!)
}
}

107.

[beta]

ViewModel.swift
private func getCurrentFrameTexture() -> MTLTexture? {
guard let videoPlayerOutput = videoPlayerOutput,
let videoPlayerItem = videoPlayerItem else { return nil }
guard let textureCache else { return nil }
let currentTime = videoPlayerItem.currentTime()
var itemTimeForDisplay = CMTime.zero
guard let buffer = videoPlayerOutput.copyPixelBuffer(forItemTime: currentTime,
itemTimeForDisplay: &itemTimeForDisplay) else { return nil }
let width = CVPixelBufferGetWidth(buffer)
let height = CVPixelBufferGetHeight(buffer)
let planeCount = CVPixelBufferGetPlaneCount(buffer)
var textureRef: CVMetalTexture?
let result = CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault, textureCache, buffer, nil,
.bgra8Unorm_srgb, width, height, planeCount, &textureRef
)
guard result == kCVReturnSuccess,
let unwrappedTextureRef = textureRef
else { return nil }
return CVMetalTextureGetTexture(unwrappedTextureRef)
}

108.

[beta]

ViewModel.swift
private func getCurrentFrameTexture() -> MTLTexture? {
guard let videoPlayerOutput = videoPlayerOutput,
let videoPlayerItem = videoPlayerItem else { return nil }
guard let textureCache else { return nil }
let currentTime = videoPlayerItem.currentTime()
var itemTimeForDisplay = CMTime.zero
guard let buffer = videoPlayerOutput.copyPixelBuffer(forItemTime: currentTime,
itemTimeForDisplay: &itemTimeForDisplay) else { return nil }
let width = CVPixelBufferGetWidth(buffer)
let height = CVPixelBufferGetHeight(buffer)
let planeCount = CVPixelBufferGetPlaneCount(buffer)
var textureRef: CVMetalTexture?
let result = CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault, textureCache, buffer, nil,
.bgra8Unorm_srgb, width, height, planeCount, &textureRef
)
guard result == kCVReturnSuccess,
let unwrappedTextureRef = textureRef
else { return nil }
return CVMetalTextureGetTexture(unwrappedTextureRef)
}

109.

クロマキー処理

110.

クロマキー処理 ImageInput HueRange SaturateRange KeyColor ValueRange

111.

Shader Graphへ値を渡す箇所

112.

[beta]

ViewModel.swift
@MainActor
func setup() async {
...
dynamicMaterial = try! await ShaderGraphMaterial(named: "/Root/DynamicMaterial",
from: "Immersive.usda", in: realityKitContentBundle)
try! dynamicMaterial!.setParameter(name: "ImageInput",
value: .textureResource(self.textureResource!))
try! dynamicMaterial!.setParameter(name: "KeyColor",
value: .color(CGColor(red: 67/255, green: 133/255, blue: 74/255, alpha: 1.0)))
try! dynamicMaterial!.setParameter(name: "HueRange", value: .float(0.03))
try! dynamicMaterial!.setParameter(name: "SaturateRange", value: .float(0.39))
try! dynamicMaterial!.setParameter(name: "ValueRange", value: .float(0.45))
entity.components.set(ModelComponent(mesh: .generatePlane(width: 0.8, height: 0.8),
materials: [dynamicMaterial!]))
...
}

113.

114.

🎉

115.

まとめ開発の情報源について Appleのサンプル群を解説！サンプルを応して独機能を作ってみる自用 X: @shmdevelop で資料、参考URL、最新情報共有しています

visionOSアプリ開発 : Swiftで挑む最前線

satoshi0212

関連スライド

Unreal EngineとiPhoneで始めるリアルタイムAR配信

visionOSで空間演出を実現する方法

Building Apps for visionOS with Swift

猫でも分かるUnreal Engineの学び方 - 超初心者向け編 - 2023 v1.0

Unreal Engine5 Lumenの仕組みと肝心なところ

最新の6.0で学ぶ！初めてのひとのためのSpring Security

各ページのテキスト

visionOSアプリ開発 : Swiftで挑む最前線

satoshi0212

関連スライド

Unreal EngineとiPhoneで始める リアルタイムAR配信

visionOSで空間演出を実現する方法

Building Apps for visionOS with Swift

猫でも分かるUnreal Engineの学び方 - 超初心者向け編 - 2023 v1.0

Unreal Engine5 Lumenの仕組みと肝心なところ

最新の6.0で学ぶ！初めてのひとのためのSpring Security

各ページのテキスト

Unreal EngineとiPhoneで始めるリアルタイムAR配信