diff --git a/Handbook/src/advanced-topics/openxr/README.md b/Handbook/src/advanced-topics/openxr/README.md index bd07bfa4..27c4e98d 100644 --- a/Handbook/src/advanced-topics/openxr/README.md +++ b/Handbook/src/advanced-topics/openxr/README.md @@ -1,9 +1,9 @@ # OpenXR -The SenseGlove Unreal Engine Plugin has provided OpenXR-compatible hand tracking by implementing XR_EXT_hand_tracking since v2.1.0. +The SenseGlove Unreal Engine Plugin has provided OpenXR-compatible hand tracking by implementing `XR_EXT_hand_tracking` since `v2.1.0`. Typically a user does not need to know anything about OpenXR to use the plugin, so this section of the handbook is for advanced users who are looking for a way to directly consume the OpenXR data coming directly from either a SenseGlove device or if enabled in the plugin settings from hand-tracking. -Since the SenseGlove Unreal Engine Plugin registers itself as an OpenXRHandTracking motion controller device it becomes a hand-tracking provider for Unreal Engine, thus the OpenXR data from SenseGlove could always be retrieved from the Unreal Engine's IXTrackingSystem with one caveat. The caveat is if another OpenXR-compatible hand-tracking plugin, e.g. Epic's own OpenXRHandTracking, is enabled simultaneously it's not guaranteed that the FXRMotionControllerData retrieved from the IXTrackingSystem::GetMotionControllerData() method is coming from SenseGlove, as this method returns the first hand-tracking plugin it could find. Thus, SenseGlove provides its own implementation of GetMotionControllerData() which guarantees the retrieved FXRMotionControllerData is coming from the SenseGlove Unreal Engine Plugin; and this is the preferred way to that. +Since the SenseGlove Unreal Engine Plugin registers itself as an `OpenXRHandTracking` motion controller device it becomes a hand-tracking provider for Unreal Engine, thus the OpenXR data from SenseGlove could always be retrieved from the Unreal Engine's `IXTrackingSystem` with one caveat. The caveat is if another OpenXR-compatible hand-tracking plugin, e.g. Epic's own OpenXRHandTracking, is enabled simultaneously it's not guaranteed that the `FXRMotionControllerData` retrieved from the `IXTrackingSystem::GetMotionControllerData()` method is coming from SenseGlove, as this method returns the first hand-tracking plugin it could find. Thus, SenseGlove provides its own implementation of `GetMotionControllerData()` which guarantees the retrieved `FXRMotionControllerData` is coming from the SenseGlove Unreal Engine Plugin; and this is the preferred way to that. -In the next sections we'll see [how we can directly consume the FXRMotionControllerData](consuming-fxrmotioncontrollerdata/README.md) to draw and animate debug virtual hands in both [Blueprint](consuming-fxrmotioncontrollerdata/blueprint.md) and [C++](consuming-fxrmotioncontrollerdata/cpp.md). +In the next sections we'll see [how we can directly consume the `FXRMotionControllerData`](consuming-fxrmotioncontrollerdata/README.md) to draw and animate debug virtual hands in both [Blueprint](consuming-fxrmotioncontrollerdata/blueprint.md) and [C++](consuming-fxrmotioncontrollerdata/cpp.md). diff --git a/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/README.md b/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/README.md index aa2db720..5af42d7d 100644 --- a/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/README.md +++ b/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/README.md @@ -1,6 +1,6 @@ # Consuming FXRMotionControllerData -Taking a closer look at the FXRMotionControllerData declaration inside the Unreal Engine's HeadMountedDisplay module at [Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h), figuring out the data structure might not seem very straightforward: +Taking a closer look at the `FXRMotionControllerData` declaration inside the Unreal Engine's `HeadMountedDisplay` module at `[Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h)`, figuring out the data structure might not seem very straightforward: ```cpp USTRUCT(BlueprintType) @@ -132,15 +132,15 @@ But, fear not, we've got you covered! ### Organization of FXRMotionControllerData -The structure is organized to encapsulate all relevant data needed for motion controller tracking in a coherent and accessible manner. Boolean flags (`bValid` and `bIsGrasped`) provide quick checks on the state of the controller data. Identifiers (`DeviceName` and `ApplicationInstanceID`) ensure the correct association of data. Positional and rotational data (`GripPosition`, `GripRotation`, `AimPosition`, and `AimRotation`) offer precise tracking of the controller's movement. Arrays (`HandKeyPositions`, `HandKeyRotations`, and `HandKeyRadii`) allow detailed hand tracking, which is critical for immersive VR experiences. Lastly, the tracking status (`TrackingStatus`) informs the system of the reliability of the data being processed and whether the motion controller is actively being tracked or it's inactive at the moment. +The structure is organized to encapsulate all relevant data needed for motion controller tracking in a coherent and accessible manner. Boolean flags `bValid` and `bIsGrasped` provide quick checks on the state of the controller data. Identifiers `DeviceName` and `ApplicationInstanceID` ensure the correct association of data. Positional and rotational data `GripPosition`, `GripRotation`, `AimPosition`, and `AimRotation` offer precise tracking of the controller's movement. Arrays `HandKeyPositions`, `HandKeyRotations`, and `HandKeyRadii` allow detailed hand tracking, which is critical for immersive VR experiences. Lastly, the tracking status `TrackingStatus` informs the system of the reliability of the data being processed and whether the motion controller is actively being tracked or it's inactive at the moment. ### Processing the Data for Drawing and Animating a Virtual Hand -In order to draw and animate a virtual hand in real-time whether the data is coming from hand-tracking or a SenseGlove device, we could consume the data from the HandKeyPositions and HandKeyRotations fields of the FXRMotionControllerData struct. +In order to draw and animate a virtual hand in real-time whether the data is coming from hand-tracking or a SenseGlove device, we could consume the data from the `HandKeyPositions` and `HandKeyRotations` fields of the `FXRMotionControllerData` struct. -Both HandKeyPositions and HandKeyRotations contain 26 elements as defined by OpenXR's [XR_HAND_JOINT_COUNT_EXT](https://registry.khronos.org/OpenXR/specs/1.1/man/html/XR_HAND_JOINT_COUNT_EXT.html) and [XrHandJointLocationsEXT](https://registry.khronos.org/OpenXR/specs/1.0/man/html/XrHandJointLocationsEXT.html), etc. +Both `HandKeyPositions` and `HandKeyRotations` contain 26 elements as defined by OpenXR's [`XR_HAND_JOINT_COUNT_EXT`](https://registry.khronos.org/OpenXR/specs/1.1/man/html/XR_HAND_JOINT_COUNT_EXT.html) and [`XrHandJointLocationsEXT`](https://registry.khronos.org/OpenXR/specs/1.0/man/html/XrHandJointLocationsEXT.html), etc. -Unreal Engine also provides an enum called EHandKeypoint naming the 26 joints, and the equivalent of XR_HAND_JOINT_COUNT_EXT as EHandKeypointCount inside [Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h) as follows: +Unreal Engine also provides an enum called `EHandKeypoint` naming the 26 joints, and the equivalent of `XR_HAND_JOINT_COUNT_EXT` as `EHandKeypointCount` inside `[Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h)` as follows: ```cpp /** @@ -187,7 +187,7 @@ So, getting the any joint's position or rotation is as easy as casting the enum ```cpp FXRMotionControllerData MotionControllerData; const bool bGotMotionControllerData = FSGXRTracker::GetMotionControllerData( - this, EControllerHand::Left, MotionControllerData); + GetWorld(), EControllerHand::Left, MotionControllerData); // Return if the struct data is invalid! if (!bGotMotionControllerData || !MotionControllerData.bValid) diff --git a/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/blueprint.md b/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/blueprint.md index d7bd87b9..2f380e7b 100644 --- a/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/blueprint.md +++ b/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/blueprint.md @@ -12,40 +12,40 @@ Before continuing this section, please ensure you've studied the [Consuming FXRM 3. You could use either hand-tracking or a SenseGlove device as the input data, or both of the inside the same project. Whether you would like to use hand-tracking or a SenseGlove device, please make sure the required steps are taken for each of those first. -4. You could add the required Blueprint code for drawing virtual hands to either your Level Buleprint or the VRPawn Blueprint Class located at /Content/VRTemplate/Blueprints/VRPawn. In this guide we are going to add the code to our VRPawn. +4. You could add the required Blueprint code for drawing virtual hands to either your Level Buleprint or the VRPawn Blueprint Class located at `/Content/VRTemplate/Blueprints/VRPawn`. In this guide we are going to add the code to our VRPawn. -5. Add a new function named Draw Hand with an input parameter of type EController Hand named Hand. +5. Add a new function named `Draw Hand` with an input parameter of type `EController Hand` named `Hand`. ![Adding the Blueprint Draw Hand function](consuming-fxrmotioncontrollerdata-blueprint-add-draw-hand-function.png "Adding the Blueprint Draw Hand function") -6. Inside this function's event graph add a Get Motion Controller Data node from SenseGlove > Tracking > XR Tracker > Get Motion Controller Data. +6. Inside this function's event graph add a `Get Motion Controller Data` node from `SenseGlove > Tracking > XR Tracker > Get Motion Controller Data`. ![Adding the Get Motion Controller Data node](consuming-fxrmotioncontrollerdata-blueprint-add-get-motion-controller-data-node.png "Adding the Get Motion Controller Data node") -7. Then connect the functions Hand input parameter to the Get Motion Controller Data's Hand input and right-click on the OutMotionControllerData parameter and use the Break XRMotionControllerData node to break the struct to it's fields. +7. Then connect the functions `Hand` input parameter to the `Get Motion Controller Data`'s `Hand` input and right-click on the `OutMotionControllerData` parameter and use the `Break XRMotionControllerData` node to break the struct to it's fields. ![Breaking the XR Motion Controller Data node](consuming-fxrmotioncontrollerdata-blueprint-break-xr-motion-controller-data.png "Breaking the XR Motion Controller Data node") -8. After this, we need to perform data validation by checking the return status of the Get Motion Controller Data function and FXRMotionControllerData's Valid field. Then, we check if the motion controller device is being tracked and indeed coming from a hand-tracking source. And, finally, we check whether we have the positions and rotations for exactly 26 joints or not. +8. After this, we need to perform data validation by checking the return status of the `Get Motion Controller Data` function and `FXRMotionControllerData`'s `Valid` field. Then, we check if the motion controller device is being tracked and indeed coming from a hand-tracking source. And, finally, we check whether we have the positions and rotations for exactly `26` joints or not. ![FXRMotionControllerData validation](consuming-fxrmotioncontrollerdata-blueprint-validation.png "FXRMotionControllerData validation") -9. OK, now it's time to draw the joints! If we check out the SenseGlove Debug module's draw option, we notice there are various ways to draw the debug virtual hand. Drawing a cube or a gizmo per joint, or draw the whole hand all at once by passing the retrieved FXRMotionControllerData to the DebugVirtualHand::Draw function! But, since the point of this tutorial is to learn how to consume the FXRMotionControllerData we ignore the last option. Between the debug cubes or gizmos, we are going to choose the gizmos since they better represent the rotations than the cubes. +9. OK, now it's time to draw the joints! If we check out the SenseGlove Debug module's draw option, we notice there are various ways to draw the debug virtual hand. Drawing a cube or a gizmo per joint, or draw the whole hand all at once by passing the retrieved `FXRMotionControllerData` to the `DebugVirtualHand::Draw` function! But, since the point of this tutorial is to learn how to consume the `FXRMotionControllerData` we ignore the last option. Between the debug cubes or gizmos, we are going to choose the gizmos since they better represent the rotations than the cubes. ![Some options for drawing a debug virtual hand](consuming-fxrmotioncontrollerdata-blueprint-debug-virtual-hand-draw-options.png "Some options for drawing a debug virtual hand") -10. In the last step inside the Draw Hand function, in order to draw a virtual hand with 26 joints, we have to first iterate through either of the Hand Key Positions or Hand Key Rotations arrays from the FXRMotionControllerData struct. Since we made sure both arrays have 26 elements before we reached this step, it's safe to just iterate over one and use the Array Index inside a For Each Loop or a For Loop to access the position and rotation of every joint. Then we use each array Get (a ref) method to access the position and rotation data inside the loop and call the Draw function from SenseGlove > Debug > Gizmo per every joint. Please note that there are two Draw functions and the only difference between the two is that one accepts an FQuat and the other a FRotator for its Rotation input parameter. In this case, we use the FQuat variant to avoid an extra conversion to FRotator. Also, please adjust the Thickness option for the Settings parameter from 1.0 to 0.2, as the default value might be too thick for drawing a joint gizmo. +10. In the last step inside the `Draw Hand` function, in order to draw a virtual hand with `26` joints, we have to first iterate through either of the `Hand Key Positions` or `Hand Key Rotations` arrays from the `FXRMotionControllerData` struct. Since we made sure both arrays have `26` elements before we reached this step, it's safe to just iterate over one and use the `Array Index` inside a `For Each Loop` or a `For Loop` to access the position and rotation of every joint. Then we use each array `Get (a ref)` method to access the position and rotation data inside the loop and call the `Draw` function from `SenseGlove > Debug > Gizmo` per every joint. Please note that there are two `Draw` functions and the only difference between the two is that one accepts an `FQuat` and the other a `FRotator` for its `Rotation` input parameter. In this case, we use the `FQuat` variant to avoid an extra conversion to `FRotator`. Also, please adjust the `Thickness` option for the `Settings` parameter from `1.0` to `0.2`, as the default value might be too thick for drawing a joint gizmo. ![Drawing a debug gizmo per each hand's joint](consuming-fxrmotioncontrollerdata-blueprint-draw-debug-gizmo-per-joint.png "Drawing a debug gizmo per each hand's joint") -11. Well, now the full implementation for the Draw Hand function insde the VRPawn should look something like this: +11. Well, now the full implementation for the `Draw Hand` function insde the `VRPawn` should look something like this: ![VRPawn Draw Hand function final implementation](consuming-fxrmotioncontrollerdata-blueprint-draw-hand-function-implementation.png "VRPawn Draw Hand function final implementation") -12. Finally, go back to VRPawn's event graph and the following code to the Tick event. Basically what we do here is call our newly implemented Draw Hand twice, once for each hand. +12. Finally, go back to `VRPawn`'s event graph and the following code to the `Tick` event. Basically what we do here is call our newly implemented `Draw Hand` twice, once for each hand. ![Attempt to draw the left and right virtual hands every frame from the VR Pawn Tick event](consuming-fxrmotioncontrollerdata-blueprint-vrpawn-draw-virtual-hands-every-frame.png "Attempt to draw the left and right virtual hands every frame from the VR Pawn Tick event") -13. Now, go back to the VRTemplateMap and use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation. +13. Now, go back to the `VRTemplateMap` and use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation. ![FXRMotionControllerData animated debug virtual hands](consuming-fxrmotioncontrollerdata-animated-debug-virtual-hands.gif "FXRMotionControllerData animated debug virtual hands") \ No newline at end of file diff --git a/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/cpp.md b/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/cpp.md index 646a60a9..1cea1d02 100644 --- a/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/cpp.md +++ b/Handbook/src/advanced-topics/openxr/consuming-fxrmotioncontrollerdata/cpp.md @@ -12,33 +12,33 @@ Before continuing this section, please ensure you've first studied the [Consumin 3. You could use either hand-tracking or a SenseGlove device as the input data, or both of the inside the same project. Whether you would like to use hand-tracking or a SenseGlove device, please make sure the required steps are taken for each of those first. -4. From the Tools menu choose New C++ class.... +4. From the `Tools` menu choose `New C++ class...`. ![Creating a new C++ class](consuming-fxrmotioncontrollerdata-cpp-new-class.png "Creating a new C++ class") -5. Choose the Unreal Engine's APawn class as the parent class for the new C++ pawn class. +5. Choose the Unreal Engine's `APawn` class as the parent class for the new C++ pawn class. ![Choosing APawn as the parent class](consuming-fxrmotioncontrollerdata-cpp-add-class-choose-name.png "Choosing APawn as the parent class") -6. Name the new pawn class DebugPawn. +6. Name the new pawn class `DebugPawn`. ![Naming the new C++ class DebugPawn](consuming-fxrmotioncontrollerdata-cpp-add-class-choose-parent.png "Naming the new C++ class DebugPawn") 7. Since we have created a new C++ class, this converts the current Blueprint VRTemplateMap project to a C++ one. That's why the Unreal Editor will give us a few prompts regarding opening the project in the default IDE and rebuilding the code. It might be simpler to just close the editor, then rebuild the source code inside your favorite IDE, and then start the editor with the converted project again. -8. Find and open the VRPawn Blueprint Class located at /Content/VRTemplate/Blueprints/VRPawn inside the Blueprint Editor and from the File menu choose the Reparent Blueprint class. +8. Find and open the VRPawn Blueprint Class located at `/Content/VRTemplate/Blueprints/VRPawn` inside the Blueprint Editor and from the `File` menu choose the `Reparent Blueprint` class. ![Reparenting the VRPawn Blueprint class](consuming-fxrmotioncontrollerdata-cpp-reparent.png "Reparenting the VRPawn Blueprint class") -9. In the new Reparent blueprint window choose DebugPawn as the new parent. +9. In the new `Reparent blueprint` window choose `DebugPawn` as the new parent. ![Reparenting the VRPawn Blueprint class to ADebugPawn](consuming-fxrmotioncontrollerdata-cpp-reparent-debug-pawn.png "Reparenting the VRPawn Blueprint class to ADebugPawn") -10. By looking at the Parent Class label located under the Blueprint Editor window control buttons verify that the ADebugPawn class has been set as the new parent. +10. By looking at the `Parent Class` label located under the Blueprint Editor window control buttons verify that the `ADebugPawn` class has been set as the new parent. ![Veifying whether the VRPawn Blueprint class set to ADebugPawn or not](consuming-fxrmotioncontrollerdata-cpp-verify-parent-class.png "Veifying whether the VRPawn Blueprint class set to ADebugPawn or not") -11. Locate the project's main Build file, in our case VirtualHandCpp/Source/VirtualHandCpp/VirtualHandCpp.Build.cs and add the InputDevice, OpenXRHMD, SenseGloveBuildHacks, SenseGloveDebug, SenseGloveSettings, and SenseGloveTracking modules as either a private or public dependency. +11. Locate the project's main Build file, in our case `VirtualHandCpp/Source/VirtualHandCpp/VirtualHandCpp.Build.cs` and add the `InputDevice`, `OpenXRHMD`, `SenseGloveBuildHacks`, `SenseGloveDebug`, `SenseGloveSettings`, and `SenseGloveTracking` modules as either a private or public dependency. ```csharp // Fill out your copyright notice in the Description page of Project Settings. @@ -74,9 +74,9 @@ public class VirtualHandCpp : ModuleRules } ``` -12. Locate the C++ header and source file for the ADebugPawn inside the project in your C++ IDE. In our case they are located at VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.h and VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.cpp. +12. Locate the C++ header and source file for the `ADebugPawn` inside the project in your C++ IDE. In our case they are located at `VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.h` and `VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.cpp`. -13. Modify the DebugPawn.h header file to look like this: +13. Modify the `DebugPawn.h` header file to look like this: ```cpp // Fill out your copyright notice in the Description page of Project Settings. @@ -121,7 +121,7 @@ private: }; ``` -14. Modify the DebugPawn.cpp implementation file to look like this: +14. Modify the `DebugPawn.cpp` implementation file to look like this: ```cpp @@ -246,6 +246,6 @@ void ADebugPawn::DrawHand(const EControllerHand Hand) const } ``` -15. Now, rebuild the source code and go back to the VRTemplateMap, then use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation. +15. Now, rebuild the source code and go back to the `VRTemplateMap`, then use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation. ![FXRMotionControllerData animated debug virtual hands](consuming-fxrmotioncontrollerdata-animated-debug-virtual-hands.gif "FXRMotionControllerData animated debug virtual hands") \ No newline at end of file