add the documentation on consuming the FXRHandTrackingState struct in both Blueprint and C++
This commit is contained in:
Mamadou Babaei
@@ -63,6 +63,9 @@ Welcom to the SenseGlove Unreal Engine Handbook!
|
||||
|
||||
- [Safe Glove Access in Blueprint](advanced-topics/safe-glove-access-blueprint/README.md)
|
||||
- [OpenXR](advanced-topics/openxr/README.md)
|
||||
- [Consuming FXRHandTrackingState](advanced-topics/openxr/consuming-fxrhandtrackingstate/README.md)
|
||||
- [Blueprint](advanced-topics/openxr/consuming-fxrhandtrackingstate/blueprint.md)
|
||||
- [C++](advanced-topics/openxr/consuming-fxrhandtrackingstate/cpp.md)
|
||||
- [Consuming FXRMotionControllerData](advanced-topics/openxr/consuming-fxrmotioncontrollerdata/README.md)
|
||||
- [Blueprint](advanced-topics/openxr/consuming-fxrmotioncontrollerdata/blueprint.md)
|
||||
- [C++](advanced-topics/openxr/consuming-fxrmotioncontrollerdata/cpp.md)
|
||||
|
||||
@@ -4,6 +4,32 @@ The SenseGlove Unreal Engine Plugin has provided OpenXR-compatible hand tracking
|
||||
|
||||
Typically a user does not need to know anything about OpenXR to use the plugin, so this section of the handbook is for advanced users who are looking for a way to directly consume the OpenXR data coming directly from either a SenseGlove device or if enabled in the plugin settings from hand-tracking.
|
||||
|
||||
Since the SenseGlove Unreal Engine Plugin registers itself as an `OpenXRHandTracking` motion controller device it becomes a hand-tracking provider for Unreal Engine, thus the OpenXR data from SenseGlove could always be retrieved from the Unreal Engine's `IXTrackingSystem` with one caveat. The caveat is if another OpenXR-compatible hand-tracking plugin, e.g. Epic's own OpenXRHandTracking, is enabled simultaneously it's not guaranteed that the `FXRMotionControllerData` retrieved from the `IXTrackingSystem::GetMotionControllerData()` method is coming from SenseGlove, as this method returns the first hand-tracking plugin it could find. Thus, SenseGlove provides its own implementation of `GetMotionControllerData()` which guarantees the retrieved `FXRMotionControllerData` is coming from the SenseGlove Unreal Engine Plugin; and this is the preferred way to that.
|
||||
Since the SenseGlove Unreal Engine Plugin registers itself as an `OpenXRHandTracking` motion controller device it becomes a hand-tracking provider for Unreal Engine, thus the OpenXR data from SenseGlove could always be retrieved from the Unreal Engine's `IXTrackingSystem` with one caveat. The caveat is if another OpenXR-compatible hand-tracking plugin, e.g. Epic's own OpenXRHandTracking, is enabled simultaneously it's not guaranteed that the `FXRMotionControllerData` and `FXRHandTrackingState` structs retrieved from the `IXTrackingSystem::GetMotionControllerData()` and `IXTrackingSystem::GetHandTrackingState()` methods are coming from SenseGlove, as these methods return the first hand-tracking plugin they could find. Thus, SenseGlove provides its own implementation of `GetMotionControllerData()` and `GetHandTrackingState()` which guarantee the retrieved `FXRMotionControllerData` or `FXRHandTrackingState` are coming from the SenseGlove Unreal Engine Plugin; and this is the preferred way to that.
|
||||
|
||||
In the next sections we'll see [how we can directly consume the `FXRMotionControllerData`](consuming-fxrmotioncontrollerdata/) to draw and animate debug virtual hands in both [Blueprint](consuming-fxrmotioncontrollerdata/blueprint.md) and [C++](consuming-fxrmotioncontrollerdata/cpp.md).
|
||||
> [!NOTE]
|
||||
> In order to retrieve the latest `FXRMotionControllerState` available, The
|
||||
> SenseGlove Unreal Engine Plugin provides an alternative implementation for
|
||||
> `IXTrackingSystem::GetMotionControllerState()` as well . However, since this
|
||||
> method does not rely on the `OpenXRHandTracking` provider, it may become
|
||||
> redundant. As a result, we might consider removing this functionality in
|
||||
> future updates in favor of the one that Unreal Engine provides.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
In the next sections we'll see:
|
||||
|
||||
- [How we can directly consume the `FXRMotionControllerData`](consuming-fxrmotioncontrollerdata/) on UE `5.2`, `5.3`, `5.4`, and `5.5` to draw and animate debug virtual hands in both [Blueprint](consuming-fxrmotioncontrollerdata/blueprint.md) and [C++](consuming-fxrmotioncontrollerdata/cpp.md).
|
||||
|
||||
- [How we can directly consume the `FXRHandTrackingState`](consuming-fxrhandtrackingstate/) on UE `5.5` to draw and animate debug virtual hands in both [Blueprint](consuming-fxrhandtrackingstate/blueprint.md) and [C++](consuming-fxrhandtrackingstate/cpp.md).
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
# Consuming FXRHandTrackingState
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand-tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
Taking a closer look at the `FXRHandTrackingState` declaration inside the Unreal Engine's `HeadMountedDisplay` module at `[Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h)`, figuring out the data structure might not seem very straightforward:
|
||||
|
||||
```cpp
|
||||
USTRUCT(BlueprintType)
|
||||
struct FXRHandTrackingState
|
||||
{
|
||||
GENERATED_USTRUCT_BODY();
|
||||
|
||||
// The state is valid if poses have ever been provided.
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
bool bValid = false;
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FName DeviceName;
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FGuid ApplicationInstanceID;
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
EXRSpaceType XRSpaceType = EXRSpaceType::UnrealWorldSpace;
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
EControllerHand Hand = EControllerHand::Left;
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
ETrackingStatus TrackingStatus = ETrackingStatus::NotTracked;
|
||||
|
||||
// The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
TArray<FVector> HandKeyLocations;
|
||||
|
||||
// The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
TArray<FQuat> HandKeyRotations;
|
||||
|
||||
// The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
TArray<float> HandKeyRadii;
|
||||
};
|
||||
```
|
||||
|
||||
Which on the Blueprint side it looks like this:
|
||||
|
||||
|
||||
|
||||
But, fear not, we've got you covered!
|
||||
|
||||
## FXRHandTrackingState in Unreal Engine
|
||||
|
||||
`FXRHandTrackingState` is a structure in Unreal Engine designed to hold detailed information about the state of a hand-tracking device at a given moment. This structure is essential for handling hand-tracking inputs in virtual reality (VR) applications, providing the necessary data to accurately track and represent the user's hand movements and actions within the virtual environment.
|
||||
|
||||
### Structure Members of FXRHandTrackingState
|
||||
|
||||
- **bValid**
|
||||
- **Description**: A boolean flag indicating whether the data is valid or not.
|
||||
- **Usage**: This is used to check if the motion controller data is correctly initialized and can be used for further processing.
|
||||
|
||||
- **DeviceName**
|
||||
- **Type**: `FName`
|
||||
- **Description**: The name of the device.
|
||||
- **Usage**: Identifies which device the data is coming from, useful when multiple devices are in use.
|
||||
|
||||
- **ApplicationInstanceID**
|
||||
- **Type**: `FString`
|
||||
- **Description**: A unique identifier for the application instance.
|
||||
- **Usage**: Helps in differentiating data from different instances of an application, ensuring the correct instance processes the data.
|
||||
|
||||
- **XRSpaceType**
|
||||
- **Type**: `EXRSpaceType`
|
||||
- **Description**: Enum specifying the type of XR space being used (e.g., unreal world or tracking space).
|
||||
- **Usage**: Specifies the coordinate system the XR Device is tracking itself in.
|
||||
|
||||
- **Hand**
|
||||
- **Type**: `EControllerHand`
|
||||
- **Description**: Enum indicating which hand is being tracked (left or right).
|
||||
- **Usage**: Helps identify whether the hand-tracking data pertains to the left or right hand, essential for hand-specific actions or interactions.
|
||||
|
||||
- **TrackingStatus**
|
||||
- **Type**: `EXRTrackingStatus`
|
||||
- **Description**: Enum indicating the tracking status of the hand-tracking device.
|
||||
- **Usage**: Shows whether the hand-tracking device is being tracked accurately, with possible statuses like `Tracked`, `NotTracked`, etc.
|
||||
|
||||
- **HandKeyLocations**
|
||||
- **Type**: `TArray<FVector>`
|
||||
- **Description**: An array of vectors representing key locations of the hand.
|
||||
- **Usage**: Provides detailed locations of key points on the hand, useful for precise hand-tracking and interaction.
|
||||
|
||||
- **HandKeyRotations**
|
||||
- **Type**: `TArray<FQuat>`
|
||||
- **Description**: An array of quaternions representing key rotations of the hand.
|
||||
- **Usage**: Complements the hand key locations with rotational data, ensuring accurate representation of hand movements.
|
||||
|
||||
- **HandKeyRadii**
|
||||
- **Type**: `TArray<float>`
|
||||
- **Description**: An array of floats representing the radii of key points of the hand.
|
||||
- **Usage**: Gives the size of the hand key points, aiding in collision detection and interaction fidelity.
|
||||
|
||||
### Organization of FXRHandTrackingState
|
||||
|
||||
The structure is organized to encapsulate all relevant data needed for hand-tracking in a coherent and accessible manner. Boolean flag `bValid` provides quick checks on the state of the controller data. Identifiers `DeviceName` and `ApplicationInstanceID` ensure the correct association of data. Arrays `HandKeyLocations`, `HandKeyRotations`, and `HandKeyRadii` allow detailed hand-tracking, which is critical for immersive VR experiences. Lastly, the tracking status `TrackingStatus` informs the system of the reliability of the data being processed and whether the hands are actively being tracked or they are inactive at the moment.
|
||||
|
||||
### Processing the Data for Drawing and Animating a Virtual Hand
|
||||
|
||||
In order to draw and animate a virtual hand in real-time whether the data is coming from hand-tracking or a SenseGlove device, we could consume the data from the `HandKeyLocations` and `HandKeyRotations` fields of the `FXRHandTrackingState` struct.
|
||||
|
||||
Both `HandKeyLocations` and `HandKeyRotations` contain 26 elements as defined by OpenXR's [`XR_HAND_JOINT_COUNT_EXT`](https://registry.khronos.org/OpenXR/specs/1.1/man/html/XR_HAND_JOINT_COUNT_EXT.html) and [`XrHandJointLocationsEXT`](https://registry.khronos.org/OpenXR/specs/1.0/man/html/XrHandJointLocationsEXT.html), etc.
|
||||
|
||||
Unreal Engine also provides an enum called `EHandKeypoint` naming the 26 joints, and the equivalent of `XR_HAND_JOINT_COUNT_EXT` as `EHandKeypointCount` inside `[Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h)` as follows:
|
||||
|
||||
```cpp
|
||||
/**
|
||||
* Transforms that are tracked on the hand.
|
||||
* Matches the enums from WMR to make it a direct mapping
|
||||
*/
|
||||
UENUM(BlueprintType)
|
||||
enum class EHandKeypoint : uint8
|
||||
{
|
||||
Palm,
|
||||
Wrist,
|
||||
ThumbMetacarpal,
|
||||
ThumbProximal,
|
||||
ThumbDistal,
|
||||
ThumbTip,
|
||||
IndexMetacarpal,
|
||||
IndexProximal,
|
||||
IndexIntermediate,
|
||||
IndexDistal,
|
||||
IndexTip,
|
||||
MiddleMetacarpal,
|
||||
MiddleProximal,
|
||||
MiddleIntermediate,
|
||||
MiddleDistal,
|
||||
MiddleTip,
|
||||
RingMetacarpal,
|
||||
RingProximal,
|
||||
RingIntermediate,
|
||||
RingDistal,
|
||||
RingTip,
|
||||
LittleMetacarpal,
|
||||
LittleProximal,
|
||||
LittleIntermediate,
|
||||
LittleDistal,
|
||||
LittleTip
|
||||
};
|
||||
|
||||
const int32 EHandKeypointCount = static_cast<int32>(EHandKeypoint::LittleTip) + 1;
|
||||
|
||||
```
|
||||
|
||||
So, getting the any joint's location or rotation is as easy as casting the enum value and passing it as the array index.
|
||||
|
||||
```cpp
|
||||
FXRHandTrackingState HandTrackingState;
|
||||
const bool bGotHandTrackingState = FSGXRTracker::GetHandTrackingState(
|
||||
GetWorld(), EXRSpaceType::UnrealWorldSpace, EControllerHand::Left, HandTrackingState);
|
||||
|
||||
// Return if the struct data is invalid!
|
||||
if (!bGotHandTrackingState || !HandTrackingState.bValid)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Return if the device is not being tracked!
|
||||
if (HandTrackingState.TrackingStatus == ETrackingStatus::NotTracked)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure that HandTrackingState.HandKeyLocations has the location data
|
||||
// for 26 joints!
|
||||
if (!ensureAlwaysMsgf(HandTrackingState.HandKeyLocations.Num()
|
||||
== EHandKeypointCount,
|
||||
TEXT("Invalid HandKeyLocations count!")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure that HandTrackingState.HandKeyRotations has the rotation data
|
||||
// for 26 joints!
|
||||
if (!ensureAlwaysMsgf(HandTrackingState.HandKeyRotations.Num()
|
||||
== EHandKeypointCount,
|
||||
TEXT("Invalid HandKeyRotations count!")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
static constexpr int32 PalmIndex = static_cast<int32>(EHandKeypoint::Palm);
|
||||
|
||||
const FVector& PalmLocation{
|
||||
HandTrackingState.HandKeyLocations[PalmIndex]
|
||||
};
|
||||
const FRotator& PalmRotation{
|
||||
HandTrackingState.HandKeyRotations[PalmIndex].Rotator()
|
||||
};
|
||||
|
||||
```
|
||||
|
||||
The equivalent Blueprint code for the above looks something like this:
|
||||
|
||||
|
||||
|
||||
OK, now that we've got a glimpse of how the virtual hand's joint data could be processed we are going to draw and animate a virtual hand in both [Blueprint](blueprint.md) and [C++](cpp.md) in the upcoming sections.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Consuming FXRHandTrackingState in Blueprint
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
Before continuing this section, please ensure you've studied the [Consuming FXRHandTrackingState](./) section, first.
|
||||
|
||||
## Drawing and Animating Virtual Hands
|
||||
|
||||
1. Create a new Virtual Reality project based [the Unreal VR Template](https://dev.epicgames.com/documentation/en-us/unreal-engine/vr-template-in-unreal-engine).
|
||||
|
||||
2. Make sure the [SenseGlove UnrealEngine plugin is installed and enabled](/getting-started/installation.md) inside your new project.
|
||||
|
||||
|
||||
|
||||
3. You could use either hand-tracking or a SenseGlove device as the input data, or both of the inside the same project. Whether you would like to use hand-tracking or a SenseGlove device, please make sure the required steps are taken for each of those first.
|
||||
|
||||
4. You could add the required Blueprint code for drawing virtual hands to either your Level Buleprint or the VRPawn Blueprint Class located at `/Content/VRTemplate/Blueprints/VRPawn`. In this guide we are going to add the code to our VRPawn.
|
||||
|
||||
5. Add a new function named `Draw Hand` with an input parameter of type `EController Hand` named `Hand`.
|
||||
|
||||
|
||||
|
||||
6. Inside this function's event graph add a `Get Hand Tracking State` node from `SenseGlove > Tracking > XR Tracker > Get Hand Tracking State`.
|
||||
|
||||
|
||||
|
||||
7. Then connect the functions `Hand` input parameter to the `Get Hand Tracking State`'s `Hand` input and right-click on the `OutHandTrackingState` parameter and use the `Break XRHandTrackingState` node to break the struct to it's fields.
|
||||
|
||||
|
||||
|
||||
8. After this, we need to perform data validation by checking the return status of the `Get Hand Tracking State` function and `FXRHandTrackingState`'s `Valid` field. Then, we check if the hand-tracking device is being tracked and indeed coming from a hand-tracking source. And, finally, we check whether we have the positions and rotations for exactly `26` joints or not.
|
||||
|
||||
|
||||
|
||||
9. OK, now it's time to draw the joints! If we check out the SenseGlove Debug module's draw option, we notice there are various ways to draw the debug virtual hand. Drawing a cube or a gizmo per joint, or draw the whole hand all at once by passing the retrieved `FXRHandTrackingState` to the `DebugVirtualHand::Draw` function! But, since the point of this tutorial is to learn how to consume the `FXRHandTrackingState` we ignore the last option. Between the debug cubes or gizmos, we are going to choose the gizmos since they better represent the rotations than the cubes.
|
||||
|
||||
|
||||
|
||||
10. In the last step inside the `Draw Hand` function, in order to draw a virtual hand with `26` joints, we have to first iterate through either of the `Hand Key Positions` or `Hand Key Rotations` arrays from the `FXRHandTrackingState` struct. Since we made sure both arrays have `26` elements before we reached this step, it's safe to just iterate over one and use the `Array Index` inside a `For Each Loop` or a `For Loop` to access the position and rotation of every joint. Then we use each array `Get (a ref)` method to access the position and rotation data inside the loop and call the `Draw` function from `SenseGlove > Debug > Gizmo` per every joint. Please note that there are two `Draw` functions and the only difference between the two is that one accepts an `FQuat` and the other a `FRotator` for its `Rotation` input parameter. In this case, we use the `FQuat` variant to avoid an extra conversion to `FRotator`. Also, please adjust the `Thickness` option for the `Settings` parameter from `1.0` to `0.2`, as the default value might be too thick for drawing a joint gizmo.
|
||||
|
||||
|
||||
|
||||
11. Well, now the full implementation for the `Draw Hand` function insde the `VRPawn` should look something like this:
|
||||
|
||||
|
||||
|
||||
12. Finally, go back to `VRPawn`'s event graph and the following code to the `Tick` event. Basically what we do here is call our newly implemented `Draw Hand` twice, once for each hand.
|
||||
|
||||
|
||||
|
||||
13. Now, go back to the `VRTemplateMap` and use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation.
|
||||
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,259 @@
|
||||
# Consuming FXRHandTrackingState in C++
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
Before continuing this section, please ensure you've first studied the [Consuming FXRHandTrackingState](./) section.
|
||||
|
||||
## Drawing and Animating Virtual Hands
|
||||
|
||||
1. Create a new Virtual Reality project based [the Unreal VR Template](https://dev.epicgames.com/documentation/en-us/unreal-engine/vr-template-in-unreal-engine).
|
||||
|
||||
2. Make sure the [SenseGlove UnrealEngine plugin is installed and enabled](/getting-started/installation.md) inside your new project.
|
||||
|
||||
|
||||
|
||||
3. You could use either hand-tracking or a SenseGlove device as the input data, or both of the inside the same project. Whether you would like to use hand-tracking or a SenseGlove device, please make sure the required steps are taken for each of those first.
|
||||
|
||||
4. From the `Tools` menu choose `New C++ class...`.
|
||||
|
||||
|
||||
|
||||
5. Choose the Unreal Engine's `APawn` class as the parent class for the new C++ pawn class.
|
||||
|
||||
|
||||
|
||||
6. Name the new pawn class `DebugPawn`.
|
||||
|
||||
|
||||
|
||||
7. Since we have created a new C++ class, this converts the current Blueprint VRTemplateMap project to a C++ one. That's why the Unreal Editor will give us a few prompts regarding opening the project in the default IDE and rebuilding the code. It might be simpler to just close the editor, then rebuild the source code inside your favorite IDE, and then start the editor with the converted project again.
|
||||
|
||||
8. Find and open the VRPawn Blueprint Class located at `/Content/VRTemplate/Blueprints/VRPawn` inside the Blueprint Editor and from the `File` menu choose the `Reparent Blueprint` class.
|
||||
|
||||
|
||||
|
||||
9. In the new `Reparent blueprint` window choose `DebugPawn` as the new parent.
|
||||
|
||||
|
||||
|
||||
10. By looking at the `Parent Class` label located under the Blueprint Editor window control buttons verify that the `ADebugPawn` class has been set as the new parent.
|
||||
|
||||
|
||||
|
||||
11. Locate the project's main Build file, in our case `VirtualHandCpp/Source/VirtualHandCpp/VirtualHandCpp.Build.cs` and add the `InputDevice`, `OpenXRHMD`, `SenseGloveBuildHacks`, `SenseGloveDebug`, `SenseGloveSettings`, and `SenseGloveTracking` modules as either a private or public dependency.
|
||||
|
||||
```csharp
|
||||
// Fill out your copyright notice in the Description page of Project Settings.
|
||||
|
||||
using UnrealBuildTool;
|
||||
|
||||
public class VirtualHandCpp : ModuleRules
|
||||
{
|
||||
public VirtualHandCpp(ReadOnlyTargetRules Target) : base(Target)
|
||||
{
|
||||
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
|
||||
|
||||
PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore" });
|
||||
|
||||
PrivateDependencyModuleNames.AddRange(new string[]
|
||||
{
|
||||
"InputDevice",
|
||||
"OpenXRHMD",
|
||||
"SenseGloveBuildHacks",
|
||||
"SenseGloveDebug",
|
||||
"SenseGloveSettings",
|
||||
"SenseGloveTracking"
|
||||
});
|
||||
|
||||
// Uncomment if you are using Slate UI
|
||||
// PrivateDependencyModuleNames.AddRange(new string[] { "Slate", "SlateCore" });
|
||||
|
||||
// Uncomment if you are using online features
|
||||
// PrivateDependencyModuleNames.Add("OnlineSubsystem");
|
||||
|
||||
// To include OnlineSubsystemSteam, add it to the plugins section in your uproject file with the Enabled attribute set to true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
12. Locate the C++ header and source file for the `ADebugPawn` inside the project in your C++ IDE. In our case they are located at `VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.h` and `VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.cpp`.
|
||||
|
||||
13. Modify the `DebugPawn.h` header file to look like this:
|
||||
|
||||
```cpp
|
||||
// Fill out your copyright notice in the Description page of Project Settings.
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "CoreMinimal.h"
|
||||
#include "GameFramework/Pawn.h"
|
||||
|
||||
#include "SGSettings/SGDebugGizmoSettings.h"
|
||||
|
||||
#include "DebugPawn.generated.h"
|
||||
|
||||
UCLASS()
|
||||
class VIRTUALHANDCPP_API ADebugPawn : public APawn
|
||||
{
|
||||
GENERATED_BODY()
|
||||
|
||||
private:
|
||||
// The virtual hand drawing settings.
|
||||
UPROPERTY(EditDefaultsOnly, Category="DebugPawn",
|
||||
meta=(AllowPrivateAccess="false"))
|
||||
FSGDebugGizmoSettings HandDrawingSettings;
|
||||
|
||||
public:
|
||||
// Sets default values for this pawn's properties
|
||||
ADebugPawn();
|
||||
|
||||
protected:
|
||||
// Called when the game starts or when spawned
|
||||
virtual void BeginPlay() override;
|
||||
|
||||
public:
|
||||
// Called every frame
|
||||
virtual void Tick(float DeltaTime) override;
|
||||
|
||||
// Called to bind functionality to input
|
||||
virtual void SetupPlayerInputComponent(class UInputComponent* PlayerInputComponent) override;
|
||||
|
||||
private:
|
||||
// The method responsible for drawing a virtual hand.
|
||||
void DrawHand(EControllerHand Hand) const;
|
||||
};
|
||||
```
|
||||
|
||||
14. Modify the `DebugPawn.cpp` implementation file to look like this:
|
||||
|
||||
|
||||
```cpp
|
||||
// Fill out your copyright notice in the Description page of Project Settings.
|
||||
|
||||
|
||||
#include "DebugPawn.h"
|
||||
|
||||
#include "SGDebug/SGDebugGizmo.h"
|
||||
#include "SGTracking/SGXRTracker.h"
|
||||
|
||||
// Sets default values
|
||||
ADebugPawn::ADebugPawn()
|
||||
{
|
||||
// Set this pawn to call Tick() every frame. You can turn this off to improve performance if you don't need it.
|
||||
PrimaryActorTick.bCanEverTick = true;
|
||||
|
||||
// Set the default virtual hand drawing settings.
|
||||
HandDrawingSettings = FSGDebugGizmoSettings{
|
||||
1.0f,
|
||||
FColor{255, 0, 0, 255},
|
||||
FColor{0, 255, 0, 255},
|
||||
FColor{0, 0, 255, 255},
|
||||
false,
|
||||
1.1f,
|
||||
0,
|
||||
0.2f,
|
||||
};
|
||||
}
|
||||
|
||||
// Called when the game starts or when spawned
|
||||
void ADebugPawn::BeginPlay()
|
||||
{
|
||||
Super::BeginPlay();
|
||||
}
|
||||
|
||||
// Called every frame
|
||||
void ADebugPawn::Tick(float DeltaTime)
|
||||
{
|
||||
Super::Tick(DeltaTime);
|
||||
|
||||
// Attempt at drawing the left/right virtual hands every frame.
|
||||
DrawHand(EControllerHand::Left);
|
||||
DrawHand(EControllerHand::Right);
|
||||
}
|
||||
|
||||
// Called to bind functionality to input
|
||||
void ADebugPawn::SetupPlayerInputComponent(UInputComponent* PlayerInputComponent)
|
||||
{
|
||||
Super::SetupPlayerInputComponent(PlayerInputComponent);
|
||||
}
|
||||
|
||||
void ADebugPawn::DrawHand(const EControllerHand Hand) const
|
||||
{
|
||||
// Get the world and cache it, if it's null we return early.
|
||||
UWorld* World{GetWorld()};
|
||||
if (!IsValid(World))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
FXRHandTrackingState HandTrackingState;
|
||||
const bool bGotHandTrackingState = FSGXRTracker::GetHandTrackingState(
|
||||
World, EXRSpaceType::UnrealWorldSpace, Hand, HandTrackingState);
|
||||
|
||||
// Return if the struct data is invalid!
|
||||
if (!bGotHandTrackingState || !HandTrackingState.bValid)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Return if the device is not being tracked!
|
||||
if (HandTrackingState.TrackingStatus == ETrackingStatus::NotTracked)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure that HandTrackingState.HandKeyLocations has the location data
|
||||
// for 26 joints!
|
||||
if (!ensureAlwaysMsgf(HandTrackingState.HandKeyLocations.Num()
|
||||
== EHandKeypointCount,
|
||||
TEXT("Invalid HandKeyLocations count!")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure that HandTrackingState.HandKeyRotations has the rotation data
|
||||
// for 26 joints!
|
||||
if (!ensureAlwaysMsgf(HandTrackingState.HandKeyRotations.Num()
|
||||
== EHandKeypointCount,
|
||||
TEXT("Invalid HandKeyRotations count!")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Iterate over the hand joint locations and rotations!
|
||||
for (int32 JointIndex = 0; JointIndex < EHandKeypointCount; ++JointIndex)
|
||||
{
|
||||
const FVector& JointLocation{
|
||||
HandTrackingState.HandKeyLocations[JointIndex]
|
||||
};
|
||||
const FQuat& JointRotation{
|
||||
HandTrackingState.HandKeyRotations[JointIndex]
|
||||
};
|
||||
|
||||
// Draw a single joint's gizmo!
|
||||
// Please note that we could alternatively:
|
||||
// Use FSGDebugCube::Draw() to draw a cube.
|
||||
// Or use the FSGDebugVirtualHand::Draw() method and pass the
|
||||
// HandTrackingState directly to draw the virtual hand
|
||||
// all at once without iterating the joints. But, that's not
|
||||
// goal of this tutorial.
|
||||
FSGDebugGizmo::Draw(World, JointLocation, JointRotation, HandDrawingSettings);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
15. Now, rebuild the source code and go back to the `VRTemplateMap`, then use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation.
|
||||
|
||||
|
||||
BIN
Binary file not shown.
@@ -1,5 +1,19 @@
|
||||
# Consuming FXRMotionControllerData
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
Taking a closer look at the `FXRMotionControllerData` declaration inside the Unreal Engine's `HeadMountedDisplay` module at `[Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h)`, figuring out the data structure might not seem very straightforward:
|
||||
|
||||
```cpp
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# Consuming FXRMotionControllerData in Blueprint
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
Before continuing this section, please ensure you've studied the [Consuming FXRMotionControllerData](./) section, first.
|
||||
|
||||
## Drawing and Animating Virtual Hands
|
||||
@@ -48,4 +62,4 @@ Before continuing this section, please ensure you've studied the [Consuming FXRM
|
||||
|
||||
13. Now, go back to the `VRTemplateMap` and use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# Consuming FXRMotionControllerData in C++
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Unreal Engine versions `5.2`, `5.3`, and `5.4` are limited to
|
||||
> `FXRMotionControllerData` since at the time of their release no
|
||||
> `FXRHandTrackingState` was available.
|
||||
> Also please keep in mind that, while `FXRMotionControllerData` is pretty much
|
||||
> usable and functional in Unreal Engine `5.5`, it is recommended to utilize
|
||||
> `FXRHandTrackingState` instead. This is because this version of UE has
|
||||
> deprecated `FXRMotionControllerData` in favor of the
|
||||
> `FXRMotionControllerState` and `FXRHandTrackingState` structs. Prior to
|
||||
> version `5.5`, `FXRMotionControllerData` handled both motion controller and
|
||||
> hand tracking data. From `5.5` onward, these responsibilities have been
|
||||
> separated into the two distinct structs, providing clearer and more
|
||||
> specialized handling of each.
|
||||
|
||||
Before continuing this section, please ensure you've first studied the [Consuming FXRMotionControllerData](./) section.
|
||||
|
||||
## Drawing and Animating Virtual Hands
|
||||
@@ -249,4 +263,4 @@ void ADebugPawn::DrawHand(const EControllerHand Hand) const
|
||||
|
||||
15. Now, rebuild the source code and go back to the `VRTemplateMap`, then use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -39,6 +39,12 @@ This is a minor release with some breaking API and ABI changes, focusing mainly
|
||||
- The hand interaction manipulation on UE `5.5+` has been revamped to utilize `FXRHandTrackingState`.
|
||||
- The virtual hand debugging system on UE `5.5+` has been revamped to utilize `FXRHandTrackingState`.
|
||||
|
||||
### Documentation
|
||||
|
||||
- Added the documentation on consuming the `FXRHandTrackingState` struct in both Blueprint and C++.
|
||||
- Updated the documentation on consuming the `FXRMotionControllerData` struct.
|
||||
- Additional minor documentation fixes and improvements that may not be listed here.
|
||||
|
||||
## [2.1.4] - 2024-10-22
|
||||
|
||||
This is a bugfix release that delivers some documentation fixes.
|
||||
|
||||
@@ -65,6 +65,9 @@ For users familiar with the basics, this section explores advanced features of t
|
||||
|
||||
- [Safe Glove Access in Blueprint](../advanced-topics/safe-glove-access-blueprint/)
|
||||
- [OpenXR](../advanced-topics/openxr/)
|
||||
- [Consuming FXRHandTrackingState](../advanced-topics/openxr/consuming-fxrhandtrackingstate/)
|
||||
- [Blueprint](../advanced-topics/openxr/consuming-fxrhandtrackingstate/blueprint.md)
|
||||
- [C++](../advanced-topics/openxr/consuming-fxrhandtrackingstate/cpp.md)
|
||||
- [Consuming FXRMotionControllerData](../advanced-topics/openxr/consuming-fxrmotioncontrollerdata/)
|
||||
- [Blueprint](../advanced-topics/openxr/consuming-fxrmotioncontrollerdata/blueprint.md)
|
||||
- [C++](../advanced-topics/openxr/consuming-fxrmotioncontrollerdata/cpp.md)
|
||||
|
||||
Reference in New Issue
Block a user