add the content for the Consuming FXRMotionControllerData section
This commit is contained in:
Mamadou Babaei
@@ -0,0 +1,9 @@
|
||||
# OpenXR
|
||||
|
||||
The SenseGlove Unreal Engine Plugin has provided OpenXR-compatible hand tracking by implementing <code>XR_EXT_hand_tracking</code> since <code>v2.1.0</code>.
|
||||
|
||||
Typically a user does not need to know anything about OpenXR to use the plugin, so this section of the handbook is for advanced users who are looking for a way to directly consume the OpenXR data coming directly from either a SenseGlove device or if enabled in the plugin settings from hand-tracking.
|
||||
|
||||
Since the SenseGlove Unreal Engine Plugin registers itself as an <code>OpenXRHandTracking</code> motion controller device it becomes a hand-tracking provider for Unreal Engine, thus the OpenXR data from SenseGlove could always be retrieved from the Unreal Engine's <code>IXTrackingSystem</code> with one caveat. The caveat is if another OpenXR-compatible hand-tracking plugin, e.g. Epic's own OpenXRHandTracking, is enabled simultaneously it's not guaranteed that the <code>FXRMotionControllerData</code> retrieved from the <code>IXTrackingSystem::GetMotionControllerData()</code> method is coming from SenseGlove, as this method returns the first hand-tracking plugin it could find. Thus, SenseGlove provides its own implementation of <codE>GetMotionControllerData()</code> which guarantees the retrieved <code>FXRMotionControllerData</code> is coming from the SenseGlove Unreal Engine Plugin; and this is the preferred way to that.
|
||||
|
||||
In the next sections we'll see [how we can directly consume the <code>FXRMotionControllerData</code>](consuming-fxrmotioncontrollerdata/README.md) to draw debug virtual hands in both [Blueprint](consuming-fxrmotioncontrollerdata/blueprint.md) and [C++](consuming-fxrmotioncontrollerdata/cpp.md).
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
# Consuming FXRMotionControllerData
|
||||
|
||||
Taking a closer look at the <code>FXRMotionControllerData</code> declaration inside the Unreal Engine's <code>HeadMountedDisplay</code> module at <code>Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h</code>, figuring out the data structure might not seem very straightforward:
|
||||
|
||||
```cpp
|
||||
USTRUCT(BlueprintType)
|
||||
struct FXRMotionControllerData
|
||||
{
|
||||
GENERATED_USTRUCT_BODY();
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
bool bValid = false;
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FName DeviceName;
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FGuid ApplicationInstanceID;
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
EXRVisualType DeviceVisualType = EXRVisualType::Controller;
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
EControllerHand HandIndex = EControllerHand::Left;
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
ETrackingStatus TrackingStatus = ETrackingStatus::NotTracked;
|
||||
|
||||
// Vector representing an object being held in the player's hand
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FVector GripPosition = FVector(0.0f);
|
||||
// Quaternion representing an object being held in the player's hand
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FQuat GripRotation = FQuat(EForceInit::ForceInitToZero);
|
||||
|
||||
// For handheld controllers, gives a vector for pointing at objects
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FVector AimPosition = FVector(0.0f);
|
||||
// For handheld controllers, gives a quaternion for pointing at objects
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FQuat AimRotation = FQuat(EForceInit::ForceInitToZero);
|
||||
|
||||
// For handheld controllers, gives a vector for representing the hand
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FVector PalmPosition = FVector(0.0f);
|
||||
// For handheld controllers, gives a quaternion for representing the hand
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
FQuat PalmRotation = FQuat(EForceInit::ForceInitToZero);
|
||||
|
||||
// The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
TArray<FVector> HandKeyPositions;
|
||||
// The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
TArray<FQuat> HandKeyRotations;
|
||||
// The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
TArray<float> HandKeyRadii;
|
||||
|
||||
UPROPERTY(BlueprintReadOnly, Category = "XR")
|
||||
bool bIsGrasped = false;
|
||||
};
|
||||
```
|
||||
|
||||
Which on the Blueprint side it looks like this:
|
||||
|
||||
|
||||
|
||||
But, fear not, we got you covered!
|
||||
|
||||
## FXRMotionControllerData in Unreal Engine
|
||||
|
||||
`FXRMotionControllerData` is a structure in Unreal Engine designed to hold detailed information about the state of a motion controller device at a given moment. This structure is essential for handling motion controller inputs in virtual reality (VR) applications, providing the necessary data to accurately track and represent the user's hand movements and actions within the virtual environment.
|
||||
|
||||
### Structure Members of FXRMotionControllerData
|
||||
|
||||
1. **bValid**
|
||||
- **Description**: A boolean flag indicating whether the data is valid or not.
|
||||
- **Usage**: This is used to check if the motion controller data is correctly initialized and can be used for further processing.
|
||||
|
||||
2. **DeviceName**
|
||||
- **Type**: `FName`
|
||||
- **Description**: The name of the device.
|
||||
- **Usage**: Identifies which motion controller device the data is coming from, useful when multiple devices are in use.
|
||||
|
||||
3. **ApplicationInstanceID**
|
||||
- **Type**: `FString`
|
||||
- **Description**: A unique identifier for the application instance.
|
||||
- **Usage**: Helps in differentiating data from different instances of an application, ensuring the correct instance processes the data.
|
||||
|
||||
4. **TrackingStatus**
|
||||
- **Type**: `EXRTrackingStatus`
|
||||
- **Description**: Enum indicating the tracking status of the motion controller.
|
||||
- **Usage**: Shows whether the controller is being tracked accurately, with possible statuses like `Tracked`, `NotTracked`, etc.
|
||||
|
||||
5. **GripPosition**
|
||||
- **Type**: `FVector`
|
||||
- **Description**: The position of the grip in world coordinates.
|
||||
- **Usage**: Provides the 3D coordinates of the controller's grip, essential for positioning the virtual representation of the controller.
|
||||
|
||||
6. **GripRotation**
|
||||
- **Type**: `FQuat`
|
||||
- **Description**: The rotation of the grip in world coordinates.
|
||||
- **Usage**: Provides the orientation of the controller's grip, allowing for accurate rotation and alignment in the virtual space.
|
||||
|
||||
7. **AimPosition**
|
||||
- **Type**: `FVector`
|
||||
- **Description**: The position of the aim point in world coordinates.
|
||||
- **Usage**: Specifies where the controller is aiming, useful for aiming or pointing actions.
|
||||
|
||||
8. **AimRotation**
|
||||
- **Type**: `FQuat`
|
||||
- **Description**: The rotation of the aim point in world coordinates.
|
||||
- **Usage**: Determines the orientation of the aim direction, important for actions like shooting or selecting objects in VR.
|
||||
|
||||
9. **HandKeyPositions**
|
||||
- **Type**: `TArray<FVector>`
|
||||
- **Description**: An array of vectors representing key positions of the hand.
|
||||
- **Usage**: Provides detailed positions of key points on the hand, useful for precise hand tracking and interaction.
|
||||
|
||||
10. **HandKeyRotations**
|
||||
- **Type**: `TArray<FQuat>`
|
||||
- **Description**: An array of quaternions representing key rotations of the hand.
|
||||
- **Usage**: Complements the hand key positions with rotational data, ensuring accurate representation of hand movements.
|
||||
|
||||
11. **HandKeyRadii**
|
||||
- **Type**: `TArray<float>`
|
||||
- **Description**: An array of floats representing the radii of key points of the hand.
|
||||
- **Usage**: Gives the size of the hand key points, aiding in collision detection and interaction fidelity.
|
||||
|
||||
12. **bIsGrasped**
|
||||
- **Type**: `bool`
|
||||
- **Description**: A boolean indicating whether the controller is currently grasping an object.
|
||||
- **Usage**: Determines if the user is holding something, affecting interactions and animations.
|
||||
|
||||
### Organization of FXRMotionControllerData
|
||||
|
||||
The structure is organized to encapsulate all relevant data needed for motion controller tracking in a coherent and accessible manner. Boolean flags (`bValid` and `bIsGrasped`) provide quick checks on the state of the controller data. Identifiers (`DeviceName` and `ApplicationInstanceID`) ensure the correct association of data. Positional and rotational data (`GripPosition`, `GripRotation`, `AimPosition`, and `AimRotation`) offer precise tracking of the controller's movement. Arrays (`HandKeyPositions`, `HandKeyRotations`, and `HandKeyRadii`) allow detailed hand tracking, which is critical for immersive VR experiences. Lastly, the tracking status (`TrackingStatus`) informs the system of the reliability of the data being processed and whether the motion controller is actively being tracked or it's inactive at the moment.
|
||||
|
||||
### Processing the Data for Drawing and Animating a Virtual Hand
|
||||
|
||||
In order to draw and animate a virtual hand in real-time whether the data is coming from hand-tracking or a SenseGlove device, we could consume the data from the <code>HandKeyPositions</code> and <code>HandKeyRotations</code> fields of the <code>FXRMotionControllerData</code> struct.
|
||||
|
||||
Both <code>HandKeyPositions</code> and <code>HandKeyRotations</code> contain 26 elements as defined by OpenXR's [<code>XR_HAND_JOINT_COUNT_EXT</code>](https://registry.khronos.org/OpenXR/specs/1.1/man/html/XR_HAND_JOINT_COUNT_EXT.html) and [<code>XrHandJointLocationsEXT</code>](https://registry.khronos.org/OpenXR/specs/1.0/man/html/XrHandJointLocationsEXT.html), etc.
|
||||
|
||||
Unreal Engine also provides an enum called <code>EHandKeypoint</code> naming the 26 joints, and the equivalent of <code>XR_HAND_JOINT_COUNT_EXT</code> as <code>EHandKeypointCount</code> inside <code>Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h</code> as follows:
|
||||
|
||||
```cpp
|
||||
/**
|
||||
* Transforms that are tracked on the hand.
|
||||
* Matches the enums from WMR to make it a direct mapping
|
||||
*/
|
||||
UENUM(BlueprintType)
|
||||
enum class EHandKeypoint : uint8
|
||||
{
|
||||
Palm,
|
||||
Wrist,
|
||||
ThumbMetacarpal,
|
||||
ThumbProximal,
|
||||
ThumbDistal,
|
||||
ThumbTip,
|
||||
IndexMetacarpal,
|
||||
IndexProximal,
|
||||
IndexIntermediate,
|
||||
IndexDistal,
|
||||
IndexTip,
|
||||
MiddleMetacarpal,
|
||||
MiddleProximal,
|
||||
MiddleIntermediate,
|
||||
MiddleDistal,
|
||||
MiddleTip,
|
||||
RingMetacarpal,
|
||||
RingProximal,
|
||||
RingIntermediate,
|
||||
RingDistal,
|
||||
RingTip,
|
||||
LittleMetacarpal,
|
||||
LittleProximal,
|
||||
LittleIntermediate,
|
||||
LittleDistal,
|
||||
LittleTip
|
||||
};
|
||||
|
||||
const int32 EHandKeypointCount = static_cast<int32>(EHandKeypoint::LittleTip) + 1;
|
||||
|
||||
```
|
||||
|
||||
So, getting the any joint's position or rotation is as easy as casting the enum value and passing it as the array index.
|
||||
|
||||
```cpp
|
||||
FXRMotionControllerData MotionControllerData;
|
||||
const bool bGotMotionControllerData = FSGXRTracker::GetMotionControllerData(
|
||||
this, EControllerHand::Left, MotionControllerData);
|
||||
|
||||
// Return if the struct data is invalid!
|
||||
if (!bGotMotionControllerData || !MotionControllerData.bValid)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Return if device is not being tracked!
|
||||
if (MotionControllerData.TrackingStatus == ETrackingStatus::NotTracked)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure that MotionControllerData.DeviceVisualType is a hand!
|
||||
if (!ensureAlwaysMsgf(MotionControllerData.DeviceVisualType
|
||||
== EXRVisualType::Hand,
|
||||
TEXT("Invalid DeviceVisualType type!")))
|
||||
{
|
||||
}
|
||||
|
||||
// Ensure that MotionControllerData.HandKeyPositions has the location data
|
||||
// for 26 joints!
|
||||
if (!ensureAlwaysMsgf(MotionControllerData.HandKeyPositions.Num()
|
||||
== EHandKeypointCount,
|
||||
TEXT("Invalid HandKeyPositions count!")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure that MotionControllerData.HandKeyPositions has the location data
|
||||
// for 26 joints!
|
||||
if (!ensureAlwaysMsgf(MotionControllerData.HandKeyRotations.Num()
|
||||
== EHandKeypointCount,
|
||||
TEXT("Invalid HandKeyRotations count!")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
static constexpr int32 PalmIndex = static_cast<int32>(EHandKeypoint::Palm);
|
||||
|
||||
const FVector& PalmPosition{
|
||||
MotionControllerData.HandKeyPositions[PalmIndex]
|
||||
};
|
||||
const FRotator& PalmRotation{
|
||||
MotionControllerData.HandKeyRotations[PalmIndex].Rotator()
|
||||
};
|
||||
```
|
||||
|
||||
|
||||
|
||||
OK, now that we've got a glimpse of how the virtual hand's joint data could be processed we are going to draw and animate a virtual hand in both [Blueprint](blueprint.md) and [C++](cpp.md) in the upcoming sections.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
# Consuming FXRMotionControllerData in Blueprint
|
||||
|
||||
Before continuing this section, please ensure you've first studied the [Consuming FXRMotionControllerData](README.md) section.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# Consuming FXRMotionControllerData in C++
|
||||
|
||||
Before continuing this section, please ensure you've first studied the [Consuming FXRMotionControllerData](README.md) section.
|
||||
|
||||
|
||||
BIN
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user