This plugin provides foundational components and utilities for simulating robotic assistive mobility devices and manipulation tasks within Unreal Engine 5. It supports powered wheelchairs, robotic arms, grippers, and accessible environment interactions with realistic physics, sensor simulation, and control interfaces.
Designed as a plugin for UE5, it can be added to any project by cloning or
adding it as a git submodule within your project's Plugins/ directory.
For an example project, see the RAMMS Sim Project.
Table of Contents
- Robotic Assistive Mobility and Manipulation Simulation (RAMMS) Core Plugin
- Clone or add this repository as a git submodule in
Plugins/. - Open your UE5 project. The plugin is automatically detected and compiled.
- Enable the plugin in Edit > Plugins if not already enabled.
- Add the desired controller component to your actor (e.g.
UKinovaGen3ControllerComponentfor a robotic arm).
If you want to contribute or customize the plugin, you can clone the repository and set it up in your Unreal project. If you want to make changes to the code, please follow the code style guidelines below.
- Ensure
clang-formatis installed - Ensure pre-commit is installed
- Set up
pre-commitfor this repository:
pre-commit installThis helps ensure that consistent code formatting is applied.
| Component | Description |
|---|---|
URammsDifferentialDriveController |
Differential drive controller for powered wheelchair simulation |
URammsDifferentialDriveLibrary |
Blueprint function library with differential drive helpers |
URammpGen1ControllerComponent |
RAMMP Gen1 wheelchair base with linear and angular actuator control |
| Component | Description |
|---|---|
UKinovaGen3ControllerComponent |
Physics-based 7-DOF robotic arm controller with IK solvers |
UGripperControllerComponent |
Gripper controller for robotic end effectors |
URammsIKLibrary |
Static Blueprint function library exposing FK and IK solvers |
| Component | Description |
|---|---|
UVanRampComponent |
Accessible van ramp with animation control |
UVanDoorComponent |
Accessible door interaction simulation |
| Component | Description |
|---|---|
URammsToFSensorComponent |
Time-of-Flight distance sensor (single-point or NxM grid) with GPU ray tracing |
URammsSonarSensorComponent |
Sonar / RADAR cone-beam distance sensor with GPU ray tracing |
URammsIMUSensorComponent |
Inertial measurement unit (accelerometer, gyroscope, orientation) |
URammsSensorRigComponent |
A robot's sensors as data (FRammsSensorMount list): builds ToF / Sonar / IMU components and streamed cameras at BeginPlay |
URammsSensingBridge |
Remote Control JSON bridge: add / list / read sensors, start camera streams over RMSS |
UKinovaGen3ControllerComponent is the primary controller for articulated
robotic arms. It reads joint angles from physics constraints on a
SkeletalMesh, runs IK solvers to compute target angles, and drives the
constraints to reach end-effector target poses.
| Property | Default | Description |
|---|---|---|
SkeletalMeshComponentName |
"ArmSkMesh" |
Name of the SkeletalMeshComponent on the owning actor |
EndEffectorBoneName |
"end_effector" |
Bone or socket name at the arm tip |
bAutoPopulateOnSkeletalMeshChange |
true |
Auto-detect joints from the skeleton |
| Property | Options | Description |
|---|---|---|
ArmControlMode |
JointControl, EndEffectorControl |
Direct joint angles vs. IK-driven end-effector targeting |
ControlMode |
PositionControl, VelocityControl, TorqueControl |
How joint targets are applied to physics constraints |
TargetActor |
(Actor reference) | Actor whose world transform is the IK target. If unset, uses TargetEndEffectorTransform |
The Joints array contains one entry per revolute (hinge) joint:
| Property | Default | Description |
|---|---|---|
BoneName |
— | Skeleton bone this joint corresponds to |
ConstraintName |
— | Physics constraint name (defaults to BoneName if unset) |
ControlledAxis |
Twist |
Constraint axis actuated: Twist (X), Swing1 (Y), or Swing2 (Z) |
bInvertAxisForIK |
false |
Flip the axis direction if it points the wrong way |
AngleOffset |
0.0 |
Calibration offset (deg) between constraint zero and reference pose |
MinAngleLimit / MaxAngleLimit |
-180 / 180 |
Joint limits in degrees |
MaxAngularSpeed |
90.0 |
Maximum angular velocity (deg/s) |
SpeedMultiplier |
1.0 |
Scale factor (0–1) on MaxAngularSpeed |
MaxTorque |
39.0 |
Maximum torque (N·m) for torque-controlled joints |
PositionStrength |
5000000.0 |
PD position gain for position-controlled joints |
PositionDamping |
0.0 |
PD damping gain |
One component per robot that gathers every IRammsControlContributor on the
actor (drive modes, linkages, seat, arm teleop, cameras, curb crossings...)
and the registry motors nobody claims (motor.<id>) into one list of
controls, and routes commands to them by id: SetControl, TriggerControl,
ReleaseControl (Blueprint and Remote Control), or the IRammsControlSink
interface (input components, RammsAccess, panels). Control kinds:
Continuous (a rate held while the input is held), Position and Velocity
(targets), Enum, Action and Text.
Every command names its source (ERammsControlSource). Autonomy outranks
Remote, which outranks local input (Keyboard, Gamepad, Touch), which outranks
Script. A Remote or Autonomy command holds its control over lower sources for
ExternalHoldSeconds (0.3 s) after it was last sent; an equal or lower
source yields at once. The source is whatever the caller passes: a Remote
Control client can send as Autonomy.
The robot's own controllers command through SetControlInternal /
TriggerControlInternal (the stance a drive mode takes, a curb crossing
holding the arm still). They are arbitrated and held like any command, but
they are not a client: the deadman below neither watches them nor counts them
as their source being there.
A Continuous or Velocity control held by a Remote or Autonomy source springs
back once nothing has been heard from that source on the surface for
DeadmanSeconds (0.3 s) plus a grace of the last two frames:
- Heard means any command from that source the surface applied (a set, a trigger, a release, of any control). It is per source, not per control: a client that sends its controls one request at a time is not taken for gone between them, and a control it set once stands while it keeps sending others. Remote Control carries no client identity, so two clients sending as the same source keep each other's commands alive.
- A Continuous control goes back as a release from its source would: through
its contributor's
ReleaseControl, orApplyControl(DefaultValue)where it cannot let go; a raw torque motor is commanded its default. A latched command (a drive's external input) goes back with it. - A Velocity control (a raw
motor.<id>of a velocity actuator, in a robot's low-level mode) is commanded its default -- 0, stop -- rather than released: a released speed servo may coast, and a MuJoCo<velocity>actuator cannot let go at all. - Either way the hold goes, so local input takes over at once.
- Not touched: Keyboard, Gamepad, Touch and Script commands; the robot's own; Position, Enum, Action and Text controls; a control another source has taken over; one a drive-mode switch moved to another contributor (the mode that stood down zeroed its own command).
- A drive-mode switch carries held drive commands into the incoming mode as
of when they were sent (
ReapplyHeldControl): a client that went quiet before the switch gets no fresh lease from it.
The deadman runs on the core ticker, only while it watches something, on
real time (FPlatformTime): it springs back while the world is paused, and
a client re-sending at 10 Hz keeps to it however the world is dilated. The
grace of the last two frames' length means a long frame (an editor throttled
to a few frames a second in the background, a hitch) is not taken for
silence.
DeadmanSeconds is a class default (EditDefaultsOnly), not
Blueprint-visible, so neither a Blueprint graph nor Remote Control's property
route can change it at run time. ExternalHoldSeconds stays
Blueprint-writable; both are used within 0.1 to 2 s, whatever they hold.
Reports go to LogRammsControlSurface: at Log (a refusal at Warning) at most
every 5 s per surface, in between at Verbose.
Clients: re-send held commands at 10 Hz or more and release them when done
(ramms-tools' hold, keep_alive and drive(seconds=) do); RammsAccess
re-sends every tick and zeroes on its own 250 ms watchdog first. Tests:
Ramms.Control.Deadman.* (RammsCoreEditor).
Simulated Time-of-Flight distance sensor modeled after devices like the VL53L0X (single point) and VL53L5CX (8×8 zone grid). The sensor fires along its local +X axis and reports distance readings in centimeters.
| Property | Default | Description |
|---|---|---|
SensorMode |
SinglePoint |
SinglePoint or Grid (NxM zones) |
MaxRange |
400.0 cm |
Maximum detection range |
MinRange |
1.0 cm |
Minimum detection range |
HorizontalFOV |
45.0° |
Horizontal field of view (Grid mode) |
VerticalFOV |
45.0° |
Vertical field of view (Grid mode) |
GridRows |
8 |
Number of grid rows (Grid mode) |
GridColumns |
8 |
Number of grid columns (Grid mode) |
UpdateRateHz |
15.0 |
Measurement rate (0 = every tick) |
TraceChannel |
Visibility |
Collision channel to trace against |
bIgnoreOwner |
true |
Ignore owning actor in traces |
bUseGPURayTracing |
true |
Use GPU compute shader when available |
DistanceNoiseStdDev |
0.0 cm |
Gaussian noise standard deviation |
Output (FToFSensorData):
Distances—TArray<float>of distance readings (row-major, -1 = no detection)NumRows/NumColumns— Grid dimensions (1×1 for single point)bAnyHit/MinDistance— Aggregate hit informationTimestamp— Seconds since game start
Blueprint API:
GetToFData()— Latest measurementGetDistanceAt(Row, Column)— Single cell distanceMeasureNow()— Immediate synchronous measurement (CPU path)OnToFDataUpdated— Delegate fired each measurement cycle
Simulated sonar / RADAR distance sensor with a cone-shaped beam approximated by multiple rays distributed in a golden-angle spiral. Fires along +X and reports the distance to the nearest obstacle.
| Property | Default | Description |
|---|---|---|
MaxRange |
400.0 cm |
Maximum detection range |
MinRange |
2.0 cm |
Minimum detection range |
BeamHalfAngle |
15.0° |
Half-angle of cone (total width = 2×) |
NumRays |
7 |
Number of rays approximating the cone |
UpdateRateHz |
40.0 |
Measurement rate |
bUseGPURayTracing |
true |
Use GPU compute shader when available |
DistanceNoiseStdDev |
0.0 cm |
Gaussian noise standard deviation |
Output (FSonarSensorData):
Distance— Closest obstacle distance (cm, -1 if none)bHit— Whether any obstacle was detectedHitLocation/HitNormal— World-space impact point and surface normal (CPU path returns true geometric normal; GPU path returns an approximate normal derived from the ray/triangle front-face direction)HitActor— The actor hit (CPU path only; null on GPU path)
Blueprint API:
GetSonarData()— Latest measurementMeasureNow()— Immediate synchronous measurement (CPU path)OnSonarDataUpdated— Delegate fired each measurement cycle
Note: The GPU ray tracing path cannot resolve
HitActor. If you need actor identification, disablebUseGPURayTracingor perform a supplementary CPU trace from theOnSonarDataUpdateddelegate.
Simulated inertial measurement unit measuring linear acceleration, angular velocity, and orientation. Attaches to any bone or socket and uses the component's world transform as the sensor frame.
| Property | Default | Description |
|---|---|---|
bIncludeGravity |
true |
Include gravitational acceleration in output |
Source |
Auto |
Where readings come from: Auto (the engine that simulates the body the sensor rides on), Chaos, MuJoCo, Kinematic (see below) |
UpdateRateHz |
100.0 |
Measurement rate |
bUsePhysicsVelocity |
true |
Use physics velocity (avoids double-differentiation jitter); off under Auto means Kinematic |
AccelSmoothingFactor |
0.0 |
EMA filter for accelerometer (0 = none, 1 = max) |
GyroSmoothingFactor |
0.0 |
EMA filter for gyroscope |
AccelDeadBand |
0.0 cm/s² |
Accelerometer dead-band threshold |
GyroDeadBand |
0.0 °/s |
Gyroscope dead-band threshold |
AccelNoiseStdDev |
0.0 cm/s² |
Accelerometer Gaussian noise |
GyroNoiseStdDev |
0.0 °/s |
Gyroscope Gaussian noise |
OrientationNoiseStdDev |
0.0° |
Orientation Gaussian noise |
AccelBias |
(0,0,0) |
Constant accelerometer bias (sensor-local, cm/s²) |
GyroBias |
(0,0,0) |
Constant gyroscope bias (sensor-local, °/s) |
Output (FIMUSensorData):
LinearAcceleration— Sensor-local frame (cm/s²)AngularVelocity— Sensor-local frame (°/s)Orientation— World-space quaternionTimestamp— Seconds since game startSource— The engine the reading came from:Chaos/MuJoCo(physics rate) orKinematicSimTimeSeconds— Chaos / MuJoCo: the engine's time of the state the reading ends at (0 forKinematic)
Blueprint API:
GetIMUData()— Latest measurementGetActiveSource()— The source of the latest measurementOnIMUDataUpdated— Delegate fired each measurement cycle
Sources. Under Auto (with bUsePhysicsVelocity) the sensor asks
URammsPhysicsImuSubsystem for its frame: MuJoCo for a URLab body (or
anything attached below one), Chaos for a simulating primitive or skeletal
bone, else Kinematic. With an engine's samples a measurement covers the
whole interval since the previous one, as a real IMU's filtered output
does: the accelerometer reads the point's velocity change over the interval
divided by the engine time it spans (gravity then added as an accelerometer
feels it), the gyroscope the mean angular velocity (the turn over the
interval divided by its time), both in the sensor frame halfway through the
turn. No new engine step since the last measurement repeats it, as does a
moment (up to a second) in which the engine has no sample (its body handed a
new physics proxy, its scene recompiled). Kinematic
differences the component's transform at the game rate (over the time since
the last measurement). Filters, dead-bands, noise, bias and bIncludeGravity
apply the same way to every source.
A world subsystem that tracks points on bodies and keeps the physics
engine's sample of each, every physics step: the IMU component's source, and
the sensing bridge's for bodies that carry no IMU component. Everything is in
world space and Unreal units at the point (FRammsImuSample: location cm,
orientation, velocity cm/s, angular velocity °/s, proper acceleration cm/s²
— what an accelerometer reads, +|g| up at rest, about 0 in free fall —,
angular acceleration °/s², the gravity the engine applies to that body, and
the engine time of the state).
TrackPose(Component, Bone, LocalPose, Preferred, IdleTimeoutSeconds)/Track(...),GetSample,Untrack,GetTrackedSource,GetTrackedBodyName,GetTrackedBody.EnableHistory(Handle)keeps every physics step's sample in a ring (HistoryCapacity= 1024, over 2 s of MuJoCo's 500 Hz);ReadHistory(Handle, Since, Max, Out)returns those from a sequence number on (Since< 0: none, just the cursor), the next cursor and how many were dropped. The sequence goes on when a point moves to another engine. The ring keeps complete samples only: a Kinematic point's first two frames (a pose, then a velocity) and an engine's first step on a body (or on its new physics proxy) carry no acceleration and are left out, so every row has a measured acceleration.- At most
MaxTrackedPoints= 64 points per world (IMU components and bridge reads together); more are refused. The sensing bridge (TrackBridgePose) may hold at mostMaxBridgePoints= 32 of them, so a Remote Control client that leaves points behind cannot take the robots' own IMU components'. A refusal is logged once, not on every retry, until a point is tracked again. - Every point is resolved again each second, and at once when its provider reports its body gone: a robot whose physics comes up later, that is handed to another engine (a URLab robot switched to Chaos), or a component moved onto another body follows.
Engines plug in as IRammsImuProviders through RammsImuProviders::Register:
| Provider | Module | Samples |
|---|---|---|
| Chaos | RammsCore | After every solver step (a sim callback's PostSolve): the body's velocity at its centre of mass and angular velocity, accelerations as their change over the step (the step that ends at the sample), the lever arm to the point. Gravity is the solver's for that body (its gravity group's; none when its gravity is off). Stamped with the step's end. A body whose physics state is recreated is followed to its new proxy on the next read (a sample or a history read). |
| MuJoCo | RammsMujocoSupport | After every URLab step: mj_rnePostConstraint, then mj_objectVelocity / mj_objectAcceleration of the body frame and the lever arm. By default the acceleration the integrator applied, (qvel after − qvel before) / timestep, over the step that starts at the sample (a step ahead of Chaos' in phase); ramms.Imu.MuJoCoAcceleration 1 reports MuJoCo's own accelerometer convention, the forward pass's qacc, instead. Stamped with the step's start (the state its forward pass computed; RK4: the end, its last stage's state). Serves a body only while its compiled scene has it: until the scene compiles, when it fails to, or once a recompile leaves the body out, the point is Kinematic. A body Unreal poses (mocap) is left to Kinematic. |
| Kinematic | RammsCore | The point's frame differenced at the game rate. |
Over Remote Control (URammsSensingBridge, Private/RammsSensingBridgeImu.cpp;
accelerations read as an accelerometer does, sensor frame; velocity and
orientation world):
ReadBodyImuJson(Actor, Body, Options)— a reading of any body (a MuJoCo body, a component, a bone;""the actor's main body), optionsoffset,rotation,source.sim_timeonly when there is a sample.ListImuBodiesJson(Actor)— the bodies an engine simulates, and which one""reads.ReadImuSamplesJson(Actor, Target, Options)— every physics step's sample since a cursor:Targetan IMU sensor's name or a body; options as above plussince(omitted or -1: start now) andmax; returnssamples(seq,sim_time,linear_acceleration,angular_velocity,angular_acceleration,linear_velocity,orientation),next,dropped,rate_hz.StartImuStreamJson(Actor, Target, Options)/StopImuStream(Actor, Target)— the same samples streamed over RMSS (play worlds only; starts the server). Options addportandchannel(-1: the lowest from 1000 that no other IMU stream and no camera stream source uses; a channel a camera source uses is refused). Each game tick with new samples sends oneFrameDatamessage, fmtimu_f64x18, w = samples, h = 1, 18 little-endian float64 per sample: seq, sim_time, linear acceleration xyz (cm/s²), angular velocity xyz (°/s), angular acceleration xyz (°/s²), linear velocity xyz (cm/s, world), orientation xyzw (world). Stream idimu/<actor name>/<target>; the metadata names the fields, units, source, actor, target, body,first_seq,count,dropped.GetStreamStatusJsonlists IMU streams with roleimu.
Points the bridge tracks are dropped after 10 s without a read; it holds at
most 32 (MaxBridgePoints) of a world's 64. At most 8 IMU streams run per
world (URammsImuStreamSubsystem::MaxStreams; a restart takes its own
place); a stream no RMSS client has subscribed to (its channel, or every
channel) for 30 s stops by itself, and nothing is built for a channel nobody
wants. The stream's port is clamped to 1..65535. Body and target names are
at most 256 characters.
A list of FRammsSensorMounts (name, kind, parent component, socket,
relative location / rotation, and the kind's settings), built into real
components at BeginPlay and at runtime through AddMount (which replaces a
mount of the same name, or changes nothing when the new one cannot be built) /
RemoveMount. Every sensor traces / looks along its local +X. A mount whose
parent component does not exist is skipped with a warning. Names are 1 to 64
characters of A-Z a-z 0-9 _ - (no .: it separates the parts of an object
path) and may not end in _depth, _dmv, _seg or _stream: those are the
names of a camera's capture passes and stream source, and a camera whose pass
names another object already holds is not built (creating a pass over another
object of its name would be fatal). A mount's location, rotation, fields of
view and ranges must be finite, its location at most 1 km from the parent
along each axis; ranges are clamped to 1 km.
| Kind | Builds |
|---|---|
ToF |
URammsToFSensorComponent (grid when Rows x Columns > 1) |
Sonar |
URammsSonarSensorComponent (SonarConeDeg is BeamHalfAngle, the cone's half-angle) |
IMU |
URammsIMUSensorComponent |
Camera |
UIntrinsicSceneCaptureComponent2D (1 to 4096 pixels each way), plus a URammsStreamSourceComponent <Name>_stream forwarding only that camera |
A camera with base channel C (StreamChannel, or -1 for the lowest C
none of whose four channels a stream source or an IMU stream in the world
uses) streams colour on C, depth on C+100, motion on C+200 and the
segmentation mask on C+300, under the stream ids camera/<Name>/color,
camera/<Name>/depth, motion/<Name>, mask/<Name> (see RammsStreaming's
RammsCameraStreams.h). Cameras that ask for a channel are built first; one
whose channels another source or an IMU stream already uses streams on a free
one instead (with a warning). An IMU stream, likewise, never takes a camera's
channel (Private/RammsStreamChannels.h keeps both apart). A camera replaced
through AddMount keeps its channel. Names are per actor, so every frame
carries actor (the owner's path): a client picks a robot's camera by actor,
never by name alone.
A rig camera costs nothing until camera streams start: it is registered with
UCameraCaptureSubsystem (render targets, depth / motion / mask passes) by
URammsSensingBridge::StartCameraStreams (or bStartCaptureOnBeginPlay), at
once when added while the streams are on, and unregistered by
StopCameraStreams; a capture session somebody else runs does not take rig
cameras in. bStartStreamServerOnBeginPlay / bStartCaptureOnBeginPlay
start the RMSS server and camera streams, turning on the channels the rig's
cameras ask for; CaptureEveryNFrames and bSaveCaptureToDisk apply only to
a capture session the rig starts.
URammsSensingBridge exposes the same to Remote Control
(/Script/RammsCore.Default__RammsSensingBridge), plain strings in and JSON
out: ListSensorsJson, AddSensorJson (snake_case mount JSON),
RemoveSensor, ReadSensorJson, ReadSensorsJson, StartCameraStreamsJson,
StopCameraStreams, GetStreamStatusJson, GetSegmentationClassesJson.
Remote Control is reachable from the network, so the bridge is bounded: actor
paths of at most 1023 characters made of names that already exist (looked up,
never added to the name table), finite numbers, at most 16 rig sensors per
actor and 8 rig cameras per world, no disk writes ("save_to_disk": true is
refused), and changes need a play world (PIE or a game), never the editor's
level or a class default. Capture state is world-wide, so camera streams only
add to it and take back exactly what they added: StartCameraStreamsJson
turns channels on (never off), changes the rate only when asked, starts the
capture session only if it is not running (one it starts writes nothing to
disk), and refuses a port another program already listens on;
StopCameraStreams unregisters the rig cameras the streams registered, stops
the session only if the streams started it, and turns back the channels, rate
and disk writes they changed. A capture session somebody started for
recording keeps running as it was. GetStreamStatusJson says so in
capture.streams_on / capture.started_by_streams.
Segmentation and RammsUI's camera projector both use the custom-depth
stencil. While segmentation is on, CameraCapture gives every primitive its
class value as stencil, overwriting the 200 a URammsCameraProjectorComponent
target surface is marked with (TargetStencilValue) and its grid mesh's
PGMCustomStencilValue (1). A world that uses both lists those values in
ReservedStencilValues under [/Script/CameraCapture.CameraCaptureSegmentationSettings]
in DefaultGame.ini: surfaces already rendering custom depth with a reserved
value keep it (and read as that value in the mask).
The ToF and Sonar sensors can optionally use GPU-accelerated ray tracing
via a compute shader with DXR inline ray tracing (TraceRayInline). This
uses hardware RT cores when available and falls back to software ray
tracing otherwise, enabling massively parallel sensor simulation without
depending on specific GPU hardware.
Sensor Component (Game Thread)
│ BuildRayInputs() ← Generate rays from sensor config
│ SubmitGPUMeasurement() ← Queue rays for GPU dispatch
▼
FRammsSensorRayTracer (Public API)
│ SubmitTraces() ← Create readback, add to view extension
▼
FRammsSensorViewExtension (Render Thread)
│ PostTLASBuild_RenderThread() ← Called after TLAS is built each frame
│ DispatchSensorTrace() ← Transform rays to TLAS space, dispatch
▼
RammsSensorTrace.usf (GPU Compute Shader)
│ TraceRayInline(TLAS, ...) ← DXR inline ray query per thread
▼
GPU Readback → HarvestGPUResults() → Sensor Data + Debug Visualization
Key files:
| File | Description |
|---|---|
Shaders/Private/RammsSensorTrace.usf |
HLSL compute shader (64 threads/group) with TraceRayInline |
Public/RammsSensorTypes.h |
FSensorRayInput/FSensorRayOutput POD structs with compile-time layout validation |
Private/RammsSensorTraceShader.h |
C++ compute shader class (FRammsSensorTraceCS) — private, not part of public API |
Public/RammsSensorRayTracer.h |
Public API: SubmitTraces, IsRequestReady, HarvestResults |
Private/RammsSensorRayTracer.cpp |
Scene View Extension, RDG dispatch, TLAS coordinate transform, readback |
TLAS coordinate space: UE 5.7 builds the TLAS in camera-relative
coordinates (offset by PreViewTranslation ≈ -CameraPosition). The
system automatically transforms ray origins from world space to TLAS
space before dispatch. Directions are unaffected.
GPU sensor ray tracing requires:
-
DirectX 12 with ray tracing support enabled:
; DefaultEngine.ini [/Script/WindowsTargetPlatform.WindowsTargetSettings] DefaultGraphicsRHI=DefaultGraphicsRHI_DX12
-
Ray tracing enabled in project settings:
; DefaultEngine.ini [/Script/Engine.RendererSettings] r.RayTracing=True
-
Lumen Hardware Ray Tracing to populate the TLAS:
; DefaultEngine.ini [/Script/Engine.RendererSettings] r.Lumen.HardwareRayTracing=True
⚠ Important: Do not use
r.RayTracing.ForceAllRayTracingEffects=1as it breaksDrawDebugMeshrendering (used for sensor shape visualization). Lumen Hardware Ray Tracing naturally populates the TLAS without this side effect.
| CVar | Default | Description |
|---|---|---|
r.Ramms.SensorGPUTrace |
1 |
Enable/disable GPU ray tracing for sensors (0 = off) |
r.Ramms.SensorDebugMode |
0 |
Shader diagnostic mode — replaces rays 0–1 with TLAS probes and input echoes. Only for debugging; corrupts the first 2 results. |
The GPU dispatch pass is named RammsSensorTraceDispatch and appears
automatically in Unreal's built-in profiling tools:
stat gpu— Per-pass GPU timings in the viewport- GPU Visualizer (
Ctrl+Shift+,) — Detailed GPU timeline
Typical cost is negligible for small ray counts (64–256 rays). For
large-scale simulations, increase NumRays or GridRows/GridColumns
while monitoring the GPU timeline.
If the GPU path is unavailable (no DX12, no RT support, or
r.Ramms.SensorGPUTrace=0), sensors automatically fall back to CPU
LineTraceSingleByChannel. The MeasureNow() Blueprint function always
uses the CPU path for immediate synchronous results.
⚠ Collision filtering differences: The GPU ray tracing path traces against the full scene TLAS and does not honor
TraceChannelorbIgnoreOwner. These properties are greyed out in the editor whenbUseGPURayTracingis enabled. If per-channel or per-actor filtering is required, disable GPU tracing on that sensor to use the CPU path. The GPU path also cannot resolveHitActor(alwaysnullptr).
Both ToF and Sonar sensors support shape visualization in the editor
viewport and during gameplay, following the same pattern as the
CameraCapture plugin's IntrinsicSceneCaptureComponent2D.
- ToF (Grid mode) draws a frustum defined by
HorizontalFOV × VerticalFOVwith near/far planes atMinRange/MaxRange - ToF (SinglePoint mode) draws a line along +X with a crosshair at
MinRange - Sonar draws a cone with circular cross-sections at near and far distances
| Property | Default (ToF) | Default (Sonar) | Description |
|---|---|---|---|
bDrawShapeInEditor |
true |
true |
Show shape in editor viewport |
bDrawShapeInGame |
false |
false |
Show shape during gameplay |
ShapeColor |
Cyan | Orange | Outline color |
ShapeLineThickness |
0.5 |
0.5 |
Line width |
bDrawShapePlanes |
true |
true |
Draw filled translucent planes |
ShapePlaneColor |
Cyan (5% alpha) | Orange (5% alpha) | Fill color |
When bEnableDebugDisplay is enabled during gameplay, hit rays are drawn
in green, hit points as red dots, and miss rays in yellow
(dark yellow for ToF).
Note: Shape visualization draws in the editor viewport when
bDrawShapeInEditoris enabled, and during gameplay whenbDrawShapeInGameis enabled. The sensing pipeline (GPU/CPU traces, logging) only runs in game worlds.
All three solvers (DLS, FABRIK, CCD) share the same core:
ComputeChainKinematics— Forward kinematics from joint angles to world-space joint positions, axes, and end-effector transform.ApplyAngleLimits— Joint limit enforcement with optional escape margin.RotationErrorAxisAngleRad— Shortest-path rotation error as a world-space axis-angle vector.- Bone-derived kinematic chain —
JointLocalTransforms,JointAxesLocal, andEndEffectorOffsetare derived from the reference skeleton so FK matches the physics simulation exactly.
- Start from
BaseTransform(world-space arm base). - For each joint
i(root to tip):- Apply
JointLocalTransforms[i](parent-to-joint at q=0). - Record
JointPosWorld[i]and transformJointAxesLocal[i]to world. - Apply revolute rotation
FQuat(AxisWorld, q_i)about the axis.
- Apply
- Apply
EndEffectorOffsetafter the last joint.
These apply regardless of which solver is selected:
| Parameter | Default | Description |
|---|---|---|
IKSolverType |
DLS |
Active solver: DLS, FABRIK, CCD |
IKPositionTolerance |
1.0 cm |
Position convergence threshold |
IKRotationTolerance |
5.0 deg |
Rotation convergence threshold |
TaskSpaceMask |
{true x6} |
Which DOFs to solve: [X, Y, Z, Rx, Ry, Rz] |
IKTargetChangePosThreshold |
0.01 cm |
Min target position change to trigger re-solve |
IKTargetChangeRotThreshold |
0.1 deg |
Min target rotation change to trigger re-solve |
Set the orientation entries of TaskSpaceMask to false for position-only
solving (faster, fewer DOFs to satisfy).
| DLS | FABRIK | CCD | |
|---|---|---|---|
| Algorithm | Full Jacobian pseudoinverse | Per-joint scalar DLS (Gauss-Seidel) | Exact hinge-plane angles |
| FK per iteration | 1 | 1 | N (per joint) |
| Joint update | All at once (matrix solve) | Sequential, linearized with residual tracking | Sequential, exact geometry |
| Orientation | Coupled in 6×N Jacobian | Separate refinement pass | Blended per-joint |
| Cost per iter | 1 FK + matrix solve | 1 FK + N dot products | N FK evaluations |
| Convergence | Fast (global optimal) | Fast (linearized) | Moderate (greedy) |
| Best for | General-purpose, highest quality | Lightweight, no matrix library needed | Simple, robust |
The default and most capable solver. Builds the full 6×N Jacobian each
iteration and solves: dq = W J^T (J W J^T + λ²I)⁻¹ e.
| Parameter | Default | Description |
|---|---|---|
MaxIKIterations |
300 |
Maximum iterations per solve |
IKDampingFactor |
0.1 |
Damping λ — higher = more stable near singularities, less precise |
IKStepClip |
0.2 rad |
Per-joint step clamp per iteration (~11.5°) |
JointWeights |
(uniform) | Per-joint weight diagonal W. Higher = more responsive |
bEnableNullSpaceOptimization |
false |
Bias angles toward preferred pose without affecting EE |
NullSpaceGain |
0.1 |
Null-space bias strength (0–1) |
NullSpaceBias |
(empty) | Preferred joint angles (deg) for null-space optimization |
Tuning:
- Raise
IKDampingFactor(0.2–0.5) if the arm oscillates near singularities. - Lower
IKStepClipfor smoother but slower convergence. - Use
JointWeightsto make wrist joints lighter than shoulder joints.
A per-joint Jacobian-column solver using the same math as DLS but without the matrix solve. For each joint (tip to root), it computes a scalar DLS step from the Jacobian velocity column, then subtracts its predicted EE movement from the residual so the next joint sees the corrected error:
Jv_i = axis_i × (EE - joint_i) // Jacobian velocity column
dq_i = (Jv_i · dp_residual) / (Jv_i · Jv_i + λ²) // scalar DLS
dp_residual -= Jv_i * dq_i // Gauss-Seidel update
This gives Gauss-Seidel-like convergence from a single FK evaluation per iteration.
| Parameter | Default | Description |
|---|---|---|
FABRIKMaxIterations |
200 |
Maximum iterations |
FABRIKPositionTolerance |
1.0 cm |
Position convergence threshold |
FABRIKAngleGain |
1.0 |
Multiplier on the scalar DLS step (1.0 = optimal) |
FABRIKMaxAngleStepDeg |
12.0° |
Per-joint step clamp per iteration |
FABRIKLimitEscapeDeg |
2.0° |
Max step to move off a joint limit when stuck |
FABRIKOrientationIterations |
10 |
Separate orientation refinement passes after position |
FABRIKOrientationGain |
0.5 |
Orientation refinement strength |
Tuning:
- Increase
FABRIKMaxIterationsorFABRIKMaxAngleStepDegif the arm doesn't reach the target. AngleGain > 1.0is more aggressive but can overshoot.- Set
FABRIKOrientationIterations = 0for position-only solving.
Classic CCD constrained to 1-DOF hinge joints. Unlike FABRIK, it recomputes FK for each joint during the sweep, giving exact (non-linearized) geometry. For each joint it projects the joint-to-EE and joint-to-target vectors onto the hinge plane (perpendicular to the joint axis) and computes the exact signed angle between them.
When orientation is enabled, each joint blends a position delta and an orientation delta (rotation error projected onto the joint axis).
| Parameter | Default | Description |
|---|---|---|
CCDMaxIterations |
100 |
Maximum iterations |
CCDPositionGain |
1.0 |
Scale on hinge-plane angle (1.0 = full correction) |
CCDOrientationGain |
0.3 |
Orientation blend weight (0 = position only) |
CCDMaxAngleStepDeg |
12.0° |
Per-joint step clamp per iteration |
Tuning:
- CCD costs more per iteration (N FK evals) but needs fewer iterations for simple targets.
- Increase
CCDOrientationGainif EE orientation matters more than position. - Lower
CCDPositionGain(0.5–0.8) to reduce oscillation.
All solvers and FK are exposed as static BlueprintCallable functions:
// Forward kinematics
static FTransform ComputeForwardKinematics(
const FTransform& BaseTransform,
const TArray<float>& JointAnglesDeg,
const TArray<FTransform>& JointLocalTransforms,
const TArray<FVector>& JointAxesLocal,
const FTransform& EndEffectorOffset,
bool bReturnEEOnly);
// IK solvers — all return FIKSolveResult
static FIKSolveResult SolveIK_DLS(...);
static FIKSolveResult SolveIK_FABRIK(...);
static FIKSolveResult SolveIK_CCD(...);| Field | Type | Description |
|---|---|---|
bSuccess |
bool |
Whether convergence tolerances were met |
JointAngles |
TArray<float> |
Solved joint angles (degrees) |
PositionError |
float |
Final position error (cm) |
RotationError |
float |
Final rotation error (degrees) |
IterationsUsed |
int32 |
Iterations consumed |
The plugin includes Python scripts for converting between Unreal Engine Physics Assets and URDF (Unified Robot Description Format) files. This enables interoperability with ROS, MoveIt, PyBullet, MuJoCo, and other robotics tools.
The scripts run inside the UE Editor Python console and require no external dependencies beyond the Python standard library.
- Enable the Python Editor Script Plugin in Edit > Plugins > Scripting.
- Open the Output Log (Window > Developer Tools > Output Log) and switch the input dropdown from Cmd to Python.
- Select a PhysicsAsset in the Content Browser.
- Run:
from urdf.urdf_editor_utils import export_selected_to_urdf
export_selected_to_urdf()The URDF file is written to <ProjectDir>/urdf/<asset_name>.urdf.
export_selected_to_urdf() exports the currently selected Physics Asset.
It automatically resolves the associated skeletal mesh and uses the
skeleton's reference pose for bone transforms.
from urdf.urdf_editor_utils import export_selected_to_urdf
# Defaults — auto-names from the asset, writes to <Project>/urdf/
export_selected_to_urdf()
# Full options
export_selected_to_urdf(
robot_name="my_robot", # <robot name="...">
output_dir="C:/output", # output directory
include_collision=True, # include <collision> geometry
include_inertia=True, # include <inertial> properties
mapping_file="C:/map.json", # optional bone↔link name overrides
skeletal_mesh_path="/Game/MyMesh", # explicit skeletal mesh (auto-detected if omitted)
)You can also call the exporter directly with an asset path:
from urdf.urdf_exporter import export_urdf
export_urdf(
physics_asset_path="/Game/Robots/Arm/gen3_6dof_PhysicsAsset",
output_path="C:/output/kinova_gen3.urdf",
robot_name="kinova_gen3",
)What the exporter does:
- Exports the T3D representation of the Physics Asset to extract body setups (collision shapes, mass) and constraint templates (joint types, limits, axes).
- Reads bone world transforms from the skeleton reference pose.
- Builds a kinematic tree from constraint parent/child relationships, detecting and excluding loop-closure constraints (URDF requires a tree).
- Converts collision geometry (boxes, spheres, capsules, convex hulls →
cylinders) into URDF
<collision>elements. - Outputs joint types:
revolute,prismatic,fixed, orcontinuousbased on constraint settings and limits. - Applies automatic corrections for misaligned bone orientations and snaps near-cardinal joint axes to clean values.
import_urdf_to_selected() applies URDF data to the currently selected
Physics Asset.
from urdf.urdf_editor_utils import import_urdf_to_selected
# Physics-only: apply mass, inertia, and joint limits
import_urdf_to_selected("C:/path/to/robot.urdf")
# Full import: also create collision bodies from URDF geometry
import_urdf_to_selected("C:/path/to/robot.urdf", mode="full")| Mode | What it configures |
|---|---|
physics_only |
Mass, inertia tensors, joint limits, drive parameters |
full |
Everything above + collision shapes from URDF geometry |
The tooling automatically matches URDF link names to UE bone names using case-insensitive comparison with underscore/prefix stripping and fuzzy matching. For cases where auto-matching isn't sufficient, provide a JSON override file. Both flat and nested formats are supported:
{ "base_link": "root_bone", "link_1": "shoulder_bone" }{
"links": { "base_link": "root_bone", "link_1": "shoulder_bone" },
"joints": { "joint_1": "shoulder_constraint" }
}Pass it via the mapping_file parameter on export or import.
| URDF | Unreal Engine | |
|---|---|---|
| Units | meters, radians, kg | centimeters, degrees, kg |
| Handedness | Right-handed, Z-up | Left-handed, Z-up |
| Y axis | Points left | Points right (negated) |
All conversions are handled automatically. The Y axis is negated when transforming positions, quaternions, and joint axes between the two coordinate systems.
All scripts live in Content/Python/urdf/ within the plugin directory.
| Module | Description |
|---|---|
urdf_editor_utils.py |
One-click export/import entry points for the UE Editor |
urdf_exporter.py |
Core export logic (Physics Asset → URDF) |
urdf_importer.py |
Core import logic (URDF → Physics Asset) |
urdf_parser.py |
Pure-Python URDF XML parser and writer |
t3d_physics_parser.py |
Parser for UE T3D Physics Asset export format |
urdf_utils.py |
Unit conversions and coordinate transforms |
name_mapping.py |
Bone ↔ URDF link name matching and JSON overrides |
The test suite can be run outside of Unreal Engine (no unreal module
required). pytest is needed (pip install pytest):
cd Plugins/RammsCore/Content/Python
python -m pytest urdf/test_urdf.py -v
# or without pytest:
python urdf/test_urdf.py- Eigen — Matrix operations for DLS and rotation error computation. Included as a third-party dependency.
- PhysicsCore / Chaos — Physics constraint reading and actuation.
- RHI / RHICore / RenderCore / Renderer — GPU ray tracing infrastructure
(inline RT compute shader dispatch, RDG graph, TLAS access). Only used
when
bUseGPURayTracingis enabled on sensor components. - Projects — Plugin shader directory registration.