Core UI plugin for the RAMMP robotics project, built on Unreal Engine 5.7. Provides a full widget library, camera visualization pipeline, GPU-accelerated 3D projection (PGM), robot command dispatch, orientation-aware layout management, a data-asset theme system, and a central event bus.
Depends on: RammsStreaming (bridges external RMSS streams into the camera pipeline).
flowchart TB
subgraph External["External Systems"]
RMSS["RMSS Server<br/>(Python / ROS)"]
RemoteAPI["UE Remote Control<br/>HTTP :30010 / WS :30020"]
end
subgraph Streaming["RammsStreaming Plugin"]
Sink["URammsStreamSink<br/>(TCP :30030)"]
Bridge["URammsStreamCameraBridge<br/>demand-driven forwarding"]
end
subgraph Providers["Camera Providers"]
ProvBase["URammsCameraProviderBase<br/>(actor-based)"]
ProvComp["URammsCameraProviderComponent<br/>(component-based)"]
IProvider{{"IRammsCameraProvider<br/>OnCameraFrameReady"}}
end
subgraph Projection["Projection Pipeline"]
ProjMgr["URammsCameraProjectionManager<br/>auto-discover · create · link"]
Projector["URammsCameraProjectorComponent"]
Decal["UDecalComponent<br/>2D decal projection"]
PGM["Projective Grid Mesh<br/>3D RGBD reconstruction"]
end
subgraph Subsystem["URammsUISubsystem (World Subsystem)"]
Registry["Robot Controller<br/>Registry"]
EventBus["Event Bus<br/>(delegates)"]
PropStore["Property Store<br/>(key-value)"]
ThemeMgr["Theme Manager<br/>SetTheme / Dark / Light"]
end
subgraph Controllers["Robot Controllers"]
IRobot{{"IRammsRobotController"}}
RobotActor["Robot Actor<br/>(BP or C++)"]
end
subgraph Widgets["UI Widget Layer"]
LayoutHost["URammsLayoutHost<br/>orientation · crossfade"]
LayoutBase["URammsLayoutBase<br/>slot injection"]
CamWidget["URammsCameraWidget<br/>Fullscreen / Windowed / Corner"]
StatusPanel["URammsStatusPanel"]
ArmCtrl["URammsArmController"]
RammpGen1Ctrl["URammsRammpGen1Controller"]
Toolbar["URammsToolbar"]
OtherW["Other Widgets<br/>(Button, Slider, TextBlock, …)"]
end
subgraph Style["Theme System"]
StyleAsset["URammsUIStyle<br/>(DataAsset)"]
ThemeLib["URammsThemeLibrary<br/>(static BP library)"]
end
%% External → Streaming
RMSS -- "binary frames" --> Sink
Sink --> Bridge
RemoteAPI -. "commands / queries" .-> Subsystem
%% Bridge → Provider interface
Bridge --> IProvider
ProvBase --> IProvider
ProvComp --> IProvider
%% Provider → Camera Widget (2D display)
IProvider -- "texture + metadata" --> CamWidget
%% Provider → Projection (3D scene)
IProvider -- "texture + metadata" --> ProjMgr
ProjMgr -- "creates & manages" --> Projector
Projector --> Decal
Projector --> PGM
%% Subsystem ↔ Controllers
RobotActor --> IRobot
IRobot -- "register" --> Registry
EventBus -- "commands" --> IRobot
IRobot -- "state" --> EventBus
%% Subsystem ↔ Widgets
EventBus <-- "events / state" --> Widgets
ThemeMgr -- "propagate style" --> Widgets
%% Layout
LayoutHost -- "inject pool widgets" --> LayoutBase
LayoutBase -- "contains" --> CamWidget
LayoutBase -- "contains" --> StatusPanel
%% Widgets → Subsystem (commands)
ArmCtrl -- "SendCommand" --> EventBus
RammpGen1Ctrl -- "SendCommand" --> EventBus
Toolbar -- "OnToolbarItemClicked" --> EventBus
%% Theme
StyleAsset --> ThemeMgr
ThemeLib -. "create / reset / copy" .-> StyleAsset
%% Status panel reads state
EventBus -- "OnRobotStateChanged" --> StatusPanel
PropStore -- "OnPropertyChanged" --> StatusPanel
| Path | Flow |
|---|---|
| Camera frames | RMSS → StreamSink → Bridge → IRammsCameraProvider → CameraWidget (2D) / ProjectionManager (3D) |
| Robot commands | Widget → URammsUISubsystem → IRammsRobotController actor |
| Robot state | Controller → OnRobotStateChanged → StatusPanel / any subscriber |
| Theme changes | SetTheme() → subsystem propagates to all live URammsBaseWidget instances |
| Layout transitions | URammsLayoutHost detects orientation or receives request → crossfade → reparent pool widgets |
| Remote control | HTTP/WS → UE Remote Control API → URammsRemoteBridge → subsystem / widgets |
| Service | Port |
|---|---|
| UE Remote Control HTTP | 30010 |
| UE Remote Control WebSocket | 30020 |
| RMSS Binary Streaming | 30030 |
Provider → Widget → Display architecture with demand-driven data forwarding.
Internal: URammsCameraProviderBase ──► URammsCameraWidget ──► Display Mode
External: RMSS Stream ──► URammsStreamCameraBridge ──► IRammsCameraProvider ──► URammsCameraWidget
| Class | Role |
|---|---|
IRammsCameraProvider |
Interface — any actor can provide camera frames via OnCameraFrameReady |
URammsCameraProviderBase |
Actor-based provider with socket and transform support |
URammsCameraProviderComponent |
Actor component implementing IRammsCameraProvider |
URammsStreamCameraBridge |
Bridges RammsStreaming sink → IRammsCameraProvider; demand-driven raw-data forwarding, auto-registers streams with intrinsics/extrinsics/group/role metadata. Tracks channel→stream_id mapping and re-registers when a stream_id override changes mid-session. |
URammsCameraWidget |
Display widget — Fullscreen / Windowed / Corner modes, aspect-ratio maintenance, collapsible header, drag-to-move, multi-view (RGB / Data / SideBySide / Overlay), material-based data visualization |
Real-time camera-feed projection onto scene surfaces using Deferred Decals.
| Class | Role |
|---|---|
URammsCameraProjectorComponent |
Per-camera projector — internal UDecalComponent driven by projection material |
URammsCameraProjectionManager |
Auto-discovers providers, creates projectors, links depth streams, propagates settings |
- Provider discovery — Manager scans for
IRammsCameraProvideractors and components onBeginPlay+ deferred timer. - Projector creation — One
URammsCameraProjectorComponentper color stream. Depth-only streams are linked to color projectors but never get their own decal. - Decal rendering — Each projector creates a
UDecalComponentpositioned at the camera extrinsic (world pose). The frustum extent is computed from intrinsics andMaxProjectionDistance. - Material reprojection — The deferred decal material (
RammsProjection.ush) samples the camera texture and reprojects it into world space using the camera intrinsics and pose vectors. - Stencil masking & edge fade — Per-projector stencil value and UV-space edge fade prevent bleeding.
| Parameter | Type | Description |
|---|---|---|
CameraTexture |
Texture | Current video frame |
Intrinsics |
Vector4 | (fx, fy, cx, cy) — camera calibration in pixels |
ImageSize |
Vector4 | (width, height, 0, 0) |
CameraWorldPos |
Vector3 | Extrinsic world position |
CameraForward / CameraRight / CameraUp |
Vector3 | Extrinsic orientation axes |
FadeWidth |
Scalar | Normalized edge fade [0..0.5] |
TargetStencil |
Scalar | Scene stencil mask (0 = no filtering) |
| Property | Default | Description |
|---|---|---|
ProjectionMaterial |
— | Deferred Decal material (Translucent) |
bAutoCreateProjectors |
true |
Auto-discover providers and create projectors |
ExcludeStreamIDs |
— | Stream IDs to skip |
DefaultMaxDistance |
5000 cm | Frustum max depth |
DefaultFadeWidth |
0.05 | Edge fade (5%) |
DefaultTargetStencil |
200 | Decal stencil value |
Primary: GroupID + Role matching. The manager links depth streams to color projectors by matching GroupID (set in stream metadata) and checking StreamRole. A depth stream (StreamRole::Depth) is linked to the color projector in the same group that has StreamRole::Color (falling back to !bIsDepth only when role is Other/unset). When a depth stream is linked, its DepthFormat (from FRammsCameraStreamInfo) is applied to the projector via ApplyDepthStreamFormat(), automatically configuring DepthScaleToCM (e.g. 0.1 for uint16-mm → cm).
Fallback: channel ± 100 convention (deprecated). For legacy streams without group/role metadata, the manager falls back to the old channel + 100 convention (e.g. stream/0 → stream/100).
Auto-linking occurs at three points: initial projector creation (CreateProjectorsForProvider), runtime stream activation (OnCameraStreamStatus), and first-frame depth delivery (OnCameraFrameReady).
Real-time RGBD point-cloud reconstruction projected into the 3D viewport.
| Class | Role |
|---|---|
URammsCameraProjectorComponent |
Per-camera projector with two rendering paths (see below) |
URammsCameraProjectionManager |
Manages multiple projectors — auto-discovers providers, propagates settings, dynamic projector creation |
Static grid mesh + World Position Offset material reads the depth texture and deprojects vertices in the vertex shader via RammsPGM.ush. No CPU readback.
Per-frame vertex position + color computation from raw RGBD data on the CPU.
Deprojection from depth, parallax-corrected RGB UV lookup, NaN degenerate-vertex collapse, neighbor edge-stretch check.
Material parameters: DepthTexture, CameraTexture, PGMIntrinsics, PGMImageSize, PGMDepthConfig, PGMParallax, PGMColorIntrinsics, PGMGridStep, PGMDepthUnnormalize.
PGMDepthUnnormalize is automatically derived from DetectedDepthFormat (set via stream info) or the depth texture's pixel format (PF_G16 → 65535.0, otherwise 1.0). This ensures uint16 depth textures (which are GPU-normalized to [0,1]) are correctly rescaled.
Projector properties: Decimation, SensorBaselineY, MaxEdgeStretchCM, DepthScaleToCM, MinDepthCM / MaxDepthCM, DetectedDepthFormat, SyncThresholdMS for frame synchronization. DetectedDepthFormat is configured by ApplyDepthStreamFormat() when a depth stream is linked.
Orientation-aware layout management with crossfade transitions and a shared widget pool.
- Widget pool: shared
URammsBaseWidgetinstances are reparented into the active layout'sNamedSlotcontainers on every transition. - Orientation detection: auto-detects portrait / landscape from viewport aspect ratio.
- Pair mapping:
PortraitLayoutMap(TMap<FName, FName>) + suffix fallback (_Portrait/_Landscape) for orientation variants. - Crossfade: overlay-based animated transition (
CrossfadeDuration, default 0.3 s).
enum class ERammsOrientationOverride { Auto, ForceLandscape, ForcePortrait };| Delegate | Signature |
|---|---|
OnLayoutTransitionStarted |
(FName NewLayout, FName PreviousLayout) |
OnLayoutTransitionCompleted |
(FName NewLayout, FName PreviousLayout) |
OnOrientationChanged |
(ERammsOrientation NewOrientation) |
Abstract base for individual layouts. Auto-discovers injection slots (NamedSlot, Overlay, SizeBox) via DiscoverSlots(). Provides InjectWidget(), ClearSlot(), and RequestLayoutTransition().
The robot HUD's default layout (URammsSimLayout, spawned by URammsControlHUDSubsystem) puts each panel in a URammsFloatingPanel the user arranges. On by default; turn it off with URammsControlHUDSettings::bFloatingPanels (Project Settings > Plugins > Ramms Control HUD) to get the fixed columns back.
- Title bar: drag it to move the panel,
-/+to minimise it to the bar,xto hide it, the arrow (↗) to pop it out into a window of its own. Close that window to dock the panel back. - Resize grip: the bottom-right corner of the robot panel and any provider panel.
- Panel tray (top centre): a slim strip with a button per panel, each marked with its state: ● shown, ▬ minimised, ○ hidden, ↗ in its own window. Click a shown panel to hide it, a hidden one to show it, a minimised one to restore it, and a popped-out one to bring its window forward. After a divider come Hide all / Show all and Reset.
- Remembered per user. Each panel's place, size, minimised / hidden / popped-out state and window rectangle are stored in
Saved/Config/<Platform>/GameUserSettings.ini(URammsHUDLayoutUserSettings), never in the project config. A panel's place is a fraction of the room the viewport leaves around it, so a panel against an edge stays there when the window is resized. - Console:
ramms.UI.ResetLayoutandramms.UI.Panels [show|hide|toggle]. - Pop-outs are desktop only. They are turned off by
URammsControlHUDSettings::bAllowPopOutand whenever the command line asks for Pixel Streaming, which streams only the game viewport. Keyboard focus moves into a popped-out window when you click it, so click back into the game view to drive with the keyboard.
Content can take part in the chrome by implementing IRammsFloatingPanelContent: it supplies the title-bar text and drops its own frame while hosted. URammsControlSurfacePanel and URammsCameraCapturePanel both do.
Panels from components. A component on the controlled pawn that implements IRammsHUDPanelProvider (RammsControl) offers panels as Slate content, and says whether each is wanted now. The layout hosts each one in a floating panel, wrapped in a URammsSlateHostWidget, and shows it while it is wanted. The curb controller's "Curb view" graphs come in this way.
Centralized styling via UDataAsset.
| Struct | Contents |
|---|---|
FRammsColorPalette |
Primary, Secondary, Background, Surface, Text (Primary/Secondary/Disabled), Border, Success, Warning, Error, Info |
FRammsTypography |
HeadingLarge/Medium/Small, Body, Caption, Monospace (FSlateFontInfo) |
FRammsSpacing |
XSmall (2) → XXLarge (32) |
FRammsBorderStyle |
Border widths, corner radii (Small / Medium / Large) |
FRammsScrollBarStyle |
Thickness, corner radius, thumb/track colors, padding |
FRammsSliderStyle |
Thumb size, bar thickness, active/inactive/hover colors |
FRammsCheckBoxStyle |
Size, corner radius, checked/unchecked/hover/border colors, check mark color |
FRammsAnimationCurve |
Duration, easing (ERammsUIEasing), delay — presets for fade, slide, scale, expand/collapse |
Built-in themes: CreateDefaultDarkTheme() / CreateDefaultLightTheme().
Static Blueprint Function Library (no world context needed — usable from Editor Utility Widgets):
| Function | Description |
|---|---|
CreateDarkThemeAsset(AssetPath) |
Save a new DataAsset pre-filled with dark defaults |
CreateLightThemeAsset(AssetPath) |
Save a new DataAsset pre-filled with light defaults |
DuplicateThemeAsAsset(Source, AssetPath) |
Clone any theme to a new DataAsset |
CopyTheme(Source, Target) |
Copy all values from one theme into another |
Instance methods on URammsUIStyle: ResetToDarkDefaults(), ResetToLightDefaults(), CopyFrom(Source).
URammsUISubsystem owns the current theme. SetTheme() / SetDarkTheme() / SetLightTheme() fire OnThemeChanged and propagate to every live widget. Widgets auto-apply the subsystem theme in NativeConstruct when no explicit style is set (bAutoApplyStyle).
All widgets inherit from URammsBaseWidget (UUserWidget), which provides:
- Style application + child propagation
- Animation helpers:
FadeIn/Out,SlideIn/Out,ScaleIn/OutwithERammsUIEasing - Robot controller auto-resolution via
URammsUISubsystem - Programmatic widget-tree construction (
BuildWidgetTree())
| Widget | Description |
|---|---|
URammsButton |
Variant-driven button: Primary / Secondary / Success / Warning / Error / Ghost |
URammsSlider |
Styled numeric slider, themed via FRammsSliderStyle |
URammsCollapsibleContainer |
Animated expand/collapse container with styled header |
URammsStatusPanel |
Data-driven status table with FRammsStatusField rows — per-field source (RobotState / PropertyStore / Custom), format, threshold coloring, animated collapse |
URammsToolbar |
Horizontal / Vertical toolbar with FRammsToolbarItem, per-item toggle state |
URammsAxisControl |
Per-axis slider with FRammsAxisConfig, ± increment buttons, reset |
URammsJoystickWidget |
2D joystick input for movement commands |
URammsRadioButton |
Mutual-exclusion radio group, themed checkboxes via FRammsCheckBoxStyle |
URammsTextBlock |
Themed text block — variant (HeadingLarge … Monospace), color mode (Primary / Secondary / Accent / Custom / …), auto-updates on theme change |
URammsColorPicker |
HSV colour picker with hue rainbow gradient slider, optional saturation/brightness sliders, live colour preview swatch, and hex value display. Gradient textures are generated procedurally. |
URammsImageButton |
Icon button with image background |
URammsNotificationWidget |
Single toast notification with severity level |
URammsNotificationContainer |
Viewport container that stacks and auto-dismisses notifications |
URammsPanel |
Simple styled container panel |
URammsFloatingPanel |
Window chrome around any widget: move, resize, minimise, hide, pop out into its own window; state kept per user (see Floating panels) |
URammsSlateHostWidget |
Hosts a Slate widget made elsewhere (an IRammsHUDPanelProvider's content) as a UMG widget |
URammsArmController |
Kinova Gen3 7-joint arm control widget |
URammsRammpGen1Controller |
RAMMP Gen1 motor control widget |
URammsArmTaskWidget |
Arm task selection UI |
URammsTaskWidget |
General task selection UI |
URammsSeatController |
Seat axis adjustment controls |
URammsLayoutHost |
Root layout manager (see Layout System) |
URammsLayoutBase |
Layout base class (see Layout System) |
URammsLayoutManager |
Legacy responsive layout component |
UWorldSubsystem acting as the central event hub.
Auto-discovery: actors implementing IRammsRobotController register/unregister via RegisterRobotController() / UnregisterRobotController(). Query with FindRobotController(), FindRobotControllerByName(), GetAllRobotControllers().
| Delegate | Payload |
|---|---|
OnTaskAction |
ERammsTaskAction |
OnArmTaskChanged |
ERammsArmTask NewTask, ERammsArmTask PreviousTask |
OnRobotStateChanged |
const FRammsRobotState& |
OnSeatStateChanged |
ERammsSeatAxis, float |
OnToolbarItemClicked |
FName ItemID |
OnToolbarToggle |
FName ItemID, bool bIsActive |
OnOverlayToggle |
ERammsOverlayType, bool bVisible |
OnVisualizationLayerToggle |
ERammsVisualizationLayer, bool bEnabled |
OnHighlight |
ERammsHighlightTarget, bool bHighlighted, FLinearColor |
OnLayoutTransitionRequest |
FName LayoutName, bool bAnimated |
OnCustomUIEvent |
const FRammsUIEvent& (rich payload — see below) |
OnPropertyChanged |
FName Key, const FString& Value |
OnRobotCommandSent |
FName CommandName, bool bAccepted |
OnThemeChanged |
URammsUIStyle* NewStyle |
Rich payload struct: EventName, typed fields (bEnabled, FloatValue, IntValue, ColorValue, VectorValue, StringValue), an FInstancedStruct StructPayload, and a TMap<FName, FString> Properties bag with typed getters (GetPayloadFloat, GetPayloadBool, …).
SetProperty() / GetProperty() with typed convenience: GetPropertyAsFloat(), GetPropertyAsInt(), GetPropertyAsBool(), GetPropertyAsByte(). Changes fire OnPropertyChanged.
| Method | Scope |
|---|---|
SendRobotCommand(FName) |
First registered controller |
SendRobotCommandWithValue(FName, uint8) |
First registered controller |
SendRobotCommandWithPayload(FName, FInstancedStruct) |
First registered controller |
BroadcastRobotCommand(FName) |
All controllers |
BroadcastRobotCommandWithValue(FName, uint8) |
All controllers |
BroadcastRobotCommandWithPayload(FName, FInstancedStruct) |
All controllers |
BroadcastLayoutTransitionRequest(FName, bool bAnimated) — triggers OnLayoutTransitionRequest.
SetTheme() / SetDarkTheme() / SetLightTheme() — see Theme System.
| Interface | Purpose |
|---|---|
IRammsCameraProvider |
Camera frame source (OnCameraFrameReady) |
IRammsRobotController |
Robot command sink — SendCommand, SendCommandWithValue, SendCommandWithPayload, arm/RAMMP Gen1/seat control, state queries. Full C++ / Blueprint polymorphism via BlueprintNativeEvent. |
IRammsStateProvider |
Robot state source |
IRammsDataSource |
Generic data source |
IRammsCommandSink |
Generic command sink |
| File | Purpose |
|---|---|
/RammsUI/Private/RammsPGM.ush |
GPU PGM deprojection, parallax-corrected UV, degenerate-vertex collapse, edge-stretch check |
/RammsUI/Private/RammsRoundedCorners.ush |
SDF-based rounded-corner masking |
/RammsUI/Private/RammsProjection.ush |
Camera projection utilities |
/RammsUI/Private/RammsOutline.usf |
Outline post-process effect |
/RammsUI/Private/RammsOutlineCommon.ush |
Shared outline utilities |
URammsUISubsystem* UI = GetWorld()->GetSubsystem<URammsUISubsystem>();
// Robot state changes
UI->OnRobotStateChanged.AddDynamic(this, &AMyActor::HandleRobotState);
// Toolbar toggle
UI->OnToolbarToggle.AddDynamic(this, &AMyActor::HandleToolbarToggle);
// Custom events with typed payload
UI->OnCustomUIEvent.AddDynamic(this, &AMyActor::HandleCustomEvent);URammsUISubsystem* UI = GetWorld()->GetSubsystem<URammsUISubsystem>();
// Built-in presets
UI->SetDarkTheme();
UI->SetLightTheme();
// Custom theme data asset
UI->SetTheme(MyCustomStyleAsset);
// React to changes
UI->OnThemeChanged.AddDynamic(this, &AMyHUD::OnThemeSwitch);URammsLayoutHost* Host = /* your layout host widget */;
// Register two layouts
Host->AddLayout(UMyDriveLayout::StaticClass(), TEXT("Drive"));
Host->AddLayout(UMyArmLayout::StaticClass(), TEXT("Arm"));
// Map portrait orientation variants
Host->PortraitLayoutMap.Add(TEXT("Drive"), TEXT("Drive_Portrait"));
Host->PortraitLayoutMap.Add(TEXT("Arm"), TEXT("Arm_Portrait"));
// Add shared widgets to the pool — they follow the active layout
Host->AddPoolWidget(TEXT("CameraFeed"), MyCameraWidget);
Host->AddPoolWidget(TEXT("StatusBar"), MyStatusPanel);
// Transition with crossfade
Host->TransitionToLayout(TEXT("Drive"), /*bAnimated=*/ true);URammsUISubsystem* UI = GetWorld()->GetSubsystem<URammsUISubsystem>();
// Simple command
UI->SendRobotCommand(TEXT("EStop"));
// Command with value
UI->SendRobotCommandWithValue(TEXT("SetSpeed"), 3);
// Command with struct payload (broadcast to all controllers)
FInstancedStruct Payload;
Payload.InitializeAs<FMyWaypointData>(TargetLocation, Tolerance);
UI->BroadcastRobotCommandWithPayload(TEXT("NavigateTo"), Payload);| Class | Description |
|---|---|
URammsRemoteBridge |
Static function library for UE Remote Control API integration — widget discovery, status panel updates, and notification dispatch |
See LICENSE in this directory.