Ultra Engine Splitscreen / Stereo Vision

[Slastraf]

Hello
I would like to know how to render only half the screen or a specified area for a camera.
Alternatively, render a camera to a texture and have a plane with that texture perfectly placed in front of the main camera.

This could be used for stereo vision or for split screen / minimap etc

[Slastraf]

https://www.ultraengine.com/learn/Camera_SetRenderTarget?lang=cpp

 

 

This seems to be a good way of rendering multiple cameras. How do I display a texturebuffer in a third camera ? or directly inside a window ?

[Josh]

There is no 2D rendering in Ultra. There is only cameras with orthographic projection:

auto orthocamera = CreateCamera(world, PROJECTION_ORTHOGRAPHIC)
orthocamera->SetPosition(float(framebuffer->size.x) * 0.5f, float(framebuffer->size.y) * 0.5f, 0);

All cameras that render to a texture buffer are drawn first, then other cameras are rendered, in the order they are created.

You can create a sprite, create a material, get the texture buffer's color texture and apply it to the material, apply the material to the sprite, and place the sprite in front of your final orthographic camera, and that will make the two renders appear onscreen.

The reason it's done this way is because it gives you a lot of power to do anything you want. Post-processing effects can be applied underneath or on top of 2D graphics. You can add 2D graphics in the background. You can make multiple camera renders appear on screen.

This method can be used to control which entities are visible to which cameras:
https://www.ultraengine.com/learn/Entity_SetRenderLayers

The second example here might be helpful:
https://www.ultraengine.com/learn/CreateInterface

 

[Slastraf]

Here an example for stereo vision 3d or 2d VR example:
You need the UI.zip models
 

#include "UltraEngine.h"
#include "Components/Player/CameraControls.hpp"

using namespace UltraEngine;


int main(int argc, const char* argv[])
{
    //Get the displays
    auto displays = GetDisplays();

    //Create a window
    auto windowSize = Vec2(1920, 640);
    auto window = CreateWindow("Ultra Engine", 0, 0, windowSize.x, windowSize.y, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR);

    //Create a world
    auto world = CreateWorld();

    //Create a framebuffer
    auto framebuffer = CreateFramebuffer(window);

    //Create light
    auto light = CreateDirectionalLight(world);
    light->SetRenderLayers(2);
    light->SetRange(-10, 10);
    light->SetRotation(0, 0, 90);
    light->SetColor(100);

    auto light2 = CreateDirectionalLight(world);
    light->SetRotation(15, 15, 0);
    light->SetColor(5);

    //Create camera
    auto camera = CreateCamera(world);
    camera->SetRenderLayers(2);
    camera->SetClearColor(0.125);
    camera->SetPosition(0,0,-60);
    //camera->AddComponent<CameraControls>();
    camera->SetFov(2);

    //Create scenery
    auto test = LoadModel(world, "Models/UI/test.glb");
    
    auto scale = Vec3(.1, .1, .1);

    auto planeLeft = LoadModel(world, "Models/UI/LeftHalf.glb");
    planeLeft->SetRenderLayers(2);
    //planeLeft->SetPosition(0, 0, 0);
    //planeLeft->SetRotation(0, 0, 0);
    //b2->SetColor(0, 0, 1);
    planeLeft->SetScale(scale);

    auto planeRight = LoadModel(world, "Models/UI/RightHalf.glb");
    planeRight->SetRenderLayers(2);
    //planeRight->SetPosition(0, 0, 0);
    //planeRight->SetRotation(0, 0, 0);
    //b1->SetColor(1, 0, 0);
    planeRight->SetScale(scale);

    //Create camera and texture buffer
    auto texBufferLeft = CreateTextureBuffer(windowSize.x * 0.5, windowSize.y);
    auto texBufferRight = CreateTextureBuffer(windowSize.x * 0.5, windowSize.y);
    
    auto player = CreatePivot(world);
    player->AddComponent<CameraControls>();

    auto eyeDistance = .5;

    auto gameCameraLeftEye = CreateCamera(world);
    gameCameraLeftEye->SetClearColor(1, 1, 1);
    gameCameraLeftEye->SetRenderTarget(texBufferLeft);
    gameCameraLeftEye->SetPosition(-eyeDistance * 0.5, 0, 0);

    auto gameCameraRightEye = CreateCamera(world);
    gameCameraRightEye->SetClearColor(1, 1, 1);
    gameCameraRightEye->SetRenderTarget(texBufferRight);
    gameCameraRightEye->SetPosition(eyeDistance * 0.5, 0, 0);
    
    gameCameraLeftEye->SetParent(player);
    gameCameraRightEye->SetParent(player);

    player->SetPosition(0, 0, -5);
    //cam2->AddComponent<CameraControls>();

    //Create material
    auto mtll = CreateMaterial();
    auto mtlr = CreateMaterial();
    auto texl = texBufferLeft->GetColorAttachment();
    auto texr = texBufferRight->GetColorAttachment();
    mtll->SetTexture(texl);
    mtlr->SetTexture(texr);
    planeRight->SetMaterial(mtlr);
    planeLeft->SetMaterial(mtll);

    auto calibratedToResolution = false;
  	// at some settings or resolution, taking windowSize will not always work, so using projection of the center
    Vec3 pickPos = camera->Project(planeLeft->GetPosition(),framebuffer);
    // calibrate the canvas plane halfs to the screen size
    if (!calibratedToResolution)
    {
        auto steps = 0.001;
        // pick a pixel until the plane is adjusted to x and y
        while (!camera->Pick(framebuffer, 0, pickPos.y).success)
        {
            scale += Vec3(steps, 0, 0);
            planeLeft->SetScale(scale);
        }
        while (!camera->Pick(framebuffer, pickPos.x, 0).success)
        {
            scale += Vec3(0, steps, 0);
            planeLeft->SetScale(scale);
        }
        planeRight->SetScale(scale);
        calibratedToResolution = true;
    }

	// NEVER SPAWN MODELS BEFORE THIS POINT

    while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false)
    {
        //Orient the texturebuffer camera
        test->Turn(0, 1, 0);

        world->Update();
        world->Render(framebuffer);
    }
    return 0;
}

 

UI.zip

[Josh]

Nice job! How well does it run?

[Slastraf]

Very good. It runs as smooth as a single camera. Also can scale it up to 8k and wide screen, no problem.

[Josh]

There is a Camera::SetProjectionMatrix() command you can use if you need to override the default calculation. A lot of headsets use different angles for right/left and up/down: https://github.com/ValveSoftware/openvr/issues/1798

I'm sure everyone is very curious to know, which engine were you trying to do this with before Ultra, and how did its performance compare to the framerates you are enjoying now with Ultra Engine?

[Slastraf]

Unreal Engine VR Template scene + Ndisplay plugin endless headache and set up. Hovering at 10-20 fps (!). I did not bother to build it. In Unreal Engine you can use dynamic resolution and ai upscaling so we got about 20 fps and 30 with weird artifacts in the image.

I need 2 x  3840 by 2400 , so  a resolution of 7680  * 4800 for a specific stereo 3d beamer, for each eye.

Unity is what we use here usually but the graphics need to be tuned down for it to work smoothly aswell, and it needs extra plugins.

I programmed the 3d stereo vision not using two windows but one that renders each eye on the half of the screen using texturebuffer. (no plugin, and im not good at c++ but the documentation had examples that need to be adjusted to my case and that was it.)
The performance did not drop even at high resolution, and today I tried it out on the beamer.

The good thing is, Ultra Enginne builds in seconds so developing is very fast when you need to change pixel perfect values.