Unity: Extending The Lightweight Render Pipeline To Support Mobile AR

As of October 4th 2018, AR Foundation Preview 18 now supports the LWRP natively. Note that Unity 2018.3 and LWRP 3.3.0 are required for the official patch. Please refer to this Unity forum post.

The article below is maintained for historical purposes, and for those unable to upgrade to Unity 2018.3.

Unity’s new Lightweight Render Pipeline (LWRP) is great for games and applications targetting platforms where performance is critical, such as mobile devices. Unfortunately, as of writing, it does not support Unity’s mobile augmented reality stack (either AR Foundation or the individual ARCore/ARKit plugins).

Thankfully, it’s fairly easy to extend the LWRP to get it working.

Setup

This method has been tested with Unity 2018.2 and the following package versions:

  • LWRP 3.0.0-preview (might also work with 2.0.0-preview)
  • AR Foundation 1.0.0-preview.14
  • ARCore XR Plugin 1.0.0-preview.18

Note that the following are not supported:

  • Post-processing (tested with Unity’s Post-processing 2.0.9-preview package)
  • The HDR functionality of the LWRP.

I have not yet tested this on iOS/ARKit, but it should hopefully work the same. YMMV.

If you don’t already have a project with AR enabled and functioning, you can test this method by starting with the Unity-Technologies/arfoundation-samples repository, and adding the LWRP via the Package Manager.

Feel free to reach out on Twitter (@NolanScobie) if you have any issues.

Explanation

In order to render the phone’s camera feed behind your AR content, AR Foundation uses Unity’s built-in UnityEngine.XR.ARBackgroundRenderer class. As we can see from the source code, ARBackgroundRenderer functions by queueing CommandBuffers onto the scene camera, which should render the phone’s camera feed to the background when they are executed by the rendering pipeline. Unity’s default/legacy rendering pipeline would go ahead and execute these when appropriate, but the LWRP does not. As part of the Scriptable Render Pipeline team’s overhaul to the rendering system, they’ve decided to move away from this system of doing things, and therefore don’t currently execute CommandBuffers when rendering cameras.

Fortunately, LWRP v2+ supports the ability for developer to add custom rendering passes to the LWRP, without actually modifying the package’s code. To support AR, we’ll need to add a custom pass to execute our camera’s CommandBuffers when appropriate, and then tell the LWRP to use our custom pass in the right spot via a custom IRendererSetup attached to our camera.

Note that all of these packages are currently in preview mode, and will likely all support eachother flawlessly at some point in the future. Until then, we just need to hack things together a little bit.

Code

Start by creating the render pass that will execute the CommandBuffers on our camera:

ExecuteCommandBuffersPass.cs

using UnityEngine.Rendering;
using UnityEngine.Experimental.Rendering;
using UnityEngine.Experimental.Rendering.LightweightPipeline;

public class ExecuteCommandBuffersPass : ScriptableRenderPass
{
    public ExecuteCommandBuffersPass(LightweightForwardRenderer renderer) : base(renderer) { }

    public override void Execute(ref ScriptableRenderContext context, ref CullResults cullResults, ref RenderingData renderingData)
    {
        //Note: The following only supports FORWARD rendering!
        //However, as of July 2018, the LWRP doesn't support deferred rendering, so this is fine.
        foreach (var cmd in renderingData.cameraData.camera.GetCommandBuffers(CameraEvent.BeforeForwardOpaque))
        {
            context.ExecuteCommandBuffer(cmd);
        }
    }
}

Next we need to tell the LWRP to use our new rendering pass. We can do this by copying the DefaultRendererSetup.cs file found in the root of the LWRP package and modifying it to add our custom render pass we just created:

HandheldArRendererSetup.cs

using System;
using UnityEngine;
using UnityEngine.Experimental.Rendering;
using UnityEngine.Experimental.Rendering.LightweightPipeline;
using UnityEngine.Rendering;

public class HandheldArRendererSetup : MonoBehaviour, IRendererSetup
{
    private DepthOnlyPass m_DepthOnlyPass;
    private DirectionalShadowsPass m_DirectionalShadowPass;
    private LocalShadowsPass m_LocalShadowPass;
    private SetupForwardRenderingPass m_SetupForwardRenderingPass;
    private ScreenSpaceShadowResolvePass m_ScreenSpaceShadowResovePass;
    private CreateLightweightRenderTexturesPass m_CreateLightweightRenderTexturesPass;
    private BeginXRRenderingPass m_BeginXrRenderingPass;
    private SetupLightweightConstanstPass m_SetupLightweightConstants;
    private ExecuteCommandBuffersPass m_ExecuteCommandBuffersPass; //Our custom code
    private RenderOpaqueForwardPass m_RenderOpaqueForwardPass;
    private OpaquePostProcessPass m_OpaquePostProcessPass;
    private DrawSkyboxPass m_DrawSkyboxPass;
    private CopyDepthPass m_CopyDepthPass;
    private CopyColorPass m_CopyColorPass;
    private RenderTransparentForwardPass m_RenderTransparentForwardPass;
    private TransparentPostProcessPass m_TransparentPostProcessPass;
    private FinalBlitPass m_FinalBlitPass;
    private EndXRRenderingPass m_EndXrRenderingPass;

#if UNITY_EDITOR
    private SceneViewDepthCopyPass m_SceneViewDepthCopyPass;
#endif


    private RenderTargetHandle Color;
    private RenderTargetHandle DepthAttachment;
    private RenderTargetHandle DepthTexture;
    private RenderTargetHandle OpaqueColor;
    private RenderTargetHandle DirectionalShadowmap;
    private RenderTargetHandle LocalShadowmap;
    private RenderTargetHandle ScreenSpaceShadowmap;

    [NonSerialized]
    private bool m_Initialized = false;

    private void Init(LightweightForwardRenderer renderer)
    {
        if (m_Initialized)
            return;

        m_DepthOnlyPass = new DepthOnlyPass(renderer);
        m_DirectionalShadowPass = new DirectionalShadowsPass(renderer);
        m_LocalShadowPass = new LocalShadowsPass(renderer);
        m_SetupForwardRenderingPass = new SetupForwardRenderingPass(renderer);
        m_ScreenSpaceShadowResovePass = new ScreenSpaceShadowResolvePass(renderer);
        m_CreateLightweightRenderTexturesPass = new CreateLightweightRenderTexturesPass(renderer);
        m_BeginXrRenderingPass = new BeginXRRenderingPass(renderer);
        m_SetupLightweightConstants = new SetupLightweightConstanstPass(renderer);
        m_ExecuteCommandBuffersPass = new ExecuteCommandBuffersPass(renderer); //Our custom code
        m_RenderOpaqueForwardPass = new RenderOpaqueForwardPass(renderer);
        m_OpaquePostProcessPass = new OpaquePostProcessPass(renderer);
        m_DrawSkyboxPass = new DrawSkyboxPass(renderer);
        m_CopyDepthPass = new CopyDepthPass(renderer);
        m_CopyColorPass = new CopyColorPass(renderer);
        m_RenderTransparentForwardPass = new RenderTransparentForwardPass(renderer);
        m_TransparentPostProcessPass = new TransparentPostProcessPass(renderer);
        m_FinalBlitPass = new FinalBlitPass(renderer);
        m_EndXrRenderingPass = new EndXRRenderingPass(renderer);

#if UNITY_EDITOR
        m_SceneViewDepthCopyPass = new SceneViewDepthCopyPass(renderer);
#endif

        // RenderTexture format depends on camera and pipeline (HDR, non HDR, etc)
        // Samples (MSAA) depend on camera and pipeline
        Color.Init("_CameraColorTexture");
        DepthAttachment.Init("_CameraDepthAttachment");
        DepthTexture.Init("_CameraDepthTexture");
        OpaqueColor.Init("_CameraOpaqueTexture");
        DirectionalShadowmap.Init("_DirectionalShadowmapTexture");
        LocalShadowmap.Init("_LocalShadowmapTexture");
        ScreenSpaceShadowmap.Init("_ScreenSpaceShadowMapTexture");

        m_Initialized = true;
    }

    public void Setup(LightweightForwardRenderer renderer, ref ScriptableRenderContext context,
        ref CullResults cullResults, ref RenderingData renderingData)
    {
        Init(renderer);

        renderer.Clear();

        renderer.SetupPerObjectLightIndices(ref cullResults, ref renderingData.lightData);
        RenderTextureDescriptor baseDescriptor = renderer.CreateRTDesc(ref renderingData.cameraData);
        RenderTextureDescriptor shadowDescriptor = baseDescriptor;
        shadowDescriptor.dimension = TextureDimension.Tex2D;

        bool requiresCameraDepth = renderingData.cameraData.requiresDepthTexture;
        bool requiresDepthPrepass = renderingData.shadowData.requiresScreenSpaceShadowResolve ||
                                    renderingData.cameraData.isSceneViewCamera ||
                                    (requiresCameraDepth &&
                                        !LightweightForwardRenderer.CanCopyDepth(ref renderingData.cameraData));

        // For now VR requires a depth prepass until we figure out how to properly resolve texture2DMS in stereo
        requiresDepthPrepass |= renderingData.cameraData.isStereoEnabled;

        if (renderingData.shadowData.renderDirectionalShadows)
        {
            m_DirectionalShadowPass.Setup(DirectionalShadowmap);
            renderer.EnqueuePass(m_DirectionalShadowPass);
        }

        if (renderingData.shadowData.renderLocalShadows)
        {

            m_LocalShadowPass.Setup(LocalShadowmap);
            renderer.EnqueuePass(m_LocalShadowPass);
        }

        renderer.EnqueuePass(m_SetupForwardRenderingPass);

        if (requiresDepthPrepass)
        {
            m_DepthOnlyPass.Setup(baseDescriptor, DepthTexture, SampleCount.One);
            renderer.EnqueuePass(m_DepthOnlyPass);
        }

        if (renderingData.shadowData.renderDirectionalShadows &&
            renderingData.shadowData.requiresScreenSpaceShadowResolve)
        {
            m_ScreenSpaceShadowResovePass.Setup(baseDescriptor, ScreenSpaceShadowmap);
            renderer.EnqueuePass(m_ScreenSpaceShadowResovePass);
        }

        bool requiresDepthAttachment = requiresCameraDepth && !requiresDepthPrepass;
        bool requiresColorAttachment =
            LightweightForwardRenderer.RequiresIntermediateColorTexture(
                ref renderingData.cameraData,
                baseDescriptor,
                requiresDepthAttachment);
        RenderTargetHandle colorHandle = (requiresColorAttachment) ? Color : RenderTargetHandle.CameraTarget;
        RenderTargetHandle depthHandle = (requiresDepthAttachment) ? DepthAttachment : RenderTargetHandle.CameraTarget;

        var sampleCount = (SampleCount)renderingData.cameraData.msaaSamples;
        m_CreateLightweightRenderTexturesPass.Setup(baseDescriptor, colorHandle, depthHandle, sampleCount);
        renderer.EnqueuePass(m_CreateLightweightRenderTexturesPass);

        if (renderingData.cameraData.isStereoEnabled)
            renderer.EnqueuePass(m_BeginXrRenderingPass);

        Camera camera = renderingData.cameraData.camera;
        bool dynamicBatching = renderingData.supportsDynamicBatching;
        RendererConfiguration rendererConfiguration = LightweightForwardRenderer.GetRendererConfiguration(renderingData.lightData.totalAdditionalLightsCount);

        renderer.EnqueuePass(m_SetupLightweightConstants);

        renderer.EnqueuePass(m_ExecuteCommandBuffersPass); //Our custom code

        m_RenderOpaqueForwardPass.Setup(baseDescriptor, colorHandle, depthHandle, LightweightForwardRenderer.GetCameraClearFlag(camera), camera.backgroundColor, rendererConfiguration, dynamicBatching);
        renderer.EnqueuePass(m_RenderOpaqueForwardPass);

        if (renderingData.cameraData.postProcessEnabled &&
            renderingData.cameraData.postProcessLayer.HasOpaqueOnlyEffects(renderer.postProcessRenderContext))
        {
            m_OpaquePostProcessPass.Setup(baseDescriptor, colorHandle);
            renderer.EnqueuePass(m_OpaquePostProcessPass);
        }

        if (camera.clearFlags == CameraClearFlags.Skybox)
            renderer.EnqueuePass(m_DrawSkyboxPass);

        if (depthHandle != RenderTargetHandle.CameraTarget)
        {
            m_CopyDepthPass.Setup(depthHandle, DepthTexture);
            renderer.EnqueuePass(m_CopyDepthPass);
        }

        if (renderingData.cameraData.requiresOpaqueTexture)
        {
            m_CopyColorPass.Setup(colorHandle, OpaqueColor);
            renderer.EnqueuePass(m_CopyColorPass);
        }

        m_RenderTransparentForwardPass.Setup(baseDescriptor, colorHandle, depthHandle, ClearFlag.None, camera.backgroundColor, rendererConfiguration, dynamicBatching);
        renderer.EnqueuePass(m_RenderTransparentForwardPass);

        if (renderingData.cameraData.postProcessEnabled)
        {
            m_TransparentPostProcessPass.Setup(baseDescriptor, colorHandle);
            renderer.EnqueuePass(m_TransparentPostProcessPass);
        }
        else if (!renderingData.cameraData.isOffscreenRender && colorHandle != RenderTargetHandle.CameraTarget)
        {
            m_FinalBlitPass.Setup(baseDescriptor, colorHandle);
            renderer.EnqueuePass(m_FinalBlitPass);
        }

        if (renderingData.cameraData.isStereoEnabled)
        {
            renderer.EnqueuePass(m_EndXrRenderingPass);
        }

#if UNITY_EDITOR
        if (renderingData.cameraData.isSceneViewCamera)
        {
            m_SceneViewDepthCopyPass.Setup(DepthTexture);
            renderer.EnqueuePass(m_SceneViewDepthCopyPass);
        }
#endif
    }
}

Now add the HandheldArRendererSetup component to your main camera, and you should be done.

Make sure to disable all post-processing components you may have when in AR mode, and disable HDR in your LWRP config.

Final Notes

Many thanks to iBicha for explaining how to fix this issue with an earlier version of the lightweight render pipeline.

You can track this issue on Unity’s issue tracker, and vote for an official fix.

Please contact me via Twitter @NolanScobie if you have any questions!

Related