qtquick3d-architecture.qdoc

Go to the documentation of this file.

// Copyright (C) 2020 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
 
/*!
\page qtquick3d-architecture.html
\title Qt Quick 3D Architecture
\brief An overview of the architecture of Qt Quick 3D
\ingroup explanations-2dand3dgraphics
 
Qt Quick 3D extends Qt Quick to support the rendering of 3D content. It adds
extensive functionality, including several new public QML imports, as well as
a new internal scene graph and renderer. This document describes the
architecture of Qt Quick 3D from the public API to the details of how the
rendering pipeline works.
 
\section1 Module Overview
 
Qt Quick 3D consists of several modules and plugins that expose the
additional 3D APIs as well as utilities for conditioning and importing
existing 3D assets.
 
\section2 QML Imports
 
\list
    \li QtQuick3D - The main import which contains all the core components of
    Qt Quick 3D
    \li  \l{QtQuick3D.AssetUtils QML Types}{QtQuick3D.AssetUtils} - A library for importing 3D assets at runtime
    \li \l{Qt Quick 3D Helpers QML Types}{QtQuick3D.Helpers} - A library of additional components which can be
    used to help design 3D and debug 3D scenes.
\endlist
 
\section2 C++ Libraries
 
\list
    \li \l{Qt Quick 3D C++ Classes}{QtQuick3D} - The only public C++ module.
    Contains the definitions of all types exposed to the QtQuick3D QML import
    as well as a few C++ APIs
    \list
        \li QQuick3DGeometry - Subclass to create procedural mesh data
        \li QQuick3DTextureData - Subclass to create procedural texture data
        \li QQuick3D::idealSurfaceFormat - used to get the ideal surface format
    \endlist
    \li \c QtQuick3DAssetImport - An internal and private library to aid in
    importing assets and convert assets to QML.
    \li \c QtQuick3DRuntimeRender - An internal and private library that
    contains the spatial scene graph nodes and renderer.
    \li \c QtQuick3DUtils - An internal and private library used as a common
    utility library by all of the other C++ modules.
\endlist
 
\section2 AssetImporters Plugins
The asset import tooling is implemented using a plugin based architecture. The
plugins shipped with Qt Quick 3D extend the functionality of the asset importer
library and tool, \l{Balsam Asset Import Tool}{Balsam}.
\list
    \li Assimp - This plugin uses the 3rd party library libAssimp to convert
    3D assets in 3D interchange formats to Qt Quick 3D QML components.
\endlist
 
\section1 How does Qt Quick 3D fit into the Qt Graphics Stack
 
\image quick3d-graphics-stack.drawio.svg
 
The above diagram illustrates how Qt Quick 3D fits into the larger Qt
graphics stack. Qt Quick 3D works as an extension to the 2D Qt Quick API, and
when using 3D scene items in conjunction with View3D the scene will be
rendered via the Qt Rendering Hardware Interface (RHI).  The RHI will
translate API calls into the correct native rendering hardware API calls for
a given platform. The diagram above shows the options available for
each platform. If no native backend is explicitly defined, then Qt Quick will
default to a sensible native backend for rendering for each platform.
 
The integration between the Qt Quick 3D components of the stack and the Qt Quick
stack are described below in the next sections.
 
\section1 3D in 2D Integration
 
Displaying 3D content in 2D is the primary purpose of the Qt Quick 3D API. The
primary interface for integrating 3D content into 2D is the View3D component.
 
The View3D component works like any other QQuickItem derived class with
content and implements the virtual function QQuickItem::updatePaintNode. Qt
Quick calls updatePaintNode for all "dirty" items in the Qt Quick scenegraph
during the synchronization phase. This includes the 3D items managed by a
View3D, which also undergo their synchronization phase as a result of the
updatePaintNode call.
 
The updatePaintNode method of View3D performs the following actions:
\list
    \li Set up a renderer and render target if one doesn't exist already
    \li Synchronize items in the 3D scene via SceneManager
    \li Update any "dynamic" textures that were rendered by Qt Quick (\l {Texture Path}{2D in 3D Texture path} below)
\endlist
 
The rendering of the 3D scene, however, does not occur in the View3D
updatePaintNode method. Instead updatePaintNode returns a QSGNode subclass
containing the renderer for Qt Quick 3D, which will render the 3D scene during
the preprocess phase of the Qt Quick render process.
 
The plumbing for how Qt Quick 3D will render depends on which
View3D::renderMode is used:
 
\section2 Offscreen
 
The default mode for View3D is \l {View3D::renderMode}{Offscreen}. When using offscreen mode
View3D becomes a texture provider by creating an offscreen surface and
rendering to it. This surface can be mapped as a texture in Qt Quick and
rendered with a QSGSimpleTextureNode.
 
This pattern is very close to how QSGLayerNodes work already in Qt Quick.
 
\section2 Underlay
 
When using the \l {View3D::renderMode}{Underlay} mode the 3D scene is directly rendered to the
QQuickWindow containing the View3D. Rendering occurs as a result of the signal
QQuickWindow::beforeRenderPassRecording() which means that everything else in
Qt Quick will be rendered on top of the 3D content.
 
\section2 Overlay
 
When using the \l {View3D::renderMode}{Overlay} mode the 3D scene is directly rendered to the
QQuickWindow containing the View3D. Rendering occurs as a result of the signal
QQuickWindow::afterRenderPassRecording() which means that the 3D content will
be rendered on top of all other Qt Quick content.
 
\section2 Inline
 
The \l {View3D::renderMode}{Inline} render mode uses QSGRenderNode, which enables direct
rendering to Qt Quick's render target without using an offscreen surface. It
does this by injecting the render commands inline during the 2D rendering of
the Qt Quick Scene.
 
This mode can be problematic because it uses the same depth buffer as the
Qt Quick renderer, and z values mean completely different things in Qt Quick
vs Qt Quick 3D.
 
\section1 2D in 3D Integration
 
When rendering a 3D scene, there are many scenarios where there is a need to
embed 2D elements into 3D. There are two different ways to integrate 2D
content inside of 3D scenes, each of which has its own path to get to the
screen.
 
\section2 Direct Path
 
The direct path is used to render 2D Qt Quick content as if it existed as an
flat item in the 3D scene. For example, consider the following scene
definition:
 
\code
Node {
    Text {
        text: "Hello world!"
    }
}
\endcode
 
What happens here is: when a child component is set on
a spatial node of type QQuickItem, it is first wrapped by a
QQuick3DItem2D, which is just a container that adds 3D coordinates to a 2D item.
This sets the base 3D transformation for how all further 2D children are
rendered so that they appear correctly in the 3D scene.
 
When the time comes to render the scene, these 2D items' QSGNodes are passed to
the Qt Quick Renderer to generate the appropriate render commands. Because the
commands are done inline and take the current 3D transformation into
consideration, they are rendered exactly the same as in the 2D renderer, but
show up as if they were rendered in 3D.
 
The drawback of this approach is that no lighting information of the 3D scene
can be used to shade the 2D content, because the Qt Quick 2D renderer has no
concept of lighting.
 
\section2 Texture Path
 
The texture path uses a 2D Qt Quick scene to create dynamic texture
content. Consider the following Texture definition:
 
\code
Texture {
    sourceItem: Item {
        width: 256
        height: 256
        Text {
            anchors.centerIn: parent
            text: "Hello World!"
        }
    }
}
\endcode
 
This approach works in the same way that Layer items work in Qt Quick, in that
everything is rendered to an offscreen surface the size of the top-level Item,
and that offscreen surface is then usable as a texture that can be reused
elsewhere.
 
This Texture can then be used by materials in the scene to render Qt Quick
content on items.
 
\section1 Scene Synchronization
 
\section2 Scene Manager
 
The scene manager in Qt Quick 3D is responsible for keeping track of spatial
items in a 3D scene, and for making sure that items are updating their
corresponding scene graph nodes during the synchronize phase.  In Qt Quick,
this role is performed by QQuickWindow for the 2D case. The scene manager is
the primary interface between the frontend nodes and the backend scene graph
objects.
 
Each View3D item will have at least one Scene Manager, as one is created and
associated with the built-in scene root on construction. When spatial nodes
are added as children of the View3D, they are registered with the View3D's
scene manager. When using an imported scene, a second SceneManager is created
(or referenced if one exists already) to manage the nodes that are not direct
children of the View3D. This is needed because, unlike the View3D, an
imported scene doesn't exist on a QQuickWindow until it is referenced. The
additional SceneManager makes sure that assets belonging to the imported
scene are created at least once per QQuickWindow they are referenced in.
 
While the scene manager is an internal API, it is important to know that the
scene manager is responsible for calling updateSpatialNode on all objects that
have been marked dirty by calling the update() method.
 
\section2 Frontend/Backend Synchronization
 
The objective of synchronization is to make sure that the states set on the
frontend (Qt Quick) match what is set on the backend (Qt Quick Spatial Scene
Graph Renderer). By default the frontend and backend live in separate threads:
the frontend in the Qt Main thread, and the backend in Qt Quick's render thread. The
synchronization phase is where the main thread and render thread can safely
exchange data. During this phase, the scene manager will call updateSpatialNode for each dirty
node in the scene. This will either create a new backend node or update an
existing one for use by the renderer.
 
\section2 Qt Quick Spatial Scene Graph
 
Qt Quick 3D is designed to use the same frontend/backend separation pattern
as Qt Quick: frontend objects are controlled by the Qt Quick engine, while
backend objects contain state data for rendering the scene. Frontend objects
inherit from QObject and are exposed to the Qt Quick engine. Items in QML
source files map directly to frontend objects.
 
As the properties of these frontend objects are updated, one or more backend nodes
are created and placed into a scenegraph. Because rendering 3D scenes
involves a lot more state than rendering 2D, there is a separate set of specialized scene
graph nodes for representing the state of the 3D scene objects.
This scene graph is know as the Qt Quick Spatial Scene Graph.
 
Both the frontend objects and backend nodes can be categorized into two classes.
The first are spatial, in the sense that they exist somewhere in the in 3D space.
What this means in practice is that each of these types contains a transform
matrix. For spatial items the parent child relationship is significant because
each child item inherits the transform of its parents.
 
The other class of items are resources. Resource items do not have a position
in 3D space, but rather are just state that is used by other items. There can
be a parent-child relationship between these items, but it has no other meaning
than ownership.
 
Unlike the 2D scene graph in Qt Quick, the spatial scene graph exposes resource
nodes to the user directly. So for example in Qt Quick, while QSGTexture is
public API, there is no QQuickItem that exposes this object directly. Instead
the user must either use an Image item, which describes both where the texture
comes from as well as how to render it, or write C++ code to operate on the
QSGTexture itself. In Qt Quick 3D these resources are exposed directly in the
QML API. This is necessary because resources are an important part of the scene
state. These resources can be referenced by many objects in the scene: for
example, many Materials could use the same Texture.  It is also possible to
set properties of a Texture at runtime that would directly change how a texture
is sampled, for example.
 
\section3 Spatial Objects
 
All spatial Objects are subclasses of the Node component, which contains the
properties defining the position, rotation, and scale in 3D space.
 
\list
    \li  \l [QtQuick3D QML] {Node}{Node}
    \li  \l [QtQuick3D QML] {Light}{Light}
        \list
            \li DirectionalLight
            \li PointLight
            \li SpotLight
        \endlist
    \li \l [QtQuick3D QML] {Camera}{Camera}
        \list
            \li PerspectiveCamera
            \li OrthographicCamera
            \li FrustumCamera
            \li CustomCamera
        \endlist
    \li \l [QtQuick3D QML] {Model}{Model}
    \li Loader3D
    \li Repeater3D
    \li \l [QtQuick3D QML] {Skeleton}{Skeleton}
    \li \l [QtQuick3D QML] {Joint}{Joint}
\endlist
 
\section3 Resource Objects
 
Resource objects are subclasses of the Object3D component. Object3D is just a
QObject subclass with some special helpers for use with the scene manager.
Resource objects do have parent/child associations, but these are mostly useful
for resource ownership.
 
\list
    \li \l [QtQuick3D QML] {Texture}{Texture}
    \li \l [QtQuick3D QML] {TextureData}{TextureData}
    \li \l [QtQuick3D QML] {Geometry}{Geometry}
    \li \l [QtQuick3D QML] {Material}{Material}
        \list
            \li DefaultMaterial
            \li PrincipledMaterial
            \li CustomMaterial
        \endlist
    \li \l [QtQuick3D QML] {Effect}{Effect}
    \li SceneEnvironment
\endlist
 
\section3 View3D and Render Layers
 
With regard to the frontend/backend separation, View3D is the separation
point from the user perspective because a View3D is what defines what scene
content to render. In the Qt Quick Spatial Scene Graph, the root node for a
scene that will be rendered is a Layer node. Layer nodes are created by the
View3D using a combination of the the View3D's properties and the properties
of the SceneEnvironment. When rendering a scene for a View3D, it is this Layer
node that is being passed to the renderer to render a scene.
 
\section1 Scene Rendering
 
\image qtquick3d-rendergraph.drawio.svg
 
\section2 Set up Render Target
 
The first step in the rendering process is to determine and set up the scene
render target. Depending on which properties are set in the SceneEnvironment,
the actual render target will vary. The first decision is whether content is
being rendered directly to a window surface, or to an offscreen texture.
By default, View3D will render to an offscreen texture. When using post
processing effects, rendering to an offscreen texture is mandatory.
 
Once a scene render target is determined, then some global states are set.
\list
    \li window size - if rendering to a window
    \li viewport - the size of the scene area being rendered
    \li scissor rect - the subset of a window that the viewport should be
    clipped to
    \li clear color - what color to clear the render target with, if any.
\endlist
 
\section2 Prepare for Render
 
The next stage of rendering is the prepare stage where the renderer does
house-keeping to figure out what needs to be rendered for a given frame,
and that all necessary resources are available and up to date.
 
The prepare stage itself has two phases: the high-level preparation of
determining what is to be rendered and what resources are needed; and the
low-level preparation that uses RHI to actually set up rendering pipelines and
buffers, as well as setting up the rendering dependencies of the main scene pass.
 
\section3 High level render preparation
 
The purpose of this phase is to extract the state of the spatial scene graph
into something that can be used to create render commands. The overview here is
that the renderer is creating lists of geometry and material combinations to
render from the perspective of a single camera with a set of lighting states.
 
The first thing that is done is to determine the global common state for all
content. If the SceneEnvironment defines a \l {SceneEnvironment::lightProbe}{lightProbe}, then it checks if the
environment map associated with that light probe texture is loaded, and if its
not, a new environment map is is loaded or generated.  The generation of
an environment will itself be a set of passes to convolve the source texture
into a cube map. This cube map will contain both specular reflection information
as well as irradiance, which is used for material shading.
 
The next thing is that the renderer needs to determine which camera in the
scene to use. If an active camera is not explicitly defined by a View3D, the
first camera available in the scene is used. If there are no cameras
in the scene, then no content is rendered and the renderer bails out.
 
With a camera determined, it is possible to calculate the projection matrix
for this frame. The calculation is done at this point because each renderable
needs to know how to be projected. This also means that it is now possible to
calculate which renderable items should be rendered. Starting with the list of
all renderable items, we remove all items that are not visible because they
are either disabled or completely transparent. Then, if frustum culling is
enabled on the active camera, each renderable item is checked to see if it is
completely outside of the view of the camera's frustum, and if so it is
removed from the renderable list.
 
In addition to the camera projection, the camera direction is also calculated
as this is necessary for lighting calculations in the shading code.
 
If there are light nodes in the scene, these are then gathered into a list the
length of the maximum available lights available. If more light nodes exist in
the scene than the amount of lights the renderer supports, any additional
light nodes over that limit are ignored and don't contribute to the lighting of
the scene.  It is possible to specify the scope of light nodes, but note that
even when setting a scope the lighting state of each light is still sent to
every material which has lighting, but for lights not in scope the brightness
will be set to 0, so in practice those lights will not contribute to the
lighting of those materials.
 
Now with a hopefully shorter list of renderables, each of these items need to
be updated to reflect the current state of the scene. For each renderable we
check that a suitable material is loaded, and if not a new one is created.
A material is a combination of shaders and a rendering pipeline, and it is needed
for creating a draw call. In addition the renderer makes sure that any
resources needed to render a renderable is loaded, for example geometry and
textures set on the Model. Resources that are not loaded already are
loaded here.
 
The renderables list is then sorted into 3 lists.
\list
    \li Opaque Items: these are sorted from front to back, or in other words
    from items that are closest to the camera to items that are furthest from the
    camera. This is done to take advantage of hardware occlusion culling or
    early z detection in the fragment shader.
    \li 2D Items: these are QtQuick Items that are rendered by the Qt Quick
    renderer.
    \li Transparent Items: these are sorted from back to front, or in other
    words from items that are farthest from the camera to items that are nearest
    to the camera. This is done because transparent items need to be blended
    with all items that are behind them.
\endlist
 
\section3 Low Level render preparation
 
Now that everything that needs to be considered for this frame has been
determined, the plumbing and dependencies for the main render pass can be
addressed. The first thing that is done in this phase is to render any
pre-passes that are required for the main pass.
 
\list
    \li Render DepthPass - Certain features like Screen Space Ambient Occlusion
    and Shadowing require a depth pre-pass. This pass consists of all opaque
    items being rendered to a depth texture.
 
    \li Render SSAOPass - The objective of the Screen Space Ambient Occlusion
    pass is to generate an ambient occlusion texture. This texture is used
    later by materials to darken certain areas when shading.
 
    \li Render ShadowPasses - Each light in the scene that has shadow enabled,
    contributes to an additional shadow pass. There are two different shadowing
    techniques employed by the renderer, so depending on the light types there
    will be different passes. When rendering shadows from a directional light,
    the scene is rendered to a 2D occlusion texture from a combination of the
    directional light's direction and the size of the camera frustum. When
    rendering shadows from a point or spot light the light's occlusion texture is
    a cube map representing the occlusion contribution relative to each face
    direction of the light.
 
    \li Render ScreenTexture - This pass will only occur when using a
    CustomMaterial that requires a screen texture, which can be used for
    rendering tecniques such as refraction. This pass works like a depth pass,
    but instead renders all opaque items to a color texture.
\endlist
 
After the dependency renders are done, the rest of the passes are prepared but
not rendered. This preparation involves taking the state gathered in the
high-level prep stage and translating that to graphics primitives like
creating/updating uniform buffers values, associating samplers with dependency
textures, setup for shader resource bindings, and everything else involved in
creating a pipeline state necessary for performing a draw call.
 
\section2 Scene Rendering
 
Now that the hard work of preperation is done, the easy part is running the
commands that contribute to the main scene's content. That rendering works
in this order:
 
\list
    \li Clear Pass - This isn't really a pass, but depending on what
    backgroundMode is set on SceneEnvironment, different things can happen here.
    If the background mode is either transparent or color, then the color buffer
    will be cleared with either transparency or the color specified. If, however,
    the background mode is set to SkyBox, then a pass will be run that renders
    the SkyBox from the perspective of the camera, which will also fill the buffer
    with initial data.
 
    \li Opaque Pass - Next all opaque items will be drawn. This just involves
    setting the pipeline state, and running the draw command for each item in
    the order in the list since they are already sorted at this point.
 
    \li 2D Pass - If there are any 2D items in the scene, then the Qt Quick
    renderer is invoked to generate the render commands necessary to render
    those items.
 
    \li Transparent Pass - Then finally the transparent items in the scene are
    rendered one by one in the same manner as the opaque items.
\endlist
 
This concludes the rendering of the scene.
 
\section2 Post-Processing
 
If any post-processing functionality is enabled, then it can be assumed that the
result of the scene renderer was a texture that is an input for the post
processing phase. All post-processing methods are additional passes that
operate on this scene input texture.
 
All steps of the Post-Processing phase are optional, and if no built-in
features and no user-defined effects are enabled, the output of the scene
render is what is used by the final render target. Note however that
\l{ExtendedSceneEnvironment::tonemapMode}{tonemapping} is enabled by default.
 
\image qtquick3d-postprocess-graph.drawio.svg
 
\section3 Built-in Post-Processing
 
\l ExtendedSceneEnvironment and its parent type \l SceneEnvironment offer the
most common effects used in 3D scenes, as well as tonemapping that is used to
map the high dynamic range color values generated by the renderer to the 0-1
LDR range. The effects include depth of field, glow/bloom, lens flare,
vignette, color adjustment and grading, fog, and ambient occlusion.
 
\section3 Post-Processing Effects
 
Applications can specify their own custom post-processing effects as an ordered
list in the SceneEnvironment::effects property. When this list is non-empty,
the effects in it are applied \e before the built-in effects provided by \l
ExtendedSceneEnvironment. Each post-processing effect is part of a chain such
that the output of the previous effect is the input for the next. The first
effect in this chain gets its input directly from the output of the scene
renderer step. It is also possible for effects to access the depth texture
output of the scene renderer.
 
Each effect in this process can consist of multiple sub-passes, which means it
is possible to render content into intermediate buffers. The final pass of a
multi-pass effect is expected to output a single texture containing the color
data to be used by the next steps of the post-processing phase.
 
\section3 Temporal and Progressive Antialiasing
 
The Temporal and Progressive antialiasing steps are optionally enabled by
setting properties in the SceneEnvironment. While not strictly a part of the
post-processing phase, the actual results of Temporal and Progressive
antialiasing are realized during the post-processing phase.
 
Temporal Antialiasing is performed when a scene is being actively updated.
When enabled, the active camera makes very small adjustments to the camera
direction for each frame while drawing the scene. The current frame is then
blended with the previously rendered frame to smooth out what was rendered.
 
Progressive Antialiasing is only performed when a scene is not being updated.
When enabled, an update is forced and the current state of the scene is
rendered with very small adjustments to the active cameras direction. Up to 8
frames are accumulated and blended together with pre-defined weights. This has
the effect of smoothing out a non-animating scene, but comes at a
performance cost because several extra frames will be rendered for each update.
 
\section3 Super Sampling Antialiasing (SSAA)
 
Super Sampling Antialiasing is a brute force way of smoothing out a scene. It
works by rendering to a texture that is a multiple of the requested size of
the scene, and then afterwards downsampling it to the target size. So for
example if 2X SSAA is requested, then the scene would be rendered to a texture
that is 2 times the intended size, and then downsampled as part of this
phase. This can have a huge impact on performance and resource usage so
should be avoided if possible.  It's also possible for the View3D size to be
too large to use this method, since the texture needed for this method may be
larger than what is supported by the rendering hardware.
 
*/