Structured Graph Variational Autoencoders for

Indoor Furniture layout Generation

Aditya Chattopadhyay ∗ Xi Zhang

Department of Computer Science [Link], Inc.
Johns Hopkins University xizhn@[Link]
achatto1@[Link]
arXiv:2204.04867v3 [[Link]] 22 Jul 2022

David Paul Wipf Himanshu Arora René Vidal

[Link], Inc. [Link], Inc. [Link], Inc.
daviwipf@[Link] arorah@[Link] rvidal@[Link]

Abstract
We present a graph variational autoencoder with a structured prior for generating
the layout of indoor 3D scenes. Given the room type (e.g., living room or library)
and the room layout (e.g., room elements such as floor and walls), our architecture
generates a collection of objects (e.g., furniture items such as sofa, table and chairs)
that is consistent with the room type and layout. This is a challenging problem
because the generated scene should satisfy multiple constrains, e.g., each object
should lie inside the room and two objects should not occupy the same volume.
To address these challenges, we propose a deep generative model that encodes
these relationships as soft constraints on an attributed graph (e.g., the nodes capture
attributes of room and furniture elements, such as class, pose and size, and the
edges capture geometric relationships such as relative orientation). The architecture
consists of a graph encoder that maps the input graph to a structured latent space,
and a graph decoder that generates a furniture graph, given a latent code and
the room graph. The latent space is modeled with auto-regressive priors, which
facilitates the generation of highly structured scenes. We also propose an efficient
training procedure that combines matching and constrained learning. Experiments
on the 3D-FRONT dataset show that our method produces scenes that are diverse
and are adapted to the room layout.

1 Introduction
The last few years have seen significant advances in image generation powered by the emergence
of deep generative models such as GANs Goodfellow u. a. (2014) and VAEs Kingma und Welling
(2014). State-of-the-art methods are able to generate images of a single object category (e.g., faces)
with amazingly realistic quality (e.g., Karras u. a. (2020)). However, the problem of generating
images of complex scenes composed of multiple objects in diverse arrangements remains a challenge.
As an example, images of indoor scenes consist of room elements (floor, walls, etc.) and furniture
items (table, chairs, beds, etc.) arranged in different ways depending on the room type (living room,
bedroom, etc.). Moreover, room elements and furniture items should satisfy geometric constraints,
e.g., each object must lie inside the room and on the floor, two objects cannot occupy the same
volume, some objects tend to co-occur in particular orientations relative to the room layout.
Recent work on indoor scene image generation Gadde u. a. (2021) aims to address the challenge
of generating complex indoor scenes by using GANs with multiple discriminators that specialize
∗
This work was done during Aditya’s internship with Amazon
in localizing different objects within an image. By adding a “broker” to mediate among such
discriminators, Gadde u. a. (2021) achieves state-of-the-art (SOTA) results on synthesizing images of
living rooms. However, such SOTA image generation models are far from capturing the rich structure
present in indoor scenes. For example, Para u. a. (2021) notice that these models fail to respect
the relationships between scene objects and often cannot preserve certain shapes like axis-aligned
polygons. We contend that addressing such complex image generation problems requires reasoning
about the scene content in 3D space.
As a stepping stone, this paper focuses on the problem of conditional generation of the scene’s 3D
layout, rather than a 2D image, though we can synthesize images using a renderer given the layout.
Specifically, we assume we are given the room type (e.g., living room or bedroom) and the room
layout (spatial arrangement of walls, windows and doors), and our goal is to generate a collection of
objects (e.g., furniture items such as sofa, coffee table and chairs) that is consistent with the room type
and layout. For example, a bedroom must consist of a bed, typically placed in the center of the longest
wall in the room. Moreover, we expect the generator to synthesize diverse object arrangements for
the same room. This problem of conditional 3D layout generation is important in applications such
as room decoration, where the goal is to produce diverse decors for a given room.
Recent work Armeni u. a. (2019); Wang u. a. (2018); Keshavarzi u. a. (2020) aims to address this
problem using supervision in the form of scene hierarchies or relational graphs. However, the
contextual space of possible arrangements objects in a room is simply too large to be modeled using
hand-crafted heuristics or hierarchies. This has led to recent efforts on training networks directly
from data using autoregressive models based on CNNs Ritchie u. a. (2019) or Transformers Wang
u. a. (2020); Paschalidou u. a. (2021). These models, while being adept at generating indoor scenes,
lack the advantages of a learnt latent space as in Variational Autoencoders (VAEs). The VAE latent
space is often a good representation of data which can bootstrap several downstream applications. In
this work, we show one such application of furniture recommendation given a floorplan by retrieving
the most “appropriate" furnished room from a database curated by human designers and adapting it
to the new floorplan. The presence of a learnt latent space allows us to naturally define a notion of
“appropriateness" which is otherwise a non-trivial problem (more details in §4). Moreover, users can
often traverse the latent space to manipulate generated samples allowing more controlled generations
Higgins u. a. (2016); Kumar u. a. (2017) (see Figure 4). Such manipulations are not easy to implement
in autoregressive models. Unfortunately, we found in our experiments that existing graph-based
VAE architectures are insufficient for indoor scene generation. This observation is echoed by Para
et [Link] u. a. (2021) in their work on 2D layout generation, and they conjecture that current VAE
architectures struggle with the discrete nature of graphs and layouts.
To remedy this, we propose a graph-based VAE model for synthesis of 3D indoor scenes conditioned
on the room type and layout (floorplan). We represent both the room and furniture layouts with
an attributed graph. We then present a scene generative model consisting of a graph encoder that
maps the input graph to a latent space, and a graph decoder that generates a furniture graph, given
a latent code and the room graph. Our model considerably reduces the performance gap between
VAEs and state-of-the-art autoregressive models Paschalidou u. a. (2021) for indoor scene synthesis.
Specifically, we make the following contributions:

1. A structured auto-regressive prior for graphs: This is our main contribution. Contemporary
graph-VAE architectures typically encode the graph into a single latent vector and then use a
multi-layered perceptron (MLP) to decode it back to a graph Simonovsky und Komodakis (2018);
Kipf und Welling (2016a). In contrast, we propose to have a separate latent code for each furniture
item. This has been previously explored in Luo u. a. (2020) where the authors assume an i.i.d.
Gaussian prior over the latents. This limits performance since the graph decoder struggles to learn
complex relationship between different furniture nodes from i.i.d latent codes as input.2 Instead,
we propose a novel autoregressive prior based on linear Gaussian models which allows the model
to learn a dependency structure between the different latent variables corresponding to different
furniture items in the scene. We also propose an efficient way to compute the KL divergence term
in the VAE objective which requires a matching procedure since there is no canonical ordering of
graph nodes.
2. Learning graph-VAEs under constraints: To facilitate learning, we use simple intuitive constraints
like limiting the relative distances between furniture items, such as a chair and a table. These
2
This is corroborated by our experiments in §4 where we consider this architecture as Baseline B1.

2
 can be easily computed from training data. We then train our VAE model under these constraints
utilizing a recently introduced constrained learning framework Chamon und Ribeiro (2020).
3. Experimental evaluation: Through extensive experiments we show that our proposed model
achieves considerable improvement over baseline VAE architectures bringing the performance
of latent-variable models at scene synthesis closer to what can be achieved using autoregressive
models. Moreover, we show a unique application of our latent-variable model which is not possible
with autoregressive models. Finally, we show how one can edit the generated scenes post-hoc by
traversing the latent space.

2 Related Work

Graph-based inference: Graphical representation of scenes and graph-based inference has been
extensively studied in the past. Early works Fisher u. a. (2012, 2011); Yeh u. a. (2012); Jiang u. a.
(2018); Qi u. a. (2018) employed “shallow" methods like hand-crafted graph kernels or probabilistic
graphical models to learn the furniture arrangements. Recent works leverage deep generative models
to learn good scene representations directly from spatial data. The community has explored avenues
for combining graphs with VAEs to synthesize 3D scenes Li u. a. (2019); Zhang u. a. (2020); Luo u. a.
(2020). However, all these methods rely on strong heuristics on defining object relations. For instance,
Luo u. a. (2020) relies on user-defined scene-graphs as input, Li u. a. (2019) requires hand-crafted
hierarchies, and Purkait u. a. (2020) uses heuristics to extract context-free grammars from data which
are then used to train a grammar-VAE Kusner u. a. (2017). Our proposed VAE is different from all
these methods in that we do not use any such strong heuristics on object relations, but only a check
of association between a furniture item and a room element using a distance measure. On the other
hand, Wang et [Link] u. a. (2019) uses graphs for high-level planning of the furniture layout of the
room in a 2-stage approach where they train a generator to synthesize scene graphs followed by a
CNN to propose consistent furniture poses. Their model has no latent variables and is slower due to
the 2-stage process.
Autoregressive Scene Generation. Recent successful models for indoor scene synthesis are all
autoregressive in nature Wang u. a. (2018); Ritchie u. a. (2019); Wang u. a. (2020); Paschalidou
u. a. (2021). Wang et al. Wang u. a. (2020) introduced an autoregressive scene generation pipeline,
Sceneformer, using multiple transformers Vaswani u. a. (2017) which predict objects’ category,
location and size separately. Concurrently, FastSynth Ritchie u. a. (2019) introduced a similar
pipeline, where the authors train separate CNNs based on a top-down representation of the scene to
sequentially insert objects into the scene. Their method however requires auxilary supervision in the
form of depth and semantic segmentation maps. Another recent transformer-based autoregressive
approach was proposed in Paschalidou u. a. (2021), which replaces the multiple trained models of
past works with a single unified model. Unlike these models, we learn an end-to-end latent variable
model to generate 3D indoor scenes trained from spatial data.
Expressive latent distributions for VAEs. There has been extensive work into designing expressive
distributions for VAEs. For example, Klushyn u. a. (2019) proposes a hierarchial prior, Rezende
und Mohamed (2015) uses normalizing flows to model a more expressive posterior distribution over
latents, Chen u. a. (2016) uses an autoregressive prior, and Tomczak und Welling (2018) advocates the
use of a mixture distribution for the prior based on the posterior distribution of the encoder. However,
learning expressive distributions for latent spaces which are expressed as graphs is challenging due to
the absence of any canonical ordering between the different nodes of a graph. This makes computing
the required KL divergence term in the ELBO notoriously difficult. In this work we propose to model
the latent space as an autoregressive linear Gaussian model which allows us to formulate the ordering
as a Quadratic Assignment problem for which we also propose an efficient approximation.

3 Our Approach

This section describes the proposed model. First, we describe how we represent the 3D scene layout
with an attributed graph. Next, we describe our architecture. The VAE encoder is a Graph Neural
Network (GNN) that processes the graph and produces latent variables which are then passed to
another GNN which serves as the VAE decoder. Then, we describe how we parameterize the prior
on the latent space using an autoregressive model which is learnt. Finally, we describe the proposed

3
Figure 1: (a) Overview of the proposed model. The input scene graph includes the furniture and
room sub-graphs and is complete (we omit depicting certain edges to prevent clutter). The Encoder
predicts a mean and variance per furniture node, which are then used to sample latents for proposing
furniture nodes by the decoder. During inference we use the proposed autoregressive prior to generate
the latents, which are then subsequently processed by the decoder for scene synthesis; (b) Generated
scenes. Sample generations for our model for a bedroom (row 1) and a library (row 2). Green
rectangles indicate doors while blue rectangles indicate windows

training methodology, which includes some constraints for faster convergence. A schematic depiction
of our approach is given in Figure 1a with more details in the Appendix.

3.1 Indoor scene representation as a graph

We represent an indoor scene as an attributed graph G = (V, E, X). Here the nodes V denote
room layout components (floor, wall, windows) and furniture items (sofa, chair, bed), the edges
E ⊆ V × V denote relationships between the nodes (e.g., relative orientation of sofa to wall), and
the attributes X denote features associated with the nodes and edges (e.g., location of furniture
items or relative orientation between bed and wall). The graph has two node types (room nodes
VR , furniture nodes VF ), three edge types (room-room edges ERR , room-furniture edges ERF
and furniture-furniture edges EF F ), and five attribute types corresponding to these node and edge
types (XR , XF , XRR , XRF and XF F ). In this work, we consider the graph as complete, that is,
ERF = VR × VF , EF F = VF × VF and ERR = VR × VR .
We will identify two main subgraphs of G. The room layout graph GR = (VR , ER , XR , XRR )
consists of nR := |VR | nodes (or room elements) and eR := |ER | edges, where the node attributes

4
XR ∈ RnR ×dR denote the class, location, orientation and size of the room element, and the edge
attributes XRR ∈ ReR ×dRR encode geometric or functional relationships between two room elements
(relative location, relative orientation etc.). The room type T is also encoded as a categorical feature
into XR . Similarly, the furniture layout graph GF = (VF , EF , XF , XF F ) consists of nF := |VF |
nodes (or furniture items) and eF = |EF | edges, where the node attributes XF ∈ RnF ×dF denote
the class of the furniture item, its location, orientation, size, and its 3D shape descriptor, and its
edge attributes XF F ∈ ReF ×dF F encode geometric or functional relationships between two furniture
items. We obtain the shape descriptors of each furniture item by processinga a 3D point cloud of the
item through PointNet Qi u. a. (2017) pretrained on the Stanford ShapeNet dataset Chang u. a. (2015).

3.2 Proposed Generative Model

We would like to design and learn a probabilistic model p(GF | GR , T, nF ) that generates a furniture
layout GF given the room layout GR , type T (say, bedroom or library room) and number of furniture
items nF to place in the room. We assume there exists latent Z such that p(GF | GR , T, nF ) =
Z
p(GF |Z, nF , GR , T )p(Z|nF , GR , T )dZ. (1)

Our proposed model consists of three main ingredients:

1. An encoder, qϕ (Z | G, T, nF ), which maps both room and furniture layouts (GR and GF resp.)
as well as room type and number of furniture items to a latent variable Z, which captures the
diversity of room-aware furniture layouts. The parameters of the encoder are denoted as ϕ.
2. A decoder, pθ′ (GF | Z, nF , GR , T ), which maps the number of furniture items, the latent variable
as well as the room layout and type to a furniture layout. The parameters of the decoder are
denoted as θ′ .
3. A prior model pθ′′ (Z | nF , GR , T ). The difference between the prior model and the encoder is
that the prior model only considers the room layout GR and not G. The parameters of the prior
model are denoted as θ′′ .

The encoder, decoder and prior model are parameterized with GNNs which we will describe in the
next subsection.
Given a training set, consisting of indoor scenes in the form of attributed graphs G =
{G1 , G2 , ..., Gn }, we learn the parameters {θ′ , θ′′ , ϕ} by optimizing the following empirical av-
erage over the training set:
n
′ ′′ 1X
L(θ , θ , ϕ) = L (Gi , θ′ , θ′′ , ϕ), (2)
n i=1

where, for any G ∈ G, L(G, θ′ , θ′′ , ϕ) denotes the Evidence Lower Bound (ELBO) defined as:
L (G, θ′ , θ′′ , ϕ)
= EZ∼qϕ (Z|G,T,nF ) [log pθ′ (GF | Z, nF , GR , T )] (3)
− KL(qϕ (Z | G, T, nF ) || pθ′′ (Z | nF , GR , T )).
As mentioned in §1, we optimize (2) under constraints to speed up convergence, as we will describe
in subsection 3.3.
Graph Encoder. The encoder models the approximate posterior of a latent variable Z given
(G, T, nF ). We assume that the distribution of Z, qϕ (Z | G, T, nF ), is Gaussian with mean
µϕ (G, T, nF ) and a diagonal covariance matrix with diagonal entries σϕ (G, T, nF ). The distri-
bution parameters (µϕ , σϕ ) are modeled as the output of an attention-based message passing graph
neural network (MP-GNN) with weights ϕ. The design of the MP-GNN is inspired by Mavroudi u. a.
(2020), where each layer l = 1, . . . , L of the MP-GNN maps a graph Gl−1 to another graph Gl by
updating the graph’s node and edge features. Specifically, let hli and hlij denote the features of node i
and edge (i, j) of graph Gl , respectively. Let the input to the network be the graph G0 = G, so that
h0i and h0ij denote the node features (rows of XR and XF ) and edge features (rows of XRR , XF F
and XRF ), respectively. At each iteration of node and edge refinement, the MP-GNN: (1) adapts
the scalar edge weights by using an attention mechanism; (2) updates the edge attributes depending

5
on the edge type, the attention-based edge weights, the attributes of the connected nodes and the
previous edge attribute; and (3) updates the node attribute by aggregating attributes from incoming
edges.
After L layers of refinement, we obtain a graph GL , whose node features are mapped via a lin-
ear layer with weights Wµ , Wσ to obtain the parameters of the Gaussian model as µiϕ (G, T ) =
i , σϕ (G, T ) = exp(Wσ hi ) where i = 1, . . . , nF . Note that there is a different Gaus-
W µ hL i L

sian for each node of the graph. Therefore, the output of the encoder will be two matrices
µϕ (G, T, S) ∈ RnF ×dF and σϕ (G, T, S) ∈ RnF ×dF corresponding to the mean and standard
deviation vectors of the latent variable matrix Z ∈ RnF ×dF .
Graph Decoder. The decoder maps (nF , Z) and the room layout, type (GR , T ) to a desired furniture
layout via the distribution pθ′ (GF | Z, nF , GR , T ). The generative process proceeds as follows,
An initial fully connected furniture graph G0F is instantiated. Each node of G0F is associated with
a feature of dimension dF corresponding to one of the rows of Z. Each edge (i, j) of G0F of type
ϵ ∈ {RF, F F } is associated with a feature Zij = (Zi , Zj ) as the concatenation of the node features.
As a result, we obtain an initial graph G0 that includes both the initial furniture graph G0F as well as
the given room graph GR as subgraphs. The initial graph G0 is passed to a MP-GNN, which follows
the same operations as the encoder MP-GNN.
The output of the MP-GNN is the furniture subgraph GL F of the final graph G . Each furniture node
L

of G is then individually processed through a MLP to produce parameters for the furniture layout
L

graph distribution pθ′ (GF | Z, nF , GR , T ), which can be factorized in the following way:

pθ′ (GF | Z, nF , GR , T ) (4)

YnF
= pθ′ (shapei | Z, GR , T )pθ′ (orieni | Z, GR , T )
i=1
pθ′ (loci | Z, GR , T )pθ′ (sizei | shapei )pθ′ (cati | shapei ).

More specifically, we assume that given latent Z, room layout GR and type T , the furniture features
are independent of each other (a standard assumption in the VAE literature). For a furniture, we
further assume that shape, orientation and location features are independent given Z, GF and T .
However, since the PointNet shape features implicitly capture the configuration of the furniture item
in 3D space we condition the size and category distributions on the shape feature.
We parameterize the shape and location features as a normal distribution, the size feature as a
lognormal distribution whose support is restricted to be positive valued, category and orientation
features as categorical distribtions. Since both the Encoder and Decoder graphs have nF nodes that
are in one-to-one correspondence, we can define our reconstruction loss (first term in (3)) by simply
comparing their node features without the need for an explicit matching procedure.
Graph Prior. Recall our latent space Z is modelled such that there is a latent variable corresponding
to each furniture node in the graph. Many popular graph VAE models assume an i.i.d. normal prior
for each node Kipf und Welling (2016b); Luo u. a. (2020). However, such a model is restrictive for
our purposes. MP-GNNs achieve permutation equivariance by sharing the weight matrices across
every node in the graph. When the graph is complete, as is the case here, the marginal distribution
of every output node after L GNN layers will be identical if they are initialized as i.i.d. Gaussian at
the input layer. Since, the output nodes of the decoder GNN after L layers correspond to different
furniture features, having identical marginals is detrimental. This claim is supported by experiments
(Figure 2) where the i.i.d. prior baseline models struggle to learn proper furniture placements. To
remedy this, we propose to parameterize prior distribution as an autoregressive model based on linear
gaussian models Bishop (2006). More specifically,

p(Z 0 | GR , T ) = N (µθ′′ (GR , T ), σθ0′′ (GR , T )), (5)

X
p(Z i |Z k<i , GR , T ) = N ( Akθ′′ (GR , T )Z k , σθi ′′ (GR , T )).
k<i
(6)

6
where Z i refers to the latent corresponding to the ith furniture node. Thus, the ith furniture node
latent is given by a Gaussian whose mean is a linear function of all the latents k < i. Such a structure
ensures that all the latent variables are jointly Gaussian. This allows us to analytically compute the
KL divergence term and thus was favoured over more expressive probabilistic models which would
introduce more stochasticity in the objective due to the need of estimating the KL divergence term
via sampling. We implement (5) (see also Figure 1) with two networks:

• Room Aggregator for p(Z 0 | GR , T ): The room aggregator is an MP-GNN with the same
architecture as the Graph Encoder. except that the input to the network is just (GR , T ) with the
node and edge features initialized to XR and XRF , respectively. After L GNN layers, all the room
node features are aggregated by a mean pooling operation to obtain a global representation of the
agg agg
room layout plan XR . This XR is then passed through an MLP to compute µθ′′ (GR , T ) and
0
σθ′′ (GR , T ).
• RNN Prior for p(Z i |Z k<i , GR , T ): We use a recurrent neural network to predict the matrix
agg
Akθ′′ (GR , T ) and the variance σθi ′′ (GR , T ) at each node index. The RNN is initialized with XR .
We need additional constraints on each Aθ′′ (GR , T ) to prevent the dynamics model in (5) from
k

diverging to infinity. This is typically done by controlling the spectral radius or its proxy, the
spectral norm Lacy und Bernstein (2003), of the matrices {Akθ′′ (GR , T ) : k ∈ [1, 2, ..., nF ]}. Thus,
Ak (GR ,T )
the predicted matrix is taken to be ||Ak
θ ′′
(G R ,T )||2
, where ||A||2 is the spectral norm of some matrix
θ ′′
A.

Figure 2: Qualitative comparison of our method with ATISS and baselines. Windows and doors and
indicated by white rectangles

3.3 Computing the KL term via matching and constrained learning

Note that the proposed autoregressive prior could in principle be reexpressed as a more traditional
i.i.d. Gaussian prior, which is then passed through an additional non-equivariant transformation
layer that can be absorbed into the decoder. But a significant difference emerges in practice when
facing the key challenge of incorporating this non-equivariant factor into subsequent model training,
given that there is no longer a canonical ordering between the different nodes in a graph. When the
proposed autoregressive prior formulation is adopted, such an ordering is only required to evaluate
pθ′′ (Z | GR , T, nF ) for any Z sampled from the posterior qϕ (Z | G, T, nF ) to compute the KL
divergence term in the ELBO (3). However, by design this term can be expressed analytically, and
as we will soon demonstrate, an efficient compensatory permutation can be efficiently computed.
In contrast, with an alternative autoregressive decoder formulation, the search for an appropriate
ordering is instead needed for computing the VAE reconstruction term (i.e., evaluating the decoder
pθ′ (GF | Z, nF , GR , T ) for any Z sampled from the posterior), and hence becomes entangled with
the non-analytic stochastic sampling required for obtaining approximate reconstructions.
Computing the KL divergence term. Let Z = {Z 1 , Z 2 , ..., Z nF } be the set of latent variables
correponding to nF furniture items to be placed in the room. Let π denote the ordering among these

7
variables. Given π, the likelihood of observing Z under our proposed prior is defined as
nF
Y
p (Z | GR , π, nF , T ) =
θ ′′ pθ′′ (Z π(i) | Z π(j<i) , GR , T ), (7)
i=1

However, for Z ∼ qϕ (Z | G, T, nF ) (the approximate posterior) we do not know this ordering π.

Thus, given Z, we define the optimal order π ∗ to be

π ∗ = arg min KL(qϕ (Z | G, T, nF ) || pθ′′ (Z | GR , π, nF , T )). (8)

π
Q nF
Recall qϕ (Z | G, T, nF ) = i=1 N (Z i ; µiϕ (G), σϕi (G)) Since both the prior and posterior are
jointly Gaussian, computing (8) reduces to solving a Quadratic Assignment Problem (QAP). For
simplicity let us denote the distributions as
qϕ (Z | G, T, nF ) = N (µ0 , Σ0 )
(9)
pθ′′ (Z | GR , π, nF , T ) = N (π̃µ1 , π̃Σ1 π̃ T ),
where π̃ = π ⊗ IdF ×dF . Note π ∈ RnF ×nF and the kronecker product comes from the fact that we
are only allowed to permute blocks of µ1 and Σ1 , of size dF and dF × dF respectively, where dF
is the dimension of the latent variable used for each furniture node. In other words, we can only
permute latent vectors corresponding to furniture nodes as a whole and not the intra dimensions of Z
within any furniture node. Here µ0 , µ1 ∈ RnF dF . Similarly, Σ0 , Σ1 ∈ RnF dF ×nF dF . Note that due
to the autoregressive structure Σ1 will not be a diagonal matrix.
We show in the Appendix (§6.4.1) that (8) is equivalent to
min T r Σ1−1 π̃ T Σ0 + µ0 µT0 π̃ − 2T r π̃ T µ0 µT1 Σ−1 (10)
 


1 .
π

Notice that (10) is a Quadratic Assignment Problem (QAP) which is known to be NP-Hard. For this
we propose to use a fast approximation algorithm, called FAQ, introduced in Vogelstein u. a. (2015).
FAQ first relaxes the optimization problem from set of permutation matrices to the set of all doubly
stochastic matrices. It then iteratively proceeds by solving linearizations of the objective (10) using
the Franke-Wolfe method. These linearizations reduce the QAP to just a linear assignment problem
(LAP), which can be efficiently solved by the Hungarian algorithm. After Franke-Wolfe terminates,
we project the doubly stochastic solution back to the set of permutations by solving another LAP.
FAQ has a runtime complexity that is cubic in the number of nodes per iteration which is faster than
the quartic complexity of the matching procedure used in Simonovsky und Komodakis (2018). More
details in Appendix §6.4.2.
We need to compute the optimal π ∗ for each graph in a mini-batch, and then compute the KL
divergence term in (3) analytically given this ordering. We observe no significant gains in performance
in running the FAQ algorithm more than 1 step per graph, which further speeds our method.
Learning under constraints: To facilitate faster convergence and also ensure fidelity of the learned
solution, we enforce certain constraints on the reconstructed room. These constraints are derived
from training data and do not require external annotations. Given input furniture graph GF to the
encoder and the reconstructed graph G̃F by the decoder, we enforce the relative positions of predicted
furnitures in G̃F to be “close" to the ground truth relative positions in GF . Similarly, we apply
constraints on the relative position of the predicted furniture items with the room walls, windows and
doors. Finally we apply a constraint penalizing the relative orientations between different furniture
items from being too “far" away from the relative orientations in G. We explain these constraints
more clearly in Appendix 6.5.
Having described all the ingredients in our model, we finally present the complete optimization
objective as

n
1X
max L(θ′ , θ′′ , ϕ) s.t. Constr(Gi ) ≤ ϵ. (11)
θ ,θ ,ϕ
′ ′′ n i=1
Here ϵ is a user-defined hyperparameter that determines the strictness of enforcing these constraints.
L(θ′ , θ′′ , ϕ) is as defined in (2) and i is in iterator over the scene graphs in the training set. We employ

8
 Top 3 scenes retrieved
in terms of latent code
likelihood

Empty Room Layout

<latexit sha1_base64="m2hUtA69YDIW92+oykCmvWKJwG0=">AAAB8nicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Vj0oMcq9gPSUDabTbt0sxt2J0IJ/RlePCji1V/jzX/jts1BWx8MPN6bYWZemApuwHW/nZXVtfWNzdJWeXtnd2+/cnDYNirTlLWoEkp3Q2KY4JK1gINg3VQzkoSCdcLRzdTvPDFtuJKPME5ZkJCB5DGnBKzk94CLiOW3k/5Dv1J1a+4MeJl4BamiAs1+5asXKZolTAIVxBjfc1MIcqKBU8Em5V5mWEroiAyYb6kkCTNBPjt5gk+tEuFYaVsS8Ez9PZGTxJhxEtrOhMDQLHpT8T/PzyC+CnIu0wyYpPNFcSYwKDz9H0dcMwpibAmhmttbMR0STSjYlMo2BG/x5WXSrte8i1r9/rzauC7iKKFjdILOkIcuUQPdoSZqIYoUekav6M0B58V5dz7mrStOMXOE/sD5/AFZKJFN</latexit>

G̃R
Database
Lookup

Pass Latent
through GNN
Decoder to
<latexit sha1_base64="m2hUtA69YDIW92+oykCmvWKJwG0=">AAAB8nicbVBNS8NAEN34WetX1aOXxSJ4KkkR9Vj0oMcq9gPSUDabTbt0sxt2J0IJ/RlePCji1V/jzX/jts1BWx8MPN6bYWZemApuwHW/nZXVtfWNzdJWeXtnd2+/cnDYNirTlLWoEkp3Q2KY4JK1gINg3VQzkoSCdcLRzdTvPDFtuJKPME5ZkJCB5DGnBKzk94CLiOW3k/5Dv1J1a+4MeJl4BamiAs1+5asXKZolTAIVxBjfc1MIcqKBU8Em5V5mWEroiAyYb6kkCTNBPjt5gk+tEuFYaVsS8Ez9PZGTxJhxEtrOhMDQLHpT8T/PzyC+CnIu0wyYpPNFcSYwKDz9H0dcMwpibAmhmttbMR0STSjYlMo2BG/x5WXSrte8i1r9/rzauC7iKKFjdILOkIcuUQPdoSZqIYoUekav6M0B58V5dz7mrStOMXOE/sD5/AFZKJFN</latexit>

furnish G̃R

Figure 3: Manipulating Latent Space for Design Recommendations from Database.

the learning under constraints framework introduced by Chamon und Ribeiro (2020) which results in
a primal-dual saddle point optimization problem. We give exact algorithmic details in the Appendix
(Algorithm 1).

3.4 Inference

For inference, we start with a room layout graph GR and type T along with the number of furnitures
to be placed in the room nF . We then use the learnt autoregressive prior to sample nF latent variables
recursively. This latent Z along with (nF , GR , T ) is processed by the graph decoder to generate the
funiture layout subgraph GF . For scene rendering, we use the predicted shape descriptor for each
furniture item and perform a nearest neighbour lookup using the ℓ2 distance against a database of 3D
furniture mesh objects indexed by there respective PointNet feature. We then use the predicted size
and orientation to place furnitures in the scene. Note the inclusion of shape descriptors allows the
model to reason about different appearance of furniture items that is more room aware as opposed to
retrieval based on just furniture category & size as in Wang u. a. (2020); Paschalidou u. a. (2021).

4 Experiments
In this section, we evaluate the effectiveness of the proposed approach qualitatively and quantitatively.
More details and figures provided in the Appendix.
Dataset: We use the 3D-FRONT dataset Fu u. a. (2021) for all our experiments. The dataset consists
of roughly 14k rooms, furnished with 3D mesh furniture items. We consider 4 room types - bedroom,
living room, library & dining room.
Training Protocols: All models were trained on a 80 : 20 train-test split. Since we use PointNet
shape descriptors, we only predict 7 “super-categories"3 of furniture items.4 Since furniture is mostly
axis-aligned, we discretized the orientation into four categories [0◦ , 90◦ , 180◦ , 270◦ ]. We also rotate
3
Cabinet/Shelf, Bed, Chair, Table, Sofa, Pier/Stool, Lighting.
4
This is contrary to prior work Paschalidou u. a. (2021) which explicitly models the 34 fine-grained categories
in 3D-FRONT. We made this design choice since we use shape descriptors for each furniture item which has
the fine-grained label information encoded in them. The 7 super-categories are only used to provide high-level

9
Table 1: Quantitative comparison of our method with other models and baselines. Scene classification
accuracy closer to 0.5 is better

Category KL Divergence (↓) Scene Classification Accuracy

Bedroom Living Dining Library Bedroom Living Dining Library
ATISS 0.01 0.00 0.01 0.01 0.75 0.82 0.75 0.86
Baseline B1 0.02 0.02 0.05 0.09 0.94 0.93 0.90 0.91
Baseline B2 0.02 0.02 0.05 0.07 0.93 0.91 0.85 0.90
Ours 0.01 0.01 0.02 0.02 0.83 0.88 0.78 0.82

the room by multiples of 90◦ as data augmentations. We train all models using the ADAM optimizer
for 1500 epochs with a batch size of 128.
Baselines: We compare our VAE model with baselines which have the same Encoder-Decoder
architecture but employ the commonly used i.i.d. prior. We compare against two variants (inspired
from Luo u. a. (2020)), (i) Standard Prior (B1): Each Z i ∼ N (0, I) for i ∈ [1, ..., nF ], (ii)
Non-autoregressive Learnt Prior (B2): Each Z i ∼ N (µ(GR ), σ(GR )). The mean and variance
parameters of this distribution are learnt by an MP-GNN which processes just the room subgraph.
Scene Generation: In Figure 1b, we illustrate the effectiveness of the proposed method at generating
diverse furniture recommendations given a room layout. Notice how our model generates diverse
furniture arrangements and 3D appearances e.g. sampled beds, cabinets and ceiling lights look
distinct in row 1. In the library (row 2), our model proposes diverse arrangements such as a simple
study room or a lounge with a couch and chair. In Figure 2 we qualitatively compare our generations
for different rooms with ATISS and VAE baselines. The figure shows that our baseline models have
inconsistent overlapping furniture placements. In contrast, our model generates plausible furniture
arrangements, bridging the performance gap between latent-variable models and autoregressive ones
(ATISS).
In Table 1, we provide quantitative metrics comparing our model with baseline VAEs & ATISS.
The Category KL divergence measures how well the model captures the frequency of categories
present in an indoor scene using all 34 fine-grained furniture labels compared to the ground truth. To
obtain a fine-grained label for a generated furniture, we look up the corresponding category of the
nearest-neighbour furniture retrieved using the predicted shape features. On this metric, our method
is competitive with ATISS showing that even though we explicitly only model “super-categories",
our model is able to capture well the fine-grained object label frequencies of the test set via the shape
descriptors. Scene classification accuracy (close to 0.5 is better) tests the ability of a network to
distinguish between real and synthetic scenes. Different from prior work Paschalidou u. a. (2021) we
evaluate this metric by training a GNN directly on the synthesized 3D scene graphs v/s ground truth
graphs, instead of classifying the rendered 2D top-down orthographic projections. This is because at
the image level, CNNs are not able to explicitly reason about 3D spatial relationships compared to a
GNN. Moreover, classifying 2D projections is sensitive to the rendering used and thus will not be
comparable across different works. On this metric, our method performs significantly better than
baselines (7-10% improvement) and is competitive with ATISS.
Generation Time: The average run-time for scene synthesis for our method is 130.26ms v/s
148.51ms for ATISS (measured on NVIDIA GeForce GTX 2080 Ti machine).
Do the constraints help? Employing constraints facilitates faster training. We illustrate this in the
Appendix (Figure 5). Without the constraints, the network struggles to learn the object location &
orientation even after 70k iters.
Manipulating Latent Space for Design Recommendations from Database. In the previous sub-
section we showed our model’s ability to generate indoor scenes given room layouts. However, apart
from AI-synthesized recommendations one might also wish to obtain recommendations from human
interior designers. This can be automated by having a large database curated using hand-designed
interior rooms and then given an empty floor-plan by the user, recommend “appropriate" designs from
supervision to distinguish between furnitures which may have similar shapes, for example, a chair and a sofa-
chair. During synthesis, our model predicts the shape descriptors, which is used for rendering, and thus can
generate furniture’s belonging to all the 34 categories.

10
this database. However, the current practice is for the user to manually browse through configurations
to find a match ??. Instead, through our learnt latent space, we can use our model to suggest the best
designs from the database to choose from. To simulate this, we consider the 3D-FRONT training
set as our database. We use our learnt GNN encoder to convert each of these scenes into their
corresponding latent code and store in the database. Then, given an empty room layout G̃R we
retrieve the latent code with the highest likelihood under our autoregressive prior (5) (conditioned
on G̃R and its type).5 Finally, we pass this latent through the GNN decoder along with G̃R . This
ensures that the empty room G̃R is furnished according to the design of the retrieved scene while
respecting the constraints imposed by G̃R e.g. furniture items should not go outside the walls. Figure
3 shows results for this experiment.
Scene Editing. Our model also allows the user to traverse along the latent space to edit the scene
content post-generation. Given a synthesized scene, we can convert a given furniture with label c1
to another furniture labelled c2 ensuring all other objects are approximately in the same geometric
configuration. This is a non-trivial problem since depending on the room layout we also need to
decide the size and orientation of the new furniture labelled c2 . In our formulation, this can be
−µ1
done by finding a latent direction v := ||µµ22−µ 1 ||2
that transforms furniture from label c1 → c2 ,
where {µ1 , µ2 } are the sample mean of the latent representations of furnitures with label {c1 , c2 }.
Specifically, recall the GNN encoder in our model maps the input graph to a latent space with every
furniture node having a separate latent code. We first compute the latent embeddings for all scenes in
the training set. We then compute µi by averaging over all latent codes corresponding to furnitures
with label ci across all scenes in the training set, where i ∈ {1, 2}. Now at inference time, we
synthesize a scene S with a furniture labelled c1 using our autoregressive prior. Post-synthesis we
can select the latent ẑ corresponding to furniture c1 and translate it to ẑ ′ = ẑ + αv, where α controls
the magnitude of morphing c1 into c2 . This updated latent ẑ ′ along with the latent codes of all other
furniture items in S are passed through the GNN decoder again. The end result, furniture c1 gets
morphed into c2 while keeping the relative spatial arrangement of all other furniture’s the same. This
process is elucidated in Figure 4.

5 Conclusion
We have presented a latent-variable model for generating 3D indoor scenes given the room type and
layout which is competitive with purely autoregressive models. Moreover, we show how a learnt
latent space can be utilized to recommend designs from a database. In future work, we wish to explore
more expressive non-linear autoregressive priors to improve generations; however the KL divergence
cannot be computed analytically and the subsequent matching will no longer be quadratic. Finally, the
matching procedure introduced in this paper can be potentially useful in other permutation-invariant
domains like sets.

References
[Armeni u. a. 2019] A RMENI, Iro ; H E, Zhi-Yang ; G WAK, JunYoung ; Z AMIR, Amir R. ; F ISCHER, Martin ;
M ALIK, Jitendra ; S AVARESE, Silvio: 3D Scene Graph: A Structure for Unified Semantics, 3D Space, and
Camera. In: IEEE/CVF International Conference on Computer Vision, 2019, S. 5664–5673
[Bishop 2006] B ISHOP, Christopher M.: Pattern recognition. In: Machine learning 128 (2006), Nr. 9
[Chamon und Ribeiro 2020] C HAMON, Luiz ; R IBEIRO, Alejandro: Probably approximately correct
constrained learning. In: Advances in Neural Information Processing Systems 33 (2020)
[Chang u. a. 2015] C HANG, Angel X. ; F UNKHOUSER, Thomas ; G UIBAS, Leonidas ; H ANRAHAN, Pat ;
H UANG, Qixing ; L I, Zimo ; S AVARESE, Silvio ; S AVVA, Manolis ; S ONG, Shuran ; S U, Hao ; X IAO,
Jianxiong ; Y I, Li ; Y U, Fisher: ShapeNet: An Information-Rich 3D Model Repository / Stanford University
— Princeton University — Toyota Technological Institute at Chicago. 2015 (arXiv:1512.03012 [[Link]]). –
Forschungsbericht
[Chen u. a. 2016] C HEN, Xi ; K INGMA, Diederik P. ; S ALIMANS, Tim ; D UAN, Yan ; D HARIWAL, Prafulla ;
S CHULMAN, John ; S UTSKEVER, Ilya ; A BBEEL, Pieter: Variational lossy autoencoder. In: arXiv preprint
arXiv:1611.02731 (2016)
5
We evaluate (8) to compute the ordering π of the latents for evaluating its likelihood under the autoregressive
prior. More details in Appendix.

11
Figure 4: Scene Editing. Col 1 is a scene generated by our model. In row 1, we morph the bed into a
chair by changing the α parameter as explained in text (§4). In row 2, we morph the top cabinet into
a chair.

[Fisher u. a. 2012] F ISHER, Matthew ; R ITCHIE, Daniel ; S AVVA, Manolis ; F UNKHOUSER, Thomas ;
H ANRAHAN, Pat: Example-based synthesis of 3D object arrangements. In: ACM Transactions on Graphics
(TOG) 31 (2012), Nr. 6, S. 1–11

[Fisher u. a. 2011] F ISHER, Matthew ; S AVVA, Manolis ; H ANRAHAN, Pat: Characterizing structural
relationships in scenes using graph kernels. In: ACM SIGGRAPH 2011 papers. 2011, S. 1–12

[Fu u. a. 2021] F U, Huan ; C AI, Bowen ; G AO, Lin ; Z HANG, Ling-Xiao ; WANG, Jiaming ; L I, Cao ;
Z ENG, Qixun ; S UN, Chengyue ; J IA, Rongfei ; Z HAO, Binqiang u. a.: 3d-front: 3d furnished rooms with
layouts and semantics. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021,
S. 10933–10942

[Gadde u. a. 2021] G ADDE, Raghudeep ; F ENG, Qianli ; M ARTINEZ, Aleix M.: Detail Me More: Improving
GAN’s Photo-Realism of Complex Scenes. In: Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), October 2021, S. 13950–13959

[Goodfellow u. a. 2014] G OODFELLOW, Ian ; P OUGET-A BADIE, Jean ; M IRZA, Mehdi ; X U, Bing ;
WARDE -FARLEY, David ; O ZAIR, Sherjil ; C OURVILLE, Aaron ; B ENGIO, Yoshua: Generative
Adversarial Nets. In: G HAHRAMANI, Z. (Hrsg.) ; W ELLING, M. (Hrsg.) ; C ORTES, C. (Hrsg.) ;
L AWRENCE, N. (Hrsg.) ; W EINBERGER, K. Q. (Hrsg.): Advances in Neural Information Processing Systems
Bd. 27, Curran Associates, Inc., 2014. – URL [Link]
[Link]

[Higgins u. a. 2016] H IGGINS, Irina ; M ATTHEY, Loic ; PAL, Arka ; B URGESS, Christopher ; G LOROT,
Xavier ; B OTVINICK, Matthew ; M OHAMED, Shakir ; L ERCHNER, Alexander: beta-vae: Learning basic
visual concepts with a constrained variational framework. (2016)

[Jiang u. a. 2018] J IANG, Chenfanfu ; Q I, Siyuan ; Z HU, Yixin ; H UANG, Siyuan ; L IN, Jenny ; Y U, Lap-Fai ;
T ERZOPOULOS, Demetri ; Z HU, Song-Chun: Configurable 3d scene synthesis and 2d image rendering with
per-pixel ground truth using stochastic grammars. In: International Journal of Computer Vision 126 (2018),
Nr. 9, S. 920–941

[Karras u. a. 2020] K ARRAS, Tero ; L AINE, Samuli ; A ITTALA, Miika ; H ELLSTEN, Janne ; L EHTINEN,
Jaakko ; A ILA, Timo: Analyzing and Improving the Image Quality of StyleGAN. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

12
[Keshavarzi u. a. 2020] K ESHAVARZI, Mohammad ; PARIKH, Aakash ; Z HAI, Xiyu ; M AO, Melody ; C ALDAS,
Luisa ; YANG, Allen: Scenegen: Generative contextual scene augmentation using scene graph priors. In:
arXiv preprint arXiv:2009.12395 (2020)
[Kingma u. a. 2014] K INGMA, Diederik P. ; M OHAMED, Shakir ; R EZENDE, Danilo J. ; W ELLING, Max:
Semi-supervised learning with deep generative models. In: Advances in neural information processing
systems, 2014, S. 3581–3589
[Kingma und Welling 2014] K INGMA, Diederik P. ; W ELLING, Max: Auto-Encoding Variational Bayes. In:
CoRR abs/1312.6114 (2014)
[Kipf und Welling 2016a] K IPF, Thomas ; W ELLING, Max: Variational Graph Auto-Encoders. In: ArXiv
abs/1611.07308 (2016)
[Kipf und Welling 2016b] K IPF, Thomas N. ; W ELLING, Max: Variational graph auto-encoders. In: arXiv
preprint arXiv:1611.07308 (2016)
[Klushyn u. a. 2019] K LUSHYN, Alexej ; C HEN, Nutan ; K URLE, Richard ; C SEKE, Botond ; S MAGT, Patrick
van der: Learning hierarchical priors in vaes. In: arXiv preprint arXiv:1905.04982 (2019)
[Kumar u. a. 2017] K UMAR, Abhishek ; S ATTIGERI, Prasanna ; BALAKRISHNAN, Avinash: Variational
inference of disentangled latent concepts from unlabeled observations. In: arXiv preprint arXiv:1711.00848
(2017)
[Kusner u. a. 2017] K USNER, Matt J. ; PAIGE, Brooks ; H ERNÁNDEZ -L OBATO, José M.: Grammar variational
autoencoder. In: International Conference on Machine Learning PMLR (Veranst.), 2017, S. 1945–1954
[Lacy und Bernstein 2003] L ACY, Seth L. ; B ERNSTEIN, Dennis S.: Subspace identification with guaranteed
stability using constrained optimization. In: IEEE Transactions on automatic control 48 (2003), Nr. 7,
S. 1259–1263
[Li u. a. 2019] L I, Manyi ; PATIL, Akshay G. ; X U, Kai ; C HAUDHURI, Siddhartha ; K HAN, Owais ; S HAMIR,
Ariel ; T U, Changhe ; C HEN, Baoquan ; C OHEN -O R, Daniel ; Z HANG, Hao: Grains: Generative recursive
autoencoders for indoor scenes. In: ACM Transactions on Graphics (TOG) 38 (2019), Nr. 2, S. 1–16
[Luo u. a. 2020] L UO, Andrew ; Z HANG, Zhoutong ; W U, Jiajun ; T ENENBAUM, Joshua B.: End-to-End
Optimization of Scene Layout. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2020, S. 3754–3763
[Mavroudi u. a. 2020] M AVROUDI, Effrosyni ; H ARO, Benjamín Béjar ; V IDAL, René: Representation
learning on visual-symbolic graphs for video understanding. In: European Conference on Computer Vision
Springer (Veranst.), 2020, S. 71–90
[Para u. a. 2021] PARA, Wamiq ; G UERRERO, Paul ; K ELLY, Tom ; G UIBAS, Leonidas J. ; W ONKA,
Peter: Generative layout modeling using constraint graphs. In: Proceedings of the IEEE/CVF International
Conference on Computer Vision, 2021, S. 6690–6700
[Paschalidou u. a. 2021] PASCHALIDOU, Despoina ; K AR, Amlan ; S HUGRINA, Maria ; K REIS, Karsten ;
G EIGER, Andreas ; F IDLER, Sanja: ATISS: Autoregressive Transformers for Indoor Scene Synthesis. In:
Advances in Neural Information Processing Systems (NeurIPS), 2021
[Purkait u. a. 2020] P URKAIT, Pulak ; Z ACH, Christopher ; R EID, Ian: SG-VAE: Scene Grammar Variational
Autoencoder to generate new indoor scenes. In: European Conference on Computer Vision Springer (Veranst.),
2020, S. 155–171
[Qi u. a. 2017] Q I, Charles R. ; S U, Hao ; M O, Kaichun ; G UIBAS, Leonidas J.: Pointnet: Deep learning on
point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision
and pattern recognition, 2017, S. 652–660
[Qi u. a. 2018] Q I, Siyuan ; Z HU, Yixin ; H UANG, Siyuan ; J IANG, Chenfanfu ; Z HU, Song-Chun: Human-
centric indoor scene synthesis using stochastic grammar. In: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2018, S. 5899–5908
[Rezende und Mohamed 2015] R EZENDE, Danilo ; M OHAMED, Shakir: Variational inference with normaliz-
ing flows. In: International conference on machine learning PMLR (Veranst.), 2015, S. 1530–1538
[Ritchie u. a. 2019] R ITCHIE, Daniel ; WANG, Kai ; L IN, Yu-An: Fast and Flexible Indoor Scene Synthesis
via Deep Convolutional Generative Models. In: IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 2019, S. 6175–6183

13
[Simonovsky und Komodakis 2018] S IMONOVSKY, Martin ; KOMODAKIS, Nikos: Graphvae: Towards
generation of small graphs using variational autoencoders. In: International conference on artificial neural
networks Springer (Veranst.), 2018, S. 412–422

[Tomczak und Welling 2018] T OMCZAK, Jakub ; W ELLING, Max: VAE with a VampPrior. In: International
Conference on Artificial Intelligence and Statistics PMLR (Veranst.), 2018, S. 1214–1223

[Vaswani u. a. 2017] VASWANI, Ashish ; S HAZEER, Noam ; PARMAR, Niki ; U SZKOREIT, Jakob ; J ONES,
Llion ; G OMEZ, Aidan N. ; K AISER, Ł u. ; P OLOSUKHIN, Illia: Attention is All you Need. In: G UYON,
I. (Hrsg.) ; L UXBURG, U. V. (Hrsg.) ; B ENGIO, S. (Hrsg.) ; WALLACH, H. (Hrsg.) ; F ERGUS, R. (Hrsg.) ;
V ISHWANATHAN, S. (Hrsg.) ; G ARNETT, R. (Hrsg.): Advances in Neural Information Processing Systems
Bd. 30, Curran Associates, Inc., 2017. – URL [Link]
[Link]

[Vogelstein u. a. 2015] VOGELSTEIN, Joshua T. ; C ONROY, John M. ; LYZINSKI, Vince ; P ODRAZIK, Louis J. ;
K RATZER, Steven G. ; H ARLEY, Eric T. ; F ISHKIND, Donniell E. ; VOGELSTEIN, R J. ; P RIEBE, Carey E.:
Fast approximate quadratic programming for graph matching. In: PLOS one 10 (2015), Nr. 4, S. e0121002

[Wang u. a. 2019] WANG, Kai ; L IN, Yu-An ; W EISSMANN, Ben ; S AVVA, Manolis ; C HANG, Angel X. ;
R ITCHIE, Daniel: Planit: Planning and instantiating indoor scenes with relation graph and spatial prior
networks. In: ACM Transactions on Graphics (TOG) 38 (2019), Nr. 4, S. 1–15

[Wang u. a. 2018] WANG, Kai ; S AVVA, Manolis ; C HANG, Angel X. ; R ITCHIE, Daniel: Deep convolutional
priors for indoor scene synthesis. In: ACM Transactions on Graphics (TOG) 37 (2018), Nr. 4, S. 1–14
[Wang u. a. 2020] WANG, Xinpeng ; Y ESHWANTH, Chandan ; N IESSNER, Matthias: Sceneformer: Indoor
scene generation with transformers. In: arXiv preprint arXiv:2012.09793 (2020)

[Yeh u. a. 2012] Y EH, Yi-Ting ; YANG, Lingfeng ; WATSON, Matthew ; G OODMAN, Noah D. ; H ANRAHAN,
Pat: Synthesizing open worlds with constraints using locally annealed reversible jump mcmc. In: ACM
Transactions on Graphics (TOG) 31 (2012), Nr. 4, S. 1–11

[Zhang u. a. 2020] Z HANG, Zaiwei ; YANG, Zhenpei ; M A, Chongyang ; L UO, Linjie ; H UTH, Alexander ;
VOUGA, Etienne ; H UANG, Qixing: Deep generative modeling for scene synthesis via hybrid representations.
In: ACM Transactions on Graphics (TOG) 39 (2020), Nr. 2, S. 1–21

14
6 Appendix
6.1 Feature engineering

As explained in §3.1, our indoor scene representation is an attributed graph. We now describe these
attributes in detail.

• Room node features XR ,

1. Node Type: A 4D 1-hot embedding of the node type. The four categories are wall,
door, floor, window.
2. Room Type: A 4D 1-hot embedding of the room type. The four categories are living
room, bedroom, dining room, library.
3. Location: A 6D vector representing the minimum and maximum vertex positions of
the 3D bounding box corresponding to the room node.
4. Normal: A 3D vector representing the direction of the normal vector corresponding to
the room node.
• Furniture node features XF ,
1. Super category: A 7D 1-hot embedding of the node type. The seven categories are
Cabinet/Shelf, Bed, Chair, Table, Sofa, Pier/Stool, Lighting.
2. Shape: A 1024D embedding of the 3D shape descriptor obtained by processing a 3D
point cloud of the furniture item through PointNet.
3. Location: A 3D vector representing the centroid of the furniture item.
4. Orientation: A 3D vector representing the direction the “front" side of the furniture is
facing.
5. Size: A 3D vector representing the dimensions of the furniture along each axis.
• Furniture-furniture edge features XF F , let us arbitrarily chose a furniture-furniture edge
and label its source node as s with receiver node as r. We will describe the features for this
edge.
1. Center-center distance: A scalar representing the distance between the centroids of
furniture s and r.
2. Relative orientation: A scalar representing the signed dot product between the orienta-
tion feature of s and r.
3. Center-center orientation: A 3D vector representing the unit vector connecting the
centroids of r and s.
4. Orientation: A 3D vector representing the direction the “front" side of the furniture is
facing.
5. Bbox-bbox distance: The shortest distance between the corners of the bounding box
of s and r.
• Room-furniture edge features XRF , let us arbitrarily chose a room-furniture edge and
label its source node as s with receiver node as r. We will describe the features for this edge.
1. Center-room distance: A scalar representing the distance between the centroid of
furniture r and room node s. This is computed in 2D as a point to line distance. Every
wall/window/door can be treated as a line segment in 2D.
2. Center-room center: A scalar representing the distance between the centroids of r and
s.
3. Bbox-room dist A scalar representing the shortest distance between corners of the
bounding box of furniture item r to the room node s. This is computed in 2D as a
point-to-line distance.
4. Bbox-room center A scalar representing the shortest distance between corners of the
bounding box of furniture item r to the centroid of the room node s.
5. Relative orientation: A signed dot product between the normal of s with the orientation
attribute of r.

15
 • Room-room edge features XRR , let us arbitrarily chose a room-room edge and label its
source node as s with receiver node as r. We will describe the features for this edge.
1. Center-center distance: A scalar representing the distance between the centroids of
room nodes s and r.
2. Relative orientation: A scalar representing the signed dot product between the normal
vectors of s and r.
3. Longest distance: A scalar representing the longest distance between the corners of
the bounding boxes of r and s.
4. Shortest distance: A scalar representing the shortest distance between the corners of
the bounding boxes of r and s.

6.2 Architecture Details

6.2.1 Message-Passing Graph Neural Network (MP-GNN)

Here we describe the message-passing mechanism employed in the Graph Neural Networks used in
this work. The design of the MP-GNN is inspired by the work of Mavroudi u. a. (2020), where each
layer l = 1 . . . , L of the MP-GNN maps a graph Gl−1 to another graph Gl by updating the graph’s
node and edge features. Specifically, let hli and hlij denote the features of node i and edge (i, j) of
graph Gl , respectively. Let the input to the network be the graph G0 = G, so that h0i and h0ij denote
the node features and edge features, respectively. At each iteration of node and edge refinement,
the MP-GNN: (1) adapts the scalar edge weights by using an attention mechanism; (2) updates the
edge attributes depending on the edge type, the attention-based edge weights, the attributes of the
connected nodes and the previous edge attribute; and (3) updates the node attribute by aggregating
attributes from incoming edges , as described next.
Computing attention coefficients: At each step l of refinement, we update the scalar edge weights by
computing an attention coefficient alij that measures the relevance of node j for node i as follows:
l
γij = ρ(waϵij ⊤ [Wrνi hli ; Wsνj hlj ; Wrs
ϵij l
hij ]
l
exp(γij ) (12)
alij = P l
, ∀(i, j) ∈ E.
k∈Nϵ (i) exp(γik )

In the first equation, each edge (i, j) of Gl = (V, E, Xl ) is associated with a score, γij
l
, obtained
by applying a nonlinearity ρ (e.g., a leaky ReLU) to the dot product between a vector of attention
ϵ
weights waij for edge type ϵij ∈ {RR, RF, F F }, and the concatenation of five vectors: (1) the
receiver node feature hli ∈ Rdi weighted by a matrix Wrνi for node type νi ∈ {R, F }, (2) the sender
v
node feature hlj weighted by a matrix Ws j for node type vj , and (3) the receiver-sender edge feature
ϵ
hlij weighted by a matrix Wrsij for edge type ϵij . All weight matrices are also indexed by the GNN
layer l which is suppressed to prevent notational clutter.
Updating the node and edge features: At each step l of refinement, we update the edge and node
features as: ϵij
hlij = alij (Uedge Wsνj hjl−1 + Wrs ϵij l−1
hij ), ∀(i, j) ∈ E
ϵ (13)
X
hli = ρ(hil−1 + ij
Unode Ws hlik ), ∀i ∈ V.
k∈Nϵij (i)

The first equation updates the features of edge (i, j) by taking a weighted combination of the features
ν ϵ
of node j and the features of edge (i, j) in the previous layer, with weight matrices Ws j and Wrsij ,
ϵij
respectively. The matrix Uedge is a learnable projection matrix to change the dimensionality of the
first term (transformed node feature) with that of the second term (transformed edge feature).
The second equation updates the feature of node i as the sum of the feature of node i from the
previous layer (a residual connection) and the sum of the features of all edges connected to node i
ϵij
after applying a non-linearity ρ, such as a ReLU or leaky-ReLU. Here, Unode denotes a learnable
projection matrix for edge type ϵij .
All weight matrices are also indexed by the GNN layer l which is suppressed to prevent notational
clutter.

16
 Table 2: Encoder architecture. The network parameters are as defined in Section 6.2.1

6.2.2 Encoder, Decoder and Room Aggregator networks

We employ three MP-GNNs in this work. One for the VAE encoder, one for the decoder and one for
the Room Aggregator component of the prior network. Each of these MP-GNNs are three layered
graph neural networks. In Table 2, 3, 4 we summarize the architecture of the Encoder, Decoder and
Room Aggregator networks respectively.
The RNN Prior network (second component of our Graph Prior) is implemented by a single layer
Gated Recurrent Unit (GRU) with hidden vector of size 64 × 64 (the latent space is 64D).

6.3 The Reconstruction Loss of the ELBO

The reconstruction loss is the first term in the ELBO (3).

EZ∼qϕ (Z|G,T,nF ) [log pθ′ (GF | Z, nF , GR , T )]

In subsection 3.2 we described the distribution pθ′ (GF | Z, nF , GR , T ) as

pθ′ (GF | Z, nF , GR , T )
YnF
= pθ′ (shapei | Z, GR , T )pθ′ (orieni | Z, GR , T )
i=1
pθ′ (loci | Z, GR , T )pθ′ (sizei | shapei )pθ′ (cati | shapei ).
Here shapei , orieni , loci , sizei and cati refer to the ground truth values of the 3D shape descriptor
(PointNet features), orientation, location, size and category respectively for furniture node i in GF .
We will now describe each of these distributions in detail along with the corresponding loss term. For
the normal and lognormal distributions used we assume the variance parameter is a fixed constant
equal to 1 and do not learn them.

17
 Table 3: Decoder architecture. The network parameters are as defined in Section 6.2.1

Table 4: Room Aggregator architecture. This network only processes the room subgraph GR and
hence all the weight matrices associated with furniture nodes are absent. The network parameters
are as defined in §6.2.1

18
 pθ′ (shapei | Z, GR , T ) = N (µshape
θ′ (Z, GR , T ), const.)
 2
log pθ′ (shapei | Z, GR , T ) = − µshape
θ ′ (Z, GR , T ) − shape i

pθ′ (orienti | Z, GR , T ) = Categorical(Θorient

θ′ (Z, GR , T ))
log pθ′ (orienti | Z, GR , T ) = −H(orienti , Θorient
θ′ (Z, GR , T ))
orienti is a 4D 1-hot embedding of the ground truth orientation and Θ is the parameter of the
categorical distribution. H(., .) denotes the cross entropy between two distributions.

pθ′ (loci | Z, GR , T ) = N (µloc

θ ′ (Z, GR , T ), const.)

2
log pθ′ (loci | Z, GR , T ) = − µloc
θ ′ (Z, GR , T ) − loci

shape
pθ′ (sizei | shapei ) = LogN ormal(µsize
θ ′ (µθ ′ (Z, GR , T )), const.)
 2
shape
log pθ′ (sizei | shapei ) = − µsize
θ ′ (µ θ ′ (Z, G R , T )) − ln size i

shape
pθ′ (cati | shapei ) = Categorical(Θcat
θ ′ (µθ ′ (Z, GR , T )))
shape
log pθ′ (cati | shapei ) = −H(cati , Θcat
θ ′ (µθ ′ (Z, GR , T )))

Each furniture node feature in the output of the MP-GNN Decoder is individually processed using a
multi-layer Perceptron (MLP). Specifically, let ĥL
i be the output feature of the decoder for furniture
node i. In our experiments ĥL
i ∈ R 1034
.

µshape
θ′ (Z, GR , T ) = Linearθ′ (1034, 1024)
Θorient
θ′ (Z, GR , T )) = SoftMax(Linearθ′ (1034, 4))

µloc
θ ′ (Z, GR , T ) = Linearθ ′ (1034, 3)

µsize
θ′ = Linearθ′ (1024, 512) → ReLU → Linearθ′ (512, 3)
This is a three-layer MLP with ReLU activations and 1024 input neurons since the input is the mean
of the shape features µshape
θ′ (Z, GR , T )).

Θcat
θ ′ = Linearθ ′ (1024, 512) → ReLU → Linearθ ′ (512, 7)

The output is 7D since there are 7 super-categories in our dataset, Cabinet/Shelf, Bed, Chair, Table,
Sofa, Pier/Stool, Lighting.
Linearθ′ (x, y) denotes a linear layer with x input neurons and y input neurons. Recall θ′ denotes all
the parameters of the Decoder including the GNN and the output MLPs.

6.4 Computing the KL divergence term

6.4.1 Derivation for (10)

Rewriting (8)
π ∗ = arg min KL(qϕ (Z | G, T, nF ) || pθ′′ (Z | GR , π, nF , T )) (14)
π

For simplicity let us denote the two distributions as

qϕ (Z | G, T, nF ) = N (µ0 , Σ0 )
pθ′′ (Z | GR , π, nF , T ) = N (π̃µ1 , π̃Σ1 π̃ T ),

19
where π̃ = π ⊗ IdF ×dF .
The KL divergence between two Gaussian distributions is known analytically,
KL(qϕ (Z | G, T, nF ) || pθ′′ (Z | GR , π, nF , T ))
1
T r(π̃Σ−1 T T −1 T


= 1 π̃ Σ0 ) + (π̃µ1 − µ0 ) π̃Σ1 π̃ (π̃µ1 − µ0 )
2
det(π̃Σ1 π̃ T )
   
1
+ ln − nF d F
2 det(Σ0 )
≡ T r(π̃Σ1 π̃ Σ0 ) + (π̃µ1 − µ0 )T π̃Σ−1
−1 T T


1 π̃ (π̃µ1 − µ0 )
(15)
= T r(Σ−1 T T −1
1 π̃ Σ0 π̃) + µ1 Σ µ1 − µT1 Σ−1 T
1 π̃ µ0
− µT0 π̃Σ−1 T −1 T
1 µ1 + µ0 π̃Σ1 π̃ µ0
≡ T r(Σ−1 T T −1 T T −1
1 π̃ Σ0 π̃) − µ1 Σ1 π̃ µ0 − µ0 π̃Σ1 µ1
+ µT0 π̃Σ−1 T
1 π̃ µ0
≡ T r(Σ−1 T T −1 T T −1 T
1 π̃ Σ0 π̃) − 2µ1 Σ1 π̃ µ0 + µ0 π̃Σ1 π̃ µ0
The third equivalence is obtained by observing that the constant nF dF (which is the dimension of the
support of the two Gaussian distributions) does not affect the solution of (14) and that permuting the
rows and column of a matrix by the same permutation does not change its determinant and hence the
minimizer π ∗ will also not depend on this term. By similar reasoning, we also ignore the factor of 12 .
In the fifth equivalence we again ignore terms that don’t affect π ∗ . Using the cyclic property of the
T race operator we can rewrite the last term on the RHS of (15) as
µT0 π̃Σ−1 T T −1 T −1 T T



1 π̃ µ0 = T r µ0 π̃Σ1 π̃ µ0 = T r Σ1 π̃ µ0 µ0 π̃
Substituting this results in (15) we obtain the desired result.
KL(qϕ (Z | G, T, nF ) || pθ′′ (Z | GR , π, nF , T ))
≡ T r(Σ−1 T T −1 T −1 T T


1 π̃ Σ0 π̃) − 2µ1 Σ1 π̃ µ0 + T r Σ1 π̃ µ0 µ0 π̃
= T r(Σ−1 T
Σ0 + µ0 µT0 π̃) − 2µT1 Σ−1 T
 
1 π̃ 1 π̃ µ0
= T r(Σ−1 T
Σ0 + µ0 µT0 π̃) − 2T r µT1 Σ−1 T
 

1 π̃ 1 π̃ µ0
= T r(Σ−1 T
Σ0 + µ0 µT0 π̃) − 2T r π̃ T µ0 µT1 Σ−1
 

1 π̃ 1
We again used the cyclic property of the T race operator for the last equality. This concludes our
derivation for (10).

6.4.2 Solving (10) using the FAQ approximation

. The FAQ algorithm proceeds as follows,
T
1. Choose an initial point: We choose the uninformative π0 = 1.1 nF as the initial estimate for
the algorithm. Note, here π is a doubly stochastic matrix and not a permutation matrix.
0

2. Find a local solution: In each iteration i, we linearize the objective at the current iterate π i
f˜i (π) := f (π i ) + T r ∇f (π i )T (π − π i )
 

Here, f (π) = T r Σ−1 T

− 2T r π̃µ0 µT1 Σ−1 . We then solve the fol-
  T


1 π̃ Σ0 + µ0 µ0 π̃ 1
lowing subproblem
minT r(∇f (π i )T (π)
π (16)
s.t. π ∈ D
Here D is the set of all doubly stochastic matrices. Notice (16) is just a Linear Assignment
Problem and can be efficiently solved by the Hungarian Algorithm. Let q i denote the
arg min of (16).
3. Choose the next iterate: Finally choose π i+1 = απ i + (1 − α)q i , where α ∈ [0, 1]. Here
α is chosen by solve the 1D optimization problem analytically.
min f (απ i + (1 − α)q i ) s.t. α ∈ [0, 1] (17)
α

20
 4. Repeat steps 2 and 3 untill convergence. Steps 2-3 are repeated iteratively untill some
termination criteria is met, say ||π i − π i−1 ||F < ϵ. In practice, we do not run steps 2-3
untill convergence but terminate after just 1 iteration of the FAQ algorithm. Running for
longer iterations did not result in any significant gains in performance.
5. Project onto the set of permutation matrices. Upon termination, the final solution
is obtained by projecting π f inal onto the space of permutation matrices by solving
minπ∈P −T r(π f inal π T ). Here P is the set of all permutation matrices of size nF . This is
against solved by the Hungarian Algorithm.

In step 2, we need to compute the gradient of f (π i ). This can be done analytically. We will start our
derivation by stating some facts. First, the gradient of the dot product of two matrices A and X with
respect to X is,
d
⟨A, X⟩ = A (18)
dX
Second, for every linear operator M , its adjoint operator M † is defined such that
⟨A, M (X)⟩ = ⟨M † (A), X⟩ (19)

Recall,
f (π) = T r Σ−1 T
− 2T r π̃µ0 µT1 Σ−1
  T


1 π̃ Σ0 + µ0 µ0 π̃ 1
.
We will fist look at the second term,
T r π̃µ0 µT1 Σ−1 = ⟨Σ−1 T


1 1 µ1 µ0 , M (π)⟩
×nF
nFX   
= πij T r Σ−11 µ µ T
1 0 ij (20)
ij

= ⟨M † Σ−1 T


1 µ1 µ0 , π⟩

Here, M (π) := π ⊗ IdF ×dF . Given a matrix A ∈ RnF dF ×nF dF , we define the operation [A]ij (used
in the second equality) as follows. First divide the matrix A into non-overlapping blocks of size
dF × dF , there are nF × nF such blocks. Now [A]ij denotes the ij th block. In the last equality, we
defined the adjoint L† (A) as Â. Here  is a nF × nF matrix obtained from A whose ij th entry is,
 
Âij = T r [A]ij .
.
Combining (20) with (18) we conclude
d
T r π̃µ0 µT1 Σ−1 = M † Σ−1 T
(21)



1 1 µ1 µ0
dπ
The gradient of the first term is calculated similarly,

T r Σ−1 T
= ⟨Σ−1
  T
 T

1 π̃ Σ0 + µ0 µ0 π̃ 1 π̃ Σ0 + µ0 µ0 , M (π)⟩
d (22)
T r Σ−1 T
= 2M † (Σ−1
  T
 T

1 π̃ Σ0 + µ0 µ0 π̃ 1 π̃ Σ0 + µ0 µ0 )
dπ
Here M (π) is again defined as π ⊗ IdF ×dF . Putting it all together,

∇f (π) = 2M † (Σ−1 T †
Σ−1 T
(23)
 

1 π̃ Σ0 + µ0 µ0 ) − 2M 1 µ1 µ0

6.5 Learning Under Constraints

Let furniture graph GF be the input to the encoder and G̃F be the furniture graph reconstructed by
the decoder. Recall, EF F denotes the edges between furniture nodes in GF (§3.1) and G̃F has the
same structure as GF . We employ the following constraints,

21
 • furniture-furniture distance constraint: For every (vi , vj ) ∈ EF F , we define there rel-
ative position as cij = ||loc(vi ) − loc(vj )||2 , where loc(vi ) denotes the 3D centroid of
furniture item vi from GF . We define cpred
ij = ||locpred (vi ) − locpred (vj )||2 as the relative
distance computed from the corresponding location mean prediction by the decoder from
G̃F . Finally we define dij pred
F F := M SE(cij , cij ). Here M SE refers to mean squared error.

• furniture-room distance constraint: This constraint restricts the relative position of the
predicted furniture items with the room nodes.

– For the windows and doors, dij

RF is defined analogously as
ij pred
dRF = M SE(cij , cij ) where (vi , vj ) ∈ ERF , with the exception that for computing
cpred
ij we use the ground truth room node location since they are not predicted by the
decoder.

– For walls, cpred

ij is computed as the signed distance between the ith wall node and the
j furniture centroid. This ensures the decoder predicts furniture items on the correct
th

side of the wall.

• furniture-furniture relative orientation constraint: This constraint enforces the relative

orientation between two predicted furniture centroid locations to be close to the ground truth.
Specifically we define ∀(vi , vj ) ∈ EF F oij T
F F = orient(vi , vj ) orient
pred
(vi , vj ). Here
orient(vi , vj )T is the unit vector pointing in the direction loc(vi ) − loc(vj ) computed from
GF . Similarly, orientpred (vi , vj ) is the unit vector pointing in the direction locpred (vi ) −
locpred (vj ) computed from the reconstruction G̃F .

We now present the complete optimization objective as

max L(θ′ , θ′′ , ϕ)

θ ′ ,θ ′′ ,ϕ
 
n
1 X X
s.t.  dij
FF
≤ϵ
n i=1
(vi ,vj )∈EF F
 
1 X n X (24)
s.t.  dij
RF
≤ϵ
n i=1
(vi ,vj )∈ERF
 
n
1 X X
s.t.  oij
FF
≥1−ϵ
n i=1
(vi ,vj )∈EF F

Here ϵ is a user-defined hyperparameter that determines the strictness of enforcing these constraints6 .
L(θ′ , θ′′ , ϕ) is as defined in (2) and i is in iterator over the scene graphs in the training set. We employ
the learning under constraints framework introduced by Chamon und Ribeiro (2020) which results

6
One could also have used a different ϵi per constraint.

22
in a primal-dual saddle point optimization problem. For completeness, we will now describe this
algorithm in detail. We begin by explicitly writing out the empirical lagrangian L̂θ′ ,θ′′ ,ϕ,λ1 ,λ2 ,λ3 ,
 
n
1 X X ij
g1 (θ′ , θ′′ , ϕ) :=  dF F 
n i=1
(vi ,vj )∈EF F
 
n
1 X X ij
g2 (θ′ , θ′′ , ϕ) :=  dRF 
n i=1

(vi ,vj )∈ERF
 (25)
n
′ ′′ 1 X X ij 
g3 (θ , θ , ϕ) := oF F
n i=1
(vi ,vj )∈EF F

L̂θ′ ,θ′′ ,ϕ,λ1 ,λ2 ,λ3 := L(θ , θ , ϕ) + λ3 (1 − ϵ − g3 (θ′ , θ′′ , ϕ))

′ ′′

+ λ1 (g1 (θ′ , θ′′ , ϕ) − ϵ) + λ2 (g2 (θ′ , θ′′ , ϕ) − ϵ)

Here λ1 , λ2 , λ3 are the dual variables. The empirical dual problem is then defined as,
D̂∗ := max min L̂θ′ ,θ′′ ,ϕ,λ1 ,λ2 ,λ3 .
λ1 ,λ2 ,λ3 θ ′ ,θ ′′ ,ϕ

It was shown in Chamon und Ribeiro (2020) that a saddle-point optimization of this empirical dual
will give an approximate solution to (24). Algorithm 1 describes the exact steps. At step 5 we require
a ρ-optimal minimizer to the empirical Lagrangian, in practice this is done by running the ADAM
optimizer for one epoch. After each epoch t, the dual variables are updated depending on the slack
evaluated with current parameters (θ′(t−1) , θ′′(t−1) , ϕ(t−1) ). At an intuitive level, the algorithm is
similar to regularized optimization with adaptive Lagrange multipliers (the dual variables). In (25)
each term after L(θ′ , θ′′ , ϕ) can be thought of a regularizer (one corresponding to each constraint).
The Langrange multipliers are updated in each epoch to enforce or relax the regularizer depending on
whether the corresponding constraint is violated or satisfied.

Algorithm 1 Learning under constraints

Require: Initializations: θ′(0) , θ′′(0) , ϕ(0) ; Learning rate for dual variables η; Number of steps T
(0)
1: λ1 = 0
(0)
2: λ2 = 0
(0)
3: λ3 = 0
4: for t = 1, . . . , T do
5: Obtain θ′(t−1) , θ′′(t−1) , ϕ(t−1) such that,
L̂θ′ ,θ′′ ,ϕ,λ1 ,λ2 ,λ3 ≤ ′min L̂θ′ ,θ′′ ,ϕ,λ1 ,λ2 ,λ3 + ρ.
θ ,θ ,ϕ
′′

6: Update dual variables

h i
(t) (t−1)
λ1 = λ1 + η(g1 (θ′(t−1) , θ′′(t−1) , ϕ(t−1) ) − ϵ)
+
h i
(t) (t−1)
λ2 = + η(g2 (θ′(t−1) , θ′′(t−1) , ϕ(t−1) ) − ϵ)
λ2
+
h i
(t) (t−1)
λ3 = λ3 + η(1 − ϵ − g3 (θ′(t−1) , θ′′(t−1) , ϕ(t−1) ))
+
7: end for

In Figure 5, we show training curves for ELBO objective (3) and the ELBO objective with constraints
(24) which clearly show the benefit of using constraints. Empirically on the 3D-FRONT dataset,
getting good solutions is impossible (even after 900 epochs (70k iterations)) without using constraints.
The network performance without any constraints is comparable, to that with constraints, for the
furniture category, shape and size loss terms. These are arguably much easier to learn than the
orientation and position in 3D.

23
Figure 5: Ablation studies showing the effect of constraints on training. The orange curve in-
dicates learning under constraints whereas the blue curve indicates learning without any con-
straints(a) Training curves for the terms in the reconstruction loss of the ELBO (first term in (3); (b)
Objective values of the constraints as training progresses. Note the x-axis is the number of iterations
rather than epoch (one iteration is one batch processed).

6.6 Diverse furniture layout recommendations for the same room layout

More examples for Figure 1b in Figure 6

24
Figure 6: Diverse furniture layout recommendations for the same room layout. Each row depicts
a specific floor-plan, row 1: library; row 2, 3: living room, row 4: bedroom. The first column is the
ground truth design from the test set. The remaining columns are recommendations made by our
proposed model. All rooms are top-down rendering of the scene. We mark ceiling lamps with a white
asterisk in images where we believe it is hard to recognize from our top-down rendering. In row 1,
column 4, the green furniture on top of the sofa is a overhead cabinet.

6.7 Manipulating Latent Space for Design Recommendations from Database.

In this subsection, we show more Examples for Figure 3 in Figure 7. The procedure is as follows,

1. Create a Database:

• Iterate over the training set.

• For each scene, pass it through the GNN encoder to obtain the latent code. Store both
the scene and the corresponding latent code in the database.

2. Retrieving closest matched scenes

• Given an empty room layout G˜R , we iterate over each scene i in the database.
nF i
• For every scene i, the corresponding latent Zi = {Zi1 , . . . Zi } where nFi is the
number of furniture items in scene i.

25
 • Since Zi was sampled using our approximate posterior qϕ (Z | Gi , Ti , nFi ) we solve
(8) using the FAQ algorithm and the Graph prior pθ′′ (Z | G˜R , π, Ti , nFi )7 to find the
optimal ordering π ∗ . Here G˜R is the given empty room layout.
• We then evaluate the likelihood of Zi under our prior and optimal ordering π ∗
• Finally we choose the top 3 scenes which have the highest likelihood under the prior as
the closest "match". In other words, these latent codes are very likely under the prior
for room layout G˜R and thus would result in good designs when passed through our
GNN decoder.

In Figure 3 we showed multiple retrieval results for a library, here we give two more examples a
bedroom and a living room. Notice that the retrieved scenes can have very different room shape
compared to G̃R however the furniture arrangements can still look plausible in G̃R .

6.8 Qualitative comparison of our method with ATISS and baselines

In this subsection, we show more Examples for Figure 2 in Figure 8.

6.9 More results for scene editing

In this subsection, we show more Examples for Figure 4 in Figure 9.

7
Recall, given any ordering of the latent variables, π, the Graph Prior simulates and auto-regressive model
based on π. See (7)

26
Figure 7: Manipulating Latent Space for Design Recommendations from Database. We show
results for two room types (a) bedroom and (b) living room. The GNN decoder ensures the synthesized
design in G̃R has approximately the same spatial arrangement as the design of the retrieved scene.
However in some cases (middle room in (a) and top room in (b), the bed and sofa are rotated
respectively to ensure the satisfaction of constraints levied by the room layout, for example, sofa
should be parallel to the closest wall.

6.10 Analyzing matched furniture nodes for the trained autoregressive prior

In this subsection we analyze the following question - After training, what is the category of the
furniture’s latent, taken from an encoded scene using the GNN encoder, that gets matched to the first
Z 1 sampled using our autoregressive prior?
To answer this we carry out the following steps,

27
Figure 8: More results for Qualitative comparison of our method with ATISS and baselines. Row 1,2
are bedrooms. Row 3 is a living room and Row 4 is a library.

• Iterate over the scenes in the test set.

• Ever scene is a tuple consisting of the attributed scene graph, the room type and the number
of furniture’s in the room (G, T, nF ). For each scene,
– Pass it through the trained GNN encoder to obtain the parameters for qϕ (Z | G, T, nF ).
Here Z := {Z 1 , Z 2 , . . . Z nF } is the set of latent variables corresponding to the nF
furniture items to be placed in the room.
– Next solve (8) using the FAQ algorithm and the Graph prior pθ′′ (Z | GR , π, T, nF )8
to find the optimal ordering π ∗ . Here GR refers to the room layout graph derived from
the input scene graph.
– This π ∗ is the optimal assignment between Z i ’s obtained from the approximate poste-
rior qϕ (Z | G, T, nF ) and the sequential latent nodes sampled using the auto-regressive
prior (See (5)).
• Since each Z i corresponds to a furniture item, we analyze the frequencies (over the test set)
with which a certain category is mapped to the first latent sampled by the auto-regressive
prior. This can give insights into what the model has learnt and whether the prior’s modelling
of furniture placement in indoor scenes correlates with how interior designers begin planning
room layouts. In Table 5 we present these results. We found that the first bedroom item
matched by our model is a nightstand/bed/light with probability 0.91 and the first living
room item matched is a coffee-table/sofa/light with probability 0.76. This is interesting
since human designers often start planning with these items.

8
Recall, given any ordering of the latent variables, π, the Graph Prior simulates and auto-regressive model
based on π. See (7)

28
Figure 9: Scene Editing. Col 1 is a scene generated by our model. In row 1, we morph the bottom-left
cabinet into a sofa by changing the α parameter as explained in text (§4). In row 2 we morph the
yellow chair in the top-center into a sofa. In row 3, we morph the cabinet (marked with a yellow spot)
into a chair.

Table 5: Frequency of categories mapped to the first latent sampled by the auto-regressive prior

Categories Bedroom Livingroom

Bed 0.30 0.0
Light 0.19 0.19
Night-stand 0.43 0.0
Chair 0.02 0.11
Sofa 0.01 0.20
Table 0.04 0.10
Pier 0.01 0.01
Coffee-Table 0.0 0.39

29