L-MAGIC: Language Model Assisted Generation of Images with Coherence (2025)

3.1 Warping

To ensure the local inpainting consistency of each perspective view, we use a large field of view (FoV) and adjust the rotation angles so that both known and unknown regions are reasonably large after warping. In practice, a FoV of 100 degrees with roughly 40 degrees of rotation between adjacent views works well. To further reduce the iterative error accumulation, we expand the scene alternatively from both sides of the input image, rather than expanding in a single direction. To ensure a smooth $360^{\circ}$ loop closure, we tune the rotation angles so that the final view has a large incomplete region at the center, resulting in a sequence of views with rotation angles of $\{0^{\circ},41^{\circ},-41^{\circ},82^{\circ},-82^{\circ},123^{\circ},200.5^{$.

3.2 Inpainting with Language Model Assistance

The inpainting step completes a warped view with a consistent local style and a coherent $360^{\circ}$ scene layout.We utilize the Stable Diffusion v2 inpainting model[19], which can effectively extrapolate the large missing region of each warped view while maintaining local style consistency. However, naive inpainting without any prior will generate severe artifacts. One common prior explored before[11, 25] is a user-provided text description of the scene or input image.Yet, using the same description in different views may generate duplicate objects such as multiple beds in a bedroom (see Fig.4), since perspective inpainting methods have no mechanism to split the layout into different views. Please refer to Sec.4.3 for a detailed ablation study.

To address these problems, we use a vision language model ${\mathcal{L}}_{\text{v}}(\cdot)$ (BLIP-2[13]) and a language model ${\mathcal{L}}(\cdot)$ (ChatGPT4[4]) to guide the inpainting process.In the following, we describe our method (Alg.1) in detail. The exact prompts used in Alg.1 to interact with language and diffusion models are provided in AppendixA.

Before warping and inpainting, we first prompt ${\mathcal{L}}_{\text{v}}(\cdot)$ to generate the description $d_{\mathcal{I}}$ for the input image ${\mathcal{I}}$ (line2).We ask two questions so that $d_{\mathcal{I}}$ contains both coarse and fine levels of detail.Next, we ask ${\mathcal{L}}(\cdot)$ to imagine the global scene layout $d_{360}$ (line3) based on $d_{\mathcal{I}}$, where each line of $d_{360}$ corresponds to the description of a specific view. To avoid duplicate objects, we request a compact description of individual views without mentioning objects in other views.

$d_{360}$ mostly contains objects of individual views. Using such descriptions as the inpainting prompt can lead to inconsistent style at distant views. Hence, we ask ${\mathcal{L}}(\cdot)$ to remove objects from $d_{\mathcal{I}}$ and obtain the final scene-level description $d_{\text{scene}}$, e.g., ‘a bedroom with a wooden bed’ becomes ‘a bedroom’ (line4). $d_{\text{scene}}$ is later used together with $d_{360}$ to ensure a consistent multi-view style.

Though $d_{\text{scene}}$ ensures the multi-view style consistency, the training data bias of diffusion models may still result in objects commonly associated with a particular scene being generated, even if not explicitly mentioned in $d_{360}$. For example, a bed is often generated with the word ‘bedroom’ in the prompt, resulting in duplicate beds in multiple views. To avoid this problem, we further let ${\mathcal{L}}(\cdot)$ automatically determine whether there are some objects in the scene that require repetition avoidance (line5).

After each warping step, we use the outputs from lines2 to5 to automatically generate the prompt for text-conditioned inpainting (line12). Specifically, for the warped view with $0^{\circ}$ rotation (i = 1), we use $d_{\text{scene}}$ as the prompt ($d_{i}$) for text-conditioned inpainting (line7). For other views (line13), if there is no object in $d_{\text{repeat}}$, i.e., no repetition avoidance required, we perform inpainting with the prompt ‘a peripheral view of $<$$d_{\text{scene}}$$>$ where we see $<$the corresponding description in $d_{360}$$>$’. If any object exists in $d_{\text{repeat}}$, we use the positive prompt of ‘a peripheral view of $<$$d_{\text{scene}}$$>$ where we only see $<$the corresponding description in $d_{360}$$>$’, and the negative prompt of ‘any type of $<$the object in $d_{\text{repeat}}$$>$’ (one sentence for each object in $d_{\text{repeat}}$). The positive prompt template prevents Stable Diffusion from generating common objects of an environment (e.g., the bed in a bedroom). The negative prompt template avoids duplication of objects mentioned in $d_{\text{repeat}}$.

Bias exists in the training data of diffusion models — an image with the caption of ‘a bedroom’ mostly contains a bed. Therefore, repeated objects can still be generated even with constraints from the prompt. To further alleviate this problem, we use ${\mathcal{L}}_{\text{v}}(\cdot)$ to check whether each inpainted image ${\mathcal{I}}_{i}$ contains objects mentioned in $d_{\text{repeat}}$ (line15).If the answer is ‘yes’, we re-run inpainting until the answer becomes ‘no’ or the maximum number of trials $c$ is reached.

3.3 Quality and Resolution Enhancement

Several techniques are also proposed to enhance the quality and resolution of the final panorama.

As shown in Appendix.B, adjacent pixels at the center of an image have a larger angular distance than the ones at the side of an image. When warping a completed view to a novel view, the original central region becomes the side region, making the rendered image blurry due to interpolation. Meanwhile, the resolution of the Stable Diffusion output is $512*512$, the panorama created by these images has a low resolution. To address both problems, we apply super-resolution[3] to the output ${\mathcal{I}}_{i}$ of each inpainting step, increasing the resolution of ${\mathcal{I}}_{i}$ to $2048*2048$. Then, we warp the high-resolution image to a low-resolution novel view so that no (strong) interpolation is required. After performing all warping and inpainting steps, we simply fuse the super-resolution images to generate a high-resolution panorama.

During warping and panorama generation, multiple perspective images might have overlaps at the same region. To avoid sharp boundaries when merging them, we perform a weighted average, i.e., given multiple warped pixels at the same location with colors ${\mathbf{c}}_{i}$, the final merged pixel color is ${\mathbf{c}}_{\text{merge}}=\frac{\sum_{i}w_{i}{\mathbf{c}}_{i}}{\sum_{i}w_{i}}$, where the weight $w_{i}$ is computed as the distance to the nearest image boundary at the original view $i$. This strategy effectively down-weights the pixels near the warping boundaries, ensuring a smooth transition during multi-view fusion.

To create the final panorama (line22), we first project each view to the unit sphere same as in Sec.3.1. Then, we perform the equirectangular projection[5] to warp multiple projected views to the same equirectangular plane, and merge them into a single equirectangular image.

3.4 Discussion

L-MAGIC is fully automatic — no human interaction is required to link language models and diffusion models. This is realized by 1) careful prompt engineering, which enables language models to output texts that can be automatically converted into the inpainting prompt, and 2) handling the edge cases where language models or diffusion models do not generate outputs that satisfy the requirements in the input prompt. For example, ChatGPT sometimes still outputs the layout description $d_{360}$ with an erroneous format, making automatic prompt extraction fail catastrophically. We automatically detect such cases, and re-run line3 to ensure the algorithm flow, see AppendixA for more details.

L-MAGIC requires no model fine-tuning, which ensures the zero-shot performance and makes it capable of accepting other types of inputs leveraging conditional generative models (see Sec.4 for results). This advantage also allows individual modules to be replaced by future methods to enhance the performance, e.g., change BLIP-2+ChatGPT to GPT-4V[1], or use other inpainting models.

4.1 Experimental Setup

Implementation Details. L-MAGIC is implemented with PyTorch[15] and the official release of BLIP-2[13], Stable Diffusion[19] and the ChatGPT4 API[4]. It takes 2-5 minutes to generate a $360^{\circ}$ panorama depending on the number of repetitions in line17 of Alg.1. We take the official code and model for all other methods. In text-to-panorama, L-MAGIC uses Stable Diffusion v2 conditioned on the given text prompt to generate the input image.

Data. To evaluate the in-the-wild performance, we collect data that does not overlap with the training data of any method for both tasks. For image-to-panorama, we use 20 indoor and 20 outdoor images from tanks-and-temples[12] and RealEstate10K[31]. For text-to-panorama, we use ChatGPT to generate 20 random scene descriptions (10 indoor and 10 outdoor, see AppendixC).

Metrics. To evaluate different methods with respect to quality and multi-view diversity, we compute the Inception Score (IS)[20] for the perspective views of the panorama. Since existing quantitative metrics do not capture all aspects of human perception of quality[7], we follow existing works[11, 8, 14] for a complementary human evaluation. To this end, we set up a voting web page that shows side-by-side panoramic scenes generated using the same input, one with our method and one with a baseline. We ask 15 anonymous voters to choose which panorama has higher quality and scene structure (see AppendixD for an example voting page). To minimize voting bias, we randomly shuffle the order of the side-by-side panoramas and hide the generation method names. We use the votes to compute a preference score from 0 to 1 for our method compared to the baselines. This score is simply the percentage of votes for our method with respect to quality and structure.

4.2 Main Results

As shown in Fig.3, our method performs better for both image-to-panorama and text-to-panorama, even compared to task-specific methods. To further understand the performance of different methods, we provide in Fig.4 and5 the qualitative results for both tasks. Stable Diffusion v2 treats a panorama as a single image. It cannot close the $360^{\circ}$ loop since equirectangular projection[5] splits the loop-closure area to two sides of an image (moved to the middle of the rendered panorama for better visualization). In the meantime, due to the lack of large-scale panorama training data, it still generates unnecessarily repeated objects such as multiple beds in a bedroom. Text2room and MVDiffusion separate a panorama into multiple perspective views. Inpainting them using the same prompt results in unreasonably repeated objects in multiple views. Due to the limited panorama training data, Text2light cannot fully understand zero-shot scene descriptions generated by ChatGPT, resulting in scenes not consistent with the input prompt. Similar to Stable Diffusion v2, treating panorama as a single image makes it fail on loop closure. LDM3D is fine-tuned on top of a perspective latent diffusion model. Though better than Text2light, it still cannot close the loop and sometimes fails to generate scenes that are consistent with the details of the prompt (e.g., generating a non-modern living room when asked for a modern one). Our method works robustly on various inputs, generating panoramic scenes with high perspective rendering quality and reasonable $360^{\circ}$ scene layouts (see supplementary videos for more details).

4.3 Analysis

We further analyze different components of our method in this section. The analysis is conducted from 3 aspects: 1) the scene prior, 2) the inpainting method and 3) quality enhancement techniques. For each aspect, we remove a component or replace it with other methods, and perform the same evaluation as in the main experiments. We use the same data used in the image-to-panorama main experiment for analysis. The results are reported in Fig.6. Please refer to AppendixE for the visualization comparisons.

For the scene prior, we remove the prior from chatGPT (no GPT) and all text guidance (no prompt) respectively. Without the global layout prior from chatGPT, the structure of the outputs becomes worse. Without any prompt guidance, both the scene layout and the perspective view rendering quality degrade severely.

For the inpainting method, we replace the Stable Diffusion v2 model with 3 state-of-the-art text-conditioned inpainting methods, namely, Blended Latent Diffusion (BLD inpaint)[6], Deep Floyd (DF inpaint)[2] and Stable Diffusion XL (SDXL inpaint)[16]. Interestingly, though some of the methods[16] are published later than Stabld Diffusion v2, their capacity to perform large mask inpainting is limited on in-the-wild images, resulting in worse performance in terms of both scene layouts and rendering quality. Note that the inception scores for some methods (e.g., BLD inpaint) are higher than ours despite a much worse performance from human evaluation. This is caused by the adversarial samples generated by these methods (see AppendixF for examples), where the local patches of the adversarial samples are not consistent with the input image, yet leading to a high diversity in the inception score. Similar issues have also been discovered in other problems[7], which shows the importance of human evaluation.

For the quality enhancement techniques, we remove respectively the super-resolution (no SR) and smoothing (no smooth) techniques mentioned in Sec.3.3. Though the scene layout does not degrade much, the perspective view rendering quality is lower due to the blur or artifacts.

4.4 Applications

Combining matured computer vision techniques makes our pipeline applicable to a wide range of applications.

Anything-to-panorama. Conditional diffusion models[29, 19] can now generate an image from diverse types of inputs. The strong zero-shot performance of L-MAGIC makes it possible to generate panoramic scenes from potentially any inputs compatible with conditional diffusion models. As shown in Fig.7, we can use[29] to generate a single image from 1) a depth map, 2) a sketch drawing or 3) a colored script or segmentation image. Then, this generated image can be used in L-MAGIC to produce realistic panoramic scenes. This flexibility makes L-MAGIC beneficial to a wide range of design applications.

3D scene generation. Applying state-of-the-art depth estimation models, we can further generate 3D scenes from the output of L-MAGIC. Specifically, we compute the depth map for multiple perspective views, then we merge the depth maps into the equirectangular image plane by aligning all views to the initial view. Then we convert the corresponding panoramic depth map to a 3D point cloud. We use Metric3D[27] and DPT-hybrid[17] to estimate the depth for indoor and outdoor scenes respectively. The alignment is done by optimizing the scale and shift of each depth map to enforce the depth from multiple views to be the same at the same pixel. Fig.8 shows sampled results. Despite some artifacts caused by the limitation of monocular depth models, L-MAGIC can generate diverse indoor and outdoor point clouds from various types of inputs.

Immersive video. One can also render immersive videos from our panorama. Specifically, we first generate a panorama using our pipeline, and then generate a series of camera poses for individual video frames. Then we warp the panorama to each frame view according to the camera poses. To further enable scene fly-through with camera translations, we apply depth-based warping when translation is involved in a frame, and inpaint the missing region introduced after translation. See AppendixG for implementation details and the supplementary videos for the results.