OneDiffusion Demo without captioner

Welcome to the OneDiffusion Demo!

This application allows you to generate images based on your input prompts for various tasks. Here's how to use it:

1. Select Task Type: Choose the type of task you want to perform from the "Task Type" dropdown menu.

2. Upload Images: Drag and drop images directly onto the upload area, or click to select files from your device.

3. Generate Captions: If you upload any images, Click the "Generate Captions" button to format the text prompt according to chosen task. In this demo, you will NEED to provide the caption of each source image manually. We recommend using Molmo for captioning.

4. Configure Generation Settings: Expand the "Advanced Configuration" section to adjust parameters like the number of inference steps, guidance scale, image size, and more.

5. Generate Images: After setting your preferences, click the "Generate Image" button. The generated images will appear in the "Generated Images" gallery.

6. Manage Images: Use the "Delete Selected Images" or "Delete All Images" buttons to remove unwanted images from the gallery.

Notes:

• Check out the Prompt Guide.

• For text-to-image:

  • simply enter your prompt in this format "[[text2image]] your/prompt/here" and press the "Generate Image" button.

• For boundingbox2image/semantic2image/inpainting etc tasks:

  • To perform condition-to-image such as semantic map to image, follow above steps
  • For image-to-condition e.g., image to depth, change the denoise_mask checkbox before generating images. You must UNCHECK image_0 box and CHECK image_1 box. Caption is not required for this task.

• For FaceID tasks:

  • Use 3 or 4 images if single input image does not give satisfactory results.
  • All images will be resized and center cropped to the input height and width. You should choose height and width so that faces in input images won't be cropped.
  • Model works best with close-up portrait (input and output) images. For example add some keywords such as detailed face, closeup portrait
  • If the model does not conform your text prompt, try using shorter caption for source image(s).
  • If you have non-human subjects and does not get satisfactory results, try "copying" part of caption of source images where it describes the properties of the subject e.g., a monster with red eyes, sharp teeth, etc.

• For Multiview generation:

  • The input camera elevation/azimuth ALWAYS starts with 0. If you want to generate images of azimuths 30,60,90 and elevations of 10,20,30 (wrt input image), the correct input azimuth is: 0, 30, 60, 90; input elevation is 0,10,20,30. The camera distance will be 1.5,1.5,1.5,1.5
  • Only support square images (ideally in 512x512 resolution).
  • Ensure the number of elevations, azimuths, and distances are equal.
  • The model generally works well for 2-5 views (include both input and generated images). Since the model is trained with 3 views on 512x512 resolution, you might try scale_factor of [1.1; 1.5] and scale_watershed of [100; 400] for better extrapolation.
  • For better results:
    1. try increasing num_inference_steps to 75-100.
    2. avoid aggressively changes in target camera poses, for example to generate novel views at azimuth of 180, (simultaneously) generate 4 views with azimuth of 45, 90, 135, 180.

Enjoy creating images with OneDiffusion!