Inpainting on NightCafe

Inpainting on NightCafe

Original article by creator @Weresl0th. Big thanks from the NightCafe team!

What is Inpainting in AI Image Generation?

Inpainting is a technique of changing an image that involves filling in the missing or undesired parts of the image, or removing the undesired object to create an image that meets the creator’s wants and needs.  This is accomplished by using a simple interface to paint a “mask” that tells the algorithm what parts of the image you want changed using text prompts.

Examples of inpainting include, but are not limited to:

  1. Minor alterations to fix images (removing additional limbs, unwanted objects)
  2. Changing/Removing a border or frame
  3. Removing text

What Inpainting can’t do easily:

  1. Change the color or style of an image
  2. Add specific, coherent text (this has the same limitations as other generations)

Other Constraints to think about for inpainting:

  1. Inpainting will use Stable Diffusion 1.5 and SDXL 1.0 checkpoints. You can create an image with a non-Stable Diffusion checkpoint/algorithm, but you would need to switch checkpoints before having the available options.

How to use Inpainting on NightCafe

Inpainting refers to masking your start image such that only a specific part of it gets affected by your prompt.


Inpainting is an experimental feature - not every use of it will yield the results you want, so please be ready for results which are unwanted. With that said, this guide will help you learn to use it to improve images that require changes. We've found that it works well when evolving a creation, but you can also add a start image and use inpainting to change it.

Setting the Advanced Mode toggle 

To access inpainting, you must have advanced mode turned on.  The advanced mode toggle is located at the top left (on Desktop) and top of the page (on Mobile) of the studio view.


(left image - Advanced mode Off, right image - Advanced mode On)

To set it on, click or tap on the tag next to the words “Advanced mode” and check that it is now green with the toggle to the right. This will open up access to the advanced mode settings, including the advanced prompt editor.  


For this example, an image of a holographic teacup and ship will be altered, changing the ship floating on top of the teacup to a cat.

Step 1: Begin by evolving the creation.

  1. The prompt of the original creation
  2. The settings of the original creation
  3. The creation’s image result shown at the time you clicked/tapped will be set as the Start Image if the source was a Stable Diffusion creation.

As a reminder, if the source wasn’t Stable Diffusion on NightCafe, you will need to choose a different checkpoint/algorithm.  Stable Diffusion 1.5 and SDXL 1.0 checkpoints are supported.

Step 2: Create a Mask

You will see a setting for Start Images in the Studio Panel. Clicking on that will bring you to the Start Image view, which will include a section for Inpainting. From there, click on the “Choose/Create Mask” button. This will bring you to an interface to “paint” a simple mask.



The options in the bottom left corner are, from right to left:
  1. Paint - add masked sections in RED (just to show where you've masked) 
  2. Eraser - erase what you've masked (this is not to erase parts of the start image) 
  3. Pan - move the image around the screen; can be useful if you're zoomed in 
  4. Brush Size - resize the brush using a slider 
  5. Zoom in - zoom in on the image; helps to mask fine areas 
  6. Zoom out - the opposite; helps to view the whole image and use the larger brush sizes to quickly mask more of the image
  7. Toggle - show just the masked areas without the image shown; can be toggled back and forth
  8. Feathered Mask - By default, NightCafe adds feathering to the outer edges of your masked area.  This can allow for more blending at those edges.  You can toggle this to turn it on or off.

Keyboard Shortcuts

You can also access these via keyboard shortcuts:

Key Shortcut
Action
K
Toggle Shortcuts
Hold Alt
Eraser
Hold Space
Pan
Z + Click
Zoom In
Alt + Z + Click
Zoom Out
[
Brush Size -
]
Brush Size +


Step 3: Paint the Mask

With your finger or mouse pointer, you can paint a mask that will show up as red. This is the area you want changed by the Text Prompt (shown below).


If you have feathered masking turned on, it will look similar to this:

Hints/Tips:

  1. NightCafe will allow any amount of the image to be inpainted, including having separate areas.  Your text prompt will be applied to all areas masked.  If you need to have distinct, different changes made you may need to do this in multiple inpainted creations.
  2. While any shape or amount, including separate sections can be masked, for cohesion having contiguous areas joined may be helpful for the composition.
  3. It may be better to mask slightly beyond the area to be changed to allow for NightCafe to try to blend your results.

Step 4: Apply the mask.

When done with painting the mask, you must click the green done button to apply the mask. Skipping this will result in your entire image being changed by the prompt.


After clicking the green button, you will be returned to the Start Image Interface.  The red masked area will now appear black.

Step 5: Adjust the Noise Weight and other settings (optional)

You can adjust how much NightCafe will change the masked area by adjusting the Noise Weight.  Higher percentage amounts means the image may change more than what is currently present in the image.


Notes:

  • This concept applies to using start images on NightCafe regardless of whether you’ve applied a mask for inpainting.

  • Even at 0% the masked area will change.


Before or after applying the mask and adjusting the Noise Weight, you can write or edit the Text Prompt.  If you’ve evolved an image then the original prompt will be present.

In this example we’re replacing the floating ship with a cat.  This will be done by changing the portions of the prompt that referenced a ship or other subject features, and replacing them with references to a cat. Ideas/Concepts to consider and utilize:
  1. The same text-to-image concepts for new creations apply here.  In other words, write what you want NightCafe to replace the masks sections with.
  2. Avoid providing commands like “erase, replace”.  NightCafe is already doing this.
  3. Unless trying to create a mix of styles in the same creation, keep the same artistic modifiers (artist name, styles, mediums) in the prompt.
  4. Change/Refine the modifiers for the subject.
  5. Use negative weighted prompts to refine what you don’t want NightCafe to create.

Step 6: Hit Create when you are done - and see what happens!


    • Related Articles

    • What is Inpainting in AI and how to use it?

      What is Inpainting in AI? Inpainting is a technique of filling in missing regions of images that involves filling in the missing or damaged parts of an image, or removing the undesired object to construct a complete image. However with the power of ...

    • Negative Prompts on NightCafe

      Original article by creator @Weresl0th. Big thanks from the NightCafe team! What is a Negative Prompt in AI? Negative weighted prompts (also referred to as negative prompts) are a way to instruct the algorithm that you do not want the creation to be ...

    • Introducing Fine-Tuning on NightCafe

      Fine-tuning is a feature that offers a more personalized AI experience. This allows you to train a model to understand and recreate specific styles, faces, or objects. How to Get Started with Fine-Tuning Access My Models: PC: Find it in the main menu ...

    • The Coherent algorithm - CLIP-Guided Diffusion text-to-image AI Art Generator

      NightCafe now also offers a second text-to-image AI algorithm called CLIP-Guided Diffusion, which we refer to as the "Coherent" algorithm. This algorithm is fantastic at generating more realistic images, composed in a believable way, to look more ...

    • How to upload start image?

      A start image allows you to initialise the creation process with an image rather than random pixels. To upload an image from your device, please follow these steps: 1. Make sure the Advanced mode on your Create interface is toggled on. 2. Clicking on ...