AI Image Tools

Polotno includes the editor side of four AI image tools: Expand, Generative fill (replace or remove an object), Upscale and Background removal. The SDK draws the frames and selections, prepares the pixels, shows progress, applies the result in one undo step and handles cancellation. You connect the AI provider.

The SDK does not include an AI provider for Expand, Generative fill or Upscale, and it never uploads anything. Your handler sends the request to your provider, handles authentication, limits and storage, and returns the image.

Live demo

The demo connects every tool to Polotno's AI endpoints, with a daily limit of edits per visitor. Select the photo and open AI edit in the toolbar.

Stock UI

When an image is selected, the image toolbar shows an AI edit menu. Each entry appears only when its handler is registered:

If no entry is available, the menu is hidden. Crop needs no handler.

The tools work on editable image elements. Expand also needs a resizable element. Users with the viewer role do not see the tools.

Register handlers

Register the handlers once, at startup. They are global and need no store:

import {
  setExpandFunc,
  setReplaceObjectFunc,
  setRemoveObjectFunc,
  setUpscaleFunc,
  setRemoveBackgroundFunc,
} from 'polotno/config';

setExpandFunc((req) => api.expand(req));
setReplaceObjectFunc((req) => api.replace(req));
setRemoveObjectFunc((req) => api.remove(req));
setUpscaleFunc((req) => api.upscale(req));
setRemoveBackgroundFunc((src, signal) => api.removeBackground(src, signal));

Pass null to disable a handler and hide its entry. For background removal, null restores the Polotno Cloud default. See Remove Image Background for the Cloud option.

Requests

Expand, Generative fill and Upscale handlers receive a request with these fields:

Each tool adds its own fields:

• Expand adds frame and an optional prompt. frame is the new area in coordinates relative to image: { x: -0.25, y: 0, width: 1.5, height: 1 } adds a quarter of the width on each side. image is not padded. Use prepareImageInput and padImageForFrame to build the padded image and its mask.
• Replace adds mask, region and prompt.
• Remove adds mask and region, without a prompt.

mask is a PNG Blob with the same pixel size as image. White pixels are the area to edit. Black pixels must stay the same. region is the bounding box of the selection, in coordinates relative to image. The mask can select less than the full box, so send the mask to your provider.

A background removal handler receives the original image URL instead of pixels:

setRemoveBackgroundFunc(async (src, signal, request) => {
  // request is { src, signal, options }
  return await api.removeBackground(src, signal); // a URL or { src }
});

Results

Handlers return Promise<{ src }>. src must be a durable URL that the browser can load with CORS, or a data URL. blob: URLs are rejected, because they stop working when the page closes. Store the result on your own server when you do not want large data URLs in the design JSON.

The result must cover the full requested view, not only the selection:

• For Generative fill and Upscale, the result must have the aspect ratio of image.
• For Expand, the result must have the aspect ratio of the full frame.

The SDK accepts up to 2% relative deviation from that aspect ratio and fits the result into the element with a centered crop. A result outside that tolerance rejects with invalid-result.

The result replaces the visible image at the resolution your provider returns. Expand also enlarges the element to the new frame. Filters and effects stay on the element and stay editable. Pixels that were hidden by the crop are kept only in the undo history. Background removal changes only the element's src.

Custom results

A handler can return an apply callback to change how the result is applied. The callback replaces the default update:

setReplaceObjectFunc(async (req) => {
  const src = await api.generateTransparentObject(req);
  return {
    src,
    apply({ element, selection }) {
      // Add the generated object as a new layer above the photo.
      const layer = element.parent.addElement(
        { type: 'image', src, ...selection },
        { skipSelect: true },
      );
      layer.setZIndex(element.zIndex + 1);
    },
  };
});

setUpscaleFunc(async (req) => ({
  src: await api.upscale(req),
  apply({ element, applyDefault }) {
    applyDefault();
    element.set({ custom: { upscaled: true } });
  },
}));

The callback receives:

• element: the image the request started from.
• src: the loaded result.
• selection: for Generative fill, the selected region in document units with rotation. For the other tools, null.
• applyDefault(): applies the result as the SDK would, including the aspect check.

The SDK loads src and checks that the request is still current before it calls apply. All changes in apply make one undo step. apply must be synchronous and return nothing, so finish all asynchronous work before the handler returns. If you do not call applyDefault(), your callback controls how the image fits, its parent, its stacking order and the selection. If apply throws, the SDK reverts its changes and rejects with handler-failed.

Cancellation

The SDK aborts signal when the user cancels the edit, and when the image changes in a way that makes the request invalid, for example after undo. Changes to other elements do not cancel the request. Pass signal to fetch or to your provider SDK, so the request stops too:

setUpscaleFunc(async ({ image, signal }) => {
  const body = new FormData();
  body.append('image', image, 'image.png');
  const res = await fetch('/api/upscale', { method: 'POST', body, signal });
  if (!res.ok) throw new Error(`Upscale failed (HTTP ${res.status})`);
  const { url } = await res.json();
  return { src: url };
});

If a handler ignores signal, the edit still ends right away and the SDK ignores the late result.

A handler can also stop the edit quietly by throwing an AbortError, for example after it shows an upgrade dialog. The edit then rejects as canceled and the UI shows no error.

Error messages

When a handler throws, the stock UI shows a generic failure message. To show your own text, put a safe, localized userMessage on the error:

if (res.status === 429) {
  throw Object.assign(new Error('Rate limited'), {
    userMessage: 'Daily limit reached. Try tomorrow.',
  });
}

Keep userMessage short: the stock UI shows it in a one-line status label on the image.

Failed edits reject with structured errors. TOOL_FAILED identifies a tool failure, and details.reason tells you why:

If the source image or the result cannot be loaded or prepared, the edit rejects with IMAGE_FAILED instead. The stock UI shows no error for canceled, stale-result and busy.

Image helpers

polotno/utils/image-edit has helpers for provider adapters:

• prepareImageInput(input, { maxSide, roundTo }) resizes a request's image to fit your provider. maxSide is the longest side in pixels, up to 4096, and roundTo makes the dimensions multiples of that number. It resizes mask together with the image. For an Expand request, it returns the layout of the padded output, and maxSide applies to the full padded output.
• padImageForFrame(prepared) takes the Expand result of prepareImageInput and returns { image, mask, size }: the padded image and a mask where the new area is white.
• formatMask(mask, { selection }) redraws a mask for providers with another convention. 'black' inverts it. 'transparent' makes the selection transparent and the rest opaque, as OpenAI image edits expect.

The helpers throw TypeError or RangeError for arguments they cannot use.

import { setExpandFunc } from 'polotno/config';
import { padImageForFrame, prepareImageInput } from 'polotno/utils/image-edit';

setExpandFunc(async ({ image, size, frame, prompt, signal }) => {
  const prepared = await prepareImageInput(
    { image, size, frame },
    { maxSide: 2048 },
  );
  const padded = await padImageForFrame(prepared);
  // padded.image and padded.mask have the same pixel size
  return { src: await api.expand({ ...padded, prompt, signal }) };
});

import { setReplaceObjectFunc } from 'polotno/config';
import { formatMask } from 'polotno/utils/image-edit';

setReplaceObjectFunc(async ({ image, mask, prompt, signal }) => {
  const openaiMask = await formatMask(mask, { selection: 'transparent' });
  return {
    src: await api.openaiEdit({ image, mask: openaiMask, prompt, signal }),
  };
});

Build your own UI

You can run the tools from your own UI. store.setTool() starts a tool session on an image without opening a panel:

const expand = store.setTool('expand', { elementId });
expand.setFrame({ x: -0.25, y: 0, width: 1.5, height: 1 });
await expand.run({ prompt: 'Continue the landscape' });

const inpaint = store.setTool('inpaint', { elementId });
inpaint.setRegion({ x: 0.2, y: 0.2, width: 0.4, height: 0.4 });
await inpaint.run({ operation: 'remove' }); // Or 'replace' with a prompt.

if (store.canUseTool('upscale', elementId)) {
  const upscale = store.setTool('upscale', { elementId });
  await upscale.run({ options: { model: 'creative', scale: 4 } });
}

• setTool() supports crop, expand, inpaint, upscale and remove-background. More tool names can come in minor releases, so give an exhaustive switch a default branch.
• store.canUseTool(name, elementId) tells you if setTool() would succeed. Use it to show or hide your own buttons.
• run() resolves after the result is in the design. run({ options }) passes your data, such as a model or price tier, to the handler as req.options.
• store.activeTool is the open session. session.cancel() discards the draft and cancels a running request.
• element.pendingEdit is the running request on an image, with tool and cancel(). cancel() stops only the request.

Put credits, plans and model choices in your own UI. You can replace the toolbar menu with Toolbar components={{ ImageEdit: MyMenu }}, the controls of an open tool with ImageToolActions, or the tool panel with a side panel section named image-edit. The SDK still draws the frames, selections, progress and status on the canvas.

Hide stock entries

• Do not register a handler, or pass null, to hide its entry.
• Call setRemoveBackgroundEnabled(false) to hide Remove background, also when you registered your own handler.
• Hide the full AI edit menu from the toolbar:

<Toolbar store={store} components={{ ImageEdit: () => null }} />