SawadaTomoya

Abstract

Stroke-based rendering aims to reconstruct an input image into an oil painting style by predicting brush stroke sequences. Conventional methods perform this prediction stroke-by-stroke or require multiple inference steps due to the limitations of a predictable number of strokes. This procedure leads to inefficient translation speed, limiting their practicality. In this study, we propose MambaPainter, capable of predicting a sequence of over 100 brush strokes in a single inference step, resulting in rapid translation. We achieve this sequence prediction by incorporating the selective state-space model. Additionally, we introduce a simple extension to patch-based rendering, which we use to translate high-resolution images, improving the visual quality with a minimal increase in computational cost. Experimental results demonstrate that MambaPainter can efficiently translate inputs to oil painting-style images compared to state-of-the-art methods. The codes are available at this URL.

Demo

Source Code

https://github.com/STomoya/MambaPainter

Reference

Tomoya Sawada and Marie Katsurai, “MambaPainter: Neural Stroke-Based Rendering in a Single Step,” in SIGGRAPH Asia 2024 Posters (SA ’24), Association for Computing Machinery, New York, NY, USA, Article 98, 1–2, 2024. DOI: 10.1145/3681756.3697906.

Abstract

Super-deformation in character design refers to a simplified modeling of character illustrations that are drawn in detail. Such super-deformation requires both texture and geometrical translation. However, directly adopting conventional image-to-image translation methods for super-deformation is challenging as these methods use a pixel-wise loss which makes the translated images highly dependent on the spatial information of the input image. This study proposes a novel deep architecture-based method for the super-deformation of illustrated character faces using an unpaired dataset of detailed and super-deformed character face images collected from the Internet. First, we created a dataset construction pipeline based on image classification and character face detection using deep learning. Then, we designed a generative adversarial network (GAN) that was trained using two discriminators, each for detailed and super-deformed images, and a single generator, capable of synthesizing identical pairs of characters with different textural and geometrical appearance. As ornaments are an important element in character identification, we further introduced ornament augmentation to enable the generator to synthesize a variety of ornaments on the generated character faces. Finally, we constructed a loss function to project character illustrations provided by the user to the learned GAN latent space, which can find an identical super-deformed version. The experimental results show that compared to baseline methods, the proposed method can successfully translate character illustrations to identical super-deformed versions. The codes are available on the Internet.

Demo

The following visualization shows how the source image (left) is gradually translated into the target domain (right).

Source Code

https://github.com/STomoya/ChibiGAN

Reference

Tomoya Sawada, Marie Katsurai, “Illustrated Character Face Super-Deformation via Unsupervised Image-to-Image Translation,” Multimedia Systems 30, 63, 2024. DOI: 10.1007/s00530-023-01255-y.