A RESEARCH SCIENTIST
GenerativeAI
Diffusion/GAN based Generative Models
Implicit/Explicit Nueral Rendering
Vision-Multimodal Learning
Multi-node Distributed Training

1 minute read

Manipulating Latent vectors

In this project, we trained the StyleGAN generator and manipulated the latent space of the generator.

Interpolating face identities and corresponding Non-Adversarial Domain Adaptation (NADA)

We trained our own generator, i.e. a variant of StyleGAN’s mapper and synthesizer, and then interpolates randomly sampled latent vectors. In addition, we applied the StyleGAN-NADA approach where driving prompts are “character”, “caricature”, characture, “dc comics”, “marvel comics”, and “disney 3D”, respectively.

test0 (click above and make sure full-screen mode)

We defined three stages to edit latents:

  • Sampling: A stage that samples z, w, w+, and image.
  • Inversion: Optional stage for conditioning on a given image. If there is an input image we want to edit, we first apply inversion from the image to get a style-latent w+.
  • Editing: Manipulating the latent vector acquired from the previous stage. In this stage, we adopt a flow-based (i.e. StyleFlow), driving prompt-based (CLIP and LAION) editing. Note that not only editing a latent vector but fine-tuning our synthesizer as well, if necessary.

Steerable attributes we trained are geometry, lighting, age, gender, race, glasses, hat, makeup, facial expression, etc. Here we showcase various face-related attributes manipulation from our trained generator and manipulator. The key is that face identity must not be changed when conducting editing.

  • Geometry (Yaw, Pitch)

  • Re-lighting (Light location, strength)

  • Aging

  • Eye Closing

  • Beard

  • Expressions

  • Makeup

  • ,and so on …

Face-Attribute Dataset Generation

The latent manipulation technique can be extended to generate a face-attributes dataset. For example, we can generate various face images with all wearing glasses and closing their eyes. Another example is results of generating male faces. (Red rectangles denote mis-generated images).

  • Steered attributes: wearing a glasses, closed eyes

  • Steered attributes: Male

Keywords:

Generative Adverserial Networks (GANs), Face Editing, Latent Manipulation, StyleGAN, StyleFlow, StyleCLIP, StyleGAN-NADA, CLIP, LAION, Dataset Generation

Updated:

You may also enjoy