Challenge
How can I easily transform my art creations into some specific drawing styles?
Design
An AI model that could transform and colorize the artwork empowered by Pix2Pix and CycleGAN
Duration
2 months
Tools
Kaggle, PyTorch, CycleGAN model, Pix2Pix model
My Contribution
Data collection, data cleaning, user research, model training, wireframing
Team
In collaboration with Guangya Zhu
Design Outcome
The objective is an AI model I built in PyTorch through collaboratin with Guangya. Users can upload and transform any picture they like from one style to another
Input a picture of one style, and change it to another with only 1 click away. Our AI model helps you with the work.
My Challenge in This Project
It’s very time-consuming to collect data, clean them and train the AI models before it came to a satisfying outcome. The project is one of the most tech-savvy ones I ever worked on. All the models you could see here went through over 80+ hours of training. Besides starting from a user’s point at the very start of “what they need”, I also need to start from another end of “what we could provide with current techniques” at the same time and try every effort to make ends meet.
User Needs
Luna, Drawing Skill 1/5
“I love Ukiyoe’s style of painting. I wonder how beautiful the world will look like in that style.”
Jake, Drawing Skill 3/5
“I always get stuck when I try to colorize the sketches.”
Ann, Drawing Skill 4/5
“When I want to change my style from one to another, I need some inspiration as the guidance.”
Gather Inspirations from Other Products
01 Venn Diagrams
a tool that turns users’ simple hand-drawings into matured, splended diagrams
02 Petalica Paint
a tool that colorizes the black-and-white sketches to colorized ones.
03 Justin Pinkney StyleGAN
Ukiyoe yourself and Toonify yourself
The Goals
For novice drawers, we hope to make drawing accessible to everyone.
For experienced drawers, we hope to inspire them on colorization and stylistic drawing.
The Key Steps
Step 01: Simple sketch to complicated sketch
Step 02: Complicated sketch to colorized sketch
Step 03: Colorized sketch to different styles
Technical Trials
We decided to focus on human profiles because through reading papers, and running models, we found that the human face database has the best details and info that could produce the best result in the model-training process.
01 Collect Data and Clean Data from Kaggle
1700 Anime + 1400 Ukiyoe + 1400 Metface + 2000 Colored/Uncolored Paired Anime + 1200 Iconic + 700 Simple Sketch + 1100 American Comics + 1200 Complicated Sketch + 900 Iconic Paintings
02 Different Trials of Transformation (200 epochs for each trial)
🌟Simple -> Complicated by Pix2Pix 😢Score 2/5
🌟Sketch -> Color empowered by Pix2Pix 😄Score 5/5
🌟Metface <-> Anime by CycleGAN 😄Score 5/5
Final outcome
Process
🌟Metface <-> Photos by CycleGAN 😢Score 2/5
🌟Metface <-> Ukiyoe by CycleGAN 😄Score 5/5
Final outcome
Process
🌟Anime <-> Comics by CycleGAN 😢Score 3/5
What Failed
A Drawing Tool Design(Working on it👷♀️👷)
Users can upload the pictures they like, then edit and transform the pictures to any style they love
Feedback/Reflections & Next Steps
There is no real-time interactive drawing for the current design. Drawers who tried our prototype mentioned they would love to draw and get drawing suggestions along the whole process.
Limited drawing styles for now. Drawers hope to turn the drawings into more styles, but currently, we only have 6 different styles to provide.
Gender inclusion. American comics have more male figures, while Anime has more female figures. Ukiyoe and Metface have an even number. This difference greatly influences the model training process and the outcome. When turning a drawing into the style of Anime, more female features are shown; when transforming drawings into American Comics, more male features show up.