PoseCrafts: Transforming Text into 2D Characters Animation with Openpose Generation🏃

What is PoseCrafts?

PoseCrafts is a motion model that generate 2D motion in openpose format from textual description for using it to generate 2d animation spritesheets for 2d characters in StableDiffusion with Controlnet

Demo ▶️

Text Input: "a girl walking forward" Pose generated from PoseCrafts Result generated from stable diffusion

Text Input: "a girl running forward" Pose generated from PoseCrafts Result generated from stable diffusion

Text Input: "a girl hurting and stagger" Pose generated from PoseCrafts Result generated from stable diffusion

Trying Demo: PoseCrafts Demo Web

!!! Note📢 !!!

The demo only generate openpose sequence frames if you want 2d pixel art generated like gif image above please use StableDiffusion to generate

Datasets 📦

Create my own dataset by using Pose Estimation Controlnet to estimate spritesheets image collected from Sprite Resources and editing its result from Pose Estimation to correcting shape and label text description in file name. I collected data around 2 weeks and I got 400 samples then I collect more data to get more 200 samples without sleeping😪. You can see the datasets here: Openpose Motion Dataset

Process 🧪

I use Sentence Transformer for text encoding then I used encoded text to train my model. I've experimented on many models such as Simple Dense, CNN, RNN and LSTM and from the Evaluation using MAE LSTM perform best. For full process of this AI please read full medium blog below

• Sentence Transformer sentence-transformers all-MiniLM-L6-v2
• Full medium blog Medium Blog

Deployment🚀

• Backend
  • Deploy API using Huggingface Spaces🤗 and Docker🐋.
  • Github Repo: PoseCrafts-API
• Frontend
  • Fork from https://github.com/huchenlei/sd-webui-openpose-editor to create demo client that can generate poses.
  • Github Repo: posecrafts-sd-webui-openpose-editor