Xuanke Shi bio photo

Email

Github

Google Scholar

ResearchGate

Lastest Update: 18th Jan 2024  中文版本 (Chinese Version)

Projects


3D Face Reconstruction

3D face reconstruction from a singe photo.


And I modeled my grandma’s face using this technique.


CurlingHunter

A multi-target multi-camera curling tracking system, termed CurlingHunter, is proposed, which can be applied in actual curling games in real time to assist athletes to compete, enhance the interest of the game, etc. Due to the regulations of curling game, no auxiliary equipment can be added to the curling stones, hence only non-contact measurement methods such as machine vision can be used in CurlingHunter. CurlingHunter has solved these problems:

  1. The problem of accurately capturing relatively small curling stones through long-sighted distance (> 20 m) in the super-large space environment with many occlusions;
  2. The problem of lens distortion correction in large scenes without interfering with the ice tracks;
  3. The problem of occlusions which would interfere with tracking and accuracy, while curling stone is easily blocked by athletes wiping ice, other peoples or objects during games;
  4. The problem of tracking and re-identifying multiple curling stones due to that all curling stones have identical appearance features;


As the first system to be applied to curling game, CurlingHunter demonstrated excellent performances in 2022 Beijing Winter Olympic Games of curling and 2022 Beijing Winter Paralympic Games of curling. Although we focus on curling, our system is readily transferable to other sports.

Face Stylization

This project converts the given image into a dozen of different styles while keeping the characteristics of the input person: Sketch, American Comic, Manga, Oil painting, CG style, Disney style, etc..


3D Face Cartoonization

This project converts the given image into a 3D cartoon avatar.


Avatar Head Pose Estimation

This project estimates the head pose of the person in the video and ensure the obtained head pose accurate and smooth.


Avatar Auto-Creation and Blendshape Generation

Avatar automatically pinches faces and automatically generates blendshape. Note that there is no requirement for the topology of the model.



Generated avatar is driven by video. Please note that the generated blendshape does not have any cross-mold phenomenon.


ToF Body Depth Completation

A real-time body depth completion for AR Effect


Audio Driven Digital Human

Given text or audio, we generate the visual-audio synchronized 2D avatar. All parts of the head look natural when speaking without any weird mouth, teeth and nose movements.

At the same time, as a core researcher, I independently completed (a) Outdoor digital humans (b) Video translation as HeyGen’s competitors (c) Dynamic local segment video fusion, etc. (d) Lightweight small model for mobile terminal


We used this technology to reproduce the talk show performance of Professor Tang, the founder of SenseTime, at the annual meeting. Video Link

Look forward, don't look back.


VIMI: Controllable character video generation

Given a reference image, action video and expression video, generate the corresponding video of the reference image.



3D Vison and Robotics

This project is about object 6-DOF estimation and robotics manipulation.The project has been going on for too long, so I just selected some remaining drawings.

  1. Something about point segmentation and object pose estimation.



  1. Something about robotics and manipulation.