You can do Hand Pose Classification by machine learning on Mac.
This time, as a demo, I will create a pose classification model with CreateML and create a smartphone app that recognizes “finger heart” and “peace sign”.
This demo’s GitHub repository (Xcode project): HandPoseClassificationAR
1. Collect data
Image data is required for learning.
Collect and divide into folders.
Required images:
-Images of poses you want to identify (“fingerHeart” and “peace” this time)
-Images of
poses you don’t need to identify (“background”)
are divided into folders for each class.
The “background” contains various poses that you don’t need to identify and images of half-hearted hands as you move to the poses you want to classify.
Prepare images of different skin colors, ages, genders, and lighting conditions.
This time around 200 images were prepared for each class.
2. Train your model with CreateML
Open CreateML.
(Control-click Xcode to open it from OpenDeveloperTools.)
Select HandPose Classification.
(HandPoseClassification can be used in macOS Monterey / Xcode13 or higher environment.)
Select a dataset (Training Data).
If you do not specify the validation data, it will be generated automatically.
Set the dataset expansion (Augmentations: inflating the image by rotating it) as appropriate. This time, it doesn’t matter whether the hand is on the left or right, so I added horizontal inversion and rotation.
Press Train to start learning.
Learning will be completed in a few minutes.
After training, you can test with your own images from the Preview tab. You can also live preview with a mac camera.
Get the model from the Output tab.
3. Use the model in the app
Drop the model into your Xcode project to initialize it.
The Vision framework detects the points of the hand (positions of key points such as fingertips and joints: the positions of the points in the CreateML preview above) and then inputs them to the model.
The acquired hand point data is converted into a MultiArray (multidimensional array) and input / inferred to the model with CoreML.
label: fingerHeart
confidence: 0.999993645120239
4. Add AR according to the pose of the hand
Switch the process according to the acquired classification label.
To make the effect appear at the finger position, map the finger position acquired by Vision 20 cm behind the camera.
Use SceneKit to animate 3D objects.
This demo’s GitHub repository (Xcode project): HandPoseClassificationAR
About getting hand points:
Hand Pose Detection with Vision Hand tracking: MLBoy Daisuke
🐣
I’m a freelance engineer.
Feel free to contact me for work consultation 🐥
rockyshikoku@gmail.com
I am making an app that uses Core ML and ARKit.
We send machine learning / AR related information.
Thank you very much.
