Makeup apps and hair color apps can be created on iOS. Machine learning separates facial parts.

How to get facial parts using machine learning

Image from face-parsing

We want to automatically separate only the hair and only the eyes

If we can finely separate the targeted parts such as hair and lips, we can change the color and make a makeup app.
It is troublesome to have the user trace the outline and cut it.
We want to do it automatically.
Moreover, We want to carefully cut the hair as finely as possible.
However, is it possible to cut it into elaborate pieces automatically?

There aren’t many frameworks that can take the parts you’re aiming for.

There is a field called semantic segmentation in machine learning.
It is a technology that can finely separate images for each object.
Due to technological improvements, it has become possible to cut into fairly fine pieces.

iOS also has a framework that can use segmentation.
However, although you can get Portrait Matte at AVFoundation, you can get the whole body, skin and teeth, but you cannot separate only the hair and eyes.
Moreover, it is necessary to set it when shooting with the camera.
Vision Person Segmentation also cutout whole body.

Machine learning solves it in no time

Now, what is useful there is a semantic segmentation model called face-parsing, which learns masks for each part of the face.

Original project:

This finely separates the hair, left and right eyes, lips and hat, and the accuracy is quite good.

If we use this on iOS, you can make a make-up app.

So I prepared a model converted for iOS. The accuracy does not change either.

Run on iOS

Specific procedure

The converted face-parsing model for iOS can be downloaded from Core ML-Models .

After that, you can execute the model by requesting with Vision.

The result is a number of pixels that correspond to the part.
There are 512 * 512 pixels, for example, 1 is the skin and 2 is the left eyebrow.

This array of numbers is converted into a black-and-white image and used.

Click here for sample code from model execution to image conversion and makeup.

Hair color and make-up can be done by synthesizing the acquired mask image.

hair color and lipstick color

🐣

I’m a freelance engineer.
Work consultation
Please feel free to contact us with a brief development description.
rockyshikoku@gmail.com

I am making an app that uses Core ML and ARKit.
We send machine learning / AR related information.

GitHub

Twitter
Medium

--

--

--

Freelance iOS developer. You can ask me for a job from any country. rockyshikoku@gmail.com https://github.com/john-rocky https://twitter.com/JackdeS11

Love podcasts or audiobooks? Learn on the go with our new app.

How to apply a filter to an existing video

Swift ViewController Lifecycle

YouTube ends picture-in-picture iOS test for Premium subscribers : Gadget Game News

How do use foregroundStyle to apply gradients to SwiftUI views?

conic.gradient.intro.png

How to Fix Black Screen Camera iphone 7: 6 Technical Steps

SwiftUI: how to send local notifications

Working with the didSet Property Observer in Swift

Swift High Order Functions

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
MLBoy

MLBoy

Freelance iOS developer. You can ask me for a job from any country. rockyshikoku@gmail.com https://github.com/john-rocky https://twitter.com/JackdeS11

More from Medium

How to create a model based on Emotion Recognition using the LiDAR sensor behind FaceID

Classifying Sounds with the SoundAnalysis framework and custom CoreML models in Swift

Visual Debugging of GameplayKit GKComponent

Visual Debugging of GameplayKit GKComponent

Thirty Days of Metal — Day 21: Point Lights