Commit 7579972a authored by paul's avatar paul
Browse files

add info on the neural network

parent 98442e06
# Video Matting Example
Realtime background substraction using MobileNetV3. Model provided by https://github.com/PeterL1n/RobustVideoMatting
Realtime background substraction for video or webcam using MobileNetV3.
Model provided by https://github.com/PeterL1n/RobustVideoMatting
Shout out to [Natxopedreira 2021](https://github.com/natxopedreira) for providing the initial example.
Example made with love by Natxopedreira 2021
https://github.com/natxopedreira
![](../media/video_matting.gif)
### Notes
Please check out the [official documentation](https://github.com/PeterL1n/RobustVideoMatting/blob/master/documentation/inference.md) for this model.
Starting with the Turing architecture (e.g. RTX 2070), GPUs can utilize the FP16 data type and are therefore much faster!
##### Choosing the downsample factor
The table provides a general guideline. Please adjust based on your video content.
| Resolution | Portrait | Full-Body |
| ------------- | ------------- | -------------- |
| <= 512x512 | 1 | 1 |
| 1280x720 | 0.375 | 0.6 |
| 1920x1080 | 0.25 | 0.4 |
| 3840x2160 | 0.125 | 0.2 |
Internally, the model resizes down the input for stage 1. Then, it refines at high-resolution for stage 2.
Set `downsample_ratio` so that the downsampled resolution is between 256 and 512. For example, for `1920x1080` input with `downsample_ratio=0.25`, the resized resolution `480x270` is between 256 and 512.
![](../media/video_matting.gif)
### Other References
......
......@@ -43,14 +43,21 @@ void ofApp::setup() {
};
model.setup(inputNames, outputNames);
// parameters for the neural network
float downsampleRatio = 0.25f;
float videoWidth = video.getWidth();
float videoHeight = video.getHeight();
float batchSize = 1.0f;
float numChannels = 3.0f;
// model-specific inputs
inputs = {
cppflow::tensor({0.25f}), // downsample ratio
cppflow::tensor({downsampleRatio}),
cppflow::tensor({0.0f}), // r1i
cppflow::tensor({0.0f}), // r2i
cppflow::tensor({0.0f}), // r3i
cppflow::tensor({0.0f}), // r4i
cppflow::tensor({1.0f, nnHeight, nnWidth, 3.0f}) // src
cppflow::tensor({batchSize, videoHeight, videoWidth, numChannels})
};
imgMask.allocate(video.getWidth(), video.getHeight(), OF_IMAGE_GRAYSCALE);
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment