Commit 0743b7f7 authored by Dan Wilcox's avatar Dan Wilcox
Browse files

Merge branch 'main' of git.zkm.de:Hertz-Lab/Research/intelligent-museum/ofxTensorFlow2

parents 86a2c2ff 4e278e50
1.3.0: 2022 Feb 22
* added support for frozen graphs (TF < 1.15 model format)
* added basic frozen graph example
* added updated version of ofxMSATensorFlow example_char_rnn
* fixed ofxTF2::Model::runMultiModel() more less inputs than initially named
* update to latest cppflow with frozen graph support (Paul Bethge)
* improved example model download script to accept list of zip filenames
* added example model download script -l/--list flag to print known model zip
filenames
1.2.1: 2022 Feb 16
* fixed incorrect changelog version
......
......@@ -14,13 +14,13 @@ WARRANTIES, see the file, "LICENSE.txt," in this distribution.
Selected examples:
| Style Transfer | Keyword Spotting |
| :--: | :--:
| ![](media/style_transfer.gif) | ![](media/keyword_spotting.gif) |
| Style Transfer | Keyword Spotting | CharRnn (Frozen Graph) |
| :--: | :--: | :--: |
| ![](media/style_transfer.gif) | ![](media/keyword_spotting.gif) | ![](media/charRnn.gif) |
| Pose Estimation | Pix2Pix |
| :--: | :--: |
![](media/movenet.gif) | ![](media/pix2pix.gif) |
| Pose Estimation | Pix2Pix |
| :--: | :--: |
| ![](media/movenet.gif) | ![](media/pix2pix.gif) |
| Video Matting |
| :--: |
......@@ -243,7 +243,7 @@ The example projects are located in the `example_XXXX` directories.
### Downloading Pre-Trained Models
Each example contains code to create a neural network and export it in the [SavedModel format](https://www.tensorflow.org/guide/saved_model). Neural networks require training which may take hours or days in order to produce a satisfying output, therefore we provide pre-trained models which you can download as ZIP files, either from the release page on GitHub or from a public shared link here:
Each example contains code to create a neural network and export it in the [SavedModel format](https://www.tensorflow.org/guide/saved_model) or previous frozen GraphDef format. Neural networks require training which may take hours or days in order to produce a satisfying output, therefore we provide pre-trained models which you can download as ZIP files, either from the release page on GitHub or from a public shared link here:
<https://cloud.zkm.de/index.php/s/gfWEjyEr9X4gyY6>
......@@ -297,7 +297,7 @@ Simply select ofxTensorFlow2 from the available addons in the OF ProjectGenerato
#### Model Format
ofxTensorFlow2 works with the TensorFlow 2 [SavedModel format](https://www.tensorflow.org/guide/saved_model).
ofxTensorFlow2 works with the TensorFlow [SavedModel format](https://www.tensorflow.org/guide/saved_model) (preferred) and the older TensorFlow 1 frozen GraphDef format (legacy).
When referring to the "SavedModel" we mean the parent folder of the exported neural network containing two subfolder `assets` and `variables` and a `saved_model.pb` file. Do not change anything inside this folder, however renaming the folder is permitted. Keep in mind to use the correct file path within the application.
......
# Attempt to load a config.make file.
# If none is found, project defaults in config.project.make will be used.
ifneq ($(wildcard config.make),)
include config.make
endif
# make sure the the OF_ROOT location is defined
ifndef OF_ROOT
OF_ROOT=$(realpath ../../..)
endif
# call the project makefile!
include $(OF_ROOT)/libs/openFrameworksCompiled/project/makefileCommon/compile.project.mk
# ofxTensorFlow2
include $(OF_ROOT)/addons/ofxTensorFlow2/addon_targets.mk
# Basic example
This is a very basic example wich demonstrates how to load and infer a _Frozen Graph_ created using TensorFlow.
Frozen GraphDef is the standard of storing a graph for versions of TensorFlow < 1.15.
We wanted to support this feature for legacy reasons.
### TensorFlow2
We will create a dummy graph for demonstration purposes.
To create the model simply run the following command:
```
python main.py
```
### openFrameworks
By default `ofxTF2::Model` will use the `SavedModel` format. To use the frozen GraphDef format you may either add the type to the constructor or call `setModelType()` afterwards.
As the default names differ from the names in the `SavedModel` format make sure to overwrite names of the ins and outs by calling `setup()`.
```c++
// set model type and i/o names
model.setModelType(cppflow::model::TYPE::FROZEN_GRAPH);
model.setup({{"x:0"}}, {{"Identity:0"}});
```
Afterwards you can load the _pb file_ using the `load()` function.
```c++
// load the model, bail out on error
if(!model.load("model.pb")) {
std::exit(EXIT_FAILURE);
}
```
Everything else should work the same.
Please understand that we wont be able to invest a lot of time in supporting this feature in the future.
\ No newline at end of file
################################################################################
# CONFIGURE PROJECT MAKEFILE (optional)
# This file is where we make project specific configurations.
################################################################################
################################################################################
# OF ROOT
# The location of your root openFrameworks installation
# (default) OF_ROOT = ../../..
################################################################################
# OF_ROOT = ../../..
################################################################################
# PROJECT ROOT
# The location of the project - a starting place for searching for files
# (default) PROJECT_ROOT = . (this directory)
#
################################################################################
# PROJECT_ROOT = .
################################################################################
# PROJECT SPECIFIC CHECKS
# This is a project defined section to create internal makefile flags to
# conditionally enable or disable the addition of various features within
# this makefile. For instance, if you want to make changes based on whether
# GTK is installed, one might test that here and create a variable to check.
################################################################################
# None
################################################################################
# PROJECT EXTERNAL SOURCE PATHS
# These are fully qualified paths that are not within the PROJECT_ROOT folder.
# Like source folders in the PROJECT_ROOT, these paths are subject to
# exlclusion via the PROJECT_EXLCUSIONS list.
#
# (default) PROJECT_EXTERNAL_SOURCE_PATHS = (blank)
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_EXTERNAL_SOURCE_PATHS =
################################################################################
# PROJECT EXCLUSIONS
# These makefiles assume that all folders in your current project directory
# and any listed in the PROJECT_EXTERNAL_SOURCH_PATHS are are valid locations
# to look for source code. The any folders or files that match any of the
# items in the PROJECT_EXCLUSIONS list below will be ignored.
#
# Each item in the PROJECT_EXCLUSIONS list will be treated as a complete
# string unless teh user adds a wildcard (%) operator to match subdirectories.
# GNU make only allows one wildcard for matching. The second wildcard (%) is
# treated literally.
#
# (default) PROJECT_EXCLUSIONS = (blank)
#
# Will automatically exclude the following:
#
# $(PROJECT_ROOT)/bin%
# $(PROJECT_ROOT)/obj%
# $(PROJECT_ROOT)/%.xcodeproj
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_EXCLUSIONS =
################################################################################
# PROJECT LINKER FLAGS
# These flags will be sent to the linker when compiling the executable.
#
# (default) PROJECT_LDFLAGS = -Wl,-rpath=./libs
#
# Note: Leave a leading space when adding list items with the += operator
#
# Currently, shared libraries that are needed are copied to the
# $(PROJECT_ROOT)/bin/libs directory. The following LDFLAGS tell the linker to
# add a runtime path to search for those shared libraries, since they aren't
# incorporated directly into the final executable application binary.
################################################################################
# PROJECT_LDFLAGS=-Wl,-rpath=./libs
################################################################################
# PROJECT DEFINES
# Create a space-delimited list of DEFINES. The list will be converted into
# CFLAGS with the "-D" flag later in the makefile.
#
# (default) PROJECT_DEFINES = (blank)
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_DEFINES =
################################################################################
# PROJECT CFLAGS
# This is a list of fully qualified CFLAGS required when compiling for this
# project. These CFLAGS will be used IN ADDITION TO the PLATFORM_CFLAGS
# defined in your platform specific core configuration files. These flags are
# presented to the compiler BEFORE the PROJECT_OPTIMIZATION_CFLAGS below.
#
# (default) PROJECT_CFLAGS = (blank)
#
# Note: Before adding PROJECT_CFLAGS, note that the PLATFORM_CFLAGS defined in
# your platform specific configuration file will be applied by default and
# further flags here may not be needed.
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_CFLAGS =
################################################################################
# PROJECT OPTIMIZATION CFLAGS
# These are lists of CFLAGS that are target-specific. While any flags could
# be conditionally added, they are usually limited to optimization flags.
# These flags are added BEFORE the PROJECT_CFLAGS.
#
# PROJECT_OPTIMIZATION_CFLAGS_RELEASE flags are only applied to RELEASE targets.
#
# (default) PROJECT_OPTIMIZATION_CFLAGS_RELEASE = (blank)
#
# PROJECT_OPTIMIZATION_CFLAGS_DEBUG flags are only applied to DEBUG targets.
#
# (default) PROJECT_OPTIMIZATION_CFLAGS_DEBUG = (blank)
#
# Note: Before adding PROJECT_OPTIMIZATION_CFLAGS, please note that the
# PLATFORM_OPTIMIZATION_CFLAGS defined in your platform specific configuration
# file will be applied by default and further optimization flags here may not
# be needed.
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_OPTIMIZATION_CFLAGS_RELEASE =
# PROJECT_OPTIMIZATION_CFLAGS_DEBUG =
################################################################################
# PROJECT COMPILERS
# Custom compilers can be set for CC and CXX
# (default) PROJECT_CXX = (blank)
# (default) PROJECT_CC = (blank)
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_CXX =
# PROJECT_CC =
import tensorflow as tf
from tensorflow.python.framework.convert_to_constants import (
convert_variables_to_constants_v2,
)
input = tf.keras.Input(shape=(5,))
output = tf.keras.layers.Dense(5, activation=tf.nn.relu)(input)
output = tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)(output)
model = tf.keras.Model(inputs=input, outputs=output)
# Create frozen graph
x = tf.TensorSpec(model.input_shape, tf.float32, name="x")
concrete_function = tf.function(lambda x: model(x)).get_concrete_function(x)
frozen_model = convert_variables_to_constants_v2(concrete_function)
# Check input/output node name
print(f"{frozen_model.inputs}")
print(f"{frozen_model.outputs}")
# Save the graph as protobuf format
directory = "."
tf.io.write_graph(frozen_model.graph, directory, "../bin/data/model.pb", as_text=False)
\ No newline at end of file
tensorflow==1.14
\ No newline at end of file
#include "ofMain.h"
#include "ofApp.h"
//========================================================================
int main() {
ofSetupOpenGL(1024, 768, OF_WINDOW); // <-------- setup the GL context
// this kicks off the running of my app
// can be OF_WINDOW or OF_FULLSCREEN
// pass in width and height too:
ofRunApp(new ofApp());
}
/*
* ofxTensorFlow2
*
* Copyright (c) 2021 ZKM | Hertz-Lab
* Paul Bethge <bethge@zkm.de>
* Dan Wilcox <dan.wilcox@zkm.de>
*
* BSD Simplified License.
* For information on usage and redistribution, and for a DISCLAIMER OF ALL
* WARRANTIES, see the file, "LICENSE.txt," in this distribution.
*
* This code has been developed at ZKM | Hertz-Lab as part of „The Intelligent
* Museum“ generously funded by the German Federal Cultural Foundation.
*/
#include "ofApp.h"
//--------------------------------------------------------------
void ofApp::setup() {
ofSetFrameRate(60);
ofSetVerticalSync(true);
ofSetWindowTitle("example_basics_frozen_graph");
// create an input tensor of an arbitrary shape and fill it
auto input = cppflow::fill({10, 5}, 1.0f);
// set model type and i/o names
model.setModelType(cppflow::model::TYPE::FROZEN_GRAPH);
model.setup({{"x:0"}}, {{"Identity:0"}});
// load the model, bail out on error
if(!model.load("model.pb")) {
std::exit(EXIT_FAILURE);
}
// inference
auto output = model.runModel(input);
// use the ofxTF2 namespace for some useful functions like conversion
ofxTF2::tensorToVector(output, outputVector);
ofxTF2::tensorToVector(input, inputVector);
// print summary to console
ofLog() << "Flattened Input:";
ofLog() << ofxTF2::vectorToString(inputVector);
ofLog() << "Flattened Output:";
ofLog() << ofxTF2::vectorToString(outputVector);
// load a font for displaying strings
font.load(OF_TTF_SANS, 14);
}
//--------------------------------------------------------------
void ofApp::update() {
}
//--------------------------------------------------------------
void ofApp::draw() {
// draw summary to screen
font.drawString("Flattened Input:", 20, 20);
font.drawString(ofxTF2::vectorToString(inputVector), 40, 40);
font.drawString("Flattened Output:", 20, 60);
font.drawString(ofxTF2::vectorToString(outputVector), 40, 80);
}
//--------------------------------------------------------------
void ofApp::keyPressed(int key) {
}
//--------------------------------------------------------------
void ofApp::keyReleased(int key) {
}
//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y) {
}
//--------------------------------------------------------------
void ofApp::mouseDragged(int x, int y, int button) {
}
//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button) {
}
//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button) {
}
//--------------------------------------------------------------
void ofApp::mouseEntered(int x, int y) {
}
//--------------------------------------------------------------
void ofApp::mouseExited(int x, int y) {
}
//--------------------------------------------------------------
void ofApp::windowResized(int w, int h) {
}
//--------------------------------------------------------------
void ofApp::gotMessage(ofMessage msg) {
}
//--------------------------------------------------------------
void ofApp::dragEvent(ofDragInfo dragInfo) {
}
/*
* ofxTensorFlow2
*
* Copyright (c) 2021 ZKM | Hertz-Lab
* Paul Bethge <bethge@zkm.de>
* Dan Wilcox <dan.wilcox@zkm.de>
*
* BSD Simplified License.
* For information on usage and redistribution, and for a DISCLAIMER OF ALL
* WARRANTIES, see the file, "LICENSE.txt," in this distribution.
*
* This code has been developed at ZKM | Hertz-Lab as part of „The Intelligent
* Museum“ generously funded by the German Federal Cultural Foundation.
*/
#pragma once
#include "ofMain.h"
#include "ofxTensorFlow2.h"
class ofApp : public ofBaseApp {
public:
void setup();
void update();
void draw();
void keyPressed(int key);
void keyReleased(int key);
void mouseMoved(int x, int y);
void mouseDragged(int x, int y, int button);
void mousePressed(int x, int y, int button);
void mouseReleased(int x, int y, int button);
void mouseEntered(int x, int y);
void mouseExited(int x, int y);
void windowResized(int w, int h);
void dragEvent(ofDragInfo dragInfo);
void gotMessage(ofMessage msg);
ofxTF2::Model model;
std::vector<float> inputVector;
std::vector<float> outputVector;
ofTrueTypeFont font;
};
# Attempt to load a config.make file.
# If none is found, project defaults in config.project.make will be used.
ifneq ($(wildcard config.make),)
include config.make
endif
# make sure the the OF_ROOT location is defined
ifndef OF_ROOT
OF_ROOT=$(realpath ../../..)
endif
# call the project makefile!
include $(OF_ROOT)/libs/openFrameworksCompiled/project/makefileCommon/compile.project.mk
# ofxTensorFlow2
include $(OF_ROOT)/addons/ofxTensorFlow2/addon_targets.mk
# Frozen Graph Legacy Support - CharRnn
This example demonstrates how to load and run a frozen graph developed in TensorFlow. More specific, we demonstrate how to port an example of _ofxMSATensorFlow_ to _ofxTensorFlow2_.
![](../media/charRnn.gif)
### TensorFlow
From Memo Akten's [example_char_rnn](https://github.com/memo/ofxMSATensorFlow/tree/master/example-char-rnn):
> Models are trained and saved in python with this code (https://github.com/memo/char-rnn-tensorflow)
and loaded in openframeworks for prediction.
I'm supplying a bunch of pretrained models (bible, cooking, erotic, linux, love songs, shakespeare, trump),
and while the text is being generated character by character (at 60fps!) you can switch models in realtime mid-sentence or mid-word.
(Drop more trained models into the folder and they'll be loaded too).
Note, all models are trained really quickly with no hyperparameter search or cross validation,
using default architecture of 2 layer LSTM of size 128 with no dropout or any other regularisation.
### openFrameworks
By default `ofxTF2::Model` will use the `SavedModel` format. To use the `FrozenGraph` format you may either add the type to the constructor or call `Model::setModelType()` afterwards.
As the default names differ from the names in the `SavedModel` format make sure to overwrite names of the ins and outs by calling `Model::setup()`.
```c++
// set model type and i/o names
model.setModelType(cppflow::model::TYPE::FROZEN_GRAPH);
std::vector<std::string> inputNames = {
"data_in",
"state_in",
};
std::vector<std::string> outputNames = {
"data_out",
"state_out",
};
model.setup(inputNames, outputNames);
```
Afterwards you can load the _pb file_ using the `Model::load()` function.
```c++
if(!model.load(model_path)) {
std::exit(EXIT_FAILURE);
}
```
This approach is different to _ofxMSATensorFlow_ where you would name the input and output tensors when running the model.
Lets have a look at the way we ran the model in _ofxMSATensorFlow_:
```c++
// run the model (ofxMSATensorFlow)
std::vector<tensorflow::Tensor> t_out;
std::vector<std::string> fetch_tensors = { "data_out", "state_out" };
session->Run({ { "data_in", t_data_in }, { "state_in", t_state_in } }, fetch_tensors, {}, &t_out);
```
In _ofxTensorFlow2_ we can use the `Model::run()` function (if we have a single input and output tensor) or the more general `Model::runMultiModel()` function to execute the model on some input data:
```c++
// run the model (ofxTensorFlow2)
std::vector<cppflow::tensor> vectorOfInputTensors = {t_data_in, t_state_in};
auto vectorOfOutputTensors = model.runMultiModel(vectorOfInputTensors);
```
The class for the tensor also differs a little bit. In _ofxTensorFlow2_ we use the `cppflow::tensor` class instead of `tensorflow::tensor`.
Similar to _ofxMSATensorFlow_, we provide functions for converting from a tensor to `ofImage, ofPixels` and `std::vector`, and the other way around, e.g. `ofxTF2::vectorToTensor()`. Regarding other utilities, there are now some new handy functionalities like `ofxTF2::setGPUMaxMemory()` and `ofxTF2::ThreadedModel`, while other less general functions have been dropped.
For this example we have put dropped functions into the source code of the example.
__Note:__ With the current version of _cppflow_, calling `tensor::get_data()` or `tensor::shape()` on a default (uninitialized) `cppflow::tensor` will result in a segfault. Wile you usually don't need to call them yourself, this effects the previously mentioned conversion functions.
Therefore, we've changed a few lines of code when priming the model with new data.
Please understand that we wont be able to invest a lot of time in supporting this feature in the future.
\ No newline at end of file
################################################################################
# CONFIGURE PROJECT MAKEFILE (optional)
# This file is where we make project specific configurations.
################################################################################
################################################################################
# OF ROOT
# The location of your root openFrameworks installation
# (default) OF_ROOT = ../../..
################################################################################
# OF_ROOT = ../../..
################################################################################
# PROJECT ROOT
# The location of the project - a starting place for searching for files
# (default) PROJECT_ROOT = . (this directory)
#
################################################################################
# PROJECT_ROOT = .
################################################################################
# PROJECT SPECIFIC CHECKS
# This is a project defined section to create internal makefile flags to
# conditionally enable or disable the addition of various features within
# this makefile. For instance, if you want to make changes based on whether
# GTK is installed, one might test that here and create a variable to check.
################################################################################
# None
################################################################################
# PROJECT EXTERNAL SOURCE PATHS
# These are fully qualified paths that are not within the PROJECT_ROOT folder.
# Like source folders in the PROJECT_ROOT, these paths are subject to
# exlclusion via the PROJECT_EXLCUSIONS list.
#
# (default) PROJECT_EXTERNAL_SOURCE_PATHS = (blank)
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_EXTERNAL_SOURCE_PATHS =
################################################################################
# PROJECT EXCLUSIONS
# These makefiles assume that all folders in your current project directory
# and any listed in the PROJECT_EXTERNAL_SOURCH_PATHS are are valid locations
# to look for source code. The any folders or files that match any of the
# items in the PROJECT_EXCLUSIONS list below will be ignored.
#
# Each item in the PROJECT_EXCLUSIONS list will be treated as a complete
# string unless teh user adds a wildcard (%) operator to match subdirectories.
# GNU make only allows one wildcard for matching. The second wildcard (%) is
# treated literally.
#
# (default) PROJECT_EXCLUSIONS = (blank)
#
# Will automatically exclude the following:
#
# $(PROJECT_ROOT)/bin%
# $(PROJECT_ROOT)/obj%
# $(PROJECT_ROOT)/%.xcodeproj
#
# Note: Leave a leading space when adding list items with the += operator
################################################################################
# PROJECT_EXCLUSIONS =
################################################################################
# PROJECT LINKER FLAGS
# These flags will be sent to the linker when compiling the executable.
#
# (default) PROJECT_LDFLAGS = -Wl,-rpath=./libs