Mask-RCNN h5 weights to frozen graph
The mystery of life isn’t a problem to solve, but a reality to experience ~Frank Herbert, Dune
The article aims at converting the Mask-RCNN weights (.h5) into frozen graph format (.pb). Unlike normal methods used to convert keras weights into the frozen graph format, the method for Mask-RCNN weights is different. By Mask-RCNN weights, I refer to the custom models trained using Matterport’s implementation. I found plethora of experiments in the same direction but none worked for me. I share my solution for ResNet101 here (for ResNet50 refer to this forum thread). I will demonstrate the method to obtain the frozen graph of the sample h5 weights provided by NVIDIA; then repeating the method for our custom model.
For the Google Colab version of the same, head towards the bottom.
Methodology
We leverage TensorRT tools/scripts even though the purpose of these tools isn’t just obtaining the frozen graph but optimising the Mask-RCNN models using Uff parsing. But the method to do so includes the following conversion:
h5 weights -> frozen graph (.pb) -> Uff model
We use the first half of the process without going for the other half (only needed if we need faster inferences of our model on NVIDIA devices… more articles on that).
Install Pre-requisites
- We require an NVIDIA-GPU-enabled Ubuntu 18.04 host system.
- TensorFlow 1.15 (TensorFlow-gpu), keras 2.1.3, h5py 2.10.0
pip uninstall tensorflow-gpu
pip uninstall keras-nightly
pip install --upgrade tensorflow-gpu==1.15
pip install keras==2.1.3 #2.1.6
pip install h5py==2.10.0Check version using a Python3 console
import tensorflow as tf
tf.__version__- CUDA 10.0 + CUDNN 7.6.2
Check current CUDA version using nvidia-smi. If the current version is other than 10.0, uninstall it and install desired version.
# Uninstall
sudo apt-get --purge remove cuda nvidia* libnvidia-*
sudo dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge
sudo apt-get remove cuda-*
sudo apt autoremove
# Installing CUDA 10.0 (be ready for prompts...)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo apt-get update
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
sudo apt install -y ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
sudo apt-get update
# Install NVIDIA driver
sudo apt-get -y install nvidia-driver-418
# Install development and runtime libraries (~4GB)
sudo apt-get install -y \
cuda-10-0 \
libcudnn7=7.6.2.24-1+cuda10.0 \
libcudnn7-dev=7.6.2.24-1+cuda10.0 --allow-change-held-packagesCheck installed CUDNN version using
# Find CUDNN version.
cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2An output like
Find the version under
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 2means version is 7.6.2.
- TensorRT 7.0.0.11 Download the deb package from here: https://developer.nvidia.com/nvidia-tensorrt-7x-download. Select
TensorRT 7.0.0.11 for Ubuntu 1804 and CUDA 10.0 DEB local repo package.
# Install the deb file
sudo dpkg -i "/PATH/TO/DEB/FILE/nv-tensorrt-repo-ubuntu1804-cuda10.0-trt7.0.0.11-ga-20191216_1-1_amd64.deb"
sudo apt-key add /var/nv-tensorrt-repo-cuda10.0-trt7.0.0.11-ga-20191216/7fa2af80.pub
sudo apt-get updateBy default, the system will try to upgrade the libnvinfer versions to the most latest ones (including upgrading CUDA to version 11.0). So we first install the necessary versions and then place a hold on them to restrict any automatic upgrade attempts that may cause unmet dependencies.
sudo apt-get install libnvinfer7=7.0.0-1+cuda10.0 libnvonnxparsers7=7.0.0-1+cuda10.0 libnvparsers7=7.0.0-1+cuda10.0 libnvinfer-plugin7=7.0.0-1+cuda10.0 libnvinfer-dev=7.0.0-1+cuda10.0 libnvonnxparsers-dev=7.0.0-1+cuda10.0 libnvparsers-dev=7.0.0-1+cuda10.0 libnvinfer-plugin-dev=7.0.0-1+cuda10.0 python-libnvinfer=7.0.0-1+cuda10.0 python3-libnvinfer=7.0.0-1+cuda10.0
sudo apt-mark hold libnvinfer7 libnvonnxparsers7 libnvparsers7 libnvinfer-plugin7 libnvinfer-dev libnvonnxparsers-dev libnvparsers-dev libnvinfer-plugin-dev python-libnvinfer python3-libnvinfer
sudo apt-get install tensorrt 7.0Ensure the following packages were installed using dpkg -l | grep TensorRT:
ii libnvinfer-bin 7.0.0-1+cuda10.0 amd64 TensorRT binaries
hi libnvinfer-dev 7.0.0-1+cuda10.0 amd64 TensorRT development libraries and headers
ii libnvinfer-doc 7.0.0-1+cuda10.0 all TensorRT documentation
hi libnvinfer-plugin-dev 7.0.0-1+cuda10.0 amd64 TensorRT plugin libraries
hi libnvinfer-plugin7 7.0.0-1+cuda10.0 amd64 TensorRT plugin libraries
ii libnvinfer-samples 7.0.0-1+cuda10.0 all TensorRT samples
hi libnvinfer7 7.0.0-1+cuda10.0 amd64 TensorRT runtime libraries
hi libnvonnxparsers-dev 7.0.0-1+cuda10.0 amd64 TensorRT ONNX libraries
hi libnvonnxparsers7 7.0.0-1+cuda10.0 amd64 TensorRT ONNX libraries
hi libnvparsers-dev 7.0.0-1+cuda10.0 amd64 TensorRT parsers libraries
hi libnvparsers7 7.0.0-1+cuda10.0 amd64 TensorRT parsers libraries
hi python-libnvinfer 7.0.0-1+cuda10.0 amd64 Python bindings for TensorRT
hi python3-libnvinfer 7.0.0-1+cuda10.0 amd64 Python 3 bindings for TensorRT
ii tensorrt 7.0.0.11-1+cuda10.0 amd64 Meta package of TensorRT- TensorRT OSS (Open Source Software)
git clone -b master https://github.com/nvidia/TensorRT TensorRT
cd TensorRT
git submodule update --init --recursive
git checkout 4c99d07 # point HEAD to 7.0- uff & graphsurgeon utilities
cd TensorRT/samples/opensource/sampleUffMaskRCNN/converted/
pip3 install -r requirements.txt
pip3 install --no-cache-dir --extra-index-url https://pypi.ngc.nvidia.com uff
pip3 install --no-cache-dir --extra-index-url https://pypi.ngc.nvidia.com graphsurgeonMake changes for ResNet101
- Modify the conv2d_transpose conversion function in UFF library. Find the location of the library using a python3 console using
import uff
uff.__file__ #/usr/local/lib/python3.7/dist-packages/uff/__init__.pyChange the part around line 570 to
uff_graph.conv_transpose(
inputs[0], inputs[2], inputs[1],
strides, padding,
dilation=None, number_groups=number_groups,
left_format=lhs_fmt, right_format=rhs_fmt,
name=name, fields=fields
)Download the Mask R-CNN repo
git clone https://github.com/matterport/Mask_RCNN.git
export PYTHONPATH=$PYTHONPATH:$PWD/Mask_RCNNApply the patch into Mask R-CNN repo to update the model from NHWC to NCHW. Notice that the patch already exists in the previous directory.
cd Mask_RCNN
git checkout 3deaec5
patch -p1 < ../0001-Update-the-Mask_RCNN-model-from-NHWC-to-NCHW.patch
cd -Download the pre-trained Keras model
wget https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5Convert .h5 to .pb and .uff
Make the following changes to TensorRT/samples/opensource/sampleUffMaskRCNN/converted/mrcnn_to_trt_single.py
index c9533e6..8901cac 100644
--- a/mrcnn_to_trt_single.py
+++ b/modified.py
@@ -88,7 +88,7 @@ class CocoConfig(Config):
# GPU_COUNT = 8 # Number of classes (including background)
- NUM_CLASSES = 1 + 80 # COCO has 80 classes
+ NUM_CLASSES = 1 + 3 # COCO has 80 classes # only change if your custom model has classes not equal to 81 class InferenceConfig(CocoConfig):
# Set batch size to 1 since we'll be running inference on
@@ -123,7 +123,7 @@ def main(args=None):
text=True, list_nodes=list_nodes)
-def convert_model(inference_model, output_path, output_nodes=[], preprocessor=None, text=False,
+def convert_model(inference_model, output_path, output_nodes=[], preprocessor=None, text=True,
list_nodes=False):
# convert the keras model to pb
orig_output_node_names = [node.op.name for node in inference_model.outputs]
@@ -157,7 +157,7 @@ def convert_model(inference_model, output_path, output_nodes=[], preprocessor=No
debug_mode = False
)- os.remove(temp_pb_path)
+ #os.remove(temp_pb_path)
Now convert
cd TensorRT/samples/opensource/sampleUffMaskRCNN/converted
python3 mrcnn_to_trt_single.py -w 'mask_rcnn_coco.h5' -o mrcnn_nchw.uff -p ./config.pyAn output like the following indicates success:
DEBUG [/usr/local/lib/python3.7/dist-packages/uff/converters/tensorflow/converter.py:143] Marking ['mrcnn_detection', 'mrcnn_mask/Sigmoid'] as outputs
No. nodes: 3049
UFF Output written to mrcnn_nchw.uff
UFF Text Output written to mrcnn_nchw.pbtxtFor custom model
The only changes are the change in model parameters that are used to train our custom model. For e.g., my custom model involved 4 classes instead of 81 classes (the default model). This is what I change:
- In the file
TensorRT/samples/opensource/sampleUffMaskRCNN/converted/test_mrcnn_to_trt_single.py, change the number of classes used to train the model. In my case, it was 4. So make the Corresponding modification:
# Number of classes (including background)
NUM_CLASSES = 1 + 3 # COCO has 80 classes- Make changes to
TensorRT/samples/opensource/sampleUffMaskRCNN/converted/config.py
mrcnn_detection = gs.create_plugin_node("mrcnn_detection", op="DetectionLayer_TRT", num_classes=4, keep_topk=100, score_threshold=0.7, iou_threshold=0.3)- Also, make changes to
TensorRT/samples/opensource/sampleUffMaskRCNN/converted/mrcnn/config.py
NUM_CLASSES = 4 # Override in sub-classesThe entire method was implemented in a Google Colab notebook that can be found here: trt2uff.ipynb.
If I could help you, you can also send me some crypto like Solana, Ethereum, Doge or coins on BSC. 🤠
SOL wallet: 2e233GG81VBZFeiSj65aYf1Z6uMmJUb13Y85Geht8scy
BSC wallet: 0x47acC4B652166AE55fb462e4cD7DD26eFa97Da04
ETH wallet: 0x47acC4B652166AE55fb462e4cD7DD26eFa97Da04
Doge wallet: DFT6Pydzw7RAYG1ww4RRfctBKqyiXbbBwk
