profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/markweberdev/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

google-research/deeplab2 485

DeepLab2 is a TensorFlow library for deep labeling, aiming to provide a unified and state-of-the-art TensorFlow codebase for dense pixel labeling tasks.

markweberdev/pytorch-retinanet 1

Pytorch implementation of RetinaNet object detection.

aljosaosep/bmtt2021 0

BMTT6 workshop webpage

markweberdev/4D-PLS 0

4D Panoptic Lidar Segmentation

markweberdev/deeplab2 0

DeepLab2 is a TensorFlow library for deep labeling, aiming to provide a unified and state-of-the-art TensorFlow codebase for dense pixel labeling tasks.

markweberdev/pytorch-YOLOv4 0

Minimal PyTorch implementation of YOLOv4

markweberdev/research-group-web 0

a template for research group sites

push eventaljosaosep/bmtt2021

Mark Weber

commit sha e77344a3b8dd685d91a1b7da840ab69ae4123ff7

Add winning reports.

view details

push time in 5 days

issue commentgoogle-research/deeplab2

What is your environment for testing your model?

@lxtGH Thanks a lot for pointing this out. You are right, it's 11095. I will correct it in the paper!

lxtGH

comment created time in 6 days

push eventaljosaosep/bmtt2021

Mark Weber

commit sha ad08f0fe8dc51ce2e9f18da5d3452cc4d253ee96

Update schedule

view details

push time in 11 days

issue commentgoogle-research/deeplab2

What is your environment for testing your model?

Please find the class wise scores attached. Please note, that the results are obtained with a ResNet50 os32 backbone. classwise_scores_kitti_step.csv

lxtGH

comment created time in 13 days

issue commentgoogle-research/deeplab2

STEP dataset questions

Hi @HarborYuan,

This is quite strange behaviour, If you have time I would highly appreciate it, if you can try the following:

  1. Use CUDA 11.2.2 and cudnn v8.2.1.32.
  2. Clone this repository.
  3. For protobuf, orbit and cocoapi, please follow the install guide: here For tensorflow models/orbit, my commit is 5e81b5288b9fc8d7d07d1d8db5161ae09c3b110c
  4. Instead of installing the libraries from the install guide, try to make an environment with this file. snapshot.txt It is not as clean as a minimal install, but the one I am using (only valid for Panoptic-DeepLab/Motion-DeepLab)
  5. Compile protobufs. 4.5 Compile the GPU op (otherwise you can't run all the tests)
  6. Run tests.
  7. Prepare Cityscapes and KITTI datasets.
  8. Download the pertained models.
  9. Modify the config files as needed.
  10. Run evaluation for both Cityscapes and KITTI. 10 Could you attach one of the visualised outputs so we can see whether network predictions or evaluation is causing trouble?

Alternatively, do you happen to have access to an NVIDIA TITAN X (12GB) and could try it there?

Thanks a lot for your help and patience in investigating this issue.

lxtGH

comment created time in 14 days

issue commentgoogle-research/deeplab2

What is your environment for testing your model?

That's unfortunate. I can eval that for you, could you please specify whether you would like to have per class PQ scores from Panoptic-DeepLab or Motion-DeepLab on KITTI-STEP?

lxtGH

comment created time in 14 days

issue commentgoogle-research/deeplab2

STEP dataset questions

Hi @HarborYuan,

You seem to encounter similar issues than others, however at this stage I am unable to reproduce these issues.

  • Could you please specify which TF version, which CUDA and which CuDNN version you are using?
  • Can you execute all the tests in the repository?
  • What is the command you are using to evaluate?

For me, the output looks like this: https://github.com/google-research/deeplab2/issues/64#issuecomment-938432325

lxtGH

comment created time in 15 days

issue commentgoogle-research/deeplab2

What is your environment for testing your model?

Dear @lxtGH,

My setup is CUDA 11.2.2 with tf 2.5.0. I don't need to do any changes. I get the same layout error, but everything still runs fine.

The following example is the Motion-DeepLab trained on KITTI checkpoint that we provide evaluated on KITTI:

python trainer/train.py --config_file="./configs/kitti/motion_deeplab/resnet50_os32.textproto" --num_gpus=1 --mode=eval --model_dir="/**retracted**/models/deeplab2/kitti/"

I don't have access to a RTX-3090, so I am unable to verify that It runs there too.

2021-10-08 09:12:02.213287: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 I1008 09:12:05.060556 139644910475072 train.py:65] Reading the config file. I1008 09:12:05.065931 139644910475072 train.py:69] Starting the experiment. 2021-10-08 09:12:05.068325: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2021-10-08 09:12:05.160309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:83:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2021-10-08 09:12:05.160363: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-10-08 09:12:05.346183: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2021-10-08 09:12:05.346345: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2021-10-08 09:12:05.393284: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10 2021-10-08 09:12:05.518453: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10 2021-10-08 09:12:05.607084: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11 2021-10-08 09:12:05.683005: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11 2021-10-08 09:12:05.686791: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-10-08 09:12:05.693328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2021-10-08 09:12:05.694196: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-10-08 09:12:05.699542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:83:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2021-10-08 09:12:05.706127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2021-10-08 09:12:05.706191: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-10-08 09:12:06.506011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-08 09:12:06.506057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 2021-10-08 09:12:06.506064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N 2021-10-08 09:12:06.513654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11436 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:83:00.0, compute capability: 6.1) I1008 09:12:06.516476 139644910475072 train_lib.py:105] Using strategy <class 'tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy'> with 1 replicas I1008 09:12:06.715466 139644910475072 motion_deeplab.py:53] Synchronized Batchnorm is used. I1008 09:12:06.718576 139644910475072 axial_resnet_instances.py:144] Axial-ResNet final config: {'num_blocks': [3, 4, 6, 3], 'backbone_layer_multiplier': 1.0, 'width_multiplier': 1.0, 'stem_width_multiplier': 1.0, 'output_stride': 32, 'classification_mode': True, 'backbone_type': 'resnet', 'use_axial_beyond_stride': 0, 'backbone_use_transformer_beyond_stride': 0, 'extra_decoder_use_transformer_beyond_stride': 32, 'backbone_decoder_num_stacks': 0, 'backbone_decoder_blocks_per_stage': 1, 'extra_decoder_num_stacks': 0, 'extra_decoder_blocks_per_stage': 1, 'max_num_mask_slots': 128, 'num_mask_slots': 128, 'memory_channels': 256, 'base_transformer_expansion': 1.0, 'global_feed_forward_network_channels': 256, 'high_resolution_output_stride': 4, 'activation': 'relu', 'block_group_config': {'attention_bottleneck_expansion': 2, 'drop_path_keep_prob': 1.0, 'drop_path_beyond_stride': 16, 'drop_path_schedule': 'constant', 'positional_encoding_type': None, 'use_global_beyond_stride': 0, 'use_sac_beyond_stride': -1, 'use_squeeze_and_excite': False, 'conv_use_recompute_grad': False, 'axial_use_recompute_grad': True, 'recompute_within_stride': 0, 'transformer_use_recompute_grad': False, 'axial_layer_config': {'query_shape': (129, 129), 'key_expansion': 1, 'value_expansion': 2, 'memory_flange': (32, 32), 'double_global_attention': False, 'num_heads': 8, 'use_query_rpe_similarity': True, 'use_key_rpe_similarity': True, 'use_content_similarity': True, 'retrieve_value_rpe': True, 'retrieve_value_content': True, 'initialization_std_for_query_key_rpe': 1.0, 'initialization_std_for_value_rpe': 1.0, 'self_attention_activation': 'softmax'}, 'dual_path_transformer_layer_config': {'num_heads': 8, 'bottleneck_expansion': 2, 'key_expansion': 1, 'value_expansion': 2, 'feed_forward_network_channels': 2048, 'use_memory_self_attention': True, 'use_pixel2memory_feedback_attention': True, 'transformer_activation': 'softmax'}}, 'bn_layer': functools.partial(<class 'tensorflow.python.keras.layers.normalization_v2.SyncBatchNormalization'>, momentum=0.9900000095367432, epsilon=0.0010000000474974513), 'conv_kernel_weight_decay': 0.0} I1008 09:12:06.923306 139644910475072 motion_deeplab.py:109] Setting pooling size to (13, 40) I1008 09:12:06.923628 139644910475072 aspp.py:135] Global average pooling in the ASPP pooling layer was replaced with tiled average pooling using the provided pool_size. Please make sure this behavior is intended. I1008 09:12:06.923738 139644910475072 aspp.py:135] Global average pooling in the ASPP pooling layer was replaced with tiled average pooling using the provided pool_size. Please make sure this behavior is intended. 2021-10-08 09:12:11.757189: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. I1008 09:12:11.760688 139644910475072 controller.py:362] restoring or initializing model... restoring or initializing model... I1008 09:12:12.072492 139644910475072 controller.py:368] initialized model. initialized model. I1008 09:12:12.073319 139644910475072 controller.py:252] eval | step: 0 | running complete evaluation... eval | step: 0 | running complete evaluation... 2021-10-08 09:12:12.287843: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2) 2021-10-08 09:12:12.308308: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3399950000 Hz WARNING:tensorflow:From /usr/wiss/webermar/anaconda3/envs/deeplab_pip/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version. Instructions for updating: The validate_indices argument has no effect. Indices are always validated on CPU and never validated on GPU. W1008 09:12:19.992389 139644910475072 deprecation.py:534] From /usr/wiss/webermar/anaconda3/envs/deeplab_pip/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version. Instructions for updating: The validate_indices argument has no effect. Indices are always validated on CPU and never validated on GPU. 2021-10-08 09:12:25.851721: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:808] layout failed: Invalid argument: Size of values 3 does not match size of permutation 4 @ fanin shape inMotionDeepLab/PostProcessor/StatefulPartitionedCall/while/body/_231/while/SelectV2_1-1-TransposeNHWCToNCHW-LayoutOptimizer 2021-10-08 09:12:27.338178: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-10-08 09:12:28.743844: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8201 2021-10-08 09:12:30.845198: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2021-10-08 09:12:31.859844: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 I1008 09:37:38.998472 139644910475072 api.py:446] Creating COCO objects for AP eval... creating index... index created! Loading and preparing results... DONE (t=12.72s) creating index... index created! I1008 09:37:54.079124 139644910475072 api.py:446] Running COCO evaluation... Running per image evaluation... Evaluate annotation type segm DONE (t=98.34s). Accumulating evaluation results... DONE (t=3.74s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.375 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.651 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.356 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.150 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.481 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.676 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.141 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.438 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.439 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.195 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.562 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.764 I1008 09:39:38.485969 139644910475072 controller.py:261] eval | step: 0 | eval time: 1646.4 sec | output: {'evaluation/ap/AP_Mask': 0.3751787, 'evaluation/iou/IoU': 0.63153327, 'evaluation/pq/FN': 734.8947, 'evaluation/pq/FP': 622.7895, 'evaluation/pq/PQ': 0.4207694, 'evaluation/pq/RQ': 0.52304685, 'evaluation/pq/SQ': 0.77444, 'evaluation/pq/TP': 1501.0526, 'evaluation/step/AQ': 0.5277102021206063, 'evaluation/step/IoU': 0.6308144803396004, 'evaluation/step/STQ': 0.5769638090215155, 'losses/eval_center_loss': 0.06295311, 'losses/eval_motion_loss': 0.0750779, 'losses/eval_regression_loss': 0.016829815, 'losses/eval_semantic_loss': 2.098444, 'losses/eval_total_loss': 2.2533038} eval | step: 0 | eval time: 1646.4 sec | output: {'evaluation/ap/AP_Mask': 0.3751787, 'evaluation/iou/IoU': 0.63153327, 'evaluation/pq/FN': 734.8947, 'evaluation/pq/FP': 622.7895, 'evaluation/pq/PQ': 0.4207694, 'evaluation/pq/RQ': 0.52304685, 'evaluation/pq/SQ': 0.77444, 'evaluation/pq/TP': 1501.0526, 'evaluation/step/AQ': 0.5277102021206063, 'evaluation/step/IoU': 0.6308144803396004, 'evaluation/step/STQ': 0.5769638090215155, 'losses/eval_center_loss': 0.06295311, 'losses/eval_motion_loss': 0.0750779, 'losses/eval_regression_loss': 0.016829815, 'losses/eval_semantic_loss': 2.098444, 'losses/eval_total_loss': 2.2533038}

Could you provide your full config as well as full log when you evaluate with unchanged code?

lxtGH

comment created time in 18 days

issue commentgoogle-research/deeplab2

STEP dataset questions

@lxtGH The command I am using is the following:

python trainer/train.py --config_file="./configs/kitti/panoptic_deeplab/resnet50_os32.textproto" --num_gpus=1 --mode=eval --model_dir="/**retracted**/models/deeplab2/kitti/"

If you prepare the KITTI dataset correctly and download the pretrained models, you will get the scores we report.

lxtGH

comment created time in 20 days

push eventaljosaosep/bmtt2021

Mark Weber

commit sha 16fb3237083b1aa8922416d8237788e0a8f18e45

Add submission instructions

view details

push time in 22 days

issue commentgoogle-research/deeplab2

Issues about KITTI-STEP dataset

Hi HarborYuan,

Indeed, only the classes 11 and 13 have track annotations (see here: https://github.com/google-research/deeplab2/blob/main/data/dataset.py#L162).

I'm not sure I understand the question right. There is no guarantee that for example, an image only contains one bicycle. However, in case there are two bicycles these would not have different track IDs.

You only need to consider 11 and 13 as things, all other classes are considered to be stuff.

Please let me know, if that answers your question.

HarborYuan

comment created time in a month

push eventaljosaosep/bmtt2021

Mark Weber

commit sha 5bfc77352a275e26a2362ed28829aa38056788b1

Update deadline

view details

push time in a month

issue commentgoogle-research/deeplab2

STEP dataset questions

That I simply don't know, sorry. Please feel free to let us know if that's similar in case you try it. Then you should be able to compare it with the test set.

lxtGH

comment created time in a month

issue commentgoogle-research/deeplab2

STEP dataset questions

Hi,

Please note that STEP and MOTS are two different things.

  • STEP (Segmenting and Tracking Every Pixel) contains labels for every pixel that contain semantic class and potentially tracking labels.
  • MOTS (Multi-Object Tracking and Segmentation) does not contain labels for every pixel, but only for certain object classes.

We are offering KITTI-STEP and MOTChallenge-STEP labels.

Please find the guides here:

  • KITTI-STEP. This includes also how to split the training and validation set.
  • MOTChallenge-STEP. This does not have an official validation split due to the low number of sequences. Both training sequences do not contain the full label set, hence validation is difficult. Instead one could split each training sequence in two parts and train on one part while evaluating on the other.

Please let me know if that answers your question.

lxtGH

comment created time in a month

push eventaljosaosep/bmtt2021

Mark Weber

commit sha a21e35515960d2b499749e4e8969da2b07e57d17

Update website for numpy code and challenge deadline specification.

view details

push time in a month

delete branch google-research/deeplab2

delete branch : stq_np

delete time in 2 months

push eventgoogle-research/deeplab2

Mark Weber

commit sha 4c79c611b29d65f7ce9e0c20df99df29be5e12ee

Clarify doc-string to contain more information.

view details

push time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import Mapping, MutableMapping, Sequence, Text, Any+import numpy as np++_EPSILON = 1e-15+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  Please see the following paper for more details about the metric:++  "STEP: Segmenting and Tracking Every Pixel", Weber et al., arXiv:2102.11859, +  2021.+++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an

Please note that this encoding ( * label_divisor) is only (!) used internally in DeepLab2. Following our guides, e.g., for creating KITTI-STEP, the ground-truth is in 3 channel format (RGB) with R containing the semantic class. Furthermore, everything that is saved to disk is again 3 channel output (or at least not single-channel).

The test servers expect 3 channel files. (Explained clearly at each test server.) Hence, I don't think it's a problem. For Deeplab2, the TF version must be used anyway.

What do you think?

markweberdev

comment created time in 2 months

PullRequestReviewEvent

push eventgoogle-research/deeplab2

Mark Weber

commit sha 140bc3115270e0b7971bff502702eea391e87cf1

Revise numpy STQ implementation according to review.

view details

push time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')

removed as not used.

markweberdev

comment created time in 2 months

PullRequestReviewEvent

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1++    label_mask = np.zeros_like(semantic_label, dtype=np.bool)+    prediction_mask = np.zeros_like(semantic_prediction, dtype=np.bool)+    for things_class_id in self._things_list:+      label_mask = np.logical_or(label_mask,+                                 semantic_label == things_class_id)+      prediction_mask = np.logical_or(+          prediction_mask, semantic_prediction == things_class_id)++    # Select the `crowd` region of the current class. This region is encoded+    # instance id `0`.+    is_crowd = np.logical_and(instance_label == 0, label_mask)+    # Select the non-crowd region of the corresponding class as the `crowd`+    # region is ignored for the tracking term.+    label_mask = np.logical_and(label_mask, np.logical_not(is_crowd))+    # Do not punish id assignment for regions that are annotated as `crowd` in+    # the ground-truth.+    prediction_mask = np.logical_and(prediction_mask, np.logical_not(is_crowd))++    seq_preds = self._predictions[sequence_id]+    seq_gts = self._ground_truth[sequence_id]+    seq_intersects = self._intersections[sequence_id]++    # Compute and update areas of ground-truth, predictions and intersections.+    _update_dict_stats(seq_preds, y_pred[prediction_mask])+    _update_dict_stats(seq_gts, y_true[label_mask])++    non_crowd_intersection = np.logical_and(label_mask, prediction_mask)+    intersection_ids = (+        y_true[non_crowd_intersection] * self._offset ++        y_pred[non_crowd_intersection])+    _update_dict_stats(seq_intersects, intersection_ids)++  def result(self) -> Dict[Text, Any]:

Switched to mapping. I am not sure which immutable dict-like type you're referring to. I tried to recreate the TF interface as close as possible.

markweberdev

comment created time in 2 months

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Tests for segmentation_and_tracking_quality.py"""++import unittest+import numpy as np++import segmentation_and_tracking_quality as stq

Does that make sense if people want to use this in a stand-alone fashion? For example, I imagine people just copying this folder and it would be great, if it still works.

If yes, I am happy to change it.

markweberdev

comment created time in 2 months

PullRequestReviewEvent

PR opened google-research/deeplab2

Reviewers
Add numpy implementation of Segmentation and Tracking Quality (STQ)

This PR contains the numpy implementation of STQ.

Please be aware that due to a speed-up of over 2x, the panoptic label format required is:

panoptic_id = (class_id << label_shift) + track_id.

+571 -0

0 comment

4 changed files

pr created time in 2 months

push eventgoogle-research/deeplab2

Mark Weber

commit sha 07fabe0a2c0b9dbac54f91a00827576d6fc6d407

Add changelog.

view details

push time in 2 months

create barnchgoogle-research/deeplab2

branch : stq_np

created branch time in 2 months