profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/ClaireXie/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

google-research/deeplab2 485

DeepLab2 is a TensorFlow library for deep labeling, aiming to provide a unified and state-of-the-art TensorFlow codebase for dense pixel labeling tasks.

ClaireXie/modeling_3d 34

A simple demo for 3D Object Modeling based on an RGB-D camera

ClaireXie/edgeGuidedSDSP 29

Edge Guided Single Depth Image Super-resolution

ClaireXie/denseCRF_matlab 19

A lightweight MATLAB wrapper to Philipp Krähenbühl's dense (fully connected) CRFs with gaussian edge potentials.

ClaireXie/face2sketch 15

This code is the MATLAB implementation of converting face to sketch and vice versa

ClaireXie/ffmpeg-Opencv 7

Wrapper class of using OpenCV in ffmpeg

ClaireXie/strokeRender 2

Stroke Render from Aaron Hertzmann's Implementation

ClaireXie/google-research 0

Google Research

ClaireXie/styleguide 0

Style guides for Google-originated open-source projects

startedautonomousvision/kitti360Scripts

started time in a month

startedautonomousvision/kitti360LabelTool

started time in a month

issue openedsssdddwww2/vspw_dataset_download

Why there are images/masks' name starting with '._'?

Can we ignore those images?

created time in a month

push eventgoogle-research/deeplab2

Mark Weber

commit sha 43979b08f4d858b0b988340400b3bb5844a89f86

Project import generated by Copybara. PiperOrigin-RevId: 395568778

view details

push time in 2 months

push eventgoogle-research/deeplab2

Mark Weber

commit sha 03d5ee9dd7523bc5f9edf4882923617d8fc89eef

Add numpy implementation of Segmentation and Tracking Quality (STQ) (#52) * Add numpy version of the evaluation script. * Add changelog. * Revise numpy STQ implementation according to review. * Clarify doc-string to contain more information. Co-authored-by: Mark Weber <sindorf2012@gmail.com>

view details

push time in 2 months

PR merged google-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ) cla: yes

This PR contains the numpy implementation of STQ.

Please be aware that due to a speed-up of over 2x, the panoptic label format required is:

panoptic_id = (class_id << label_shift) + track_id.

+584 -0

0 comment

4 changed files

markweberdev

pr closed time in 2 months

PullRequestReviewEvent

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Tests for segmentation_and_tracking_quality.py"""++import unittest+import numpy as np++import segmentation_and_tracking_quality as stq

I see your point. The only issue is when we convert the code back to google codebase, import segmentation_and_tracking_quality as stq might create some dependency issues. But we should be able to add some rules to manually convert it.

markweberdev

comment created time in 2 months

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1++    label_mask = np.zeros_like(semantic_label, dtype=np.bool)+    prediction_mask = np.zeros_like(semantic_prediction, dtype=np.bool)+    for things_class_id in self._things_list:+      label_mask = np.logical_or(label_mask,+                                 semantic_label == things_class_id)+      prediction_mask = np.logical_or(+          prediction_mask, semantic_prediction == things_class_id)++    # Select the `crowd` region of the current class. This region is encoded+    # instance id `0`.+    is_crowd = np.logical_and(instance_label == 0, label_mask)+    # Select the non-crowd region of the corresponding class as the `crowd`+    # region is ignored for the tracking term.+    label_mask = np.logical_and(label_mask, np.logical_not(is_crowd))+    # Do not punish id assignment for regions that are annotated as `crowd` in+    # the ground-truth.+    prediction_mask = np.logical_and(prediction_mask, np.logical_not(is_crowd))++    seq_preds = self._predictions[sequence_id]+    seq_gts = self._ground_truth[sequence_id]+    seq_intersects = self._intersections[sequence_id]++    # Compute and update areas of ground-truth, predictions and intersections.+    _update_dict_stats(seq_preds, y_pred[prediction_mask])+    _update_dict_stats(seq_gts, y_true[label_mask])++    non_crowd_intersection = np.logical_and(label_mask, prediction_mask)+    intersection_ids = (+        y_true[non_crowd_intersection] * self._offset ++        y_pred[non_crowd_intersection])+    _update_dict_stats(seq_intersects, intersection_ids)++  def result(self) -> Dict[Text, Any]:

Yes Mapping should be good.

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:

use np.ndarray instead of np.array

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Tests for segmentation_and_tracking_quality.py"""++import unittest+import numpy as np++import segmentation_and_tracking_quality as stq

from deeplab2.evaluation.numpy import segmentation_and_tracking_quality as stq

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1++    label_mask = np.zeros_like(semantic_label, dtype=np.bool)+    prediction_mask = np.zeros_like(semantic_prediction, dtype=np.bool)+    for things_class_id in self._things_list:+      label_mask = np.logical_or(label_mask,+                                 semantic_label == things_class_id)+      prediction_mask = np.logical_or(+          prediction_mask, semantic_prediction == things_class_id)++    # Select the `crowd` region of the current class. This region is encoded+    # instance id `0`.+    is_crowd = np.logical_and(instance_label == 0, label_mask)+    # Select the non-crowd region of the corresponding class as the `crowd`+    # region is ignored for the tracking term.+    label_mask = np.logical_and(label_mask, np.logical_not(is_crowd))+    # Do not punish id assignment for regions that are annotated as `crowd` in+    # the ground-truth.+    prediction_mask = np.logical_and(prediction_mask, np.logical_not(is_crowd))++    seq_preds = self._predictions[sequence_id]+    seq_gts = self._ground_truth[sequence_id]+    seq_intersects = self._intersections[sequence_id]++    # Compute and update areas of ground-truth, predictions and intersections.+    _update_dict_stats(seq_preds, y_pred[prediction_mask])+    _update_dict_stats(seq_gts, y_true[label_mask])++    non_crowd_intersection = np.logical_and(label_mask, prediction_mask)+    intersection_ids = (+        y_true[non_crowd_intersection] * self._offset ++        y_pred[non_crowd_intersection])+    _update_dict_stats(seq_intersects, intersection_ids)++  def result(self) -> Dict[Text, Any]:+    """Computes the segmentation and tracking quality.++    Returns:+      A dictionary containing:+        - 'STQ': The total STQ score.+        - 'AQ': The total association quality (AQ) score.+        - 'IoU': The total mean IoU.+        - 'STQ_per_seq': A list of the STQ score per sequence.+        - 'AQ_per_seq': A list of the AQ score per sequence.+        - 'IoU_per_seq': A list of mean IoU per sequence.+        - 'Id_per_seq': A list of sequence Ids to map list index to sequence.

maybe mention that ids are strings

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).

Can you refer to the STEP paper either here or at the top?

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1

At least two spaces before "#"

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1++    label_mask = np.zeros_like(semantic_label, dtype=np.bool)+    prediction_mask = np.zeros_like(semantic_prediction, dtype=np.bool)+    for things_class_id in self._things_list:+      label_mask = np.logical_or(label_mask,+                                 semantic_label == things_class_id)+      prediction_mask = np.logical_or(+          prediction_mask, semantic_prediction == things_class_id)++    # Select the `crowd` region of the current class. This region is encoded+    # instance id `0`.+    is_crowd = np.logical_and(instance_label == 0, label_mask)+    # Select the non-crowd region of the corresponding class as the `crowd`+    # region is ignored for the tracking term.+    label_mask = np.logical_and(label_mask, np.logical_not(is_crowd))+    # Do not punish id assignment for regions that are annotated as `crowd` in+    # the ground-truth.+    prediction_mask = np.logical_and(prediction_mask, np.logical_not(is_crowd))++    seq_preds = self._predictions[sequence_id]+    seq_gts = self._ground_truth[sequence_id]+    seq_intersects = self._intersections[sequence_id]++    # Compute and update areas of ground-truth, predictions and intersections.+    _update_dict_stats(seq_preds, y_pred[prediction_mask])+    _update_dict_stats(seq_gts, y_true[label_mask])++    non_crowd_intersection = np.logical_and(label_mask, prediction_mask)+    intersection_ids = (+        y_true[non_crowd_intersection] * self._offset ++        y_pred[non_crowd_intersection])+    _update_dict_stats(seq_intersects, intersection_ids)++  def result(self) -> Dict[Text, Any]:+    """Computes the segmentation and tracking quality.++    Returns:+      A dictionary containing:+        - 'STQ': The total STQ score.+        - 'AQ': The total association quality (AQ) score.+        - 'IoU': The total mean IoU.+        - 'STQ_per_seq': A list of the STQ score per sequence.+        - 'AQ_per_seq': A list of the AQ score per sequence.+        - 'IoU_per_seq': A list of mean IoU per sequence.+        - 'Id_per_seq': A list of sequence Ids to map list index to sequence.+        - 'Length_per_seq': A list of the length of each sequence.+    """+    # Compute association quality (AQ)+    num_tubes_per_seq = [0] * len(self._ground_truth)+    aq_per_seq = [0] * len(self._ground_truth)+    iou_per_seq = [0] * len(self._ground_truth)+    id_per_seq = [''] * len(self._ground_truth)++    for index, sequence_id in enumerate(self._ground_truth):+      outer_sum = 0.0+      predictions = self._predictions[sequence_id]+      ground_truth = self._ground_truth[sequence_id]+      intersections = self._intersections[sequence_id]+      num_tubes_per_seq[index] = len(ground_truth)+      id_per_seq[index] = sequence_id++      for gt_id, gt_size in ground_truth.items():+        inner_sum = 0.0+        for pr_id, pr_size in predictions.items():+          tpa_key = self._offset * gt_id + pr_id+          if tpa_key in intersections:+            tpa = intersections[tpa_key]+            fpa = pr_size - tpa+            fna = gt_size - tpa+            inner_sum += tpa * (tpa / (tpa + fpa + fna))++        outer_sum += 1.0 / gt_size * inner_sum+      aq_per_seq[index] = outer_sum++    aq_mean = np.sum(aq_per_seq) / np.maximum(np.sum(num_tubes_per_seq), 1e-15)

Can you make 1e-15 as a constant (epsilon) at the top?

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1++    label_mask = np.zeros_like(semantic_label, dtype=np.bool)+    prediction_mask = np.zeros_like(semantic_prediction, dtype=np.bool)+    for things_class_id in self._things_list:+      label_mask = np.logical_or(label_mask,+                                 semantic_label == things_class_id)+      prediction_mask = np.logical_or(+          prediction_mask, semantic_prediction == things_class_id)++    # Select the `crowd` region of the current class. This region is encoded+    # instance id `0`.+    is_crowd = np.logical_and(instance_label == 0, label_mask)+    # Select the non-crowd region of the corresponding class as the `crowd`+    # region is ignored for the tracking term.+    label_mask = np.logical_and(label_mask, np.logical_not(is_crowd))+    # Do not punish id assignment for regions that are annotated as `crowd` in+    # the ground-truth.+    prediction_mask = np.logical_and(prediction_mask, np.logical_not(is_crowd))++    seq_preds = self._predictions[sequence_id]+    seq_gts = self._ground_truth[sequence_id]+    seq_intersects = self._intersections[sequence_id]++    # Compute and update areas of ground-truth, predictions and intersections.+    _update_dict_stats(seq_preds, y_pred[prediction_mask])+    _update_dict_stats(seq_gts, y_true[label_mask])++    non_crowd_intersection = np.logical_and(label_mask, prediction_mask)+    intersection_ids = (+        y_true[non_crowd_intersection] * self._offset ++        y_pred[non_crowd_intersection])+    _update_dict_stats(seq_intersects, intersection_ids)++  def result(self) -> Dict[Text, Any]:

Mapping[...]? Use immutable if possible

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Tests for segmentation_and_tracking_quality.py"""

period.

markweberdev

comment created time in 2 months

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')+    """+    self._name = name+    self._num_classes = num_classes+    self._ignore_label = ignore_label+    self._things_list = things_list+    self._label_bit_shift = label_bit_shift+    self._bit_mask = (2 ** label_bit_shift) - 1++    if ignore_label >= num_classes:+      self._confusion_matrix_size = num_classes + 1+      self._include_indices = np.arange(self._num_classes)+    else:+      self._confusion_matrix_size = num_classes+      self._include_indices = np.array(+          [i for i in range(num_classes) if i != self._ignore_label])++    self._iou_confusion_matrix_per_sequence = collections.OrderedDict()+    self._predictions = collections.OrderedDict()+    self._ground_truth = collections.OrderedDict()+    self._intersections = collections.OrderedDict()+    self._sequence_length = collections.OrderedDict()+    self._offset = offset+    lower_bound = num_classes << self._label_bit_shift+    if offset < lower_bound:+      raise ValueError('The provided offset %d is too small. No guarantess '+                       'about the correctness of the results can be made. '+                       'Please choose an offset that is higher than num_classes'+                       ' * max_instances_per_category = %d' % lower_bound)++  def get_semantic(self, y: np.array) -> np.array:+    """Returns the semantic class from a panoptic label map."""+    return y >> self._label_bit_shift++  def update_state(self, y_true: np.ndarray, y_pred: np.ndarray,+                   sequence_id=0):+    """Accumulates the segmentation and tracking quality statistics.++    IMPORTANT: When encoding the parameters y_true and y_pred, please be aware+    that the `+` operator binds higher than the label shift `<<` operator.++    Args:+      y_true: The ground-truth panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      y_pred: The predicted panoptic label map for a particular video frame+        (defined as (semantic_map << label_bit_shift) + instance_map).+      sequence_id: The optional ID of the sequence the frames belong to. When no+        sequence is given, all frames are considered to belong to the same+        sequence (default: 0).+    """+    y_true = y_true.astype(np.int64)+    y_pred = y_pred.astype(np.int64)++    semantic_label = self.get_semantic(y_true)+    semantic_prediction = self.get_semantic(y_pred)+    # Check if the ignore value is outside the range [0, num_classes]. If yes,+    # map `_ignore_label` to `_num_classes`, so it can be used to create the+    # confusion matrix.+    if self._ignore_label > self._num_classes:+      semantic_label = np.where(+          semantic_label != self._ignore_label, semantic_label,+          self._num_classes)+      semantic_prediction = np.where(+          semantic_prediction != self._ignore_label,+          semantic_prediction, self._num_classes)+    if sequence_id in self._iou_confusion_matrix_per_sequence:+      idxs = (np.reshape(semantic_label, [-1]) << self._label_bit_shift) + np.reshape(+          semantic_prediction, [-1])+      unique_idxs, counts = np.unique(idxs, return_counts=True)+      self._iou_confusion_matrix_per_sequence[sequence_id][+          unique_idxs >> self._label_bit_shift, unique_idxs & self._bit_mask] += counts+      self._sequence_length[sequence_id] += 1+    else:+      self._iou_confusion_matrix_per_sequence[sequence_id] = np.zeros(+          (self._confusion_matrix_size, self._confusion_matrix_size), dtype=np.int64)+      idxs = np.stack([np.reshape(semantic_label, [-1]), +                np.reshape(semantic_prediction, [-1])], axis=0)+      np.add.at(self._iou_confusion_matrix_per_sequence[sequence_id], tuple(idxs), 1)++      self._predictions[sequence_id] = {}+      self._ground_truth[sequence_id] = {}+      self._intersections[sequence_id] = {}+      self._sequence_length[sequence_id] = 1++    # instance_label = y_true % self._max_instances_per_category+    instance_label = y_true & self._bit_mask # 0xFFFF == 2 ^ 16 - 1++    label_mask = np.zeros_like(semantic_label, dtype=np.bool)+    prediction_mask = np.zeros_like(semantic_prediction, dtype=np.bool)+    for things_class_id in self._things_list:+      label_mask = np.logical_or(label_mask,+                                 semantic_label == things_class_id)+      prediction_mask = np.logical_or(+          prediction_mask, semantic_prediction == things_class_id)++    # Select the `crowd` region of the current class. This region is encoded+    # instance id `0`.+    is_crowd = np.logical_and(instance_label == 0, label_mask)+    # Select the non-crowd region of the corresponding class as the `crowd`+    # region is ignored for the tracking term.+    label_mask = np.logical_and(label_mask, np.logical_not(is_crowd))+    # Do not punish id assignment for regions that are annotated as `crowd` in+    # the ground-truth.+    prediction_mask = np.logical_and(prediction_mask, np.logical_not(is_crowd))++    seq_preds = self._predictions[sequence_id]+    seq_gts = self._ground_truth[sequence_id]+    seq_intersects = self._intersections[sequence_id]++    # Compute and update areas of ground-truth, predictions and intersections.+    _update_dict_stats(seq_preds, y_pred[prediction_mask])+    _update_dict_stats(seq_gts, y_true[label_mask])++    non_crowd_intersection = np.logical_and(label_mask, prediction_mask)+    intersection_ids = (+        y_true[non_crowd_intersection] * self._offset ++        y_pred[non_crowd_intersection])+    _update_dict_stats(seq_intersects, intersection_ids)++  def result(self) -> Dict[Text, Any]:+    """Computes the segmentation and tracking quality.++    Returns:+      A dictionary containing:+        - 'STQ': The total STQ score.+        - 'AQ': The total association quality (AQ) score.+        - 'IoU': The total mean IoU.+        - 'STQ_per_seq': A list of the STQ score per sequence.+        - 'AQ_per_seq': A list of the AQ score per sequence.+        - 'IoU_per_seq': A list of mean IoU per sequence.+        - 'Id_per_seq': A list of sequence Ids to map list index to sequence.+        - 'Length_per_seq': A list of the length of each sequence.+    """+    # Compute association quality (AQ)+    num_tubes_per_seq = [0] * len(self._ground_truth)+    aq_per_seq = [0] * len(self._ground_truth)+    iou_per_seq = [0] * len(self._ground_truth)+    id_per_seq = [''] * len(self._ground_truth)++    for index, sequence_id in enumerate(self._ground_truth):+      outer_sum = 0.0+      predictions = self._predictions[sequence_id]+      ground_truth = self._ground_truth[sequence_id]+      intersections = self._intersections[sequence_id]+      num_tubes_per_seq[index] = len(ground_truth)+      id_per_seq[index] = sequence_id++      for gt_id, gt_size in ground_truth.items():+        inner_sum = 0.0+        for pr_id, pr_size in predictions.items():+          tpa_key = self._offset * gt_id + pr_id+          if tpa_key in intersections:+            tpa = intersections[tpa_key]+            fpa = pr_size - tpa+            fna = gt_size - tpa+            inner_sum += tpa * (tpa / (tpa + fpa + fna))++        outer_sum += 1.0 / gt_size * inner_sum+      aq_per_seq[index] = outer_sum++    aq_mean = np.sum(aq_per_seq) / np.maximum(np.sum(num_tubes_per_seq), 1e-15)+    aq_per_seq = aq_per_seq / np.maximum(num_tubes_per_seq, 1e-15)++    # Compute IoU scores.+    # The rows correspond to ground-truth and the columns to predictions.+    # Remove fp from confusion matrix for the void/ignore class.+    total_confusion = np.zeros(+        (self._confusion_matrix_size, self._confusion_matrix_size),+        dtype=np.int64)+    for index, confusion in enumerate(+        self._iou_confusion_matrix_per_sequence.values()):+      confusion = confusion

individual_confusion = confusion? Just don't assign the same var to itself.

markweberdev

comment created time in 2 months

PullRequestReviewEvent

Pull request review commentgoogle-research/deeplab2

Add numpy implementation of Segmentation and Tracking Quality (STQ)

+# coding=utf-8+# Copyright 2021 The Deeplab2 Authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Implementation of the Segmentation and Tracking Quality (STQ) metric."""++import collections+from typing import MutableMapping, Sequence, Dict, Text, Any+import numpy as np+++def _update_dict_stats(stat_dict: MutableMapping[int, np.ndarray],+                       id_array: np.ndarray):+  """Updates a given dict with corresponding counts."""+  ids, counts = np.unique(id_array, return_counts=True)+  for idx, count in zip(ids, counts):+    if idx in stat_dict:+      stat_dict[idx] += count+    else:+      stat_dict[idx] = count+++class STQuality(object):+  """Metric class for the Segmentation and Tracking Quality (STQ).++  The metric computes the geometric mean of two terms.+  - Association Quality: This term measures the quality of the track ID+      assignment for `thing` classes. It is formulated as a weighted IoU+      measure.+  - Segmentation Quality: This term measures the semantic segmentation quality.+      The standard class IoU measure is used for this.++  Example usage:++  stq_obj = segmentation_tracking_quality.STQuality(num_classes, things_list,+    ignore_label, label_bit_shift, offset)+  stq_obj.update_state(y_true_1, y_pred_1)+  stq_obj.update_state(y_true_2, y_pred_2)+  ...+  result = stq_obj.result()+  """++  def __init__(self,+               num_classes: int,+               things_list: Sequence[int],+               ignore_label: int,+               label_bit_shift: int,+               offset: int,+               name='stq'+               ):+    """Initialization of the STQ metric.++    Args:+      num_classes: Number of classes in the dataset as an integer.+      things_list: A sequence of class ids that belong to `things`.+      ignore_label: The class id to be ignored in evaluation as an integer or+        integer tensor.+      label_bit_shift: The number of bits the class label is shifted as an+        integer: (class_label << bits) + trackingID+      offset: The maximum number of unique labels as an integer or integer+        tensor.+      name: An optional name. (default: 'st_quality')

optional scope name

markweberdev

comment created time in 2 months

PullRequestReviewEvent
CommitCommentEvent
CommitCommentEvent
CommitCommentEvent
CommitCommentEvent
CommitCommentEvent
CommitCommentEvent
CommitCommentEvent