profile
viewpoint

niemeyer/gopkg 479

Source code for the gopkg.in service.

niemeyer/godeb 281

godeb transforms upstream tarballs for the Go language in deb packages and installs them.

python-constraint/python-constraint 181

Constraint Solving Problem resolver for Python

niemeyer/hsandbox 128

Hacking Sandbox: Multi-language interactive edit [> compile] > run hacking and experimenting tool.

niemeyer/golang 36

The Go Language

canonical/operator 23

The Python library behind great charms.

CanonicalLtd/landscape-client 16

The Landscape Client is the agent which communicates with the Landscape service.

niemeyer/flex 12

Bootstrap of flex project.

Pull request review commentcanonical/operator

Add JujuLogHandler to support the juju-log tool

+# Copyright 2020 Canonical Ltd.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++import logging+++class JujuLogHandler(logging.Handler):+    """A handler for sending logs to Juju via juju-log."""++    def __init__(self, model_backend, level=logging.DEBUG):

See the other note on the default level.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add JujuLogHandler to support the juju-log tool

+# Copyright 2020 Canonical Ltd.

Can we please name this file as "log.py" instead? We should be able to put other logging things here as well if necessary, instead of creating another one.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add JujuLogHandler to support the juju-log tool

+# Copyright 2020 Canonical Ltd.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++import logging+++class JujuLogHandler(logging.Handler):+    """A handler for sending logs to Juju via juju-log."""++    def __init__(self, model_backend, level=logging.DEBUG):+        super().__init__(level)+        self.model_backend = model_backend++    def emit(self, record):+        self.model_backend.juju_log(record.levelname, self.format(record))+++def setup_default_logging(model_backend):+    logger = logging.getLogger()+    logger.setLevel(logging.DEBUG)

Debugging is generally not a great default. Debugging messages are in-depth, and people (devs, admins) don't expect them to be generated unless someone is trying to debug something. Defaulting to it will mean people will learn to use debug as info.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add JujuLogHandler to support the juju-log tool

+# Copyright 2020 Canonical Ltd.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++import logging+++class JujuLogHandler(logging.Handler):+    """A handler for sending logs to Juju via juju-log."""++    def __init__(self, model_backend, level=logging.DEBUG):+        super().__init__(level)+        self.model_backend = model_backend++    def emit(self, record):+        self.model_backend.juju_log(record.levelname, self.format(record))+++def setup_default_logging(model_backend):

I think this should be setup_root_logger instead. It's the logger that is being setup, and the unnamed logger is called the root logger IIRC.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add JujuLogHandler to support the juju-log tool

+#!/usr/bin/python3++# Copyright 2020 Canonical Ltd.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++import unittest+import importlib++import logging+import ops.jujulog+++class FakeModelBackend:++    def __init__(self):+        self._calls = []++    def calls(self, clear=False):+        calls = self._calls+        if clear:+            self._calls = []+        return calls++    def juju_log(self, message, level):+        self._calls.append((message, level))+++class TestJujuLog(unittest.TestCase):++    def setUp(self):+        self.backend = FakeModelBackend()++        logging.shutdown()+        importlib.reload(logging)++    def test_default_logging(self):+        ops.jujulog.setup_default_logging(self.backend)++        logger = logging.getLogger()+        self.assertEqual(logger.level, logging.DEBUG)+        self.assertIsInstance(logger.handlers[0], ops.jujulog.JujuLogHandler)++        test_cases = [(+            lambda: logger.critical('critical'), ('CRITICAL', 'critical')+        ), (+            lambda: logger.error('error'), ('ERROR', 'error')+        ), (+            lambda: logger.warning('warning'), ('WARNING', 'warning'),+        ), (+            lambda: logger.info('info'), ('INFO', 'info'),+        ), (+            lambda: logger.debug('debug'), ('DEBUG', 'debug')+        )]++        for do, res in test_cases:+            do()+            calls = self.backend.calls(clear=True)+            self.assertEqual(len(calls), 1)+            self.assertEqual(calls[0], res)++    def test_handler_filtering(self):+        logger = logging.getLogger()+        logger.setLevel(logging.INFO)+        logger.addHandler(ops.jujulog.JujuLogHandler(self.backend, logging.WARNING))+        logger.info('debug')+        self.assertEqual(self.backend.calls(), [])

This is missing the positive case of the filter. That filter could be broken filtering everything that the test would still pass.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, relation):+        if not isinstance(relation, Relation):+            raise ModelError(f'a relation argument passed is not a Relation: {type(relation).__name__}')+        binding = self._data.get(relation)+        if binding is None:+            self._data[relation] = binding = Binding(relation.name, relation.id, self._backend)+        return binding+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        if not isinstance(relation_id, int):+            raise ModelError(f'relation_id must be an int, not {type(relation_id).__name__}')+        self._relation_id = relation_id+        self._backend = backend+        self._network = None++    @property+    def network(self):+        if self._network is None:+            self._network = Network(self._backend.network_get(self.name, self._relation_id))+        return self._network+++class Network:+    """A representation of a unit's view of the network associated with a network space binding."""
"""Network space details."""

This is the network space itself. The binding.network is what refers to it, not the opposite.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, relation):+        if not isinstance(relation, Relation):+            raise ModelError(f'a relation argument passed is not a Relation: {type(relation).__name__}')+        binding = self._data.get(relation)+        if binding is None:+            self._data[relation] = binding = Binding(relation.name, relation.id, self._backend)+        return binding+++class Binding:+    """A representation of a binding to a space."""
"""Binding to a network space."""

Most things here will end up being "A representation of ...", so we can just drop it.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, relation):+        if not isinstance(relation, Relation):+            raise ModelError(f'a relation argument passed is not a Relation: {type(relation).__name__}')+        binding = self._data.get(relation)+        if binding is None:+            self._data[relation] = binding = Binding(relation.name, relation.id, self._backend)+        return binding+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        if not isinstance(relation_id, int):+            raise ModelError(f'relation_id must be an int, not {type(relation_id).__name__}')+        self._relation_id = relation_id+        self._backend = backend+        self._network = None++    @property+    def network(self):+        if self._network is None:+            self._network = Network(self._backend.network_get(self.name, self._relation_id))+        return self._network+++class Network:+    """A representation of a unit's view of the network associated with a network space binding."""++    def __init__(self, network_info):+        self.interfaces = []+        # Treat multiple addresses on an interface as multiple logical interfaces with the same name.+        for interface_info in network_info['bind-addresses']:+            interface_name = interface_info['interface-name']+            for address_info in interface_info['addresses']:+                self.interfaces.append(NetworkInterface(interface_name, address_info))+        self.ingress_addresses = []+        for address in network_info['ingress-addresses']:+            self.ingress_addresses.append(ipaddress.ip_address(address))+        self.egress_subnets = []+        for subnet in network_info['egress-subnets']:+            self.egress_subnets.append(ipaddress.ip_network(subnet))

This ended up looking good. Glad we spent the time to discuss it.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def __init__(self, relation_name, relation_id, is_peer, our_unit, backend, cache     def __repr__(self):         return f'<{type(self).__module__}.{type(self).__name__} {self.name}:{self.id}>' +    def __hash__(self):+        return hash((self.id, self.name))

I believe the ID itself is unique, but needs double checking.

I'm also curious about why this was required. We hopefully shouldn't have more than a single object representing the same relation inside the same logic. If that happens, we'll have other bugs even if we clash them like this.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""

Comment is wrong. Let's please just drop it instead of tweaking this much more so, so that we can evolve this code in-tree.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, relation):+        if not isinstance(relation, Relation):+            raise ModelError(f'a relation argument passed is not a Relation: {type(relation).__name__}')+        binding = self._data.get(relation)+        if binding is None:+            self._data[relation] = binding = Binding(relation.name, relation.id, self._backend)+        return binding+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        if not isinstance(relation_id, int):+            raise ModelError(f'relation_id must be an int, not {type(relation_id).__name__}')

Why are we doing such type checking at apparently arbitrary places, and on particular parameters only? Why do we validate the id, and not the name, or the backend?

This is an internal type.. the data shouldn't reach that far without it being right, if we really are worried about people misusing it.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, relation):+        if not isinstance(relation, Relation):+            raise ModelError(f'a relation argument passed is not a Relation: {type(relation).__name__}')
"expected Relation instance, got {type(relation).__name__}"
dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add modeling of network primitives

 def get_relation(self, relation_name, relation_id=None):         """         return self.relations._get_unique(relation_name, relation_id) +    def get_binding(self, relation):+        """Get a network space binding for a Relation object."""

I expect people to want names here as well, for reasons we discussed previously and that are in the chat history today (Feb 21st 2020). I would suggest merging it like this, though, and discussing later, otherwise this is a never-ending exercise.

dshcherb

comment created time in 2 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metrics(self, metrics, labels=None):+        cmd = ['add-metric']+        key_re = re.compile(r'^[a-zA-Z][a-zA-Z0-9_]*$')++        def is_valid_metric_value(value):+            try:+                float(value)+                return True+            except ValueError:+                return False++        def is_valid_label_value(value):+            # Label values cannot be empty, contain commas or equal signs as those are used by add-metric as separators.+            return value and all(c not in str(value) for c in ',=')++        if labels:+            label_args = []+            for k, v in labels.items():+                if not key_re.match(k):+                    raise ModelError(f'invalid label key "{k}": must start from an ASCII letter and contain alphanumeric characters or underscores only')+                elif not is_valid_label_value(v):+                    raise ModelError('metric label values must not contain "," or "=".')+                else:+                    label_args.append(f'{k}={v}')+            cmd.extend(['--labels', ','.join(label_args)])++        metric_args = []+        for k, v in metrics.items():+            if not key_re.match(k):+                raise ModelError(f'invalid metric key "{k}": must start from an ASCII letter and contain alphanumeric characters or underscores only')+            elif not is_valid_metric_value(v):+                raise ModelError(f'invalid value "{v}" provided for key "{k}": must be a real number')

s/real/float/; also needs repr

dshcherb

comment created time in 5 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def on_foo_bar_action(self, event):         self._state['observed_event_types'].append(type(event))         self._write_state() +    def on_collect_metrics(self, event):+        self._state['on_collect_metrics'].append(type(event))+        self._state['observed_event_types'].append(type(event))+        event.add_metrics({'foo': 42}, {'bar': 4.2})

Is it required for add-metrics to be called inside the collect-metrics hook? What happens if it's called by other hooks?

dshcherb

comment created time in 5 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metrics(self, metrics, labels=None):+        cmd = ['add-metric']+        key_re = re.compile(r'^[a-zA-Z][a-zA-Z0-9_]*$')++        def is_valid_metric_value(value):+            try:+                float(value)+                return True+            except ValueError:+                return False++        def is_valid_label_value(value):+            # Label values cannot be empty, contain commas or equal signs as those are used by add-metric as separators.+            return value and all(c not in str(value) for c in ',=')++        if labels:+            label_args = []+            for k, v in labels.items():+                if not key_re.match(k):+                    raise ModelError(f'invalid label key "{k}": must start from an ASCII letter and contain alphanumeric characters or underscores only')+                elif not is_valid_label_value(v):+                    raise ModelError('metric label values must not contain "," or "=".')+                else:+                    label_args.append(f'{k}={v}')+            cmd.extend(['--labels', ','.join(label_args)])++        metric_args = []+        for k, v in metrics.items():+            if not key_re.match(k):+                raise ModelError(f'invalid metric key "{k}": must start from an ASCII letter and contain alphanumeric characters or underscores only')

Same as above.

dshcherb

comment created time in 5 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metrics(self, metrics, labels=None):+        cmd = ['add-metric']+        key_re = re.compile(r'^[a-zA-Z][a-zA-Z0-9_]*$')++        def is_valid_metric_value(value):+            try:+                float(value)+                return True+            except ValueError:+                return False++        def is_valid_label_value(value):+            # Label values cannot be empty, contain commas or equal signs as those are used by add-metric as separators.+            return value and all(c not in str(value) for c in ',=')++        if labels:+            label_args = []+            for k, v in labels.items():+                if not key_re.match(k):+                    raise ModelError(f'invalid label key "{k}": must start from an ASCII letter and contain alphanumeric characters or underscores only')+                elif not is_valid_label_value(v):+                    raise ModelError('metric label values must not contain "," or "=".')

No dots at the end of error messages.

dshcherb

comment created time in 5 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metrics(self, metrics, labels=None):+        cmd = ['add-metric']+        key_re = re.compile(r'^[a-zA-Z][a-zA-Z0-9_]*$')++        def is_valid_metric_value(value):+            try:+                float(value)+                return True+            except ValueError:+                return False++        def is_valid_label_value(value):+            # Label values cannot be empty, contain commas or equal signs as those are used by add-metric as separators.+            return value and all(c not in str(value) for c in ',=')++        if labels:+            label_args = []+            for k, v in labels.items():+                if not key_re.match(k):+                    raise ModelError(f'invalid label key "{k}": must start from an ASCII letter and contain alphanumeric characters or underscores only')
f'invalid label key {repr(k)}: must start with [a-zA-Z] and contain [a-zA-Z0-9_] only'

Same for the respective message below. Without the repr this may get confusing depending on the content, and that's bad if we're precisely trying to warn problems about awkward content.

dshcherb

comment created time in 5 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, binding_name):+        if isinstance(binding_name, Relation):+            key = binding_name+            name = binding_name.name+            relation_id = binding_name.id+        elif isinstance(binding_name, str):+            name = key = binding_name+            relation_id = None+        else:+            raise ModelError(f'binding_name must be a str or Relation, not {type(binding_name).__name__}')+        binding = self._data.get(key)+        if binding is None:+            binding = Binding(name, relation_id, self._backend)+            self._data[key] = binding+        return binding+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        self._relation_id = relation_id+        self._backend = backend+        self._network = None++    @property+    def network(self):+        if self._network is None:+            try:+                if self._relation_id is None:+                    network_info = self._backend.network_get(self.name)+                else:+                    network_info = self._backend.network_get(self.name, self._relation_id)+            except RelationNotFoundError:+                # If a relation is dead, we can still get network info associated with an endpoint itself+                # not associated with a particular relation id.+                network_info = self._backend.network_get(self.name)+            self._network = Network(network_info['bind-addresses'], network_info['ingress-addresses'], network_info['egress-subnets'])+        return self._network+++class Network:+    """A representation of a unit's view of the network associated with a network space binding."""++    def __init__(self, device_info, ingress_addresses, egress_subnets):+        self.devices = []+        for netdev in device_info:+            self.devices.append(NetworkDevice(netdev['interface-name'], netdev['addresses']))++        self.ingress_addresses = []+        for address in ingress_addresses:+            self.ingress_addresses.append(ipaddress.ip_address(address))+        self.egress_subnets = []+        for subnet in egress_subnets:+            self.egress_subnets.append(ipaddress.ip_network(subnet))++    @property+    def bind_address(self):+        return self.devices[0].addresses[0]++    @property+    def ingress_address(self):+        return self.ingress_addresses[0]+++class NetworkDevice:++    def __init__(self, name, address_info):+        self.name = name+        self.addresses = []+        self.cidrs = []+        for entry in address_info:+            address = entry['value']+            self.addresses.append(ipaddress.ip_address(address))+            cidr = entry.get('cidr')+            if cidr:+                self.cidrs.append(ipaddress.ip_interface(cidr))+            else:+                # At least provide a /32 if there is no CIDR provided.+                self.cidrs.append(ipaddress.ip_interface(entry['value']))

Why is it not using ip_network here? Having to index both address and cidrs by the same value to be able to associate one with the other feels a bit uncomfortable.

dshcherb

comment created time in 5 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, binding_name):+        if isinstance(binding_name, Relation):+            key = binding_name+            name = binding_name.name+            relation_id = binding_name.id+        elif isinstance(binding_name, str):+            name = key = binding_name+            relation_id = None+        else:+            raise ModelError(f'binding_name must be a str or Relation, not {type(binding_name).__name__}')

binding_name must be a Relation? That's a good hint.

dshcherb

comment created time in 6 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, binding_name):+        if isinstance(binding_name, Relation):+            key = binding_name+            name = binding_name.name+            relation_id = binding_name.id+        elif isinstance(binding_name, str):+            name = key = binding_name+            relation_id = None+        else:+            raise ModelError(f'binding_name must be a str or Relation, not {type(binding_name).__name__}')+        binding = self._data.get(key)+        if binding is None:+            binding = Binding(name, relation_id, self._backend)+            self._data[key] = binding+        return binding+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        self._relation_id = relation_id+        self._backend = backend+        self._network = None++    @property+    def network(self):+        if self._network is None:+            try:+                if self._relation_id is None:+                    network_info = self._backend.network_get(self.name)+                else:+                    network_info = self._backend.network_get(self.name, self._relation_id)+            except RelationNotFoundError:+                # If a relation is dead, we can still get network info associated with an endpoint itself+                # not associated with a particular relation id.+                network_info = self._backend.network_get(self.name)+            self._network = Network(network_info['bind-addresses'], network_info['ingress-addresses'], network_info['egress-subnets'])

Why are we splitting it up here if we're passing that data to the object that precisely is the one that knows about these attributes, and also knows about the mapping itself given it's digging further into it?

dshcherb

comment created time in 6 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping:+    """A map of binding names to lists of Binding instances."""++    def __init__(self, backend):+        self._backend = backend+        self._data = {}++    def get(self, binding_name):+        if isinstance(binding_name, Relation):+            key = binding_name+            name = binding_name.name+            relation_id = binding_name.id+        elif isinstance(binding_name, str):+            name = key = binding_name

This type needs a bit of love as a whole. The idea is pretty simple, yet it manages to mix things up somewhat.

The map should be more consistent, and the variable names should be more indicative of what they hold. binding_name.name? binding_name.id? name = binding_name? There's no other name being handled by this method other than binding names, so why do we have two binding names to hold the same name? Etc.

dshcherb

comment created time in 6 days

Pull request review commentcanonical/operator

Rework DB lock handling

 def test_helper_properties(self):         self.assertEqual(my_obj.meta, framework.meta)         self.assertEqual(my_obj.charm_dir, framework.charm_dir) +    def test_ban_concurrent_frameworks(self):+        framework = self.create_framework()+        with self.assertRaises(Exception) as cm:+            framework_copy = self.create_framework()+            self.fail(f'frameworks {framework} and {framework_copy} got instantiated successfully')+        self.assertTrue('database is locked' in str(cm.exception))

This is better, as when it fails it will tell you what actually was in the string instead:

self.assertIn('database is locked', str(cm.exception))

But in this case, it looks like it might be simply:

self.assertEqual('database is locked', str(cm.exception))
dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Rework DB lock handling

 def test_helper_properties(self):         self.assertEqual(my_obj.meta, framework.meta)         self.assertEqual(my_obj.charm_dir, framework.charm_dir) +    def test_ban_concurrent_frameworks(self):+        framework = self.create_framework()+        with self.assertRaises(Exception) as cm:+            framework_copy = self.create_framework()+            self.fail(f'frameworks {framework} and {framework_copy} got instantiated successfully')

As a side note, this line doesn't seem to be adding any useful insight. It be saying something about what error condition was expected, but instead it's literally saying that the line above worked fine, which is what assertRaises will already do by default.

dshcherb

comment created time in 9 days

CommitCommentEvent
CommitCommentEvent

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metric(self, metrics, labels=None):

The point was probably not clear. Imagine you have a comma in a key, or an equal sign.. what happens?

Given that, we need to restrict the vocabulary. What is the exact vocabulary accepted by juju, so we're not diverging? I assume it's not a completely lenient string handling.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Rework DB lock handling

 def test_helper_properties(self):         self.assertEqual(my_obj.meta, framework.meta)         self.assertEqual(my_obj.charm_dir, framework.charm_dir) +    def test_ban_concurrent_frameworks(self):+        framework = self.create_framework()+        with self.assertRaises(OperationalError):+            framework_copy = self.create_framework()

This is a case where I would actually look at the exception string instead of the type. We don't care as much about "OperationalError" as we do care about "database is locked". It's also in a test, so that conversation we had doesn't apply. If that error message changes we'll know it because we're always verifying it, and at the right time (testing, not in production).

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Rework DB lock handling

 def __str__(self):  class SQLiteStorage: +    DB_LOCK_TIMEOUT = timedelta(hours=1).total_seconds()+     def __init__(self, filename):-        self._db = sqlite3.connect(str(filename), isolation_level="EXCLUSIVE")+        # The isolation_level argument is set to None such that autocommit behavior of the sqlite3 module is disabled.+        self._db = sqlite3.connect(str(filename), isolation_level=None, timeout=self.DB_LOCK_TIMEOUT)         self._setup()      def _setup(self):-        c = self._db.execute("BEGIN")+        # Make sure that the database is locked until the connection is closed, not until the transaction ends.+        c = self._db.execute("PRAGMA locking_mode=EXCLUSIVE")+        c = self._db.execute("BEGIN EXCLUSIVE")

If I understand the SQLite documentation correctly, EXCLUSIVE shouldn't be required here if we have the pragma.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Rework DB lock handling

  from ops.framework import (     Framework, Handle, EventSource, EventsBase, EventBase, Object, PreCommitEvent, CommitEvent,-    NoSnapshotError, StoredState, StoredList, BoundStoredState, StoredStateData+    NoSnapshotError, StoredState, StoredList, BoundStoredState, StoredStateData, SQLiteStorage ) +from sqlite3 import OperationalError+  class TestFramework(unittest.TestCase):      def setUp(self):         self.tmpdir = Path(tempfile.mkdtemp())         self.addCleanup(shutil.rmtree, self.tmpdir) +        default_timeout = SQLiteStorage.DB_LOCK_TIMEOUT+

Nitpick: empty line is a bit unexpected here. That's very tightly related to the next few lines.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Rework DB lock handling

 def __str__(self):  class SQLiteStorage: +    DB_LOCK_TIMEOUT = timedelta(hours=1).total_seconds()

Sounds a bit better to keep that as a timedelta, and convert where necessary.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Rework DB lock handling

 def __str__(self):  class SQLiteStorage: +    DB_LOCK_TIMEOUT = timedelta(hours=1).total_seconds()+     def __init__(self, filename):-        self._db = sqlite3.connect(str(filename), isolation_level="EXCLUSIVE")+        # The isolation_level argument is set to None such that autocommit behavior of the sqlite3 module is disabled.+        self._db = sqlite3.connect(str(filename), isolation_level=None, timeout=self.DB_LOCK_TIMEOUT)         self._setup()      def _setup(self):-        c = self._db.execute("BEGIN")+        # Make sure that the database is locked until the connection is closed, not until the transaction ends.+        c = self._db.execute("PRAGMA locking_mode=EXCLUSIVE")+        c = self._db.execute("BEGIN EXCLUSIVE")         c.execute("SELECT count(name) FROM sqlite_master WHERE type='table' AND name='snapshot'")         if c.fetchone()[0] == 0:             # Keep in mind what might happen if the process dies somewhere below.             # The system must not be rendered permanently broken by that.             self._db.execute("CREATE TABLE snapshot (handle TEXT PRIMARY KEY, data BLOB)")             self._db.execute("CREATE TABLE notice (sequence INTEGER PRIMARY KEY AUTOINCREMENT, event_path TEXT, observer_path TEXT, method_name TEXT)")-            self._db.commit()+            self.commit()

Looks like a leftover change.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Update the LICENSE to be Apache 2.0.

+# Copyright 2019 Canonical Ltd.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+ from setuptools import setup +with open("README.md", "r") as fh:+    long_description = fh.read()  setup(     name="ops",     version="0.0.1",+    description="The Python library behind great charms",+    long_description=long_description,+    long_description_content_type="text/markdown",+    license='Apache-2.0',     url="https://github.com/canonical/operator",     packages=['ops'],+    classifiers=[+        'Development Status :: 4 - Beta',++        "Programming Language :: Python :: 3",+        "Programming Language :: Python :: 3.6",+        "Programming Language :: Python :: 3.7",+        "Programming Language :: Python :: 3.8",++        "License :: OSI Approved :: Apache Software License",

Strings are being double-quotes and single-quoted back and forth.

jameinel

comment created time in 9 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def test_setup_action_links(self):         action_hook = self.JUJU_CHARM_DIR / 'actions' / 'test'         self.assertTrue(action_hook.exists()) +    def test_collect_metrics_concurrent(self):+        indicator_file = self.JUJU_CHARM_DIR / 'indicator'

That's looks like a very complex test to verify something that is completely unrelated to the way we handle that hook. That's nothing about concurrency that is special for that hook, as I understand it. So what are we testing? That the framework is opened in exclusive mode? That should be done in that other PR, and we don't need threads/etc to check that, per today's call. Just open it twice, wait for the error.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def on_foo_bar_action(self, event):         self._state['observed_event_types'].append(type(event))         self._write_state() +    def on_collect_metrics(self, event):+        self._state['on_collect_metrics'].append(type(event))+        self._state['observed_event_types'].append(type(event))+        event.add_metrics({'foo': 'bar'}, {'dead': ' beef '})+        self._write_state()++    def on_ha_relation_departed(self, event):+        indicator_file = pathlib.Path(self._charm_config['INDICATOR_FILE'])+        indicator_file.touch()+        with open(self._state_file, 'w+') as state_fd:+            fcntl.flock(state_fd, fcntl.LOCK_EX)+            fcntl.flock(state_fd, fcntl.LOCK_UN)

How's that related to the metric collection?

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metric(self, metrics, labels=None):

I think we need some validation that keys/values are what we expect. If they contain commas, equal signs, spaces (maybe?), it'll create unexpected commands. What's the actual format accepted by juju for such data?

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 def network_get(self, endpoint_name, relation_id=None):             if 'relation not found' in str(e):                 raise RelationNotFoundError() from e             raise++    def add_metric(self, metrics, labels=None):+        cmd = ['add-metric']

Is the command actually add_metric and not add_metrics? I think add_metrics is more correct since it accepts a set. It's okay for the command itself to be different in this case, since the backend doesn't have to match one-to-one with commands run.

dshcherb

comment created time in 9 days

Pull request review commentcanonical/operator

Add support for collect-metrics events

 class LeaderSettingsChangedEvent(HookEvent):     pass  +class CollectMetricsEvent(HookEvent):++    def add_metrics(self, metrics, labels=None):+        self.framework.model._backend.add_metric(metrics, labels)

add_metrics vs. add_metric? Which one is right?

dshcherb

comment created time in 9 days

issue commentcanonical/operator

ActiveStatus() doesn't accept a message

@stub42 Yes, again, I know there are dozens of different interesting pieces of information that are put in the status. That's exactly why this is not a proper user interface or experience. As I said, I also realize that people do that because there's no better way.

Anyway, I see we're hitting a nerve and I the operator framework should feel great, and not liking it because of this simple issue would be a terrible outcome.

So here is what I suggest we do next:

  1. We'll support a message in the operator framework.
  2. We'll add support in juju for providing structured information to admins
  3. Juju itself will stop communicating the active message, so setting it becomes irrelevant, via the operator or otherwise.
barryprice

comment created time in 10 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):+        binding_list = self._data[binding_name]+        if binding_list is None:+            binding_list = self._data[binding_name] = []+            # A relation id can be None which means a binding is not associated with a relation.+            binding_list.append(Binding(binding_name, None, self._backend))+            for relation_id in self._backend.relation_ids(binding_name):+                binding_list.append(Binding(binding_name, relation_id, self._backend))+        return binding_list++    def _get_unique(self, binding_name, relation_id=None):+        if not isinstance(relation_id, (int, type(None))):+            raise ModelError(f'relation name {relation_id} must be int or None not {type(relation_id).__name__}')+        for binding in self[binding_name]:+            if binding.relation_id == relation_id:+                return binding+        raise ModelError(f'a Binding object named "{binding_name}" with a relation id "{relation_id}" does not exist')+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        self.relation_id = relation_id+        self._network_info = None+        self._backend = backend++    def _cache_network_info(self):+        try:+            if self.relation_id is None:+                self._network_info = self._backend.network_get(self.name)+            else:+                self._network_info = self._backend.network_get(self.name, self.relation_id)+        except RelationNotFoundError:+            # If a relation is dead, we can still get network info associated with an endpoint itself+            # not associated with a particular relation id.+            self._network_info = self._backend.network_get(self.name)++    @property+    def network_info(self):+        if not self._network_info:+            self._cache_network_info()+        return self._network_info++    @property+    def bind_address(self):+        return self.network_info['bind-addresses'][0]['addresses'][0]['value']++    @property+    def ingress_address(self):+        return self.network_info['ingress-addresses'][0]

Same.. doesn't it feel a bit strange to have both:

binding.network_info["ingress-addresses"][0]

and

binding.ingress_address

?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):+        binding_list = self._data[binding_name]+        if binding_list is None:+            binding_list = self._data[binding_name] = []+            # A relation id can be None which means a binding is not associated with a relation.+            binding_list.append(Binding(binding_name, None, self._backend))+            for relation_id in self._backend.relation_ids(binding_name):+                binding_list.append(Binding(binding_name, relation_id, self._backend))+        return binding_list++    def _get_unique(self, binding_name, relation_id=None):+        if not isinstance(relation_id, (int, type(None))):+            raise ModelError(f'relation name {relation_id} must be int or None not {type(relation_id).__name__}')+        for binding in self[binding_name]:+            if binding.relation_id == relation_id:+                return binding

What if it's not unique? This logic is significantly different from what we offer on get_relation.

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def action_log(self, message):     def action_fail(self, message=''):         self._run(f'action-fail', f"{message}") -    def network_get(self, endpoint_name, relation_id=None):-        """Return network info provided by network-get for a given endpoint.+    def network_get(self, binding_name, relation_id=None):

Name or key?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):+        binding_list = self._data[binding_name]+        if binding_list is None:+            binding_list = self._data[binding_name] = []+            # A relation id can be None which means a binding is not associated with a relation.+            binding_list.append(Binding(binding_name, None, self._backend))+            for relation_id in self._backend.relation_ids(binding_name):+                binding_list.append(Binding(binding_name, relation_id, self._backend))+        return binding_list++    def _get_unique(self, binding_name, relation_id=None):+        if not isinstance(relation_id, (int, type(None))):+            raise ModelError(f'relation name {relation_id} must be int or None not {type(relation_id).__name__}')+        for binding in self[binding_name]:+            if binding.relation_id == relation_id:+                return binding+        raise ModelError(f'a Binding object named "{binding_name}" with a relation id "{relation_id}" does not exist')+++class Binding:+    """A representation of a binding to a space."""++    def __init__(self, name, relation_id, backend):+        self.name = name+        self.relation_id = relation_id+        self._network_info = None+        self._backend = backend++    def _cache_network_info(self):+        try:+            if self.relation_id is None:+                self._network_info = self._backend.network_get(self.name)+            else:+                self._network_info = self._backend.network_get(self.name, self.relation_id)+        except RelationNotFoundError:+            # If a relation is dead, we can still get network info associated with an endpoint itself+            # not associated with a particular relation id.+            self._network_info = self._backend.network_get(self.name)++    @property+    def network_info(self):+        if not self._network_info:+            self._cache_network_info()+        return self._network_info++    @property+    def bind_address(self):+        return self.network_info['bind-addresses'][0]['addresses'][0]['value']

That's a pretty deep value just to find the default address. It also feels slightly unusual that this is a Binding with network_info["bind-addresses"][0]["addresses"] but bind_address is in the Binding and not in the network_info in the first place. We should probably talk for a moment about how to shape this object.

Meanwhile, where do I find docs about the format of the network-get result?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):+        binding_list = self._data[binding_name]+        if binding_list is None:+            binding_list = self._data[binding_name] = []+            # A relation id can be None which means a binding is not associated with a relation.+            binding_list.append(Binding(binding_name, None, self._backend))+            for relation_id in self._backend.relation_ids(binding_name):+                binding_list.append(Binding(binding_name, relation_id, self._backend))

This logic seems a bit suspect. Why is it returning all bindings for all relations established plus the one that isn't associated with any relation at all? Imagine you now have three bindings, for different networks, and a binding for no relation at all. How do we expect people to make a good use of this result?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):+        binding_list = self._data[binding_name]+        if binding_list is None:+            binding_list = self._data[binding_name] = []+            # A relation id can be None which means a binding is not associated with a relation.+            binding_list.append(Binding(binding_name, None, self._backend))+            for relation_id in self._backend.relation_ids(binding_name):+                binding_list.append(Binding(binding_name, relation_id, self._backend))+        return binding_list++    def _get_unique(self, binding_name, relation_id=None):+        if not isinstance(relation_id, (int, type(None))):

What's the benefit of this check given what follows?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):+        binding_list = self._data[binding_name]+        if binding_list is None:+            binding_list = self._data[binding_name] = []+            # A relation id can be None which means a binding is not associated with a relation.

This looks like documentation of Binding itself, so doesn't have to be here.

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):+        return key in self._data++    def __len__(self):+        return len(self._data)++    def __iter__(self):+        return iter(self._data)++    def __getitem__(self, binding_name):

Names or keys? :)

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.

Bindings names or binding keys?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def _get_unique(self, relation_name, relation_id=None):             raise TooManyRelatedAppsError(relation_name, num_related, 1)  +class BindingMapping(Mapping):+    """A map of binding names to lists of Binding instances."""++    def __init__(self, binding_names, backend):+        """+        binding_names -- Relation endpoint names and names of extra-bindings.+        backend -- An instance of a model backend implementation.+        """+        self._backend = backend+        self._data = {binding_name: None for binding_name in binding_names}++    def __contains__(self, key):

keys or names?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def get_relation(self, relation_name, relation_id=None):         """         return self.relations._get_unique(relation_name, relation_id) +    def get_binding(self, binding_key):+        """Get a network space binding.++        Providing a Relation instance is preferred because network info exposed in a Binding instance+        can be different depending on whether a relation is a cross-model relation or not.

This seems a bit confusing. If a space is bound to a relation, it shouldn't matter whether this relation is cross-model or not. What this should really be saying is whether the space is bound to an individual established relation or to all of the relations established against that endpoint, I suspect?

dshcherb

comment created time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def get_relation(self, relation_name, relation_id=None):         """         return self.relations._get_unique(relation_name, relation_id) +    def get_binding(self, binding_key):

Is it a binding_key or a binding_name? We should be consistent.

dshcherb

comment created time in 13 days

push eventcanonical/operator

mthaddon

commit sha c3c9224504d33f9264396d377c1c233ad21895f7

Multiple README improvements (#115)

view details

push time in 13 days

PR merged canonical/operator

Reviewers
Clarify how Juju events map to operator framework functions

There are some potentially controversial changes here, so happy to discuss here or in another forum if appropriate:

  • Switching the import path to avoid needing to add to sys.path before imports (avoids violating E402).
  • Shows how to publish to the charmstore as well as local deploy.
  • Clarifies that because the operator framework requires python 3.6 or greater it only supports charms for bionic or later.
+50 -19

3 comments

1 changed file

mthaddon

pr closed time in 13 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def get_unit(self, unit_name):     def get_app(self, app_name):         return self._cache.get(Application, app_name) -    @property-    def bindings(self):-        if self._bindings is None:-            binding_keys = list(self._meta.relations) + list(self._meta.extra_bindings)-            for endpoint_relations in self.relations.values():-                if endpoint_relations:-                    binding_keys.extend(endpoint_relations)-            self._bindings = BindingMapping(binding_keys, self._backend)-        return self._bindings+    def get_binding(self, binding_name):+        """Get a binding without specifying a relation id."""+        return self.bindings._get_unique(binding_name)++    def get_relation_binding(self, relation):+        """Get a binding for a specific relation.++        Using this method is preferred if a Relation instance can be provided because network info+        exposed in a Binding instance can be different depending on whether a relation is a cross-model relation or not.+        """+        return self.bindings._get_unique(relation.name, relation.id)

What's the rationale behind these methods? How an someone specify a different space for a relation depending on its ID? If that's impossible, why would we have any other interfaces besides the one that looks up a binding via its relation name?

dshcherb

comment created time in 13 days

push eventcanonical/operator

Dmitrii Shcherbakov

commit sha 5dafe293b9c382a0bcd7c306399f6d334bf26039

Add network-get support to ModelBackend (#108) Hook tool handling for network-get which may be a base for further endpoint modeling in the model.

view details

push time in 13 days

PR merged canonical/operator

Reviewers
Add network-get support to ModelBackend

Hook tool handling for network-get which may be a base for further endpoint modeling in the model.

+69 -0

0 comment

2 changed files

dshcherb

pr closed time in 13 days

push eventcanonical/operator

Dmitrii Shcherbakov

commit sha 45f5ffe8875b7f3b54d420f419c1ff9fb83aed52

Fix .app population for single-unit peer relations (#107) Currently .app property is None when a peer relation contains only one unit (relation-list returns an empty list). This change addresses the issue.

view details

push time in 13 days

PR merged canonical/operator

Reviewers
Fix .app population for single-unit peer relations

Currently .app property is None when a peer relation contains only one unit (relation-list returns an empty list). This change addresses the issue.

+70 -75

0 comment

2 changed files

dshcherb

pr closed time in 13 days

Pull request review commentcanonical/operator

Add a makefile showing how to run lint and tests, and fix lint

 def __lt__(self, other):         elif not isinstance(other, JujuVersion):             raise RuntimeError(f'cannot compare Juju version "{self}" with "{other}"') -        if self.major != other.major:-            return self.major < other.major-        elif self.minor != other.minor:-            return self.minor < other.minor-        elif self.tag != other.tag:-            if not self.tag:-                return False-            elif not other.tag:-                return True-            return self.tag < other.tag-        elif self.patch != other.patch:-            return self.patch < other.patch-        elif self.build != other.build:-            return self.build < other.build+        for prop in ['major', 'minor', 'tag', 'patch', 'build']:+            lt = self._less_than_property(other, prop)+            if lt is not None:+                return lt

The original version had 14 lines.. the new version has 14 lines, except it jumps across two different functions back and forth repeatedly, uses more indirection, and is harder to read. It's also a bit of a lie as it pretends to be general until it's not, by special casing the handling of "tag".

I'd recommend preserving the existing function.

mthaddon

comment created time in 17 days

PR closed canonical/operator

Add optional way for observing an event

Be explicit about an alternative and DRY-er way of observing an event.

+24 -3

6 comments

1 changed file

relaxdiego

pr closed time in 17 days

pull request commentcanonical/operator

Add optional way for observing an event

Tentatively closing based on the above comment from two days ago. Please feel free to comment further or reopen if it feels undue.

relaxdiego

comment created time in 17 days

Pull request review commentcanonical/operator

Clarify how Juju events map to operator framework functions

 Your charm directory should have the following overall structure: |   +-- charm.py +-- hooks/     +-- install -> ../src/charm.py+    +-- start -> ../src/charm.py  # for k8s charms per below ``` -The `mod/` directory will contain the operator framework dependency as a git+The `mod/` directory should contain the operator framework dependency as a git submodule:  ``` git submodule add https://github.com/canonical/operator mod/operator ``` -Other dependencies included as git submodules can be added there as well.--The `lib/` directory will then contain symlinks to subdirectories of your-submodule dependencies to enable them to be imported into the charm:+Then symlink from the git submodule for the operator framework into the `lib/`+directory of your charm so it can be imported at run time:  ``` ln -s ../mod/operator/ops lib/ops ``` +Other dependencies included as git submodules can be added in the `mod/`+directory and symlinked into `lib/` as well.++You can sync subsequent changes from the framework and other submodule+dependencies by running:++```+git submodule update+```++Those cloning and checking out the source for your charm for the first time+will need to run:++```+git submodule update --init+```+ Your `src/charm.py` is the entry point for your charm logic. It should be set-to executable and use Python 3.6 or greater. At a minimum, it needs to define a-subclass of `CharmBase` and pass that into the framework's `main` function:+to executable and use Python 3.6 or greater (as such, operator charms can only

How about just keeping the way it is right now? We require Python 3.6+, and that's it. Everyone can figure their own environment.

mthaddon

comment created time in 17 days

Pull request review commentcanonical/operator

Clarify how Juju events map to operator framework functions

 if __name__ == "__main__":     main(MyCharm) ``` -This charm does nothing, though, so you'll typically want to observe some Juju-events, such as `start`:+This charm does nothing, because the `MyCharm` class passed to the operator+framework's `main` function is empty. Functionality can be added to the charm+by instructing it to observe particular Juju events when the `MyCharm` object+is initialized. In the example below we're observing the `start` event. The+name of the event is a function of `self.on`, so in this case `self.on.start`+is passed to `self.framework.observe`. This will look for method named+`on_start` when that Juju event is triggered.  ```python class MyCharm(CharmBase):     def __init__(self, *args):         super().__init__(*args)-        self.framework.observe(self.on.start, self.on_start)+        self.framework.observe(self.on.start, self)       def on_start(self, event):-        # Handle the event here.+        # Handle the start event here. ```  Every standard event in Juju may be observed that way, and you can also easily-define your own events in your custom types.+define your own events in your custom types. For example, to observe the+`config-changed` Juju event:

Can we please drop this extra example and respective introductory sentence? This was added to document the the shorthand notation, but it's now simply repeating the same exact example described above.

mthaddon

comment created time in 17 days

Pull request review commentcanonical/operator

Clarify how Juju events map to operator framework functions

 if __name__ == "__main__":     main(MyCharm) ``` -This charm does nothing, though, so you'll typically want to observe some Juju-events, such as `start`:+This charm does nothing, because the `MyCharm` class passed to the operator+framework's `main` function is empty. Functionality can be added to the charm+by instructing it to observe particular Juju events when the `MyCharm` object+is initialized. In the example below we're observing the `start` event. The+name of the event is a function of `self.on`, so in this case `self.on.start`+is passed to `self.framework.observe`. This will look for method named+`on_start` when that Juju event is triggered.  ```python class MyCharm(CharmBase):     def __init__(self, *args):         super().__init__(*args)-        self.framework.observe(self.on.start, self.on_start)+        self.framework.observe(self.on.start, self)       def on_start(self, event):-        # Handle the event here.+        # Handle the start event here. ```  Every standard event in Juju may be observed that way, and you can also easily-define your own events in your custom types.+define your own events in your custom types. For example, to observe the+`config-changed` Juju event: -The `hooks/` directory will then contain symlinks to your `src/charm.py` entry+```python+class MyCharm(CharmBase):+    def __init__(self, *args):+        super().__init__(*args)+        self.framework.observe(self.on.start, self)+        self.framework.observe(self.on.config_changed, self)++     def on_start(self, event):+        # Handle the start event here.++     def on_config_changed(self, event):+        # Handle the config-changed event here.+```++The `hooks/` directory must contain a symlink to your `src/charm.py` entry point so that Juju can call it. You only need to set up the `hooks/install` link (`hooks/start` for K8s charms, until [lp#1854635](https://bugs.launchpad.net/juju/+bug/1854635)

Let's please not hold @mthaddon's collaboration based on what else might be improved on the README. Those can come in a separate PR.

mthaddon

comment created time in 17 days

Pull request review commentcanonical/operator

Add bindings to network spaces to the model

 def get_unit(self, unit_name):     def get_app(self, app_name):         return self._cache.get(Application, app_name) +    @property+    def bindings(self):+        if self._bindings is None:+            binding_keys = list(self._meta.relations) + list(self._meta.extra_bindings)+            for endpoint_relations in self.relations.values():+                if endpoint_relations:+                    binding_keys.extend(endpoint_relations)+            self._bindings = BindingMapping(binding_keys, self._backend)

Why is this diverging from the other patterns we have?

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Add network-get support to ModelBackend

 def test_storage_tool_errors(self):                 run()             self.assertEqual(fake_script_calls(self, clear=True), calls) +    def test_network_get(self):+        network_get_out = ('{"bind-addresses":[{"mac-address":"","interface-name":"",'+                           '"addresses":[{"hostname":"","value":"192.0.2.2","cidr":""}]}]'+                           ',"egress-subnets":["192.0.2.2/32"],"ingress-addresses":["192.0.2.2"]}')

Not the usual quoting here and elsewhere in the PR, plus point by @chipaca is still open

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Add network-get support to ModelBackend

 def function_log(self, message):      def function_fail(self, message=''):         self._run(f'{self._function_cmd_prefix}-fail', f"{message}")++    def network_get(self, endpoint_name, relation_id=None):+        """Return network info provided by network-get for a given endpoint.++        endpoint_name -- A name of an endpoint (relation name or extra-binding name).+        relation_id -- An optional relation id to get network info for.+        """+        cmd = ['network-get', endpoint_name]+        if relation_id is not None:+            cmd.extend(['-r', str(relation_id)])+        try:+            return self._run(*cmd, return_output=True, use_json=True)+        except ModelError as e:+            if 'relation not found' in str(e):+                raise RelationNotFoundError() from e

Sorry, my bad. I forgot that this was a case we already agreed on.

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def __getitem__(self, relation_name):         relation_list = self._data[relation_name]         if relation_list is None:             relation_list = self._data[relation_name] = []-            relation_role = self._relations_meta[relation_name].role+            our_app = self._our_unit.app if self._relations_meta[relation_name].role == 'peers' else None             for relation_id in self._backend.relation_ids(relation_name):-                relation_list.append(Relation(relation_name, relation_id, relation_role, self._our_unit, self._backend, self._cache))+                relation_list.append(Relation(relation_name, relation_id, our_app, self._our_unit, self._backend, self._cache))         return relation_list   class Relation:-    def __init__(self, relation_name, relation_id, role, our_unit, backend, cache):+    def __init__(self, relation_name, relation_id, our_app, our_unit, backend, cache):         self.name = relation_name         self.id = relation_id-        self.role = role         self.app = None         self.units = set()          # For peer relations, both the remote and the local app are the same.-        if self.role == 'peers':-            self.app = our_unit.app+        if our_app is not None:+            self.app = our_app

Alternatively, we might have relation_app as a parameter, instead of our_app.

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def __getitem__(self, relation_name):         relation_list = self._data[relation_name]         if relation_list is None:             relation_list = self._data[relation_name] = []-            relation_role = self._relations_meta[relation_name].role+            our_app = self._our_unit.app if self._relations_meta[relation_name].role == 'peers' else None             for relation_id in self._backend.relation_ids(relation_name):-                relation_list.append(Relation(relation_name, relation_id, relation_role, self._our_unit, self._backend, self._cache))+                relation_list.append(Relation(relation_name, relation_id, our_app, self._our_unit, self._backend, self._cache))         return relation_list   class Relation:-    def __init__(self, relation_name, relation_id, role, our_unit, backend, cache):+    def __init__(self, relation_name, relation_id, our_app, our_unit, backend, cache):         self.name = relation_name         self.id = relation_id-        self.role = role         self.app = None         self.units = set()          # For peer relations, both the remote and the local app are the same.-        if self.role == 'peers':-            self.app = our_unit.app+        if our_app is not None:+            self.app = our_app

Instead of

self.app = None
(...)
if our_app is not None:
    self.app = our_app

This can be simply:

self.app = our_app

But! This shows a real problem in my recommendation: this is awkward because this is not our app. This is the relations app.. it makes no sense to tell the relation what the app is only sometimes.

Your suggestion of having an is_peer attribute seems more reasonable now. Sorry for taking you into this misleading path.

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def get_relation(self, relation_name, relation_id=None):                     return relation             else:                 # The relation may be dead, but it is not forgotten.-                return Relation(relation_name, relation_id, self._meta.relations[relation_name].role, self.unit, self._backend, self._cache)+                our_app = self.app if self._meta.relations[relation_name].role == 'peers' else None

The straight version of this is much easier to read:

if self._meta.relations[relation_name].role == 'peers'
    our_app = self.app
else:
    our_app = None

Also, this logic seems to be distributed in two different places without benefit. Instead of doing it here, and then having a new _meta attribute, and then handing the underlying relations value down into relations which is also stored, let's move all of this into the relations value itself. We might call an underlying private method in the RelationMapping value, but instead of:

model.get_relation(name, id)

I think we can simply have:

model.relations.get(name, id)

With id still defaulting to None. Seems nice and follows a recognized pattern.

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def __getitem__(self, key): class RelationMapping(Mapping):     """Map of relation names to lists of Relation instances.""" -    def __init__(self, relation_names, our_unit, backend, cache):+    def __init__(self, relations_meta, our_unit, backend, cache):+        self._relations_meta = relations_meta         self._our_unit = our_unit         self._backend = backend         self._cache = cache-        self._data = {relation_name: None for relation_name in relation_names}+        self._data = {relation_name: None for relation_name in list(relations_meta)}

Why transforming relations_meta into a list before iterating over it?

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def __getitem__(self, relation_name):         relation_list = self._data[relation_name]         if relation_list is None:             relation_list = self._data[relation_name] = []-            relation_role = self._relations_meta[relation_name].role+            our_app = self._our_unit.app if self._relations_meta[relation_name].role == 'peers' else None

Related to the first comment. I suggest avoiding ternary operators in such cases.

dshcherb

comment created time in 17 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def __getitem__(self, key): class RelationMapping(Mapping):     """Map of relation names to lists of Relation instances.""" -    def __init__(self, relation_names, our_unit, backend, cache):+    def __init__(self, relations_meta, our_unit, backend, cache):+        self._relations_meta = relations_meta

Same here.. we're also changing from getting some data we just iterate over to build the type into storing part of the meta map. Yet, we don't actually use the map much. It'd be faster and nice to use if we did something like this I think:

self._peers = set()
for name, data in relations_meta:
    if data.role == 'peers':
        self._peers.add(name)

Now name in self._peers is all we need.

dshcherb

comment created time in 17 days

PR closed canonical/operator

Allow ActiveStatus to set an output message

This potentially resolves #126

+7 -14

1 comment

2 changed files

barryprice

pr closed time in 19 days

pull request commentcanonical/operator

Allow ActiveStatus to set an output message

Per discussion in #126, I think we need to have a more clear idea of how and when we want to organize things in that area before we make that promise in this new API. I'll keep the bug open to track the progress of those discussions.

barryprice

comment created time in 19 days

PR closed canonical/operator

Add optional suggested status to ModelError

The error classes in the MySQLClient component were very similar to existing errors in the framework save for having a suggested status. This seems like a pattern we want to recommend and the errors and suggested statuses seem reusable.

+20 -5

1 comment

1 changed file

johnsca

pr closed time in 19 days

pull request commentcanonical/operator

Add optional suggested status to ModelError

Closing this for now based on the above feedback.

johnsca

comment created time in 19 days

Pull request review commentcanonical/operator

Rename functions back to actions

 def main(charm_class):         # the hook setup code is not triggered on config-changed.         # 'start' event is included as Juju does not fire the install event for K8s charms (see LP: #1854635).         if juju_event_name in ('install', 'start', 'upgrade_charm') or juju_event_name.endswith('_storage_attached'):-            _setup_event_links(charm_dir, charm, functions_dir)+            _setup_event_links(charm_dir, charm)

Yes, indeed!

dshcherb

comment created time in 19 days

issue commentcanonical/operator

ActiveStatus() doesn't accept a message

It seems wrong to take away functionality that is currently provided by Juju and relied upon by many charms

That's exactly the rationale applying here. The operator framework has never offered this functionality, so this is the time to discuss how we want charms to be designed. The conversation here is not about how things used to work, but how we want them to work.

In those terms, every single time I've had this conversation so far, people came up with completely different reasons why they want the message in place. How many nodes are in place, whether the node is slave or master, whether it's degraded or fully working, etc. This smells like bad design on our side. What if you want to say both whether it's slave or master, and whether it's degraded or not, and also the node count?

We are the designers. Please join me making juju fantastic.

barryprice

comment created time in 19 days

Pull request review commentcanonical/operator

Add network-get support to ModelBackend

 def function_log(self, message):      def function_fail(self, message=''):         self._run(f'{self._function_cmd_prefix}-fail', f"{message}")++    def network_get(self, endpoint_name, relation_id=None):+        """Return network info provided by network-get for a given endpoint.++        endpoint_name -- A name of an endpoint (relation name or extra-binding name).+        relation_id -- An optional relation id to get network info for.+        """+        cmd = ['network-get', endpoint_name]+        if relation_id is not None:+            cmd.extend(['-r', str(relation_id)])+        try:+            return self._run(*cmd, return_output=True, use_json=True)+        except ModelError as e:+            if 'relation not found' in str(e):+                raise RelationNotFoundError() from e

Digging into error messages is a bad idea as we discussed in similar cases before.

Why is that necessary?

dshcherb

comment created time in 19 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def test_unit_relation_data(self):         ])      def test_remote_app_relation_data(self):-        self.backend = ops.model.ModelBackend()-        meta = ops.charm.CharmMeta()-        meta.relations = {'db0': None, 'db1': None, 'db2': None}-        self.model = ops.model.Model('myapp/0', meta, self.backend)

Why are these completely irrelevant, and copy & pasted so much? Error?

dshcherb

comment created time in 19 days

Pull request review commentcanonical/operator

Fix .app population for single-unit peer relations

 def __getitem__(self, relation_name):         relation_list = self._data[relation_name]         if relation_list is None:             relation_list = self._data[relation_name] = []+            relation_role = self._relations_meta[relation_name].role             for relation_id in self._backend.relation_ids(relation_name):-                relation_list.append(Relation(relation_name, relation_id, self._our_unit, self._backend, self._cache))+                relation_list.append(Relation(relation_name, relation_id, relation_role, self._our_unit, self._backend, self._cache))         return relation_list   class Relation:-    def __init__(self, relation_name, relation_id, our_unit, backend, cache):+    def __init__(self, relation_name, relation_id, role, our_unit, backend, cache):         self.name = relation_name         self.id = relation_id+        self.role = role

Having "peers", "requires", and "provides" here doesn't seem great. We might use the same logic as our_unit and have our_app here, avoiding the use of such strings and having a "role" at all in this place for now.

dshcherb

comment created time in 19 days

MemberEvent

issue commentcanonical/operator

ActiveStatus() doesn't accept a message

@sajoupa @barryprice I think what you're both saying is that communicating information to the administrator can be very useful, which is of course true and vital for proper operation of systems. What I found out while talking to people is that this message is being used to carry all kinds of information. Some examples were provided above:

the Postgres charm displays whether it's running as master or slave, the Livepatch charm displays the current patch status, and so on

The purpose of this field is to say whether the unit is waiting for something, or functioning normally. Not as a conveyor of arbitrary data for the charm, as this would be far from ideal. My recommendation is to preserve the active status empty, and eventually disable it in juju altogether so that it doesn't show anything anymore (it would not break, for compatibility, but it wouldn't show anything). At the same time, let's work with the juju team to have a proper way to feed such valuable information to the administrator in a structured way.

barryprice

comment created time in 19 days

issue closedgo-yaml/yaml

Question: Is it correct to assume that MappingNode key/value children share the same hierarchy level?

Not sure if an issue is the best place to ask a question, but could not find a better way.

I'm currently working on a project that needs to use the new Node intermediate interface. While playing with it to understand its internals, I found something that looked a bit counter-intuitive to me, but that could be well part of the yaml spec, so I just want to confirm my assumption is correct.

So, given this file:

foo: 5
bar: 6

What I get is a top level DocumentNode, which is composed of a single child MappingNode. All good until here. The unexpected was when I found that the children are 4 ScalarNode in the sequence of key-value-key-value (foo, 5, bar, 6), as if they were inside a SequenceNode.

Can I safely assume that all the even indexes of MappingNode children are keys, whilst the odd indexes are the values?

Thank you.

closed time in 24 days

Lantero

issue commentgo-yaml/yaml

Question: Is it correct to assume that MappingNode key/value children share the same hierarchy level?

That's exactly right. This preserves the ordering and allows a more compact and fast representation. It won't change in v3 as that would break the API, and most likely won't change in v4+ either. Sorry that it's not yet documented properly.

Lantero

comment created time in 24 days

created taggo-yaml/yaml

tagv2.2.8

YAML support for the Go language.

created time in a month

issue closedgo-yaml/yaml

Consistent marshaling of generated code such as protobuf would be helpful

I am using protobufs extensively, but protobuf code generation in go does not specify how to handle YAML marshaling. In this case, I would like to pass the JSON tag to the YAML parser to ensure that the name matches the proto definition itself to ensure consistency across various platforms.

ex.)

type InstanceGroup struct {
	Name              string   'protobuf:"bytes,1,opt,name=name" json:"name,omitempty"'
	NumberOfInstances int32    'protobuf:"varint,2,opt,name=numberOfInstances" json:"numberOfInstances,omitempty"`
}

Ideally, I would be able to feed alternative tags to use for marshaling.

closed time in a month

ebilling

issue commentgo-yaml/yaml

Consistent marshaling of generated code such as protobuf would be helpful

I can understand the desire, but JSON is not YAML, and the json package is not the yaml package either. There are several things that one needs to do to make such a parser work correctly and as expected, and I'm not keen on having to forever maintain all of the json package behavior inside the yaml package, and all the awkward edge cases that this will create.

With that said, if you want the behavior of the json package as a YAML file, it's trivial to use the json package to create a generic map of the data, and then ask the yaml package to marshal it out for you.

ebilling

comment created time in a month

issue commentgo-yaml/yaml

Curious list formatting when using indent of 3

Okay, I've seen a few more files with awkward formatting and I'm convinced you're right.

I'll change v3 soon to avoid that wart. Thanks for the feedback.

manute

comment created time in a month

issue closedgo-yaml/yaml

Mark v3 branch as default.

This is mainly a request and discussion proposal: It has been around a year now since v3 is out. Is v3 not enough stable to mark it as default branch rather than v2?

closed time in a month

willemavjc

issue commentgo-yaml/yaml

Mark v3 branch as default.

Actually, we already have #487 for this. Closing this one.

willemavjc

comment created time in a month

issue commentgo-yaml/yaml

Please tag a 3.0.0 release

The time is coming. I want to review a few last details before tagging it, but we're close.

thallgren

comment created time in a month

issue commentgo-yaml/yaml

Question: What to use instead of MapSlice and map[string]interface{} in v3?

That's a valid concern, thanks for the input.

I think the correct solution here is to implement something like Node.Encode, which is the opposite of Node.Decode: it would accept an interface{}, and make the Node represent the Go value provided. This way it would be trivial to implement even MapSlice itself again locally, if that's desired, or any similar variation that would fit the problem at hand.

vsapronov

comment created time in a month

issue closedgo-yaml/yaml

Force selected strings to marshal as string enclosed with double quotes

Hello, I have a requirement where a mapping with key "key" and value as string like 12:34 should be marshalled as key: "12.34". I searched for all possible documentation but could not find it. Could you please suggest how to implement this requirement. What I tried while debugging is changing the value of "canUsePlain" in encode.go at runtime to false and it worked as expected. But there is no certain way to enforce this behavior for a selected property that I could find.

closed time in a month

sachintaksande

issue commentgo-yaml/yaml

Force selected strings to marshal as string enclosed with double quotes

Marshaling will not quote values unless they change their meaning when unquoted. At the moment if you want to force this, you can either use yaml.Node in v3 or define an Unmarshaler interface that returns a Node with the custom styling.

sachintaksande

comment created time in a month

issue closedgo-yaml/yaml

DOS-style newlines not allowed in v3, allowed in v2

package main                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  

import (
        "fmt"

        "gopkg.in/yaml.v2"
)

type Yerson struct {
        Provider struct {
                RootURL string `yaml:"rootURL"`
        } `yaml:"provider"`
}

func main() {
        var p1 Yerson
        err := yaml.Unmarshal([]byte("---\r\nprovider:\r\n  rootURL: abc\r\n"), &p1)
        if err != nil {
                fmt.Printf("err: %v\n", err)
                return
        }
        fmt.Printf("%#v\n", p1)
}

The above works correctly, but changing v2 to v3 causes it to fail with

err: yaml: line 3: mapping values are not allowed in this context

This seems to violate YAML-1.2 section 5.4.

closed time in a month

djmitche

issue commentgo-yaml/yaml

DOS-style newlines not allowed in v3, allowed in v2

It seems to work fine here, and also in the playground:

  • https://play.golang.org/p/0SPlsdOXaL9

We also have unit tests covering this since at least last july (674ba3e).

Maybe you had an old branch?

djmitche

comment created time in a month

issue closedgo-yaml/yaml

Special character swap error

` package main

import ( "fmt" "io/ioutil" "log"

"gopkg.in/yaml.v2"

)

//Nginx nginx 配置 type Nginx struct { Port int yaml:"Port" LogPath string yaml:"LogPath" Path string yaml:"Path" }

//Config 系统配置配置 type Config struct { Name string yaml:"SiteName" Addr string yaml:"SiteAddr" HTTPS bool yaml:"Https" SiteNginx []Nginx yaml:"Nginx" }

func main() { var setting Config config, err := ioutil.ReadFile("./test.yaml") if err != nil { fmt.Print(err) } yaml.Unmarshal(config, &setting) data, err := yaml.Marshal(&setting) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(data)) ioutil.WriteFile("./out.yaml", data, 0777) } `

Raw data:

SiteName: seeta SiteAddr: BeiJing Https: true Nginx: - Port: 443 LogPath: > "/var/log//nginx/nginx.log" Path: "/opt/nginx/"

Output after execution

` SiteName: seeta SiteAddr: BeiJing Https: true Nginx:

  • Port: 443 LogPath: | "/var/log//nginx/nginx.log" Path: /opt/nginx/ ` the right arrow becomes vertical "> " ==> "|"

============================================= region: !region_component ======> region: ""

closed time in a month

qimingxingyuliyang

issue commentgo-yaml/yaml

Special character swap error

Yes, it's not a "character swap". Your data is exactly the same. What you are seeing is that the formatting of the output is not the same as the input, and that's impossible in this case since there's no information in your structure that would allow preserving the output. If you want to make the output closer to the input, your best bet is looking into yaml.Node in v3, but even that will not preserve 100% of the formatting in the input.

qimingxingyuliyang

comment created time in a month

more