profile
viewpoint

banker/mongulator 69

Try MongoDB! A simple MongoDB shell with tutorial that runs in the web browser.

dcrosta/dabble 25

Simple A/B testing framework for Python

dcrosta/cookbook-preferred 9

Chef LWRP to dynamically set attributes.

brettlangdon/ferrite 3

A very fast kyoto cabinet powered memcache interface http proxy caching server.

dcrosta/browsershame 3

Web browsers use too many resources -- shame on you!

dcrosta/cookbook-disco 2

Chef Cookbook to install the Disco map-reduce framework

dcrosta/asyncmongo 1

An asynchronous library for accessing mongo with tornado.ioloop

dcrosta/cacheorbust 1

Read-behind HTTP API cache with memcached protocol support

dcrosta/chef 1

A systems integration framework, built to bring the benefits of configuration management to your entire infrastructure.

Pull request review commenttox-dev/tox-docker

Randomize container names to avoid conflcits

 def tox_runtest_pre(venv):  # noqa: C901                 healthcheck=healthcheck,                 labels={"tox_docker_container_name": container_name},                 links=links,-                name=container_name,+                name=container_name + f"{random.randint(0, 100000)}",

To make the names both "more random" and predictable length (maybe this is just my OCD speaking), you could do:

                name=f"{container_name}_{random.randint(100000, 999999)}",
chaitu-tk

comment created time in 7 days

PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Randomize container names to avoid conflcits

 def tox_runtest_pre(venv):  # noqa: C901                 healthcheck=healthcheck,                 labels={"tox_docker_container_name": container_name},                 links=links,-                name=container_name,+                name=container_name + f"{random.randint(0, 100000)}",

Since you're using f-strings anyway, might be clearer to do

                name=f"{container_name}_{random.randint(0, 100000)}",
chaitu-tk

comment created time in 7 days

PullRequestReviewEvent

issue commenttox-dev/tox-docker

Document using port 0 for host port to let docker assign an available high port

Hm, so this is when you don't want to expose all ports, but also don't want to pick a particular port? Yeah, that seems reasonable.

thenewguy

comment created time in 2 months

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from typing import Dict, List++from tox.config.cli.parser import ToxParser+from tox.config.main import Config+from tox.config.sets import ConfigSet+from tox.config.types import EnvList+from tox.execute.api import Outcome+from tox.plugin import impl+from tox.tox_env.api import ToxEnv++from tox_docker.config import ContainerConfig, RunningContainers+from tox_docker.plugin import (+    docker_health_check,+    docker_pull,+    docker_run,+    get_env_vars,+    HealthCheckFailed,+    stop_containers,+)+from tox_docker.tox4.config import (+    discover_container_configs,+    EnvRunningContainers,+    parse_container_config,+)+from tox_docker.tox4.log import log++CONTAINER_CONFIGS: Dict[str, ContainerConfig] = {}+ENV_CONTAINERS: EnvRunningContainers = {}+++@impl+def tox_add_core_config(core_conf: ConfigSet, config: Config) -> None:+    container_config_names = discover_container_configs(config)++    # validate command line options+    for container_name in config.options.docker_dont_stop:+        if container_name not in container_config_names:+            raise ValueError(+                f"Container {container_name!r} not found (from --docker-dont-stop)"+            )++    for container_name in container_config_names:+        CONTAINER_CONFIGS[container_name] = parse_container_config(+            config, container_name, container_config_names+        )+++@impl+def tox_before_run_commands(tox_env: ToxEnv) -> None:+    tox_env.conf.add_config(+        keys=["docker"],+        of_type=EnvList,+        default=EnvList([]),+        desc="docker image configs to load",+        post_process=list,  # type: ignore+    )++    container_names = tox_env.conf["docker"]+    env_container_configs = []++    seen = set()+    for container_name in container_names:+        if container_name not in CONTAINER_CONFIGS:+            raise ValueError(f"Missing [docker:{container_name}] in tox.ini")+        if container_name in seen:+            raise ValueError(f"Container {container_name!r} specified more than once")+        seen.add(container_name)+        env_container_configs.append(CONTAINER_CONFIGS[container_name])++    for container_config in env_container_configs:+        docker_pull(container_config, log)

I expect many folks will not upgrade immediately, so I'm going to support both versions for some time.

dcrosta

comment created time in 2 months

PullRequestReviewEvent

push eventtox-dev/tox-docker

Dan Crosta

commit sha efab3fe161f1f6bf2a9baab0a707f579a34c20a5

clean up logging mess

view details

Dan Crosta

commit sha 062c07edb8849fc13b5b03642b1ca632c6d740ad

move custom config handling to tox_add_env_config hook

view details

push time in 2 months

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from typing import Dict, List++from tox.config.cli.parser import ToxParser+from tox.config.main import Config+from tox.config.sets import ConfigSet+from tox.config.types import EnvList+from tox.execute.api import Outcome+from tox.plugin import impl+from tox.tox_env.api import ToxEnv++from tox_docker.config import ContainerConfig, RunningContainers+from tox_docker.plugin import (+    docker_health_check,+    docker_pull,+    docker_run,+    get_env_vars,+    HealthCheckFailed,+    stop_containers,+)+from tox_docker.tox4.config import (+    discover_container_configs,+    EnvRunningContainers,+    parse_container_config,+)+from tox_docker.tox4.log import log++CONTAINER_CONFIGS: Dict[str, ContainerConfig] = {}+ENV_CONTAINERS: EnvRunningContainers = {}+++@impl+def tox_add_core_config(core_conf: ConfigSet, config: Config) -> None:+    container_config_names = discover_container_configs(config)++    # validate command line options+    for container_name in config.options.docker_dont_stop:+        if container_name not in container_config_names:+            raise ValueError(+                f"Container {container_name!r} not found (from --docker-dont-stop)"+            )++    for container_name in container_config_names:+        CONTAINER_CONFIGS[container_name] = parse_container_config(+            config, container_name, container_config_names+        )+++@impl+def tox_before_run_commands(tox_env: ToxEnv) -> None:+    tox_env.conf.add_config(+        keys=["docker"],+        of_type=EnvList,+        default=EnvList([]),+        desc="docker image configs to load",+        post_process=list,  # type: ignore+    )++    container_names = tox_env.conf["docker"]+    env_container_configs = []++    seen = set()+    for container_name in container_names:+        if container_name not in CONTAINER_CONFIGS:+            raise ValueError(f"Missing [docker:{container_name}] in tox.ini")+        if container_name in seen:+            raise ValueError(f"Container {container_name!r} specified more than once")+        seen.add(container_name)+        env_container_configs.append(CONTAINER_CONFIGS[container_name])++    for container_config in env_container_configs:+        docker_pull(container_config, log)

It won't pull if it's already local -- just as the docker pull CLI command won't.

This can't be moved to the config set, it needs to support tox3. The things in tox_docker/ but not in the tox3 or tox4 sub-packages are "generic" code to support both versions.

dcrosta

comment created time in 2 months

PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from logging import Logger+from typing import Optional++from tox.report import LOGGER, ToxHandler++_HANDLER = None+++def _get_handler() -> ToxHandler:+    global _HANDLER+    if _HANDLER is None:+        logger: Optional[Logger] = LOGGER+        while logger:+            for handler in logger.handlers:+                if isinstance(handler, ToxHandler):+                    _HANDLER = handler+                    logger = None+                    break+            else:+                logger = logger.parent if logger.propagate else None++    assert _HANDLER is not None

my mistake -- closing this thread per https://github.com/tox-dev/tox-docker/pull/113#discussion_r720482493

dcrosta

comment created time in 2 months

PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from logging import Logger+from typing import Optional++from tox.report import LOGGER, ToxHandler++_HANDLER = None+++def _get_handler() -> ToxHandler:+    global _HANDLER+    if _HANDLER is None:+        logger: Optional[Logger] = LOGGER+        while logger:+            for handler in logger.handlers:+                if isinstance(handler, ToxHandler):+                    _HANDLER = handler+                    logger = None+                    break+            else:+                logger = logger.parent if logger.propagate else None++    assert _HANDLER is not None+    return _HANDLER+++def log(line: str) -> None:+    with _get_handler().with_context("docker"):

hm, I could swear when I first wrote this, just using a logging.warning() didn't print with the right format. it works now, so will clean this mess up.

dcrosta

comment created time in 2 months

PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from logging import Logger+from typing import Optional++from tox.report import LOGGER, ToxHandler++_HANDLER = None+++def _get_handler() -> ToxHandler:+    global _HANDLER+    if _HANDLER is None:+        logger: Optional[Logger] = LOGGER+        while logger:+            for handler in logger.handlers:+                if isinstance(handler, ToxHandler):+                    _HANDLER = handler+                    logger = None+                    break+            else:+                logger = logger.parent if logger.propagate else None++    assert _HANDLER is not None

I couldn't figure how else to produce output in a style that's consistent with what tox itself prints.

dcrosta

comment created time in 2 months

PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from typing import Dict, List++from tox.config.cli.parser import ToxParser+from tox.config.main import Config+from tox.config.sets import ConfigSet+from tox.config.types import EnvList+from tox.execute.api import Outcome+from tox.plugin import impl+from tox.tox_env.api import ToxEnv++from tox_docker.config import ContainerConfig, RunningContainers+from tox_docker.plugin import (+    docker_health_check,+    docker_pull,+    docker_run,+    get_env_vars,+    HealthCheckFailed,+    stop_containers,+)+from tox_docker.tox4.config import (+    discover_container_configs,+    EnvRunningContainers,+    parse_container_config,+)+from tox_docker.tox4.log import log++CONTAINER_CONFIGS: Dict[str, ContainerConfig] = {}+ENV_CONTAINERS: EnvRunningContainers = {}+++@impl+def tox_add_core_config(core_conf: ConfigSet, config: Config) -> None:+    container_config_names = discover_container_configs(config)++    # validate command line options+    for container_name in config.options.docker_dont_stop:+        if container_name not in container_config_names:+            raise ValueError(+                f"Container {container_name!r} not found (from --docker-dont-stop)"+            )++    for container_name in container_config_names:+        CONTAINER_CONFIGS[container_name] = parse_container_config(+            config, container_name, container_config_names+        )+++@impl+def tox_before_run_commands(tox_env: ToxEnv) -> None:+    tox_env.conf.add_config(

I see. That wasn't really at all clear. I think a lot of work could be done to make the docs clear for plugin authors... I've had to figure a lot of this by trial and error.

dcrosta

comment created time in 2 months

PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from typing import Dict, List++from tox.config.cli.parser import ToxParser+from tox.config.main import Config+from tox.config.sets import ConfigSet+from tox.config.types import EnvList+from tox.execute.api import Outcome+from tox.plugin import impl+from tox.tox_env.api import ToxEnv++from tox_docker.config import ContainerConfig, RunningContainers+from tox_docker.plugin import (+    docker_health_check,+    docker_pull,+    docker_run,+    get_env_vars,+    HealthCheckFailed,+    stop_containers,+)+from tox_docker.tox4.config import (+    discover_container_configs,+    EnvRunningContainers,+    parse_container_config,+)+from tox_docker.tox4.log import log++CONTAINER_CONFIGS: Dict[str, ContainerConfig] = {}+ENV_CONTAINERS: EnvRunningContainers = {}+++@impl+def tox_add_core_config(core_conf: ConfigSet, config: Config) -> None:+    container_config_names = discover_container_configs(config)++    # validate command line options+    for container_name in config.options.docker_dont_stop:+        if container_name not in container_config_names:+            raise ValueError(+                f"Container {container_name!r} not found (from --docker-dont-stop)"+            )++    for container_name in container_config_names:+        CONTAINER_CONFIGS[container_name] = parse_container_config(+            config, container_name, container_config_names+        )+++@impl+def tox_before_run_commands(tox_env: ToxEnv) -> None:+    tox_env.conf.add_config(+        keys=["docker"],+        of_type=EnvList,+        default=EnvList([]),+        desc="docker image configs to load",+        post_process=list,  # type: ignore+    )++    container_names = tox_env.conf["docker"]+    env_container_configs = []++    seen = set()+    for container_name in container_names:+        if container_name not in CONTAINER_CONFIGS:+            raise ValueError(f"Missing [docker:{container_name}] in tox.ini")+        if container_name in seen:+            raise ValueError(f"Container {container_name!r} specified more than once")+        seen.add(container_name)+        env_container_configs.append(CONTAINER_CONFIGS[container_name])++    for container_config in env_container_configs:+        docker_pull(container_config, log)++    ENV_CONTAINERS.setdefault(tox_env, {})+    containers: RunningContainers = ENV_CONTAINERS[tox_env]++    for container_config in env_container_configs:+        container = docker_run(container_config, containers, log)+        containers[container_config.name] = container++    for container_name, container in containers.items():+        container_config = CONTAINER_CONFIGS[container_name]+        try:+            docker_health_check(container_config, container, log)+        except HealthCheckFailed:+            tox_env.interrupt()+            clean_up_containers(tox_env)+            raise++    for container_name, container in containers.items():+        container_config = CONTAINER_CONFIGS[container_name]+        tox_env.conf["set_env"].update(get_env_vars(container_config, container))+++@impl+def tox_after_run_commands(+    tox_env: ToxEnv, exit_code: int, outcomes: List[Outcome]+) -> None:+    clean_up_containers(tox_env)+++def clean_up_containers(tox_env: ToxEnv) -> None:+    env_containers: RunningContainers = ENV_CONTAINERS.get(tox_env, {})+    containers_and_configs = [+        (CONTAINER_CONFIGS[name], container)+        for name, container in env_containers.items()+    ]+    stop_containers(containers_and_configs, log)

It may not be worth the trouble, but my thought was to have a "business logic" definition in tox_docker/plugin.py, and then in the tox3 and tox4 versions to have adapters to the tox version. The "business logic" in question here is checking the stop flag (based on a command-line option, but propagated through to the ContainerConfig class), so admittedly it's maybe a weak case for DRY, but that's what I was after.

dcrosta

comment created time in 2 months

PullRequestReviewEvent

push eventtox-dev/tox-docker

Dan Crosta

commit sha eceef1f161caa1474234209913966053b6212d80

don't use egrep

view details

push time in 2 months

push eventtox-dev/tox-docker

Dan Crosta

commit sha b4651d8d6a7a959bf7dde9e40c5c803fb03b7d28

fix CI for healthcheck-failing

view details

push time in 2 months

push eventtox-dev/tox-docker

Dan Crosta

commit sha 70d054ed83851a380b16e39e9f6a4f808334f7e7

error message on healthcheck fail

view details

push time in 2 months

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from typing import Dict, List++from tox.config.cli.parser import ToxParser+from tox.config.main import Config+from tox.config.sets import ConfigSet+from tox.config.types import EnvList+from tox.execute.api import Outcome+from tox.plugin import impl+from tox.tox_env.api import ToxEnv++from tox_docker.config import ContainerConfig, RunningContainers+from tox_docker.plugin import (+    docker_health_check,+    docker_pull,+    docker_run,+    get_env_vars,+    HealthCheckFailed,+    stop_containers,+)+from tox_docker.tox4.config import (+    discover_container_configs,+    EnvRunningContainers,+    parse_container_config,+)+from tox_docker.tox4.log import log++CONTAINER_CONFIGS: Dict[str, ContainerConfig] = {}+ENV_CONTAINERS: EnvRunningContainers = {}+++@impl+def tox_add_core_config(core_conf: ConfigSet, config: Config) -> None:+    container_config_names = discover_container_configs(config)++    # validate command line options+    for container_name in config.options.docker_dont_stop:+        if container_name not in container_config_names:+            raise ValueError(+                f"Container {container_name!r} not found (from --docker-dont-stop)"+            )++    for container_name in container_config_names:+        CONTAINER_CONFIGS[container_name] = parse_container_config(+            config, container_name, container_config_names+        )+++@impl+def tox_before_run_commands(tox_env: ToxEnv) -> None:+    tox_env.conf.add_config(+        keys=["docker"],+        of_type=EnvList,+        default=EnvList([]),+        desc="docker image configs to load",+        post_process=list,  # type: ignore+    )++    container_names = tox_env.conf["docker"]+    env_container_configs = []++    seen = set()+    for container_name in container_names:+        if container_name not in CONTAINER_CONFIGS:+            raise ValueError(f"Missing [docker:{container_name}] in tox.ini")+        if container_name in seen:+            raise ValueError(f"Container {container_name!r} specified more than once")+        seen.add(container_name)+        env_container_configs.append(CONTAINER_CONFIGS[container_name])++    for container_config in env_container_configs:+        docker_pull(container_config, log)++    ENV_CONTAINERS.setdefault(tox_env, {})+    containers: RunningContainers = ENV_CONTAINERS[tox_env]++    for container_config in env_container_configs:+        container = docker_run(container_config, containers, log)+        containers[container_config.name] = container++    for container_name, container in containers.items():+        container_config = CONTAINER_CONFIGS[container_name]+        try:+            docker_health_check(container_config, container, log)+        except HealthCheckFailed:+            tox_env.interrupt()+            clean_up_containers(tox_env)+            raise++    for container_name, container in containers.items():+        container_config = CONTAINER_CONFIGS[container_name]+        tox_env.conf["set_env"].update(get_env_vars(container_config, container))+++@impl+def tox_after_run_commands(+    tox_env: ToxEnv, exit_code: int, outcomes: List[Outcome]+) -> None:+    clean_up_containers(tox_env)+++def clean_up_containers(tox_env: ToxEnv) -> None:+    env_containers: RunningContainers = ENV_CONTAINERS.get(tox_env, {})+    containers_and_configs = [+        (CONTAINER_CONFIGS[name], container)+        for name, container in env_containers.items()+    ]+    stop_containers(containers_and_configs, log)+++@impl+def tox_add_option(parser: ToxParser) -> None:+    # command line flag to keep docker containers running+    parser.add_argument(+        "--docker-dont-stop",+        default=[],

It works ¯_(ツ)_/¯

dcrosta

comment created time in 2 months

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commenttox-dev/tox-docker

Support tox 4 (as well as 3)

+from collections import defaultdict+from typing import Callable, Container, Dict, List, Sequence++from tox.config.loader.section import Section+from tox.config.main import Config+from tox.config.sets import ConfigSet+from tox.tox_env.api import ToxEnv++from tox_docker.config import (+    ContainerConfig,+    RunningContainers,+    validate_link,+    validate_port,+    validate_volume,+)++# nanoseconds in a second; named "SECOND" so that "1.5 * SECOND" makes sense+SECOND = 1000000000++EnvRunningContainers = Dict[ToxEnv, RunningContainers]+++class MissingRequiredSetting(Exception):+    pass+++def required(setting_name: str) -> Callable[[str], str]:+    def require_value(val: str) -> str:+        if not val:+            raise MissingRequiredSetting(setting_name)+        return val++    return require_value+++class EnvDockerConfigSet(ConfigSet):+    def register_config(self) -> None:+        self.add_config(+            keys=["docker"],+            of_type=List[str],+            default=[],+            desc="docker image configs to load",+        )+++class DockerConfigSet(ConfigSet):+    def register_config(self) -> None:+        self.add_config(+            keys=["image"],+            of_type=str,+            default="",+            post_process=required("image"),+            desc="docker image to run",+        )+        self.add_config(+            keys=["environment"],+            of_type=Dict[str, str],+            default={},+            desc="environment variables to pass to the docker container",+        )+        self.add_config(+            keys=["ports"],+            of_type=List[str],+            default=[],+            desc="ports to expose",+            post_process=list,+        )+        self.add_config(+            keys=["links"],+            of_type=List[str],+            default=[],+            desc="containers to link",+            post_process=list,+        )+        self.add_config(+            keys=["volumes"],+            of_type=List[str],+            default=[],+            desc="volumes to attach",+            post_process=list,+        )++        self.add_config(+            keys=["healthcheck_cmd"],+            of_type=str,+            default="",

See https://github.com/tox-dev/tox/issues/2241

dcrosta

comment created time in 2 months

push eventtox-dev/tox-docker

Dan Crosta

commit sha 4fbb1538dd81a56a02907305e7b8f2b5d980a880

go all in on ConfigSets in tox4

view details

Dan Crosta

commit sha c90ef3b09e6b3cfc5cc23a2b90042708c764f922

use new objects in tox3

view details

Dan Crosta

commit sha 99ad2c3a9fcbe58aad82b3ffc057ffbfbd58e653

remove unused stuff

view details

Dan Crosta

commit sha 486d5bb449cb09de5bad77b13ada70a99e04ba2e

fix some typing

view details

push time in 2 months

issue openedtox-dev/tox

tox4: typing error when add_config's of_type is Optional[...]

With the following code:

class MyConfigSet(ConfigSet):
    def register_config(self) -> None:
        self.add_config(
            keys=["key"],
            of_type=Optional[str],
            default=None,
            desc="desc",
        )

I get the following mypy error:

error: Argument "of_type" to "add_config" of "ConfigSet" has incompatible type "object"; expected "Type[None]"

if I change the default to any string (eg ""), the error changes to:

error: Argument "of_type" to "add_config" of "ConfigSet" has incompatible type "object"; expected "Type[str]"

created time in 2 months

issue commenttox-dev/tox

tox4: typing error in add_config with factory and container types

Oh sorry, didn't realize you had created this already!

dcrosta

comment created time in 2 months

more