profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/Lothiraldan/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

comet-ml/comet-examples 78

Examples of Machine Learning code using Comet.ml

comet-ml/comet-for-mlflow 47

Comet-For-MLFlow Extension

Lothiraldan/balto 20

BAlto is a Language independent Test Orchestrator

comet-ml/co2-tracker-utils 2

CO2 Tracker helpers

comet-ml/comet-recipes 1

A collection of cookiecutter recipes for integrating ML frameworks with Comet in Python and R

comet-ml/fastai 1

The fastai deep learning library, plus lessons and and tutorials

comet-ml/websocket-client 1

websocket client for python

comet-ml/dulwich 0

Pure-Python Git implementation

Pull request review commentmlco2/codecarbon

fix RAPL.end() unit

 from dataclasses import dataclass -from codecarbon.core.units import Energy+from codecarbon.core.units import Energy, Power, Time   @dataclass class RAPLFile:     name: str     path: str     energy_reading: Energy = Energy(0)-    power_measurement: float = 0+    power_measurement: float = 0  # kW

I think it is a good idea, mypy would be able to check for mixing of units. If we want to go further, we can even have a custom classes that don't wrapper float and convert them to float at the last moment.

vict0rsch

comment created time in 3 days

PullRequestReviewEvent

issue commentmlco2/codecarbon

Same as ContextualVersionConflict exception in Google Colab issue #108

@grimelda After installing packages on Google Colab (I recommend using %pip install -U codecarbon to do so), you need to restart the Runtime. Google Colab seems to start from a non-empty state, some packages are pre-installed, you can see which ones with %pip list. From this list, it looks like tzlocal is preinstall in version 1.5.1. While installing codecarbon, we install apscheduler which requires a higher version so the version tzlocal-2.1 is installed instead. Then when loading apscheduler, an automatic check happens on the dependencies:

/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py in require(self, *requirements)
    884         included, even if they were already activated in this working set.
    885         """
--> 886         needed = self.resolve(parse_requirements(requirements))

And it looks like that the old version 1.5.1 is still in memory somewhere. After restarting the Runtime, the version on disk is 2.1 so everything works as expected.

chrstnsusai

comment created time in 16 days

issue commentteejee2008/timeshift

GLib-GIO-CRITICAL

I've tried to run the cron script in debug mode to understand why daily snapshots weren't created (still digging), but I think I found some leads regarding the root cause of the issue, extract from the output of timeshift --check --debug --scripted:

juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (D: Main: load_app_config())
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (App config loaded: /etc/timeshift.json)
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (D: IconManager: init())
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (D: bin_path: /bin/timeshift)
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (D: found images directory: /usr/share/timeshift/images)
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT ()
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT ((process:256272): GLib-GIO-CRITICAL **: 20:04:02.024: g_file_get_path: assertion 'G_IS_FILE (file)' failed)
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (D: base_path: (null))
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT ()
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (** (process:256272): CRITICAL **: 20:04:02.024: tee_jee_file_system_path_combine: assertion 'path1 != NULL' failed)
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT ()
juil. 15 20:04:02 fedora CROND[256271]: (root) CMDOUT (** (process:256272): CRITICAL **: 20:04:02.024: tee_jee_file_system_dir_exists: assertion 'dir_path != NULL' failed)

I think because the bin path is /bin/timeshift, the following line https://github.com/teejee2008/timeshift/blob/08d0e5912b617009f2f0fdb61fb4173cb3576ed4/src/Utility/IconManager.vala#L78 will returns null, hence triggering the errors. I've installed using the fedora system package.

System:

  • Linux Distribution Name and Version: Fedora 34
  • Desktop: Gnome
  • Application Version: v20.03
g0rdonL

comment created time in 17 days

startedteejee2008/timeshift

started time in 17 days

startedvinceliuice/Orchis-theme

started time in 17 days

PR opened mlco2/codecarbon

Simplify args rebased
+160 -121

0 comment

6 changed files

pr created time in 2 months

create barnchmlco2/codecarbon

branch : simplify-args-rebased

created branch time in 2 months

issue commentPyCQA/isort

Bug: Isort dropping # noqa comment with star import

Thanks @anirudnits and @timothycrosley for the prompt workaround and prompt fix, you rocks!

Lothiraldan

comment created time in 2 months

PullRequestReviewEvent

pull request commentmlco2/codecarbon

[WIP] Count RAM consumption

@Lothiraldan my main concern is actually about https://github.com/mlco2/codecarbon/pull/166/files#diff-cf42f273abce88a8c0b9a5f3133f00f5b53a8aa0f8542f0ab10830bf0b6fecc6R190 do you think it's a good enough way of getting the memory used by the main process (where the tracker was instantiated) and its children?

It looks ok but I would check two things manually:

  • How fast it can retrieves the memory of lots processes
  • How it behaves if a children dies between the time it was returned in the children list and the time we try to get its RSS
vict0rsch

comment created time in 2 months

pull request commentmlco2/codecarbon

Simplify attribute setting from arguments & conf

@Lothiraldan any idea beyond extracting from the package what we need so that it's not a dependency anymore?

I would avoid using it altogether. I'm not sure how it would support new Python versions, alternative Python implementations (like Pypy) and I have high doubt it would be fast and effective.

That would likely means that instead of calling: _get_conf(log_level, "info") we will do _get_conf(log_level, "log_level", "info") which is a bit repetitive but not that much.

vict0rsch

comment created time in 2 months

issue commentmlco2/codecarbon

Use PyPI Extras to reduce dependencies in server-based use

The usual way of using extras in Pypi packages is to use the extra_requires like here: https://github.com/mlflow/mlflow/blob/5508e6784afe0442705471bf4c10cd1ca13bafe3/setup.py#L89. It's much more easier to use when installing a package on command-line or in a requirements.txt file.

I think MLFlow used an environment variable to avoid most users having to install mlflow[not-skinny].

In addition, we have a Conda package that needs to be taken care of. It seems that mlflow on Conda has two variants https://github.com/conda-forge/mlflow-feedstock/blob/master/recipe/conda_build_config.yaml which is then used like this https://github.com/conda-forge/mlflow-feedstock/blob/master/recipe/meta.yaml. It seems doable to have both codecarbon and codecarbon-server conda packages.

mdekstrand

comment created time in 2 months

issue openedPyCQA/isort

Bug: Isort dropping # noqa comment with star import

Version: 5.8.0 Python 3.7.5 (running through pre-commit but not sure) Description Isort drops # noqa comment while sorting, when two imports from the same file/module exists, and one of them is a star import. I think it might be related to https://github.com/PyCQA/isort/issues/1594 but I preferred opening a new bug if that's not the case.

Steps to reproduce Create one file with content as below. Sort test.py with isort

test.py

from typing import *  # noqa
from typing import IO, BinaryIO, Union # noqa

Actual result:

from typing import *  # noqa
from typing import IO, BinaryIO, Union

As you can see, #noqa comment was dropped from the second line, resulting in linter failling.

Expected result: Imports should be kept as is, both # noqa comments should be kept as well.

Thank you for your amazing work!

created time in 2 months

Pull request review commentmlco2/codecarbon

WIP : API Call

+"""++Based on https://kernelpanic.io/the-modern-way-to-call-apis-in-python++TODO : use async call to API+"""+# from httpx import AsyncClient+import dataclasses+import json+import time+from datetime import timedelta, tzinfo++import arrow+import requests++from codecarbon.core.schemas import EmissionCreate, RunCreate+from codecarbon.external.logger import logger++# from codecarbon.output import EmissionsData+++class ApiClient:  # (AsyncClient)+    """+    This class call the Code Carbon API+    Note : The project, team and organization must have been created in the interface.+    """++    run_id = None+    _previous_call = time.time()++    def __init__(self, experiment_id, endpoint_url, api_key):+        """+        :project_id: ID of the existing project+        :api_ley: Code Carbon API_KEY+        """+        # super().__init__(base_url=endpoint_url) # (AsyncClient)+        self.url = endpoint_url+        self.experiment_id = experiment_id+        self.api_key = api_key+        self._create_run(self.experiment_id)++    def add_emission(self, carbon_emission: dict):+        self._previous_call = time.time()+        if self.run_id is None:+            # TODO : raise an Exception ?+            logger.error(+                "add_emissionadd_emission need a run_id : the initial call may have failed. Retrying..."+            )+            self._create_run(self.experiment_id)+        if carbon_emission["duration"] < 1:+            logger.warning(+                "Warning : emission not send because of a duration smaller than 1."+            )+            return False+        emission = EmissionCreate(+            timestamp=self.get_datetime_with_timezone(),+            run_id=self.run_id,+            duration=int(carbon_emission["duration"]),+            emissions=carbon_emission["emissions"],+            energy_consumed=carbon_emission["energy_consumed"],+        )+        try:+            payload = dataclasses.asdict(emission)+            r = requests.put(url=self.url + "/emission", json=payload)

Let's add a timeout here

benoit-cty

comment created time in 2 months

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentmlco2/codecarbon

WIP : API Call

+"""++Based on https://kernelpanic.io/the-modern-way-to-call-apis-in-python++TODO : use async call to API+"""+# from httpx import AsyncClient+import dataclasses+import json+import time+from datetime import datetime++import requests++from codecarbon.core.schemas import EmissionCreate, RunCreate+from codecarbon.external.logger import logger++# from codecarbon.output import EmissionsData+++class ApiClient:  # (AsyncClient)+    """+    This class call the Code Carbon API+    Note : The project, team and organization must have been created in the interface.+    """++    run_id = None+    _previous_call = time.time()++    def __init__(self, experiment_id, endpoint_url, api_key):+        """+        :project_id: ID of the existing project+        :api_ley: Code Carbon API_KEY+        """+        # super().__init__(base_url=endpoint_url) # (AsyncClient)+        self.url = endpoint_url+        self.experiment_id = experiment_id+        self.api_key = api_key+        self._create_run(self.experiment_id)++    def add_emission(self, carbon_emission: dict):+        self._previous_call = time.time()+        if self.run_id is None:+            # TODO : raise an Exception ?+            logger.error(+                "add_emissionadd_emission need a run_id : the initial call may have failed. Retrying..."+            )+            self._create_run(self.experiment_id)+        if carbon_emission["duration"] < 1:+            logger.warning(+                "Warning : emission not send because of a duration smaller than 1."+            )+            return False+        emission = EmissionCreate(+            timestamp=datetime.now().isoformat(),

We might want to send UTC timestamps

benoit-cty

comment created time in 2 months

PullRequestReviewEvent

Pull request review commentmlco2/codecarbon

WIP : API Call

+"""++Based on https://kernelpanic.io/the-modern-way-to-call-apis-in-python++TODO : use async call to API+"""+# from httpx import AsyncClient+import dataclasses+import json+import time+from datetime import datetime++import requests++from codecarbon.core.schemas import EmissionCreate, RunCreate+from codecarbon.external.logger import logger++# from codecarbon.output import EmissionsData+++class ApiClient:  # (AsyncClient)+    """+    This class call the Code Carbon API+    Note : The project, team and organization must have been created in the interface.+    """++    run_id = None+    _previous_call = time.time()++    def __init__(self, experiment_id, endpoint_url, api_key):+        """+        :project_id: ID of the existing project+        :api_ley: Code Carbon API_KEY+        """+        # super().__init__(base_url=endpoint_url) # (AsyncClient)+        self.url = endpoint_url+        self.experiment_id = experiment_id+        self.api_key = api_key+        self._create_run(self.experiment_id)++    def add_emission(self, carbon_emission: dict):+        self._previous_call = time.time()+        if self.run_id is None:+            # TODO : raise an Exception ?+            logger.error(+                "add_emissionadd_emission need a run_id : the initial call may have failed. Retrying..."+            )+            self._create_run(self.experiment_id)+        if carbon_emission["duration"] < 1:+            logger.warning(+                "Warning : emission not send because of a duration smaller than 1."+            )+            return False+        emission = EmissionCreate(+            timestamp=datetime.now().isoformat(),+            run_id=self.run_id,+            duration=int(carbon_emission["duration"]),+            emissions=carbon_emission["emissions"],+            energy_consumed=carbon_emission["energy_consumed"],+        )+        try:+            payload = dataclasses.asdict(emission)+            r = requests.put(url=self.url + "/emission", json=payload)+            if r.status_code != 201:+                self._log_error(payload, r)+            assert r.status_code == 201+        except Exception as e:+            logger.error(e, exc_info=True)+            return False+        return True++    def _create_run(self, experiment_id):+        """+        Create the experiment for project_id+        # TODO : Allow to give an existing experiment_id+        """+        try:+            run = RunCreate(+                timestamp=datetime.now().isoformat(), experiment_id=experiment_id+            )+            payload = dataclasses.asdict(run)+            r = requests.put(url=self.url + "/run", json=payload)+            if r.status_code != 200:+                self._log_error(payload, r)+            assert r.status_code == 200+            self.run_id = r.json()["id"]+        except Exception as e:+            logger.error(e, exc_info=True)++    def _log_error(self, payload, response):+        logger.error(f" Error when calling the API with : {json.dumps(payload)}")+        logger.error(+            f" API return http code {response.status_code} and answer : {response.json()}"

We might receives non-JSON response in case the URL is wrong, the server is down or a random proxy/firewall blocks the connection.

            f" API return http code {response.status_code} and answer : {response.text}"
benoit-cty

comment created time in 2 months

PullRequestReviewEvent

issue commentwebsocket-client/websocket-client

Official Apache 2.0 License Transition

Yes, I consent to the change to the Apache 2.0 license

engn33r

comment created time in 2 months

delete branch comet-ml/websocket-client

delete branch : fix-closing-thread-safety

delete time in 2 months

issue commentmlco2/codecarbon

Recipe on conda-forge

Hi @PertuyF, thank you for your ticket and suggestion.

We already have conda recipes in the repo who are slightly different: https://github.com/mlco2/codecarbon/blob/master/.conda/py36/meta.yaml and https://github.com/mlco2/codecarbon/blob/master/.conda/py37-plus/meta.yaml. The separate recipe for Python 3.6 is needed to install the dataclasses package for this particular Python version. I didn't find a better way to handle that in a single conda recipe, maybe you can help?

How does it works when we add a new dependency? Are we supposed to make a PR on the conda-forge repository?

PertuyF

comment created time in 3 months

issue commentmlco2/codecarbon

Codecarbon Database Schema

I can discuss how our namespaces are structured. Each comet_ml.Experiment have an unique ID, generated on the client side (with uuid.uuid4() basically). Each experiment uses a single API Key attached to a single user. Each user has a default workspace (named after the user) and inside that workspace there is a default (private) project named general. That means when sending data (and if the user don't provides a workspace or project_name), we can find "where" to store the data.

Users can also provide a workspace name and/or a project name. If the workspace exists and if the user has permission to write in it, we store the data in the provided workspace and/or project.

Projects (and experiment) are automatically created but workspaces are not. Permission are managed at the workspace level and each workspace admin can add other members to read and/or write to that workspace.

Namespaces (or team/project/organizations) design often comes to solves permissions requirements, how are you envisioning read/write permissions?

vict0rsch

comment created time in 3 months