profile
viewpoint
Maximilian Hils mhils Innsbruck, Austria | Berkeley, CA https://hi.ls Information security researcher, mitmproxy developer, PhD student.

ffalcinelli/pydivert 105

A Python binding for WinDivert driver

AppPETs/developer-guidelines 8

Developer Guidelines for Privacy Aware Mobile Apps

mhils/backports.socketpair 2

Python 2 support for socket.socketpair() on Windows

honeynet/stickers 1

Hexagon stickers for @honeynet projects.

mhils/backbone 1

Give your JS App some Backbone with Models, Views, Collections, and Events

mhils/arbtt 0

arbtt, the automatic rule-based time-tracker

mhils/awesome-for-beginners 0

A list of awesome beginners-friendly projects.

mhils/BDFProxy 0

Patch Binaries via MITM: BackdoorFactory + mitmProxy.

mhils/catawampus 0

Unofficial mirror of https://gfiber.googlesource.com/vendor/google/catawampus/

mhils/chroma 0

A general purpose syntax highlighter in pure Go

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+import mimetypes+import re+import typing+import urllib+from pathlib import Path++from mitmproxy import exceptions+from mitmproxy import ctx+from mitmproxy import http+from mitmproxy.addons.modifyheaders import parse_modify_spec, ModifySpec+++class MapLocal:+    def __init__(self):+        self.replacements: typing.List[ModifySpec] = []++    def load(self, loader):+        loader.add_option(+            "map_local", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/file-or-directory", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        if "map_local" in updated:+            self.replacements = []+            for option in ctx.options.map_local:+                try:+                    spec = parse_modify_spec(option, True, True)+                except ValueError as e:+                    raise exceptions.OptionsError(f"Cannot parse map_local option {option}: {e}") from e++                self.replacements.append(spec)++    def construct_candidate_path(self, base_path, path_components, filename):+        candidate_path = base_path.joinpath("/".join(path_components + [filename]))+        return str(candidate_path)++    def sanitize_candidate_path(self, candidate_path, base_path):+        try:+            candidate_path = candidate_path.resolve(strict=True)+            if base_path == candidate_path or base_path in candidate_path.parents:+                return candidate_path+        except FileNotFoundError:+            pass+        return None++    def file_candidates(self, url: str, spec: ModifySpec) -> typing.List[Path]:+        replacement = spec.replacement+        candidates = []++        if replacement.is_file():+            candidates.append(replacement)++        elif replacement.is_dir():+            parsed_url = urllib.parse.urlparse(url)++            path_components = parsed_url.path.lstrip("/").split("/")+            filename = path_components.pop()++            # todo: this can be improved (e.g., also consider index.htm)+            if not filename:+                filename = 'index.html'++            # construct all possible paths+            while True:+                candidates.append(+                    self.construct_candidate_path(replacement, path_components, filename)+                )++                if not path_components:+                    break++                path_components.pop()++        return candidates++    def get_mime_type(self, file_path):+        mimetype = (+            mimetypes.guess_type(file_path)[0]+            or "text/plain"

'application/octet-stream' is a bit more useful. The stuff that generally isn't recognized is binary blobs, which should trigger a download. :)

mplattner

comment created time in 4 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+import mimetypes+import re+import typing+import urllib+from pathlib import Path++from mitmproxy import exceptions+from mitmproxy import ctx+from mitmproxy import http+from mitmproxy.addons.modifyheaders import parse_modify_spec, ModifySpec+++class MapLocal:+    def __init__(self):+        self.replacements: typing.List[ModifySpec] = []++    def load(self, loader):+        loader.add_option(+            "map_local", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/file-or-directory", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        if "map_local" in updated:+            self.replacements = []+            for option in ctx.options.map_local:+                try:+                    spec = parse_modify_spec(option, True, True)+                except ValueError as e:+                    raise exceptions.OptionsError(f"Cannot parse map_local option {option}: {e}") from e++                self.replacements.append(spec)++    def construct_candidate_path(self, base_path, path_components, filename):+        candidate_path = base_path.joinpath("/".join(path_components + [filename]))+        return str(candidate_path)++    def sanitize_candidate_path(self, candidate_path, base_path):

Looking closer at this, it might actually be the best approach to just call werkzeug's safe_join. We could then also already call sanitization in file_candidates so that it only returns a list of safe paths. What do you think?

mplattner

comment created time in 4 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+import mimetypes+import re+import typing+import urllib+from pathlib import Path++from mitmproxy import exceptions+from mitmproxy import ctx+from mitmproxy import http+from mitmproxy.addons.modifyheaders import parse_modify_spec, ModifySpec+++class MapLocal:+    def __init__(self):+        self.replacements: typing.List[ModifySpec] = []++    def load(self, loader):+        loader.add_option(+            "map_local", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/file-or-directory", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        if "map_local" in updated:+            self.replacements = []+            for option in ctx.options.map_local:+                try:+                    spec = parse_modify_spec(option, True, True)+                except ValueError as e:+                    raise exceptions.OptionsError(f"Cannot parse map_local option {option}: {e}") from e++                self.replacements.append(spec)++    def construct_candidate_path(self, base_path, path_components, filename):+        candidate_path = base_path.joinpath("/".join(path_components + [filename]))+        return str(candidate_path)++    def sanitize_candidate_path(self, candidate_path, base_path):

Here's some existing work on that problem:

  • I believe this is what Flask/werkzeug is doing: https://github.com/pallets/werkzeug/blob/0d0d1e7263013c4e14750d8fb91984cf1584e0e0/src/werkzeug/security.py#L202 https://github.com/pallets/werkzeug/blob/09a248649069780b4adfb910fb69f6fb304f2ea3/tests/test_security.py#L69-L82
  • And here's aiohttp: https://github.com/aio-libs/aiohttp/blob/2a924cae6c19dbbfde220bbd461fc446764ad156/aiohttp/web_urldispatcher.py#L585
mplattner

comment created time in 4 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+from pathlib import Path++from mitmproxy.addons.maplocal import MapLocal+from mitmproxy.test import taddons+from mitmproxy.test import tflow++from mitmproxy.addons.modifyheaders import parse_modify_spec+++class TestMapLocal:+    def test_file_candidates(self, tmpdir):

See comment above: Eliminate I/O, then we don't need tmpdir.

mplattner

comment created time in 5 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+from pathlib import Path++from mitmproxy.addons.maplocal import MapLocal+from mitmproxy.test import taddons+from mitmproxy.test import tflow++from mitmproxy.addons.modifyheaders import parse_modify_spec+++class TestMapLocal:+    def test_file_candidates(self, tmpdir):+        ml = MapLocal()++        url = "https://example.org/img/topic/subtopic/test.jpg"+        spec = parse_modify_spec(":/img/jpg:" + str(tmpdir), True, True)+        file_candidates = ml.file_candidates(url, spec)+        assert file_candidates[0] == str(tmpdir) + "/img/topic/subtopic/test.jpg"+        assert file_candidates[1] == str(tmpdir) + "/img/topic/test.jpg"+        assert file_candidates[2] == str(tmpdir) + "/img/test.jpg"+        assert file_candidates[3] == str(tmpdir) + "/test.jpg"++        url = "https://example.org/img/topic/subtopic/"+        spec = parse_modify_spec(":/img:" + str(tmpdir), True, True)+        file_candidates = ml.file_candidates(url, spec)+        assert file_candidates[0] == str(tmpdir) + "/img/topic/subtopic/index.html"+        assert file_candidates[1] == str(tmpdir) + "/img/topic/index.html"+        assert file_candidates[2] == str(tmpdir) + "/img/index.html"+        assert file_candidates[3] == str(tmpdir) + "/index.html"++        url = "https://example.org"+        spec = parse_modify_spec(":org:" + str(tmpdir), True, True)+        file_candidates = ml.file_candidates(url, spec)+        assert file_candidates[0] == str(tmpdir) + "/index.html"

We can make this (and test_sanitize_candidate_path) nicer with pytest's parametrize (see here for example):

@pytest.mark.parametrize("url,spec,expected_candidates", [
(
    "https://example.org/foo/bar.jpg", 
    ":example.com/foo:/tmp", 
    ["/tmp/bar.jpg", ...]
),
])
def test_file_candidates(url: str, spec: str, expected_candidates: List[str]):
    spec = parse_modify_spec(spec, True, True)
    candidates = file_candidates(url, spec)
    assert [str(x) for x in candidates] == expected_candidates
mplattner

comment created time in 4 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+import mimetypes+import re+import typing+import urllib+from pathlib import Path++from mitmproxy import exceptions+from mitmproxy import ctx+from mitmproxy import http+from mitmproxy.addons.modifyheaders import parse_modify_spec, ModifySpec+++class MapLocal:+    def __init__(self):+        self.replacements: typing.List[ModifySpec] = []++    def load(self, loader):+        loader.add_option(+            "map_local", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/file-or-directory", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        if "map_local" in updated:+            self.replacements = []+            for option in ctx.options.map_local:+                try:+                    spec = parse_modify_spec(option, True, True)+                except ValueError as e:+                    raise exceptions.OptionsError(f"Cannot parse map_local option {option}: {e}") from e++                self.replacements.append(spec)++    def construct_candidate_path(self, base_path, path_components, filename):+        candidate_path = base_path.joinpath("/".join(path_components + [filename]))+        return str(candidate_path)++    def sanitize_candidate_path(self, candidate_path, base_path):+        try:+            candidate_path = candidate_path.resolve(strict=True)+            if base_path == candidate_path or base_path in candidate_path.parents:+                return candidate_path+        except FileNotFoundError:+            pass+        return None++    def file_candidates(self, url: str, spec: ModifySpec) -> typing.List[Path]:+        replacement = spec.replacement+        candidates = []++        if replacement.is_file():+            candidates.append(replacement)++        elif replacement.is_dir():

I would move the is_file and is_dir check directly into request and invoke file_candidates only for directories. Then we can make sure that there are no decisions involving I/O in file_candidates, which makes it easier to test.

Also, we can make this @staticmethod or move it out of the addon (I personally prefer the latter, but YMMV). This thing is complex, so we want to reduce the complexity of the mental model one needs to build as much as possible.

mplattner

comment created time in 5 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

 def parse_modify_spec(option, subject_is_regex: bool) -> ModifySpec:         except re.error as e:             raise ValueError(f"Invalid regular expression {subject!r} ({e})") +    if replacement_is_path:+        path = Path(replacement)+        try:+            replacement = path.expanduser().resolve(strict=True)

This doesn't fly - we say replacement is a str, so we can't just make it a Path here. I think this needs to happen on-demand in the maplocal addon.

mplattner

comment created time in 5 hours

Pull request review commentmitmproxy/mitmproxy

MapLocal addon

+import mimetypes+import re+import typing+import urllib+from pathlib import Path++from mitmproxy import exceptions+from mitmproxy import ctx+from mitmproxy import http+from mitmproxy.addons.modifyheaders import parse_modify_spec, ModifySpec+++class MapLocal:+    def __init__(self):+        self.replacements: typing.List[ModifySpec] = []++    def load(self, loader):+        loader.add_option(+            "map_local", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/file-or-directory", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        if "map_local" in updated:+            self.replacements = []+            for option in ctx.options.map_local:+                try:+                    spec = parse_modify_spec(option, True, True)+                except ValueError as e:+                    raise exceptions.OptionsError(f"Cannot parse map_local option {option}: {e}") from e++                self.replacements.append(spec)++    def construct_candidate_path(self, base_path, path_components, filename):

Let's make sure that we use type annotations where it's easily possible. :) We also really want to return Path objects, not str.

mplattner

comment created time in 5 hours

push eventmitmproxy/mitmproxy

gorogoroumaru

commit sha 0e998e9b52cabad3221cce4825bcdbb6fd6ad71c

Fix typespec_to_str funcion to process typing.Optional[int] (#4066) * Fix typespec_to_str funcion to process typing.Optional[int] * Add test * Add test Co-authored-by: gorogoroumaru <zokutyou2@gmail.com>

view details

push time in 3 days

PR merged mitmproxy/mitmproxy

Fix typespec_to_str funcion to process typing.Optional[int]

Fix #4059

The file examples/addons/options-configure.py uses typespec "typing.Optional[int]" which was not implemented in typespec_to_str function. So, I implemented it.

+3 -0

0 comment

2 changed files

gorogoroumaru

pr closed time in 3 days

issue closedmitmproxy/mitmproxy

Mitmweb fails with addons/options-configure.py example.

I am new to learn it, but i follow official demo, it can't working?

Proxy server listening at http://*:8888
ERROR:tornado.application:Uncaught exception GET /options.json (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:8081', method='GET', uri='/options.json', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\tornado\web.py", line 1697, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\mitmproxy\tools\web\app.py", line 453, in get
    self.write(optmanager.dump_dicts(self.master.options))
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\mitmproxy\optmanager.py", line 469, in dump_dicts
    t = typecheck.typespec_to_str(o.typespec)
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\mitmproxy\utils\typecheck.py", line 85, in typespec_to_str
    raise NotImplementedError
NotImplementedError
ERROR:tornado.access:500 GET /options.json (127.0.0.1) 3.91ms

closed time in 3 days

jekoie

push eventmitmproxy/mitmproxy

Vane11ope

commit sha 7806a3485893e5b54d701fd38e3cb62d044d4091

List was not cycled right for tab autocompletion

view details

Vane11ope

commit sha e43c4e3cb1227963a401170b0b147854705c60fb

Fix a type error

view details

Vane11ope

commit sha 2e0c078e7cc2b8cd0722f27a7612a016838aa829

Use -1 as an initial value for the pos

view details

Vane11ope

commit sha ecf076cb2413bd71083841fee1b522131a9dea40

Add tests for list completer

view details

Maximilian Hils

commit sha 65318603ae216353a3de7a70f0ccb240dcb18b64

Merge pull request #4058 from Vane11ope/vane11ope/fix List was not cycled right for tab auto-completion

view details

push time in 3 days

PR merged mitmproxy/mitmproxy

List was not cycled right for tab auto-completion

Description

Options was not cycled right for tab auto-completion. Using tabs and shift-tabs mixed up when you are in cut.clip for instance. You would see the options just don't go back right when you press shift-tab, and it wouldn't go forward right for some occasion caused by the same bug.

The code before did not even care if tab or shift tab was pressed and set the previously calculated position as the new position of the options, then calculate the next position after it.

So fix was fairly simple. Check if tab or shift tab was pressed and calculate the new position according to it. A little difficult part was how to deal with the first time the tab or shift tab was pressed, but it could be handled by setting sys.maxsize as the initial value for the position.

Probably the code itself would explain you better :)

Checklist

  • [x] I have updated tests where applicable.
  • [ ] I have added an entry to the CHANGELOG.
+22 -8

5 comments

2 changed files

Vane11ope

pr closed time in 3 days

pull request commentmitmproxy/mitmproxy

List was not cycled right for tab auto-completion

Awesome, thanks! 🍰

Vane11ope

comment created time in 3 days

issue commentbluejekyll/trust-dns

how to send a valid MessageResponse with ResponseHandler

I ran into the same problem and what worked for me was to explicitly specify types:

let records = vec![record];
let response = MessageResponseBuilder::new(
    Some(request_message.raw_queries())
).build(
    header,
    Box::new(records.iter()) as Box<dyn Iterator<Item=&Record> + Send>,
    Box::new(iter::empty()) as Box<dyn Iterator<Item=&Record> + Send>,
    Box::new(iter::empty()) as Box<dyn Iterator<Item=&Record> + Send>,
    Box::new(iter::empty()) as Box<dyn Iterator<Item=&Record> + Send>,
);

let result = response_handle.send_response(response);

I'm not much of a Rust expert though, @bluejekyll's answer is probably more correct but less easy to parse for me :)

Sherlock-Holo

comment created time in 3 days

issue commentmitmproxy/mitmproxy

Transparent mode failure: FileNotFoundError(2, 'No such file or directory')

See https://github.com/mitmproxy/mitmproxy/issues/3533#issuecomment-640811676.

TeddJohnson

comment created time in 4 days

issue commentmitmproxy/mitmproxy

Transparent mode failure: FileNotFoundError(2, 'No such file or directory')

what is passed when this method is called? I'd like to dummy-test it, see what I can figure out.

We pass a regular socket.socket object for the connection obtained using socket.accept(). csock.getsockopt pretty much does a getsockopt syscall and not much more. There's either an issue with your network/iptables configuration, CentOS 8 not supporting SO_ORIGINAL_DST, or a rare race condition where the socket is already closed (unlikely).

TeddJohnson

comment created time in 4 days

delete branch mitmproxy/mitmproxy

delete branch : codecov-fails-too-much

delete time in 4 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha b46ffe1395efa246f4ccdc0aeb1d7d79eaf01b3d

don't fail on covecov errors this happens quite a bit, so let's stop failing tests for that.

view details

Maximilian Hils

commit sha d52b17f2006559ecc4bb6d065e46009f10456c1c

Merge pull request #4065 from mitmproxy/codecov-fails-too-much Don't fail on covecov errors

view details

push time in 4 days

PR merged mitmproxy/mitmproxy

Don't fail on covecov errors

This happens quite a bit, so let's stop failing tests for that.

+2 -2

0 comment

1 changed file

mhils

pr closed time in 4 days

issue commentmitmproxy/mitmproxy

Transparent mode failure: FileNotFoundError(2, 'No such file or directory')

Thanks! That pretty much confirms my hypothesis of the syscall not working as intended. This also happens if you run mitmproxy as root?

The FileNotFoundError generated by Python doesn't really make sense, but that would need to be fixed in Python, not mitmproxy.

TeddJohnson

comment created time in 4 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha 96ce21687e91e3530b130f22c90871fd07c3e767

blacklist -> blocklist

view details

push time in 4 days

issue commentmitmproxy/mitmproxy

Transparent mode failure: FileNotFoundError(2, 'No such file or directory')

"Transparent mode failure" indicates that we likely fail somewhere in here: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/platform/linux.py

Maybe you can debug that the professional way using print statements. My guess would be this is a CentOS problem with one of the syscalls.

TeddJohnson

comment created time in 4 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha b46ffe1395efa246f4ccdc0aeb1d7d79eaf01b3d

don't fail on covecov errors this happens quite a bit, so let's stop failing tests for that.

view details

push time in 4 days

issue closedmitmproxy/mitmproxy

Use markdown code blocks for documentation

Is your feature request related to a problem? Please describe. The {{< highlight none >}} :~q:foo:bar {{< / highlight >}} blocks don't work on GitHub. It's confusing and makes the code examples almost useless on GitHub itself. And also in your code editor if you have Markdown syntax highlighting with embedded highlighting for nested code.

Describe the solution you'd like

It'd be neat if we could switch to using the standard triple backticks. I didn't search for it, I guess this has to do with how the website is generated from the markdown files. Can this be updated?

closed time in 4 days

Prinzhorn

push eventmitmproxy/mitmproxy

Tedd Johnson

commit sha c3d3b75517c05d0b0600d96d886f6437fa9e0b40

Update concepts-certificates.md {{< highlight bash >}} doesn't render properly on the wiki, so converted to codeblock

view details

Maximilian Hils

commit sha c60c3dbb75c344fb5b422e5ac137e64518a94b3e

docs: consistently use backticks for highlighting

view details

Maximilian Hils

commit sha 491123df2380ee84c8470a57ff7ba2b3a2b8a0a8

Merge pull request #4064 from TeddJohnson/patch-1 Update concepts-certificates.md

view details

push time in 4 days

PR merged mitmproxy/mitmproxy

Update concepts-certificates.md

Description

{{< highlight bash >}} doesn't render properly on the wiki, so converted to codeblock

Checklist

  • [ ] I have updated tests where applicable.
  • [ ] I have added an entry to the CHANGELOG.
+180 -180

2 comments

16 changed files

TeddJohnson

pr closed time in 4 days

pull request commentmitmproxy/mitmproxy

List was not cycled right for tab auto-completion

Thanks @Vane11ope!

Could you please

  1. Add some tests in https://github.com/mitmproxy/mitmproxy/blob/master/test/mitmproxy/tools/console/test_commander.py#L28?
  2. Use something else for the "not set" offset value? I would favor None or -1. Semantically, sys.maxsize has nothing to do with that, so it's a bit of an odd choice.
Vane11ope

comment created time in 4 days

PR opened mitmproxy/mitmproxy

Reviewers
Don't fail on covecov errors

This happens quite a bit, so let's stop failing tests for that.

+2 -2

0 comment

1 changed file

pr created time in 4 days

create barnchmitmproxy/mitmproxy

branch : codecov-fails-too-much

created branch time in 4 days

pull request commentmitmproxy/mitmproxy

Update concepts-certificates.md

I just ran this over the entire documentation, thanks again.

fixes #3946

TeddJohnson

comment created time in 4 days

push eventTeddJohnson/mitmproxy

Maximilian Hils

commit sha c60c3dbb75c344fb5b422e5ac137e64518a94b3e

docs: consistently use backticks for highlighting

view details

push time in 4 days

delete branch mhils/mitmproxy

delete branch : fix-tests

delete time in 4 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha 0ee0cf566817fe842d3997445f6d891a5a1a90f6

minor addon improvements, fix tests

view details

Maximilian Hils

commit sha 75ef914c05aad7b4cf269617511386250bc80667

Merge pull request #4062 from mhils/fix-tests Minor addon improvements, fix tests

view details

push time in 4 days

PR merged mitmproxy/mitmproxy

Minor addon improvements, fix tests

This PR fixes the Windows tests and introduces some minor code-reuse in the modify* and map* addons.

@mplattner: There shouldn't be much controvery here, feel free to take a look and merge if you are happy! 😃

+146 -177

1 comment

6 changed files

mhils

pr closed time in 4 days

push eventTeddJohnson/mitmproxy

Maximilian Hils

commit sha b99ee181006d5a634e6cab38d7b8124c71723624

docs: consistently use backticks for highlighting

view details

push time in 4 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha 50d7b4e84fd7d83acc8579362f56d34c530e65c9

don'

view details

push time in 4 days

push eventmhils/mitmproxy

Maximilian Hils

commit sha 0ee0cf566817fe842d3997445f6d891a5a1a90f6

minor addon improvements, fix tests

view details

push time in 4 days

pull request commentmitmproxy/mitmproxy

Minor addon improvements, fix tests

Codecov we could ignore, individual coverage we can't. Will fix asap! :)

mhils

comment created time in 4 days

pull request commentmitmproxy/mitmproxy

Update concepts-certificates.md

Thanks! Can you please make the start tag ```bash so that syntax highlighting remains?

TeddJohnson

comment created time in 4 days

issue commentmitmproxy/mitmproxy

mitmproxy dictionary changed size during iteration

Hi, how can reproduce this?

CGpythoner

comment created time in 5 days

issue commentmitmproxy/mitmproxy

official demo not working???

Thanks for reporting this. It's is a bug in mitmweb only, mitmproxy and mitmdump are unaffected.

Repro:

mitmweb -s examples/addons/options-configure.py
jekoie

comment created time in 5 days

issue closedmitmproxy/mitmproxy

How to update options using python script without restart

I want to start mitmproxy, and later when I set option startInterception=True I should be able to record all the flows to a file, further when I set option to startInterception=False, recording should stop.

I started mitmproxy using subprocess command and set option using subprocess.Popen('mitmdump -s /Users/hemantj/PycharmProjects/OneRobot/record_script.py --set startInterception=True')

How can I set startInterception=False without stopping mitmproxy?

closed time in 5 days

hemantj

issue commentmitmproxy/mitmproxy

How to update options using python script without restart

There is no startInterception setting in mitmproxy, OneRobot/record_script.py is not part of mitmproxy. Also, please don't cross-post between StackOverflow and here.

hemantj

comment created time in 5 days

PR opened mitmproxy/mitmproxy

Reviewers
Minor addon improvements, fix tests

This PR fixes the Windows tests and introduces some minor code-reuse in the modify* and map* addons.

@mplattner: There shouldn't be much controvery here, feel free to take a look and merge if you are happy! 😃

+134 -175

0 comment

6 changed files

pr created time in 5 days

push eventmhils/mitmproxy

Maximilian Hils

commit sha adaf12ae461316c15a044b6aa68d8839df87b94a

minor addon improvements, fix tests

view details

push time in 5 days

push eventmhils/mitmproxy

Maximilian Hils

commit sha 3142c10be1834b37983eb1f0fb0377dd2fb56d6b

minor addon improvements, fix tests

view details

push time in 5 days

create barnchmhils/mitmproxy

branch : fix-tests

created branch time in 5 days

push eventmitmproxy/mitmproxy

Martin Plattner

commit sha 89fad1e2b1a5a84672aa22e128c39165059351c9

add mapremote addon to modify request URLs

view details

Maximilian Hils

commit sha 417c213dd720312ba554c5bb7c258fe65fe4e52a

minor docs improvements

view details

Martin Plattner

commit sha cee4da4e8c1600119115cd624bdbea3da7e0d6f2

fix docs and url encoding

view details

Maximilian Hils

commit sha cf158022d9c2fde6d94d896506abacc661e96f67

Merge pull request #4060 from mplattner/mapremote-addon add mapremote addon to modify request URLs

view details

push time in 5 days

PR merged mitmproxy/mitmproxy

add mapremote addon to modify request URLs

Description

Added a new addon (MapRemote) to rewrite URLs of requests before the request is performed.

refs #3948

Checklist

  • [x] I have updated tests where applicable.
  • [x] I have added an entry to the CHANGELOG.
+203 -21

0 comment

6 changed files

mplattner

pr closed time in 5 days

Pull request review commentmitmproxy/mitmproxy

add mapremote addon to modify request URLs

 The _separator_ is arbitrary, and is defined by the first character.  ### Examples -Replace `example.org` in all request URLs to with `mitmproxy.org`:+Map all requests ending with `.jpg` to `https://placedog.net/640/480?random`: -{{< highlight none  >}}-/example.org/mitmproxy.org-{{< / highlight >}}+```+:.*\.jpg$:https://placedog.net/640/480?random

Good catch on the pattern. 👍

The HTTP2 issue is a super annoying one, which we'll hopefully be able to fix with sans-io. Let's put in a general note for the addon for now.

mplattner

comment created time in 5 days

Pull request review commentmitmproxy/mitmproxy

add mapremote addon to modify request URLs

+import os+import re+import typing++from mitmproxy import exceptions+from mitmproxy import ctx+from mitmproxy.utils import strutils+from mitmproxy.addons.modifyheaders import parse_modify_spec, ModifySpec+++class MapRemote:+    def __init__(self):+        self.replacements: typing.List[ModifySpec] = []++    def load(self, loader):+        loader.add_option(+            "map_remote", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/[@]replacement", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        if "map_remote" in updated:+            self.replacements = []+            for option in ctx.options.map_remote:+                try:+                    spec = parse_modify_spec(option)+                    try:+                        re.compile(spec.subject)+                    except re.error:+                        raise ValueError(f"Invalid regular expression: {spec.subject}")+                except ValueError as e:+                    raise exceptions.OptionsError(+                        f"Cannot parse map_remote option {option}: {e}"+                    ) from e++                self.replacements.append(spec)++    def request(self, flow):+        if not flow.reply.has_message:+            for spec in self.replacements:+                if spec.matches(flow):+                    self.replace(flow.request, spec.subject, spec.replacement)++    def replace(self, obj, search, repl):+        """+        Replaces all matches of the regex search in the url of the request with repl.++        Returns:+            The number of replacements made.+        """+        if repl.startswith(b"@"):+            path = os.path.expanduser(repl[1:])+            try:+                with open(path, "rb") as f:+                    repl = f.read()+            except IOError:+                ctx.log.warn("Could not read replacement file: %s" % repl)+                return++        replacements = 0+        obj.url, replacements = re.subn(search, repl, strutils.escaped_str_to_bytes(obj.pretty_url), flags=re.DOTALL)

The URL is surrogate-escaped UTF-8, so we should do obj.pretty_url.encode("utf8", "surrogateescape"). We're doing a bit of a weird encoding dance here that's tricky to explain, but escaped_str_to_bytes would interpret escape sequences in a URL as raw bytes, which would fail in some edge cases.

mplattner

comment created time in 5 days

push eventmplattner/mitmproxy

Maximilian Hils

commit sha 417c213dd720312ba554c5bb7c258fe65fe4e52a

minor docs improvements

view details

push time in 5 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha 45ffad8acd420c49017957593acd410ecd32c665

[sans-io] h1: raise on HTTP trailers

view details

push time in 5 days

issue commentmitmproxy/mitmproxy

Unify Replacement UX

For the map addon we plan to have two separate addons, MapRemote and MapLocal, right?

Yes! I think these are very distinct use-cases.

Regarding MapRemote, I thought it'd be nice to intercept the request before it is sent to the server, such that the "original" resource isn't (wastefully) requested, before it is replaced with the response of the replaced URL. I looked at eventsequence.Events and there does not seem to be something like "before-request". Am I following the wrong route here?

request triggers once we have received the request, before it is sent upstream. We essentially want to do this:

https://github.com/mitmproxy/mitmproxy/blob/73163c876c7bd50da3c73c67c1a41d26eae575ef/examples/addons/http-redirect-requests.py#L5-L10

The request is not transmitted to the original host in this example.

For MapLocal, it's a bit different:

https://github.com/mitmproxy/mitmproxy/blob/73163c876c7bd50da3c73c67c1a41d26eae575ef/examples/addons/http-reply-from-proxy.py#L6-L11

By setting .response we can avoid the entire upstream side.

Regarding this, another question is whether we want to replace only the body only or the entire response (including headers)?

MapRemote: return whatever upstream sent us, MapLocal: We probably need to make up some mime-type headers? Let's tackle MapRemote first! :)

mhils

comment created time in 6 days

push eventmitmproxy/mitmproxy

Martin Plattner

commit sha 29f0bd02761217db122b0cdccf0f98624cd817ca

replacements addon: only replace in body

view details

Martin Plattner

commit sha b608b0ef3e8dfa79bc4f6808475321714ac802da

replacements addon: improve namings

view details

Martin Plattner

commit sha 96e756ead0c42d237bbba21855c831b48a63fcb9

remove not needed replacement code

view details

Martin Plattner

commit sha b263b0decef36564b153cd5ef95a7f4fa26d7796

replacements addon: rename to ModifyBody

view details

Martin Plattner

commit sha 4cffbec291a4b672cea86b5351acdca05a46cb2f

modifybody addon: update docs

view details

Martin Plattner

commit sha 48dcc6e073a6097157c405c3e8cd89867d5952b3

revert modify headers parameter order

view details

Martin Plattner

commit sha 0a124793998a4ee52818c4d004ce213e578e3fea

modify addons: cleanup redundant code and error handling

view details

Martin Plattner

commit sha 2ed77707162ce94773bd0e3c03e35ac7e49a856f

modify body addon: fix typo in docs

view details

Martin Plattner

commit sha 2a408c93797d0ef22eb36a4db56e2d9191585ca5

refactor modify addons Use a universal ModifySpec class to represent rules. ModifyHeaders now supports reading the header value from a file.

view details

Maximilian Hils

commit sha 2a4933eb9deb0c82c3ddf2b821cae88219c755a7

minor fixes

view details

Martin Plattner

commit sha aee1b812600b4f95472cf39b3973b6a5f4f971ed

fix test coverage

view details

Martin Plattner

commit sha a782bb47a776ff74de6a3101f3428c73cb1b5204

fix linting error

view details

Martin Plattner

commit sha fe662b42823ce559d92f7239dd287e2465732d85

improve modify addons docs fixes #3945

view details

Maximilian Hils

commit sha 73163c876c7bd50da3c73c67c1a41d26eae575ef

Merge pull request #4046 from mplattner/modify-body-addon Replace replacements addon with new modify body addon

view details

push time in 6 days

PR closed mitmproxy/mitmproxy

Improve replacements docs

I thought #3940 was about using files to replace bodies, but it turns out replacements already does that. I completely missed this multiple times because the docs are not meant for scanning. But nobody actually reads walls of text and some sentences are incredibly long.

  1. That patt is optional was completely missing
  2. That replacement can be a @file_path is now completely obvious and right in your face without reading any of the text
  3. I broke up the wall of text so that you don't have to actually read the thing if you're just looking for a particular thing

Now your brain can go pew pew "patt -> filter expression", nice gotcha. "replace -> string literal" neat. You get the idea.

+13 -6

1 comment

1 changed file

Prinzhorn

pr closed time in 6 days

PR merged mitmproxy/mitmproxy

Replace replacements addon with new modify body addon

The new addon only replaces data in the body using --modify-body.

refs #3948

+489 -567

6 comments

22 changed files

mplattner

pr closed time in 6 days

pull request commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

🎉 🍰

mplattner

comment created time in 6 days

pull request commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

(@Kriechi's draconian individual coverage checks are still complaining, but that should be a very easy fix)

mplattner

comment created time in 7 days

pull request commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

Awesome! Next stop: map remote editor 🎉

mplattner

comment created time in 7 days

push eventmplattner/mitmproxy

Maximilian Hils

commit sha 2a4933eb9deb0c82c3ddf2b821cae88219c755a7

minor fixes

view details

push time in 7 days

pull request commentmitmproxy/mitmproxy

Update changelog and add PR template

Fair enough. I've reduced the word count a bit, does it work for you this way?

Kriechi

comment created time in 7 days

push eventKriechi/mitmproxy

Maximilian Hils

commit sha 5169962fdfa1ffcda7d174f3259105457978aedd

alternative PR template wording proposal

view details

push time in 7 days

pull request commentmitmproxy/mitmproxy

Update changelog and add PR template

I'm a bit torn on this to be honest. This is nice for a slightly easier release process, but it also adds more friction and iteration to PRs. We can try this, but I feel like it's a PITA to have PRs which are ready to merge but miss a changelog entry. Is going through the commit log once really that bad?

Kriechi

comment created time in 7 days

push eventmitmproxy/mitmproxy

Maximilian Hils

commit sha 89670a19baba7a0a687e0915d170c16485afe155

[sans-io] h2 client (wip)

view details

push time in 8 days

issue openedmitmproxy/mitmproxy

How to handle conflicting HTTP Host information?

Let's assume the following HTTP/1 exchange with a regular HTTP proxy:

CONNECT example.com:80 HTTP/1.1
Host: example.org:80 

HTTP/1.1 200 OK

GET http://foo.bar/ HTTP/1.0
Host: qux.dev
Host: qux.xyz

In this example, we observe five conflicting destinations: example.com, example.org, foo.bar, and qux.dev, and qux.xyz. This is of course a bit contrived, but should illustrate the problem. For a more practical example, let's think of CONNECT + an HTTP/2 request with :authority and host headers.

Current Behavior

We first ignore the Host header in the CONNECT request entirely. Then we fail because we receive an absolute-form request while expecting a relative-form one. Absolute-form is valid HTTP, so we shouldn't continue doing this.

If we assume GET / HTTP/1.1 instead, mitmproxy will set .host is example.com and .pretty_host to qux.dev, qux.xyz.

Desired Behavior

I think some of these problems are relatively easy:

  • Host header in CONNECT requests: Host headers never leave the proxy here, we can just keep ignoring these.
  • Conflicting Host headers: This problem is a bit of a tangent here - we can just forward those as-is if other host information is present.

This leaves us with example.com, foo.bar and qux.dev. What do we want to do here? Some ideas:

  • Introduce an additional request.authority attribute. .pretty_host would then try .authority, host header and .host (in that order). .host would update .authority if it is present.
  • Raise if :authority and Host are both present and do not match. A bit of an open question is how we determine what to send upstream?
  • Use .host for :authority. Among other things, this'd mean that we have spoofable information in .host. Not what we want I guess.

@mitmproxy/devs, any ideas?

See also:

  • https://www.icir.org/vern/papers/host-of-troubles.ccs16.pdf
  • https://github.com/nghttp2/nghttp2/issues/317

refs #848

created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 conversation, where requests may have been made concurrently. You may want to use client-side replay in conjunction with the `anticache` option, to make sure the server responds with complete data. -## Proxy Authentication--Asks the user for authentication before they are permitted to use the proxy.-Authentication headers are stripped from the flows, so they are not passed to-upstream servers. For now, only HTTP Basic authentication is supported. The-proxy auth options are not compatible with the transparent, socks or reverse-proxy mode.- -## Replacements+## Modify Body -The `replacements` option lets you specify an arbitrary number of patterns that-define text replacements within flows. A replacement pattern looks like this:+The `modify-body` option lets you specify an arbitrary number of patterns that

Agreed, that's actually a really good point! Let's track this in #4054.

mplattner

comment created time in 8 days

issue openedmitmproxy/mitmproxy

Inform the user when underscore-formatted options are used as a cli argument

It might make sense to inform the user when underscore-formatted options are used as a cli argument, similar to the messages on deprecated commands. At the moment it outputs: mitmproxy: error: unrecognized arguments: --modify_body asdf.

Originally posted by @mplattner in https://github.com/mitmproxy/mitmproxy/pull/4046

created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

+import os+import re+import typing++from mitmproxy import exceptions+from mitmproxy import flowfilter+from mitmproxy import ctx+from mitmproxy.addons.modifyheaders import parse_modify_hook+++class ModifyBody:+    def __init__(self):+        self.lst = []++    def load(self, loader):+        loader.add_option(+            "modify_body", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/[@]replacement", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        """+            .modify_body is a list of tuples (flow_filter_pattern, regex, repl):++            flow_filter_pattern: a string specifying a flow filter pattern.+            regex: a regular expression, as string.+            repl: the replacement string+        """+        if "modify_body" in updated:+            lst = []+            for rep in ctx.options.modify_body:+                try:+                    flow_filter_pattern, regex, repl = parse_modify_hook(rep)+                except ValueError as e:+                    raise exceptions.OptionsError(+                        "Invalid modify_body option: %s" % rep

Make this f"Cannot parse modify_body option {rep}: {e}" to include the ValueError's message, then move the checks below into parse_modify_hook and make their messages generic ("Invalid modify_body flow filter: %s" % flow_filter_pattern -> Invalid filter pattern: {pattern}). This should solve the problem mentioned in https://github.com/mitmproxy/mitmproxy/pull/4046#discussion_r447724841, or am I misunderstanding things? :)

mplattner

comment created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

  from mitmproxy import exceptions from mitmproxy import flowfilter+from mitmproxy.utils import strutils from mitmproxy import ctx  -def parse_modify_headers(s):+def parse_modify_hook(s):     """-        Returns a (header_name, header_value, flow_filter) tuple.+        Returns a (flow_filter, header_name, header_value) tuple.

If we re-use this (which is good 👍), we should probably signal somehow that it's not just headers. We can also add a bit more type information to make things easier to grasp:

class ModifySpec(typing.NamedTuple):
    matches: flowfilter.TFilter,
    match_str: str
    key: bytes  # those could potentially have better names if you can think of some
    replacement: bytes


def parse_modify_spec(s: str) -> ModifySpec:
    # ...
    filter = flowfilter.parse(filter_str)
    return ModifySpec(filter, filter_str, header_name, header_value)

then, below:

class ModifyHeaders:
    def __init__(self):
        replacements: List[ModifySpec] = []    # replaces self.lst
    
    # ...
    
    def run(self, f, hdrs):
        for spec in self.replacements:
            if spec.matches(f):
                hdrs.pop(spec.key, None)

I realize this is just a very rough sketch. Does it make sense anyway?

mplattner

comment created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

+import os+import re+import typing++from mitmproxy import exceptions+from mitmproxy import flowfilter+from mitmproxy import ctx+from mitmproxy.addons.modifyheaders import parse_modify_hook+++class ModifyBody:+    def __init__(self):+        self.lst = []++    def load(self, loader):+        loader.add_option(+            "modify_body", typing.Sequence[str], [],+            """+            Replacement pattern of the form "[/flow-filter]/regex/[@]replacement", where+            the separator can be any character. The @ allows to provide a file path that+            is used to read the replacement string.+            """+        )++    def configure(self, updated):+        """+            .modify_body is a list of tuples (flow_filter_pattern, regex, repl):++            flow_filter_pattern: a string specifying a flow filter pattern.+            regex: a regular expression, as string.+            repl: the replacement string+        """

Let me be a bit pendantic here - this comment doesn't make too much sense to me:

  • .modify_body would indicate that there's a modify_body attribute. As far as I can see, there isn't?
  • If we want to document the type of .lst (that thing could probably be renamed, yes...), it would make sense to do this at the definition and not here. I'll add some ideas on how we can do this better in the next comment.
mplattner

comment created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 import pytest  from mitmproxy.addons import modifybody+from mitmproxy.addons.modifyheaders import parse_modify_hook from mitmproxy.test import taddons from mitmproxy.test import tflow   class TestModifyBody:-    def test_parse_modify_body(self):-        x = modifybody.parse_modify_body("/foo/bar/voing")-        assert x == ("foo", "bar", "voing")-        x = modifybody.parse_modify_body("/foo/bar/vo/ing/")-        assert x == ("foo", "bar", "vo/ing/")-        x = modifybody.parse_modify_body("/bar/voing")-        assert x == (".*", "bar", "voing")-        with pytest.raises(Exception, match="Invalid modify_body specifier"):-            modifybody.parse_modify_body("/")+    def test_parse_modify_hook(self):+        x = parse_modify_hook("/foo/bar/voing")+        assert x == ("foo", b"bar", b"voing")+        x = parse_modify_hook("/foo/bar/vo/ing/")+        assert x == ("foo", b"bar", b"vo/ing/")+        x = parse_modify_hook("/bar/voing")+        assert x == (".*", b"bar", b"voing")+        with pytest.raises(Exception, match="Invalid modify_\\* specifier"):+            parse_modify_hook("/")      def test_configure(self):         mb = modifybody.ModifyBody()         with taddons.context(mb) as tctx:             tctx.configure(mb, modify_body=["one/two/three"])

Generally speaking, having this line here asserts that it runs without error. In an ideal world, you would probably also check that the addon has parsed it properly, but "doesn't error" can sometimes be good enough.

mplattner

comment created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 conversation, where requests may have been made concurrently. You may want to use client-side replay in conjunction with the `anticache` option, to make sure the server responds with complete data. -## Proxy Authentication--Asks the user for authentication before they are permitted to use the proxy.-Authentication headers are stripped from the flows, so they are not passed to-upstream servers. For now, only HTTP Basic authentication is supported. The-proxy auth options are not compatible with the transparent, socks or reverse-proxy mode.- -## Replacements+## Modify Body -The `replacements` option lets you specify an arbitrary number of patterns that-define text replacements within flows. A replacement pattern looks like this:+The `modify-body` option lets you specify an arbitrary number of patterns that

(if you have any ideas on how we could have this more consistent, I'd love to hear them - what we currently have is the best solution we came up with so far)

mplattner

comment created time in 8 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 conversation, where requests may have been made concurrently. You may want to use client-side replay in conjunction with the `anticache` option, to make sure the server responds with complete data. -## Proxy Authentication--Asks the user for authentication before they are permitted to use the proxy.-Authentication headers are stripped from the flows, so they are not passed to-upstream servers. For now, only HTTP Basic authentication is supported. The-proxy auth options are not compatible with the transparent, socks or reverse-proxy mode.- -## Replacements+## Modify Body -The `replacements` option lets you specify an arbitrary number of patterns that-define text replacements within flows. A replacement pattern looks like this:+The `modify-body` option lets you specify an arbitrary number of patterns that

We auto-translate _ to - for command line arguments, yes. However, modify_body is what you would set in your config.yaml or in mitmproxy's options editor, and what we consistently use in our docs (see https://docs.mitmproxy.org/stable/concepts-options/). You could also do --set modify_body=... for that matter. The inconsistency is a bit ugly, but there really aren't many better options around.

mplattner

comment created time in 8 days

pull request commentmitmproxy/mitmproxy

New File has been renamed to clear all

Thanks! :)

yogeshojha

comment created time in 9 days

push eventmitmproxy/mitmproxy

Yogesh Ojha

commit sha 285b1f5feff504ed2d0647374ac40861ba1d0f2e

New file renamed to Clear all

view details

Yogesh Ojha

commit sha 0be9dc305b4dad0d97c025cf43cfa6b15b12970e

minified

view details

Yogesh Ojha

commit sha 936f349bf7d2d6c6f7ad9c9e5ce7ff3674236cc7

styles/scripts vendor minified

view details

Yogesh Ojha

commit sha 4863825737e6ca62a614b045c95d47d30bf084e5

ran npm test

view details

Maximilian Hils

commit sha b9108b271e43fe9447fd30802619880443558dd0

Merge pull request #4051 from yogeshojha/issue/clearall New File has been renamed to clear all

view details

push time in 9 days

PR merged mitmproxy/mitmproxy

New File has been renamed to clear all

Issue #4025, #3665 has been fixed by changing the New File to Clear All

Screenshot 2020-06-29 at 6 18 18 PM

+5 -5

5 comments

3 changed files

yogeshojha

pr closed time in 9 days

pull request commentmitmproxy/mitmproxy

New File has been renamed to clear all

Yes, that should go away with your next push. I'm talking about https://github.com/mitmproxy/mitmproxy/pull/4051/checks?check_run_id=818305276#step:8:189.

yogeshojha

comment created time in 9 days

issue commentmitmproxy/mitmproxy

Addons in mitmweb stops working when an addon adds a boolean option and that option is set in config.yml

Thanks for the clean minimal example! This definitely looks like a bug. Is mitmdump/mitmproxy affected, too?

andreas-wolf

comment created time in 9 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

+import os+import re+import typing++from mitmproxy import exceptions+from mitmproxy import flowfilter+from mitmproxy import ctx+from mitmproxy.utils import strutils+++def parse_modify_body(s):

Have the function in one addon file and import it in the other?

yup, let's keep it simple. :)

You mean a ValueError in the parse method and an OptionsError in the Configure method? Does that make sense here, i.e., having 2 subsequently thrown exceptions for the same error? (Trying to learn here, not to correct you :))

Yes. The very Pythonic way would be to re-raise as follows:

def foo():
    raise ValueError("qux")

def bar():
    try:
        foo()
    except ValueError as e:
        raise BarError("cannot foo() today") from e

This ensure the following: 1) You can make sure that bar() only raises BarErrors. 2) You get a very nice chained traceback:

Traceback (most recent call last):
  File "<pyshell#55>", line 3, in bar
    foo()
  File "<pyshell#49>", line 2, in foo
    raise ValueError("qux")
ValueError: qux

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<pyshell#56>", line 1, in <module>
    bar()
  File "<pyshell#55>", line 5, in bar
    raise BarError("cannot foo() today") from e
BarError: cannot foo() today

These methods are now applied in replace(). You suggest to move them to the parse method, right?

Yes. Reading from file should stay inside replace() however, it can be very useful that file contents are not cached. :)

mplattner

comment created time in 9 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 def check():      for option in REPLACED.splitlines():         if option in args:+            if isinstance(REPLACEMENTS.get(option), list):+                new_options = REPLACEMENTS.get(option)+            else:+                new_options = [REPLACEMENTS.get(option)]             print(                 "{} is deprecated.\n"                 "Please use `{}` instead.".format(                     option,-                    REPLACEMENTS.get(option)+                    "` or `".join(new_options)

Yeah, I can't read.

mplattner

comment created time in 9 days

pull request commentmitmproxy/mitmproxy

Added OpenSSL.crypto.X509.get_version()

wrong button, oops.

Ronophobia

comment created time in 9 days

PullRequestEvent

pull request commentmitmproxy/mitmproxy

Added OpenSSL.crypto.X509.get_version()

Hi, may I ask what's the purpose of this addition? We don't use .version in mitmproxy code, so that just means more maintenance cost for us.

Ronophobia

comment created time in 9 days

issue closedmitmproxy/mitmproxy

mitmproxy has crashed

Problem Description

test@test-VirtualBox:~$ ./mitmproxy --set block_global=false --ssl-insecure Traceback (most recent call last): File "mitmproxy/master.py", line 86, in run_loop File "urwid/main_loop.py", line 287, in run File "urwid/main_loop.py", line 385, in _run File "urwid/main_loop.py", line 1494, in run File "urwid/compat.py", line 58, in reraise File "asyncio/events.py", line 88, in _run File "urwid/main_loop.py", line 1454, in faux_idle_callback File "urwid/main_loop.py", line 574, in entering_idle File "urwid/main_loop.py", line 588, in draw_screen File "urwid/widget.py", line 145, in cached_render File "urwid/container.py", line 1086, in render File "urwid/widget.py", line 145, in cached_render File "urwid/decoration.py", line 226, in render File "urwid/widget.py", line 145, in cached_render File "urwid/container.py", line 1086, in render File "urwid/widget.py", line 145, in cached_render File "urwid/listbox.py", line 457, in render File "urwid/widget.py", line 145, in cached_render File "urwid/widget.py", line 1761, in render File "urwid/widget.py", line 145, in cached_render File "urwid/container.py", line 1554, in render File "urwid/widget.py", line 145, in cached_render File "urwid/container.py", line 2125, in render File "urwid/widget.py", line 145, in cached_render File "mitmproxy/tools/console/common.py", line 173, in render File "urwid/canvas.py", line 358, in init urwid.canvas.CanvasError: Canvas text is wider than the maxcol specified 15 [23] [b'\xe2\x80\xa6o -3] \xe5\x9f\x9f\xe5\x90\x8d\xe8\xa7\xa3\xe6\x9e\x90\xe6\x9a\x82\xe6\x97\xb6\xe5\xa4\xb1\xe8\xb4\xa5']

mitmproxy has crashed! Please lodge a bug report at: https://github.com/mitmproxy/mitmproxy test@test-VirtualBox:~$

Steps to reproduce the behavior:

  1. there maybe some non-http inside tls in my test.
  2. some https can be dump correctly before crash.

System Information

Paste the output of "mitmproxy --version" here. version 5.1.1 ubuntu 20.04

closed time in 9 days

dragonforeverliu

issue commentmitmproxy/mitmproxy

mitmproxy has crashed

Thanks! This looks like a duplicate of #2894.

dragonforeverliu

comment created time in 9 days

issue closedmitmproxy/mitmproxy

client certs - build for win32 like 4.0.4 instead of win64 add ecc compat

  1. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] Can we get some client certs to be taken in consideration and exclude the rest? Situation is like I added a client cert for 1 domain. The mitmproxy forwards the client cert. In some situations the client has more than 1 client certificates for multiple domains. Can we get an option where other client certificates are pulled directly from the clients browser if we do not add it to mitmproxy client certs folder? Even if the traffic cannot be intercepted, at least the client does not suffer of DoS because, like now, there is no option to let the client to select his certificate for some domain, or for any of them. If the client certificate does not exist selected in mitmproxy then the client has DoS because cannot authenticate in any way. Can an option be made to ignore mitm for any other domain where an client certificate is requested but does not exist selected in the folder of mitm client cert ?in case of client cert requests? Like no interception but, at least, the client does auth succesfully.

  2. Can there be builds for win32 instead of win64? 4.0.4 is only one build for win32 and works for both win32 and win64. There is no effort in changing the build back to win32 and let all users enjoy an easy turnkey mitmproxy build.

  3. Can we all get a Elliptic Curve certificates compatibility?

  4. Question: If the default key size for RSA is 2048, could a 4096 RSA certificate work for some intercepted sites?

closed time in 9 days

alexalwks

issue commentmitmproxy/mitmproxy

client certs - build for win32 like 4.0.4 instead of win64 add ecc compat

Hi,

You're dumping a large list of questions and feature requests here, without putting too much effort into it. You kind of expect us to do your work for free, which is a bit rude. I'm closing this here because it's too much to answer and in many cases already discussed elsewhere.

Please ask future questions on StackOverflow. Thanks!

alexalwks

comment created time in 9 days

pull request commentmitmproxy/mitmproxy

New File has been renamed to clear all

Could you please make sure that the tests pass so that we can get this merged? Thanks!

yogeshojha

comment created time in 9 days

pull request commentmitmproxy/mitmproxy

New File has been renamed to clear all

Awesome, thanks! 🍰 😃

yogeshojha

comment created time in 9 days

issue commentmitmproxy/mitmproxy

mitmweb: Rename File -> New to File -> Clear/Clear All/Clear Flows

Maybe there is a misunderstanding: I'm proposing to rename File -> New to File -> Clear All.

GamiDPC

comment created time in 9 days

issue commentmitmproxy/mitmproxy

mitmweb: Rename File -> New to File -> Clear/Clear All/Clear Flows

yeah this one is a duplicate. Let's just rename the existing button, otherwise one may reasonably ask what the difference between those two is.

GamiDPC

comment created time in 9 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

+import os+import re+import typing++from mitmproxy import exceptions+from mitmproxy import flowfilter+from mitmproxy import ctx+from mitmproxy.utils import strutils+++def parse_modify_body(s):

This looks suspiciously similar to parse_modify_headers now. We can probably just merge those two?

  • We can raise a plain ValueError here and re-raise an OptionsError in each addon with an individual error message.
  • Let's also include strutils.escaped_str_to_bytes() and flowfilter.parse here.
mplattner

comment created time in 9 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 conversation, where requests may have been made concurrently. You may want to use client-side replay in conjunction with the `anticache` option, to make sure the server responds with complete data. -## Proxy Authentication--Asks the user for authentication before they are permitted to use the proxy.-Authentication headers are stripped from the flows, so they are not passed to-upstream servers. For now, only HTTP Basic authentication is supported. The-proxy auth options are not compatible with the transparent, socks or reverse-proxy mode.- -## Replacements+## Modify Body -The `replacements` option lets you specify an arbitrary number of patterns that-define text replacements within flows. A replacement pattern looks like this:+The `modify-body` option lets you specify an arbitrary number of patterns that

modify-body -> modify_body

mplattner

comment created time in 9 days

Pull request review commentmitmproxy/mitmproxy

Replace replacements addon with new modify body addon

 def check():      for option in REPLACED.splitlines():         if option in args:+            if isinstance(REPLACEMENTS.get(option), list):+                new_options = REPLACEMENTS.get(option)+            else:+                new_options = [REPLACEMENTS.get(option)]             print(                 "{} is deprecated.\n"                 "Please use `{}` instead.".format(                     option,-                    REPLACEMENTS.get(option)+                    "` or `".join(new_options)

This fails for the existing options, see e.g. " or ".join("--intercept").

mplattner

comment created time in 9 days

create barnchmhils/pdf-presenter

branch : master

created branch time in 12 days

created repositorymhils/pdf-presenter

an in-browser PDF rendering + pressure-sensitive annotation proof-of-concept

created time in 12 days

delete branch mhils/mitmproxy

delete branch : sans-io-adjustments

delete time in 12 days

more