profile
viewpoint

iSECPartners/yontma-mac 228

You'll Never Take Me Alive!

ghacksuserjs/TorZillaPrint 35

Firefox & Tor Browser only, current ESR+ only, client-side only [for now], results only, fingerprint testing

tomrittervg/ct-tools 23

Random tools to perform Certificate Transparency-related stuff

IncludeSecurity/RTSPhuzz 11

RTSPhuzz - An RTSP Fuzzer written using the Boofuzz framework

mozilla/foundation-security-advisories 11

Canonical source for Mozilla Foundation Security Advisories. http://www.mozilla.org/security/announce/

mozilla-services/third-party-library-alert 8

A tool to alert when third party libraries embedded in the FF source code go out of date.

push eventmozilla-services/updatebot

Tom Ritter

commit sha fc8f4b8821234b6ce53d684b47053ba3d9b3e6e5

Refactor the functionality test to de-duplicate a ton of code

view details

push time in 4 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha d251b326f12b5d79cd883829d6914f591f01dfa9

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha 00951468816d0d8774a3b9ba22fc58c0bdeb97bd

Get rid the notion of a 'project' - our project is try.

view details

Tom Ritter

commit sha 61b43ded8345992e2f3b7e2abe46c06e13461696

Strip the newline off the end of the library version identifier

view details

Tom Ritter

commit sha 3c049de90b6f88b04ad82fc0c75d6cdae0648ed9

Refactor the phabricator output matching to be more robust

view details

Tom Ritter

commit sha 013d4adc2163df3a4f52a4121451ce6e790d08a8

Process Existing Job

view details

Tom Ritter

commit sha 4722cb250f121d02a3d72121b0a2a2cb492aa1d3

Add a testExistingJobSucceeded test

view details

Tom Ritter

commit sha 886832f4907112da2f4f0f9053319122c967d9ce

Refactor JOBSTATUS into two separate parts: Status and Outcome Status is what updatebot needs to do for it next. Outcome is how the job completed. Outcome is really needed more for testing purposes than anything else because in theory we won't care how it ended as long as it ended but that's a short-sighted view because in the future we will definetly want to display that information on a dashboard.

view details

Tom Ritter

commit sha 8ef4d46a8d2fcad833d52ae8c8a88b6cc3ceb215

Add a test for a build failure

view details

Tom Ritter

commit sha ff4609c5d7b5b11118a014c8c976fc632078c69b

Add a test for a test failing that's classified intrmittent.

view details

Tom Ritter

commit sha 02748fe7db3b28b87edccfbb9525f3dbc6a884b8

Improve the MockTreeherderServer to allow unlimited responses for a url

view details

Tom Ritter

commit sha 8be8c296f7fc8f9c143e347fdb204937f3b5021e

Add a test for retriggers and unclassified failures.

view details

Tom Ritter

commit sha e2dfad759b298e7cb676ae119adcc7c93a904e72

Refactor the functionality test to de-duplicate a ton of code

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha d251b326f12b5d79cd883829d6914f591f01dfa9

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha 00951468816d0d8774a3b9ba22fc58c0bdeb97bd

Get rid the notion of a 'project' - our project is try.

view details

Tom Ritter

commit sha 61b43ded8345992e2f3b7e2abe46c06e13461696

Strip the newline off the end of the library version identifier

view details

Tom Ritter

commit sha 3c049de90b6f88b04ad82fc0c75d6cdae0648ed9

Refactor the phabricator output matching to be more robust

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha e2ca10af5ba67939fbe8d56dc858f1bc214864e3

Refactor the functionality test to de-duplicate a ton of code

view details

push time in 6 days

pull request commentmozilla-services/updatebot

2020 07 30 existing job

Oh; this patch series contains all the commits from 07-14 as well - so once that PR lands I will rebase this if needed and then the diff will make more sense.

tomrittervg

comment created time in 6 days

PR opened mozilla-services/updatebot

Reviewers
2020 07 30 existing job

Here is the big one where we add all the actual logic for processing an existing job.

I've tested this on try a little bit - specifically the 'Broken Build' case because.... that's what happens presently when you try to vendor dav1d. The other scenarios.... well they're probably broken in some subtle way I don't see right now and can be fixed up in a future commit?

+2286 -106

0 comment

25 changed files

pr created time in 6 days

pull request commentmozilla-services/updatebot

2020 07 14 existing job

Okay I think this will pass. I added a few more tiny patches to the end of the series as well.

tomrittervg

comment created time in 6 days

Pull request review commentmozilla-services/updatebot

2020 07 14 existing job

 def testFunctionalityWithRealDatabase(self):             # Not Mocked At All             'Phabricator': PhabricatorProvider,         }-        u = Updatebot(configs, providers)++    def testAllNewJobs(self):+        # Setup+        try_revision = "try_rev"  # Doesn't matter+        expected_values = DEFAULT_EXPECTED_VALUES(try_revision)+        self.configs['Bugzilla']['filed_bug_id'] = expected_values.filed_bug_id++        command_mappings = {+            "./mach vendor": expected_values.library_version_id,+            "./mach try auto": TRY_OUTPUT(try_revision),+            "arc diff --verbatim": ARC_OUTPUT+        }+        self.configs['Command']['test_mappings'] = command_mappings++        # Make it+        u = Updatebot(self.configs, self.providers)++        # Ensure we don't have a dirty database with existing jobs

I noodled around on this a bit, wrote a patch, threw it out, wrote a SO question, threw it out, and finally came up with something I think is an improvement. But it'll be a real bitch to rebase into here, so I'd like to leave it to the end of the next patchset where I add like 4 more tests to this file.

tomrittervg

comment created time in 6 days

Pull request review commentmozilla-services/updatebot

2020 07 14 existing job

 def submit_to_try(self, library):             raise Exception("Could not find the try link in output:\n" + output)         try_link = try_link[try_link.index("#/jobs?repo=try&revision=") + len("#/jobs?repo=try&revision="):]         return try_link++    # =================================================================+    # =================================================================++    @staticmethod+    def _transform_job_list(property_names, job_list):+        new_job_list = []+        for j in job_list:+            d = {}+            for i in range(len(property_names)):+                d[property_names[i]] = j[i]+            new_job_list.append(Struct(**d))+        return new_job_list++    # =================================================================++    def _get_failure_classifications(self):+        if not self._failure_classifications:+            self.logger.log("Requesting failure classifications", level=LogLevel.Info)+            r = requests.get(self.url_treeherder + "/api/failureclassification", headers=self.HEADERS)+            j = r.json()++            failureclassifications = {}+            for f in j:+                failureclassifications[f['id']] = f['name']+            self._failure_classifications = failureclassifications+        return self._failure_classifications++    def _set_failure_classifications(self, v):+        self._failure_classifications = v++    def _del_failure_classifications(self):+        del self._failure_classifications++    failure_classifications = property(_get_failure_classifications, _set_failure_classifications, _del_failure_classifications)++    # =================================================================+    # =================================================================++    @logEntryExit+    def get_job_details(self, revision):+        push_list_url = self.url_treeherder + "/api/" + self.project + "push/?revision=%s" % revision+        self.logger.log("Requesting revision %s from %s" % (revision, push_list_url), level=LogLevel.Info)++        r = requests.get(push_list_url, headers=self.HEADERS)+        push_list = r.json()+        try:+            push_id = push_list['results']['id']+        except Exception as e:+            raise Exception("Could not find the expected ['results']['id'] from %s" % r.text) from e++        job_list = []+        property_names = []+        job_details_url = self.url_treeherder + "/api/jobs/?push_id=%s" % push_id+        try:+            while job_details_url:+                self.logger.log("Requesting push id %s from %s" % (push_id, job_details_url), level=LogLevel.Info)+                r = requests.get(job_details_url, headers=self.HEADERS)+                j = r.json()++                job_list.extend(j['results'])+                if not property_names:+                    property_names = j['job_property_names']+                else:+                    for i in range(len(property_names)):+                        if len(property_names) != len(j['job_property_names']):+                            raise Exception("The first j['job_property_names'] was %i elements long, but a subsequant one was %i for url %s" % (len(property_names), len(j['job_property_names']), job_details_url))+                        elif property_names[i] != j['job_property_names'][i]:+                            raise Exception("Property name %s (index %i) doesn't match %s" % (property_names[i], i, j['job_property_names'][i]))++                job_details_url = j['next'] if 'next' in j else None+        except Exception as e:+            raise Exception("Could not obtain all the job results for push id %s" % push_id) from e++        new_job_list = TaskclusterProvider._transform_job_list(property_names, job_list)++        return new_job_list++    # =================================================================+    # =================================================================++    @logEntryExitNoArgs+    def retrigger_jobs(self, job_list, retrigger_list):+        # Go through the jobs and find the decision task+        decision_task = None+        for j in job_list:+            if "Gecko Decision Task" == j.job_type_name:+                decision_task = j+                break+        assert decision_task is not None++        # Download its actions.json+        artifact_url = self.url_taskcluster + "api/queue/v1/task/%s/runs/0/artifacts/public/actions.json" % (decision_task.task_id)+        r = requests.get(artifact_url, headers=self.HEADERS)+        actions = r.json()

I didn't add it in a separate commit, instead I added it in-line

tomrittervg

comment created time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 081a4b78827cce6c436ceaa1d721a470075a8059

Add Treeherder APIs for retrieving and transforming job details

view details

Tom Ritter

commit sha d910170338fe497d0f9c8e9305820ed03f90fecb

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha f562cfe1c6df676dcd6e4bc545286c0b5ad931ec

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

Tom Ritter

commit sha cb219fb862a818f29a009595cb5cebbb7ea560c2

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha 10052d4f5f7436b1321a34c38b3d99d33220e4df

Get rid the notion of a 'project' - our project is try.

view details

Tom Ritter

commit sha 4c7a36cde19f03bceabb6309e220c7c82e159c1c

Strip the newline off the end of the library version identifier

view details

Tom Ritter

commit sha f0ebe96f4a666b10f2818ab75d7017db6d95f2df

Refactor the phabricator output matching to be more robust

view details

Tom Ritter

commit sha 73df7a952876d72b7f95a5193a48972b4d8bd884

Process Existing Job

view details

Tom Ritter

commit sha 0ed9bcfb7293b4fa669e8eecd3b962e9c5d4db38

Add a testExistingJobSucceeded test

view details

Tom Ritter

commit sha 13f7a459de1968b782a3602ca68b1211dbb85571

Refactor JOBSTATUS into two separate parts: Status and Outcome Status is what updatebot needs to do for it next. Outcome is how the job completed. Outcome is really needed more for testing purposes than anything else because in theory we won't care how it ended as long as it ended but that's a short-sighted view because in the future we will definetly want to display that information on a dashboard.

view details

Tom Ritter

commit sha 635b320913d3d7bbb304dfc7df8af08aa77c7970

Add a test for a build failure

view details

Tom Ritter

commit sha 82292ced1a5d949ecc7211a37da9e3087cd56f96

Add a test for a test failing that's classified intrmittent.

view details

Tom Ritter

commit sha 742d124d6234befe8051f98e30c7a50698d34f33

Improve the MockTreeherderServer to allow unlimited responses for a url

view details

Tom Ritter

commit sha 1feeeaaa7a958aff540fee8a9eead5b84a270a38

Add a test for retriggers and unclassified failures.

view details

Tom Ritter

commit sha 13df0d11afdd23b1a4fe913a83d998f14a1e6c91

Refactor the functionality test to de-duplicate a ton of code

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 081a4b78827cce6c436ceaa1d721a470075a8059

Add Treeherder APIs for retrieving and transforming job details

view details

Tom Ritter

commit sha d910170338fe497d0f9c8e9305820ed03f90fecb

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha f562cfe1c6df676dcd6e4bc545286c0b5ad931ec

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

Tom Ritter

commit sha cb219fb862a818f29a009595cb5cebbb7ea560c2

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha 10052d4f5f7436b1321a34c38b3d99d33220e4df

Get rid the notion of a 'project' - our project is try.

view details

Tom Ritter

commit sha 4c7a36cde19f03bceabb6309e220c7c82e159c1c

Strip the newline off the end of the library version identifier

view details

Tom Ritter

commit sha f0ebe96f4a666b10f2818ab75d7017db6d95f2df

Refactor the phabricator output matching to be more robust

view details

push time in 6 days

Pull request review commentmozilla-services/updatebot

2020 07 14 existing job

 def submit_to_try(self, library):             raise Exception("Could not find the try link in output:\n" + output)         try_link = try_link[try_link.index("#/jobs?repo=try&revision=") + len("#/jobs?repo=try&revision="):]         return try_link++    # =================================================================+    # =================================================================++    @staticmethod+    def _transform_job_list(property_names, job_list):+        new_job_list = []+        for j in job_list:+            d = {}+            for i in range(len(property_names)):+                d[property_names[i]] = j[i]+            new_job_list.append(Struct(**d))+        return new_job_list++    # =================================================================++    def _get_failure_classifications(self):+        if not self._failure_classifications:+            self.logger.log("Requesting failure classifications", level=LogLevel.Info)+            r = requests.get(self.url_treeherder + "/api/failureclassification", headers=self.HEADERS)+            j = r.json()++            failureclassifications = {}+            for f in j:+                failureclassifications[f['id']] = f['name']+            self._failure_classifications = failureclassifications+        return self._failure_classifications++    def _set_failure_classifications(self, v):+        self._failure_classifications = v++    def _del_failure_classifications(self):+        del self._failure_classifications++    failure_classifications = property(_get_failure_classifications, _set_failure_classifications, _del_failure_classifications)++    # =================================================================+    # =================================================================++    @logEntryExit+    def get_job_details(self, revision):+        push_list_url = self.url_treeherder + "/api/" + self.project + "push/?revision=%s" % revision+        self.logger.log("Requesting revision %s from %s" % (revision, push_list_url), level=LogLevel.Info)++        r = requests.get(push_list_url, headers=self.HEADERS)+        push_list = r.json()+        try:+            push_id = push_list['results']['id']+        except Exception as e:+            raise Exception("Could not find the expected ['results']['id'] from %s" % r.text) from e++        job_list = []+        property_names = []+        job_details_url = self.url_treeherder + "/api/jobs/?push_id=%s" % push_id+        try:+            while job_details_url:+                self.logger.log("Requesting push id %s from %s" % (push_id, job_details_url), level=LogLevel.Info)+                r = requests.get(job_details_url, headers=self.HEADERS)+                j = r.json()++                job_list.extend(j['results'])+                if not property_names:+                    property_names = j['job_property_names']+                else:+                    for i in range(len(property_names)):+                        if len(property_names) != len(j['job_property_names']):+                            raise Exception("The first j['job_property_names'] was %i elements long, but a subsequant one was %i for url %s" % (len(property_names), len(j['job_property_names']), job_details_url))+                        elif property_names[i] != j['job_property_names'][i]:+                            raise Exception("Property name %s (index %i) doesn't match %s" % (property_names[i], i, j['job_property_names'][i]))++                job_details_url = j['next'] if 'next' in j else None+        except Exception as e:+            raise Exception("Could not obtain all the job results for push id %s" % push_id) from e++        new_job_list = TaskclusterProvider._transform_job_list(property_names, job_list)++        return new_job_list++    # =================================================================+    # =================================================================++    @logEntryExitNoArgs+    def retrigger_jobs(self, job_list, retrigger_list):+        # Go through the jobs and find the decision task+        decision_task = None+        for j in job_list:+            if "Gecko Decision Task" == j.job_type_name:+                decision_task = j+                break+        assert decision_task is not None++        # Download its actions.json+        artifact_url = self.url_taskcluster + "api/queue/v1/task/%s/runs/0/artifacts/public/actions.json" % (decision_task.task_id)+        r = requests.get(artifact_url, headers=self.HEADERS)+        actions = r.json()

Yeah this is the behavior we get when we get a 404 or other HTTP error. I'm going to add a commit to handle this type of error all over the place....

tomrittervg

comment created time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 131225149d0c7bf32459c09843e772d8f1c7f13d

Fix failure classification

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 76a37a279b6405f3b346d1055acf77ccfdc28ee7

Results is a list

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 9421e60f553a0efc1681e2d3a9b3a24c6657380d

Fix taskcluster api route

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 3100896f848a8895481b97d314d824e57a7dee50

Get rid the notion of a 'project' - our project is try.

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha e85349c53881f1343026ecf83eddec95f61ac1fa

Refactor the phabricator output matching to be more robust

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha ceae2d1d5d885aeb222435475103928a10812b53

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha 82fc2ba1846c4d583a137538160573c1997c7b26

Process Existing Job

view details

Tom Ritter

commit sha ecceacbe59c8e1ffb8787941c580702b09b0ef4f

Add a testExistingJobSucceeded test

view details

Tom Ritter

commit sha 9699f3de9e95e1812b107ad596d75ca9086e7d43

Refactor JOBSTATUS into two separate parts: Status and Outcome Status is what updatebot needs to do for it next. Outcome is how the job completed. Outcome is really needed more for testing purposes than anything else because in theory we won't care how it ended as long as it ended but that's a short-sighted view because in the future we will definetly want to display that information on a dashboard.

view details

Tom Ritter

commit sha 76954dda572a53dab1189cbe62b1ed684c4a9cf7

Add a test for a build failure

view details

Tom Ritter

commit sha a78b64517e4d51d18366127dcd832ddba7e86d12

Add a test for a test failing that's classified intrmittent.

view details

Tom Ritter

commit sha c25a7eeccc50d07984b5638d60957a56ab389e4f

Improve the MockTreeherderServer to allow unlimited responses for a url

view details

Tom Ritter

commit sha a0bbe0eb33741008923e4b925b5cb218033c3864

Add a test for retriggers and unclassified failures.

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha ceae2d1d5d885aeb222435475103928a10812b53

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha b74215a345176289a1f264009d136f4c81eb1e37

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha 0d1145a05eed56d2e2464874099988de23fee404

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

Tom Ritter

commit sha e33c6009c600b9e21de5977230c6b7c7adf5d23a

Process Existing Job

view details

Tom Ritter

commit sha 06387fcd36496cf995e20bd7a8139af57d0417a5

Add a testExistingJobSucceeded test

view details

Tom Ritter

commit sha c29854a43f9c0857637ab787688cf3e6dc6dd294

Refactor JOBSTATUS into two separate parts: Status and Outcome Status is what updatebot needs to do for it next. Outcome is how the job completed. Outcome is really needed more for testing purposes than anything else because in theory we won't care how it ended as long as it ended but that's a short-sighted view because in the future we will definetly want to display that information on a dashboard.

view details

Tom Ritter

commit sha 6c8087380c7ee75f05ecf26ba50e5734bf0353ce

Add a test for a build failure

view details

Tom Ritter

commit sha 8ba41ee5f4b3e223d56d64c8acae95c5db8a63cc

Add a test for a test failing that's classified intrmittent.

view details

Tom Ritter

commit sha 135704a31d728fd741c412dcba8a96d92d119431

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha f2200127562d2905843bf23874f2bbb3a9cf8789

Improve the MockTreeherderServer to allow unlimited responses for a url

view details

Tom Ritter

commit sha e2bb63056b143742a49fcae0356eca2008569bc5

Add a test for retriggers and unclassified failures.

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha b74215a345176289a1f264009d136f4c81eb1e37

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha 0d1145a05eed56d2e2464874099988de23fee404

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha a9cd094336694940531a484bdde94fec90ce09e6

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha 4ce0c858574f596c954486e3013777ef4bdb469c

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

Tom Ritter

commit sha ba5391bc5519bb48c55d2296e5f506e080b12b01

Process Existing Job

view details

Tom Ritter

commit sha 908b4eb45a22ada85972724fa75a1210d522f3c7

Add a testExistingJobSucceeded test

view details

Tom Ritter

commit sha 6eb7533e874a4e033cb8ae3243a03319230d6de7

Refactor JOBSTATUS into two separate parts: Status and Outcome Status is what updatebot needs to do for it next. Outcome is how the job completed. Outcome is really needed more for testing purposes than anything else because in theory we won't care how it ended as long as it ended but that's a short-sighted view because in the future we will definetly want to display that information on a dashboard.

view details

Tom Ritter

commit sha 90a7eeff7ccab356f257cb144dc085ce5c42d55b

Add a test for a build failure

view details

Tom Ritter

commit sha db649b3688096e7380147390a69e098f7b8480f7

Add a test for a test failing that's classified intrmittent.

view details

Tom Ritter

commit sha f2eaead5f00eacad813052fa9bcdaec5f886b357

Rename the taskcluster/treeherder url because we are going to need two of them.

view details

Tom Ritter

commit sha 9e10dc8cfa6968be2a9be5654b0e9b31950b3c7c

Improve the MockTreeherderServer to allow unlimited responses for a url

view details

Tom Ritter

commit sha e2939ed1492ce297e692252519fda36ed3daf9fe

Add a test for retriggers and unclassified failures.

view details

push time in 6 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha a9cd094336694940531a484bdde94fec90ce09e6

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha 4ce0c858574f596c954486e3013777ef4bdb469c

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

push time in 6 days

PR opened mozilla-services/updatebot

Reviewers
2020 07 14 existing job

I think I have all the existing job code working. Well; that is to say, it passes tests locally. I haven't tried it in-tree yet.

But I think this prep-work branch is good to review.

+463 -50

0 comment

14 changed files

pr created time in 7 days

create barnchmozilla-services/updatebot

branch : 2020-07-30-existing-job

created branch time in 7 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha d562afe6c76d7059179fa34ba159773327ed8fc8

Impove the TestCommandProvider

view details

Tom Ritter

commit sha 60ea0c1aff689bb34bce9b4159248ab47565aa19

Implement phabricator set_reviewer and abandon

view details

Tom Ritter

commit sha 58c50513380cb2eae36a30da07f76a53e3244cb4

Add JSON-E, needed for Taskcluster API interaction

view details

Tom Ritter

commit sha cb3e682380421eeb6d4bab93283f24bc48395442

Add Treeherder APIs for retrieving and transforming job details

view details

Tom Ritter

commit sha 7020e44e1bd42fcf40652c4fb21e4fe97d29d9c3

Implement Treeherder/Taskcluster retrigger_jobs call

view details

Tom Ritter

commit sha 1cb8156fc655157dd89b45a369c5264daa03a188

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

push time in 7 days

issue openedmozilla-services/updatebot

Address infinite reprocessing of a lbrary update that is failing.

If the retrigger of jobs fails, we raise an exception but don't update the job state. We'll try again. If it fails again, we will repeat. If the retrigger always fails, we'll never abort this library update.

created time in 7 days

push eventmozilla/foundation-security-advisories

Tom Ritter

commit sha abb866d5c0d144ce87479bed52eff127e6c6897d

Add Thunderbird 68.11 advisories and correct a few things in FF 68.11

view details

push time in 7 days

PR opened google/oss-fuzz

Add two more team members to rnp's cc list
+2 -0

0 comment

1 changed file

pr created time in 7 days

create barnchtomrittervg/oss-fuzz

branch : rnp-more-ccs

created branch time in 7 days

push eventmozilla/foundation-security-advisories

Garvan Keeley

commit sha 0c49ec70026f042f8384bbf95978cda36c4d74eb

Advisories for iOS Release 28

view details

push time in 8 days

push eventtomrittervg/oss-fuzz

Tom Ritter

commit sha ffc37b9eed866d90a8bb69b6c254e434526e00f2

Add fuzz_dump and fuzz_keyring to the exception list These binaries - from https://github.com/rnpgp/rnp/tree/master/src/fuzzing - are very tiny and just call into the library with a single API, and are not staticly compiled - so they have very few UBSAN calls.

view details

push time in 8 days

push eventtomrittervg/oss-fuzz

push time in 9 days

PR opened google/oss-fuzz

[rnp] Add afl engine
+2 -1

0 comment

1 changed file

pr created time in 9 days

create barnchtomrittervg/oss-fuzz

branch : 2020-07-28-rnp-afl

created branch time in 9 days

push eventtomrittervg/oss-fuzz

Max Moroz

commit sha 70d278bb53147c23c4fc7f2c4528e485c50cfe1f

[infra] Rename msan-builder to msan-libs-builder (#3388). (#4190) * [infra] Rename msan-builder to msan-libs-builder and add (broken) dfsan-libs-builder (#3388). * remove dfsan-libs-builder for now * presubmit format

view details

AdamKorcz

commit sha 793ed709079f9b409c06413c196df4d94c105b6d

[Radon] Added project (#4183)

view details

Abhishek Arya

commit sha dbeab81f5d10a2b0c8c1715c48132fc631c52699

Add corpus-dir to run_fuzzer, for later use in coverage cmd. (#4191) * Add corpus-dir to run_fuzzer, for later use in coverage cmd. * Update as per comment.

view details

Micah Snyder

commit sha 7f9866ee90ac8be46c1e4673342b6d81cf0d81b4

[clamav] Add optional autogen.sh step; fix build (#4193) ClamAV recently removed autotools generated materials (configure, Makefile.in, etc) from the git repo. This commit adds tools and calls necessary to generate those files if autogen.sh is present.

view details

Oliver Chang

commit sha bc7d478d4a64e93e1f0eb7022100ef3af75829d4

build: Project sync fixes. (#4194) - Add some more logging. - Use GitHub client ID/secret rather than personal access token. - Fix function deploy wrt "--project" argument.

view details

Oliver Chang

commit sha cf1ce595ba720177c73c441da81181b502d0ec3d

Disable coverage job sync. (#4195) Migrating to new infratructure for requesting coverage jobs.

view details

Joachim Metz

commit sha 2c34387f359f47b1ed2a33c03990625dcc7cc3e4

Added libfplist, libfsapfs, libbde, libqcow, libvhdi and libvmdk to libyal project configuration (#4198)

view details

binaryFate

commit sha da87fd89448b929ba8c58a67078d54f4a618516d

Monero: fix corpus zip file creation anew; enable honggfuzz&afl (#4200) Co-authored-by: binaryFate <binaryfate@users.noreply.github.com>

view details

Tom Ritter

commit sha 490a59781e2f6e53a2b0ba43951b1270fb75dda5

[rnp] Fuzzer fixes and improvements for rnp (#4189) Undefined sanitizer should now work, and the honggfuzz engine as well. Upstream changes supported enabling these, and when I did so I renamed one of the variables.

view details

Oliver Chang

commit sha 24b4bb32cb76397fb5af096a24117b5009937dfb

Fix builds status requirements.txt (#4203)

view details

werdnaH

commit sha 69869e098e0ae0e2f2ed87b8383c621f125a24e4

Add DDL Fuzzer (#4202) Co-authored-by: Zepeng Hu <zepenghu@google.com>

view details

Kim Kulling

commit sha c22ec3a33da652ad58f7975ae90a6122d0202a35

Enable assimp-fuzzing test (#4181)

view details

Everett Maus

commit sha 6806b74519a573f0ca66d5c2d9208ccdec17dbd4

Removing no-longer valid email and updating sanitizer definitions (#4207) * Removing no-longer valid email and updating sanitizer definitions * Updating CCs to a team rather than an individual

view details

Matt Johnston

commit sha 951403b0753467751c2cbf41394163717169566b

[dropbear] Change Mercurial repo to hg.ucc.asn.au (#4204)

view details

Stephen Frost

commit sha 56a3b89eae1a86cbc269fb0b14d4a45325e4d7c2

Add gmail.com account to PostgreSQL project (#4206) This is to add my (Stephen Frost's) gmail.com account to the PostgreSQL project to allow me to view the issues that are being opened by the fuzz tool.

view details

Eugene Kliuchnikov

commit sha 3c234c8f1c2319f27a328656c84c7eaf7767a561

Add fuzzer for Brunsli decoder (#4205)

view details

seabeepea

commit sha e1920a468d7c77b93bcda2b7cb9ab344e10855c0

move Header constructor inside try/catch (#4208) Signed-off-by: Cary Phillips <seabeepea@gmail.com>

view details

Oliver Chang

commit sha 4e2a7c07e578d7b68605b2b7e21707217cbe8ced

Many build infra fixes. (#4209) - Remove base image project argument from deploy.sh. Deploy all functions to the main oss-fuzz image. - Reduce max instances of functions to 1 to avoid rate limiting issues. - Fix missing ndb context initialization in request_build.py - Fix incorrect tags in BuildHistory entities. "-" was doubled. - Fix base build deployment with incorrect schedule variable. - Add scripts for requesting builds locally.

view details

Oliver Chang

commit sha 7bb74407dc037fdc2cd373e29eff729c74ab784b

Fix an Python 3 exception in build_project.py. (#4210)

view details

Oliver Chang

commit sha eb19f16419f81c392f7638d422a23fff183d78e1

Add queueTtl argument to cloud build request. (#4211) To prevent expiry of queued builds.

view details

push time in 9 days

push eventtomrittervg/rnp

Tom Ritter

commit sha f1cc774ac9d1cfc9512d5e464fb2eccbdd99ad95

Swallow the dump output to speed up fuzzing.

view details

push time in 9 days

pull request commentrnpgp/rnp

Swallow the dump output to speed up fuzzing.

I don't believe that test failure :)

tomrittervg

comment created time in 9 days

PR opened rnpgp/rnp

Swallow the dump output to speed up fuzzing.

While this works, it might be possible to do this better but I'm not sure about how to use rnp_dump_packets_to_output.

+3 -11

0 comment

1 changed file

pr created time in 9 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 692c2f7c13f80e6588a73e2f5ad3bed637e0ab6d

Swallow the dump output to speed up fuzzing.

view details

push time in 9 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 8c9ad4a039e75d3dfc146f233c7231234d822480

Swallow the dump output to speed up fuzzing.

view details

push time in 9 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 4759d4b75937b33080bb08af56a831058d732c8d

no_writer

view details

push time in 9 days

push eventtomrittervg/rnp

Tom Ritter

commit sha c30dffc279b069a2482077377cbcd5636c9738fb

Redirect the dump output to /dev/null to speed up fuzzing

view details

push time in 9 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 46bb9a1171f818d1455265b54175f17ea64e4bae

fix

view details

push time in 10 days

push eventtomrittervg/rnp

Tom Ritter

commit sha e9ade14fc21de556a69ef4b3953df8c35ae41b90

debug

view details

push time in 10 days

create barnchtomrittervg/rnp

branch : 2020-07-28-fuzz-dump-devnull

created branch time in 10 days

PR opened rnpgp/rnp

Memory leak from unknown pk algo
=================================================================
==1==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 245 byte(s) in 1 object(s) allocated from:
    #0 0x51e1bd in malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:145:3
    #1 0x7fae0309f210 in stream_read_packet_body(pgp_source_t*, pgp_packet_body_t*) /src/rnp/src/librepgp/stream-packet.cpp:681:36
    #2 0x7fae030a6434 in stream_parse_pk_sesskey(pgp_source_t*, pgp_pk_sesskey_t*) /src/rnp/src/librepgp/stream-packet.cpp:1115:16
    #3 0x7fae03067daa in stream_dump_pk_session_key(rnp_dump_ctx_t*, pgp_source_t*, pgp_dest_t*) /src/rnp/src/librepgp/stream-dump.cpp:918:16
    #4 0x7fae03062cf0 in stream_dump_packets_raw(rnp_dump_ctx_t*, pgp_source_t*, pgp_dest_t*) /src/rnp/src/librepgp/stream-dump.cpp:1215:19
    #5 0x7fae030619a3 in stream_dump_packets(rnp_dump_ctx_t*, pgp_source_t*, pgp_dest_t*) /src/rnp/src/librepgp/stream-dump.cpp:1323:11
    #6 0x7fae0327682d in rnp_dump_packets_to_output /src/rnp/src/lib/rnp.cpp:7217:24
    #7 0x5506f8 in LLVMFuzzerTestOneInput /src/rnp/src/fuzzing/dump.c:53:11
    #8 0x457da1 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:558:15
    #9 0x4574e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:470:3
    #10 0x459487 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:770:7
    #11 0x459689 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:799:3
    #12 0x4493c5 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:846:6
    #13 0x470f62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19:10
    #14 0x7fae0024082f in __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/libc-start.c:291

SUMMARY: AddressSanitizer: 245 byte(s) leaked in 1 allocation(s).

In stream_parse_pk_sesskey, we allocate pkt->data in stream_read_packet_body (called here.) But when an unknown pk algo is specified, we do an early return instead of free-ing the memory.

+1 -1

0 comment

1 changed file

pr created time in 10 days

create barnchtomrittervg/rnp

branch : 2020-07-27-leak

created branch time in 10 days

push eventmozilla-services/updatebot

Tom Ritter

commit sha 2247bc6434483ccd518cdeb259f8554f8bcc1f8d

Fix logging categories .bind_category was being called on the same instance shared among every Provider. Therefore the last provider to initialize set the category for every other provider. We copy the logging object and provider an overridden log function with category defined. We have to update the test because we are not comparing identical objects anymore, so we just make sure that it was populated. Although, since we are duplicating the logging object we could just keep the whole bind_category idea instead of using a partial, but I think the partial is better?

view details

Tom Ritter

commit sha 38561ddf26b4df9ad11a5ef8f8ab1581e0ad04aa

Check the database during Updatebot initialization. This is needed to allow us to call things like get_all_libraries before run.

view details

Tom Ritter

commit sha 7d94d60dbd12b95b5e4b4294740754b8045196bb

Add a project field that is needed for the TreeHerder API

view details

Tom Ritter

commit sha e4e8a2c049b7b091381836141c0af8a075d88445

Add a library_filter parameter to Updatebot.run()

view details

Tom Ritter

commit sha 277a3ec4d32d6501cf5b9358f44392875617fd6c

Add additional logging to library processing

view details

Tom Ritter

commit sha 40a20adc89e0bb190f51082147be078fbcdc14ff

Improve logging for functions with large arguments

view details

Tom Ritter

commit sha 874c62faaee33d20045aba8e1f8fbc5b08afbecc

Create an update_job_status database API

view details

Tom Ritter

commit sha 8410f3930ea73403006dc20b0e764d5072a8d759

Improve how TaskclusterProvider finds the try revision This is needed because hardcoded try result output will use the real url but when communicating to a mocked treeherder api, the url will be localhost.

view details

Tom Ritter

commit sha d0b666efbbb6702c3fd3c1e7402b741df3da4a1a

Add a separate column for the maintainer's phabricator username The api bmoexternalaccount.search ought to do what I need, but I can't get it to work. moz-phab uses this api (although they don't have a bmo email to look so that isn't a concrete comparison).

view details

Tom Ritter

commit sha 5cebef8c2e4b404a61a162932d24574409cc2a8c

Implement phabricator set_reviewer and abandon

view details

Tom Ritter

commit sha 01f8a8cae9c29843c5e56e7622bd6f722d9578d0

Implement Treeherder Retrigger Job call

view details

Tom Ritter

commit sha 15ae42b66eb7f12c6f370891ebada28634358086

Add Treeherder APIs for retrieving and transforming job details

view details

Tom Ritter

commit sha 664013a2e694fc215369474d5cdf18783147482a

Process Existing Job

view details

Tom Ritter

commit sha fe8e062b2a5196a82cad9a5545de8b8d4ab51cf5

Refactor the MockTreeHerderServer for later use

view details

Tom Ritter

commit sha cacc9604ef1837b42fbb0705313691b626b11548

Refactor how expected values work for functionality tests This will support us running tests with different expected values.

view details

Tom Ritter

commit sha a35f11f61674083e2dfcf8604e909db056afc777

Add a testExistingJobSucceeded test

view details

Tom Ritter

commit sha 0d115bb2298d5dc73ce4c496a2ff805c1bf4d812

Refactor JOBSTATUS into two separate parts: Status and Outcome Status is what updatebot needs to do for it next. Outcome is how the job completed. Outcome is really needed more for testing purposes than anything else because in theory we won't care how it ended as long as it ended but that's a short-sighted view because in the future we will definetly want to display that information on a dashboard.

view details

Tom Ritter

commit sha 5f6a101cb3e449029d077bdf83b7d37e948168f4

Add a test for a build failure

view details

Tom Ritter

commit sha 906750b624256754be76fa5febca2836fd659ae3

Add a test for a test failing that's classified intrmittent.

view details

Tom Ritter

commit sha 27cd57780015f96d311a99a03429ad550cc43f52

Add a test for retriggers and unclassified failures.

view details

push time in 10 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

@inferno-chromium What's the path forward here? You probably don't want me editing the ubsan check; but I think it's a false negative....

tomrittervg

comment created time in 10 days

PR opened rnpgp/rnp

When we're compiling for AFL, use C++'s stdlib because AFL's library uses C++ code

This commit will allow us to use the AFL driver on OSS-Fuzz

+4 -0

0 comment

1 changed file

pr created time in 10 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 0bac0ab248bd43e137b1ae731822898ba94c625e

When we're compiling for AFL, use C++'s stdlib because AFL's library uses C++ code

view details

push time in 10 days

push eventtomrittervg/rnp

Tom Ritter

commit sha b1d195c9897275525a954d4164bbce85b2e1851f

try this

view details

push time in 10 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 7909da75814f6cf103b253084d9fd3958e36476e

try this

view details

Tom Ritter

commit sha 7df74aacdcc7beba6b97aa37cf416c418285950e

dbl check

view details

push time in 10 days

push eventtomrittervg/rnp

Tom Ritter

commit sha ea7a45e13c50daca0b52942dbbea07ddbf56c37e

strequal

view details

push time in 11 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 2e4e61718c9b8372c66b3c2896db4090d2cbe569

conditionally

view details

push time in 11 days

push eventtomrittervg/rnp

Tom Ritter

commit sha e832da57e37ad9b69572694bb824beabce8132cb

-stdlib=libc++

view details

push time in 11 days

issue closedgoogle/oss-fuzz

honggfuzz and afl engines give an error '-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer'

While trying to add these engines to rnp ( https://github.com/google/oss-fuzz/tree/master/projects/rnp ) I am getting an error:

$ python infra/helper.py check_build rnp                                                                      1 ↵
Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e ARCHITECTURE=x86_64 -v /home/tom/Documents/play/rnp/oss-fuzz/build/out/rnp:/out -t gcr.io/oss-fuzz-base/base-runner test_all
INFO: performing bad build checks for /tmp/not-out/fuzz_keyring.
INFO: performing bad build checks for /tmp/not-out/fuzz_dump.
Broken fuzz targets (2):
fuzz_dump:
BAD BUILD: /tmp/not-out/fuzz_dump seems to have either startup crash or exit:
-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer.
Please either migrate to a compiler that supports -fsanitize=fuzzer
or use an older version of libFuzzer
fuzz_keyring:
BAD BUILD: /tmp/not-out/fuzz_keyring seems to have either startup crash or exit:
-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer.
Please either migrate to a compiler that supports -fsanitize=fuzzer
or use an older version of libFuzzer
ERROR: 100% of fuzz targets seem to be broken. See the list above for a detailed information.
Check build failed.

AFAICT these flags come from the oss-fuzz config; not anything I am using. I am building in the oss-fuzz containers:

$ python infra/helper.py build_image $PROJECT_NAME && python infra/helper.py build_fuzzers --sanitizer address --engine honggfuzz $PROJECT_NAME

Full log: https://ritter.vg/misc/transient/log2.txt

Sample compile command:

/usr/local/bin/clang++
-O1
-fno-omit-frame-pointer
-gline-tables-only
-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
-fsanitize=address
-fsanitize-address-use-after-scope
-fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp
-stdlib=libc++
-O3
-DNDEBUG
CMakeFiles/rnpkeys.dir/rnpkeys.cpp.o
CMakeFiles/rnpkeys.dir/tui.cpp.o
CMakeFiles/rnpkeys.dir/main.cpp.o
CMakeFiles/rnpkeys.dir/__/rnp/rnpcfg.cpp.o
CMakeFiles/rnpkeys.dir/__/rnp/fficli.cpp.o
../common/CMakeFiles/rnp-common.dir/list.cpp.o
../common/CMakeFiles/rnp-common.dir/str-utils.cpp.o
../common/CMakeFiles/rnp-common.dir/file-utils.cpp.o
-o rnpkeys
-Wl,-rpath,/src/rnp-build/src/lib:
../lib/librnp-0.so.0.0.0
/usr/lib/x86_64-linux-gnu/libjson-c.so
/usr/lib/libbotan-2.so
/usr/lib/x86_64-linux-gnu/libjson-c.so
/usr/lib/x86_64-linux-gnu/libbz2.so
/usr/lib/x86_64-linux-gnu/libz.so

I know I must be doing something wrong; but I'm not sure what...

closed time in 11 days

tomrittervg

issue commentgoogle/oss-fuzz

honggfuzz and afl engines give an error '-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer'

Thanks all, I wasn't properly using LIB_FUZZING_ENGINE

tomrittervg

comment created time in 11 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 6e88c619fb6d04d04986aa4bdef9df04d55276b5

Try link_libraries

view details

push time in 11 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 046f81478ff0bac038beee468496666886714fe9

Remove static ubsan lib

view details

Tom Ritter

commit sha 2bc70290ebe5a038b736f42ed92a2705945b0fca

Add lstdc++ to fuzz drivers

view details

push time in 11 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

objdump -dC fuzz_dump | egrep "callq\s+[0-9a-f]+\s+<__ubsan" -c 163 objdump -dC fuzz_dump | egrep "callq\s+[0-9a-f]+\s+<__asan" -c 0

Looks like I have UBSAN but the check doesn't catch it

tomrittervg

comment created time in 11 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

Well... that's weird. The build succeeds. When I run the fuzzer binaries it finds Undefined Behavior from UBSAN. But the check build says it's missing UBSAN....

tomrittervg

comment created time in 12 days

push eventtomrittervg/oss-fuzz

Tom Ritter

commit sha 3faff754c864a7470d6721697ff7728659437775

[rnp] Fuzzer fixes and improvements for rnp Undefined sanitizer should now work, and the honggfuzz engine as well. Upstream changes supported enabling these, and when I did so I renamed one of the variables.

view details

push time in 12 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

I have been looking at other projects. I've seen a few solutions:

  • OpenSSL in #195 has a LDCMD variable to override the linker. Helpful, doesn't work for me.
  • s2opc uses -DCMAKE_C_LINK_EXECUTABLE="$CXX <FLAGS> <CMAKE_C_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>" which does work for me, but it overrides the internal CMAKE link command entirely, so it seems really brittle to me and I hope to avoid it....
  • libpsl disabled vptr...
  • imagemagick added -lubsan
  • A lot of baxel-based projects all use the same solution of --linkopt=$(find $(llvm-config --libdir) -name libclang_rt.ubsan_standalone_cxx-x86_64.a | head -1)

Of these options, I'd really prefer doing -lubsan or the libclang_rt.a because it seems a lot less brittle than the other options. But when I do that I'm hitting the error

/usr/bin/ld: decrypt: hidden symbol `_ZN7__ubsan14TypeCheckKindsE' in /usr/local/lib/clang/11.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a(ubsan_handlers.cpp.o) is referenced by DSO

So I think I'll do the CMAKE override....

tomrittervg

comment created time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha fc16f28497f55828eaf5653f35cae7a4c751a460

remove lubsan

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha d2528426baf030f176dc991b27ac13f50febfcd1

Specify exact lib

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 5345d48c5536781e0318b85f43fc89af6e02963f

Compiler-rt

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha dae4ff677d2fdb29c75800ed9dc188853e652e84

try link libraries

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 32e434e067fa2a92dc34de53ec6fcd0443ce7b3b

unconditional

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 5c542ca1d60fd459b2086a37cce0717528a2a6fc

fix

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 36e6251c9b3fc0557e8aff1ccc083075eebd9d66

Try adding lubsan

view details

push time in 12 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 4dc6869541e11cd12e28452d14f748a4653ed34c

Try adding lubsan

view details

push time in 13 days

create barnchtomrittervg/rnp

branch : 2020-07-24-fuzzing-fixes

created branch time in 13 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

Battling with trying to set the linker, which is surprisingly hard... Ideally in the CMakeLists.txt so it applied to a single directory, but that doesn't seem feasible. And -DCMAKE_LINKER= in build.sh doesn't work surprisingly...

tomrittervg

comment created time in 13 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

Looks like I am probably hitting #713 / #195 and need to use clang++ for linking when compiling with UBSAN.

tomrittervg

comment created time in 13 days

pull request commentgoogle/oss-fuzz

Fuzzer fixes and improvements for rnp

Uhg, I thought UBSAN was working. Will debug....

tomrittervg

comment created time in 14 days

PR opened google/oss-fuzz

Fuzzer fixes and improvements for rnp

Undefined sanitizer should now work, and the honggfuzz engine as well. Upstream changes supported enabling these, and when I did so I renamed one of the variables.

+7 -2

0 comment

2 changed files

pr created time in 14 days

create barnchtomrittervg/oss-fuzz

branch : 2020-07-24-rnp-fixes

created branch time in 14 days

push eventtomrittervg/oss-fuzz

Tom Ritter

commit sha e49213a34858e65031525effb3ece1ca8b673ae9

Add a new fuzz target to osquery for sql queries, and add dictionaries and seed corpuses for both targets. (#2992)

view details

Catena cyber

commit sha f060db655d0931e8bc70ff568e87d1898cc96b0d

[ndpi] Integrate NDPI with pcap (#2994)

view details

Catena cyber

commit sha c57a63b3ed4c0f0a2f6bac2c0c65b3e92d0609ed

Adds gonids project (#2995)

view details

Teddy Reed

commit sha bdd592b1b36f507c1e2b35489ac4f66207afd98b

[osquery][lz4][zstd] Add more fb auto_ccs (#2998) Cc: mhl@fb.com

view details

Bhargava Shastry

commit sha 048ccb79a0092c208bc9069f58f7c621ee6a3cc6

mruby: Move LPM build to dockerfile (#3000)

view details

Paul Dreik

commit sha c4f6e0122ff50731f263b528faa5fc279f461486

[simdjson] initial integration (#2999) * [simdjson] initial attempt * fix name mismatch * fix bad path * add corpus build script * drop setting the c++ standard

view details

Regina Obe

commit sha d739d654605fb9e07981438bc3e6a45a827218c5

Update Dockerfile (#3005) gogs is the old url that redirects to new gitea url

view details

Guido Vranken

commit sha 82dcfabe3d9dbf0ca595cccf330e4d306d56b1d0

[cryptofuzz] wolfCrypt (#3004)

view details

Roman Lebedev

commit sha 06cd6f44311cdf3e114463bfd670b5e1ca30fbd8

[LibRawSpeed] Pull in newer cmake to fix the build (#3003) As per https://apt.kitware.com/

view details

Abhishek Arya

commit sha 4035d0d62f995482ec33c6ed9cd9c368a50413fd

Speculative fix for ndpi broken build

view details

jonathanmetzman

commit sha af0ca618de71174e5daaa2104ed8ca2b27823aa9

Get run_minijail working for i386 (#2997)

view details

Guido Vranken

commit sha c729e2677091f77d1e84280a004b6bcea0262ae9

[cryptofuzz] Enable wolfCrypt 32 bit builds (#3008)

view details

Abhishek Arya

commit sha 4ab29a23696b7a7a55cef0bb73180043ed9ca922

Remove unused aosp project. (#3009)

view details

Paul Dreik

commit sha 12b7ac580ff857e9a75bfdc6998fa2f9621aae51

[simdjson] move the build script into the simdjson repo (#3013) this means it can be maintained and bugfixed upstream

view details

Catena cyber

commit sha c839326cd27003f11b1f319ed579554c5632aff3

Debugs mysql server fuzz targets (#3012)

view details

Oliver Chang

commit sha 98e4472c453e099b8e471a033fbcb02127e7503c

honggfuzz: Include libhfcommon in $LIB_FUZZING_ENGINE. (#3011)

view details

David Bryant

commit sha ce05d64476830f9447a8c06fde337980152bbd56

Add wavpack to oss-fuzz (#3017)

view details

Catena cyber

commit sha ad44b580d94b3a6caae00bceab2ab1664d552777

do not analyze too long inpus for gnupg list fuzz target (#3007)

view details

Andreas Schneider

commit sha 14bd043a8c4abe7d425de0acb72efdee100f7d7e

[libssh] Fix building libssh as a static library (#3020)

view details

Guido Vranken

commit sha 16ddb85aafd645739aa6ec192210fb088c21f552

[cryptofuzz] ARM mbed crypto (#3021)

view details

push time in 14 days

issue commentghacksuserjs/ghacks-user.js

FYI: don't use dFPI with FPI [1631676: resolved 78]

If FPI is enabled, it should override privacy.partition.network_state in all cases. Although I admit it's a bit difficult to see how that happens from the code...

Thorin-Oakenpants

comment created time in 15 days

PR opened rnpgp/rnp

Improve fuzzing configuration to support more fuzzers

Also take care of TODOs from #1199

+22 -39

0 comment

5 changed files

pr created time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 9385ce66c175e18f5489dee29f364e9c0e714c24

Add oss-fuzz's linking command to the build if it is present. Otherwise, add the libfuzzer flags.

view details

Tom Ritter

commit sha 57202ebae0d4cbde37ace895d4437f330acc2b6d

Correct copyright years

view details

Tom Ritter

commit sha 009f9573f728bdd105c9117a9bbebdccad7ca9c3

ENABLE_FUZZING -> ENABLE_FUZZERS to match SANITIZERS

view details

Tom Ritter

commit sha 1e5fd630c915e03d9318ec48411860f21d43fc4a

Update fuzzing Documentation

view details

push time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 05785ae189bf7cd38963f1fa4e5cee580c133e91

Correct env var usage again again again again

view details

push time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha cc5215b4a4c1c64be6a5e6fb84ac1c3632d9dec2

Correct env var usage again again again

view details

push time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha aa7fcb03b707df09afe55233d6f8b49e799c62dd

Correct env var usage again again

view details

push time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 7a23944a6c9ad0fda0abede128b9f28d723516d8

Correct env var usage again

view details

push time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 1e0a51f2310af85d28d8c0b12d0f0867160fae05

Correct env var usage

view details

push time in 15 days

push eventtomrittervg/rnp

Tom Ritter

commit sha ee5f4624c962a924ba3e32e505c023c8e04cf8ef

Fix whitespace

view details

push time in 16 days

push eventtomrittervg/rnp

Tom Ritter

commit sha bc9d15a81da189a6b2fffc68d00c9c517c814528

Follow oss-fuzz's instructions

view details

push time in 16 days

push eventtomrittervg/rnp

Tom Ritter

commit sha b3cac7d9f64804d8eaceaa3beedb6f5cfb0fde2a

Follow oss-fuzz's instructions

view details

push time in 16 days

issue openedgoogle/oss-fuzz

honggfuzz and afl engines give an error '-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer'

While trying to add these engines to rnp ( https://github.com/google/oss-fuzz/tree/master/projects/rnp ) I am getting an error:

$ python infra/helper.py check_build rnp                                                                      1 ↵
Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e ARCHITECTURE=x86_64 -v /home/tom/Documents/play/rnp/oss-fuzz/build/out/rnp:/out -t gcr.io/oss-fuzz-base/base-runner test_all
INFO: performing bad build checks for /tmp/not-out/fuzz_keyring.
INFO: performing bad build checks for /tmp/not-out/fuzz_dump.
Broken fuzz targets (2):
fuzz_dump:
BAD BUILD: /tmp/not-out/fuzz_dump seems to have either startup crash or exit:
-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer.
Please either migrate to a compiler that supports -fsanitize=fuzzer
or use an older version of libFuzzer
fuzz_keyring:
BAD BUILD: /tmp/not-out/fuzz_keyring seems to have either startup crash or exit:
-fsanitize-coverage=trace-pc-guard is no longer supported by libFuzzer.
Please either migrate to a compiler that supports -fsanitize=fuzzer
or use an older version of libFuzzer
ERROR: 100% of fuzz targets seem to be broken. See the list above for a detailed information.
Check build failed.

AFAICT these flags come from the oss-fuzz config; not anything I am using. I am building in the oss-fuzz containers:

$ python infra/helper.py build_image $PROJECT_NAME && python infra/helper.py build_fuzzers --sanitizer address --engine honggfuzz $PROJECT_NAME

Full log: https://ritter.vg/misc/transient/log2.txt

Sample compile command:

/usr/local/bin/clang++
-O1
-fno-omit-frame-pointer
-gline-tables-only
-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
-fsanitize=address
-fsanitize-address-use-after-scope
-fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp
-stdlib=libc++
-O3
-DNDEBUG
CMakeFiles/rnpkeys.dir/rnpkeys.cpp.o
CMakeFiles/rnpkeys.dir/tui.cpp.o
CMakeFiles/rnpkeys.dir/main.cpp.o
CMakeFiles/rnpkeys.dir/__/rnp/rnpcfg.cpp.o
CMakeFiles/rnpkeys.dir/__/rnp/fficli.cpp.o
../common/CMakeFiles/rnp-common.dir/list.cpp.o
../common/CMakeFiles/rnp-common.dir/str-utils.cpp.o
../common/CMakeFiles/rnp-common.dir/file-utils.cpp.o
-o rnpkeys
-Wl,-rpath,/src/rnp-build/src/lib:
../lib/librnp-0.so.0.0.0
/usr/lib/x86_64-linux-gnu/libjson-c.so
/usr/lib/libbotan-2.so
/usr/lib/x86_64-linux-gnu/libjson-c.so
/usr/lib/x86_64-linux-gnu/libbz2.so
/usr/lib/x86_64-linux-gnu/libz.so

I know I must be doing something wrong; but I'm not sure what...

created time in 16 days

push eventtomrittervg/oss-fuzz

Tom Ritter

commit sha 719f414ba77b5f2dfcc6f879fb655222ab09f3ad

Add rnp

view details

push time in 18 days

push eventtomrittervg/oss-fuzz

Tom Ritter

commit sha c59bd45bc021de47c255404b505c8c1795617ca6

Add rnp

view details

push time in 19 days

PR opened google/oss-fuzz

Add rnp

rnp is the OpenPGP library that will be directly integrated into Thunderbird in the coming months. (See https://wiki.mozilla.org/Thunderbird:OpenPGP#Testing ) This commit adds it as a project, with two small fuzzers. More extensive fuzzers, corpus, dictionaries to come.

cc @ronaldtse

+94 -0

0 comment

3 changed files

pr created time in 19 days

create barnchtomrittervg/oss-fuzz

branch : rnp

created branch time in 19 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 6dddef3d1eb05b7901746a3faa80c3dc8f5442dc

Follow oss-fuzz's instructions

view details

push time in 19 days

push eventtomrittervg/rnp

Tom Ritter

commit sha 72782a49d08e403343586d66a1f2ae1863c914e7

Let the shared libraries be loaded from the current directory

view details

push time in 19 days

create barnchtomrittervg/rnp

branch : 2020-07-18-fuzzing-fixes

created branch time in 20 days

issue commentrnpgp/rnp

Update Fuzzer section of `develop.adoc` due to #1195

I think we should change -DENABLE_FUZZING to -DENABLE_FUZZERS also.

ronaldtse

comment created time in 20 days

more