actually this is not web panel problem, seams all events are queued and not processing by workers, the problem is when I reset queue it works, but after a few minutes stops working

I just upgraded to the latest self-hosted version, 21.1.0. When I try to upload an avatar for my organization, I don't see any logs coming from the web container, but I do see the following from nginx:

nginx_1                                     | 2021/01/16 00:20:46 [warn] 6#6: *237 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000001, client: 172.31.21.73, server: , request: "PUT /api/0/organizations/allvision/avatar/ HTTP/1.1", host: "sentry.allvision.io", referrer: "https://sentry.allvision.io/settings/allvision/"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:20:47 +0000] "PUT /api/0/organizations/allvision/avatar/ HTTP/1.1" 200 815 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"

The flash message makes it look like the upload succeeded, but refreshing the page produces these logs:

nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:00 +0000] "GET /settings/allvision/ HTTP/1.1" 200 5047 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:00 +0000] "GET /_static/1610755751/sentry/dist/vendor.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:00 +0000] "GET /_static/1610755751/sentry/dist/app.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:00 +0000] "GET /_static/1610755751/sentry/js/ads.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /_static/1610755751/sentry/dist/AccountSettingsLayout~CommandPalette~HelpSearchModal~OrganizationSettingsLayout~ProjectSettingsLayou~456989bd.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /_static/1610755751/sentry/dist/OrganizationSettingsLayout.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /_static/1610755751/sentry/dist/AccountSettingsLayout~OrganizationSettingsLayout~ProjectSettingsLayout~SettingsIndex.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /_static/1610755751/sentry/dist/AccountSettingsLayout~CommandPalette~DiscoverV2Details~GroupEventDetails~HelpSearchModal~Organizatio~6244bbe6.js HTTP/1.1" 304 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /_static/1610755751/sentry/images/logos/apple-touch-icon-152x152.png HTTP/1.1" 499 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /_static/1610755751/sentry/images/favicon-dark.png HTTP/1.1" 499 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
web_1                                       | [uwsgi-http key:  client_addr: 172.25.0.29 client_port: 55983] hr_read(): Connection reset by peer [plugins/http/http.c line 917]
web_1                                       | [uwsgi-http key: sentry.allvision.io client_addr: 172.25.0.29 client_port: 55471] hr_write(): Broken pipe [plugins/http/http.c line 565]
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "POST /api/1/envelope/?sentry_key=14485c536df64a6a98534427140a832f&sentry_version=7 HTTP/1.1" 200 2 "https://sentry.allvision.io/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /api/0/internal/health/ HTTP/1.1" 200 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /api/0/organizations/?member=1 HTTP/1.1" 200 817 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /api/0/assistant/?v2 HTTP/1.1" 200 107 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /api/0/organizations/allvision/?detailed=1 HTTP/1.1" 200 10925 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /api/0/organizations/allvision/broadcasts/ HTTP/1.1" 200 2 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
web_1                                       | Traceback (most recent call last):
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
web_1                                       |     response = get_response(request)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
web_1                                       |     response = self._get_response(request)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
web_1                                       |     response = self.process_exception_by_middleware(e, request)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
web_1                                       |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
web_1                                       |     return callback(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
web_1                                       |     return self.dispatch(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
web_1                                       |     return handler(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/web/frontend/base.py", line 553, in get
web_1                                       |     photo_file = photo.getfile()
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 338, in getfile
web_1                                       |     impl = self._get_chunked_blob(mode, prefetch)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 330, in _get_chunked_blob
web_1                                       |     delete=delete,
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 482, in __init__
web_1                                       |     self.open()
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 519, in open
web_1                                       |     self.seek(0)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 579, in seek
web_1                                       |     self._curfile.seek(pos - self._curidx.offset)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/files/utils.py", line 20, in <lambda>
web_1                                       |     seek = property(lambda self: self.file.seek)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/filestore/s3.py", line 166, in _get_file
web_1                                       |     self._file.write(self.obj.get()["Body"].read())
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
web_1                                       |     response = action(self, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
web_1                                       |     response = getattr(parent.meta.client, operation_name)(**params)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
web_1                                       |     return self._make_api_call(operation_name, kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 586, in _make_api_call
web_1                                       |     raise error_class(parsed_response, operation_name)
web_1                                       | botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
web_1                                       | 00:24:01 [ERROR] django.request: Internal Server Error: /organization-avatar/bef17f592ac246568950bf7dc1a90f5c/ (status_code=500 request=<WSGIRequest: GET '/organization-avatar/bef17f592ac246568950bf7dc1a90f5c/?s=120'>)
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /organization-avatar/bef17f592ac246568950bf7dc1a90f5c/?s=120 HTTP/1.1" 500 5910 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.21.73 - - [16/Jan/2021:00:24:01 +0000] "GET /api/0/organizations/allvision/auth-provider/ HTTP/1.1" 204 0 "https://sentry.allvision.io/settings/allvision/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0" "24.148.95.206"
nginx_1                                     | 172.31.39.172 - - [16/Jan/2021:00:24:01 +0000] "POST /api/1/store/ HTTP/1.1" 200 41 "-" "sentry.python/0.19.5" "3.140.186.225"
web_1                                       | Traceback (most recent call last):
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
web_1                                       |     response = get_response(request)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
web_1                                       |     response = self._get_response(request)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
web_1                                       |     response = self.process_exception_by_middleware(e, request)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
web_1                                       |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
web_1                                       |     return callback(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
web_1                                       |     return self.dispatch(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
web_1                                       |     return handler(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/web/frontend/base.py", line 553, in get
web_1                                       |     photo_file = photo.getfile()
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 338, in getfile
web_1                                       |     impl = self._get_chunked_blob(mode, prefetch)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 330, in _get_chunked_blob
web_1                                       |     delete=delete,
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 482, in __init__
web_1                                       |     self.open()
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 519, in open
web_1                                       |     self.seek(0)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/models/file.py", line 579, in seek
web_1                                       |     self._curfile.seek(pos - self._curidx.offset)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/django/core/files/utils.py", line 20, in <lambda>
web_1                                       |     seek = property(lambda self: self.file.seek)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/sentry/filestore/s3.py", line 166, in _get_file
web_1                                       |     self._file.write(self.obj.get()["Body"].read())
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
web_1                                       |     response = action(self, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
web_1                                       |     response = getattr(parent.meta.client, operation_name)(**params)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
web_1                                       |     return self._make_api_call(operation_name, kwargs)
web_1                                       |   File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 586, in _make_api_call
web_1                                       |     raise error_class(parsed_response, operation_name)
web_1                                       | botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
web_1                                       | 00:24:01 [ERROR] django.request: Internal Server Error: /organization-avatar/bef17f592ac246568950bf7dc1a90f5c/ (status_code=500 request=<WSGIRequest: GET '/organization-avatar/bef17f592ac246568950bf7dc1a90f5c/'>)

The file is not getting uploaded to S3, but I can't tell why. My hunch is it has something to do with the tmp file in nginx. My filestore settings are the same as above.

Summary

Update Sentry automatically, to further reduce the delta between SaaS and self-hosted.

Motivation

Sentry exists in several deployment environments:

1. SaaS - primary target, sentry.io cloud
2. single-tenant - FSI†-managed, isolated from (1)
3. self-hosted - single-node, docker-compose-based ... this repo! 😁
4. custom - anything else (unsupported by FSI†)

† Functional Software, Inc.

(1) is a continuous deployment (CD) environment (using freight). (3) is a non-CD environment. (2) is somewhere in between(?).

With the move to CalVer we brought self-hosted Sentry installations within reach of a CD implementation, if the self-hoster so chooses to set it up that way. The proposal here is to teach Self-Hosted Sentry (SHS) to automatically update itself by default out of the box, so that it stays close to current with SaaS. This ensures that SHS users are getting a comparable experience to SaaS users.

Additional Context

n/a

This might help someone else dealing with this issue but I was able to resolve this by making the following change:

In replay/config.yml the default upstream is set too: relay: upstream: "http://web:9000/" But I noticed that the docker-compose for relay did not have web setup in the depends_on and since a custom networks is not being used in the docker-compose relay container had no context on where web is pointing too. Setting up web under depends_on would cause a circular dependency so you can not do that.

I am running in AWS in a VPC where the EC2 is in a private subnet. So to fix it I swaped web for the host internal IP for that EC2 instance.

After that I was able to receive events. Example: relay: upstream: "http://[host-ip]:9000/"

Follow-up to #144.

https://github.com/getsentry/.github/issues/8

• Use envvar for authentication (better place for secrets).
• Skip validation for staff (less noise when we go to sentry).
• Link to "closing" comment at the end.

I guess we can use error code.

$ gh api "https://api.github.com/orgs/getsentry/members/chadwhitacrejjj"
{
gh: Not Found (HTTP 404)
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/reference/orgs#check-organization-membership-for-a-user"
}
$ echo $?
1

$ gh api "https://api.github.com/orgs/getsntry/members/chadwhitacre" { "message": "Not Found", gh: Not Found (HTTP 404) "documentation_url": "https://docs.github.com/rest/reference/orgs#check-organization-membership-for-a-user" } $ echo $?
1 $ gh api "https://api.github.com/orgs/getsentry/members/chadwhitacre"
$ echo $?
0 $

size-limit report