profile
viewpoint
Felix Bünemann felixbuenemann @LOUISINTERNET Germany https://keybase.io/felixbuenemann Head of Engineering @LOUISINTERNET

dweidenfeld/plexdrive 1005

Plexdrive mounts your Google Drive FUSE filesystem (optimized for media playback)

amerine/acts_as_tree 518

ActsAsTree -- Extends ActiveRecord to add simple support for organizing items into parent–children relationships.

felixbuenemann/bombevent 9

HTML5/Websockets/Eventmachine Bomberman

felixbuenemann/active_enum 2

Define enum classes in Rails and use them to enumerate ActiveRecord attributes

felixbuenemann/active_admin 1

The administration framework for Ruby on Rails applications.

felixbuenemann/active_model_serializers 0

ActiveModel::Serializer implementation and Rails hooks

felixbuenemann/acts_as_ordered_tree 0

ActiveRecord extension for ordered adjacency trees support

felixbuenemann/acts_as_solr_reloaded 0

ActsAsSolr with new features

felixbuenemann/acts_as_tree 0

ActsAsTree plugin

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha ab7546987e0903bf5f417d3b737e0d2ce042a6b5

Use additional read buffer per download thread This should increase throughput by avoiding long locking times when downloading directly into chunk storage.

view details

Felix Bünemann

commit sha 1423e6d12d5cbf8b2eaaa862ac485aff53387592

Tweak storage tests and add parallel benchmark

view details

push time in a day

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 7d2f3cf7941ac1f6344f1fd0bcc1f6e41ca1dfd2

Use additional read buffer per download thread This should increase throughput by avoiding long locking times when downloading directly into chunk storage.

view details

Felix Bünemann

commit sha cdf410c3a3f1935d0193e55130fcbd7a89158169

Tweak storage tests and add parallel benchmark

view details

push time in a day

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 2e5427c16f412f7aa39aa92c8c0ce2649ca70b98

Use fine-grained rw mutex for each chunk

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha cb89d645a9237961ecb3bc7beec39479c6778d44

Download chunks directly into storage This should make the memory usage bounded by chunk size * max chunks. As a side effect it's possible that downloads must wait for a storage chunk to become available, so it might make sense to increase max chunks.

view details

Felix Bünemann

commit sha 4a1d9c3dada752a23c99c8b16a97631dbfb6c180

Allow storage init with open file

view details

Felix Bünemann

commit sha ebc41411a563c814bdabaf6e8313c4ea50e09e99

Add chunk storage tests and benchmarks

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 4ade6389411ad3bf97490410aeb4c7760a7ce446

Download chunks directly into storage This should make the memory usage bounded by chunk size * max chunks. As a side effect it's possible that downloads must wait for a storage chunk to become available, so it might make sense to increase max chunks.

view details

Felix Bünemann

commit sha f1bcb5dfee1067d053729466a080577e71363d45

Allow storage init with open file

view details

Felix Bünemann

commit sha a4b0f3ef857a613a930515ca5a3494c091a7b747

Add chunk storage tests and benchmarks

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 08ff362167f3461b394c28098dc5af202487a03c

Download chunks directly into storage This should make the memory usage bounded by chunk size * max chunks. As a side effect it's possible that downloads must wait for a storage chunk to become available, so it might make sense to increase max chunks.

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 076b3515eee945d1cc6802d7ea9f1ccb9b291537

Download chunks directly into storage This should make the memory usage bounded by chunk size * max chunks. As a side effect it's possible that downloads must wait for a storage chunk to become available, so it might make sense to increase max chunks.

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha cc451bb13d8f84bfd33d3cbd11d3e0fda7d5c903

Allow parallel reads from storage

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 2facd81b074a6a9a0a7ae6616488e400f1edfcc9

Add buffer pool for download bodies This should reduce memory allocation and gc pressure.

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha a45f7ceee00d507503d77094e24dc29f8d6d8256

Add buffer pool for download bodies This should reduce memory allocation and gc pressure.

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 641c2f60a5fd078f8442d653ab64bbb04e9f9adb

Add buffer pool for download bodies This should reduce memory allocation and gc pressure.

view details

push time in 2 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha feb03ed959f2c3f07f26876508bcfeee5e9aac65

WIP: Implement MMAP backed file cache for chunks

view details

push time in 3 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha c02407ed7dad52e1cc354a84aedbf07ee798416f

Remove allow_root mount option This is no longer supported, because fuse.AllowRoot() was removed: https://github.com/bazil/fuse/commit/e0b89709bc6481f1345b6e19363cf18766e53fc7

view details

Felix Bünemann

commit sha f5837b6facaa0f427e8be04ceb1a96e565a0c66a

Bump CI to at least Go 1.13 This is needed to build against the latest version of basil.org/fuse.

view details

Dominik Weidenfeld

commit sha 6a8661b3a2248791496ad92229af27dd6fc91056

Merge pull request #340 from felixbuenemann/remove-allow-root-mount-option Remove allow_root mount option

view details

Felix Bünemann

commit sha f12bbe593d6d07ecbfd050596076109b7610cdfd

Add --drive-id option for team drives This fixes the problem where --root-node-id was broken for non-team drives and also should allow mounting a sub.directory of a team drive by specifying both --drive-id and --root-node-id.

view details

Felix Bünemann

commit sha 1486bf3d53de9e708c973ea6e2a54b8892b9242d

Ignore changes with changeType drive This should fix the unexpected change records without a file resource. Additionally we request to include the file resource for removals, which could be another reason of missing file resource in changes.

view details

Felix Bünemann

commit sha d2a14ee3bb27b5c9049ddfa795be7088acfa51cd

Update API to support all shared drives

view details

Felix Bünemann

commit sha afa5e4e8c125e2758398dc72ab19dc7ea74e1a8c

Merge pull request #342 from felixbuenemann/team-drive-fixes Team Drive Fixes

view details

Felix Bünemann

commit sha 90f7cd975926edc6691ef53d2fb6427742775708

Only store each downloaded chunk once This makes sure the download callback is only run once. This reduces memory pressure, lock contention and cpu usage for storing chunks.

view details

Felix Bünemann

commit sha 54f74cde21450e210478886a2fc4a6af461864eb

WIP: Implement MMAP backed file cache for chunks

view details

push time in 3 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 39a488889bcb795b026b77334af71ad81fc9ef6d

WIP: Implement MMAP backed file cache for chunks

view details

push time in 3 days

push eventfelixbuenemann/better_errors

Felix Bünemann

commit sha 641da2ac4d383a781dfc0acb23b9c7e5f17220f9

Fix spec errors with binding of caller For some weird reason with binding of caller loaded the inspected value for strings is no longer quoted, breaking the specs, so this changes the specs back to using all symbols.

view details

push time in 5 days

pull request commentBetterErrors/better_errors

Allow skipping variable inspection by class name

Seems like with the BOC specs there's a different string representation for inspect_value, which lacks the quotes. I'll rewrite the spec changes to work around that.

felixbuenemann

comment created time in 5 days

pull request commentBetterErrors/better_errors

Clear error page state before each request

CI errors are unrelated, see #450.

felixbuenemann

comment created time in 5 days

PR opened BetterErrors/better_errors

Fix Travis CI Build Problems

Update Rubies to latest minor version and exclude Rails 4.2 specs on Ruby 2.3, since the default bundler 2.1.2 gem breaks the downgrade.

Note that I tried to fix this first by using export BUNDLER_VERSION=1.3, which works fine locally to prefer Bundler 1.x, even if 2.x is installed, but for some reason that didn't work on Travis CI either.

If we really wanted to we could install bundler 1.17.3 and oveeride the install step to run bundle _1.17.3_ install …, but since Rails 4.2 is EOL imho the easiest thing is to just run the 4.2 specs on Ruby 2.2.10.

+15 -19

0 comment

1 changed file

pr created time in 5 days

push eventfelixbuenemann/better_errors

Felix Bünemann

commit sha da5e6556b13a30634912405c56b725a0ee20174c

Fix Travis CI Build Problems Update Rubies to latest minor version and exclude Rails 4.2 specs on Ruby 2.3, since the default bundler 2.1.2 gem breaks the downgrade.

view details

push time in 5 days

push eventfelixbuenemann/better_errors

Felix Bünemann

commit sha 4fe3793d99e193ae9170c45b4122065c4ff1ddc9

Fix Travis CI Rails 4.2 bundler problems

view details

push time in 5 days

push eventfelixbuenemann/better_errors

Felix Bünemann

commit sha 35f6e398043f2492d19301bc7b4691e74c4122f3

Fix Travis CI Rails 4.2 bundler problems

view details

push time in 5 days

push eventfelixbuenemann/better_errors

Felix Bünemann

commit sha 53f3795ac764d75025814ec93f5bd4a727fcf52e

Fix Travis CI Rails 4.2 bundler problems

view details

push time in 5 days

create barnchfelixbuenemann/better_errors

branch : fix-rails-4.2-ci

created branch time in 5 days

issue commentdweidenfeld/plexdrive

Windows 10

@fuomag9 You can ask https://github.com/bazil/fuse about adding support for WinFsp, because that is the Go package that Plexdrive uses for interacting with FUSE:

sneakypete73

comment created time in 5 days

PR opened BetterErrors/better_errors

Allow skipping variable inspection by class name

This adds a new global setting BetterErrors.ignored_classes that will skip inspecting the value of local and instance variables of this type.

The default setting ignores action dispatch request and response instances, which are usually not printed anyways due to exceeding the limit of BetterErrors.maximum_variable_inspect_size, but also avoids all the memory allocations and database queries that are triggered by calling #inspect on variables.

The default setting was chosen to avoid excessive database queries for unloaded relations, because both the ActionDispatch::Request and the ActionDispatch::Response objects hold many references to the controller instance, causing the same instance variables to be inspected over and over again. (See https://github.com/rails/rails/pull/38445 for details.)

This is a safer alternative to the undocumented setting BetterErrors.ignored_instance_variables, which could be set to [:@_request, :@_response] for Rails. It also doesn't completely hide the variables, but instead shows a message why the value was omitted, similar to maximum_variable_inspect_size.

+41 -6

0 comment

4 changed files

pr created time in 5 days

push eventfelixbuenemann/better_errors

Felix Bünemann

commit sha faa39ec4a2f8233a194a33bbd383a7c302f25dc1

Allow skipping variable inspection by class name This adds a new global setting `BetterErrors.ignored_classes` that will skip inspecting the value of local and instance variables of this type. The default setting ignores action dispatch request and response instances, which are usually not printed anyways due to exceeding the limit of `BetterErrors.maximum_variable_inspect_size`, but also avoids all the memory allocations and database queries that are triggered by calling `#inspect` on variables. The default setting was chosen to avoid excessive database queries for unloaded relations, because both the `ActionDispatch::Request` and the `ActionDispatch::Response` objects hold many references to the controller instance, causing the same instance variables to be inspected over and over again.

view details

push time in 5 days

create barnchfelixbuenemann/better_errors

branch : ignore-variables-by-class-name

created branch time in 5 days

PR opened BetterErrors/better_errors

Clear error page state before each request

This clears out the @error_page state before each request to avoid dirtying the error state of subsequent requests. For example a name error in the controller will cause an inspect call on the controller instance that will then also inspect the @error_page ivar referenced through the rack env in the controller. One possible side-effect of this are excessive database queries for unloaded relations.

This also should reduce memory usage, since the app can garbage collect the no longer referenced state, delete temp files, etc.

For more info see the discussion on: https://github.com/rails/rails/pull/38445

+1 -0

0 comment

1 changed file

pr created time in 5 days

create barnchfelixbuenemann/better_errors

branch : fix-error-page-leak

created branch time in 5 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

I've also found out why the problem gets so much worse when using better_errors.

The first problem is that by default Better Errors inspects the @_request and @_response ivars, which both hold 14 references to the controller instance, so each trigger 14 additional inspect calls per ivar. These can be worked around by setting BetterErrors.ignored_instance_variables = [:@_request, :@_response] in an initializer.

The second problem occurs only from the second request after an error, because BetterErrors::Middleware stores its error state / session in an @error_page ivar on the middleware instance, causing 45 references per ivar when NameError#message is called, because it ends up being referenced in the ivars of the controller.

That last problem can easily be fixed by resetting the @error_page ivar in the middleware prior to each normal request, so I'm going to submit a PR to better_errors to fix that.

felixbuenemann

comment created time in 5 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

I've inspected the MRI C code for NameError and what happens is this:

NameError#message will call #inspect on the object where the name error occurred, unless it is true, false or nil. Since the exception in this case happened in the controller, it will call inspect on the controller, which in turn calls inspect on all its instance variables. Then it checks to see if the result of the inspect call is longer than 65 bytes and if yes, it just calls #to_s and uses that in the error message, which means all the work done by inspect was for naught.

felixbuenemann

comment created time in 5 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

I spoke too soon, the did_you_mean gem was cloaking the original call locations, with RUBYOPT="--disable-did_you_mean" things became clearer:

  • 1x inspect from ActiveSupport::Notifications::Instrumenter#instrument caused by the line payload[:exception] = [e.class.name, e.message] (activesupport-6.0.2.1/lib/active_support/notifications/instrumenter.rb:26:in `inspect')
  • 1x inspect from Airbrake::NestedException#as_json calling exception.message (yeah, not default middleware, forgot to remove that one)
  • 1x inspect from ActionDispatch::DebugExceptions#log_error caused by message << "#{exception.class} (#{exception.message}):" (actionpack-6.0.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:150:in `inspect')

The actual calls are triggered by calling NameError#message, so it can't really be avoided and anything working with exceptions will trigger the inspect calls.

felixbuenemann

comment created time in 5 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

I'm assuming our default middleware stack needs to inspect those things but I don't remember where those inspect calls happen. Can you point them out?

Thanks for the suggestion. Actually all inspect calls with the default middleware stack are caused by the did_you_mean gem that comes bundled with ruby:

ActiveRecord::Relation#inspect called 3 times from /Users/felix/.rvm/rubies/ruby-2.6.5/lib/ruby/gems/2.6.0/gems/did_you_mean-1.3.0/lib/did_you_mean/core_ext/name_error.rb:8:in `inspect'

NameError (undefined local variable or method `xxx' for #ProductFamiliesController:0x00007fd28e689c10):

felixbuenemann

comment created time in 5 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

This is going to degrade the experience we want to give to the Rails console.

User.all in a console should records, not show nothing.

I'm not convinced that we need to memoize here.

So would you be ok with triggering a load on inspect then, instead of doing a separate limit query?

Unless the 4x with the default middleware stack are happening in production, or it makes development unbearable, it is simpler for the code and as behavior to not memoize.

On the app I was working on the default middleware stack caused about 200-300 ms additional delay without memoization, which is not so bad, but due to spamming the log with multiple pages of query logs it was very hard to find the actual error output in the logs, since it was before all the inspect calls.

With both meta-request and better_errors active the inspect calls caused several seconds additional delay and easily exhausted the 10,000 line scroll back buffer in my terminal.

Better Errors certainly has its own problems, since it sometimes does 100 inspects per ivar and additional ones when navigating the stack, so I've replaced it with web-console for now.

The instance variables with unloaded relations contain some deeply nested preloading for rendering navigation that is usually only loaded if the relevant fragment caches get busted.

Since the app uses UUID PKs the SQL for preload queries can become quite long.

felixbuenemann

comment created time in 6 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

I've tried to look at the problem from a different perspective:

If I inspect something, I would like to see it's current state, not trigger side-effects.

What do you think about the following logic:

  • If the relation is loaded, show the first 10 records in the inspect output.
  • If the relation is not loaded, show no records (or empty records collection)

Additionally it might be worthwile to include the loaded flag in the inspect output.

This way we never do unneccessary queries and calling inspect on unloaded relations is much cheaper and free of side effects and we can skip the memoization.

felixbuenemann

comment created time in 7 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

Just using normal load behavior would work. The idea of the original code was to load less records for inspect, since only the first 10 are shown in the inspect output.

It would certainly be simpler, if we removed that logic and always loaded the original relation and do the limiting in ruby.

felixbuenemann

comment created time in 7 days

Pull request review commentrails/rails

Memoize ActiveRecord::Relation#inspect

 def values     end      def inspect-      subject = loaded? ? records : self-      entries = subject.take([limit_value, 11].compact.min).map!(&:inspect)+      @inspect ||= begin+        subject = loaded? ? records : self

No, if the relation is not yet loaded the subject.take triggers a limit query each time inspect is called.

felixbuenemann

comment created time in 7 days

issue commentdweidenfeld/plexdrive

Getting invalid_client

You can remove ~/.plexdrive/config.json and ~/.plexdrive/token.json, then you should be able to redo the setup steps.

sogeniusio

comment created time in 8 days

pull request commentrails/rails

Memoize ActiveRecord::Relation#inspect

Updates to include Changelog entry.

felixbuenemann

comment created time in 8 days

push eventfelixbuenemann/rails

Felix Bünemann

commit sha 6d1203966b44e8c825ff829f02c29c8e09394e9b

Memoize ActiveRecord::Relation#inspect Avoid excessive database queries when unloaded relations are inspected. This commonly happens if there are lazy loaded relations assigned to instance variables in a controller and an exception occurs, causing many inspect calls per instance variable in the exception handlers.

view details

push time in 8 days

push eventfelixbuenemann/rails

Felix Bünemann

commit sha 458cc9328dc96020954cb950ffd3ac463b2a84e8

Memoize ActiveRecord::Relation#inspect Avoid excessive database queries when unloaded relations are inspected. This commonly happens if there are lazy loaded relations assigned to instance variables in a controller and an exception occurs, causing many inspect calls per instance variable in the exception handlers.

view details

push time in 8 days

PR opened rails/rails

Memoize ActiveRecord::Relation#inspect

Summary

This adds memoization to ActiveRecord::Relation#inspect.

The following methods invalidate the cached @inspect text:

  • ActiveRecord::Relation#reset
  • ActiveRecord::Relation#load (unless already loaded)
  • ActiveRecord::Associations::CollectionProxy#reset_scope

Other Information

The memoization avoids excessive database queries when unloaded relations are inspected.

This commonly happens if there are lazy loaded relations assigned to instance variables in a controller and an exception occurs, causing many inspect calls per instance variable in the exception handlers.

Here are some metrics on how often controller instance variables are inspected when an exception occurs in development:

  • 4x with the default middleware stack
  • 7x with meta-request active
  • 2 x 33x with better_errors active (and additional times when navigating the stack)

Another area were this helps is when working with IRB in the rails console, because every return statement is inspected automatically by IRB.

+56 -5

0 comment

4 changed files

pr created time in 8 days

create barnchfelixbuenemann/rails

branch : memoize-activerecord-relation-inspect

created branch time in 8 days

push eventfelixbuenemann/rails

Felix Bünemann

commit sha bfd2497d6ff9e9c0bd8601a7c037faf3d4a2eeb8

Memoize ActiveRecord::Relation#inspect Avoid excessive database queries when unloaded relations are inspected. This commonly happens if there are lazy loaded relations assigned to instance variables in a controller and an exception occurs, causing many inspect calls per instance variable in the exception handlers.

view details

push time in 8 days

startedankane/strong_migrations

started time in 11 days

issue commentfelixbuenemann/xlsxtream

Money type

I'd like to avoid coupling xlsxtream to third-party Gems and there are many different ways to format monetary values (leading / trailing symbols, different rounding, thousands separators etc.), so having a one-fits-all format for that doesn't really work.

I've done a spike on adding support for custom formatters, but ran out of time and I'm not yet sure of the performance implications due to the additional indirection, so I'm not sure of I'll add it, since performance is a number one goal of this gem.

In the meantime the easiest workaround would be to store the amount and currency of the money instance in different columns of the worksheet, which also makes it much easier to work with in Excel.

Nerian

comment created time in 12 days

starteddoitsujin/fand

started time in 13 days

issue commentdweidenfeld/plexdrive

Does Partial Scan work with PD

That doesn't matter, both directories work fine.

rrbailey89

comment created time in 16 days

issue commentdweidenfeld/plexdrive

Does Partial Scan work with PD

Oh and the plexwatcher.service in the gist assumes that your plexdrive is mounted by a systemd unit called plexdrive.service, so you need to change that if yours is called differently.

rrbailey89

comment created time in 16 days

issue commentdweidenfeld/plexdrive

Does Partial Scan work with PD

The plexwatcher.service i posted already runs as user/group plex.

You can put the file in /lib/systemd/system/plexwatcher.service and then use systemctl daemon-reload and systemctl start plexwatcher.service. You can then run systemctl status plexwatcher.service to see if it runs OK and if everything is fine you can enable it permanently with systemctl enable plexwatcher.service.

I think it is normal that you see no status spinner in the web frontend when manually calling the Plex Media Scanner, but the new media should pop up once it is done.

rrbailey89

comment created time in 16 days

issue commentdweidenfeld/plexdrive

Does Partial Scan work with PD

Hmm, it's strange that those files are owned by root and not the plex user.

Can you run ps aux | grep 'Plex Media Server' and check what user the process /usr/lib/plexmediaserver/Plex Media Server is running at?

If the files in /var/lib/plexmediaserver are also owned by root you will probably run into permission problems, when you try to run the script as user plex.

rrbailey89

comment created time in 16 days

issue commentdweidenfeld/plexdrive

Does Partial Scan work with PD

It looks like your missing the '-pipe=true' command line option for watcher. Also make sure the LD_LIBRARY_PATH in plexwatcher.py is correct or the Plex Media Scanner will crash.

rrbailey89

comment created time in 18 days

issue commentdweidenfeld/plexdrive

TeamDrive support from #319 seems to have broken root-node-id for non-teamdrives

Please retry with current master.

Nothing4You

comment created time in 18 days

delete branch felixbuenemann/plexdrive

delete branch : team-drive-fixes

delete time in 18 days

push eventdweidenfeld/plexdrive

Felix Bünemann

commit sha f12bbe593d6d07ecbfd050596076109b7610cdfd

Add --drive-id option for team drives This fixes the problem where --root-node-id was broken for non-team drives and also should allow mounting a sub.directory of a team drive by specifying both --drive-id and --root-node-id.

view details

Felix Bünemann

commit sha 1486bf3d53de9e708c973ea6e2a54b8892b9242d

Ignore changes with changeType drive This should fix the unexpected change records without a file resource. Additionally we request to include the file resource for removals, which could be another reason of missing file resource in changes.

view details

Felix Bünemann

commit sha d2a14ee3bb27b5c9049ddfa795be7088acfa51cd

Update API to support all shared drives

view details

Felix Bünemann

commit sha afa5e4e8c125e2758398dc72ab19dc7ea74e1a8c

Merge pull request #342 from felixbuenemann/team-drive-fixes Team Drive Fixes

view details

push time in 18 days

PR merged dweidenfeld/plexdrive

Team Drive Fixes

This fixes a couple of problems with the Team Drive support:

  • Team Drive support now uses a new option --drive-id instead of overloading the --root-node-id option, which fixes #334 and makes --root-node-id work with subfolders of a team drive.
  • Ignore change records of type "drive" and always include the file resource on change records for removals.
  • Update drive API calls for shared drives to remove deprecated usage (the new calls are also deprecated some time in 2020, at which time shared drives are always included).
+38 -27

2 comments

3 changed files

felixbuenemann

pr closed time in 18 days

issue closeddweidenfeld/plexdrive

TeamDrive support from #319 seems to have broken root-node-id for non-teamdrives

Hi,

I used the 5.0.0 release so far, now I wanted to try the current master to also mount a teamdrive next to my normal drive. The newer version including #319 (current master) does not allow me to mount a custom subfolder as root mount anymore:

plexdrive 5.0.0

[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Opening cache connection
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Authorizing against Google Drive API
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Loading token from cache
[PLEXDRIVE.BAK] [2019-09-15 12:58] INFO   : Mounting path /media/plexdrive
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Checking for changes
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Getting start page token from cache
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Last change id found, continuing getting changes (15854177)
[PLEXDRIVE.BAK] [2019-09-15 12:58] INFO   : First cache build process started...
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Getting root from API
[PLEXDRIVE.BAK] [2019-09-15 12:58] INFO   : Processed 2 items / deleted 0 items / updated 2 items
[PLEXDRIVE.BAK] [2019-09-15 12:58] DEBUG  : Storing page token 15854182 in cache
[PLEXDRIVE.BAK] [2019-09-15 12:58] INFO   : First cache build process finished!

plexdrive bf61a476178d450e969bbf8fcc910c4c4c2aa25b

[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Opening cache connection
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Authorizing against Google Drive API
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Loading token from cache
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Checking for changes
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Getting start page token from cache
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Last change id found, continuing getting changes (15854182)
[PLEXDRIVE] [2019-09-15 12:59] INFO   : First cache build process started...
[PLEXDRIVE] [2019-09-15 12:59] INFO   : Mounting path /media/plexdrive
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : Getting root from API
[PLEXDRIVE] [2019-09-15 12:59] DEBUG  : googleapi: Error 404: Shared drive not found: some_folder_in_my_drive, notFound
[PLEXDRIVE] [2019-09-15 12:59] WARNING: Could not get changes
[PLEXDRIVE] [2019-09-15 12:59] INFO   : First cache build process finished!

I don't think this duplicates #333 as my problem doesn't involve using team drives.

closed time in 18 days

Nothing4You

issue commentfelixbuenemann/xlsxtream

Money type

So what exactly do you expect to happen?

The OpenXML Spreadsheet format has no Type for money, only numbers. The only thing ine could do is apply one of the pre-defined number formats if instances of Money are detected.

I'm also not sure why you are calling Money#format instead of #to_f or #to_d.

Nerian

comment created time in 20 days

pull request commentxbmc/xbmc

Fix Linux GL Renderer color primaries detection

This PR has been stale for a while – is there anything left to do for me here?

felixbuenemann

comment created time in 20 days

issue commentdweidenfeld/plexdrive

Does Partial Scan work with PD

I'm using watcher and a custom python script and systemd service to do polling and partial library updates.

It's been running since March 2019 without problems.

rrbailey89

comment created time in 20 days

pull request commentdweidenfeld/plexdrive

Only store each downloaded chunk once

@dweidenfeld It would be great to get your thoughts on this.

I have a local feature branch that implements a mmapped backing store for the chunk cache and I noticed that chunks were stored over and over again while developing it, sometimes 10 times for the same chunk.

This also happens with the current memory only cache and causes quite some stress on the garbage collector, because the same chunk gets stored and evicted multiple times in a row.

I'm not sure if this last-callback-wins change is the right solution to the problem, maybe you've got a better idea.

felixbuenemann

comment created time in 21 days

pull request commentdweidenfeld/plexdrive

Team Drive Fixes

@dweidenfeld If we get these changes and #343 merged I think we're good to go for a 5.1.0 release with Team Drive support.

felixbuenemann

comment created time in 21 days

pull request commentdweidenfeld/plexdrive

Use go modules to lock dependencies

Closing this in favor of #340 which was merged.

felixbuenemann

comment created time in 21 days

PR closed dweidenfeld/plexdrive

Use go modules to lock dependencies

This is an alternative to #340 and uses Go Modules introduced in Go 1.11 to lock down dependencies to compatible versions.

Specifically bazil.org/fuse is locked down to bazil/fuse@742914fa44b872401579c0619775301997526e3a to keep the fuse.AllowRoot option and compatibility with Go < 1.13.

Since Go Modules requires 1.11+ we drop older Go versions on Travis CI.

+149 -1

3 comments

3 changed files

felixbuenemann

pr closed time in 21 days

PR opened dweidenfeld/plexdrive

Remove allow_root option from README

This option was removed in #340.

+0 -1

0 comment

1 changed file

pr created time in 21 days

create barnchdweidenfeld/plexdrive

branch : remove-allow-root-from-readme

created branch time in 21 days

PR opened dweidenfeld/plexdrive

Team Drive Fixes

This fixes a couple of problems with the Team Drive support:

  • Team Drive support now uses a new option --drive-id instead of overloading the --root-node-id option, which fixes #334 and makes --root-node-id work with subfolders of a team drive.
  • Ignore change records of type "drive" and always include the file resource on change records for removals.
  • Update drive API calls for shared drives to remove deprecated usage (the new calls are also deprecated some time in 2020, at which time shared drives are always included).
+38 -27

0 comment

3 changed files

pr created time in 21 days

create barnchfelixbuenemann/plexdrive

branch : team-drive-fixes

created branch time in 21 days

issue commentdweidenfeld/plexdrive

TeamDrive support from #319 seems to have broken root-node-id for non-teamdrives

I think the way to go is to add a new Option like --team-drive-id that enables team drive mode, which also allows to use a different root-node-id with team drives.

Nothing4You

comment created time in 25 days

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

@dweidenfeld I bumped Travis to Go 1.13+ to make the build happy. Do you mind reviewing so I can merge the fix? (I can't review my own PRs)

felixbuenemann

comment created time in 25 days

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha f5837b6facaa0f427e8be04ceb1a96e565a0c66a

Bump CI to at least Go 1.13 This is needed to build against the latest version of basil.org/fuse.

view details

push time in 25 days

startedtrapexit/bbf

started time in 25 days

push eventfelixbuenemann/rclone

Felix Bünemann

commit sha dc398da56377452534374460e79bf91cf30e01ea

premiumize: add fast list support Since the /item/listall endpoint only returns files without directories, we need a new feature flag to enable synthesizing dirs similar to blob storage like S3.

view details

push time in 25 days

push eventfelixbuenemann/rclone

Felix Bünemann

commit sha 788b4749a7051aca84c63f9cd51df876a0ef6b14

premiumize: Add fast list support Since the /item/listall endpoint only returns files without directories, we need a new feature flag to enable synthesizing dirs similar to blob storage like S3.

view details

push time in 25 days

create barnchfelixbuenemann/rclone

branch : premiumize-fast-list-support

created branch time in 25 days

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

@dweidenfeld Just because we support older Go versions doesn't mean we can't support new Go versions.

I think it makes sense to support older Go versions that are still widely used, at least if we don't have regular releases.

If there are regular releases it doesn't really matter, since people can just download the right binaries for their system.

Do you know if it is possible and would you be OK with if contributors like me to publish new releases?

felixbuenemann

comment created time in a month

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

I've pushed an alternate solution using Go Modules to lock down dependencies in #341.

Unfortunately that also drops Go < 1.11.4 from the supported versions, since that is the minimum version with the required Go Modules support.

Potentially Plexdrive can still be built on older Go versions, if the dependencies are properly downgraded using "go get", but I have no intention of supporting such hacks in CI or introducing a third party dependency management tool.

felixbuenemann

comment created time in a month

pull request commentdweidenfeld/plexdrive

Use go modules to lock dependencies

OK, seems to be a quirk with go modules on Go < 1.11.4, so I bumped the version in the Travis CI config.

felixbuenemann

comment created time in a month

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 3f77cfb2bb1254b167834c06eb13575d8813f067

Bump Travis to go1.11 to 1.11.4 to fix build since it changed the semantics of go version checks for modules.

view details

push time in a month

pull request commentdweidenfeld/plexdrive

Use go modules to lock dependencies

Seems we also need to lock down golang.org/x/sys/unix:

go build golang.org/x/sys/unix: module requires Go 1.12

felixbuenemann

comment created time in a month

PR opened dweidenfeld/plexdrive

Use go modules to lock dependencies

This is an alternative to #340 and uses Go Modules introduced in Go 1.11 to lock down dependencies to compatible versions.

Specifically bazil.org/fuse is locked down to bazil/fuse@742914fa44b872401579c0619775301997526e3a to keep the fuse.AllowRoot option and compatibility with Go < 1.13.

Since Go Modules requires 1.11+ we drop older Go versions on Travis CI.

+149 -1

0 comment

3 changed files

pr created time in a month

push eventfelixbuenemann/plexdrive

Felix Bünemann

commit sha 17544b6abc0577bf0aa625df96388c7f263213d3

Drop go < 1.11 CI support due to go modules and update build go version matrix.

view details

push time in a month

create barnchfelixbuenemann/plexdrive

branch : use-go-modules

created branch time in a month

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

Yeah, current Ubuntu LTS 18.04 ships with 1.10, but you can add a PPA for newer versions.

felixbuenemann

comment created time in a month

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

@dweidenfeld What's you position on this? – Should we drop support for golang < 1.13 or vendor the fuse package?

felixbuenemann

comment created time in a month

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

I opened https://github.com/bazil/fuse/issues/236 to clarify backwards compatibility.

Depending on how it goes we might consider vendoring an older version of the fuse package.

felixbuenemann

comment created time in a month

issue openedbazil/fuse

errors.As not supported on go < 1.13

The commit 3bebd277fa844a81aaabccac0c6ee81b5e59205b broke support for golang < 1.13.

Is there any way to implement it without breaking backwards compatibility?

created time in a month

pull request commentdweidenfeld/plexdrive

Remove allow_root mount option

Looks like bazil.org/fuse master also broke support for go < 1.13:

bazil.org/fuse/fuse.go:410:5: undefined: errors.As

felixbuenemann

comment created time in a month

PR opened dweidenfeld/plexdrive

Remove allow_root mount option

This is no longer supported, because fuse.AllowRoot() was removed in https://github.com/bazil/fuse/commit/e0b89709bc6481f1345b6e19363cf18766e53fc7.

Fixes #339

+1 -1

0 comment

1 changed file

pr created time in a month

create barnchfelixbuenemann/plexdrive

branch : remove-allow-root-mount-option

created branch time in a month

issue commentdweidenfeld/plexdrive

mount.go: undefined: fuse.AllowRoot ???

Looks like it is caused by an update to bazil.org/fuse:

https://github.com/bazil/fuse/commit/e0b89709bc6481f1345b6e19363cf18766e53fc7

I'll look into it.

bjstrange

comment created time in a month

issue commentdweidenfeld/plexdrive

Still unable to download from plexdrive mount on 1cpu

Did you build the latest version from the master branch?

bin28281010

comment created time in a month

startedjorgemanrubia/turbolinks_render

started time in a month

startedrenderedtext/render_async

started time in a month

pull request commentxbmc/xbmc

Fix Linux GL Renderer color primaries detection

@Rechi I've fixed the clang-format violations in newly added code, but kept the existing code as is to avoid obfuscating the changes.

felixbuenemann

comment created time in a month

push eventfelixbuenemann/xbmc

Felix Bünemann

commit sha ed7ba06632a7fde9e5092b8599b067bcdb21d454

Fix Linux GL Renderer color primaries detection This makes sure the videos source primaries are passed to the color manager. Also adapted the ColorManager interface to accept AVColorPrimaries instead of CONF_FLAGS_COLPRI which required updates to the Windows renderer base.

view details

push time in a month

pull request commentxbmc/xbmc

Fix Linux GL Renderer color primaries detection

@FernetMenta I've pushed a new version with the requested changes.

The ColorManager now gets passed AVColorPrimaries instead of flags.

I've also updated the Windows renderer accordingly, but couldn't test those changes.

So it would be great if someone could test this on Windows.

felixbuenemann

comment created time in a month

push eventfelixbuenemann/xbmc

Felix Bünemann

commit sha 26d78cfa7bedbd6a7d06e0e267c526c767a4a8ca

Fix Linux GL Renderer color primaries detection This makes sure the videos source primaries are passed to the color manager. Also adapted the ColorManager interface to accept AVColorPrimaries instead of CONF_FLAGS_COLPRI which required updates to the Windows renderer base.

view details

push time in a month

pull request commentxbmc/xbmc

Fix Linux GL Renderer color primaries detection

@FernetMenta I just replicated what these renderers are currently doing:

https://github.com/xbmc/xbmc/blob/dee52eafc89569449b1be7eae200c24eb76b1655/xbmc/cores/VideoPlayer/VideoRenderers/HwDecRender/RendererDRMPRIME.cpp#L75 https://github.com/xbmc/xbmc/blob/244618bf3a127d879370776a7a0bd4fd7f0bcd40/xbmc/cores/VideoPlayer/VideoRenderers/HwDecRender/RendererMediaCodecSurface.cpp#L59 https://github.com/xbmc/xbmc/blob/770c691bfeea88d561b39d2e38348ac434db3adc/xbmc/cores/VideoPlayer/VideoRenderers/WinRenderer.cpp#L164 https://github.com/xbmc/xbmc/blob/770c691bfeea88d561b39d2e38348ac434db3adc/xbmc/cores/VideoPlayer/VideoRenderers/HwDecRender/MMALRenderer.cpp#L934

The only place I found where it seems to be updated for a buffer is here:

https://github.com/xbmc/xbmc/blob/770c691bfeea88d561b39d2e38348ac434db3adc/xbmc/cores/VideoPlayer/VideoRenderers/windows/RendererBase.cpp#L409

Do you propose that all these renderers need to be fixed?

Also can you elaborate or have an example video where the color primaries change on the fly?

Reading the code of the Linux GL renderer it looks like the primaries could changhe on the fly, but are there real world cases where that happens?

I'm a bit concerned about the cost of possibly checking if the color manager config has changes for each frame instead of once.

felixbuenemann

comment created time in a month

PR opened xbmc/xbmc

Fix Linux GL Renderer color primaries detection

Description

The m_iFlags are passed to ColorManager for detecting the color primaries of the video, so it needs to contain the color primary flags for the Colour Management -> Primaries -> Automatic to work as expected.

This is a follow-up to #17109 and fixes desaturated colours for BT.2020 videos that were previously handled at BT.709 because that is the fallback value, when no color primaries flag is set.

Motivation and Context

After #17109 color management can finally be used with the Linux GL Renderer without crashing. When testing this with 4K HDR videos with BT.2020 primaries I noticed that the colours were completely desaturated. After manually setting Colour Management -> Primaries -> UHDTV (BT.2020), the colors were accurate.

This fix is essential for playing back BT.2020 videos on non-BT.2020 capable displays (eg. sRGB or BT.709).

Since the bug also affects Leia it should be backported, similar to the changes from #17109.

How Has This Been Tested?

The fix was tested against master@f4f9d88a35 with both software decoding and VAAPI.

The bug was observed without the fix on both Leia stable and nightly builds on Ubuntu 19.10 with Gemini Lake (Intel UHD 605) graphics and both i965 and iHD VAAPI drivers as well as software decoding.

The following settings were used:

  • Color management: Enabled
  • Colour management mode: ICC Profile
  • ICC display profile: (path to monitors ICM profile extracted from windows driver, a standard sRGB.icc should work as well for testing)
  • Whitepont: D65
  • Gamma mode: BT.1886
  • Lookup table size: 4 or 5 (doesn't matter)

The following videos were tested:

  • 2160p HDR BT.2020 HEVC MKV
  • 1080p SDR BT.709 AVC MKV

For some movies both 4K and Full-HD versions were tested to compare the color timing and the BT.2020 color levels after tone mapping and color management on an Asus PA328Q (100% sRGB IPS) were compared to a Sony Z9D HDR TV as a reference.

Without the fix all BT.2020 content looked completely desaturated with either colour management disabled or set to Primaries -> Automatic, but looked fine with Primaries -> BT.2020. Conversely BT.709 videos looked ok with Automatic, but oversaturated with BT.2020 hinting at the broken auto-detection.

Screenshots (if appropriate):

Example 1080p SDR BT.709 w/o CM: lk_sdr_bt709_no_cm

Same with CM enabled and automatic primaries: lk_sdr_bt709_cm_auto_primaries

Example 2160p HDR BT.2020 Reinhard Tone Mapping to SDR w/o CM: lk_hdr_bt2020_no_cm

Same with CM enabled and automatic primaries (after fix): lk_hdr_bt2020_cm_auto_primaries

Same with CM enabled and BT.709 primaries (simulates "automatic" w/o fix): lk_hdr_bt2020_cm_bt709_primaries

Please view screenshots in fullscreen for best results.

Types of change

<!--- What type of change does your code introduce? Put an x in all the boxes that apply like this: [X] -->

  • [x] Bug fix (non-breaking change which fixes an issue)
  • [ ] Clean up (non-breaking change which removes non-working, unmaintained functionality)
  • [ ] Improvement (non-breaking change which improves existing functionality)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that will cause existing functionality to change)
  • [ ] Cosmetic change (non-breaking change that doesn't touch code)
  • [ ] None of the above (please explain below)

Checklist:

<!--- Go over all the following points, and put an X in all the boxes that apply like this: [X] --> <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->

  • [x] My code follows the Code Guidelines of this project
  • [ ] My change requires a change to the documentation, either Doxygen or wiki
  • [ ] I have updated the documentation accordingly
  • [x] I have read the Contributing document
  • [ ] I have added tests to cover my change
  • [ ] All new and existing tests passed (not tested)
+1 -0

0 comment

1 changed file

pr created time in a month

more