profile
viewpoint
Alexander Larsson alexlarsson Sweden

alexlarsson/bundler 7

Simple app bundle implementation

alexlarsson/broadway-docker 5

Sources for a docker image with broadway

alexlarsson/fedora-docker 5

Some test docker images based on fedora

alexlarsson/docker 4

Docker - the open-source application container engine

alexlarsson/dtools 3

Tools for streaming gvariants in pipes

alexlarsson/fdrpc 3

A version of golang rpc that supports passing file descriptors

alexlarsson/fedora-apps 1

experimental xdg-apps based on fedora rpms

push eventalexlarsson/flat-auth

Dan Nicholson

commit sha 14c132e84230fae7dee204064e6f181e727c97b0

Construct URLs with url_for instead of requiring baseURL Flask provides the url_for helper to construct URLs to endpoint functions. Any arguments that aren't handled by the method are appended as query arguments. An absolute URL is constructed when _external is True. The server address and protocol are properly detected from the current request. This includes a proxied request so long as the reverse proxy properly forwards the Host and X-Forwarded-Proto headers. After this baseURL is not needed, which simplifies the configuration by taking the hostname out of the picture.

view details

push time in 21 hours

PR merged alexlarsson/flat-auth

Reviewers
Construct URLs with url_for instead of requiring baseURL

Flask provides the url_for helper to construct URLs to endpoint functions. Any arguments that aren't handled by the method are appended as query arguments. An absolute URL is constructed when _external is True. The server address and protocol are properly detected current request or a proxied request so long as the reverse proxy properly forwards the Host and X-Forwarded-Proto headers. After this baseURL is not needed, which simplifies the configuration by taking the hostname out of the picture.

+10 -8

0 comment

1 changed file

dbnicholson

pr closed time in 21 hours

pull request commentflatpak/flatpak-builder

Support passing --disable-rofiles-fuse to --run

Ok then. I think any time we do builder_cache_checkout () with options.force_copy = FALSE, we should also create a file .hardlinked-cache in the checkout dir, and we should then filter this out in commit_filter() in the same file. Then flatpak-builder run should be changed to only enable rofuse if this file exists in the builddir.

barthalion

comment created time in 5 days

pull request commentflatpak/flatpak-builder

Support passing --disable-rofiles-fuse to --run

This is an extremely dangerous operation. Typically the files in /app (ie. the build dir) are hardlinks into the flatpak-builder build cache, and unless you use rofiles-fuse the container can do modifications of the files in there which changes whats in the cache breaking it in unexpected ways.

We do allow --disable-rofiles-fuse in the build stages because we then are in control of what we're running against and can ensure the cache checksouts are not hardlinks (by setting force_copy=TRUE in the cache ostree checkout operation). This makes the build slower but its safe. However, "flatpak-builder --run" just uses whatever is currently in the builddir, and there is no guarantee that what is there was built with --disable-rofiles-fuse.

Maybe we can store some kind of hint in the builddir when it contains a hardlinked cache checkout and allow disabling rofiles if that is not there. In fact, if we did that we could automatically skip the rofiles fuse when it is not necessary.

What exactly is your usecase for this?

barthalion

comment created time in 5 days

issue commentflatpak/flatpak

Sort refs before prompting to disambiguate

Related to: https://github.com/flatpak/flatpak/pull/2500 (don't show already installed refs as candidates)

wjt

comment created time in 5 days

issue commentflatpak/flatpak

Cleanup empty dirs in repo/refs/ that linger after app is uninstalled

I agree with @mwleeds. This should really be done by ostree during pruning.

uajain

comment created time in 5 days

issue closedflatpak/flatpak

Requirements for WebKit flatpak-spawn sandbox

WebKitGTK added support for a flatpak-spawn backed sandbox but it was removed because it didn't expose enough options for us. This is just a list of some changes we would like:

  • --sandbox is all or nothing

    In the end we just used --no-network which is a good start but we would like that to be expanded so for example we can selectively disable DBus access and remove all host filesystem access but keep Wayland access.

    Something as simple as --no-filesystem and --no-dbus might do; We currently pass through pulseaudio, wayland, and dri so ideally not a single other permission is exposed in the end. Maybe the reverse approach of --sandbox --socket=wayland makes more sense.

  • --sandbox-expose is too limiting

    We expose API allowing applications to mount any path in the sandbox and requiring all files be in a specific directory is not useful for us.

    I realize this is tricky but we need the ability to mount any path in the current sandbox (including inside /tmp for example) into the newly made sandbox.

closed time in 5 days

TingPing

issue commentflatpak/flatpak

Dbus services not working

@dvdhrm Flatpak creates two fixed locations where the service files are put, /var/lib/flatpak/exports/share/dbus-1/services/ for system installed flatpak and ~/.local/share/flatpak/exports/share/dbus-1/services/ for per-user installed ones. There is no real way for flatpak the app to modify the dbus daemon to pick this up, instead it is up to each packager of flatpak to use whatever setup the distro has to add these directories to the XDG_DATA_DIRS env var before the dbus daemon is started.

What I think is weird in this report is that installing a flatpak, which created a service file in one of these dirs, didn't cause dbus-broker to find the new service. However, once rebooted it did find it. Assuming flatpak was installed at the time of the last boot, the XDG_DATA_DIRS env var should have been the same in both these runs, so why did it require a reboot to pick up the new service?

damianatorrpm

comment created time in 5 days

issue closedflatpak/flatpak

Applications do not start at all if $HOME is a subdirectory of /usr

Flatpak version

1.4.2

Description of the problem

One of the machines that I use has the home directory under /usr. On that machine, trying to run any flatpak application fails with a message along the lines of:

bwrap: Can't mkdir parents for /usr/.../home/$USER/.var/app/org.darktable.Darktable/config/user-dirs.dirs: Read-only file system

I am only speculating that this is due to $HOME being under /usr but it seems like a plausible explanation to me, given the blacklisting of /usr mentioned in the documentation.

In any case, contrary to what the message indicates, it did manage to create the parent directories in question.

closed time in 5 days

sboukortt

issue commentflatpak/flatpak

Applications do not start at all if $HOME is a subdirectory of /usr

Yes, this is never going to work, because flatpak assumes that /usr is under control of the runtime. We can't in general modify what exists there, and we can't even mount the home directory there because the required mountpoint will not exist in the runtime.

sboukortt

comment created time in 5 days

issue closedflatpak/flatpak

[Suggestion] Override Context filesystems alias???

Linux distribution and version

Fedora Silverblue rawhide

Flatpak version

flatpak-1.5.2-1.fc32.x86_64

Description of the problem

Korean: 최근에 org.qutebrowser.qutebrowser app 을 사용 중입니다. 매우 만족해서 주 웹 브라우저로 사용 중입니다. DRM 비디오를 시청하기 위해 widevine 를 사용해야 하는데 /opt/google/chrome 와 같은 특정 경로를 참조하게 되어 있습니다. 그래서 아래와 같이 설정하여 사용 중입니다. English: I am using org.qutebrowser.qutebrowser app recently. It is very cool so I use it as my main web browser. To watching DRM video (widevine), it supposed to refer to a specific path like /opt/google/chrome. So I am using it by setting as below.

  • HOST: /opt/google/chrome/libwidevinecdm.so # ~/.local/share/flatpak/overrides/org.qutebrowser.qutebrowser
[Context]
filesystems=/opt/google/chrome;

Korean: 이경우 Host 에서는 실제로 사용하지 않을 경로의 디렉토리를 생성해야 합니다. 관리면에서 불편하기 때문에 몇몇 경우 아래와 같은 형태로 설정했으면 어떨까 합니다. English: In this case, Host needs to create a directory with a path that is not actually used. Because it is inconvenient for management, how about setting form as follows as necessary?

  • HOST: /path/to/want/libwidevinecdm.so
[Context]
filesystems=/path/to/want:/opt/google/chrome;

I used a translator because my English skills were poor. Sorry.

closed time in 5 days

kbjji

issue commentflatpak/flatpak

[Suggestion] Override Context filesystems alias???

For security reasons flatpak doesn't allow apps to remap paths in ways that are under the apps control, as this can be used to confuse the host in security sensitive cases like portals.

kbjji

comment created time in 5 days

issue commentflatpak/flatpak

Excessive flatpak-system-helper memory usage

One thing I find weird about this is that system-helper is meant to idle exit after 10 minutes. Does this not happen for you?

hadess

comment created time in 5 days

issue closedflatpak/flatpak

Applications inside flatpak don't work internet

I use Arch Linux And at some point the applications stopped working with the internet Applications like Skype and Spotify I've tried to reinstall, but without effect, anyone could help me?

image image

image image

closed time in 5 days

MaChInEgUn3

issue commentflatpak/flatpak

Applications inside flatpak don't work internet

Closing this then

MaChInEgUn3

comment created time in 5 days

issue closedflatpak/flatpak

[Suggestion] Environment variable to override Configuration file location

Hello,

Not an issue but I didn't know where else I could post this suggestion.

It could be useful to be able to specify a/several custom location(s) for the conf file used to define the installation location.

I'm actually installing the apps on a central location, a NFS mount, so that I do not have to install them on each one of the 40+ workstations of our company. I do so by using a conf file stored in /etc/flatpak/installation.d

That works great but in order for the users to be able to launch the apps the conf file has to be available on their workstation and I'd rather specify the location of this configuration file using an env variable than having to propagate it on each one of the stations each time I amend it.

Just an idea....

closed time in 5 days

sjkosinski

issue commentflatpak/flatpak

[Suggestion] Environment variable to override Configuration file location

There is a FLATPAK_CONFIG_DIR env var you can use.

sjkosinski

comment created time in 5 days

issue closedflatpak/flatpak

Support for Debian Buster and older

Linux distribution and version

Debian Buster or older

Flatpak version

N/A

Description of the problem

Flatpak in Debian doesn't install anything because is too old

Solution

Provide newer versions as .deb files, as is done with Ubuntu 16.04

closed time in 5 days

sudo-give-me-coffee

issue commentflatpak/flatpak

Support for Debian Buster and older

Buster seems to have 1.2.5: https://packages.debian.org/buster/flatpak

We don't currently do (or plan to) backports to debian, all debian flatpak packaging happens upstream, so you'll have to bring this up with the debian maintainers.

sudo-give-me-coffee

comment created time in 5 days

issue commentflatpak/flatpak

Command to determine if flatpak app is masked

We could add this info in flatpak info. At least for masked installed apps. Won't work for non-installed things obviously.

be2c38e286fff7df25d17e21294604a8

comment created time in 5 days

issue commentflatpak/flatpak

flatpak enter: error: Can't enter user namespace: Invalid argument

Does gentoo use a setuid bwrap or a regular one (i.e. user namespaces)? There are issues with flatpak enter and the setuid case.

devurandom

comment created time in 5 days

issue commentflatpak/flatpak

fcntl(F_ADD_SEALS): Device or resource busy

Just to isolate this, any chance you could boot into text-mode without the nvidia driver and see if this is reproducible?

aanno

comment created time in 5 days

issue commentflatpak/flatpak

fcntl(F_ADD_SEALS): Device or resource busy

But how could there be a mapping? We just created it and we didn't map it!

aanno

comment created time in 5 days

issue commentflatpak/flatpak

fcntl(F_ADD_SEALS): Device or resource busy

Oh:

       EBUSY  cmd is F_ADD_SEALS, arg includes F_SEAL_WRITE, and there
              exists a writable, shared mapping on the file referred to by
              fd.
aanno

comment created time in 5 days

issue commentflatpak/flatpak

fcntl(F_ADD_SEALS): Device or resource busy

This is the failing part:

20325 memfd_create("ld-so-conf", MFD_CLOEXEC|MFD_ALLOW_SEALING) = 17</memfd:ld-so-conf (deleted)>
20325 ftruncate(17</memfd:ld-so-conf (deleted)>, 41) = 0
20325 write(17</memfd:ld-so-conf (deleted)>, "/usr/lib/x86_64-linux-gnu/GL/def"..., 41) = 41
20325 lseek(17</memfd:ld-so-conf (deleted)>, 0, SEEK_SET) = 0
20325 fcntl(17</memfd:ld-so-conf (deleted)>, F_ADD_SEALS, F_SEAL_SEAL|F_SEAL_SHRINK|F_SEAL_GROW|F_SEAL_WRITE) = -1 EBUSY (Device or resource busy)

I have no idea why this would ever fail. It is purely an in-memory operation, no "device" or "resources" are involved.

aanno

comment created time in 5 days

issue commentflatpak/flatpak

GTK3 apps fonts unhinted on wayland

@eternal-sorrow When you compiled them, does gsettings get org.gnome.settings-daemon.plugins.xsettings hinting on the host print anything?

eternal-sorrow

comment created time in 5 days

issue commentflatpak/flatpak

"error: Invalid group" when primary group is "10001"

Can you paste the output of "flatpak run -v -v .,..", that might help show where the warning comes from.

Djiock

comment created time in 5 days

issue closedflatpak/flatpak

Flatpak Repair Does Not Verify The Existence of Directories

Distribution: Fedora Silverblue 31

Version: flatpak-1.4.3-3.fc31.x86_64

I had a lot of empty folders in my home directory so I ran a find command to find and delete all the empty directories that I had created.

The find command deleted a lot of empty directories that were actually in use by flatpak and its runtimes and applications. Afterwards, all flatpak apps stopped functioning.

I ran flatpak repair to no avail. It found no fault in the apps and runtimes and it subsequently did nothing. I had to manually run 'flatpak --user install --reinstall' on all the installed apps and runtimes to make them all work again. It would have saved me a lot of trouble if flatpak repair simply recreated the directories.

closed time in 5 days

Saroufim

issue commentflatpak/flatpak

Flatpak Repair Does Not Verify The Existence of Directories

I think flatpak repair --reinstall-all would have done what you wanted. That would redeploy the apps and create new files and directories, should any deployed file have changed.

In your specific situation it would have been possible to do this in a more efficient way by just re-deploying existing refs as (i assume) you hadn't lost any files in the ostree repo that needed to be re-downloaded. However, such an operation would have failed in the normal case where some object file was also damaged, so I don't think exposing that option is a great idea. Its not like flatpak repair is a commonly used operation.

Saroufim

comment created time in 5 days

pull request commentflatpak/flatpak

Add --without-systemd arg to make systemd optional

Thanks!

DanySpin97

comment created time in 5 days

push eventflatpak/flatpak

Danilo Spinella

commit sha f3b863a187243aa48745f91a79c5e289753ec468

Add --with-systemd arg to make systemd optional

view details

push time in 5 days

PR closed flatpak/flatpak

Download debug info on demand through portal

This is a proposal for an incremental on demand download of debuginfo.

Debug info files are big and it has been source of multiple complains for the Freedesktop SDK. There is no reason a user should download all debug info files. It is rarely used, and when it is, it only few files.

Here I show how to allow the run-time to download debug files from within the runtime through the Flatpak portal. With this solution the debuginfo files are stored for future use by flatpak. This is good as developer usually need the same debug files across different flatpak run.

The corresponding merge request for Freedesktop SDK is available there: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/merge_requests/1451

+418 -101

11 comments

15 changed files

valentindavid

pr closed time in 5 days

pull request commentflatpak/flatpak

Download debug info on demand through portal

I think the debugd integration approach is better than this, so i'm gonna close this.

valentindavid

comment created time in 5 days

push eventflatpak/flatpak

Alexander Larsson

commit sha 2d2dd37741963e2087be29c9a91893d2e05bfb12

flatpak-dir: Fix doc-comment for flatpak_deploy_data_get_subpaths

view details

Alexander Larsson

commit sha ebca05ff10927a885afd097f27dd00e15001933d

utils: Add flatpak_bytes_save()

view details

Alexander Larsson

commit sha 966c6e2a255a821a03056c11df51ebc74d0c2292

CI: Add python3-pyparsing deps

view details

Alexander Larsson

commit sha 4046741e5c649b7f98d97332eab9a3505b3097d2

Add (and dist) variant-schema-compiler to sources

view details

Alexander Larsson

commit sha 4f2c4a5b1c0da4cbd9805fe0647f44546d54c60a

Add schema for some ostree/flatpak variant type and generate header

view details

Alexander Larsson

commit sha 00283943f24985a72f6859874f712ef57131229f

flatpak_remote_state_lookup_cache: Implement using variant schemas

view details

Alexander Larsson

commit sha 93d44413e6f47d9ab0f078f03a54f786769eac19

flatpak_dir_list_all_remote_refs: Implement using variant schemas

view details

Alexander Larsson

commit sha 8fe634d047fd1765831fb8ab985c79bc7cbf09fe

Remove unused flatpak_remote_state_lookup_repo_metadata()

view details

Alexander Larsson

commit sha 0f028e5329818e1660fa7efdaf42be4c35e7a704

flatpak_remote_state_lookup_sparse_cache: Use variant schema

view details

Alexander Larsson

commit sha 7c4fd8891e9189f7f43456f9fa72b415d52f1e5c

Convert deploy data to use variant schemas

view details

Alexander Larsson

commit sha 9f6c60405d91a6edc9f529db7a37d2766e07279e

utils: Convert summary ref lookup code to variant codegen We can't use the built-in bsearch from the codegen because its an array instead of a dict, so we have to keep that but its now not using variant at least.

view details

Alexander Larsson

commit sha 61da44a5e35f39ca98d2cce4ecc57b96c64b643d

Convert flatpak_summary_lookup_ref from GVariants Now it returns a VarRefInfoRef instead of a GVariant

view details

Alexander Larsson

commit sha f29830b4a408908e6486a192265bf1dae73cb60c

Convert sparse cache API to generated variant APIs

view details

Alexander Larsson

commit sha 567bddf25ccde106a5c84d9ca803d73277154a78

Use generated variant accessors for commit objects

view details

push time in 6 days

PR merged flatpak/flatpak

Use variant-schema-compiler for some GVariant code

This replaces some use of g_variant_get_* with generated code based on a IDL description of the types.

The generated code is a lot more efficient than the individual g_variant calls, as well as easier to read due to the use of types and field names instead of magic offsets. On the other hand, this is not necessarily code that needs to be very efficient, and there is a risk in any change like this.

Is this something we want to use, and over time extend to more code?

+573 -481

7 comments

26 changed files

alexlarsson

pr closed time in 6 days

push eventflatpak/flatpak

Piotr Drąg

commit sha a97f3b74c21c75a7de5469753f42358914c0dd9a

Update Polish translation

view details

push time in 6 days

push eventflatpak/flatpak

Piotr Drąg

commit sha 9610d2ef01b7388aef0a704ed26358a91166f6da

Update Polish translation

view details

push time in 6 days

push eventalexlarsson/flatpak

AsciiWolf

commit sha fb97782d26be50dd64a90782004466429741b865

Update Czech translation

view details

Simon McVittie

commit sha b2adbe2a74e10859d5f327aa3b7364becc0c2f85

exports: Only choose bwrap --bind/--ro-bind for host FS once We can choose this once and use it repeatedly, which will be simpler when we add more directories that work this way. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 66aee5a34266acb9d6cb38c5a83a0612d10e9b6f

icon-validator: Add lib32 to usrmerged_dirs, for completeness This syncs it up with our other lists of /usr-merged directories. In particular, this could matter on Arch Linux, which uses /usr/lib and /usr/lib32 for 64- and 32-bit libraries (respectively), instead of the more common /usr/lib64 and /usr/lib. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha b34ccef1c0cc708e729ec966350e3bc018f1f709

common: Unify some lists of /usr-merged directories In some places we want a list of basenames, and in others we want a list of absolute paths. Use the absolute paths, because converting those into basenames doesn't require memory allocation. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 08d65c5414d756c957549e14dcc599a9ed5a2a91

exports: If --filesystem=host, provide /run/host/lib etc. In a host system where the /usr merge has not been implemented, these can be necessary to load or inspect libraries or executables from the host system. They are conceptually the same as /usr. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 949a3ec479d5ca0c962cf12adec70aea30bf0186

context: Generalize handling of special filesystems a bit Currently there are only "home" and "host", but I'm going to add one that represents /usr and friends (/usr, /lib, ...), and one for /etc. These differ from ordinary filesystem mounts because they are redirected into /run/host to avoid conflicting with the runtime. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha fe2536b8441858e3b22f6780dca64a516ee4e48c

exports: Add host-etc and host-os keywords These are subsets of the host keyword, which provide access to operating system files but not to users' personal files. In particular, the experimental support for namespace-based sandboxes in the Steam Runtime[1] uses the graphics stack from the host system, which requires access to the host /usr/libQUAL, /libQUAL (even if the host OS has undergone the /usr merge, the canonical paths of ELF interpreters start with /lib), /etc/ld.so.cache, and for some libraries on Debian-based systems, /etc/alternatives. It will not be possible to do similar things in Flatpak without either allowing full host filesystem access (which exposes personal files, and in any case cannot be done by the Steam app because it is incompatible with --persist=.), or adding the ability to expose /usr and related directories without including the rest of the host filesystem. To the best of my knowledge, host-etc is not necessary for anything; I've mainly provided it for symmetry, since it's the other significant thing that we mount in /run/host and cannot get via --filesystem=/path. Some notes on the security/privacy implications of the new keywords: - Neither new keyword allows anything that was not already allowed by "host". - Neither new keyword can allow anything that was not already allowed to the user outside the sandbox. - "host-os" allows enumeration of the installed packages on the host system, and often their version numbers too. A malicious app could use this to look for exploitable security vulnerabilities on the host system. An app could also use this for fingerprinting, although this is not a regression, because the systemd/D-Bus machine ID, MAC addresses, hostname, kernel boot UUID, DMI product ID and many other unique or relatively unique properties are already available inside the sandbox. - "host-os" allows read access, and possibly write access (if the user has it outside the sandbox, for example members of group 'staff' in older Debian installations), to /usr/local. - "host-etc" allows reading configuration files whose contents might be considered sensitive, such as /etc/passwd. [1] https://steamcommunity.com/app/221410/discussions/0/1638675549018366706/ Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Alexander Larsson

commit sha 8cb73d10fb0ee76de6dd3b9614948ba6e5497642

flatpak-dir: Fix doc-comment for flatpak_deploy_data_get_subpaths

view details

Alexander Larsson

commit sha e154da0ce0271b180fda2cf71d042c6038137852

utils: Add flatpak_bytes_save()

view details

Alexander Larsson

commit sha 6ce6bbd49ba961762335779c8669d8e3416de312

CI: Add python3-pyparsing deps

view details

Alexander Larsson

commit sha 2e309e520735ed2d103bc126132b4422eec78971

Add (and dist) variant-schema-compiler to sources

view details

Alexander Larsson

commit sha 793227e6e129ee3b31f315b030fc7395250bfc86

Add schema for some ostree/flatpak variant type and generate header

view details

Alexander Larsson

commit sha 7ed2143f77fe3db94572243dd8eef2a37c5dd12b

flatpak_remote_state_lookup_cache: Implement using variant schemas

view details

Alexander Larsson

commit sha b89a35a5cd2d8b7fa43cbfb4d418dc519decd150

flatpak_dir_list_all_remote_refs: Implement using variant schemas

view details

Alexander Larsson

commit sha 165886dac3aa252afc11d8870080aee7da12f26d

Remove unused flatpak_remote_state_lookup_repo_metadata()

view details

Alexander Larsson

commit sha db196afb35d93afef195955a5c0fb1108c312f6a

flatpak_remote_state_lookup_sparse_cache: Use variant schema

view details

Alexander Larsson

commit sha 8524de46cc88eace2f6ed43c955cde7b3395ecdb

Convert deploy data to use variant schemas

view details

Alexander Larsson

commit sha 1e8de484fd375c438ff080f5f9167e1b3755b04a

utils: Convert summary ref lookup code to variant codegen We can't use the built-in bsearch from the codegen because its an array instead of a dict, so we have to keep that but its now not using variant at least.

view details

Alexander Larsson

commit sha e16b9bbbcc310c94fa401a19b862238737e37a4b

Convert flatpak_summary_lookup_ref from GVariants Now it returns a VarRefInfoRef instead of a GVariant

view details

Alexander Larsson

commit sha 07a8cc3be99cda7b58b266b1622640d5aba13c23

Convert sparse cache API to generated variant APIs

view details

push time in 6 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 46c28c752b23379796e5bf72ba407a3a839641ed

Add schema for some ostree/flatpak variant type and generate header

view details

Alexander Larsson

commit sha 697df439b42a36ae0b41afeed656a7723c16d974

flatpak_remote_state_lookup_cache: Implement using variant schemas

view details

Alexander Larsson

commit sha 47744b922a551cd859c7a1ac4d8352e01f0f4784

flatpak_dir_list_all_remote_refs: Implement using variant schemas

view details

Alexander Larsson

commit sha 6c3d6eaeabfc2a071194eae69ffd04e74da5aa6b

Remove unused flatpak_remote_state_lookup_repo_metadata()

view details

Alexander Larsson

commit sha 5996e205b8f1070ab321557db26346ab8d668051

flatpak_remote_state_lookup_sparse_cache: Use variant schema

view details

Alexander Larsson

commit sha 1daab52280c59e7a168554b111abe2bf1e402e12

Convert deploy data to use variant schemas

view details

Alexander Larsson

commit sha 8d2a64697f17411fe384b4f5752e7b8e40136a5b

utils: Convert summary ref lookup code to variant codegen We can't use the built-in bsearch from the codegen because its an array instead of a dict, so we have to keep that but its now not using variant at least.

view details

Alexander Larsson

commit sha 8d028abb4e4417d6d0b045d67a38780e4efd03cc

Convert flatpak_summary_lookup_ref from GVariants Now it returns a VarRefInfoRef instead of a GVariant

view details

Alexander Larsson

commit sha b8311dff13847387231c3d1af6f1100f52d2db92

Convert sparse cache API to generated variant APIs

view details

Alexander Larsson

commit sha 7c44d85cfbda68b3f76c60ed1af32c66b8b249e0

Use generated variant accessors for commit objects

view details

push time in 6 days

PR opened containers/bubblewrap

Ensure we're always clearing the cap bounding set

In the non-setuid case if we're not running as uid 0 in the final namespace but we need devpts (e.g. use --dev) we mount the devpts as uid and then change to the actual numberical uid at the end. This final unshare(CLONE_NEWPID) will reset tha cap bounding set we previously cleared.

This change clears the cap bounding set again after the unshare call.

This is not really a security problem because we always set NO_NEW_PRIVS which is essentially a superset of capability bounds, so there is no way the container can use the bounding set to gain caps. However its nice to be consistent and not display setting which look like potential problems.

Fixes https://github.com/containers/bubblewrap/issues/350

See 6b3dd4f10c23f23a2f3c3ec0f0d27ffc1149194c for the original change the drops the cap bounding set in the first location.

+3 -0

0 comment

1 changed file

pr created time in 6 days

create barnchcontainers/bubblewrap

branch : drop-cap-bounding-set-2

created branch time in 6 days

issue commentcontainers/bubblewrap

amicontained shows that --dev re-adds all capabilities

This seems to be caused by the opt_needs_devpts workaround. I.e. we create a two layered user namespace, the first on mapping the host uid to 0 so we can mount devpts, and the second to map back to the original numerical id. I tested this by setting opt_needs_devpts to TRUE without using --dev.

This is due to the second unshare (CLONE_NEWNS) where we remap the uids again, which clears the capability bounding set. I belive we should clear it again here to be consistent, but as @smcv said its not really a problem as no_new_privs is a superset of capability bounding

Maryse47

comment created time in 6 days

push eventflatpak/flatpak

AsciiWolf

commit sha eb6602f2c1ab4ecf303f6cea3f37b359ef331eb2

Update Czech translation

view details

push time in 7 days

PR merged flatpak/flatpak

Update Czech translation [flatpak-1.6.x]

Also update the translation in a stable branch. :-)

/cc @alexlarsson

+3 -3

1 comment

1 changed file

AsciiWolf

pr closed time in 7 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha c2440e0d51c25e66e48232f2b557f5fade945359

utils: Add flatpak_bytes_save()

view details

Alexander Larsson

commit sha 36ea319869cf3f8b5c053d114fd541db9b517f58

CI: Add python3-pyparsing deps

view details

Alexander Larsson

commit sha 9525dac716063b4313d76ee97bcedc07d228588d

Add (and dist) variant-schema-compiler to sources

view details

Alexander Larsson

commit sha eb7b8f15586f43bc78c9a3f68f187d8976d815d9

Add schema for some ostree/flatpak variant type and generate header

view details

Alexander Larsson

commit sha 2f36aeb9fb732c09c80daf38b68a0572044df452

flatpak_remote_state_lookup_cache: Implement using variant schemas

view details

Alexander Larsson

commit sha 36a9258d4ce1c4d213264675e126ada6ab7f3446

flatpak_dir_list_all_remote_refs: Implement using variant schemas

view details

Alexander Larsson

commit sha c76b18e61466ce2a3ac26a68c07f2e8cf209825a

Remove unused flatpak_remote_state_lookup_repo_metadata()

view details

Alexander Larsson

commit sha 2f40c34611051459f1f21e431a3616417c62f6bd

flatpak_remote_state_lookup_sparse_cache: Use variant schema

view details

Alexander Larsson

commit sha 9e85e5111852d1676e56d3ffa4feadd1e96523c7

Convert deploy data to use variant schemas

view details

push time in 9 days

pull request commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

Rebased to master and fixes up some of the comments. I still want to do some more work before we land this. Also, i'm considering moving the generator to gnome git.

alexlarsson

comment created time in 9 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 5fac91c074cc2c6fc8b2c09caf71f0a2aac6aaa8

Convert deploy data to use variant schemas

view details

push time in 9 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 4804d7340a0341e953e094bb6879f1a3f27a8b73

flatpak-dir: Fix doc-comment for flatpak_deploy_data_get_subpaths

view details

Alexander Larsson

commit sha 6dd5506e06dcda930d80baef2b86a70a41f91b53

utils: Add flatpak_bytes_save()

view details

Alexander Larsson

commit sha 126663a7b168aeecc7210fe972d808c3b709b4d4

CI: Add python3-pyparsing deps

view details

Alexander Larsson

commit sha 1a3ccab7333b582f61733b807b8b40b454895978

Add (and dist) variant-schema-compiler to sources

view details

Alexander Larsson

commit sha fca4b4005b630a9b168dc0de2ae9a4c1d28de6cb

Add schema for some ostree/flatpak variant type and generate header

view details

Alexander Larsson

commit sha 5e8ae4a47f0fbc80af0b9471fa4a2f4e5432d1f9

flatpak_remote_state_lookup_cache: Implement using variant schemas

view details

Alexander Larsson

commit sha 6bbbf22bb082f8207aaf92733117b5feb66a81e5

flatpak_dir_list_all_remote_refs: Implement using variant schemas

view details

Alexander Larsson

commit sha eab0a43cea0bc66018434d574ec6a32ba86d94ca

Remove unused flatpak_remote_state_lookup_repo_metadata()

view details

Alexander Larsson

commit sha ea1159275781e8f5d7f8722d4a2b849a99bdbd82

flatpak_remote_state_lookup_sparse_cache: Use variant schema

view details

Alexander Larsson

commit sha f58f7ed93eb52db10a96875e5a950db52d13b1bb

Convert deploy data to use variant schemas

view details

push time in 9 days

Pull request review commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

 flatpak_dir_list_all_remote_refs (FlatpakDir         *self,    exts = var_summary_get_metadata (summary); -  collection_id = NULL;-  if (var_metadata_lookup (exts, "ostree.summary.collection-id", NULL, &v) &&-      var_variant_is_type (v, G_VARIANT_TYPE_STRING))-    collection_id = var_variant_get_string (v);-+  collection_id = var_metadata_lookup_string (exts, "ostree.summary.collection-id", NULL);

Yeah, it just snuck in due to new generated APIs being available

alexlarsson

comment created time in 9 days

Pull request review commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

 nodist_libflatpak_common_base_la_SOURCES = \ BUILT_SOURCES += $(nodist_libflatpak_common_base_la_SOURCES) CLEANFILES += $(nodist_libflatpak_common_base_la_SOURCES) +common/flatpak-variant-private.h: variant-schema-compiler/variant-schema-compiler  data/flatpak-variants.gv+	$(AM_V_GEN) variant-schema-compiler/variant-schema-compiler --outfile common/flatpak-variant-private.h --prefix var data/flatpak-variants.gv

The generated types are bit weird, not being typical GObjects and stuff, and some may conflict with other more regular types, so I picked a different prefix.

alexlarsson

comment created time in 9 days

push eventalexlarsson/flatpak

Umang Jain

commit sha 18626add023774ed482a9c3459f3e23dd00b2223

dir: Return NULL instead of boolean when querying related refs The related refs are returned as GPtrArray, hence return NULL instead of FALSE on error paths.

view details

Umang Jain

commit sha 56787325edd802c1b5d01e5e5c0c46d86d234f62

dir: Return empty array instead of NULL while querying related-refs Initialize the related-refs array with empty GPtrArray so that if the remote has 'url= ' (for e.g., in case of flatpak bundle's remotes), a empty array is returned instead of NULL. (NULL mostly implies a operation has failed and error is set) Also, this syncs the implementation of `if (*url == 0)` with that of flatak_dir_find_remote_related_for_metadata function.

view details

Matthew Leeds

commit sha c8df96a79eec29605f44fe659e2a6e183210ba2b

Merge pull request #3363 from uajain/uajain/fix-gs-crash Fix a gnome-software crash

view details

Alexander Larsson

commit sha 39903eab40920dab749a795ce2bef8ce54aacc1a

Add --device=shm permission This new permission exposes the host /dev, which is normally not visible even with --device=all, as it is not really a device node but rather a bunch of shared memory blocks available on the host. This access is needed by jack, as explained at: https://github.com/flatpak/flatpak/issues/1509 Long term I think a better solution for pro audio (like pipewire) is a better solution, but for now we should at least allow jack apps to work.

view details

Alexander Larsson

commit sha b735344644d25e1586dd9d8b4d832d80c588d80e

Correct filename in docs

view details

Alexander Larsson

commit sha 3c6c51f46b753934cff9649c0937ead9c16524d3

build-commit-from: Fix generation of download-size In flatpak-builtins-build-commit-from.c we call flatpak_repo_collect_sizes() without initializing the passed in download size to zero, which mean we sum with sizes with some random value as the start. This is fixed by having flatpak_repo_collect_sizes() always initialize the counters to 0 at the start. Fixes https://github.com/flatpak/flatpak/issues/3362

view details

Alexander Larsson

commit sha b3bd501978df5d9923caae0ae47957862e652972

update-portal: Limit which filesystem access additions we allow Don't allow adding access to things like ~/foo xdg-foo/bar or similar things just because you used to have home access, because such files may be outside the homedir (for instance, if they are symlinks or configured via xdg-user-dirs).

view details

Alexander Larsson

commit sha 50fc19daf12b9fd5088c137c8c7d8d61a2e3c124

Bump version to 1.6.1

view details

Alexander Larsson

commit sha 68d814bd01490d0c1af7230e9b0092a4175ae426

Update NEWS for release

view details

Alexander Larsson

commit sha 400a3358ec6e53e22957226f9d1cf22052fe7f4d

Update pofiles for release

view details

Simon McVittie

commit sha 1f9dc50e338ece356cbd10a4f98c46daf8577f85

exports: Fix a confusingly-named method It was called flatpak_exports_add_home_expose(), but it actually exposed the entire host filesystem, to the extent possible. Rename it to flatpak_exports_add_host_expose() to reflect that. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 851a34b355d0cda45ce3347e4abf4185da70e086

doc: Point to flatpak-metadata(5) for the meanings of filesystem keywords Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Umang Jain

commit sha fe8b3c4b33b26c57b1d49c7a27079a5b4e3532b0

nitpick: installation: Remove a blank line

view details

Kukuh Syafaat

commit sha e04795f440394918bac57a900918ac495118fdbc

Update Indonesian translation

view details

Matthew Leeds

commit sha 6aa1617e63194373a3aa403bcb84a9e9fb36ba36

Merge pull request #3376 from uajain/uajain/del-blank-line nitpick: installation: Remove a blank line

view details

Matthew Leeds

commit sha 35d8c6afb5eebae877bff2f6bc8389f6b6cc08fd

Merge pull request #3375 from smcv/doc-filesystems doc: Point to flatpak-metadata(5) for the meanings of filesystem keywords

view details

milotype

commit sha 8cd363a94aaa63d451aa2689b0083aa2c9ec3b65

Update LINGUAS with "hr" (croatian)

view details

Damian Wrobel

commit sha e801959e58196988b69723956e5a69ac20352e36

Fix out-of-tree build error Fixes the missing 'app' directory: Traceback (most recent call last): File "/data/dwrobel1/rdkv/rpi-flutter-3/build-raspberrypirdkhybrefapp/tmp/sysroots/x86_64-linux/usr/bin/gdbus-codegen", line 39, in <module> sys.exit(codegen_main.codegen_main()) File "/data/dwrobel1/rdkv/rpi-flutter-3/build-raspberrypirdkhybrefapp/tmp/sysroots/x86_64-linux/usr/share/glib-2.0/codegen/codegen_main.py", line 186, in codegen_main h = open(c_code + '.h', 'w') FileNotFoundError: [Errno 2] No such file or directory: './app/flatpak-permission-dbus-generated.h' make: *** [app/flatpak-permission-dbus-generated.c] Error 1 make: *** Waiting for unfinished jobs....

view details

Matthew Leeds

commit sha 00ed995860c187e17ea4075ff7006a0e4187e9ec

Merge pull request #3393 from dwrobel/patch-1 Fix out-of-tree build error

view details

Matthew Leeds

commit sha 423a21271c8677d285a41ee47ecabb5a2433374e

Merge pull request #3374 from smcv/fix-home-host-confusion exports: Fix a confusingly-named method

view details

push time in 9 days

Pull request review commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

 flatpak_dir_list_all_remote_refs (FlatpakDir         *self,                                   GError            **error) {   g_autoptr(GHashTable) ret_all_refs = NULL;-  g_autoptr(GVariant) ref_map = NULL;-  g_autoptr(GVariant) exts = NULL;-  g_autoptr(GVariant) collection_map = NULL;   const gchar *collection_id;-  GVariantIter iter;+  VarSummaryRef summary;+  VarMetadataRef exts;

It doesn't really have a name, its just a dict that lets you shove random extra stuff in the summary. I guess i meant extensions? I'm not sure that is a great name either though.

alexlarsson

comment created time in 9 days

Pull request review commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

 flatpak_dir_remote_has_ref (FlatpakDir *self, }  static void-populate_hash_table_from_refs_map (GHashTable *ret_all_refs, GVariant *ref_map,+populate_hash_table_from_refs_map (GHashTable *ret_all_refs, VarRefMapRef ref_map,                                    const gchar *collection_id,                                    FlatpakRemoteState *state) {-  GVariant *value;-  GVariantIter ref_iter;+  gsize len, i; -  g_variant_iter_init (&ref_iter, ref_map);-  while ((value = g_variant_iter_next_value (&ref_iter)) != NULL)+  len = var_ref_map_get_length (ref_map);+  for (i = 0; i < len; i++)     {-      /* helper for being able to auto-free the value */-      g_autoptr(GVariant) child = value;-      const char *ref_name = NULL;--      g_variant_get_child (child, 0, "&s", &ref_name);-      if (ref_name == NULL)

String elements can't be NULL in a variant. g_variant_get_child would only ever return NULL here if child 0 was not a string, but at this point we already asserted the type.

alexlarsson

comment created time in 9 days

Pull request review commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

 flatpak_dir_list_all_remote_refs (FlatpakDir         *self,    * list in that it lacks checksums). */   if (state->collection_id != NULL && state->summary == NULL)     {-      g_autoptr(GVariant) xa_cache = NULL;-      g_autoptr(GVariant) cache = NULL;+      VarCacheRef cache;       gsize i, n; -      if (!flatpak_remote_state_ensure_metadata (state, error))+      if (!flatpak_remote_state_get_cache (state, &cache, error))         return FALSE; -      if (!flatpak_remote_state_lookup_repo_metadata (state, "xa.cache", "@*", &xa_cache))-        return flatpak_fail_error (error, FLATPAK_ERROR_INVALID_DATA, _("No summary or Flatpak cache available for remote %s"),-                                   state->remote_name);--      cache = g_variant_get_child_value (xa_cache, 0);-      n = g_variant_n_children (cache);+      n = var_cache_get_length (cache);       for (i = 0; i < n; i++)         {-          g_autoptr(GVariant) child = NULL;-          g_autoptr(GVariant) cur_v = NULL;           g_autoptr(FlatpakCollectionRef) coll_ref = NULL;+          VarCacheEntryRef entry;

An Entry is the dict entry, i.e. the thing with both key and value in it. The CacheData is the value only.

alexlarsson

comment created time in 9 days

Pull request review commentflatpak/flatpak

Use variant-schema-compiler for some GVariant code

 libflatpak_common_la_SOURCES = \ 	common/flatpak-utils-private.h \ 	common/flatpak-utils.c \ 	common/valgrind-private.h \+	common/flatpak-variant-private.h \

Yes, i'd like to ship this in the tarball so people building flatpak don't have to have pyparsing installed.

alexlarsson

comment created time in 9 days

pull request commentflatpak/flatpak

Support --device=snd, for audio on ALSA-only systems.

So perhaps there should be a higher-level permission features=audio which is the equivalent of sockets=pulseaudio, and also gives access to /dev/snd and /dev/dsp?

I like that idea except it could detect which is needed: If using pipewire allow only that, if using pulseaudio only allow that, if using alsa finally allow device access.

I like this idea in theory, but how would you know what kind of sound access the app is going to need? Even if pipewire is running, the app may not work with it. I think this is a problem in general with the general "allow audio" support, unless it means "grant access to everything possible sound related". This is why historically the static permissions in flatpak has been sort of lowlevel, even if that does cause issues with presenting them in a useful way to the end user.

Historically the plan was to make pulseaudio have some kind of portal integration with access control, but that never really happened. I think the new goal is instead to support pipewire, as a portal. In that setup I think by default all apps should be allowed to produce sound, but do some permissions UI to allow recording (and never have monitor-like access).

Since pulseaudio in practice is no more safe than anything else, and adding a new features=audio that adds pulseaudio+alsa+oss that everyone would have to add seems like a lot of churn, maybe we should just mount in the alsa and oss device nodes if socket=pulseaudio is there, then apps don't need both and old apps may just work as is if the host doesn't have pulseaudio. Its sort of a hack, but so is everything that isn't the pipewire full end goal.

foresto

comment created time in 9 days

push eventflatpak/flatpak

Simon McVittie

commit sha b2adbe2a74e10859d5f327aa3b7364becc0c2f85

exports: Only choose bwrap --bind/--ro-bind for host FS once We can choose this once and use it repeatedly, which will be simpler when we add more directories that work this way. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 66aee5a34266acb9d6cb38c5a83a0612d10e9b6f

icon-validator: Add lib32 to usrmerged_dirs, for completeness This syncs it up with our other lists of /usr-merged directories. In particular, this could matter on Arch Linux, which uses /usr/lib and /usr/lib32 for 64- and 32-bit libraries (respectively), instead of the more common /usr/lib64 and /usr/lib. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha b34ccef1c0cc708e729ec966350e3bc018f1f709

common: Unify some lists of /usr-merged directories In some places we want a list of basenames, and in others we want a list of absolute paths. Use the absolute paths, because converting those into basenames doesn't require memory allocation. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 08d65c5414d756c957549e14dcc599a9ed5a2a91

exports: If --filesystem=host, provide /run/host/lib etc. In a host system where the /usr merge has not been implemented, these can be necessary to load or inspect libraries or executables from the host system. They are conceptually the same as /usr. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha 949a3ec479d5ca0c962cf12adec70aea30bf0186

context: Generalize handling of special filesystems a bit Currently there are only "home" and "host", but I'm going to add one that represents /usr and friends (/usr, /lib, ...), and one for /etc. These differ from ordinary filesystem mounts because they are redirected into /run/host to avoid conflicting with the runtime. Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

Simon McVittie

commit sha fe2536b8441858e3b22f6780dca64a516ee4e48c

exports: Add host-etc and host-os keywords These are subsets of the host keyword, which provide access to operating system files but not to users' personal files. In particular, the experimental support for namespace-based sandboxes in the Steam Runtime[1] uses the graphics stack from the host system, which requires access to the host /usr/libQUAL, /libQUAL (even if the host OS has undergone the /usr merge, the canonical paths of ELF interpreters start with /lib), /etc/ld.so.cache, and for some libraries on Debian-based systems, /etc/alternatives. It will not be possible to do similar things in Flatpak without either allowing full host filesystem access (which exposes personal files, and in any case cannot be done by the Steam app because it is incompatible with --persist=.), or adding the ability to expose /usr and related directories without including the rest of the host filesystem. To the best of my knowledge, host-etc is not necessary for anything; I've mainly provided it for symmetry, since it's the other significant thing that we mount in /run/host and cannot get via --filesystem=/path. Some notes on the security/privacy implications of the new keywords: - Neither new keyword allows anything that was not already allowed by "host". - Neither new keyword can allow anything that was not already allowed to the user outside the sandbox. - "host-os" allows enumeration of the installed packages on the host system, and often their version numbers too. A malicious app could use this to look for exploitable security vulnerabilities on the host system. An app could also use this for fingerprinting, although this is not a regression, because the systemd/D-Bus machine ID, MAC addresses, hostname, kernel boot UUID, DMI product ID and many other unique or relatively unique properties are already available inside the sandbox. - "host-os" allows read access, and possibly write access (if the user has it outside the sandbox, for example members of group 'staff' in older Debian installations), to /usr/local. - "host-etc" allows reading configuration files whose contents might be considered sensitive, such as /etc/passwd. [1] https://steamcommunity.com/app/221410/discussions/0/1638675549018366706/ Signed-off-by: Simon McVittie <smcv@collabora.com>

view details

push time in 9 days

PR merged flatpak/flatpak

Reviewers
Add host-etc, host-os exports
  • exports: Fix a confusingly-named method (#3374)

    It was called flatpak_exports_add_home_expose(), but it actually exposed the entire host filesystem, to the extent possible. Rename it to flatpak_exports_add_host_expose() to reflect that.

  • doc: Point to flatpak-metadata(5) for the meanings of filesystem keywords (#3375)

  • exports: Only choose bwrap --bind/--ro-bind for host FS once

    We can choose this once and use it repeatedly, which will be simpler when we add more directories that work this way.

  • icon-validator: Add lib32 to usrmerged_dirs, for completeness

    This syncs it up with our other lists of /usr-merged directories.

    In particular, this could matter on Arch Linux, which uses /usr/lib and /usr/lib32 for 64- and 32-bit libraries (respectively), instead of the more common /usr/lib64 and /usr/lib.

  • common: Unify some lists of /usr-merged directories

    In some places we want a list of basenames, and in others we want a list of absolute paths. Use the absolute paths, because converting those into basenames doesn't require memory allocation.

  • exports: If --filesystem=host, provide /run/host/lib etc.

    In a host system where the /usr merge has not been implemented, these can be necessary to load or inspect libraries or executables from the host system. They are conceptually the same as /usr.

  • context: Generalize handling of special filesystems a bit

    Currently there are only "home" and "host", but I'm going to add one that represents /usr and friends (/usr, /lib, ...), and one for /etc. These differ from ordinary filesystem mounts because they are redirected into /run/host to avoid conflicting with the runtime.

  • exports: Add host-etc and host-os keywords

    These are subsets of the host keyword, which provide access to operating system files but not to users' personal files.

    In particular, the experimental support for namespace-based sandboxes in the Steam Runtime[1] uses the graphics stack from the host system, which requires access to the host /usr/libQUAL, /libQUAL (even if the host OS has undergone the /usr merge, the canonical paths of ELF interpreters start with /lib), /etc/ld.so.cache, and for some libraries on Debian-based systems, /etc/alternatives. It will not be possible to do similar things in Flatpak without either allowing full host filesystem access (which exposes personal files, and in any case cannot be done by the Steam app because it is incompatible with --persist=.), or adding the ability to expose /usr and related directories without including the rest of the host filesystem.

    To the best of my knowledge, host-etc is not necessary for anything; I've mainly provided it for symmetry, since it's the other significant thing that we mount in /run/host and cannot get via --filesystem=/path.

    Some notes on the security/privacy implications of the new keywords:

    • Neither new keyword allows anything that was not already allowed by "host".
    • Neither new keyword can allow anything that was not already allowed to the user outside the sandbox.
    • "host-os" allows enumeration of the installed packages on the host system, and often their version numbers too. A malicious app could use this to look for exploitable security vulnerabilities on the host system. An app could also use this for fingerprinting, although this is not a regression, because the systemd/D-Bus machine ID, MAC addresses, hostname, kernel boot UUID, DMI product ID and many other unique or relatively unique properties are already available inside the sandbox.
    • "host-os" allows read access, and possibly write access (if the user has it outside the sandbox, for example members of group 'staff' in older Debian installations), to /usr/local.
    • "host-etc" allows reading configuration files whose contents might be considered sensitive, such as /etc/passwd.

    [1] https://steamcommunity.com/app/221410/discussions/0/1638675549018366706/


Tested on a host system with unmerged /usr (/lib, etc. are real directories), with symlinks like /lib -> /usr/lib, and with /lib -> usr/lib.

Includes #3374 and #3375.

+270 -37

1 comment

11 changed files

smcv

pr closed time in 9 days

pull request commentflatpak/flatpak

Add host-etc, host-os exports

This looks good.

smcv

comment created time in 9 days

Pull request review commentflatpak/flatpak

Add host-etc, host-os exports

                                 Available since 0.3.                             </para></listitem></varlistentry> +                            <varlistentry><term><option>host-os</option></term>+                            <listitem><para>+                                The host operating system's libraries,+                                executables and static data from+                                <filename>/usr</filename>+                                and the related directories+                                <filename>/bin</filename>,+                                <filename>/lib</filename>,+                                <filename>/lib32</filename>,+                                <filename>/lib64</filename>,+                                <filename>/sbin</filename>.+                                Additionally, this keyword provides access+                                to a subset of <filename>/etc</filename> that+                                is associated with packaged libraries and+                                executables, even if the+                                <option>host-etc</option> keyword+                                was not used:+                                <filename>/etc/ld.so.cache</filename>,+                                (used by the dynamic linker) and+                                <filename>/etc/alternatives</filename>+                                (on operating systems that use it, such as+                                Debian).+                            </para>+                            <para>+                                To avoid conflicting with the Flatpak+                                runtime, these are mounted in the sandbox+                                at <filename>/run/host/usr</filename>,+                                <filename>/run/host/etc/ld.so.cache</filename>+                                and so on.+                            </para>+                            <para>+                                Available since 1.7.

I branched 1.6 so this will be in 1.7

smcv

comment created time in 9 days

Pull request review commentflatpak/flatpak

Add host-etc, host-os exports

                                 Available since 0.3.                             </para></listitem></varlistentry> +                            <varlistentry><term><option>host-os</option></term>+                            <listitem><para>+                                The host operating system's libraries,+                                executables and static data from+                                <filename>/usr</filename>+                                and the related directories+                                <filename>/bin</filename>,+                                <filename>/lib</filename>,+                                <filename>/lib32</filename>,+                                <filename>/lib64</filename>,+                                <filename>/sbin</filename>.+                                Additionally, this keyword provides access+                                to a subset of <filename>/etc</filename> that+                                is associated with packaged libraries and+                                executables, even if the+                                <option>host-etc</option> keyword+                                was not used:+                                <filename>/etc/ld.so.cache</filename>,

I don't think those are needed as long as you have the compiled result (i.e. ld.so.cache).

smcv

comment created time in 9 days

Pull request review commentflatpak/flatpak

Add host-etc, host-os exports

                                 Available since 0.3.                             </para></listitem></varlistentry> +                            <varlistentry><term><option>host-os</option></term>+                            <listitem><para>+                                The host operating system's libraries,+                                executables and static data from+                                <filename>/usr</filename>

I think having /usr/local is fine. Its still the "os" rather than personal data.

smcv

comment created time in 9 days

Pull request review commentflatpak/flatpak

Add host-etc, host-os exports

 flatpak_exports_append_bwrap_args (FlatpakExports *exports,         }     } -  if (exports->host_fs != 0)+  if (exports->host_os != 0)     {+      const char *os_bind_mode = "--bind";+      int i;++      if (exports->host_os == FLATPAK_FILESYSTEM_MODE_READ_ONLY)+        os_bind_mode = "--ro-bind";+       if (g_file_test ("/usr", G_FILE_TEST_IS_DIR))         flatpak_bwrap_add_args (bwrap,-                                (exports->host_fs == FLATPAK_FILESYSTEM_MODE_READ_ONLY) ? "--ro-bind" : "--bind",-                                "/usr", "/run/host/usr", NULL);+                                os_bind_mode, "/usr", "/run/host/usr", NULL);++      for (i = 0; flatpak_abs_usrmerged_dirs[i] != NULL; i++)

Nah. I don't really care either.

smcv

comment created time in 9 days

push eventflatpak/flatpak

AsciiWolf

commit sha fb97782d26be50dd64a90782004466429741b865

Update Czech translation

view details

push time in 9 days

PR merged flatpak/flatpak

Update Czech translation

Just a small update, adding translation for the recently added string. :-)

+3 -3

1 comment

1 changed file

AsciiWolf

pr closed time in 9 days

push eventflatpak/flatpak

Alexander Larsson

commit sha 81665617d191b8375fa57804d2026dfdc09e75d2

Bump version on master to 1.7.1, new stable branch is flatpatk-1.6.x.

view details

push time in 9 days

create barnchflatpak/flatpak

branch : flatpak-1.6.x

created branch time in 9 days

Pull request review commentflatpak/flatpak

Actually use from-scratch deltas

 flatpak_dir_setup_extra_data (FlatpakDir                           *self,   /* If @results is set, @rev must be. */   g_assert (results == NULL || rev != NULL); -  extra_data_sources = flatpak_repo_get_extra_data_sources (repo, rev, cancellable, NULL);-  if (extra_data_sources == NULL)+  /* ostree-metadata and appstreams never have extra data, so ignore those */+  if (g_str_has_prefix ("app/", ref) || g_str_has_prefix ("runtime/", ref))     {-      /* Pull the commits (and only the commits) to check for extra data-       * again. Here we don't pass the progress because we don't want any-       * reports coming out of it. */-      if (!repo_pull (repo, repository,-                      NULL,-                      ref,-                      rev,-                      token,-                      results,-                      flatpak_flags,-                      OSTREE_REPO_PULL_FLAGS_COMMIT_ONLY,-                      NULL,-                      cancellable,-                      error))-        return FALSE;-       extra_data_sources = flatpak_repo_get_extra_data_sources (repo, rev, cancellable, NULL);+      if (extra_data_sources == NULL)+        {+          /* This is a gigantic hack where we download the commit in a temporary transaction+           * which we then abort after having read the result. We do this to avoid creating+           * a partial commit in the local repo and a ref that points to it, because that+           * causes ostree to not use static deltas.+           * See https://github.com/flatpak/flatpak/issues/3412 for details.+           */++          if (!ostree_repo_prepare_transaction (repo, NULL, cancellable, error))+            return FALSE;++          /* Pull the commits (and only the commits) to check for extra data+           * again. Here we don't pass the progress because we don't want any+           * reports coming out of it. */+          if (!repo_pull (repo, repository,+                          NULL,+                          ref,+                          rev,+                          token,+                          results,+                          flatpak_flags,+                          OSTREE_REPO_PULL_FLAGS_COMMIT_ONLY,+                          NULL,+                          cancellable,+                          error))+            return FALSE;++          extra_data_sources = flatpak_repo_get_extra_data_sources (repo, rev, cancellable, NULL);++          if (!ostree_repo_abort_transaction (repo, cancellable, error))+            return FALSE;

Does pulling a rev without a ref work with the p2p case? I was sorta assuming it didn't. Yet another reason why p2p should die in flames.

alexlarsson

comment created time in 9 days

pull request commentcontainers/bubblewrap

Mount new tmpfs filesystems with noexec flag

Not only bash, but all kinds of interpreters, including non-intended ones like buffer overruns which could be exploited to run whatever code you want (such as the memfd hack above).

Maryse47

comment created time in 9 days

pull request commentcontainers/bubblewrap

Mount new tmpfs filesystems with noexec flag

Also, i'm not sure noexec is all that great of a protection. There are so many indirect ways to create files. For example, attached here is a simple way to use memfds to execute some copied data without needing to write it to a filesystem. hack-memfd.c.txt

Maryse47

comment created time in 9 days

pull request commentcontainers/bubblewrap

Mount new tmpfs filesystems with noexec flag

This change as-is breaks the behaviour for existing users of bwrap. For example, it will no longer make files in /tmp or the fake $HOME executable in flatpak apps, which may cause problems for apps that maybe download or generate scripts. I realize this is a potential escalation in some special cases, but I'm not sure that means we should make it impossible for anyone to make executable tmpfs:es.

So, there are two issues here, one is that you might want to create a bwrap container that has a no-exec tmpfs. I think this should be an option independent on the host state, and its actually similar to https://github.com/containers/bubblewrap/issues/346 in that we want ways to set more permissions on the files/dirs/mounts bwrap creates.

The other issue is that you're setting up a system (be it a real host or some other form of container) where the user has access to the bwrap command, but doesn't have access to any writable executable location. In this case giving access to the bwrap commands would let you escape the noexec limitation. I think this should be handled by bwrap detecting this case and automatically setting noexec, in that particular case.

So, if all mounts in the current namespace are read-only or noexec, then we should make all new tmpfs:es automatically noexec. I guess this isn't 100% correct, as you might have a situation where you have access to a read-write executable mount but file permissions makes it impossible to write anything there. That seems like an unsafe thing to rely on though, and if you want to grant bwrap access it would be better to make such mounts read-only.

Maryse47

comment created time in 9 days

pull request commentostreedev/ostree

Fix handling of scratch deltas with existing commits

Actually I don't know if this is right either. The walk over the commits parent will only succeed if the direct parent of the new commit object is available locally. If you jump multiple revisions when you do the update then it will not find the intermediate parents and thus not the grandparent.

dbnicholson

comment created time in 9 days

release flatpak/flatpak

1.4.4

released time in 9 days

push eventflatpak/flatpak

Alexander Larsson

commit sha d2960c8d9437b4b58b3b515a65d2aaf2f0de972b

Bump version to 1.4.4

view details

Alexander Larsson

commit sha e68ea25b4072759e4277486b08ea3609f8339d07

Update NEWS for release

view details

Alexander Larsson

commit sha 16a03665aac921ada99a82a0a653116b24f94a00

Update pofiles for release

view details

push time in 9 days

created tagflatpak/flatpak

tag1.4.4

Linux application sandboxing and distribution framework

created time in 9 days

PR merged flatpak/flatpak

Backport scratch-delta support

This is a backport of fix to use static-deltas to the 1.4 branch. Additionally it also has some fixes to make the tests pass on python3-by-default distros and a backport of a p2p mirror-ref fix that is needed for the static-delta fix.

+62 -99

0 comment

6 changed files

alexlarsson

pr closed time in 9 days

push eventflatpak/flatpak

Alexander Larsson

commit sha 8490b19705d203e5cb99551ffe3bd3c2f2cef65c

Hardcode "python2" for tests On master we're python3 only, but there is no need to change this for the 1.4 branch. However, some recent systems have the /usr/bin/python being python3, which breaks the tests. So, we hardcode python2 to make it work on these systems.

view details

Matthew Leeds

commit sha 48eac4c26a0cce03bec2fdca52292d70025123bc

Revert "dir: Check commit signatures before resolving a ref" This reverts commit 915ad583a7bf70e03bf58bf14b9d3bdb7ef33277. This commit turned out to have unintended side effects. Specifically, with it we do a pull with OSTREE_REPO_PULL_FLAGS_MIRROR, and then flatpak_dir_setup_extra_data() does a non-mirror pull in the same transaction, so the ref being pulled ends up being written to disk under both refs/remotes/ and refs/mirrors/ in ostree_repo_commit_transaction(). This is a problem because only the remote ref is deleted during an uninstall, so the disk space is leaked, and we don't have the infrastructure in place to keep both refs up to date as they're updated. It would be nice to consistently use OSTREE_REPO_PULL_FLAGS_MIRROR for all pulls but that turns out to be a deep rabbit hole to go down; see the discussion in https://github.com/flatpak/flatpak/pull/3220 So revert the commit instead (with a few exceptions: keep a still-relevant FIXME comment, keep an assertion in the "out:" section, and keep a debug statement printing out the resolved rev). Note that this means that since we're no longer checking commit signatures during ref resolution, in theory remote B could try to set the same collection ID as remote A and serve a malicious update for something from remote A, but the signature would be found to be invalid during the pull phase due to our use of "ref-keyring-map" so the transaction would fail. All the other uses of OSTREE_REPO_PULL_FLAGS_MIRROR across the codebase should be kept I think: - flatpak create-usb uses it when pulling into the repo on the USB which works perfectly well with refs/mirrors/ (and the USB is mirroring the collection-refs!) - it's used when pulling into a temporary "child" repo in a few places and there it makes sense since the child repo is mirroring the refs so they can be pulled into the main repo. In fact, in the case of flatpak_dir_do_resolve_p2p_refs(), we need MIRROR since otherwise ostree_repo_resolve_collection_ref() gives us the commit on-disk rather than the just-pulled one that's in memory. (cherry picked from commit 13366524d82af14ad35fa4618ccf202ece1adacc)

view details

Alexander Larsson

commit sha 3e40bb00ca9a6446802ff147c0011fdcc4061de1

p2p: Don't mirror ostree-metadata refs when pulling into the child repo This breaks Deploy which can't find the ref. It used to work due to the extra non-mirroring pull in flatpak_dir_setup_extra_data, but this is not needed here. (cherry picked from commit 9aecad7f4fb32e0dffc38eb18e78629f931787e3)

view details

Alexander Larsson

commit sha 557e809cac2a3b9411f2789e7ad0b3d968343d55

Actually use from-scratch deltas As noticed in https://github.com/flatpak/flatpak/issues/3412 we regressed at some point and are no longer using from-scratch deltas. This is caused by an optimization in ostree where it decides to not use a from-scratch deltas if theres is *some* version of the ref locally available. This conflicts with some code in flatpak that pulls *only* the commit object in order to look for extra data size information so that we can get the progress reporting right. Unfortunately the existance of just the object triggers the above causing us to *never* use from-scratch deltas. We fix this by throwing away the partial pull in an aborted ostree transaction. (cherry picked from commit b371ef9007e6136dc956ac2507d30ce020827247)

view details

push time in 9 days

issue commentflatpak/flatpak

fcntl(F_ADD_SEALS): Device or resource busy

This is really weird. I've never seen it (and i'm on f31 with nvidia drivers). Can you run something like strace -o signal.log -f -y flatpak -v run org.signal.Signal and attach the signal.log file here?

aanno

comment created time in 9 days

PR opened flatpak/flatpak

Backport scratch-delta support

This is a backport of fix to use static-deltas to the 1.4 branch. Additionally it also has some fixes to make the tests pass on python3-by-default distros and a backport of a p2p mirror-ref fix that is needed for the static-delta fix.

+62 -99

0 comment

6 changed files

pr created time in 9 days

push eventalexlarsson/flatpak

Piotr Drąg

commit sha 8026160dc5a72a3317f0fccfe50e2188153166a7

Update Polish translation

view details

Piotr Drąg

commit sha 5584b46fcc76dc1010d35b47eead0351cc550322

Update Polish translation Closes: #2926 Approved by: matthiasclasen

view details

AsciiWolf

commit sha f4818db612e25b27be1aff38af5ccb652dcdb358

Update Czech translation Closes: #2940 Approved by: mwleeds

view details

Matthew Leeds

commit sha f44c70e7dcd61c0da830650506d9568b25aaed55

Merge pull request #2926 from piotrdrag/pl-update-190528-1.4.x Update Polish translation for flatpak-1.4.x

view details

Will Thompson

commit sha 5a16666bdc779994733b8deadb8006b5bb266f17

dir: include NULL url in flatpak_dir_log() call I spotted this line in the output from `flatpak history`: Jun 4 16:17:20 deploy install com.discordapp.Discord x86_64 stable system flathub 8a0fc700c701 Installed %s from %s root (test) flatpak-system-helper (gnome-software) 1.3.3 This is because the format string is passed as the 'url' parameter, the first format parameter (the ref) is passed as the 'format' parameter, and 'origin' is ignored because (fortunately) as far as I know, no character significant to printf (like '%') is permitted in ref names. Fix this by passing a NULL 'url', like the neighbouring call in flatpak_dir_deploy_update(). (cherry picked from commit dbc90df513be5cd51efd21058401f5a41932d3d6)

view details

Ryan Gonzalez

commit sha bfb7d4217ae1907133347e4e03febabbf4d5648b

app: Avoid a potential segfault when skipping columns Closes: #2942 Approved by: mwleeds (cherry picked from commit cd231503f2ee55a5f4b312a55af4534f80a20a7c)

view details

Simon McVittie

commit sha 641121440b21198cb7f86cf4822eeb44e1ae58a9

icon-validator: Remove remnants of GSpawn error handling Now that validate-icon uses execvpe(), status and error were never set, so rerun_in_sandbox() would have crashed while dereferencing a NULL error if execvpe() failed. This is reproducible with, for example: FLATPAK_BWRAP=/bin/nope flatpak-validate-icon --sandbox 48 48 /path/to/icon execvpe() does not return on success (the process image is replaced), and sets errno on failure, so behave accordingly. Also print the error message to stderr, even if G_MESSAGES_DEBUG is not set, since it's our only opportunity to indicate to a caller what has gone wrong. Signed-off-by: Simon McVittie <smcv@collabora.com> Closes: #2950 Approved by: alexlarsson (cherry picked from commit 8deef94f9d6ed4575c6c69b505e4528099e1c7e2)

view details

Ryan Gonzalez

commit sha 96eb8837a837dd0b8d0a4866a19c34993d0ec806

remote-delete: Manually delete origin remotes if no refs were removed There are a few cases where -origin remotes don't get removed when their refs are uninstalled, most notably when xa.noenumerate is set, or somehow the uninstall gets interrupted at the wrong time. Regardless of the reason, the remote could never be removed after this, unless a new ref is installed from it and then removed, or noenumerate is set. Closes: #2920 Closes: #2953 Approved by: alexlarsson (cherry picked from commit 71fcf99b2e819966e3e818df335ed9d974b74f3b)

view details

Alexander Larsson

commit sha a24e2b3e3631351120f249daaa9562d8bc7cae23

transaction: Add back support for file: uris in RuntimeRepo keys This used to work, and the gnome-software test suite relies on it, so add it back. I believe it regressed in https://github.com/flatpak/flatpak/pull/2740 This fixes https://github.com/flatpak/flatpak/issues/2955 Closes: #2956 Approved by: alexlarsson (cherry picked from commit 1029f2b1d05bf96eb6083b5879c31997fd1d53be)

view details

Richard Hughes

commit sha 8b42f89e7a3c4391f4937ff983fee89bef127e2d

Do not break ABI and cause gnome-software to crash This restores the ABI to the pre-1.4.0 version. This moves the new signal to the *end* of the struct and also correctly decrements the padding. Fixes https://github.com/flatpak/flatpak/issues/2957, although we probably need a 1.4.1 release with this included pretty quickly to avoid chaos. Closes: #2958 Approved by: alexlarsson (cherry picked from commit 4f327649fd6a6347d7ab2c385c924d31676196a8)

view details

Ryan Gonzalez

commit sha d17f4487c366da0352de68a0146e343d0988f4a2

revokefs-fuse: Fix some build warnings Closes: #2952 Approved by: alexlarsson (cherry picked from commit 2b939282859143b0a86beed90574a791e877d4e1)

view details

Alexander Larsson

commit sha 45118ab1f5ba7813af30fe8430e3da7a0f79cb17

Update Version to 1.4.1

view details

Alexander Larsson

commit sha 8857cf1d7e9e68ca56d22fadb23767258f16af1c

Update NEWS for release

view details

Alexander Larsson

commit sha 663b5cd37036e59ec8b0447df644d385984f1d06

Update pofiles for release

view details

Ryan Gonzalez

commit sha e4bf242764efd3d69911e076915489e833fdafb6

build-init: Export an extension's runtime in the metadata Without this, extensions cannot use extra-data, as there is no indication of what runtime to run apply_extra in. Closes: #2954 Approved by: alexlarsson (cherry picked from commit 7222a8367880db910ef9843dcd3a95256be92a85) Closes: #2992 Approved by: alexlarsson

view details

Ryan Gonzalez

commit sha 1ca31146d367f0d08cddd083677503b5cafbc6e4

transaction: Install an extension's required runtime Closes: #2954 Approved by: alexlarsson (cherry picked from commit c87c480a18e6a112e49c639f7b5a818135ef1e68) Closes: #2992 Approved by: alexlarsson

view details

Ryan Gonzalez

commit sha cdc8d2deb5567727c2adaa0f962cba75ab86ac1a

dir: Use ExtensionOf.runtime for apply_extra Closes: #2954 Approved by: alexlarsson (cherry picked from commit 9cd682b057206de8596bdba6303a0414207ef621) Closes: #2992 Approved by: alexlarsson

view details

Ryan Gonzalez

commit sha 40de35049a01508d5b11f4837a455ce4c20684ce

doc: Document ExtensionOf.runtime Closes: #2954 Approved by: alexlarsson (cherry picked from commit da62f665da16163ae1250f55fc26a57fd3eb2800) Closes: #2992 Approved by: alexlarsson

view details

Matthew Leeds

commit sha 3f8a68e40fad4600e692ae4c16d7d6046317cbf4

main: Handle double slashes in $XDG_DATA_DIRS When checking for Flatpak directories in $XDG_DATA_DIRS, treat /example//path/ as equivalent to /example/path/. Fixes https://github.com/flatpak/flatpak/issues/2989 Closes: #2990 Approved by: alexlarsson (cherry picked from commit 4fd7d7d2096cddbcbd2984c908a663c9347cfbdd) Closes: #2992 Approved by: alexlarsson

view details

Matthew Leeds

commit sha 944e7bf381c497111cbe10aacfc08bf4908e3932

common/flatpak-dir-private.h: Fix order of arguments This issue was introduced by commit 5da7a0411. Fixes https://github.com/flatpak/flatpak/issues/2966 Closes: #2968 Approved by: matthiasclasen (cherry picked from commit 1ce250b5bd9ccb07b601a90b8742a9f0744c268c) Closes: #2992 Approved by: alexlarsson

view details

push time in 9 days

issue commentflatpak/flatpak

"flatpak install" doesn't use static deltas when downloading repository data

Not sure, i mainly do backports for security things.

clopez

comment created time in 10 days

issue closedflatpak/flatpak

"flatpak install" doesn't use static deltas when downloading repository data

Linux distribution and version

Debian 10 (Buster) and Fedora 31 (tested both)

Flatpak version

1.2.5 (Debian) and 1.4.3 (Fedora)

Description of the problem

Flatpak documentation advertises that you can generate a repository with static deltas to make downloads much faster.

From: https://docs.flatpak.org/en/latest/hosting-a-repository.html

Flatpak supports something called static deltas. These are single files that contain all the data needed to go between two revisions (or from nothing to a revision). Creating such deltas will take up more space on the server, but will make downloads much faster. This can be done with the flatpak build-update-repo --generate-static-deltas option.

From: https://blogs.gnome.org/alexl/2017/02/10/maintaining-a-flatpak-repository/

Additionally, as objects are whole files, we still have to download the entire file even if just one byte in the file changed. In order to solve this flatpak supports something called static deltas. These are pre-generated delta files (using bsdiff) that contain all the data needed to go from one version to another (or from nothing to a version). If these files are available they are used instead of the individual objects, which allows pulls to be much more efficient. You can generate static deltas for the latest versions of all apps by passing --generate-static-deltas to flatpak build-update-repo, and I recommend everyone do this for production repositories.

The problem is that the command flatpak install seems unable to use static deltas in practice when downloading the repository data, and it downloads all the objects one by one, making the download to be very slow for a repository that contains lot of small objects (lot of small files), even when the repository has enabled static deltas.

Downloading the very same repository with ostree pull instead of with flatpak install works as expected and static deltas are used, therefore making the download much faster.

Steps to reproduce

1. test: flatpak install (static deltas not working)

Try to download the following test repository with flatpak install and see how the download speed its slow (less than 2Mbytes/sec).

rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo 
flatpak install --user webkit-sdk org.webkit.Sdk 0.1 --assumeyes --no-related

On top of the obvious slow download, you can verify that flatpak its downloading the (many small) individual objects instead of the deltas by checking the many HTTP petitions its doing, either with wireshark/tcpdump or by passing the environment variable OSTREE_DEBUG_HTTP=1 and counting the GET petitions it reports the debug log.

rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
OSTREE_DEBUG_HTTP=1 flatpak install --user webkit-sdk org.webkit.Sdk 0.1 --assumeyes --no-related 2>&1 | grep GET | wc -l

And the last command (wc -l) reports (after a long while): 85751 (that its, it did more than 85k GET petitions!, that its clearly not using deltas)

2. test: ostree pull (static deltas working)

Now let's do the same but instead of flatpak install let's use ostree pull to pull the very same repository.

rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
ostree --repo=repo pull webkit-sdk runtime/org.webkit.Sdk/x86_64/0.1

Now you can see the download speed much faster. Then you can also see how OSTree prints its downloading a few deltas only. On top of that you can verify its the case by debugging it like previously:

rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
OSTREE_DEBUG_HTTP=1 ostree --repo=repo pull webkit-sdk runtime/org.webkit.Sdk/x86_64/0.1 2>&1 | grep GET | wc -l

In this case (with ostree pull), the last command completes much faster and (wc -l) only reports: 295 GET petitions (because it used deltas for downloading)

This very same issue its reproducible with other flatpak repositories, like for example, the LibreOffice one on FlatHub, but there the issue its less obvious because the LibreOffice flatpak repository has much less objects than the one from this example. The one from this example has 85744 objects with an average size of 10kb per object, meanwhile the LibreOffice one (app/org.libreoffice.LibreOffice/x86_64/stable) has "only" 5669 objects with an average size of 46Kb per object.

Downloading the objects individually instead of using the static deltas its a problem for this kind of flatpak repositories with so many small objects. I would like "flatpak install" to support downloading it using static deltas like "ostree pull" does.

closed time in 10 days

clopez

release flatpak/flatpak

1.6.2

released time in 10 days

push eventflatpak/flatpak

Alexander Larsson

commit sha 609217650d6972aa3e408bfeb26488870178ac4c

Bump version to 1.6.2

view details

Alexander Larsson

commit sha 01a8f5ad2cbb881cff051d05da8abda86dd4e852

Update NEWS for 1.6.2

view details

Alexander Larsson

commit sha c80eae2f9158ddc6abbbb214157d0620652fe440

Update pofiles for release

view details

push time in 10 days

created tagflatpak/flatpak

tag1.6.2

Linux application sandboxing and distribution framework

created time in 10 days

pull request commentostreedev/ostree

Fix handling of scratch deltas with existing commits

That seems fine to me. Although I'm gonna do the workaround in flatpak, so its not a huge hurry for me for this change.

dbnicholson

comment created time in 10 days

push eventflatpak/flatpak

Alexander Larsson

commit sha 30636a508d75e03b95bd507f8a6cf11714adc805

system-helper: Change debug prefix from F to FH This makes it easier to see what message comes from where.

view details

Alexander Larsson

commit sha 087ba2d23f28d0d296ee3ff45d4f27fd1aa27981

system-helper: Support -vv and --ostree-verbose

view details

Alexander Larsson

commit sha b371ef9007e6136dc956ac2507d30ce020827247

Actually use from-scratch deltas As noticed in https://github.com/flatpak/flatpak/issues/3412 we regressed at some point and are no longer using from-scratch deltas. This is caused by an optimization in ostree where it decides to not use a from-scratch deltas if theres is *some* version of the ref locally available. This conflicts with some code in flatpak that pulls *only* the commit object in order to look for extra data size information so that we can get the progress reporting right. Unfortunately the existance of just the object triggers the above causing us to *never* use from-scratch deltas. We fix this by throwing away the partial pull in an aborted ostree transaction.

view details

Alexander Larsson

commit sha 9aecad7f4fb32e0dffc38eb18e78629f931787e3

p2p: Don't mirror ostree-metadata refs when pulling into the child repo This breaks Deploy which can't find the ref. It used to work due to the extra non-mirroring pull in flatpak_dir_setup_extra_data, but this is not needed here.

view details

Alexander Larsson

commit sha 2481207a6fc086ddadfb8b90f3211f450151013d

run: Fix uninitialized use warning This isn't actually used ununitialized, but gcc can't figure that out.

view details

Alexander Larsson

commit sha b03916f5bdf878ba42af7ccf569233b839d4b3d2

setup-extra-data: Avoid extra work for ostree-metadata and appstream branches We never have extra data for non app/ or runtime/ refs, so lets not do an unnecessary pull there.

view details

push time in 10 days

PR merged flatpak/flatpak

Actually use from-scratch deltas

As noticed in https://github.com/flatpak/flatpak/issues/3412 we regressed at some point and are no longer using from-scratch deltas. This is caused by an optimization in ostree where it decides to not use a from-scratch deltas if theres is some version of the ref locally available.

This conflicts with some code in flatpak that pulls only the commit object in order to look for extra data size information so that we can get the progress reporting right. Unfortunately the existance of just the object triggers the above causing us to never use from-scratch deltas.

We fix this by throwing away the partial pull in an aborted ostree transaction.

+65 -28

3 comments

3 changed files

alexlarsson

pr closed time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha f4d56f5472b83d5c2378c27e4b007dc424686b65

setup-extra-data: Avoid extra work for ostree-metadata and appstream branches We never have extra data for non app/ or runtime/ refs, so lets not do an unnecessary pull there.

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha d24649b9be40e2c7bec6d33dfc6acb393241789f

run: Fix uninitialized use warning This isn't actually used ununitialized, but gcc can't figure that out.

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha c7e13c7ab99f67490891b044ac854ce976333327

system-helper: Change debug prefix from F to FH This makes it easier to see what message comes from where.

view details

Alexander Larsson

commit sha e8547f34374b8ec1d64368a3e0c7fde9433780fc

system-helper: Support -vv and --ostree-verbose

view details

Alexander Larsson

commit sha 6084753f192a06a421f1692b4ed1cf4cda6a3a2a

Actually use from-scratch deltas As noticed in https://github.com/flatpak/flatpak/issues/3412 we regressed at some point and are no longer using from-scratch deltas. This is caused by an optimization in ostree where it decides to not use a from-scratch deltas if theres is *some* version of the ref locally available. This conflicts with some code in flatpak that pulls *only* the commit object in order to look for extra data size information so that we can get the progress reporting right. Unfortunately the existance of just the object triggers the above causing us to *never* use from-scratch deltas. We fix this by throwing away the partial pull in an aborted ostree transaction.

view details

Alexander Larsson

commit sha 63e2dc4cf06d2881e8e93035d5373a2ff8eb7cc1

p2p: Don't mirror ostree-metadata refs when pulling into the child repo This breaks Deploy which can't find the ref. It used to work due to the extra non-mirroring pull in flatpak_dir_setup_extra_data, but this is not needed here.

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha e7f9bd007be34d858c12ebcbb025ad3268ea1c46

Test CI issue workaround

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha ca77d482a3421e952ef5dd2d9fe6d2cfdc612c71

spew more in system-helper

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha b20156c6b7e28817bb77136a1e3046d7b73ec4bf

spew in _flatpak_dir_fetch_remote_state_metadata_branch

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 75fcdca399c213da0fcf764b121fe9c2276b87e4

MOAR SPEW!!!

view details

push time in 10 days

pull request commentostreedev/ostree

Fix handling of scratch deltas with existing commits

As I said here: https://github.com/ostreedev/ostree/issues/2004#issuecomment-585586281 I don't think this is quite right. The point of the original check was that if the ref is available at all locally its likely we have a previous version and thus a from-scratch-pull is likely inefficient.

With this change if you had a previous version of the ref and with flatpak downloading a COMMIT_ONLY of the latest version of the ref then the is-partial check will always be true, even if we have some other version of the ref so using the from-scratch is not a good idea.

dbnicholson

comment created time in 10 days

more