profile
viewpoint
Alexander Larsson alexlarsson Sweden

pull request commentflatpak/xdg-desktop-portal

Make document portal exit on unmount

Looks good to me. Did you try it?

matthiasclasen

comment created time in a day

Pull request review commentflatpak/flatpak

installation: Return main app as updatable if related extensions are missing

 _ostree_collection_ref_free0 (OstreeCollectionRef *ref)   ostree_collection_ref_free (ref); } +static gboolean+flatpak_installation_check_installed_ref_missing_related_ref (FlatpakInstallation *self,+                                                              const gchar         *remote_name,+                                                              gchar               *full_ref,+                                                              GCancellable        *cancellable)+{+  g_autoptr(FlatpakDir) dir  = NULL;+  g_autoptr (GPtrArray) installed_related_refs = NULL;+  g_autoptr (GPtrArray) remote_related_refs = NULL;+  g_autoptr (GError) local_error = NULL;+  guint j, k;++  dir = flatpak_installation_get_dir (self, &local_error);+  if (dir == NULL)+    return FALSE;++  installed_related_refs = flatpak_installation_list_installed_related_refs_sync (self,+                                                                                  remote_name,+                                                                                  full_ref,+                                                                                  cancellable,+                                                                                  &local_error);+  if (local_error != NULL)+    {+      g_warning ("Unable to get installed related refs for %s: %s", full_ref, local_error->message);+      return FALSE;+    }++  g_return_val_if_fail (installed_related_refs != NULL, FALSE);++  remote_related_refs = flatpak_installation_list_remote_related_refs_sync (self,

That way you can also reuse a single FlatpakRemoteState for all the calls for this remotes, avoiding a lot of i/o to re-read it.

uajain

comment created time in a day

Pull request review commentflatpak/flatpak

installation: Return main app as updatable if related extensions are missing

 _ostree_collection_ref_free0 (OstreeCollectionRef *ref)   ostree_collection_ref_free (ref); } +static gboolean+flatpak_installation_check_installed_ref_missing_related_ref (FlatpakInstallation *self,+                                                              const gchar         *remote_name,+                                                              gchar               *full_ref,+                                                              GCancellable        *cancellable)+{+  g_autoptr(FlatpakDir) dir  = NULL;+  g_autoptr (GPtrArray) installed_related_refs = NULL;+  g_autoptr (GPtrArray) remote_related_refs = NULL;+  g_autoptr (GError) local_error = NULL;+  guint j, k;++  dir = flatpak_installation_get_dir (self, &local_error);+  if (dir == NULL)+    return FALSE;++  installed_related_refs = flatpak_installation_list_installed_related_refs_sync (self,+                                                                                  remote_name,+                                                                                  full_ref,+                                                                                  cancellable,+                                                                                  &local_error);+  if (local_error != NULL)+    {+      g_warning ("Unable to get installed related refs for %s: %s", full_ref, local_error->message);+      return FALSE;+    }++  g_return_val_if_fail (installed_related_refs != NULL, FALSE);++  remote_related_refs = flatpak_installation_list_remote_related_refs_sync (self,

This creates a lot of FlatpakRef objects etc, but all you need is the ref and should_download, so i recommend using flatpak_dir_find_remote_related() directly instead.

uajain

comment created time in a day

Pull request review commentflatpak/flatpak

installation: Return main app as updatable if related extensions are missing

 flatpak_installation_list_installed_refs_for_update (FlatpakInstallation *self,                                                                             full_ref,                                                                             cancellable))             g_ptr_array_add (updates, g_object_ref (installed_ref));++          /* Check for runtime: Now see if any of the installed ref is missing its runtime component.+           * If so, return that installed ref in the updates list, so that FlatpakTransaction+           * can resolve one of its operation to install the runtime instead.+           *+           * Runtime of an app can go missing if an app upgrade makes an app dependent on a new runtime+           * entirely. We had couple of cases like that in the past, for example, before it was updated+           * to use FlatpakTransaction, updating an app in GNOME Software to a version which needs a+           * different runtime would not install that new runtime, leaving the app unusable.+           */+          deploy_data = flatpak_dir_get_deploy_data (dir, full_ref, FLATPAK_DEPLOY_VERSION_CURRENT, cancellable, &local_error);+          if (local_error != NULL)

In these kind of fixups i don't think the errors and whatnot are useful and just complicate the code. I'd recommend not passing local_error and just move all the code below into a if (deploy_data != NULL) { ... }

uajain

comment created time in a day

Pull request review commentflatpak/flatpak

installation: Return main app as updatable if related extensions are missing

 flatpak_installation_list_installed_refs_for_update (FlatpakInstallation *self,                                                                             full_ref,                                                                             cancellable))             g_ptr_array_add (updates, g_object_ref (installed_ref));++          /* Check for runtime: Now see if any of the installed ref is missing its runtime component.+           * If so, return that installed ref in the updates list, so that FlatpakTransaction+           * can resolve one of its operation to install the runtime instead.+           *+           * Runtime of an app can go missing if an app upgrade makes an app dependent on a new runtime+           * entirely. We had couple of cases like that in the past, for example, before it was updated+           * to use FlatpakTransaction, updating an app in GNOME Software to a version which needs a+           * different runtime would not install that new runtime, leaving the app unusable.+           */+          deploy_data = flatpak_dir_get_deploy_data (dir, full_ref, FLATPAK_DEPLOY_VERSION_CURRENT, cancellable, &local_error);+          if (local_error != NULL)+            {+              g_warning ("Unable to get deploy data for %s: %s", full_ref, local_error->message);+              g_clear_error (&local_error);+              continue;+            }++          runtime = flatpak_deploy_data_get_runtime (deploy_data);+          full_runtime_ref = g_strconcat ("runtime/", runtime, NULL);+          deploy_dir = flatpak_dir_get_if_deployed (dir, full_runtime_ref, NULL, cancellable);+          if (deploy_dir == NULL)+            g_ptr_array_add (updates, g_object_ref (installed_ref));

We might add installed_ref twice here.

uajain

comment created time in a day

Pull request review commentflatpak/flatpak

installation: Return main app as updatable if related extensions are missing

 _ostree_collection_ref_free0 (OstreeCollectionRef *ref)   ostree_collection_ref_free (ref); } +static gboolean+flatpak_installation_check_installed_ref_missing_related_ref (FlatpakInstallation *self,+                                                              const gchar         *remote_name,+                                                              gchar               *full_ref,+                                                              GCancellable        *cancellable)+{+  g_autoptr(FlatpakDir) dir  = NULL;+  g_autoptr (GPtrArray) installed_related_refs = NULL;+  g_autoptr (GPtrArray) remote_related_refs = NULL;+  g_autoptr (GError) local_error = NULL;+  guint j, k;++  dir = flatpak_installation_get_dir (self, &local_error);+  if (dir == NULL)+    return FALSE;++  installed_related_refs = flatpak_installation_list_installed_related_refs_sync (self,+                                                                                  remote_name,+                                                                                  full_ref,+                                                                                  cancellable,+                                                                                  &local_error);+  if (local_error != NULL)+    {+      g_warning ("Unable to get installed related refs for %s: %s", full_ref, local_error->message);+      return FALSE;+    }++  g_return_val_if_fail (installed_related_refs != NULL, FALSE);++  remote_related_refs = flatpak_installation_list_remote_related_refs_sync (self,+                                                                            remote_name,+                                                                            full_ref,+                                                                            cancellable,+                                                                            &local_error);+  if (local_error != NULL)+    {+      g_warning ("Unable to get remote related refs for %s: %s", full_ref, local_error->message);+      return FALSE;+    }++  if (remote_related_refs == NULL)+    return FALSE;++  for (j = 0; j < remote_related_refs->len; j++)+    {+      FlatpakRelatedRef *remote_related_ref = g_ptr_array_index (remote_related_refs, j);+      g_autofree gchar *remote_related_full_ref = flatpak_ref_format_ref (FLATPAK_REF (remote_related_ref));+      gboolean should_download = flatpak_related_ref_should_download (remote_related_ref);+      gboolean related_ref_found = FALSE;++      if (flatpak_dir_ref_is_masked (dir, remote_related_full_ref))+          continue;++      if (should_download)+        {+          for (k = 0; k < installed_related_refs->len; k++)

I think this is doing quite a bit of unnecessary work by finding all matching installed extensions and then comparing to them. A better approach would be to just get the remote related refs, and then call flatpak_dir_get_if_deployed (dir, remote_related_full_ref, NULL, NULL) to see if its installed already.

uajain

comment created time in a day

Pull request review commentflatpak/flatpak

installation: Return main app as updatable if related extensions are missing

 flatpak_installation_list_installed_refs_for_update (FlatpakInstallation *self,       if (remote_commit != NULL &&           g_strcmp0 (remote_commit, local_commit) != 0)         g_ptr_array_add (updates, g_object_ref (installed_ref));+      else if (flatpak_ref_get_kind (FLATPAK_REF (installed_ref)) == FLATPAK_REF_KIND_APP)

Runtimes can also have similar extensions, do we not need to do a similar thing there? For instance, mark the runtime as needs update when there is a new nvidia driver extension for it needed?

uajain

comment created time in a day

issue commentflatpak/flatpak

Flatpak repair doesn't fix invalid objects

Its really hard do do anything with so little information. Can you attach some logs of you running sudo flatpak repair?

Do you use system or per-user installs?

monterrr

comment created time in a day

pull request commentflatpak/flatpak

utils: Allow chaining OstreeAsyncProgress when pushing GMainContext

@ptomato yes, that looks good. As i mentioned in the comment you probably want a destroy notify for the FlatpakProgress user_data too.

I'd be fine to merge this for now when the ostree pr is merged though.

ptomato

comment created time in a day

CommitCommentEvent

push eventflatpak/flatpak

Alexander Larsson

commit sha 6df104837cc0cd47d756e050fca91ed1ee10d25f

Require bubblewrap 0.3.1 for --bind-try support We want to use this to work around some stuff for flatpak enter to work.

view details

Alexander Larsson

commit sha 7079c6c722366b8fa5bf304ad582c56c73abe17f

run: Keep around base user namespace reference in /run/.userns There is a kernel issue which has been fixed in linux 4.9: https://github.com/torvalds/linux/commit/e98d41370392dbc3e94c8802ce4e9eec9efdf92e Which makes it impossible (on older kernels) to mount devpts unless uid 0 is mapped in the user namespace. Bubblewrap works around this by using two namespaces, the base one which sets up everything (and thus owns all the other namespaces), and then at the end a child of that that remaps uid 0 to the real uid. Unfortunately, this makes it impossible to enter the bubblewrap user namespace, because there are no references to the intermediate user namespace we can use. To work around this we make a bind mount of the intermediate namespace during setup using --ro-bind-try which we can use for nsenter.

view details

Alexander Larsson

commit sha 180494ca41518a649eb96ae06ccd9d382aec3aad

run: Don't install polkit agent for enter/run/build Neither of these ever need a polkit agent, and run/build are somewhat performance sensitive and we don't want to connect to dbus unnecessarily. For enter this is critical though, as the dbus connection starts a thread which is not compatible with the setns syscall.

view details

Alexander Larsson

commit sha e16b40dfeca28c456e444f452dd243f8e6bffa45

enter: Use the correct pid (child pid) as the target for entering This is the one in a namespace, the other one is just the wrapper.

view details

Alexander Larsson

commit sha f568a068c6d6e836735ac1ff6ac11042c119ff5d

enter: Support non-root flatpak enter Assuming unprivileged namespaces works we can now user the .userns bindmount to access the intermediate bubblewrap user namespace. This also drops the warning about root, and make sure we drop all caps at the end.

view details

Alexander Larsson

commit sha a0cb7f69641aaf322841860668bff425d829c37e

enter: No need to fail if a pid is specified that it is not the primary pid. We can enter from any pid in the sandbox.

view details

Alexander Larsson

commit sha 47e2718833e66c398b31da1971f220d8353a0677

enter: Clean up namespace equality checks There is no need to read the links, just look at the inode nr which is the same info, and that also works on the bind-mounted .userns thing where readlink fails. Also, don't fail for non-existing namespaces.

view details

Alexander Larsson

commit sha a094aa0040751c60ec576c4eef9843fbd23e5066

enter: Add comment about why NS_GET_USERNS ioctl is not used

view details

Alexander Larsson

commit sha 6d68094879c9c9fb79f5f7149ce36dc73966920d

enter: Better error values if enter is not supported on your kernel

view details

Alexander Larsson

commit sha 4718bcccfd7af2663d59164b264048609897c7a6

enter: Make sure sudo -E works when using setuid brwap This also updates the manpage so it is correct wrt permissions and sudo.

view details

push time in a day

PR merged flatpak/flatpak

Support flatpak enter without requiring root (and fix it)

This just straight out didn't work. However, now it works, as long as your system is using unprivileged user namespaces (rather than the setuid bwrap).

This bumps the system bubblewrap dependency to 0.3.1 for the --bind-try, but the bundled version is already 0.3.3 so that works fine.

+77 -39

1 comment

5 changed files

alexlarsson

pr closed time in a day

push eventflatpak/flatpak

Matthew Leeds

commit sha e4676955498f2dee6eb0aced54258612693b0cb3

tests: Delete test-p2p-security.sh This test was intended to verify that updates from remote B can't interfere with updates from remote A even if remote B maliciously sets the same collection ID as remote A. However, the commits intended to protect against this turned out to have nasty side effects and need to be reverted.[1] A subsequent commit will revert the use of OSTREE_REPO_PULL_FLAGS_MIRROR which means this attack is not exploitable (since refs will be resolved using a refspec which includes the remote name), at the cost of not supporting more than one remote having the same collection ID configured. Since we don't support that, it doesn't make sense to keep this unit test. Also, the test seems to be failing. [1] https://github.com/flatpak/flatpak/issues/3215

view details

Matthew Leeds

commit sha 13366524d82af14ad35fa4618ccf202ece1adacc

Revert "dir: Check commit signatures before resolving a ref" This reverts commit 915ad583a7bf70e03bf58bf14b9d3bdb7ef33277. This commit turned out to have unintended side effects. Specifically, with it we do a pull with OSTREE_REPO_PULL_FLAGS_MIRROR, and then flatpak_dir_setup_extra_data() does a non-mirror pull in the same transaction, so the ref being pulled ends up being written to disk under both refs/remotes/ and refs/mirrors/ in ostree_repo_commit_transaction(). This is a problem because only the remote ref is deleted during an uninstall, so the disk space is leaked, and we don't have the infrastructure in place to keep both refs up to date as they're updated. It would be nice to consistently use OSTREE_REPO_PULL_FLAGS_MIRROR for all pulls but that turns out to be a deep rabbit hole to go down; see the discussion in https://github.com/flatpak/flatpak/pull/3220 So revert the commit instead (with a few exceptions: keep a still-relevant FIXME comment, keep an assertion in the "out:" section, and keep a debug statement printing out the resolved rev). Note that this means that since we're no longer checking commit signatures during ref resolution, in theory remote B could try to set the same collection ID as remote A and serve a malicious update for something from remote A, but the signature would be found to be invalid during the pull phase due to our use of "ref-keyring-map" so the transaction would fail. All the other uses of OSTREE_REPO_PULL_FLAGS_MIRROR across the codebase should be kept I think: - flatpak create-usb uses it when pulling into the repo on the USB which works perfectly well with refs/mirrors/ (and the USB is mirroring the collection-refs!) - it's used when pulling into a temporary "child" repo in a few places and there it makes sense since the child repo is mirroring the refs so they can be pulled into the main repo. In fact, in the case of flatpak_dir_do_resolve_p2p_refs(), we need MIRROR since otherwise ostree_repo_resolve_collection_ref() gives us the commit on-disk rather than the just-pulled one that's in memory.

view details

Matthew Leeds

commit sha eabc52456a7915c4900e3e39fdded1309969d017

Clean up duplicated mirror refs Due to bug #3215 some systems have refs in refs/mirrors/ in addition to the usual refs/remotes/ location. The remote refs are always at least as new as the mirror ones since the repo_pull() invocation in flatpak_dir_pull() which does not use OSTREE_PULL_FLAGS_MIRROR happened after the one that did. Cleaning up these mirror refs is important since otherwise when the remote ref is either updated or removed (by an uninstall) disk space will be leaked since the mirror ref will point to a no longer needed commit. So, remove (almost) all mirror refs during flatpak repair, uninstall, or update operations. And for the uninstall and update operations do it in FlatpakDir so that it happens regardless of if the CLI of libflatpak are used. Also, add a unit test for this. Fixes https://github.com/flatpak/flatpak/issues/3222

view details

Matthew Leeds

commit sha 10f6166a40866cbdb4af83eec0511cee2076b21a

tests: Add mask command to test-basic and test-completion

view details

push time in a day

issue closedflatpak/flatpak

Flatpak should clean up duplicated mirror/remote refs

As mentioned in https://github.com/flatpak/flatpak/issues/3215#issuecomment-552083779 we need Flatpak to detect and delete mirror refs which have been left behind when only the remote ref was deleted during an app/runtime uninstall.

We should also look into what happens during an update operation that happens with duplicated refs. Is only one updated? Should we delete the other at that point?

closed time in a day

mwleeds

Pull request review commentflatpak/flatpak

Add a permission-set command

+/*+ * Copyright © 2018 Red Hat, Inc+ *+ * This program is free software; you can redistribute it and/or+ * modify it under the terms of the GNU Lesser General Public+ * License as published by the Free Software Foundation; either+ * version 2.1 of the License, or (at your option) any later version.+ *+ * This library is distributed in the hope that it will be useful,+ * but WITHOUT ANY WARRANTY; without even the implied warranty of+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the GNU+ * Lesser General Public License for more details.+ *+ * You should have received a copy of the GNU Lesser General Public+ * License along with this library. If not, see <http://www.gnu.org/licenses/>.+ *+ * Authors:+ *       Matthias Clasen <mclasen@redhat.com>+ */++#include "config.h"++#include <locale.h>+#include <stdlib.h>+#include <string.h>+#include <unistd.h>+#include <errno.h>++#include <glib/gi18n.h>++#include "libglnx/libglnx.h"+#include "flatpak-permission-dbus-generated.h"++#include "flatpak-builtins.h"+#include "flatpak-table-printer.h"+#include "flatpak-utils-private.h"+#include "flatpak-run-private.h"++static GOptionEntry options[] = {+  { NULL }+};++static char **+get_ids_for_table (XdpDbusPermissionStore *store,+                   const char             *table)+{+  char **ids = NULL;++  xdp_dbus_permission_store_call_list_sync (store, table, &ids, NULL, NULL);++  return ids;+}++static const char *device_perms[] = { "yes", "no", "ask", NULL };+static const char *document_perms[] = { "read", "write", "delete", "grant-permissions", NULL };+static const char *notification_perms[] = { "yes", "no", NULL };++static const char **+get_permission_values_for_table (const char *table)+{+  if (strcmp (table, "devices") == 0)+    return device_perms;+  else if (strcmp (table, "documents") == 0)+    return document_perms;+  else if (strcmp (table, "notifications") == 0)+    return notification_perms;++  return NULL;+}++static gboolean+set_permissions (XdpDbusPermissionStore  *store,+                 const char              *table,+                 const char              *id,+                 const char              *app_id,+                 const char             **permissions,+                 GError                 **error)+{+  GVariant *perms = NULL;+  GVariant *data = NULL;+  GVariantBuilder builder;+  int i;+  GVariant *new_perms = g_variant_new_strv (permissions, -1);++  if (!xdp_dbus_permission_store_call_lookup_sync (store, table, id,+                                                   &perms, &data,+                                                   NULL, error))+    return FALSE;++  g_variant_builder_init (&builder, G_VARIANT_TYPE ("a{sas}"));+  for (i = 0; perms && i < g_variant_n_children (perms); i++)+    {+      const char *key;+      GVariant *value = NULL;++      g_variant_get_child (perms, i, "{&s@as}", &key, &value);+      if (strcmp (key, app_id) == 0)+        {+          if (permissions[0] != NULL)+            g_variant_builder_add (&builder, "{s@as}", key, new_perms);+          new_perms = NULL;+        }+      else+        g_variant_builder_add (&builder, "{s@as}", key, value);+    }++  if (new_perms != NULL)+    {+      if (permissions[0] != NULL)+        g_variant_builder_add (&builder, "{s@as}", app_id, new_perms);+    }++  if (!xdp_dbus_permission_store_call_set_sync (store, table, TRUE, id,+                                                g_variant_builder_end (&builder),+                                                data ? data : g_variant_new_byte (0),+                                                NULL, error))

This should add the permissions for the app keeping the previous ones, so I'm not sure why you read the previous values out above? Also, seems like you can use the SetPermissions call which is easier if your just doing one app.

matthiasclasen

comment created time in a day

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 054dd079643d51e8ecad500f1d6dcf97db88fc4b

enter: Make sure sudo -E works when using setuid brwap This also updates the manpage so it is correct wrt permissions and sudo.

view details

push time in a day

push eventalexlarsson/flatpak

Alexander Larsson

commit sha cbecbbed6f175206e2271dac07f4fec20278b737

enter: Clean up namespace equality checks There is no need to read the links, just look at the inode nr which is the same info, and that also works on the bind-mounted .userns thing where readlink fails. Also, don't fail for non-existing namespaces.

view details

Alexander Larsson

commit sha 2a043a50504a8e5a482f6ffcb026ec6a2f311481

enter: Add comment about why NS_GET_USERNS ioctl is not used

view details

Alexander Larsson

commit sha ed5cf86a1970d28ccdecdb969bf7d4e2702f9d58

enter: Better error values if enter is not supported on your kernel

view details

push time in a day

PR opened flatpak/flatpak

Support flatpak enter without requiring root (and fix it)

This just straight out didn't work. However, now it works, as long as your system is using unprivileged user namespaces (rather than the setuid bwrap).

This bumps the system bubblewrap dependency to 0.3.1 for the --bind-try, but the bundled version is already 0.3.3 so that works fine.

+70 -30

0 comment

4 changed files

pr created time in a day

create barnchalexlarsson/flatpak

branch : fix-enter

created branch time in a day

pull request commentflatpak/flat-manager

Support requiring tokens for some refs

I’ll have a more detailed look later, but for the token type there are two answers. First the technical, token type zero is “behave as before” (ie no tokens needed), and anything else is undefined in general. Any non zero value triggers an authenticator call passing the ref and the token type. It is then up to the Authenticator what to do about it. It could even return an empty token if it want to signal that no token was needed anyway.

The second answer is how I imagine flathub to use this. I think we’ll have two values. One for “needs purchase” and one for “needs donation flow”. Both of these will just run some webflow that end up in a possibly empty token. However the purchase one will start with a login and will always try to get a token, whereas the donation one will uses an anonymous webflow the first time and then track locally that it already did so so it can be skipped the next time.

alexlarsson

comment created time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 38ad065e14a0978a920b5a3d91b829fe74355f86

token: Remove unused variable

view details

push time in 10 days

pull request commentflatpak/flatpak

Update Ukrainian translation

This branch cannot be rebased due to conflicts

Can you rebase this, and ideally merge it to just one commit.

yurchor

comment created time in 10 days

push eventalexlarsson/flat-auth

Alexander Larsson

commit sha 9286cef8cb23a9582fec33ef8355edeca30401c6

Now with purchase timeouts

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 4a12772e8a6b222e0dc93df8ef337aa888e1b263

webflow: Clean up argument names We use base_uri + uri, to imply you can completely replace base_uri if you pass in an absolute uri instead.

view details

Alexander Larsson

commit sha 0bf7c60d5d934020e8623eabb21f8de63630b646

tokens: Generate purchase done uri on server instead of client This makes things more flexible for services.

view details

Alexander Larsson

commit sha 8e4cbc6ecd4f636c91da6f86cad1218dca8077f8

tokens: Allow a final redirec in the webflow Once we redirect to the authenticator we can do whatever is necessary to continue installing, but the redirect the browser to some service-specific "we're done" page.

view details

push time in 10 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 8a6b0a7851c41632054642b132db0fba89bb82ea

authenticator: Pass token-type for each ref This allows the authenticator to handle each token type differently. For example, this allows a "purchase" type to run the donation webflow, but not require login (and then store the fact that this was run locally).

view details

push time in 13 days

push eventflatpak/flat-manager

Bartłomiej Piotrowski

commit sha 22e5b58947161468335258ad9d82bcec039e41dc

Bump version to 0.3.6

view details

Bartłomiej Piotrowski

commit sha 2fe1fcebb9dd34cafc0721d5cacdceeb6057ed11

Add *.pyc to gitignore fixes #30

view details

Bartłomiej Piotrowski

commit sha 737b31ada49e74f4d86d0e5c13cda1ed806c8384

client: Replace requests with aiohttp

view details

Bartłomiej Piotrowski

commit sha 8cee93c56e46dcd9c5d22b8c8a07a277c5b9d143

client: missing_objects is also async function

view details

Bartłomiej Piotrowski

commit sha b2150a3aa1672352b9e483a0449f351f38093183

client: Simplify ApiError class

view details

Bartłomiej Piotrowski

commit sha d5cbbac292a98252de19cefa6c98b5084cac3f54

client: aiohttp uses "status" attribute instead of "status_code"

view details

Bartłomiej Piotrowski

commit sha 5acd3d4e7206b6da57ce0d04fc2cd6d41f298d91

client: Remove duplicate run_until_complete call

view details

Bartłomiej Piotrowski

commit sha 5133893c162609a354dd69370c87ca191134d1ea

client: Pass response body to ApiError exception

view details

Bartłomiej Piotrowski

commit sha f93d8cf8826c129512bfbdec7f8a98d3f389d65e

client: Await all API functions

view details

Bartłomiej Piotrowski

commit sha 1a5ccc9c501d4221522c029a07a4def305a88880

client: Get result from run_with_session

view details

Bartłomiej Piotrowski

commit sha 7c34bdb1501df1db4c72fd0a2ce3f2d8552a6313

client: Sort imports

view details

Bartłomiej Piotrowski

commit sha 0dc1deec47239b0a861a97d4b8c5df4a10afa9c7

client: Use tenacity to retry upload attempts

view details

Bartłomiej Piotrowski

commit sha d4ba193542db2814a1663543c10c80d9a7c96423

client: Import retry_if_exception_type from tenacity

view details

Bartłomiej Piotrowski

commit sha b27f652adc4545828774b8b411388ad7a235f759

client: Gracefully handle case when build has been already published

view details

Bartłomiej Piotrowski

commit sha 979f9903b72ed4341dfb522d7fadd89adbd8884e

client: Re-open file if it was closed

view details

Bartłomiej Piotrowski

commit sha ec93faf082a0f7c90948731d32c4fddab71ea63e

client: Increase number of attempts to 6

view details

Bartłomiej Piotrowski

commit sha 3a31b95bb4ab8c0af541f8651ed74f2400659b53

jobs: Generate flatpakref files also for runtimes/extensions

view details

Alexander Larsson

commit sha 941b730e4ac60ef1da0cfe3bd4f5d8a997a9a696

tokens: If not matching token prefix, return the id that failed

view details

Dan Nicholson

commit sha 2ea04d626bdc640030b8471530f504a7ba83e0a8

tokens: Scope middleware to AppState requests We only ever use the TokenParser middleware from requests with state. Change the implementation to be specific to AppState requests instead of generic so that the state can be used in the middleware.

view details

Dan Nicholson

commit sha 7e7d073b9d9d8c016ae39465e2fc085199aa3ee4

tokens: Add message for InvalidToken errors I had an invalid token and couldn't figure out what was wrong. Add a message for these errors to help debugging.

view details

push time in 13 days

release flatpak/flat-manager

0.3.7

released time in 13 days

created tagflatpak/flat-manager

tag0.3.7

Manager for flatpak repositories

created time in 13 days

push eventflatpak/flat-manager

Alexander Larsson

commit sha 66a7d4224f3ed8afd423f54cc1bbd9e3ff6ff010

Bump version to 0.3.7

view details

Alexander Larsson

commit sha 304bf8fb94bfa906f66c4b608375ad9250d62437

Update dependencies

view details

Alexander Larsson

commit sha 257c60731ef5a54254fecdd3ca89f1e5ce0ae0b3

cargo.toml: Enable default-run now that its stable

view details

Alexander Larsson

commit sha c25ee7bccdbb7492adbcee49b75937db1484595a

client: Add missing await and session from the asyncio conversion

view details

Alexander Larsson

commit sha 5252cb3f691d82d74b3ef5eb0276e216f215e461

client: push --publish --wait-update implies --wait Previously you had to specify both for --wait-update to work, which makes no sense.

view details

push time in 13 days

push eventflatpak/flat-manager

Alexander Larsson

commit sha 941b730e4ac60ef1da0cfe3bd4f5d8a997a9a696

tokens: If not matching token prefix, return the id that failed

view details

Dan Nicholson

commit sha 2ea04d626bdc640030b8471530f504a7ba83e0a8

tokens: Scope middleware to AppState requests We only ever use the TokenParser middleware from requests with state. Change the implementation to be specific to AppState requests instead of generic so that the state can be used in the middleware.

view details

Dan Nicholson

commit sha 7e7d073b9d9d8c016ae39465e2fc085199aa3ee4

tokens: Add message for InvalidToken errors I had an invalid token and couldn't figure out what was wrong. Add a message for these errors to help debugging.

view details

Alexander Larsson

commit sha 59ebfd62bbab834109308fe2d9b7109c716bdfcf

tokens: Log token errors on 403 too

view details

Dan Nicholson

commit sha 99df9f23e8f87718b76470dfffabba85d28f85fe

errors: Return 403 Forbidden for NotEnoughPermissions This error is thrown when the user has supplied a valid token but the token does not contain the appropriate permissions. This is different than a 401, which is used when the client has not authenticated. See https://en.wikipedia.org/wiki/HTTP_403.

view details

push time in 13 days

push eventflatpak/flat-manager

Alexander Larsson

commit sha 203ea5bbeea4b6a19bd45741d628ba9166523b03

tokens: Make some jwt token parts optional We allow missing prefixes, repos and name, as these are not going to be needed for repo specific tokens. The name is just a debug thing and this is not sensitive. The other ones don't really affect anything. They just let you avoid specifying no key instead of an empty array.

view details

Alexander Larsson

commit sha d323f549f5a696edaacaeae4726617364dd6aff8

tokens: Use simpler tokens for repo access The API token checks are as before, but the repo checks now just use "scope" to mean a list of id prefixes, and "sub" the user id (although this is not used atm). There is no need for all the other complexity in this case, and means we can create more standardized tokens in the authenticator.

view details

push time in 13 days

push eventflatpak/flat-manager

Bartłomiej Piotrowski

commit sha 3a31b95bb4ab8c0af541f8651ed74f2400659b53

jobs: Generate flatpakref files also for runtimes/extensions

view details

push time in 13 days

PR merged flatpak/flat-manager

Reviewers
jobs: Generate flatpakref files also for runtimes/extensions

We run generate_flatpakref for commit and publish jobs. If I understand correctly test builds are never actually published in flat-manager world but I'm not 100% sure about that.

+12 -3

2 comments

1 changed file

barthalion

pr closed time in 13 days

pull request commentflatpak/flatpak

utils: Allow chaining OstreeAsyncProgress when pushing GMainContext

Also, i think we should accept a NULL FlatpakProgress to mean "no callbacks" instead of all callers having to hack that in themselves.

ptomato

comment created time in 13 days

pull request commentflatpak/flatpak

utils: Allow chaining OstreeAsyncProgress when pushing GMainContext

I don't really like this solution. It requires a new ostree API which is not gonna be available in many places for quite some time, and it propagates the messiness that is all the g_object_set_data() hacks on the progress.

I think instead what we want is a proper FlatpakProgress type, that all callers create, which can then be used in the innermost operations to create an OstreeAsyncProgress during the actual pull. This will then be tied to the mainloop in use and needs no forwarding.

ptomato

comment created time in 13 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 2c6e2085c880e876539358aeaa8a632155d2fc34

authenticator: Add initial version of an authenticator This is a general authenticator that relies on a service with a pretty simple REST api with OAUTH2 like authentication. The idea is that we can ship it with flatpak so that flathub and others could use it. Its not the *only* authenticator possibly though. If you have specific integration needs you can create your own. The webservice API is undocumented and work in progress, but has a sample implementation here: https://github.com/alexlarsson/flat-auth

view details

push time in 14 days

PR closed flatpak/flatpak

flatpak-parental-controls: Properly initialize appdata_value when checking oars rating

Fix the following warning: common/flatpak-parental-controls.c: In function ‘flatpak_oars_check_rating’: common/flatpak-parental-controls.c:121:10: warning: ‘appdata_value’ may be used uninitialized in this function [-Wmaybe-uninitialized] if (appdata_value != NULL) ^

+1 -1

3 comments

1 changed file

andrunko

pr closed time in 14 days

pull request commentflatpak/flatpak

flatpak-parental-controls: Properly initialize appdata_value when checking oars rating

Already fixed by https://github.com/flatpak/flatpak/pull/3214

andrunko

comment created time in 14 days

push eventflatpak/flatpak

Will Thompson

commit sha 1f35dda1b50cceb24c471b19ac886dc91bf8a0b4

parental-controls: fix read of uninitialized variable If content_rating == NULL, then no value will be assigned to appdata_value, but its value will be used anyway – if it happens to be non-NULL, it will be dereferenced. common/flatpak-parental-controls.c: In function ‘flatpak_oars_check_rating’: common/flatpak-parental-controls.c:121:10: warning: ‘appdata_value’ may be used uninitialized in this function [-Wmaybe-uninitialized] if (appdata_value != NULL) ^

view details

push time in 14 days

PR merged flatpak/flatpak

parental-controls: fix read of uninitialized variable

If content_rating == NULL, then no value will be assigned to appdata_value, but its value will be used anyway – if it happens to be non-NULL, it will be dereferenced.

common/flatpak-parental-controls.c: In function ‘flatpak_oars_check_rating’:
common/flatpak-parental-controls.c:121:10: warning: ‘appdata_value’ may be used uninitialized in this function [-Wmaybe-uninitialized]
       if (appdata_value != NULL)
          ^
+1 -1

1 comment

1 changed file

wjt

pr closed time in 14 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha a0c9c734cf462c2326cc21f1efa7a79b964fa31e

tokens: Allow authenticators to send an error message

view details

Alexander Larsson

commit sha 601b32654f2566631198512d47d46485769a8e60

authenticator: Add initial version of an authenticator This is a general authenticator that relies on a service with a pretty simple REST api with OAUTH2 like authentication. The idea is that we can ship it with flatpak so that flathub and others could use it. Its not the *only* authenticator possibly though. If you have specific integration needs you can create your own. The webservice API is undocumented and work in progress, but has a sample implementation here: https://github.com/alexlarsson/flat-auth

view details

push time in 14 days

create barnchalexlarsson/flat-auth

branch : master

created branch time in 14 days

created repositoryalexlarsson/flat-auth

created time in 14 days

release flatpak/flatpak-xdg-utils

1.0.1

released time in 15 days

created tagflatpak/flatpak-xdg-utils

tag1.0.1

Simple portal-based commandline tools for use inside flatpak sandboxes

created time in 15 days

push eventflatpak/flatpak-xdg-utils

Alexander Larsson

commit sha 727e551bb0bf4a4335a9568f0687970c9d483c50

Bump version to 1.0.1

view details

push time in 15 days

push eventflatpak/flat-manager

Bartłomiej Piotrowski

commit sha 737b31ada49e74f4d86d0e5c13cda1ed806c8384

client: Replace requests with aiohttp

view details

Bartłomiej Piotrowski

commit sha 8cee93c56e46dcd9c5d22b8c8a07a277c5b9d143

client: missing_objects is also async function

view details

Bartłomiej Piotrowski

commit sha b2150a3aa1672352b9e483a0449f351f38093183

client: Simplify ApiError class

view details

Bartłomiej Piotrowski

commit sha d5cbbac292a98252de19cefa6c98b5084cac3f54

client: aiohttp uses "status" attribute instead of "status_code"

view details

Bartłomiej Piotrowski

commit sha 5acd3d4e7206b6da57ce0d04fc2cd6d41f298d91

client: Remove duplicate run_until_complete call

view details

Bartłomiej Piotrowski

commit sha 5133893c162609a354dd69370c87ca191134d1ea

client: Pass response body to ApiError exception

view details

Bartłomiej Piotrowski

commit sha f93d8cf8826c129512bfbdec7f8a98d3f389d65e

client: Await all API functions

view details

Bartłomiej Piotrowski

commit sha 1a5ccc9c501d4221522c029a07a4def305a88880

client: Get result from run_with_session

view details

Bartłomiej Piotrowski

commit sha 7c34bdb1501df1db4c72fd0a2ce3f2d8552a6313

client: Sort imports

view details

Bartłomiej Piotrowski

commit sha 0dc1deec47239b0a861a97d4b8c5df4a10afa9c7

client: Use tenacity to retry upload attempts

view details

Bartłomiej Piotrowski

commit sha d4ba193542db2814a1663543c10c80d9a7c96423

client: Import retry_if_exception_type from tenacity

view details

Bartłomiej Piotrowski

commit sha b27f652adc4545828774b8b411388ad7a235f759

client: Gracefully handle case when build has been already published

view details

Bartłomiej Piotrowski

commit sha 979f9903b72ed4341dfb522d7fadd89adbd8884e

client: Re-open file if it was closed

view details

Bartłomiej Piotrowski

commit sha ec93faf082a0f7c90948731d32c4fddab71ea63e

client: Increase number of attempts to 6

view details

push time in 15 days

pull request commentflatpak/flat-manager

jobs: Generate flatpakref files also for runtimes/extensions

Hmm, this will create flatpakref files for all refs though, including things like .Locale, .Debug, .Source, etc. That doesn't seem right.

barthalion

comment created time in 15 days

push eventflatpak/flat-manager

Alexander Larsson

commit sha da3c7192fd4a8f9350a767d50c347f1b61bc0b35

cargo.toml: Enable default-run now that its stable

view details

push time in 15 days

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 42cf356563cd120e2e49a3e5adeea599b2b36022

authenticator: Pass remote name to RequestRefTokens This makes it very easy to reuse a single authenticator for several remotes. This is useful for the a default authenticator implementation that we can ship with flatpak and use for e.g. flathub.

view details

push time in 17 days

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 operation_error (FlatpakTransaction            *transaction,   return TRUE; /* Continue */ } +static gboolean+webflow_start (FlatpakTransaction *transaction,+               const char         *remote,+               const char         *url,+               guint               id)+{+  FlatpakCliTransaction *self = FLATPAK_CLI_TRANSACTION (transaction);+  const char *browser;+  g_autoptr(GError) local_error = NULL;+  const char *args[3] = { NULL, url, NULL };++  if (!self->disable_interaction)+    {+      g_print (_("Authentication required for remote '%s'\n"), remote);+      if (!flatpak_yes_no_prompt (TRUE, _("Open browser?")))+        return FALSE;+    }++  browser = g_getenv ("BROWSER");+  if (browser == NULL)+    browser = "xdg-open";++  /* TODO: Use better way to find default browser */++  args[0] = browser;+  if (!g_spawn_async (NULL, (char **)args, NULL, G_SPAWN_SEARCH_PATH,+                      NULL, NULL, NULL, &local_error))+    {+      g_printerr ("Failed to spawn browser %s: %s\n", browser, local_error->message);+      return FALSE;+    }++  g_print ("Waiting for browser...\n");+

As in, it will return to the main loop spinning code.

alexlarsson

comment created time in 24 days

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 operation_error (FlatpakTransaction            *transaction,   return TRUE; /* Continue */ } +static gboolean+webflow_start (FlatpakTransaction *transaction,+               const char         *remote,+               const char         *url,+               guint               id)+{+  FlatpakCliTransaction *self = FLATPAK_CLI_TRANSACTION (transaction);+  const char *browser;+  g_autoptr(GError) local_error = NULL;+  const char *args[3] = { NULL, url, NULL };++  if (!self->disable_interaction)+    {+      g_print (_("Authentication required for remote '%s'\n"), remote);+      if (!flatpak_yes_no_prompt (TRUE, _("Open browser?")))+        return FALSE;+    }++  browser = g_getenv ("BROWSER");+  if (browser == NULL)+    browser = "xdg-open";++  /* TODO: Use better way to find default browser */

Probably

alexlarsson

comment created time in 24 days

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 operation_error (FlatpakTransaction            *transaction,   return TRUE; /* Continue */ } +static gboolean+webflow_start (FlatpakTransaction *transaction,+               const char         *remote,+               const char         *url,+               guint               id)+{+  FlatpakCliTransaction *self = FLATPAK_CLI_TRANSACTION (transaction);+  const char *browser;+  g_autoptr(GError) local_error = NULL;+  const char *args[3] = { NULL, url, NULL };++  if (!self->disable_interaction)+    {+      g_print (_("Authentication required for remote '%s'\n"), remote);+      if (!flatpak_yes_no_prompt (TRUE, _("Open browser?")))+        return FALSE;+    }++  browser = g_getenv ("BROWSER");+  if (browser == NULL)+    browser = "xdg-open";++  /* TODO: Use better way to find default browser */++  args[0] = browser;+  if (!g_spawn_async (NULL, (char **)args, NULL, G_SPAWN_SEARCH_PATH,+                      NULL, NULL, NULL, &local_error))+    {+      g_printerr ("Failed to spawn browser %s: %s\n", browser, local_error->message);+      return FALSE;+    }++  g_print ("Waiting for browser...\n");+

Its waiting for the browser to handle the url, and eventually get redirected to the local host uri telling the Authenticator the token. After that it will send the done signal and continue .

alexlarsson

comment created time in 24 days

pull request commentflatpak/flatpak

WIP: Work on authenticated downloads

I did some work on flat-manager to support the token-type data and require authentication: https://github.com/flatpak/flat-manager/pull/29

alexlarsson

comment created time in a month

pull request commentflatpak/flat-manager

Private repo handling

@dbnicholson https://github.com/flatpak/flat-manager/pull/29 is an alternative approach, can you have a look at that?

dbnicholson

comment created time in a month

pull request commentflatpak/flat-manager

Support requiring tokens for some refs

@dbnicholson Can you check this out?

alexlarsson

comment created time in a month

PR opened flatpak/flat-manager

Support requiring tokens for some refs

This is an alternative version of https://github.com/flatpak/flat-manager/pull/19/commits/c4ce11901e20984836020ce6801729a0c81d61c1 and builds on the flatpak work here: https://github.com/flatpak/flatpak/pull/3167

This assumes that some commits in the repo can have the xa.token-type set to a value != 0, and if so the flatpak authentication machinery will call into the authenticator api and can (optionally) return a bearer token that flatpak will pass when downloading the ref. The idea is that the authenticator can handle tokens differently. Some may be optional like donation requests, and trigger some nagging, but not actually send a token, whereas others do cause a token to be sent.

On the server side you can set the new require-auth-for-token-types option for the repo to a list of which token types require authentication. Additionally there is a default-token-type option that can easily make all refs private.

Whenever a commit object or a delta superblock is read we extract the token-type from it, and if it matches the required types we look at the token and ensure that the ref (which we also get from the commit object) matches the set of id prefixes in the token.

+144 -34

0 comment

4 changed files

pr created time in a month

create barnchflatpak/flat-manager

branch : tokens

created branch time in a month

push eventflatpak/flat-manager

Bartłomiej Piotrowski

commit sha 8e476d507f9ef176a32afb6081c36f917b0eb931

client: Make uploading delta files async

view details

Bartłomiej Piotrowski

commit sha 80bbcf78be3b70b62e5f66da8aea2f4a3e8efa71

client: Upload all deltas at once

view details

Bartłomiej Piotrowski

commit sha b25b225297778edaf5cbc637bdfc87e630e778ab

client: Move delta list out of the loop

view details

Bartłomiej Piotrowski

commit sha ac189fce1d008302f7c3854dc9f50216a19f65ca

client: Make upload_deltas return early if no deltas were found

view details

Bartłomiej Piotrowski

commit sha 4bddd7f626bd924fc6666130b6c59ae062050937

client: Remove NamedFilePart class

view details

Alexander Larsson

commit sha 65d821d54bbb4bff1779a57b53e07e455abbd9ab

client: Supply filename when uploading deltas

view details

Alexander Larsson

commit sha 19a4fc65ffc3e2f66c936bd59490e5dcb3a34a94

client: Don't quote filenames in multipart header when uploading

view details

push time in a month

PR merged flatpak/flat-manager

Async deltas
+29 -44

0 comment

2 changed files

alexlarsson

pr closed time in a month

PR opened flatpak/flat-manager

Async deltas
+29 -44

0 comment

2 changed files

pr created time in a month

create barnchflatpak/flat-manager

branch : async-deltas

created branch time in a month

push eventflatpak/flat-manager

Alexander Larsson

commit sha 2a448445363ac625d8f685332f1f9a7ca5583830

jobs: Properly json encode the results in case of errors In case of internal errors, we were getting: | Job failed: InternalError: Command "flatpak" "build-commit-from" "--force" "--no-update-summary" "--gpg-homedir=/vcs/gnome/flathub/gpg" "--gpg-sign=01DA958A7DC283BF1D3D208CF154092B3A7124D7" "--src-repo=build-repo/573" "repo" exited unsuccesfully: error: Writing content object: fchown: Operation not permitted | \ Job failed Unexpected JSONDecodeError exception in push: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "./flat-manager-client", line 720, in <module> result = loop.run_until_complete(args.func(args)) File "/usr/lib64/python3.7/asyncio/base_events.py", line 579, in run_until_complete return future.result() File "./flat-manager-client", line 558, in push_command reparse_job_results(publish_job) File "./flat-manager-client", line 301, in reparse_job_results job["results"] = json.loads(job.get("results", "{}")) File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) This is due to the results field in the jobresult just being the error string, while its supposed to be a stringified json object.

view details

push time in a month

push eventflatpak/flat-manager

Bartłomiej Piotrowski

commit sha 7583025fa3da26d0a9efccb05e509e217d14c2d2

client: Rework push command to use aiohttp for easy chunking

view details

Bartłomiej Piotrowski

commit sha d4957b02ccf70ae3d83151049cf57f0cd7b4686a

client: Timeout objects upload after 90 minutes

view details

Bartłomiej Piotrowski

commit sha 7e36c3807b998248c082003d5758b4707cd00a52

client: Pass response content to ApiError for async requests

view details

push time in a month

PR merged flatpak/flat-manager

Reviewers
client: Upload objects using aiohttp

Switch to aiohttp to workaround requests reading entire file sooner or later instead of chunking.

Delta part is in separate branch as I still haven't figured out why flat-manager rejects their filenames. This PR is already deployed to production so it makes sense to me to merge it to master.

+90 -43

1 comment

1 changed file

barthalion

pr closed time in a month

pull request commentflatpak/flat-manager

client: Upload objects using aiohttp

Seems ok. I'll have a look at the delta part.

barthalion

comment created time in a month

push eventflatpak/flat-manager

Alexander Larsson

commit sha eee1352bcd9adc0e22cc9c7633873241898f63f6

flat-manager-client: Avoid double reparse of result in publish If we're not waiting, the results would be reparsed twice causing an error like: Unexpected TypeError exception in push: the JSON object must be str, bytes or bytearray, not dict

view details

push time in a month

push eventflatpak/flat-manager

Alexander Larsson

commit sha b360ccdd7ca5b7884e82bf2ceb643a3f1bcc6217

Fix warnings from missing dyn This fixes a bunch of warnings like: warning: trait objects without an explicit `dyn` are deprecated By just adding the dyn as suggested.

view details

Alexander Larsson

commit sha f52f287529d202c9a01fe218f0aff5179becdc77

Stop using deprecated before_exec In favour of new pre_exec.

view details

Alexander Larsson

commit sha 7fd8e5b3217e097d9ce422c662fa611ad77a37bf

ostree: Use new pre_exec instead of deprectated before_exec

view details

Alexander Larsson

commit sha f87125eb972fa5b157309e189ba4512a5c8b5428

ostree: Check out of band for read frame offsets

view details

Alexander Larsson

commit sha a5b3232c458663d2605a5148e49f5e8da2e2bdd4

ostree: Don't include trailing 0 in parse_as_string()

view details

Alexander Larsson

commit sha b3d8d6236a6235356c990f6a7887c2aa4a7c534d

ostree: Fix alignment of commit metadata

view details

Alexander Larsson

commit sha 8cac75d4b69275fbc894fc324a6dace1e552ff75

ostree: Add toplevel Variant type that keeps the info for the SubVariants

view details

Alexander Larsson

commit sha 2a5735e1b48131508000092878a19ce9d147ccb9

ostree: No need to track overall offset in child variants The offset was only used for padding, and all containers are always aligned to the max needed for any child, so the overall offset will never affect padding.

view details

Alexander Larsson

commit sha 3e7746012f05c046b61b04caa8f71356dbdd1a00

ostree: Track (and verify) variant type strings

view details

Alexander Larsson

commit sha cef234487f2703afc1883bb6b6e42ea2d6fa6dba

ostree: Add SubVariant.copy() to make a standalone variant.

view details

Alexander Larsson

commit sha 62f7aa0861220e9953b0edc2f0092d18327b3375

ostree: Extract metadata from commit too

view details

Alexander Larsson

commit sha 88e155a8718c87c57b7df47c61ffc1c25a939b93

ostree: Expose some public type getters for Variant

view details

Alexander Larsson

commit sha 116ab42eafde147157dd913a5025cee74ed170ae

ostree: Fix warning

view details

Alexander Larsson

commit sha a03521bec5fc503194772f757baf057929508bf8

ostree: Add code to parse delta superblocks

view details

Alexander Larsson

commit sha fa4aa8bb1679ecfccdd54219a9a2eeeed3f1d7e3

Merge pull request #27 from flatpak/ostree Add various ostree helpers

view details

push time in a month

PR merged flatpak/flat-manager

Add various ostree helpers

This supercharges the code in ostree for parsing gvariants. I want to use this in https://github.com/flatpak/flat-manager/pull/19 rather than linking to glib.

+424 -88

0 comment

4 changed files

alexlarsson

pr closed time in a month

push eventflatpak/flat-manager

Alexander Larsson

commit sha 8952e55a76d7b500e0ea4c3b033902c0b5c0242d

README.md: Update eog example build docs

view details

push time in a month

pull request commentflatpak/flat-manager

Private repo handling

I'd like to avoid linking to glib. It just overall makes it easier to deploy the result. See the PR for some initial code for this.

dbnicholson

comment created time in a month

PR opened flatpak/flat-manager

Add various ostree helpers

This supercharges the code in ostree for parsing gvariants. I want to use this in https://github.com/flatpak/flat-manager/pull/19 rather than linking to glib.

+424 -88

0 comment

4 changed files

pr created time in a month

create barnchflatpak/flat-manager

branch : ostree

created branch time in a month

MemberEvent

pull request commentflatpak/flatpak

WIP: Work on authenticated downloads

@kalev @hughsie Hey, can you take a look at the API here that gnome-software would have to implement. In particular https://github.com/flatpak/flatpak/pull/3167/commits/8134dd4d0eea177840379cf18a61a61a7a33643b

Does that seem reasonable?

alexlarsson

comment created time in a month

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 23b7ed3e433d47560201844658f07ad9cec26cd4

tests: Fix installing org.flatpak.Authenticator.test.service.in

view details

Alexander Larsson

commit sha 546a7fb47ada363eb2e707759ed41d9bd1b561bc

Authenticator dbus api: Add Webflow signal When resolving the transactions we call RequestRefTokens as needed to get bearer tokens for some refs. These calls can also emit the Webflow signal on the request object with a url. It is then up to the client to show this url in some way. Once the required operations are done in the browser it will redirect to some url that will reach the authenticator, telling it that the operation is done and the final result of it. At that point the authenticator will emit the WebflowDone signal and continue. If the cliend doesn't want to do the web flow it can call the close operation on the request object.

view details

Alexander Larsson

commit sha 67ca3e43a12d5dda923fdb36aadf53585f287805

auth: Add helpers dbus helpers for sending webflow signals

view details

Alexander Larsson

commit sha 54d2b7211adcc3da928047c7ce15553a476b39f4

test-authenticator: Support requesting a web flow If request-webflow file exists, then the authenticator will listen to a local socket and start a webflow request with a uri pointing to it. If anything connects to the uri it will consider the flow ok and continue. If the client calls close() instead it will silently succeed anyway if require-webflow doesn't exists, and fail if is exists.

view details

Alexander Larsson

commit sha 8134dd4d0eea177840379cf18a61a61a7a33643b

FlatpakTransaction: Add webflow-start/done signals and abort_webflow() These signals are emitted when the authenticator needs some kind of web-based authentication. If the caller implements webflow-start and returns TRUE, then it needs to show the user the URL and allow the user to interact with it. Typically this ends with the web-page being redirected to a url to localhost or similar which tells the authenticator the result of the operations. This will cause the webflow-done signal to be emitted and the transaction operation to continue. If something goes wrong (or the signal is not handled) it will also report webflow-done, but then the transaction will fail in a normal way. Generally all users of FlatpakTransaction need to do is: On webflow-start, show a browser window with the url and return TRUE. On webflow-done, close the browser window if its still visible. If the user closes the browser window early, call flatpak_transaction_abort_webflow().

view details

Alexander Larsson

commit sha c2cb4acaacfb4f76cdb3c290812bf158c39a6ac1

cli transaction: Handle webflow signals This shows the url in the default browser. Currenly it just looks at $BROWSER and falls back to xdg-open.

view details

Alexander Larsson

commit sha f9497c31f97f880502f1385e3b99fe01df672dff

tests: Test the webflow handling We just verify that we can roundtrip via the cli by setting BROWSER=curl to "finish" the webflow.

view details

push time in a month

pull request commentalexlarsson/gthree

WiP: Initial GTK 4 support

Some things are still broken, but this seems useful

t-chaik

comment created time in a month

PR merged alexlarsson/gthree

WiP: Initial GTK 4 support

Plumbings for GTK 4 support including a naive attempt at building alibgthree-gtk4. Some examples are deactivated, some got uglified by #ifdef and none of them seems to be fully functional...

Early comments are very welcome!

+566 -671

1 comment

30 changed files

t-chaik

pr closed time in a month

push eventalexlarsson/gthree

Martin Blanchard

commit sha 230d385dcc776e1969bb6623dd5619d5e956485c

Add a configure option for libgthree-gtk3 Activated by default, deactivate using '-Dgtk3=false'.

view details

Martin Blanchard

commit sha 6dd72dba077d4b5b2185f3829e7b5daf5d6b11e7

Introduce a gtk4 dependency for libgthree-gtk4 Deactivated by default, activate using '-Dgtk4=true'.

view details

Alexander Larsson

commit sha eed432bd0a60e9d60d3e992a49497732dfd7d3c0

examples: Add examples_init () helper to clean up all examples

view details

Alexander Larsson

commit sha 9b32ceac3f42fc0f0d8a974e9bbe0eb5e4891c9f

Make cursor-using demos work on gtk4

view details

Alexander Larsson

commit sha ccad9d908f4cc115026df3ade1219317ad1bbfb1

examples: Add click event controller and us in multi example

view details

Alexander Larsson

commit sha edbe38ca41109289e93ed6b930bcbcfbccdfb9cc

examples: Make model example work on gtk4 For some reason the scroll event controller doesn't seem to work on gtk3, so keep the old event version for that in an ifdef.

view details

Alexander Larsson

commit sha bde7e8b792ecdc7a3111d8b4b39ee1a1a2a5c167

Merge pull request #70 from t-chaik/gtk4-port WiP: Initial GTK 4 support

view details

push time in a month

push eventt-chaik/gthree

Alexander Larsson

commit sha eed432bd0a60e9d60d3e992a49497732dfd7d3c0

examples: Add examples_init () helper to clean up all examples

view details

Alexander Larsson

commit sha 9b32ceac3f42fc0f0d8a974e9bbe0eb5e4891c9f

Make cursor-using demos work on gtk4

view details

Alexander Larsson

commit sha ccad9d908f4cc115026df3ade1219317ad1bbfb1

examples: Add click event controller and us in multi example

view details

Alexander Larsson

commit sha edbe38ca41109289e93ed6b930bcbcfbccdfb9cc

examples: Make model example work on gtk4 For some reason the scroll event controller doesn't seem to work on gtk3, so keep the old event version for that in an ifdef.

view details

push time in a month

pull request commentalexlarsson/gthree

WiP: Initial GTK 4 support

Hmm, i think we could clean up the examples a lot by sharing some setup code in helper functions.

t-chaik

comment created time in a month

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 73dbba74a4afca7db603106565a2a17639ab1482

tests: Add a test authenticator This is a trivial implementation of org.freedesktop.Flatpak.Authenticator that just reads the contents of the "required-token" file and returns that as the tokens for all refs.

view details

Alexander Larsson

commit sha 9a37e2169e08552a7fd29182a7fc5dd6fb748d40

add/modify-remote: Add options to set authenticator options

view details

Alexander Larsson

commit sha 47b6b0b1f24260225962dad872a5e221ea246872

transaction: Request required tokens from configured authenticator When we need a bearer token, look up the configured authenticator for the remote and ask it for tokens. Also updates the test-auth test with to use the new test authenticator instead of the previous env var hack.

view details

push time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 flatpak_repo_update (OstreeRepo   *repo,        /* Add bindings to the metadata. */       new_summary_commit_dict = g_variant_dict_new (new_summary);+      g_variant_dict_insert_value (new_summary_commit_dict, "xa.commits",+                                   g_variant_builder_end (&commits_builder));

Basically, from a practical point of view, its not going to be guaranteed that all the refs in a p2p peer were downloaded at the same time, but requiring them to match the ostree-meta from upstream requires that. And anyway, we can just verify the signatures and binding of the commit objects we download, so why would we need to verify the commits themselves against the ostree-metadata.

Now, obviously this is for the refs themselves, for the data in the ostree-meta commit we do need to know what commits it refers to, but the solution in this PR seems more efficient and clean for achieving that.

alexlarsson

comment created time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 flatpak_repo_update (OstreeRepo   *repo,        /* Add bindings to the metadata. */       new_summary_commit_dict = g_variant_dict_new (new_summary);+      g_variant_dict_insert_value (new_summary_commit_dict, "xa.commits",+                                   g_variant_builder_end (&commits_builder));

Yeah, i don't believe the change in https://github.com/ostreedev/ostree/pull/1946 is really useful/correct, and has commented so there.

alexlarsson

comment created time in a month

pull request commentostreedev/ostree

ostree-metadata commit API

The way I see it, you pull the ostree-metadata ref and the summary file from all of the p2p peers. That's how you reliably determine what subset of the remote each of them have. As I said in flatpak/flatpak#3167 (comment), since you can't verify the peer's summary file, the only source of verification it provides is the mirrored ostree-metadata file from the time when it mirrored from the remote. Having a copy of the remote's ref map in the ostree-metadata file allows you to verify that the commits in the peer are actually from the remote without fetching the commit objects from the peer.

However, I'm not the P2P expert.

Well, that is not how any ostree code is written today. The unique identified for the ref (i.e. the ref name + the collection id) can with the current repo layout only be stored once (as refs/mirrors/$collection_id/$ref), and it is pulled like any other ref, ie. from the highest priority available peer. Now, in theory one could pull it from all, but that is not how any code today works.

But, the summary isn't really necessary to verify the the ref, because the commit itself is also signed, and it includes a timestamp so that we can know if the update is a downgrade or not. So, I don't see the point of verifying it with the original summary.

The reason that flapak needs this information is because it stores info related to the refs inside the ostree-metadata itself (rather than in the commit itself), and it then needs to know what commit this data is about. As such, having the commit map in the summary could help (although the form this has uses much more space than the solution i did for flatpak only, as it duplicates every ref in the table, as well as using asciified commit ids instead of raw and including all refs rather than the required ones only.

dbnicholson

comment created time in a month

push eventalexlarsson/flatpak

Alexander Larsson

commit sha e049d3f8412eb44e08bad3b9d733a96b8c8567bd

transaction: Request required tokens from configured authenticator When we need a bearer token, look up the configured authenticator for the remote and ask it for tokens. Also updates the test-auth test with to use the new test authenticator instead of the previous env var hack.

view details

push time in a month

pull request commentostreedev/ostree

ostree-metadata commit API

I don't really understand the local ref map. Is this only about the flatpak xa.cache issue, or do you actually expect ostree to use this? If so, how?

In a situation where you're pulling N refs from M different p2p remotes you'll end up pulling the ostree-metadata ref from one of those M different remotes. And the rest of the N refs from some "random" subset depending on what refs each node has in their respective summary file. If the ostree-metadata you end up with is more recen" than some of the other peers, do you expect to fail pulling those refs? That seems like a bad idea to me.

dbnicholson

comment created time in a month

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 10e3a52109a1a1891d8cf19170cae17b0019cd91

Add initial version of 'org.freedesktop.Flatpak.Authenticator dbus API For now this just has a portal-like API for requesting tokens for a list of refs.

view details

Alexander Larsson

commit sha cd3689702bda9f798301525187cdf72073ee53f7

Add some helper functions for working with authenticators This will be used by later code in combination with the gdbus generated code.

view details

Alexander Larsson

commit sha 84636fe88566282e37f1629cf524fa5e5f712b05

tests: Add a test authenticator This is a trivial implementation of org.freedesktop.Flatpak.Authenticator that just reads the contents of the "required-token" file and returns that as the tokens for all refs.

view details

Alexander Larsson

commit sha aa9553bbea86207ed47cfa20d70eebc8b4505590

add/modify-remote: Add options to set authenticator options

view details

Alexander Larsson

commit sha adb244e8d6774e54b29211b08551b46aada7a83f

transaction: Request required tokens from configured authenticator When we need a bearer token, look up the configured authenticator for the remote and ask it for tokens. Also updates the test-auth test with to use the new test authenticator instead of the previous env var hack.

view details

push time in a month

issue commentflatpak/flatpak

Extension points used across runtime versions incorrectly pull in old runtime

Eh, only pull the runtime i mean.

TingPing

comment created time in a month

issue commentflatpak/flatpak

Extension points used across runtime versions incorrectly pull in old runtime

Hmm, this is a regression from https://github.com/flatpak/flatpak/pull/2954 Before that we never installed runtimes that an extension depends upon, because that is not generally needed, since you never run the extension directly.

I think we should change this code to only pull the extension if the extension uses extra-data.

TingPing

comment created time in a month

push eventalexlarsson/flatpak

Matthew Leeds

commit sha 1ebc94dc485f710c2b4dcbd12c3e1ddcb817145d

doc/flatpak-mask: Fix a couple typos

view details

Matthias Clasen

commit sha f1f83499b3858734264289a7bd27876e4c6212a9

Merge pull request #3174 from mwleeds/doc-mask-fix-typos doc/flatpak-mask: Fix a couple typos

view details

Bastien Nocera

commit sha 0873e5dd791b9b064cfff70b741d0f2ff8830297

docs: Fix typos

view details

Matthias Clasen

commit sha 0fd672cdc4cdc24d7fdf6ccb394c33cef8a598fc

Merge pull request #3179 from flatpak/wip/hadess/fix-typos docs: Fix typos

view details

Alexander Larsson

commit sha 1fc2de8950b3ab1a80837c32ed9d0685215ad2d5

libtest.sh: Add gdb_bt helper This is unused, but can be inserted in the tests if something is segfaulting so that you can see the backtrace.

view details

Alexander Larsson

commit sha 236c8bb78d3175a1aeeee13d3ca7e516df7e67a2

Add defines for existing summary sparse cache keys These are explicitly made short to save space, so lets have defines for them to make sure we don't mistype them, especially as we will be adding new keys.

view details

Alexander Larsson

commit sha 215e32cbd7fa76a720933bbf3ac4d706ae065ec9

Print errors on stderr in install/update If we're not doing fancy ui, print errors on stderr. This was biting me in some tests where I'd like to grep for some strings in the stderr output.

view details

Alexander Larsson

commit sha 6ba483afa82a70a83086376266c028da67eaa76b

dir: Split up the p2p resolve code into two phases Historically the p2p resolve code always did a parallel call to find all the available commits for the refs, and then it took the results and pulled only the commits for all the refs so that it could resolve against the exact commits that were available (which might not match with whatever metadata we have in the local ostree-metadata copy. This splits this into two phases, the first that uses the summary only, and a second one that pulls the commit. The reason for this is that we want to be able to do some stuff inbetween these, such as resolving some refs via the ostree-metadata and maybe requesting bearer tokens that we need for pulling the commit objects.

view details

Alexander Larsson

commit sha 6bbe5158e3a827550c78c708918a328e8509bbc6

prepare_resolve_p2p() return last_remote_commit We want this in the transaction code, to see what commit would be pulled, and thus if the data in the ostree-metadata is good enough

view details

Alexander Larsson

commit sha ebd7e36a647bcbbe83bc260b2c4ff69b7a0f66cb

p2p: Generate xa.commits array in ostree-metadata This is in the same order as the xa.cache array and contains the id of the commit that the cached data is about. This is not necessary in the non-p2p summary metadata, because in that we always have a matching ref -> commit array. However, in the p2p case this information can be useful.

view details

Alexander Larsson

commit sha 654c4d8cdaf1e895e8f1e5c4bc835648340a8b77

Optionally return commit id in flatpak_remote_state_lookup_cache This will only work if xa.commits is in the metadata, which is only available in the p2p case and was only added recently.

view details

Alexander Larsson

commit sha 3e649d643bf24496fc9c8853db3254fa42b97c81

transaction: Break out some shared code. This moves some code into helper functions so it can be easily shared. No behavioural change.

view details

Alexander Larsson

commit sha 8b6a39b1019519679848eb2d5b71f421233419f2

Add support for bearer tokens to flatpak_dir_install/update Anything passed in here will be added as a bearer token for all http requests in the operation.

view details

Alexander Larsson

commit sha a470196edcb0bb65933add34c5909c8d5652e05a

Add token-type per-commit metadata key and export it to summary We store this in the sparse cache, because we don't expect it to be set for everything.

view details

Alexander Larsson

commit sha e5606e98d3bec002f1f353c2cfc58c026b4e1f37

Transaction: non-p2p support for tokens We extract the need-token key from the summary and if set we calculate a token to use for the operation, which we then pass to install/update. For now the actual token just comes from the FLATPAK_TEST_TOKEN environment var. The details of this will be fleshed out later. Additionally, this does not support the p2p case, because there we need the token in order to request the commit during the resolve. This will also be added later.

view details

Alexander Larsson

commit sha 2acfbdd1726f8ed4ce742d77e8ea21376f8b4228

tests: Add initial tests for token-needing installs

view details

Alexander Larsson

commit sha c396d97ad9f38ee60d5fb6fbd669e9b0fe57e838

p2p resolve: Resolve the token_type data from the commit

view details

Alexander Larsson

commit sha ede7abd9c79b439efa09efea161ea91a7457179b

p2p: Add flatpak_dir_resolve_maybe_resolve_from_metadata This tries to resolve the p2p resolve operation from the info in a ostree-metadata commit. This only works if the resolve ended up on the same commit id as what was available in the ostree-metadata which may not be correct if the two are not synchronized.

view details

Alexander Larsson

commit sha 90a649036c2e773a1a154e27d67ec88ebe169ea6

p2p: When updating/installing, resolve via ostree-metadata if possible If the commit is available in the ostree-metadata and it matches what the latest available commit in the p2p results then resolve it to that, so we don't have to download the commit object.

view details

Alexander Larsson

commit sha 58664acdb9d6f798b798428e96021dda1ca69221

p2p resolve: Support tokens in flatpak_dir_finish_resolve_p2p_refs Also, since the lower level APIs don't allow you to pass different tokens for different parts change this function to support passing a subset of the resolves, so that we can pass all that need a specific token in one go, and then call this multiple times. The way we handle this is by saving all the original ref_to_checksum hashtables for all results and then re-create them with the subset of refs needed when pulling.

view details

push time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 flatpak_repo_update (OstreeRepo   *repo,        /* Add bindings to the metadata. */       new_summary_commit_dict = g_variant_dict_new (new_summary);+      g_variant_dict_insert_value (new_summary_commit_dict, "xa.commits",+                                   g_variant_builder_end (&commits_builder));

That means you have to pull each commit object for verification in the p2p case, which might require authorization for purchased refs. That means that the refs can't be reliably resolved unless the user is authenticated, which is what @alexlarsson is trying to avoid.

Do they have to be reliably resolved until you’re actually ready to go through the authenticator?

The commit object contains the metadata for the app, which says which runtime we need to download (this is what resolve means in the scope of a flatpak transaction). If the ostree-metadata is mismatched then the resolve would add a download operation for a runtime that will not match what will actually be installed.

I'm not sure if this answers your question, because I'm not sure what you mean. But we definitely need to actually download the commit to get the right deps, and for that we need the token. We don't necessarily have to have the most up-to-date info to just know that we need to get a token though, which is what the iterative workaround uses.

Do you have any objections if the collection map is included in ostree-metadata?

Yes, some mild ones: (a) it increases the size of ostree-metadata and means it has to be updated much more frequently than it currently is; (b) the collection map still needs to be in the summary file too, because otherwise it’s not possible to resolve ostree-metadata itself. Having it in two places, or parts of it in two places, is ugly and confusing.

What this commit adds is the commit id of the cached ref flatpak metadata in the ostree-metadata ref, which may be different from the commit id of the actual ref.

Sorry, I don’t understand. What do you mean by ‘cached ref’ and ‘actual ref’?

So, both these questions sound confusing. Lemme back up a bit.

Both the master and the p2p repo have a summary file. It contains a mapping from ref name to the commit name that is available (as HEAD) in each individual repo.

Additionally, the summary in the master repo contains a cache (xa.cache) for information about flatpak apps in the repo that is used for resolving. For example, its used when downloading an app to know which runtime to download. Additionally it has some "nice to have" data like installed and download size. These caches don't actually exists for all refs, just flatpak ones. xa.cache doesn't contain the commit id of the version we extracted the data from, because being inside the summary we can assume the that the cache and the regular ref->commit maps are synced, so this data is already available.

However, in the p2p case each peer generates the summary file, so it is not trustworthy for critical information (i.e. not signed). So the master also makes a copy of xa.cache inside the commit of the ostree-metadata ref, which is signed, and this is mirrored as-is to the p2p nodes. However, at this point we disconnected the cached metadata from the summary so we can't rely on it to know exactly what commit each cached metadata came from. In order to fix this the PR adds an array that matches the order of xa.cache that just contains the commit that the cached data refers to.

Now to answer: (a) yes, it slightly increases the size. Compared to the size of the rest of the cached data already in it this is pretty insignificant thought. (b) The collection map (ref->commit id) is not quite the same as the "what commit did we cache metadata from" map, and will not be used during an ostree pull. It also would never include ostree-metadata or any non-flatpak refs.

cached ref: The version of the ref that we extracted data from and put in the xa.cache. actual ref: The version of the ref that is currently available in the p2p mirror.

In a full mirror, or on the master these would be the same, but in a partial mirror where the ostree-metadata branch and some other ref were mirrored at different times these would not be the same.

alexlarsson

comment created time in a month

push eventalexlarsson/flatpak

Alexander Larsson

commit sha 785d4469189260c75138b140bad7ee49f2239956

libtest.sh: Add gdb_bt helper This is unused, but can be inserted in the tests if something is segfaulting so that you can see the backtrace.

view details

Alexander Larsson

commit sha d0338801908d015476e036eb1067308136feb6ee

Add defines for existing summary sparse cache keys These are explicitly made short to save space, so lets have defines for them to make sure we don't mistype them, especially as we will be adding new keys.

view details

Alexander Larsson

commit sha f78e833d7452c64b4fcdc1b473579add965dc5ff

Print errors on stderr in install/update If we're not doing fancy ui, print errors on stderr. This was biting me in some tests where I'd like to grep for some strings in the stderr output.

view details

Alexander Larsson

commit sha dd236c80fbb3dfa955f98f6d76fc9b51138befd8

dir: Split up the p2p resolve code into two phases Historically the p2p resolve code always did a parallel call to find all the available commits for the refs, and then it took the results and pulled only the commits for all the refs so that it could resolve against the exact commits that were available (which might not match with whatever metadata we have in the local ostree-metadata copy. This splits this into two phases, the first that uses the summary only, and a second one that pulls the commit. The reason for this is that we want to be able to do some stuff inbetween these, such as resolving some refs via the ostree-metadata and maybe requesting bearer tokens that we need for pulling the commit objects.

view details

Alexander Larsson

commit sha 77bbded786a74fc7e85682468e6f34566f460cf4

prepare_resolve_p2p() return last_remote_commit We want this in the transaction code, to see what commit would be pulled, and thus if the data in the ostree-metadata is good enough

view details

Alexander Larsson

commit sha 2da1f88bc6441762258f0788e8c71ec8dc9ac600

p2p: Generate xa.commits array in ostree-metadata This is in the same order as the xa.cache array and contains the id of the commit that the cached data is about. This is not necessary in the non-p2p summary metadata, because in that we always have a matching ref -> commit array. However, in the p2p case this information can be useful.

view details

Alexander Larsson

commit sha 75cec7445ae9f45da810c6ee54dd25eda554973a

Optionally return commit id in flatpak_remote_state_lookup_cache This will only work if xa.commits is in the metadata, which is only available in the p2p case and was only added recently.

view details

Alexander Larsson

commit sha d2b50cbedde3b57550737da55a064434a1af27e0

transaction: Break out some shared code. This moves some code into helper functions so it can be easily shared. No behavioural change.

view details

Alexander Larsson

commit sha 66add38f51a4371ae3c6ca728137f9b8a78eee88

Add support for bearer tokens to flatpak_dir_install/update Anything passed in here will be added as a bearer token for all http requests in the operation.

view details

Alexander Larsson

commit sha b4bf55a0f2699583dfcfdfd895dc368469892fa5

Add token-type per-commit metadata key and export it to summary We store this in the sparse cache, because we don't expect it to be set for everything.

view details

Alexander Larsson

commit sha 25e6596b09dae759cab04bb9297d95ffa7edd1ee

Transaction: non-p2p support for tokens We extract the need-token key from the summary and if set we calculate a token to use for the operation, which we then pass to install/update. For now the actual token just comes from the FLATPAK_TEST_TOKEN environment var. The details of this will be fleshed out later. Additionally, this does not support the p2p case, because there we need the token in order to request the commit during the resolve. This will also be added later.

view details

Alexander Larsson

commit sha 8f9651ae46d889ec8c969c0ba259d1a660761101

tests: Add initial tests for token-needing installs

view details

Alexander Larsson

commit sha 26ade3d0b4765e9a1bfae42666345f416d4767a5

p2p resolve: Resolve the token_type data from the commit

view details

Alexander Larsson

commit sha 71bd95fdd2b28c23235b9d913101c871a6f9d7f9

p2p: Add flatpak_dir_resolve_maybe_resolve_from_metadata This tries to resolve the p2p resolve operation from the info in a ostree-metadata commit. This only works if the resolve ended up on the same commit id as what was available in the ostree-metadata which may not be correct if the two are not synchronized.

view details

Alexander Larsson

commit sha 338f719c139a4daecc3df7692bbd81fd18763116

p2p: When updating/installing, resolve via ostree-metadata if possible If the commit is available in the ostree-metadata and it matches what the latest available commit in the p2p results then resolve it to that, so we don't have to download the commit object.

view details

Alexander Larsson

commit sha 34c36a7fdf3fef870dd37c101c18c997cef2cd5b

p2p resolve: Support tokens in flatpak_dir_finish_resolve_p2p_refs Also, since the lower level APIs don't allow you to pass different tokens for different parts change this function to support passing a subset of the resolves, so that we can pass all that need a specific token in one go, and then call this multiple times. The way we handle this is by saving all the original ref_to_checksum hashtables for all results and then re-create them with the subset of refs needed when pulling.

view details

Alexander Larsson

commit sha 12c9c045f0b848c64c9667bb2684aa5663bf707b

Transaction: Support tokens for all p2p pulls The p2p case is kinda weird wrt tokens. We can do most of the basics, like which refs need updating using the partial summary from the p2p mirrors, but we can't rely 100% on the ostree-metadata info for core info like permissions or dependencies, since it may be out-of-sync. So, if the information in the ostree-metadata doesn't match the commit we're resolving, the p2p resolve code actually pulls the actual commit objects as part of a resolve. Now, the commit objects are protected by bearer tokens, so we need to pass them while doing this pull. Unfortunately the information about which refs requires tokens are part of the commit, which is a circular dependency. We resolve this by relying on the (possibly stale, but probably ok) copy of the need-token info in the ostree-repo metadata. So, we do the first part of the p2p resolve, then for all the not-yet-resolved ops (i.e. ones that actually need updates) we look in the ostree-metadata for which refs need tokens, generate tokens and then do the pulling with the tokens. This is an iterative process, because resolving a ref can create more update operations, which may need more tokens.

view details

Alexander Larsson

commit sha d67a997ff8628d661bbdc1a8a4eabf5766672623

tests: Add p2p token tests Unfortunately we lose some error information when we pull multiple refs, ending with a generic "something failed" error rather than the 401 error so in the p2p case we can't verify that we get the right errors.

view details

push time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 static GOptionEntry options[] = {   { "gpg-homedir", 0, 0, G_OPTION_ARG_STRING, &opt_gpg_homedir, N_("GPG Homedir to use when looking for keyrings"), N_("HOMEDIR") },   { "end-of-life", 0, 0, G_OPTION_ARG_STRING, &opt_endoflife, N_("Mark build as end-of-life"), N_("REASON") },   { "end-of-life-rebase", 0, 0, G_OPTION_ARG_STRING, &opt_endoflife_rebase, N_("Mark build as end-of-life, to be replaced with the given ID"), N_("ID") },+  { "need-token", 0, 0, G_OPTION_ARG_INT, &opt_need_token, N_("Set value of need token"), N_("VAL") },

I actually think the default will be dependent on how the remote is set up, this is just a way to override the default for this particular ref.

alexlarsson

comment created time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 resolve_all_ops (FlatpakTransaction *self,   return TRUE; } ++static gboolean+request_tokens_for_remote (FlatpakTransaction *self,+                           const char         *remote,+                           GList              *ops,+                           GCancellable       *cancellable,+                           GError            **error)+{+  g_autoptr(GString) refs_as_str = g_string_new ("");+  GList *l;+  const char *token;++  for (l = ops; l != NULL; l = l->next)+    {+      FlatpakTransactionOperation *op = l->data;++      if (l != ops)+        g_string_append (refs_as_str, ", ");+      g_string_append (refs_as_str, op->ref);+    }++  g_debug ("Requesting tokens for remote %s, refs: %s", remote, refs_as_str->str);++  token = g_getenv ("FLATPAK_TEST_TOKEN");

I expect to rewrite this branch a fair bit before landing, but sure, that would be nice.

alexlarsson

comment created time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 flatpak_builtin_build_commit_from (int argc, char **argv, GCancellable *cancella               strcmp (key, OSTREE_COMMIT_META_KEY_ENDOFLIFE_REBASE) == 0)             continue; +          if (opt_need_token >= 0 && strcmp (key, "xa.need-token") == 0)

I guess we could use int32 in the file too, and allow an explicit -1 to mean use the default. Might make things easier.

alexlarsson

comment created time in a month

Pull request review commentflatpak/flatpak

WIP: Work on authenticated downloads

 flatpak_builtin_build_commit_from (int argc, char **argv, GCancellable *cancella               strcmp (key, OSTREE_COMMIT_META_KEY_ENDOFLIFE_REBASE) == 0)             continue; +          if (opt_need_token >= 0 && strcmp (key, "xa.need-token") == 0)

Yeah, token-type seem better. However, the domain of the metadata is guint32, it just is currently using a signed int in the tools that set it so that i can use < 0 to mean "don't set", because I want to be able to signal unset to mean "whatever is default for this repo" which we then can explicitly override with "0" if we want the default to be require token with some refs not requiring it.

alexlarsson

comment created time in a month

more