profile
viewpoint

josch/cycles_hawick_james 14

Finding all the circuits of a directed graph with self-arcs and multiple-arcs by K.A. Hawick and H.A. James

josch/cycles_johnson_meyer 14

find all circuits of a directed graph using johnson's algorithm and java implementation by frank meyer

josch/3dpcp 13

jacobs university fall 2012 3d point cloud processing homework

josch/cycles_tarjan 8

find all circuits of a directed graph using tarjan's algorithm

josch/cycles_johnson_abate 3

find all circuits of a directed graph using johnson's algorithm and ocaml implementation by pietro abate

josch/cycle_test 1

checking cycle enumeration implementations for correctness

josch/debarchwildcardtest 1

test debian architecture wildcard matching

josch/debian-n900 1

running debian on the nokia n900

pull request commentpython-pillow/Pillow

WIP: only convert GIF frames if palette actually changes

What is img2pdf actually doing with the multiple frames of gif images?

For images that contain more than one frame (so not only for GIF images) img2pdf outputs one page per frame unless the --first-frame-only option is given.

Thus, I very much welcome the fixes to Pillow concerning the automatic conversion to RGB if 256 colors are not enough to represent a frame. I think that's awesome! I just want to retain the palette in those cases were it does not change because storing an RGB image (even after paeth filter and zlib compression) in PDF requires more space than the palette based image.

Sorry, that I haven't pushed more fixes to this branch yet. I'll find more time to work on this during the weekend.

josch

comment created time in an hour

pull request commentpython-pillow/Pillow

WIP: src/PIL/GifImagePlugin.py: only convert image if palettes actually changes

software like img2pdf does not need to add special-casing to be compatible with both Pillow 9.0.0 and Pillow before 9.0.0

Could you provide some more detail on this point? If I'm understanding correctly, the img2pdf tests are failing just because they expect an indexed colorspace and are instead finding RGB. That doesn't sound like a problem for end users, just a problem for the test suite.

Yes, this is a minor argument and only becomes the larger argument if there is more software or users having similar expectations. The bigger argument is, that the files produced by img2pdf are now bigger because they are not palette images anymore even though they could be palette images because all frames share the same palette.

The larger file size and the additional processing cost (due to converting) are my main arguments.

#5966 (comment)

what can software do that wants to really figure out whether frame X of an image is P or RGB?

At the moment in the main branch, if you seek past the first frame of a GIF, it will be either RGB or RGBA.

My primary concern with this PR is that it would no longer be predictable what mode the image would be in that scenario. It might have the same palette and be P, it might not and be RGB. Surely that sounds like it would make situations like the one happening in your test suite worse, not better.

But it would be predictable. If the input GIF has the same palette for all frames, then al frames would be P. I'd just add different test cases for the different situations.

Really, the img2pdf testsuite is of little concern here. If subsequent images have the same palette as all the frames before, then that's an advantage for filesize (because we can retain the palette when converting) and for processing speed (because frames don't have to be converted from P to RGB).

josch

comment created time in 2 days

pull request commentpython-pillow/Pillow

WIP: src/PIL/GifImagePlugin.py: only convert image if palettes actually changes

Some dispose-related tests are still failing and I'm working on that (suggestions welcome!).

The advantages of only changing to the new behaviour if needed (i.e. if the palette does change from one frame to the next) are:

  • faster because we have to run convert less often
  • on all my test input the palette images were smaller than the images after they were converted to RGB(A)
  • software like img2pdf does not need to add special-casing to be compatible with both Pillow 9.0.0 and Pillow before 9.0.0. Other pieces of software might be in a similar situation.
josch

comment created time in 3 days

push eventjosch/Pillow

Johannes Schauer Marin Rodrigues

commit sha e576c3df2ae9cf9b5f468b2025a677521b574b4e

src/PIL/GifImagePlugin.py: only convert image if palettes actually change

view details

push time in 3 days

push eventjosch/Pillow

Johannes Schauer Marin Rodrigues

commit sha 1fbaf9cffa215d1681fb2125670762c58df944c1

src/PIL/GifImagePlugin.py: only convert image if palettes actually change

view details

push time in 3 days

push eventjosch/Pillow

Johannes Schauer Marin Rodrigues

commit sha a92dfc395ee5a5ead3c5a220e039c3eeed1f3f9c

src/PIL/GifImagePlugin.py: only convert image if palettes actually change

view details

push time in 3 days

PR opened python-pillow/Pillow

WIP: src/PIL/GifImagePlugin.py: only convert image if palettes actually changes

This is a WIP pull request which avoids converting image frame of GIF images in case all earlier palettes of the GIF are actually the same.

I wrote this because the suggestions by @radarhere in https://github.com/python-pillow/Pillow/issues/5966#issuecomment-1015155331 doesn't work for some images which already get converted in the _seek function. Thus, this pull request introduces a new variable that keeps track of the last used palette and only converts images if any of the former frames changed the palette (including the global palette).

What do you think?

+30 -15

0 comment

1 changed file

pr created time in 3 days

create barnchjosch/Pillow

branch : gif_same_palette

created branch time in 3 days

fork josch/Pillow

The friendly PIL fork (Python Imaging Library)

https://python-pillow.org

fork in 3 days

issue commentpython-pillow/Pillow

[9.0.0 Regression] Pillow claims GIF frame to be mode RGB instead of P

That's awesome, thank you!

Since this behaviour is intended. I'll adjust img2pdf accordingly and I guess you can close this issue. Thank you for taking the time and answering all my questions so quickly. :)

doko42

comment created time in 3 days

issue commentpython-pillow/Pillow

[9.0.0 Regression] Pillow claims GIF frame to be mode RGB instead of P

Okay, then I guess the simple answer is just that there is currently no way to get the second frame of the GIF without it being composited onto the first frame?

doko42

comment created time in 3 days

issue commentpython-pillow/Pillow

[9.0.0 Regression] Pillow claims GIF frame to be mode RGB instead of P

Yes, but I can I obtain the original image data somehow or is that information lost now?

doko42

comment created time in 3 days

issue commentpython-pillow/Pillow

[9.0.0 Regression] Pillow claims GIF frame to be mode RGB instead of P

Hi, I'm the author of img2pdf and version 9.0.0 breaks the tests due to the new handling of GIF images.

If this change is here to stay, what can software do that wants to really figure out whether frame X of an image is P or RGB?

doko42

comment created time in 3 days

push eventjosch/img2pdf

Johannes Schauer Marin Rodrigues

commit sha 6a55258804c2d74802b5746d071a8e1d8c11734f

appveyor.yml: rename pil to Pillow

view details

push time in 5 days

push eventjosch/img2pdf

Johannes Schauer Marin Rodrigues

commit sha 3cdeab08ab72a5d546b1a75071236f9649b9bf0b

appveyor.yml: also install pil so that maybe pyinstaller picks it up

view details

push time in 5 days

push eventjosch/img2pdf

Johannes Schauer Marin Rodrigues

commit sha cea7c9120b0d1035b3a38c90f80bd7ec3849b26f

tox.ini: python 3.5 and 3.6 are not supported anymore

view details

push time in 5 days

pull request commentany1/wayvnc

Debianization of wayvnc

I already packaged vnc and uploaded it to NEW.

But if you want to help maintain the package I'd be happy to sponsor your work. Create yourself an account on salsa and once the next wayvnc version is released, package it up and contact me at josch@debian.org and I'll review your packaging and upload it into unstable.

Boyuan Yang might also appreciate help with aml and neatvnc. You can contact Boyuan Yang via email at byang@debian.org.

Thanks!

jauntywunderkind

comment created time in 16 days

pull request commentany1/aml

debianization of libaml0, libaml-dev

aml has already been packaged by Boyuan Yang. You can find the packaging here: https://salsa.debian.org/debian/aml

The package is already in Debian testing and unstable: https://tracker.debian.org/pkg/aml

jauntywunderkind

comment created time in 16 days

pull request commentany1/neatvnc

debianization of libneatvnc0

neatvnc has already been packaged by Boyuan Yang. You can find the packaging here: https://salsa.debian.org/debian/neatvnc

The package was also already uploaded to NEW: https://ftp-master.debian.org/new/neatvnc_0.4.0+dfsg-1.html

jauntywunderkind

comment created time in 16 days

pull request commentbestouff/genext2fs

Enable system locale - change from the standard (C) to system locale.

Without it, handling a tarball in pax format containing utf8 characters in its filenames will result in this error message:

archive_read_next_header(): Pathname can't be converted from UTF-8 to current locale.
josch

comment created time in a month

PR opened bestouff/genext2fs

Enable system locale - change from the standard (C) to system locale.

This allows libarchive (in case it is activated) to handle filenames. We only change LC_CTYPE since libarchive only needs the charset set. We don't use LC_ALL because it causes problems on some systems. We restore the original LC_CTYPE after extraction to avoid side effects. We use uselocale instead of setlocale to avoid setting LC_CTYPE globally. See on libarchive Website for a more complete description of the issue: https://github.com/libarchive/libarchive/issues/587 https://github.com/libarchive/libarchive/wiki/Filenames https://github.com/sbabic/swupdate/blob/master/handlers/archive_handler.c#L94

+19 -0

0 comment

1 changed file

pr created time in a month

create barnchjosch/genext2fs

branch : utf8tar

created branch time in a month

created tagjosch/genext2fs

tagv1.5.0

genext2fs - ext2 filesystem generator for embedded systems

created time in a month

pull request commentmapbox/carto

lib/carto/mml.js: FIX: Error: Function yaml.safeLoad is removed in js…

This PR, as it currently stands, would be using the unsafe load method from js-yaml 3.x . We would need to make the change from safeLoad to load at the same time as upgrading the package.json file to use the 4.x series.

Yes. I'll leave it up to you to make the switch. I'm just forwarding a patch that I apply to the Debian package of carto because in Debian we can only ship one version of each package, so we have carto use the latest version of js-yaml which is 4.x.

josch

comment created time in a month

issue commentlkhq/debspawn

Comparison between debspawn and sbuild

Hi Matthias, thanks a lot for your very extensive and detailed reply!

Hi @josch ! First of all, the README is seriously outdated, there's stuff in there that is no longer true about debspawn, let alone sbuild. So, this likely needs an update (although I am actually inclined to remove everything that contrasts debspawn with other tools except for intentional differences to sbuild).

Even if it's not in the README, I think that listing what makes debspawn different/better than alternatives would be useful to have stated somewhere.

  • I just wondered how easy it would be to build a tool like this, given that systemd-nspawn does some of the more complicated stuff already. So I just built a prototype after a "it will just take me a few hours" bet with friends at Debconf ;-)

Ah, reminds me of myself when I wrote mmdebstrap 3 years ago. XD

  • I'm working on a tool called Laniakea which is a suite of tools to build Debian derivatives. It has a generic job runner called laniakea-spark (previously we used Paul Tagliamonte's debile) that was using sbuild back in the day. Setting up sbuild was always a bit annoying though, as it needed a bunch of configuration.

Yes. Today, there is the sbuild-debian-developer-setup package which hopefully makes the standard sbuild+schroot setup a bit easier.

We also had to parse the whole changelog again to extract whether builds failed reliably, and also why they failed: https://github.com/lkhq/laniakea-spark/blob/d0529862f48692483838c2e72bb4e96673fc45f7/spark/runners/sbuild.py#L138 - I thought that I could maybe simplify this by creating a tool that worked better for Laniakea

Yes, the log parsing is still an issue. There is still no machine readable way to identify reliably why a build failed without using shaky log parsing.

  • Back then at Purism we had people who never had touched Debian or built a Debian package, and setting up sbuild was still a bit more complex. With debspawn it's one command to create and image, then yet another one to build, and the environment is exactly the same as on the autobuilders, so reproducing failures is pretty easy.

Agreed. Sbuild is far from being that easy to use. I should write maybe another set of wrapper scripts to make default usage as easy as using a single command.

  • We were to run some semi-trusted builds (nothing dangerous, but something that could cause issues on accident), and systemd-nspawn being a container solution allows some neat features like limiting available memory or CPU resources to the build.

Ah, that's where the root requirement of systemd-nspawn comes from, I guess.

  • Initially I thought I could use OverlayFS for some extremely fast build setups - that did not work out though (it could be done with some modifications, but I didn't want to make those at the time as tarball decompression turned out to be rather quick).

Yes. While sbuild (and schroot) also supports overlayfs mounting, for my personal use I also resort to just unpacking chroot tarballs to a tmpfs as I found it to be sufficiently fast.

  • I wanted to integrate some Debian package caching, so we wouldn't also need to setup apt-cacher-ng on the builder machines, but have one tool handle all stuff transparently in the background.

Understandable. I still struggle with apt-cacher-ng as it seems to somehow fill up the package cache without ever cleaning up and dropping cached packages after some time...

For example the first paragraph claims that sbuild uses a plain chroot. True it can do that. But it can also use any environment supported by autopkgtest (like lxc, lxd, qemu, schroot, ssh...) as well as the unshare backend which uses linux user namespaces to do package builds.

Could it do that 4 years ago already? If so, then I must have overlooked it. I know sbuild can do these things today, and even build via QEMU, so today that statement is definitely wrong nowadays.

The autopkgtest backend was introduced in sbuild 0.69.0, released in May 2016. The unshare backend was introduced in sbuild 0.77, released in July 2018. In any case, both backends were never advertised anywhere and I haven't yet received a single bug report for those two backends, so either nobody is using those or they are bug free (very unlikely).

What advantage does systemd-nspawn have over [unshare]?

I don't know how user namespaces are implemented exactly in sbuild - is unshare a chroot with user namespaces on top? Systemd-nspawn creates a pretty neat lightweight container that virtualizes the file system hierarchy, process tree, IPC subsystems, the host and domain name and also has a system callback filter to filter out some potentially problematic syscalls (which actually can be an issue if packages need these for tests, so debspawn weakens that by default). So, systemd-nspawn gives a bit more isolation and is a nice pretty lightweight way if you want a container but don't need the complexity and dependencies of podman/docker & Co.

The ushare backend uses linux user namespaces to unshare the mount, UTS, IPC, network, PID and user namespaces. I think that's also what other container solutions like docker, lxc and systemd-nspawn do. The unshare backend cannot limit resources like CPU or RAM and it cannot limit syscalls -- all these features would require being root at some point and I wanted to avoid that. I see that it might make sense to add a systemd-nspawn backend to make these features available to sbuild users.

I'm tempted to implement a systemd-nspawn backend and/or a debspawn backend into sbuild but it seems that systemd-nspawn requires super user privileges while the sbuild unshare backend does not (not for the setup, nor for image creation and neither for running the build).

Systemd-nspawn can actually run without the need for superuser privileges if you tell it to use user namespaces - and I think debspawn should actually do that eventually. When playing with it a while back, I remember I ran into issues with it though. I can't recall what the actual problem was though which was a dealbreaker back then, which is a bit annoying (debspawn's method for dropping privileges inside of the container is also complicated, and user namespaces would have solved this). Maybe I should just try this again ;-) The user namespace support for nspawn is actually quite extensive, see https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html#User%20Namespacing%20Options for reference.

I haven't been able to make it work though. When I try to run systemd-nspawn with arguments like --private-users=pick --private-users-ownership=auto I still get the message Need to be root.. I'd welcome any pointers that allow running systemd-nspawn without being root first.

Though to get features like RAM and CPU limits, root would be definitely needed. How does debspawn handle this? Does the user need to run sudo debspawn or is there a setuid root binary somewhere?

Maybe you can point out in the README why a user would prefer to use debspawn over the sbuild unshare backend.

I am kind of tempted to drop comparisons to other tools, as they are likely not useful and get outdated sooner or later...

As the author of one of these tools, I actually find these comparisons useful.

You state that a difference between sbuild and debspawn is unicode handling. Could you point me to the debspawn code that implements this so that I can understand what this paragraph means? Maybe this functionality should be included into sbuild.

Oh, this was actually a huge debate years ago! Maybe https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873919 can give some insight. Debian's default locale is "C", and therefore some stuff, especially Python code occasionally, falls flat when parsing unicode (file)names. Dpkg doesn't want to have an opinion on this. My opinion was that UTF-8 should be default (also to test things like building in a directory consisting of random emojis to see what fails - back then I was working on some QA to make sure software was handling unicode correctly). This is implemented in debspawn by it setting LANG=C.UTF-8 in the build environment and "drawing" to the console using unicode characters to make readable build logs, unless this feature is disabled. IMHO this is a better default, but that is obviously a personal opinion. tbh, I also don't know if anything has changed in the unicode-as-default department since 2016/2017. I actually thought sbuild had a unicode option as well already, but maybe I imagined that.

Using C.UTF-8 has been the default in sbuild since 0.78.0 (released Jan 2019).

You also state that sbuild works on OSes that are not Linux. I'd be interested to know which ones you are talking about. I have not heard of people using sbuild outside of Linux.

I am not sure what past-me was thinking about there exactly, but I'm pretty sure it was Debian/kFreeBSD and Debian/Hurd. For the former, I definitely know people who use sbuild on it.

Ah okay, I misunderstood and was thinking of Linux as in GNU/Linux but yes, you are right, the Debian ports buildds for hurd and kfreebsd definitely run sbuild.

Then you state that debspawn is faster due to zstd tarballs and eatmydata. sbuild supports that as well plus lz4 tarballs which are again a tiny bit faster. Can you provide benchmarks to back up your claim?

Back then I actually benchmarked this, but LZ4 would obviously beat zstd in most settings. The eatmydata stuff may be misleading, what I mean is that with sbuild you need (needed?) to configure this manually - debspawn just does it. In general, debspawn intentionally doesn't offer a lot of configuration options, it's intended to just do whatever thing is "best" and doesn't actually let the developer mess with the build environment too much (the later versions actually don't follow this paradigm as much anymore, as debspawn is now used for package development as well, and not just as a dumb executor for an autobuilder).

I hope this clarifies things - I haven't read the README full in almost 4 years, and maybe with your feedback I can clean it up quite a bit :-)

Thanks a lot for your time! :-)

josch

comment created time in a month

push eventjosch/img2pdf

Johannes Schauer Marin Rodrigues

commit sha 95a313f43771c2a840c2e1e080476573732bc412

tox.ini: add python 3.10 to envlist

view details

Johannes Schauer Marin Rodrigues

commit sha 9eacfdaa7607d224c86e1a1bfaa8f895d22f93dd

appveyor.yml: don't run tests because we don't have imagemagick

view details

push time in a month

PR opened mapbox/carto

lib/carto/mml.js: FIX: Error: Function yaml.safeLoad is removed in js…

…-yaml 4. Use yaml.load instead, which is now safe by default.

This makes the code compatible with js-yaml >= 4.

+1 -1

0 comment

1 changed file

pr created time in a month

create barnchjosch/carto

branch : js-yaml-4

created branch time in a month

push eventjosch/carto

Johannes Schauer Marin Rodrigues

commit sha 3951644f3f54a08d1eda6c328fe10710aaa49b05

lib/carto/mml.js: FIX: Error: Function yaml.safeLoad is removed in js-yaml 4. Use yaml.load instead, which is now safe by default.

view details

push time in a month

fork josch/carto

fast CSS-like map stylesheets

https://cartocss.readthedocs.io/

fork in a month

more