lib.mkEnableOption prepends the string "Whether to enable " to the string that is provided as function argument. (source)

This way we will get "Weather to enable Enable support for the ...".

I suggest "support for the Raspberry Pi POE Hat."

Specs: https://rog.asus.com/laptops/rog-zephyrus/rog-zephyrus-g14-series/spec

I'm going to prioritise this, but rather than make it LATERAL all the time (which will likely scare other Opaleye users) I am going to rewrite lateral to use runQueryArr rather than runSimpleQueryArr. That should avoid https://github.com/circuithub/rel8/issues/72.

It needs a bit of fiddling but I hope to have it done this week.

Done 😄

Nice! Do you mind adding another fmap and getting rid of those xs? :D

I can 😛 . I was being lazy.

One moment.

Can you remove the duplication in here by doing all the others as an (fmap . fmap) f jsonFieldTypeByteParser? f will be IPT.strictDecodeUtf8 for String, for example.

• Improve JSON function documentation
• Add ByteString instances for JSON types, so query response can be used directly.
• Add Text instances for JSON types

AFAIK using a lateral join will force a nested loop (rather than hash join or merge join) unless the postgresql optimizer is able to eliminate the lateral join by pulling up the lateral subquery. The subquery is only able to be pulled up if it satisfies a number of conditions outlined here. It cannot have any sort of aggregation, limit, offset, window function, etc.

So, to give an example, consider this query:

explain
  select
    cl.relname,
    co.conkey
  from pg_class cl
  left outer join pg_constraint co on cl.oid = co.conrelid
  ;

which generates the following plan:

QUERY PLAN
Hash Left Join  (cost=1.54..22.58 rows=446 width=87)
  Hash Cond: (cl.oid = co.conrelid)
  ->  Seq Scan on pg_class cl  (cost=0.00..17.46 rows=446 width=68)
  ->  Hash  (cost=1.24..1.24 rows=24 width=27)
        ->  Seq Scan on pg_constraint co  (cost=0.00..1.24 rows=24 width=27)

We could instead write this with a lateral query

explain
  select
    cl.relname,
    t.conkey
  from pg_class cl
  left outer join lateral (
    select *
    from pg_constraint co
    where cl.oid = co.conrelid
  ) as t on true
  ;

and generate an identical plan because the subquery satisfies the is_simple_subquery predicate.

If we make the subquery slightly more complex in a way that causes it to not satisfy the is_simple_subquery predicate then we cannot eliminate the subquery and the lateral remains, which forces the planner to use a nested loop for the lateral join.

Some examples:

explain
  select
    cl.relname,
    t.conkey
  from pg_class cl
  left outer join lateral (
    select array_agg(co.conkey) as conkey
    from pg_constraint co
    where cl.oid = co.conrelid
  ) as t on true
  ;

QUERY PLAN
Nested Loop Left Join  (cost=1.31..613.99 rows=446 width=96)
  ->  Seq Scan on pg_class cl  (cost=0.00..17.46 rows=446 width=68)
  ->  Aggregate  (cost=1.31..1.32 rows=1 width=32)
        ->  Seq Scan on pg_constraint co  (cost=0.00..1.30 rows=2 width=23)
              Filter: (cl.oid = conrelid)

compared to a non-lateral equivalent (assuming cl.relname is unique)

explain
  select
    cl.relname,
    array_agg(co.conkey)
  from pg_class cl
  left outer join pg_constraint co on cl.oid = co.conrelid
  group by cl.relname
  ;

QUERY PLAN
HashAggregate  (cost=24.81..30.39 rows=446 width=96)
  Group Key: cl.relname
  ->  Hash Left Join  (cost=1.54..22.58 rows=446 width=87)
        Hash Cond: (cl.oid = co.conrelid)
        ->  Seq Scan on pg_class cl  (cost=0.00..17.46 rows=446 width=68)
        ->  Hash  (cost=1.24..1.24 rows=24 width=27)
              ->  Seq Scan on pg_constraint co  (cost=0.00..1.24 rows=24 width=27)

Or we can even force the subquery to not be pulled up with the much sillier offset 0

explain
  select
    cl.relname,
    t.conkey
  from pg_class cl
  left outer join lateral (
    select *
    from pg_constraint co
    where cl.oid = co.conrelid
    offset 0
  ) as t on true
  ;

QUERY PLAN
Nested Loop Left Join  (cost=0.00..615.10 rows=892 width=87)
  ->  Seq Scan on pg_class cl  (cost=0.00..17.46 rows=446 width=68)
  ->  Seq Scan on pg_constraint co  (cost=0.00..1.30 rows=2 width=320)
        Filter: (cl.oid = conrelid)

The example @ocharles gives above

explain
select *
from
  (values (1), (2), (3)) as s1(x)
  cross join lateral (
    select *
    from (select 0) as q
    left join lateral (
      select *
      from (values (1), (2), (3)) as s2(x)
      where s1.x = s2.x
    ) as s2 on true
  ) as q;

QUERY PLAN
Nested Loop  (cost=0.00..0.29 rows=3 width=12)
  ->  Values Scan on "*VALUES*"  (cost=0.00..0.04 rows=3 width=4)
  ->  Nested Loop Left Join  (cost=0.00..0.07 rows=1 width=8)
        ->  Result  (cost=0.00..0.01 rows=1 width=0)
        ->  Values Scan on "*VALUES*_1"  (cost=0.00..0.05 rows=1 width=4)
              Filter: ("*VALUES*".column1 = column1)

fails to satisfy the is_simple_subquery predicate for a trickier reason. Have a look at this excerpt

 * If the subquery is LATERAL, check for pullup restrictions from that.
 */
if (rte->lateral)
{
        bool      restricted;
        Relids        safe_upper_varnos;

        /*
         * The subquery's WHERE and JOIN/ON quals mustn't contain any lateral
         * references to rels outside a higher outer join (including the case
         * where the outer join is within the subquery itself).  In such a
         * case, pulling up would result in a situation where we need to
         * postpone quals from below an outer join to above it, which is
         * probably completely wrong and in any case is a complication that
         * doesn't seem worth addressing at the moment.
         */
        if (lowest_outer_join != NULL)
        {
                restricted = true;
                safe_upper_varnos = get_relids_in_jointree((Node *) lowest_outer_join,
                                                                                                   true);
        }
        else
        {
                restricted = false;
                safe_upper_varnos = NULL; /* doesn't matter */
        }

        if (jointree_contains_lateral_outer_refs((Node *) subquery->jointree,
                                                                                         restricted, safe_upper_varnos))
                return false;

I believe that this means that the subquery

(
  select *
  from (values (1), (2), (3)) as s2(x)
  where s1.x = s2.x
)

fails to be pulled up since we have a qual s1.x = s2.x with lateral references to a relation outside of the lowest outer join (the join with select 0). It would be wrong to pull this subquery up since that qual would have to be moved from the nullable side of an outer join to the non-nullable side, and would begin filtering rows from the non-nullable side, changing the query.

We then consider pulling up the lateral subquery

(
      select *
      from (select 0) as q
      left join lateral (
        select *
        from (values (1), (2), (3)) as s2(x)
        where s1.x = s2.x
      ) as s2 on true
    )

which fails for essentially the same reason: If we pulled up this subquery then we cannot have the lateral qual s1.x = s2.x, so this condition would need to be pulled up to a where clause, but we refuse to pull it out of the nullable side of an outer join as it would change the meaning of the query. So, we do not pull this subquery up.

So, I believe any sort of query with the structure:

  select
    ...
  from src
  cross join lateral (
    select
      ...
    from blerg
    left outer join (lateral or not) (
      subquery with some lateral reference to src
    )
  )

will fail to have the lateral subquery pulled up, and thus will force a nested loop.

Hi

Wouldn't it make sense to add a type-safe lens ? Using type applications it becomes quite compact.

ix' :: forall m (n :: Nat) a (f :: * -> *).
  (Functor f, KnownNat m, KnownNat n, m+1 <= n) =>
  (a -> f a) -> Vec.Vector n a -> f (Vec.Vector n a)
ix' = ix $ natToFinite (Proxy::Proxy m)

This can be then be used like so:

ix' @7

It probably would be good to have a release if you're going to merge this, because we will have to make changes to Rel8 to accommodate this, and at least if it's in a release we can set the version bounds appropriately. However, probably the more urgent problem for us is #480.

It probably would be good to have a release if you're going to merge this, because we will have to make changes to Rel8 to accommodate this, and at least if it's in a release we can set the version bounds appropriately. However, probably the more urgent problem for us is #480.

This actually can't be solved in the optimizer alone. This is what such an optimization pass would look like (not sure if 100% correct, but something like this):

optional :: Opaleye.PrimQuery' a -> Opaleye.PrimQuery' a
optional = Opaleye.foldPrimQuery Opaleye.primQueryFoldDefault
  { Opaleye.product = product
  }
  where
    product ps es = case go (toList ps) of
      [] -> Opaleye.Unit
      [p] -> snd p
      p : ps' -> Opaleye.Product (p :| ps') es
      where
        go ((x, q) : (x', Opaleye.Join jt on lb rb Opaleye.Unit rq) : rest) =
          (x <> x', Opaleye.Join jt on lb rb q rq) : rest
        go as = as

That will basically turn @ocharles's latter example into the former, but it won't be LEFT JOIN LATERAL, it will only be LEFT JOIN, which will generate an invalid query.

We've just discovered some more motivation for this. In https://github.com/circuithub/rel8/issues/72, it's reported that Rel8 generates a bad plan for a LEFT JOIN. What's happening is we're using optional :: Query a -> Query (MaybeTable a), to introduce the LEFT JOIN, which works by taking the LEFT JOIN of the current Query with whatever Query is to be made optional. That is, in something like:

foo >> optional bar

We want something like Join LeftJoin fooQ barQ.

This isn't quite what happens at the moment, because >>= is implemented in terms of lateral, which itself uses runSimpleQueryArr. This means what you actually get is:

Product [fooQ, Join LeftJoin Unit barQ]

This spurious Unit wrecks havoc with PostgreSQL's optimizer. First, here's some "ideal" SQL:

postgres=# explain select * from (values (1), (2), (3)) s1 (x) left join lateral (select * from (select 0) q, (values (1), (2), (3)) s2 (x) where s1.x = s2.x) s2 on true;
                                QUERY PLAN                                 
---------------------------------------------------------------------------
 Hash Left Join  (cost=0.08..0.15 rows=3 width=12)
   Hash Cond: ("*VALUES*".column1 = "*VALUES*_1".column1)
   ->  Values Scan on "*VALUES*"  (cost=0.00..0.04 rows=3 width=4)
   ->  Hash  (cost=0.04..0.04 rows=3 width=8)
         ->  Values Scan on "*VALUES*_1"  (cost=0.00..0.04 rows=3 width=8)
(5 rows)

But look what happens if we introduce a Unit:

explain select * from (values (1), (2), (3)) s1 (x), lateral (select * from (select 0) q left join lateral (select * from (values (1), (2), (3)) s2 (x) where s1.x = s2.x) s2 on true) q;
                                QUERY PLAN                                 
---------------------------------------------------------------------------
 Nested Loop  (cost=0.00..0.32 rows=3 width=12)
   ->  Values Scan on "*VALUES*"  (cost=0.00..0.04 rows=3 width=4)
   ->  Nested Loop Left Join  (cost=0.00..0.07 rows=1 width=8)
         ->  Result  (cost=0.00..0.01 rows=1 width=4)
         ->  Values Scan on "*VALUES*_1"  (cost=0.00..0.05 rows=1 width=4)
               Filter: ("*VALUES*".column1 = column1)
(6 rows)

The worst-case cost has doubled, and we've deteriorated to a loop.

This could be solved in the optimizer, but spotting this unit and re-arranging the query appropriately. I think the simpler option is to just do this PR. Writing optimization passes are really hard and easy to get wrong!

It's now up-to-date at https://github.com/tomjaguarpaw/haskell-opaleye/tree/endo.

Do you want this in a release or is in master good enough? I definitely will make a release with it in soon, but just checking whether you want me to expedite it.

And actually, also this comment as well. Basically, we actually do want that endo refactoring you did.

And actually, also this comment as well. Basically, we actually do want that endo refactoring you did.

Hello there, The pi connects fine to a single LG monitor, xrandr detects it, and I'm able to do customizations with it(except, I'm unable to change brightness with xrandr, so I have to manually press the monitors buttons to set the brightness). But, whenever I try to connected the pi to a 2nd LG monitor(which is the exact same model), and it stays stuck in the rainbow screen? And xrandr does not detect the 2nd monitor either? I enabled home-manager's autorandr option, but that does not fix this issue either? I'm posting this issue here, because I'm not sure if it's a hardware problem. Also I haven't messed with the pi's bootloader config.txt file, so I don't know if that would fix this issue. If you could at least give some tips on how to troubleshoot this issue. Thanks in advance for helping fix this issue.

I built nixos-21.05 with the latest kernel 5.12.11 and the bluetooth patch right now. I can confirm: wifi and bluetooth work for me. Also built-in microphone and speaker works. A PR to update would be great @terinjokes

For what it's worth, you might be interested in this comment here explaining the technique we eventually ended up using to get this working.

Pending tasks are:

• [ ] Document at least the regenerate-hackage-packages.sh script in the nixpkgs manual
• [ ] Remove the legacy hackage2nix scripts from this repository (I think they now only serve to confuse users)

@tomjaguarpaw I'm still interested in this by the way, I don't know if you're any more open to it now. I've rebased it onto the latest master anyway.

I've tried to run hackage2nix in order to test a change I made to the yaml file in Nixpkgs. I didn't make it through. I'll summarize my experience.

1. I used git add worktree and git checkout --detach instead of git clone nixpkgs to speed it up. Didn't work because update-nixpkgs.sh pulls in the nixpkgs worktree.
2. I had to add cabal, ghc, zlib and openssl to my shell before invoking update-nixpkgs.sh
3. Still, cabal failed with

   $ ./update-nixpkgs.sh 
   <command line>: can't load .so/.DLL for: libz.so (libz.so: cannot open shared object file: No such file or directory)
   cabal: Failed to build cabal2nix-2.15.0 (which is required by exe:hackage2nix
   from cabal2nix-2.15.0).
   

Proposed solution

I think the script can be nixified. I suggest adding the following to Nixpkgs:

• pin files for
  • cabal2nix repo revision
  • all-cabal-hashes repo revision
• a port of update-nixpkgs.sh, limited to generation
  • using the pins
  • no assumption about the caller's environment (nix-shell shebang or similar)
  • no git logic, just generation
• a script to update the pins and call the codegen script
• assuming the code generation is now pure, a derivation that tests whether the generated code is up to date, to prevent hand-editing and ensure reproducibility

This will let anyone regenerate the package set in a reproducible way. This also makes #219 trivial. Just mention the pin files in a comment.

Did I miss something that makes the above not work?

The steps in Nixpkgs to regenerate the packages have been reduced to essentially two steps, which is close enough for me.

My notes for a quick test with updated hackage, distilled from the more complete pkgs/development/haskell-modules/HACKING.md (or its permalink)

Test newly uploaded release

Check that all-cabal-hashes is up to date with your upload. Run:

./maintainers/scripts/haskell/update-hackage.sh
./maintainers/scripts/haskell/regenerate-hackage-packages.sh
nix-build -A haskellPackages.foo

Integrity holding at ninety percent.

Diff without documentation changes