mypy would have caught this except that lies in the AST about Optional caused it to think most of the function was unreachable. Most of this PR is actually just fixing the fallout from that. I also added parser contexts to this error while I was at it.

Fixes #1788.

The query is obviously wrong but it should fail with a proper error:

SELECT schema::ObjectType {
      name,
      is_abstract: {
        name,
      } ORDER BY @index ASC,
} FILTER .name = 'sys::Role';

Currently fails with

ERROR: InternalServerError:
  Hint: This is most likely a bug in EdgeDB. Please consider opening an issue ticket at https://github.com/edgedb/edgedb/issues/new?template=bug_report.md
  Server traceback:
      Traceback (most recent call last):
        File "/Users/yury/dev/edge/edgedb/edb/server/procpool/worker.py", line 75, in worker
          res = await meth(*args)
        File "/Users/yury/dev/edge/edgedb/edb/server/compiler/compiler.py", line 1795, in compile_eql_tokens
          return self._compile(ctx=ctx, tokens=eql_tokens)
        File "/Users/yury/dev/edge/edgedb/edb/server/compiler/compiler.py", line 1429, in _compile
          comp: dbstate.BaseQuery = self._compile_dispatch_ql(ctx, stmt)
        File "/Users/yury/dev/edge/edgedb/edb/server/compiler/compiler.py", line 1392, in _compile_dispatch_ql
          return self._compile_ql_query(ctx, ql)
        File "/Users/yury/dev/edge/edgedb/edb/server/compiler/compiler.py", line 592, in _compile_ql_query
          ir = qlcompiler.compile_ast_to_ir(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/__init__.py", line 175, in wrapper
          return func(*args, **kwargs)
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/__init__.py", line 223, in compile_ast_to_ir
          ir_set = dispatch_mod.compile(tree, ctx=ctx)
        File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/functools.py", line 874, in wrapper
          return dispatch(args[0].__class__)(*args, **kw)
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/stmt.py", line 84, in compile_SelectQuery
          stmt.result = compile_result_clause(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/stmt.py", line 1017, in compile_result_clause
          ir_result = compile_query_subject(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/stmt.py", line 1097, in compile_query_subject
          view_scls = viewgen.process_view(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/viewgen.py", line 86, in process_view
          view_scls = _process_view(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/viewgen.py", line 173, in _process_view
          pointer = _normalize_view_ptr_expr(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/viewgen.py", line 549, in _normalize_view_ptr_expr
          ptr_target = _process_view(
        File "/Users/yury/dev/edge/edgedb/edb/edgeql/compiler/viewgen.py", line 160, in _process_view
          assert isinstance(view_scls, s_objtypes.ObjectType)
      AssertionError

I started fixing some of this and it might be less bad than I feared but I also discovered that schema.get basically always is inferred as returning None which seems pretty bad

mypy would have caught this except that lies in the AST about Optional caused it to think most of the function was unreachable. Most of this PR is actually just fixing the fallout from that. I also added parser contexts to this error while I was at it.

Fixes #1788.

A lot of fields in especially our qlast but also everywhere else are not marked Optional despite being optional.

This causes two big problems:

1. We don't get typechecking related to the optionality
2. In a bunch of cases mypy will decide that branches are unreachable and not typecheck them at all.

Ideally, mypy would catch that we were putting None into these non-Optional fields, but we have evaded it.

What is the correct behavior here? Rejecting the query because target is not a link attribute?

Updated with a fix that I'm not really happy with.

This will fix an issue where fetch something twice for expr and serialized, or similar, including sometimes doing two function calls!

A fix I made in #1762 made find_path_output no longer necessary for correctness, so I removed it, but it will still important as an optimization.

Restore the optimization bit of that code, and add a comment so I don't forget and delete it again ;P.

Oh, oops. I ran the test suite locally and somehow then forgot it had failed between yesterday and this morning and it is broken. I'm going to close this for now because I think the problem is serious.

This will fix an issue where fetch something twice for expr and serialized, or similar, including sometimes doing two function calls!

A fix I made in #1762 made find_path_output no longer necessary for correctness, so I removed it, but it will still important as an optimization.

Restore the optimization bit of that code, and add a comment so I don't forget and delete it again ;P.

My intuition about it was that it clearly makes sense according to our semantics that SELECT (Object, random()) should always return the same value (and, in fact, it would be pretty wrong if it didn't!)

Actually, I take this back. The "Iterate over the set of input tuples, and on every iteration evaluate the query expression" semantics does I think imply that it ought to call random() for each row, since there will be an input tuple for each Object. But a query like SELECT ({1,2,3}, random()) would call random() once, because there is only one input tuple. (This matches what currently happens in my evaluator model.)

Is this a nonstarter in terms of performance/implementation? Anything we need to do we could only do if we detect a volatile, so maybe the hypothetical perf impact isn't that bad, but correlating with every input set seems painful.

So we need to rethink how we specify the semantics, then, too, if our desired (AKA implementable) behavior is somewhere in between what we currently specify and what we implement.

Currently set literals get compiled into a chain of UNIONs, which causes a big blowup in the nesting depth of the AST. As a result, we trip a recursion error on set literals containing 60 one-element tuples, which is a solidly medium-size input.

Fix this by compiling set literals into a tree of UNIONs instead of a chain.

Fixes #475 to my satisfaction though others may disagree

Query to repro:

WITH MODULE test
FOR X IN {
     (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), 
     (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), 
     (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), 
     (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), 
}
UNION 1;

Yeah, I hit this while thinking about #492.

My intuition about it was that it clearly makes sense according to our semantics that SELECT (Object, random()) should always return the same value (and, in fact, it would be pretty wrong if it didn't!), the obvious seeming interpretation of our semantics is that FOR x IN {1, 2, 3} UNION (x, random()) should return different values, and so it should be possible to accomplish whatever we need to do without new syntax. So I figured I'd work through some examples of what workarounds were needed, which would help illuminate to me what is needed from sugar to make it easier.

(Thinking about it a little more, the most obvious reading of our semantics I think also implies that ORDER BY random() ought to work, too, though.)

(There's a plausible semantics where it is always allowed to get factored out, also, but my understanding is there are cases where we need that to not be the case. And it's less nice, IMO.)

Currently set literals get compiled into a chain of UNIONs, which causes a big blowup in the nesting depth of the AST. As a result, we trip a recursion error on set literals containing 60 one-element tuples, which is a solidly medium-size input.

Fix this by compiling set literals into a tree of UNIONs instead of a chain.

Fixes #475 to my satisfaction though others may disagree

A few things:

• It looks like the error does get serialized properly now
• It's pretty fundamental to what we do that we are going to be recursing over user-provided nested data, so this sort of thing is always a risk
• Python's default recursion limit is 1000, which is really not very big. It's probably worth turning it up.
• If we want to make sure to prevent this sort of thing, one option is to have a check on the original AST that the nesting depth of constructs doesn't exceed some level, and then try not to blow up the nesting level beyond some factor during our compilation passes
• This example in particular blows up the nesting level because we turn the set literal into a linearly deep chain of UNION operators. We could make it logarithmically deep by making it a tree of UNIONs instead of a list. (Though note that this is still incompatible with the above bullet point's suggest strategy which only permits a constant factor blowup.)

So I think my proposed plan for RecursionErrors is:

1. Turn up the recursion limit
2. Fix anything that seems to gratuitously blow up the nesting depth but otherwise don't worry too much at this stage

(That means fixing this one to not do a linear blowup of set literals)

Both of these work now, but adding an extra row makes it fail

1. FOR x IN {1, 2, 3} UNION (x, random()); returns identical values
2. FOR x IN {1, 2, 3} UNION (SELECT (x, random())); returns different values
3. SELECT (FOR x IN {1, 2, 3} UNION (SELECT (x, random()))); returns identical values (!)

I /think/ that all 3 ought to return different values but 2 and 3 definitely should do the same thing.

What is the semantics of this query supposed to be? Currently it doesn't even parse.

@elprans is working on a change to use UUIDs as column names, which should fix this

An update on this bug: SELECT test(<optional str>{}); now returns {}, which is correct. For it to return {0}, you would need the parameter to be declared optional.

It looks like the bug is now caused by name conflict: renaming the parameter to something other than nickname makes it work. Argh!

Also: WITH X := {1, 2}, Y := X+1 SELECT (X, Y). Should produce a cross-product output with 4 results, actually produces 2.

Currently we generate invalid SQL, since we would need to lift the correlated set into CTEs, which we don't want to deal with.

Fixes #1739.

(Stacked on top of https://github.com/edgedb/edgedb/pull/1776, can't merge yet)

We should reject queries like SELECT (Person, (FOR x in {Person} UNION ((INSERT Note {name := x.name}))));

Currently we generate invalid SQL.

This fixes an issue introduced in #1757, where declaring a view for the iterator was done in a temporary scope so we lost the tree modifications.

Currently we generate invalid SQL, since we would need to lift the correlated set into CTEs, which we don't want to deal with.

Fixes #1739.

(Stacked on top of https://github.com/edgedb/edgedb/pull/1776, can't merge yet)

Great catch.

I did that, and threw some slight refactoring/rearrangement in because dropping such a big chunk of code into the same nesting level without any visual breaks was giving me the willies :P

Maybe it's also better to disallow overloading the number of positional arguments at all?

I think I am against prohibiting overloading the number of arguments, though that is also something I could easily see myself on the other side of.

If the proposal is disallow making SINGLETON/SET OF/OPTIONAL decisions based on the number of arguments, that is pretty reasonable and might be better than my original proposal. Then the type mod of any parameter would be uniquely determined by the function and its argument position/name.

Additional knock on effects in the standard library are that std::to_datetime() would need some tweaking but that is it.

I'm not sure what you propose here. To have placeholder values in those cases, or create another modifier that can be omitted but can't be an empty set? Probably latter can fix the issue I've decsribed above, but that's another quirk of the language.

@msullivan proposes to prohibit this: [...]

~My question was specifically for this part of the proposal:~ Aha, I see we actually do differntiate between OPTIONAL arguments and arguments having a default values that can be omitted, but are not OPTIONAL...

A lot of the OPTIONAL arguments aren't really OPTIONAL, I think, also; the optionality is just there to be a default value for a parameter and probably nobody ever plans to pass a potentially empty set in.

I think I was somewhat unclear here: this sentence was really just an observation, and doesn't have any proposal attached to it. This was not really

Currently type-based function overloading is allowed to select argument type modifiers (SINGLETON/SET OF/OPTIONAL). There is substantial complexity in edb/edgeql/compiler/func.py and the scope tree to handle this (a bunch added recently, in #1762).

I think that argument type modifiers (SET OF, in particular, which induces subqueries) affect the semantics of the query enough that it should be easy and straightforward to determine them.

I propose that we require that all function overloads with matching signatures (number of positional args + names of name only arguments) have the same type modifiers. We can then decide which type modifiers to use based on just the function signature, before we necessarily know the types, and then figure out which exact instance later. (I wouldn't mind banning overloading SET OF here, either, since SET OF is a big deal, but I don't have any technical arguments.)

This will let us clean up function compilation to be able to know the argument kind in advance. (We could also accomplish this by having a typechecker that directly typechecks EdgeQL in advance, which I think is probably a good idea either way but is also probably a bunch of work.) It also means that an evaluation model can get away without needing to do a static typecheck. (I think. There might be other problems.)

There are currently 4 offending functions in the stdlib: std::to_str (the deprecated array overload) and cal::to_local_{datetime,date,time}. They can all be fixed without much trouble. (The second argument to the array overload for to_str can be made optional, though it is deprecated and being removed anyway, and the cal functions can have the fmt argument made named only, or made non-OPTIONAL and given a default of "" or something).

A lot of the OPTIONAL arguments aren't really OPTIONAL, I think, also; the optionality is just there to be a default value for a parameter and probably nobody ever plans to pass a potentially empty set in.

git-config appears to lowercase keys before returning them, which meant sync.excludePaths and friends weren't getting found. Look them up lowercased, instead.

I'm not sure if this is a change in behavior on git's side (it does this as of 2.20.1, at least), so to make sure, lowercase it outselves at the ingestion point.

OK, I'll try that. Basically every single thing about this PR has changed since that version, though :P

Fixes #1743.

• EdgeDB Version: EdgeDB 1.0-alpha.6+dev.5050.gb8aa1c99
• OS Version: Linux

Steps to Reproduce:

CREATE TYPE default::Foo {
    CREATE OPTIONAL SINGLE PROPERTY name -> std::str {
        CREATE CONSTRAINT std::exclusive;
    };
};
CREATE TYPE default::Bar {
    CREATE REQUIRED SINGLE LINK foo -> default::Foo;
};
INSERT Foo {name := 'foo0'};

Then the following happens:

victor> SELECT (INSERT Bar {foo := (INSERT Foo {name := 'foo0'} UNLESS CONFLICT ON .name ELSE (SELECT Foo))}) {id};
{Object {id: 948f4c8e-ed81-11ea-955a-5b32015497c5}}
victor> SELECT (INSERT Bar {foo := (INSERT Foo {name := 'foo0'} UNLESS CONFLICT ON .name ELSE (SELECT Foo))}) {id, foo};
ERROR: InternalServerError: more than one row returned by a subquery used as an expression
  Hint: This is most likely a bug in EdgeDB. Please consider opening an issue ticket at https://github.com/edgedb/edgedb/issues/new?template=bug_report.md
  Server traceback:
      edb.errors.InternalServerError: more than one row returned by a subquery used as an expression

So it's not that the cardinality check fails, it's that the SQL causes an effect similar to a cardinality problem if foo is included in the final shape to be returned.

This branch is only for INSERTs. The assert should always be true.

Oops on the unIon! I added that to make sure this is where a particular union in the generated code was coming from :P

Fixes #1704.

Steps to Reproduce:

EdgeDB 1.0-alpha.4+dev.145.g4d9ce642
Type "\?" for help.
victor> 
victor> 
victor> create type Vector {
.......     create required property x -> float64;
.......     create required property y -> float64;
....... };
OK: CREATE
victor> alter type Vector { create constraint expression on (.x^2 + .y^2 < 25) };
error: could not resolve partial path 
  ┌─ query:1:54
  │
1 │ alter type Vector { create constraint expression on (.x^2 + .y^2 < 25) };
  │                                                      ^^^ error

Ideally, the short path should assume __subject__ as its start.

OK this isn't right yet, since WITH X := {1, 2} SELECT ((FOR x in {X} UNION (SELECT x)), X); still computes a cross product, although that bug predates the original one.

This fixes an issue introduced in #1757, where declaring a view for the iterator was done in a temporary scope so we lost the tree modifications.

https://github.com/edgedb/edgedb/pull/1757/ made this succeed by losing the correlation between the two Persons, which is also a bug!

Fixes #1743.

Fixes #1704.

This patch errs on the side of disallowing, since it is pretty easy to workaround.

We allow DML when the source is filtered, comes from a query that is not a set reference, when it is a FOR iterator, and when it is the set being inserted in the ELSE clause of INSERT ... ELSE. (FOR is allowed because the meaning seems to clearly imply applying to everything ...and because it made the implementation easy.)

Most of the test cases for things we allow are existing ones.

I can follow up with a documentation update if there is agreement on this one.

Closes #1742.