profile
viewpoint

adonovan/gopl.io 4578

Example programs from "The Go Programming Language"

google/skylark 1157

Skylark in Go: the Skylark configuration language, implemented in Go [MOVED to go.starlark.net]

google/starlark-go 1068

Starlark in Go: the Starlark configuration language, implemented in Go

google/re2j 898

linear time regular expression matching in Java

bazelbuild/starlark 689

Starlark Language

bazelbuild/bazel-skylib 127

Common useful functions and rules for Bazel

alandonovan/gopl.io 1

Example programs from "The Go Programming Language"

alandonovan/rules_haskell 1

Haskell rules for Bazel.

alandonovan/starlark 0

Starlark Language

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

issue commentgoogle/starlark-go

Possible to create a simple interactive debugger?

Not yet. At one point I started writing an asynchronous concurrent debugger with breakpoints, but got distracted. Let me see if I can find it, then perhaps we can define a reasonable API for accessing the environment to support your use case, even if we don't add complete support for breakpoints etc.

maxmcd

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentbazelbuild/starlark

Spec: document string literals

 octal_digit   = '0' … '7' . hex_digit     = '0' … '9' | 'A' … 'F' | 'a' … 'f' . ``` -TODO: define string_lit, indent, outdent, semicolon, newline, eof+### String literals++A Starlark string literal denotes a string value.+In its simplest form, it consists of the desired text+surrounded by matching single- or double-quotation marks:++```python+"abc"+'abc'+```++Literal occurrences of the chosen quotation mark character must be+escaped by a preceding backslash. So, if a string contains several+of one kind of quotation mark, it may be convenient to quote the string+using the other kind, as in these examples:++```python+'Have you read "To Kill a Mockingbird?"'+"Yes, it's a classic."+"Have you read \"To Kill a Mockingbird?\""+'Yes, it\'s a classic.'+```++#### String escapes++Within a string literal, the backslash character `\` indicates the+start of an _escape sequence_, a notation for expressing things that+are impossible or awkward to write directly.++The following *traditional escape sequences* represent the ASCII control+codes 7-13:++```+\a   \x07 alert or bell+\b   \x08 backspace+\f   \x0C form feed+\n   \x0A line feed+\r   \x0D carriage return+\t   \x09 horizontal tab+\v   \x0B vertical tab+```++A *literal backslash* is written using the escape `\\`.++An *escaped newline*---that is, a backslash at the end of a line---is ignored,+allowing a long string to be split across multiple lines of the source file.++```python+"abc\+def"			# "abcdef"+```++An *octal escape* encodes a single byte using its octal value.

This seems fine for now. Correctly documenting how the Java implementation differs is a complex and subtle project.

laurentlb

comment created time in 4 days

PullRequestReviewEvent

pull request commentbazelbuild/starlark

Spec: document string literals

Note that it doesn't exactly match the Java implementation, e.g. we reject \a. I assume you'd rather update the implementation?

Yes, useless though it may be, every language supports \a thus so should we.

laurentlb

comment created time in 4 days

Pull request review commentbazelbuild/starlark

Spec: document string literals

 octal_digit   = '0' … '7' . hex_digit     = '0' … '9' | 'A' … 'F' | 'a' … 'f' . ``` -TODO: define string_lit, indent, outdent, semicolon, newline, eof+### String literals++A Starlark string literal denotes a string value.+In its simplest form, it consists of the desired text+surrounded by matching single- or double-quotation marks:++```python+"abc"+'abc'+```++Literal occurrences of the chosen quotation mark character must be+escaped by a preceding backslash. So, if a string contains several+of one kind of quotation mark, it may be convenient to quote the string+using the other kind, as in these examples:++```python+'Have you read "To Kill a Mockingbird?"'+"Yes, it's a classic."+"Have you read \"To Kill a Mockingbird?\""+'Yes, it\'s a classic.'+```++#### String escapes++Within a string literal, the backslash character `\` indicates the+start of an _escape sequence_, a notation for expressing things that+are impossible or awkward to write directly.++The following *traditional escape sequences* represent the ASCII control+codes 7-13:++```+\a   \x07 alert or bell+\b   \x08 backspace+\f   \x0C form feed+\n   \x0A line feed+\r   \x0D carriage return+\t   \x09 horizontal tab+\v   \x0B vertical tab+```++A *literal backslash* is written using the escape `\\`.++An *escaped newline*---that is, a backslash at the end of a line---is ignored,+allowing a long string to be split across multiple lines of the source file.++```python+"abc\+def"			# "abcdef"+```++An *octal escape* encodes a single byte using its octal value.

(The reference to byte here isn't really appropriate for the Java implementation.)

laurentlb

comment created time in 4 days

Pull request review commentbazelbuild/starlark

Spec: document string literals

 octal_digit   = '0' … '7' . hex_digit     = '0' … '9' | 'A' … 'F' | 'a' … 'f' . ``` -TODO: define string_lit, indent, outdent, semicolon, newline, eof+### String literals++A Starlark string literal denotes a string value.+In its simplest form, it consists of the desired text+surrounded by matching single- or double-quotation marks:++```python+"abc"+'abc'+```++Literal occurrences of the chosen quotation mark character must be+escaped by a preceding backslash. So, if a string contains several+of one kind of quotation mark, it may be convenient to quote the string+using the other kind, as in these examples:++```python+'Have you read "To Kill a Mockingbird?"'+"Yes, it's a classic."+"Have you read \"To Kill a Mockingbird?\""+'Yes, it\'s a classic.'+```++#### String escapes++Within a string literal, the backslash character `\` indicates the+start of an _escape sequence_, a notation for expressing things that+are impossible or awkward to write directly.++The following *traditional escape sequences* represent the ASCII control+codes 7-13:++```+\a   \x07 alert or bell+\b   \x08 backspace+\f   \x0C form feed+\n   \x0A line feed+\r   \x0D carriage return+\t   \x09 horizontal tab+\v   \x0B vertical tab+```++A *literal backslash* is written using the escape `\\`.++An *escaped newline*---that is, a backslash at the end of a line---is ignored,+allowing a long string to be split across multiple lines of the source file.++```python+"abc\+def"			# "abcdef"+```++An *octal escape* encodes a single byte using its octal value.+It consists of a backslash followed by one, two, or three octal digits [0-7].+It is error if the value is greater than decimal 255.++```python+'\0'			# "\x00"  a string containing a single NUL byte+'\12'			# "\n"    octal 12 = decimal 10+'\101-\132'		# "A-Z"+'\119'			# "\t9"   = "\11" + "9"+```++A *hex escape* encodes a single byte using its hexadecimal value.

(and hex escapes aren't implemented in Java)

laurentlb

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentbazelbuild/starlark

Update spec from google/starlark-go

 values include `None`, Booleans, numbers, and strings, and tuples composed from hashable values.  Most mutable values, such as lists, and dictionaries, are not hashable, unless they are frozen. Attempting to use a non-hashable value as a key in a dictionary-results in a dynamic error, as does passing one to the built-in-`hash` function.+results in a dynamic error.

Oh, of course.

laurentlb

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentbazelbuild/starlark

Update spec from google/starlark-go

 values include `None`, Booleans, numbers, and strings, and tuples composed from hashable values.  Most mutable values, such as lists,

I noticed that the TOC that exists in Go spec has vanished from bazelbuild. Could you restore it?

laurentlb

comment created time in 4 days

Pull request review commentbazelbuild/starlark

Update spec from google/starlark-go

 values include `None`, Booleans, numbers, and strings, and tuples composed from hashable values.  Most mutable values, such as lists, and dictionaries, are not hashable, unless they are frozen. Attempting to use a non-hashable value as a key in a dictionary-results in a dynamic error, as does passing one to the built-in-`hash` function.+results in a dynamic error.

I think the old text was correct (does that mean this change needs to flow in the other direction?).

Both hash implementations accept only string, not list, say.

laurentlb

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentbazelbuild/stardoc

Where is stardoc source code

@c-parsons Could you rename this issue? Apparently I don't have write access.

UebelAndre

comment created time in 5 days

issue commentgolang/go

proposal: Go 2: language: make slices of constant strings constant if the indexes are constant

@go101 That's diabolical.

Spoiler: the difference arises from the top-down effect of the enclosing type (byte) on the shift operator when its right operand is non-constant. Thus 1 << len(s[:]) is truncated to byte. The type rules for shifts are the most complex of all...

ainar-g

comment created time in 5 days

issue commentbazelbuild/stardoc

Where is stardoc source code

Hi UebelAndre, I don't really have a tracking issue for this work, but I can sketch it briefly. The current design of skydoc is an unfortunate maintenance problem as it executes BUILD files in an environment that tries to simulate the loading phase of Bazel without actually running Bazel. This means that every change to the production code of the really quite complex BUILD environment in Bazel requires a corresponding change to the fakes in Stardoc. To reduce that burden, the entire Starlark API of the build language has been factored into an artificial separation of interface and implementation, even though there is only one real implementation, leading to sublime poetry like this: https://github.com/bazelbuild/bazel/blob/450c7ad65bfcffebf753159ee2b617e557ffb4fd/src/main/java/com/google/devtools/build/lib/starlarkbuildapi/cpp/CcBootstrap.java#L53

To me the obvious solution is: ask Bazel for the information. It already knows how to load and execute a .bzl file. Bazel query supports many ways reporting information about targets. What we need is an output format that reports information about a target's BUILD file, the .bzl file its loads (and so on recursively), and the Starlark constants, functions, rules, providers, and so on, defined in those .bzl files. Bazel already has the internal data structures. Once we have that, Skydoc can be entirely implemented as an external client of blaze. (Or, if you prefer, as a Starlark-defined build rule that invokes genquery then processes its output.) The information provided by this output format would be useful for many other tools too.

At this point I have a very rough prototype. I expect we'll finish it in Q4.

UebelAndre

comment created time in 6 days

issue commentbazelbuild/starlark

Add grammar to permit type annotations

I imagine this would be optional. It would have no impact on runtime

If the types have no dynamic semantics, then my suggestion is that those who wish to experiment with types can embed the new syntax in comments or string literals, and write an external type-checking tool. This keeps all the complexity outside the core until we have extensive experience of the type system.

It would not interact with struct or similar.

This seems like a major lost opportunity. Structs (aka Provider instances in Bazel) are perhaps the most interesting case for types.

My experiment took ~200 lines to Hack in, and the resulting Starlark code was easier to read, write and debug (for me). Not a proof it doesn't take away the simplicity, but a suggestion it's not fatal.

I am less concerned about the complexity of the implementation than I am about the complexity of the spec changes.

ndmitchell

comment created time in 7 days

issue commentbazelbuild/starlark

Add grammar to permit type annotations

The addition of types is an interesting idea---the lack of types is the biggest disadvantage of working in a Python-like language---but it is an enormous undertaking, and raises all kinds of questions, such as:

  • Should dynamic checks be optional? If so, users can't rely on them for precondition checking, and nor can compiler optimizations.
  • What is the performance cost of additional type checks, and can this be mitigated by compiler optimization?
  • How would separate compilation work? Python, unlike typed languages, allows each file to be compiled without any knowledge of the values imported from other packages. Would that change? If so, it has enormous architectural ramifications for Bazel. If not, what limits does that place on the type system?
  • To what extent have types been a success in Python3? Which parts are good and which less so?
  • Core Starlark has no struct type; that's a Bazelism. How would the type checker be made extensible to support application-defined data types with their own type-checking rules?
  • Can types be added without compromising the core benefit of Starlark, namely that it is very simple?

I don't wish to dampen your enthusiasm, but I think it is very unlikely that the Starlark team will have significant time to spend advancing any type proposal, at least in the next year.

ndmitchell

comment created time in 8 days

issue commentbazelbuild/bazel

building bazel libcpu_profiler fails to find jni_md.h (ppc64le)

The two conditions have equal definitions, so I don't see why that should matter. https://github.com/bazelbuild/bazel/blob/master/src/conditions/BUILD#L37

Linux is only a good default if you're actually using Linux. I'd rather it fail fast than use the wrong header.

clnperez

comment created time in 14 days

issue commentbazelbuild/bazel

building bazel libcpu_profiler fails to find jni_md.h (ppc64le)

@aiuto Tony, should I be able to reproduce this in a cross build? I tried bazel build --cpu=ppc src/bazel (with 3.5.0) and I got this error:

ERROR: [...]external/local_config_cc/BUILD:47:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'ppc'

Any help much appreciated.

clnperez

comment created time in 14 days

issue commentbazelbuild/bazel

rule_implementation_hash is unsound when a rule definition site is a macro in b.bzl, but that macro is invoked from a.bzl, and that invocation passes along e.g. an implementation function object

Alex, the fault was mine: you worked on the clean-up at my behest, and the suggestion to make the behavioral change along with the refactoring was mine based on a misunderstanding of the inconsistency between the label and digest portions. Thanks again for making the change, and for unmaking it when the problem emerged.

linzhp

comment created time in 14 days

PullRequestReviewEvent

PR closed adonovan/gopl.io

ch1/dup1 : Break the scanner loop for empty input

Currently the existing ch1/dup1 does not breaks for an empty input. Ctrl-C breaks the loop but ends in terminating the complete program.

The expected behavior should be that when there is an empty line as an input from user, the scanner loop should terminate.

Issue Reproduced on MacOS

image

+3 -0

1 comment

1 changed file

varunu28

pr closed time in 17 days

pull request commentadonovan/gopl.io

ch1/dup1 : Break the scanner loop for empty input

On most terminals, Ctrl-C is configured to send an interrupt signal (SIGINT) to the program, which by default terminates it. Use Ctrl-D, which on most terminals closes the file connected to stdin of the child process, causing the loop to terminate.

Example:

$ go run gopl.io/ch1/dup1
hello
world
goodbye
world
^D
2	world
varunu28

comment created time in 17 days

issue commentbazelbuild/bazel

rule_implementation_hash is unsound when a rule definition site is a macro in b.bzl, but that macro is invoked from a.bzl, and that invocation passes along e.g. an implementation function object

What is the use for that definition?

Good question; I'm struggling to page back into memory our length conversations about this, which unfortunately left no residue in the code. A superficial observation: the offending CL at least made the treatment of the label portion consistent with the digest portion. Before, the label came from the innermost and the digest from the outermost .bzl file on the call stack.

At least a part of the conversation was about the infeasibility of deeply hashing starlark function values, and the assumption that Starlark will ~soon support nested def statements with lexical scope, aka closures, which means that a rule class created by a function in Q.bzl may be a closure over values supplied by a function in P.bzl, where P loads Q. This seems to argue for the RDE of a RuleClass being a property of the label + digest of outermost frame, which is the opposite of what the change did. Alex, do you remember more?

every rule in the universe could have the same hash if they all were defined through the make_a in haxorz 's example.

If the purpose of the hash is to detect changes in the logic of a rule, then that's not necessarily wrong, as all the rules will have the same load graph of bzl sources.

linzhp

comment created time in 17 days

issue commentbazelbuild/bazel

rule_implementation_hash is unsound when a rule definition site is a macro in b.bzl, but that macro is invoked from a.bzl, and that invocation passes along e.g. an implementation function object

The rule definition environment (RDE---a concept that desperately needs an explanatory doc comment somewhere, not least because the term is overloaded as the name of an unrelated Java class) is based on the hash of the source of the .bzl file containing the immediate call to rule(), plus all bzl files it transitively loads. It is not based on the .bzl file that is being initialized. In other words, it's the innermost frame on the call stack, not the outermost. This was discussed at length in the context of cr/311173293 and the sequence of changes of which it was a part; Alex Jurkowski @alexjski has the freshest knowledge here.

BTW, Nathan: head dir/* is your friend for listing files preceded by their file name. (Add -n 999 if any file is long.)

linzhp

comment created time in 17 days

issue commentbazelbuild/bazel

Misleading error for target collisions

Glob can be used in two ways. Originally it could only be used as a shorthand for a known list of files that could in principle be typed out, even though it may be tedious and redundant to do so. More recently it has evolved another use, to match an open-ended set of files that is not statically knowable. This occurs in Bazel when a glob is applied to the files fetched by a repo rule, but it also occurs in Blaze (which has no concept of repo) when a macro in a .bzl file executes a glob on behalf of an unknown client package.

This demands good hygiene: the macro should make clear which portions of the file/target namespace it will inspect or 0define. A similar problem exists, without glob, when a macro makes decisions based on the set of rules that already exist. And a reverse variant exists when two macros use the same names for their auxiliary rules due to insufficient disambiguation. I don't think glob is the root of the problem. Such problems are at least diagnosable using blaze query.

shmish111

comment created time in 19 days

issue commentbazelbuild/bazel

Misleading error for target collisions

I agree with Klaus's analysis, but there is no way we can change glob. It is defined to return a list of strings, and it would be a profoundly surprising and incompatible change for it to return anything else. Also, glob is crucial to performance (and still needs further optimization), so we couldn't afford to change its result type so something that (say) wraps a string. In short, glob returns strings, and attr.label_list accepts strings, and there can be no out-of-band communication between them. I think fundamentally this is working as intended.

That said, we could improve the error. For example, we could observe that the cyclic path occurred in a package that uses globs, and use some heuristic to decide that the strings returned by glob are similar to the names of targets in the package, and suggest to the user that the glob was ultimately used as a set of labels (as is usual), but the heuristics may be fragile. That seems ok if they're only triggered in a fairly uncommon error situation. The implementation could be tricky.

(As Janak points out, a glob may return a directory entry such as @foo, which, if used as a label, would be misinterpreted as repository, so glob contains a hack to replace them with ./@foo. In hindsight, we should have used the syntax //@repo/dir/name for labels to avoid this problem, and keep only / and : as the only two reserved characters. Colons are exceptionally rare in file names because they have had special meaning on Windows, classic Mac, and POSIX for decades.)

shmish111

comment created time in 19 days

pull request commentbazelbuild/bazel

Expose documented function return values in stardoc

The code change seems fine to me, but @c-parsons is the expert.

segiddins

comment created time in 25 days

issue commentbazelbuild/bazel

repository_ctx.template corrupts binary files

As I commented at https://github.com/bazelbuild/bazel/issues/11858#issuecomment-686480642, yes, template will indeed not work for binary files. Strings, in the Java implementation of Starlark used by Bazel, are intended to be strings of UTF-16 codes (chars), like strings in Java, and are thus incapable of representing binary files. A copy function seems like a better approach.

guibou

comment created time in 25 days

issue commentbazelbuild/bazel

feature request: repository_ctx.copy

The Java implementation of Starlark has no datatype capable of holding byte strings, so template will indeed not work for binary file copies. (Technically strings can hold bytes, but only because of Bazel's regrettable use of Latin-1 to interpret file names and file contents, which means that bytes in file names and BUILD/.bzl files are internally represented as the chars of a Java string. This is a deep and pervasive bug that requires a deep and pervasive fix.)

A copy function seems reasonable, but I have very little experience with the repository_rule API; someone more knowledgeable should review it. Tony? @aiuto

guibou

comment created time in 25 days

issue commentthought-machine/please

Differences with Starlark

Also: there's an easily embeddable and efficient Go implementation of Starlark at go.starlark.net that you might want to re-use.

laurentlb

comment created time in a month

Pull request review commentbazelbuild/bazel-blog

3.5 release notes

+---+layout: posts+title: "Bazel 3.5"+authors:+  - aiuto+---++[Bazel 3.5](https://github.com/bazelbuild/bazel/releases/tag/3.5.0) has just been released.++Bazel 3.5 is a minor release. It is an incremental update to Bazel 3.4.++*Baseline*: 889bc0b523b47eeb38a72bf9bb6858ee525a7c7e++*Cherry picks*:+-   a7a0d48fbeb059ee60e77580e5d05baeefdd5699: Make no-op starlark transition not affect the output directory+-   b37c51c7085f0aefe04034dd451acb847605ddb5: Add `include_prefix` and `strip_include_prefix` to `cc_common.compile`+-   f6ad35fcde93f92c591778ed7db38d167f5bbc03: Delete `--experimental_transparent_compression`+-   39bc97eab295bddb35b38bfc4a2ff3d2b15d034e: Remove `--experimental_action_args`+-   b9706675a7abf6ceebb250f0b3dfa4087a0c35f6: Stop needlessly parsing WORKSPACE files from external repositories+-   e574d558da17cfd0f818e7a937a07926aa270069: Allow hyphen char in `workspace` names+-   9993785fa0c4fa4172aa31d306f3abea76833abf: Allow dot ('.') in `workspace` names++## Incompatible changes++- Remove the `--experimental_process_wrapper_wait_fix` flag.+- Remove the `--experimental_ui_deduplicate` flag.+- Remove the `--experimental_transparent_compression` flag.+- Remove the `--experimental_action_args` flag.+- Bazel now correctly prefers Xcode versions in `/Applications` over+  Other paths, by default.  This resolves an issue with accidentally+  picking up an Xcode version from a Time Machine backup or network+  disk.  If you rely on the old behavior, move your desired Xcode+  version to `/Applications`.++## New features:++- `cquery` now follows aspects when the `--include_aspects` flag is used.+- Add support to `bazel/crosstool` for building arm64 on macos.++## Important changes:++- Fix resource shrinking when \<overlayable\> tags are used.+- Remove legacy handling of `--extra_checks`+- Remove old incompatible flag `--incompatible_symlinked_sandbox_expands_tree_artifacts_in_runfiles_tree`.+- Update coverage configuration for Python, filegroup, and shell script+  rules to distinguish between source and dependency attributes.++- Add `InstrumentedFilesInfo` provider to Starlark globals.+- Make filegroup always forward `InstrumentedFilesProvider` and not+  collect any sources directly.+- *Caution*: Update Starlark error reporting and the call stack,+   which may cause regressions, such as errors that lack relevant+   location information.

If we update the reporting so we get better call stack, error locations will need to change. If people are depending on the exact error stack, their test is brittle.

By the way, the analysis test mechanism (https://docs.bazel.build/versions/master/skylark/testing.html#failure-testing) positively invites users to write fragile tests of analysis failures, that depend on the precise wording of the Bazel error message du jour.

aiuto

comment created time in a month

PullRequestReviewEvent

Pull request review commentbazelbuild/bazel-blog

3.5 release notes

+---+layout: posts+title: "Bazel 3.5"+authors:+  - aiuto+---++[Bazel 3.5](https://github.com/bazelbuild/bazel/releases/tag/3.5.0) has just been released.++Bazel 3.5 is a minor release. It is an incremental update to Bazel 3.4.++*Baseline*: 889bc0b523b47eeb38a72bf9bb6858ee525a7c7e++*Cherry picks*:+-   a7a0d48fbeb059ee60e77580e5d05baeefdd5699: Make no-op starlark transition not affect the output directory+-   b37c51c7085f0aefe04034dd451acb847605ddb5: Add `include_prefix` and `strip_include_prefix` to `cc_common.compile`+-   f6ad35fcde93f92c591778ed7db38d167f5bbc03: Delete `--experimental_transparent_compression`+-   39bc97eab295bddb35b38bfc4a2ff3d2b15d034e: Remove `--experimental_action_args`+-   b9706675a7abf6ceebb250f0b3dfa4087a0c35f6: Stop needlessly parsing WORKSPACE files from external repositories+-   e574d558da17cfd0f818e7a937a07926aa270069: Allow hyphen char in `workspace` names+-   9993785fa0c4fa4172aa31d306f3abea76833abf: Allow dot ('.') in `workspace` names++## Incompatible changes++- Remove the `--experimental_process_wrapper_wait_fix` flag.+- Remove the `--experimental_ui_deduplicate` flag.+- Remove the `--experimental_transparent_compression` flag.+- Remove the `--experimental_action_args` flag.+- Bazel now correctly prefers Xcode versions in `/Applications` over+  Other paths, by default.  This resolves an issue with accidentally+  picking up an Xcode version from a Time Machine backup or network+  disk.  If you rely on the old behavior, move your desired Xcode+  version to `/Applications`.++## New features:++- `cquery` now follows aspects when the `--include_aspects` flag is used.+- Add support to `bazel/crosstool` for building arm64 on macos.++## Important changes:++- Fix resource shrinking when \<overlayable\> tags are used.+- Remove legacy handling of `--extra_checks`+- Remove old incompatible flag `--incompatible_symlinked_sandbox_expands_tree_artifacts_in_runfiles_tree`.+- Update coverage configuration for Python, filegroup, and shell script+  rules to distinguish between source and dependency attributes.++- Add `InstrumentedFilesInfo` provider to Starlark globals.+- Make filegroup always forward `InstrumentedFilesProvider` and not+  collect any sources directly.+- *Caution*: Update Starlark error reporting and the call stack,+   which may cause regressions, such as errors that lack relevant+   location information.

Sorry I didn't see this till now. When I wrote the CL I was concerned that there would be places in which error location information had gone missing, but there seem to have been no adverse effects reported on Blaze, and in many ways the errors are much improved. I concur that it is safe to strike the warning.

aiuto

comment created time in a month

PullRequestReviewEvent

delete branch google/starlark-go

delete branch : spec-resolve

delete time in a month

push eventgoogle/starlark-go

alandonovan

commit sha 6e684ef5eeee284749e13eae206cb4dd899b3472

spec: clarify resolution of undefined names (#303)

view details

push time in a month

PR merged google/starlark-go

Reviewers
spec: clarify resolution of undefined names
+18 -2

1 comment

1 changed file

alandonovan

pr closed time in a month

push eventgoogle/starlark-go

alandonovan

commit sha a2dfe113613cc01d20ccee815bc366ac1d8bbd86

spec: clarify resolution of undefined names Change-Id: I1b1942f1fb5fffe8d1a65bc74dab53f86123d4f6

view details

push time in a month

pull request commentgoogle/starlark-go

spec: clarify resolution of undefined names

@laurentlb

alandonovan

comment created time in a month

PR opened google/starlark-go

spec: clarify resolution of undefined names
+17 -2

0 comment

1 changed file

pr created time in a month

create barnchgoogle/starlark-go

branch : spec-resolve

created branch time in a month

issue commentgolang/go

cmd/vet: warn when return value of context.WithValue is lost

Add context.WithValue to this list? https://github.com/golang/tools/blob/master/go/analysis/passes/unusedresult/unusedresult.go#L47

ancarda

comment created time in a month

issue commentgoogle/starlark-go

support first-class modules in load syntax

I don't see wildcard imports as bad for small applications comparing to big projects with hundreds of files.

Every big project with hundreds of files starts as a small application.

What do you think about allowing at most one wildcard import in order to simplify reasoning and implementation?

I still don't like. You would have to pay all the complexity of the feature but then place an artificial limit on it.

More importantly, wildcard imports are completely incompatible with the Python compilation model. They are feasible to implement in a compiled language like Java or Go because the names defined by the imported module are enumerable at compilation time. By contrast, in a Python-like language, compilation occurs one file at a time, before the module becomes available during execution, so you simply don't know what names would be bound by each import statement. This means it's impossible to tell during compilation whether a reference is defined by an import, predeclared, or undefined.

fkorotkov

comment created time in a month

issue commentgoogle/starlark-go

Wildcard load statement

Wildcard loads, in any language, are a bad idea because they make it hard to know where a name is defined. But there is a cleaner way to solve the same problem: first-class modules. In Python, you can say from os import mkdir, analogous to a Starlark load, or you can say import os and later refer to os.mkdir. Indeed, the second form is the more usual one in Python. In Bazel, many .bzl files construct an explicit struct to act as a module, like so (sticking with the os analogy):

def mkdir_(...): ...
def rmdir_(...): ...

os = struct(
   mkdir=mkdir_,
   rmdir=rmdir_,
)

If Starlark had first-class modules, then a load statement would be able to import the entire module as a value. The syntax would presumably be something like:

  load("os.bzl") # binds 'os' to the "os.bzl" module

or

 load(alt="os.bzl") binds alternative name 'alt' to the "os.bzl" module

I think this would be a worthwhile language change.

fkorotkov

comment created time in a month

PullRequestReviewEvent

pull request commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

same message

Would you mind reporting a Bazel bug about the lack of error detail in this failure? Thanks.

On Fri, 28 Aug 2020 at 10:51, ehkloaj notifications@github.com wrote:

@ehkloaj commented on this pull request.

In src/main/java/com/google/devtools/common/options/Converters.java https://github.com/bazelbuild/bazel/pull/11900#discussion_r479356602:

@@ -625,4 +632,71 @@ public String getTypeDescription() { return "Converts to a CacheBuilderSpec, or null if the input is empty"; } }

  • public static class BooleanFormulaConverter implements Converter<Expression> {

Well, that is a truly terrible error message: it has suppressed the part the really matters. (Does running with --sandbox_debug show you a Java stack trace?)

No, same message.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/pull/11900#discussion_r479356602, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAK35GFCPUPPG4F5CRG5EDSC7AARANCNFSM4PWUJLGQ .

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

         + "then they can analyze test_suite targets."   )   public boolean expandTestSuites;++  public static class BooleanFormulaConverter implements Converter<Expression> {+    @Override+    public Expression convert(String input) throws OptionsParsingException {+      if (input.isEmpty()) {+        return null;+      }+      if (isLegacySyntax(input)) {+        input = convertLegacySyntaxToBooleanFormula(input);+        GoogleLogger.forEnclosingClass().at(Level.WARNING).log(+            "Specifying a list of tags separated by comma is legacy syntax and will not be supported after" ++                " Jan 1, 2021. Consider changing to: " + input);+      }+      try {+        Expression result = Expression.parse(ParserInput.fromLines(input));+        validate(result);+        return result;+      } catch (SyntaxError.Exception e) {+        throw new OptionsParsingException("Failed to parse expression: " + e.getMessage() + " input: " + input, e);+      }+    }++    @Override+    public String getTypeDescription() {+      return "A Boolean formula over test tags (e.g. '(release or smoke) and not flaky')";+    }++    private static boolean isLegacySyntax(String input) {+      return input.contains(",") || input.contains("-");+    }++    /**+     * Converts a string containing tags separated by comma to a boolean formula.+     */+    private String convertLegacySyntaxToBooleanFormula(String input) throws OptionsParsingException {+      return input.replaceAll(" *, *", " or ").replace("-", " not ").trim();

I see. I don't think it matters. I'll leave it up to you.

(It still boggles my mind that Java's Strings depend on regular expression---and in the innocuously named 'replaceAll', no less.)

ehkloaj

comment created time in a month

PullRequestReviewEvent

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Maps;+import com.google.devtools.build.lib.syntax.BinaryOperatorExpression;

(Move this.)

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 java_library(         ],     ),     deps = [+        "//src/main/java/com/google/devtools/build/lib/syntax:frontend",

Revert this.

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public boolean equals(Object obj) {           && other.compileOneDependency == compileOneDependency           && other.buildTestsOnly == buildTestsOnly           && other.determineTests == determineTests-          && other.buildTargetFilter.equals(buildTargetFilter)+          && (other.buildTargetFilter == null && buildTargetFilter == null

Objects.equal

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public boolean getDetermineTests() {       return determineTests;     } -    public ImmutableList<String> getBuildTargetFilter() {+    public Expression getBuildTargetFilter() {

@Nullable public

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 java_test(     ),     deps = [         ":testutils",+        "//src/main/java/com/google/devtools/build/lib/syntax:frontend",

(Revert this.)

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public boolean equals(Object o) {     TestFilter f = (TestFilter) o;     return f.testSizeFilterSet.equals(testSizeFilterSet)         && f.testTimeoutFilterSet.equals(testTimeoutFilterSet)-        && f.testTagFilterList.equals(testTagFilterList)+        && (f.testTagFilter == null && testTagFilter == null

Objects.equal(f.testTagFilter, testTagFilter)

(it handles null)

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public static SkyKey key(       boolean compileOneDependency,       boolean buildTestsOnly,       boolean determineTests,-      ImmutableList<String> buildTargetFilter,+      Expression buildTargetFilter,

@Nullable Expression

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public static TestFilter forOptions(     return new TestFilter(         ImmutableSet.copyOf(options.testSizeFilterSet),         ImmutableSet.copyOf(options.testTimeoutFilterSet),-        ImmutableList.copyOf(options.testTagFilterList),+        options.testTagFilter,         ImmutableList.copyOf(options.testLangFilterList));   }    private final ImmutableSet<TestSize> testSizeFilterSet;   private final ImmutableSet<TestTimeout> testTimeoutFilterSet;-  private final ImmutableList<String> testTagFilterList;+  private final Expression testTagFilter;   private final ImmutableList<String> testLangFilterList;   private final Predicate<Target> impl;    @AutoCodec.VisibleForSerialization   TestFilter(       ImmutableSet<TestSize> testSizeFilterSet,       ImmutableSet<TestTimeout> testTimeoutFilterSet,-      ImmutableList<String> testTagFilterList,+      Expression testTagFilter,

@Nullable public

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

         + "then they can analyze test_suite targets."   )   public boolean expandTestSuites;++  public static class BooleanFormulaConverter implements Converter<Expression> {+    @Override+    public Expression convert(String input) throws OptionsParsingException {+      if (input.isEmpty()) {+        return null;+      }+      if (isLegacySyntax(input)) {+        input = convertLegacySyntaxToBooleanFormula(input);+        GoogleLogger.forEnclosingClass().at(Level.WARNING).log(+            "Specifying a list of tags separated by comma is legacy syntax and will not be supported after" ++                " Jan 1, 2021. Consider changing to: " + input);+      }+      try {+        Expression result = Expression.parse(ParserInput.fromLines(input));+        validate(result);+        return result;+      } catch (SyntaxError.Exception e) {+        throw new OptionsParsingException("Failed to parse expression: " + e.getMessage() + " input: " + input, e);+      }+    }++    @Override+    public String getTypeDescription() {+      return "A Boolean formula over test tags (e.g. '(release or smoke) and not flaky')";+    }++    private static boolean isLegacySyntax(String input) {+      return input.contains(",") || input.contains("-");+    }++    /**+     * Converts a string containing tags separated by comma to a boolean formula.+     */+    private String convertLegacySyntaxToBooleanFormula(String input) throws OptionsParsingException {+      return input.replaceAll(" *, *", " or ").replace("-", " not ").trim();

There's no need to use a regular expression here, nor remove whitespace; the parser will do that.

return input.replace(",", " or ").replace("-", " not ");

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public static SkyKey keyWithoutFilters(ImmutableList<String> targetPatterns, Str     private final boolean compileOneDependency;     private final boolean buildTestsOnly;     private final boolean determineTests;-    private final ImmutableList<String> buildTargetFilter;+    private final Expression buildTargetFilter;

@Nullable private

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public static TestFilter forOptions(     return new TestFilter(         ImmutableSet.copyOf(options.testSizeFilterSet),         ImmutableSet.copyOf(options.testTimeoutFilterSet),-        ImmutableList.copyOf(options.testTagFilterList),+        options.testTagFilter,         ImmutableList.copyOf(options.testLangFilterList));   }    private final ImmutableSet<TestSize> testSizeFilterSet;   private final ImmutableSet<TestTimeout> testTimeoutFilterSet;-  private final ImmutableList<String> testTagFilterList;+  private final Expression testTagFilter;

@Nullable private

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

         + "then they can analyze test_suite targets."   )   public boolean expandTestSuites;++  public static class BooleanFormulaConverter implements Converter<Expression> {+    @Override+    public Expression convert(String input) throws OptionsParsingException {

@Nullable public

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

    @Option(     name = "build_tag_filters",-    converter = CommaSeparatedOptionListConverter.class,+    converter = BooleanFormulaConverter.class,     defaultValue = "",     documentationCategory = OptionDocumentationCategory.UNCATEGORIZED,     effectTags = {OptionEffectTag.UNKNOWN},     help =-        "Specifies a comma-separated list of tags. Each tag can be optionally "-            + "preceded with '-' to specify excluded tags. Only those targets will be built that "-            + "contain at least one included tag and do not contain any excluded tags. This option "-            + "does not affect the set of tests executed with the 'test' command; those are be "-            + "governed by the test filtering options, for example '--test_tag_filters'"+        "See '--test_tag_filters' for instructions. "+            + "This option does not affect the set of tests executed with the 'test' command; "+            + "those are to be governed by the test filtering options, for example '--test_tag_filters'"   )-  public List<String> buildTagFilterList;+  public Expression buildTagFilter;    @Option(     name = "test_tag_filters",-    converter = CommaSeparatedOptionListConverter.class,+    converter = BooleanFormulaConverter.class,     defaultValue = "",     documentationCategory = OptionDocumentationCategory.UNCATEGORIZED,     effectTags = {OptionEffectTag.UNKNOWN},     help =-        "Specifies a comma-separated list of test tags. Each tag can be optionally "-            + "preceded with '-' to specify excluded tags. Only those test targets will be "-            + "found that contain at least one included tag and do not contain any excluded "-            + "tags. This option affects --build_tests_only behavior and the test command."+        "Specifies a boolean formula that follows the syntax: "

""" A Boolean formula over test tags that selects the tests to run. Example: '(release or smoke) and not flaky'.

Formulas have the form: expr = IDENT | expr 'or' expr | expr 'and' expr | 'not' expr | '(' expr ')'. IDENT is defined as a Starlark identifier, with the same reserved words.

The legacy syntax, a comma-separated list of tags, implicitly joined as if by 'or', each optionally preceded by '-' for negation, will be removed after Jan 1, 2021. """

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

    @Option(     name = "build_tag_filters",-    converter = CommaSeparatedOptionListConverter.class,+    converter = BooleanFormulaConverter.class,     defaultValue = "",     documentationCategory = OptionDocumentationCategory.UNCATEGORIZED,     effectTags = {OptionEffectTag.UNKNOWN},     help =-        "Specifies a comma-separated list of tags. Each tag can be optionally "-            + "preceded with '-' to specify excluded tags. Only those targets will be built that "-            + "contain at least one included tag and do not contain any excluded tags. This option "-            + "does not affect the set of tests executed with the 'test' command; those are be "-            + "governed by the test filtering options, for example '--test_tag_filters'"+        "See '--test_tag_filters' for instructions. "

"A Boolean formula over rule tags that selects the targets to build. Example: '(server or client) and not experimental'. See --test_tag_filters for details."

ehkloaj

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

The integration test of the new feature can go in third_party/bazel/src/test/shell/bazel/bazel_test_test.sh. All the tests have the form: write some files, run a blaze command, make assertions about the exit code/logs/file system.

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

         + "then they can analyze test_suite targets."   )   public boolean expandTestSuites;++  public static class BooleanFormulaConverter implements Converter<Expression> {+    @Override+    public Expression convert(String input) throws OptionsParsingException {+      if (input.isEmpty()) {+        return null;+      }+      if (isLegacySyntax(input)) {+        input = convertLegacySyntaxToBooleanFormula(input);+        GoogleLogger.forEnclosingClass().at(Level.WARNING).log(

Just use System.err.println.

ehkloaj

comment created time in a month

PullRequestReviewEvent

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public String getTypeDescription() {       return "Converts to a CacheBuilderSpec, or null if the input is empty";     }   }++  public static class BooleanFormulaConverter implements Converter<Expression> {

Well, that is a truly terrible error message: it has suppressed the part the really matters. (Does running with --sandbox_debug show you a Java stack trace?)

I tried it locally and it appears that the class must be declared public for it to be accessible to reflection; the "bazel help everything-as-html" command that failed above uses reflection to print all the flags.

So, please keep the class inside LoadingOptions, but declare it public, and precede it by a comment such as this:

/** BooleanFormulaConverter parses a boolean formula to an Expression. Declared public to make it accessible to reflection. */

ehkloaj

comment created time in a month

PullRequestReviewEvent

pull request commentbazelbuild/stardoc

Correct formatting when no parameters.

@c-parsons

thomasvl

comment created time in a month

issue commentbazelbuild/stardoc

Stardoc doesn't work with Bazel head

On Thu, 27 Aug 2020 at 04:54, larsrc-google notifications@github.com wrote:

I'm new on the Bazel team, and there's still so much to learn.

That never goes away, I'm afraid.

achew22

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public String getTypeDescription() {       return "Converts to a CacheBuilderSpec, or null if the input is empty";     }   }++  public static class BooleanFormulaConverter implements Converter<Expression> {+    @Override+    public Expression convert(String input) throws OptionsParsingException {+      if (input.isEmpty()) {+        return Expression.empty();+      }+      if (isLegacySyntax(input)) {+        input = convertLegacySyntaxToBooleanFormula(input);+      }+      try {+        Expression result = Expression.parse(ParserInput.fromLines(input));+        validate(result);+        return result;+      } catch (SyntaxError.Exception e) {+        throw new OptionsParsingException("Failed to parse expression: " + e.getMessage() + " input: " + input, e);+      }+    }++    @Override+    public String getTypeDescription() {+      return "A Boolean formula over test tags (e.g. '(release or smoke) and not flaky')";+    }++    private static boolean isLegacySyntax(String input) {+      return input.contains(",") || input.contains("-");+    }++    /**+     * Converts a string containing tags separated by comma to a boolean expression.+     */+    private String convertLegacySyntaxToBooleanFormula(String input) throws OptionsParsingException {+      return input.replaceAll(" *, *", " or ").replace("-", " not ").trim();+    }++    /**+     * Throws an OptionsParsingException when the given boolean formula does not follow the grammar:+     * expr = IDENT | expr 'or' expr | expr 'and' expr | 'not' expr | '(' expr ')'+     */+    private void validate(Expression expression) throws OptionsParsingException {+      switch (expression.kind()) {+        case IDENTIFIER:+          break;+        case BINARY_OPERATOR:+          BinaryOperatorExpression boe = (BinaryOperatorExpression) expression;+          if (boe.getOperator() == TokenKind.OR || boe.getOperator() == TokenKind.AND) {

Tip: the control structures are slightly easier to read if you reject the error first:

   if (op != OR && op != AND) {
            throw ...
    }
    validate(X)
    validate(Y)
    break

Ditto for the unary case.

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public String getTypeDescription() {       return "Converts to a CacheBuilderSpec, or null if the input is empty";     }   }++  public static class BooleanFormulaConverter implements Converter<Expression> {

(I don't know why GitHub's code review tool won't let me continue the previous discussion. Is it because the relevant declaration moved? In any case, not helpful.)

You wrote:

Making this class private will not work since the methods convert and getTypeDescription must be accessible outside of LoadingOptions. Do you still want it to be moved but stay public or is it better if it stays in Converters?

The class can be private, but its methods must be public because they override public methods defined in a superclass. This is quite normal practice in Java.

Please do move it out of Converters, for the reason I gave previously.

ehkloaj

comment created time in a month

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

+package com.google.devtools.build.lib.syntax;++/** An expression without content. */

We absolutely cannot add a new expression type to the language for this: it would be adding complexity to a widely used lower-level package just to support a single esoteric use in a higher-level package, and that is nearly always a mistake. Every existing client of the lower-level package would be immediately broken because their switch(stmt.kind()) statements would become non-exhaustive. So they would have add a case for EmptyExpression, but what should it do? What does empty expression even mean? Where does it come from? Why isn't it mentioned in the language spec? And so on.

(One could avoid null by using some trivially true expression such as "x or not x", but that's still a hack, so let's not do that.)

The obvious solution is simply to remove the non-null precondition check, as it no longer applies. Make clear that this is intentional by annotating the relevant parameters, fields, and accessor methods with @Nullable. Follow the flow of values as far as it goes so that we don't pass null to a function that requires non-null. This particular value doesn't flow very far.

ehkloaj

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentbazelbuild/bazel

Use boolean formulas to express filters in --build_tag_filters and --test_tag_filters

 public String getTypeDescription() {       return "Converts to a CacheBuilderSpec, or null if the input is empty";     }   }++  public static class ExpressionConverter implements Converter<Expression> {

Do you mean that ExpressionConverter should be moved to LoadingOptions?

Yes, exactly.

The previous solution used CommaSeparatedOptionListConverter which is also a part of Converters.java so I do not see why this new converter would not belong there as well.

The reason is that Converters is for commonly used converters, but this one is very esoteric and has only a single user (and is likely to have only a single user), so the appropriate place for the complexity is adds is with that user, not with Converters.

ehkloaj

comment created time in a month

PullRequestReviewEvent

issue commentbazelbuild/starlark

spec: clarify comprehension scope issues

The remaining task is to copy the changes from https://github.com/google/starlark-go/pull/299 into this repo.

alandonovan

comment created time in a month

issue closedgoogle/starlark-go

Is there any way to stop a running script with a timeout?

In my use case, I receive a function from the user. And I passthrough params and execute it. But I need a way to stop the function running with a timeout (e.g. 2000ms).

can starlark-go provide a function like CallContext?

thread := &starlark.Thread{
    Name:  "example",
}

globals, _ := starlark.ExecFile(thread, "test.star", f, nil)

mainFn := globals["main"]
dict := starlark.StringDict{
    "param": starlark.String("test"),
}

m := starlarkstruct.FromStringDict(starlarkstruct.Default, dict)
arg := starlark.Tuple{m}

ctx, cancel := context.WithTimeout(context.background(), time.Second)
defer cancel()
starlark.CallContext(ctx, thread, mainFn, arg, nil)

closed time in a month

Dreamacro

issue commentgoogle/starlark-go

Is there any way to stop a running script with a timeout?

Fixed. See Thread.SetMaxExecutionSteps.

Dreamacro

comment created time in a month

issue closedgoogle/starlark-go

Question: Prevent long-running scripts

Hi there. Is it possible to constrain execution time and resources? I am interested in embedding starlark in a server application that processes user-defined scripts and was interested if there were steps that could be take to mitigate long-running or malicious code.

For example to prevent something like this:

def loop():
  x = '';
  for i in range(1, 100000000):
    x = x + 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do ...'

loop()

closed time in a month

bradrydzewski

issue commentgoogle/starlark-go

Question: Prevent long-running scripts

Fixed. See Thread.SetMaxExecutionSteps.

bradrydzewski

comment created time in a month

delete branch google/starlark-go

delete branch : steps-and-cancel

delete time in a month

push eventgoogle/starlark-go

alandonovan

commit sha 949cc6f4b09716061bc5c308049b0da811aeca10

starlark: support thread cancellation and limits on computation (#298) This change adds two related features: (1) asynchronous cancellation of Starlark threads (Thread.Cancel), and (2) recording and limiting of the number of abstract computation steps. Fixes issue #252 Fixes issue #160

view details

push time in a month

PR merged google/starlark-go

starlark: support thread cancellation and limits on computation

This change adds two related features: (1) asynchronous cancellation of Starlark threads (Thread.Cancel), and (2) recording and limiting of the number of abstract computation steps.

Fixes issue #252 Fixes issue #160

+120 -0

0 comment

3 changed files

alandonovan

pr closed time in a month

push eventgoogle/starlark-go

alandonovan

commit sha a121eb0ff8a7b6644240d139113ab0d29ac7e12a

starlark: support thread cancellation and limits on computation This change adds two related features: (1) asynchronous cancellation of Starlark threads (Thread.Cancel), and (2) recording and limiting of the number of abstract computation steps. Fixes issue #252 Fixes issue #160 Change-Id: Ia2e594d742363f18e650a04a0e3745678182f0e7

view details

push time in a month

Pull request review commentgoogle/starlark-go

starlark: support thread cancellation and limits on computation

 type Thread struct { 	// See example_test.go for some example implementations of Load. 	Load func(thread *Thread, module string) (StringDict, error) +	// Steps is a count of abstract computation steps executed by+	// this thread. It is incremented by the interpreter. If after+	// being incremented, its value is zero, the interpreter calls+	// thread.Cancel("too many steps").+	//+	// Steps may be used as a measure the approximate cost of+	// Starlark execution, by computing the difference in its value+	// before and after a computation. It may also be used to limit+	// a computation to k steps by setting its initial value to+	// 1<<64 - k.+	Steps uint64

Fair point about encapsulation; I've changed the API to a pair of functions ExecutionSteps and SetMaxExecutionSteps. (No set or reset operation is necessary.) The check must now do an extra load of 'maxSteps', but it's in the same cache line as 'steps'.

alandonovan

comment created time in a month

Pull request review commentgoogle/starlark-go

starlark: support thread cancellation and limits on computation

 type Thread struct { 	// See example_test.go for some example implementations of Load. 	Load func(thread *Thread, module string) (StringDict, error) +	// Steps is a count of abstract computation steps executed by+	// this thread. It is incremented by the interpreter. If after+	// being incremented, its value is zero, the interpreter calls+	// thread.Cancel("too many steps").+	//+	// Steps may be used as a measure the approximate cost of+	// Starlark execution, by computing the difference in its value+	// before and after a computation. It may also be used to limit+	// a computation to k steps by setting its initial value to+	// 1<<64 - k.

Done.

alandonovan

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogle/starlark-go

starlark: support thread cancellation and limits on computation

 func (fn *Function) CallInternal(thread *Thread, args Tuple, kwargs []Tuple) (Va 	code := f.Code loop: 	for {+		thread.Steps+++		if thread.Steps == 0 {+			thread.Cancel("too many steps")+		}+		if reason := atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&thread.cancelReason))); reason != nil {

I measured it on x86 and it wasn't noticeable (sorry I didn't keep the data). The address is always in L1 cache, the branch is extremely predictable, and on Intel at least, every load is an atomic load. (The atomic load is much cheaper than selecting on the Done channel of a context: even though fundamentally that too boils down to an atomic load, compare, and branch, the function call overheads are significant.)

I considered using a local variable but it complicates the logic, and I suspect it could increase register pressure in the rest of the bytecode interpreter.

alandonovan

comment created time in a month

PullRequestReviewEvent
more